2024-03-12T05:27:04.870 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-12T05:27:04.871 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} --archive /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400 -- /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400/orig.config.yaml 2024-03-12T05:27:04.929 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T05:27:05.033 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591400' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 94a539707733325322964c6a187c6a1b41be1daf subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b6a4997be2b8bb74411268cc42753f29ae9a78ad owner: scheduled_teuthology@teuthology priority: 250 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8179 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 297/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b6a4997be2b8bb74411268cc42753f29ae9a78ad syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi076.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBqka8M8jzDWF3C2lqo7awi+U11UcjBC8Zs7qH3YtWdp+1zsFMhetUtBoJIewgFOT9yQ6NU4TTrYHmpoQYmfUMg= smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpcbTekqurm/P/LdvPVYPk/tK/Ff68jezglLEGdGJKJ+GszihZUSUNetFE2IBnaSh5KjxJMRvGf7TC69MVdLIQ= smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMT5r1Iy6rqvFFcdWQB5pSnaz720cyku351a7vLpZdiM2w8akjnZYm6UE5UWSAv9nZOOznlQsnPrCEu85hCYyhI= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_14:02:35 tube: smithi user: teuthology verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-12T05:27:05.034 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa; will attempt to use it 2024-03-12T05:27:05.034 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks 2024-03-12T05:27:05.035 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-12T05:27:05.036 INFO:teuthology.task.internal:Checking packages... 2024-03-12T05:27:05.059 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-12T05:27:05.060 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-12T05:27:05.060 INFO:teuthology.packaging:ref: None 2024-03-12T05:27:05.060 INFO:teuthology.packaging:tag: None 2024-03-12T05:27:05.060 INFO:teuthology.packaging:branch: squid 2024-03-12T05:27:05.060 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T05:27:05.060 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-12T05:27:05.213 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-12T05:27:05.214 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-12T05:27:05.221 INFO:teuthology.task.internal:no buildpackages task found 2024-03-12T05:27:05.221 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-12T05:27:05.229 INFO:teuthology.task.internal:Saving configuration 2024-03-12T05:27:05.251 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-12T05:27:05.256 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T05:27:05.284 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 05:21:34.347497', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBqka8M8jzDWF3C2lqo7awi+U11UcjBC8Zs7qH3YtWdp+1zsFMhetUtBoJIewgFOT9yQ6NU4TTrYHmpoQYmfUMg='} 2024-03-12T05:27:05.305 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 05:21:34.349469', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpcbTekqurm/P/LdvPVYPk/tK/Ff68jezglLEGdGJKJ+GszihZUSUNetFE2IBnaSh5KjxJMRvGf7TC69MVdLIQ='} 2024-03-12T05:27:05.329 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 05:21:34.351240', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMT5r1Iy6rqvFFcdWQB5pSnaz720cyku351a7vLpZdiM2w8akjnZYm6UE5UWSAv9nZOOznlQsnPrCEu85hCYyhI='} 2024-03-12T05:27:05.329 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-12T05:27:05.336 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-12T05:27:05.336 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-12T05:27:05.336 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-12T05:27:05.336 INFO:teuthology.run_tasks:Running task console_log... 2024-03-12T05:27:05.462 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f591f49a160>, signals=[15]) 2024-03-12T05:27:05.463 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-12T05:27:05.469 INFO:teuthology.task.internal:Opening connections... 2024-03-12T05:27:05.469 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2024-03-12T05:27:05.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T05:27:05.565 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2024-03-12T05:27:05.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T05:27:05.657 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2024-03-12T05:27:05.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T05:27:05.745 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-12T05:27:05.751 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2024-03-12T05:27:05.769 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2024-03-12T05:27:05.769 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2024-03-12T05:27:05.825 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Stream" 2024-03-12T05:27:05.825 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="9" 2024-03-12T05:27:05.825 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2024-03-12T05:27:05.826 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2024-03-12T05:27:05.826 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="9" 2024-03-12T05:27:05.826 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el9" 2024-03-12T05:27:05.826 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T05:27:05.826 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2024-03-12T05:27:05.826 INFO:teuthology.orchestra.run.smithi076.stdout:LOGO="fedora-logo-icon" 2024-03-12T05:27:05.826 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T05:27:05.826 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://centos.org/" 2024-03-12T05:27:05.826 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T05:27:05.826 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T05:27:05.826 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T05:27:05.827 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2024-03-12T05:27:05.849 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2024-03-12T05:27:05.867 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2024-03-12T05:27:05.868 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2024-03-12T05:27:05.925 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="CentOS Stream" 2024-03-12T05:27:05.925 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="9" 2024-03-12T05:27:05.925 INFO:teuthology.orchestra.run.smithi082.stdout:ID="centos" 2024-03-12T05:27:05.925 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="rhel fedora" 2024-03-12T05:27:05.925 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="9" 2024-03-12T05:27:05.925 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el9" 2024-03-12T05:27:05.925 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T05:27:05.925 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31" 2024-03-12T05:27:05.926 INFO:teuthology.orchestra.run.smithi082.stdout:LOGO="fedora-logo-icon" 2024-03-12T05:27:05.926 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T05:27:05.926 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://centos.org/" 2024-03-12T05:27:05.926 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T05:27:05.926 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T05:27:05.926 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T05:27:05.926 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2024-03-12T05:27:05.950 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2024-03-12T05:27:05.968 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2024-03-12T05:27:05.968 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2024-03-12T05:27:06.025 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream" 2024-03-12T05:27:06.025 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="9" 2024-03-12T05:27:06.025 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2024-03-12T05:27:06.025 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2024-03-12T05:27:06.025 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="9" 2024-03-12T05:27:06.025 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el9" 2024-03-12T05:27:06.025 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T05:27:06.025 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2024-03-12T05:27:06.026 INFO:teuthology.orchestra.run.smithi088.stdout:LOGO="fedora-logo-icon" 2024-03-12T05:27:06.026 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T05:27:06.026 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/" 2024-03-12T05:27:06.026 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T05:27:06.026 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T05:27:06.026 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T05:27:06.026 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2024-03-12T05:27:06.049 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-12T05:27:06.061 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-12T05:27:06.066 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-12T05:27:06.066 DEBUG:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2024-03-12T05:27:06.069 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2024-03-12T05:27:06.072 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2024-03-12T05:27:06.087 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-12T05:27:06.094 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-12T05:27:06.094 DEBUG:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2024-03-12T05:27:06.122 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2024-03-12T05:27:06.127 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2024-03-12T05:27:06.441 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-12T05:27:06.525 INFO:teuthology.run_tasks:Running task kernel... 2024-03-12T05:27:06.540 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-12T05:27:06.541 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-12T05:27:06.541 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-12T05:27:06.541 DEBUG:teuthology.orchestra.run.smithi076:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T05:27:06.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T05:27:06.558 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-03-12T05:27:06.612 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-361.el9.x86_64 2024-03-12T05:27:06.612 INFO:teuthology.task.kernel:Running kernel on smithi076: 5.14.0-361.el9.x86_64 2024-03-12T05:27:06.612 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum install -y kernel 2024-03-12T05:27:08.798 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-12T05:27:10.466 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-12T05:27:12.248 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-03-12T05:27:14.241 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-12T05:27:26.541 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 714 kB/s | 6.1 MB 00:08 2024-03-12T05:27:28.067 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 15 kB 00:00 2024-03-12T05:30:24.285 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 120 kB/s | 21 MB 02:56 2024-03-12T05:30:29.209 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-12T05:30:31.408 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T05:30:31.408 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T05:30:31.502 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-12T05:30:31.507 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-12T05:30:31.507 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2024-03-12T05:30:31.507 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-12T05:30:31.507 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-12T05:30:31.507 INFO:teuthology.orchestra.run.smithi076.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T05:30:31.508 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-12T05:30:31.508 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T05:30:31.508 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T05:30:31.508 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T05:30:31.508 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:30:31.508 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-12T05:30:31.508 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-12T05:30:31.508 INFO:teuthology.orchestra.run.smithi076.stdout:Install 4 Packages 2024-03-12T05:30:31.508 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:30:31.509 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 100 M 2024-03-12T05:30:31.509 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 123 M 2024-03-12T05:30:31.509 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-12T05:30:32.177 INFO:teuthology.orchestra.run.smithi076.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-03-12T05:30:32.761 INFO:teuthology.orchestra.run.smithi076.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 18 MB/s | 21 MB 00:01 2024-03-12T05:30:33.428 INFO:teuthology.orchestra.run.smithi076.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 22 MB/s | 39 MB 00:01 2024-03-12T05:30:33.787 INFO:teuthology.orchestra.run.smithi076.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 21 MB/s | 33 MB 00:01 2024-03-12T05:30:33.787 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-12T05:30:33.787 INFO:teuthology.orchestra.run.smithi076.stdout:Total 44 MB/s | 100 MB 00:02 2024-03-12T05:30:34.673 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-12T05:30:35.054 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-12T05:30:35.055 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-12T05:30:36.666 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-12T05:30:36.666 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-12T05:30:39.639 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-12T05:30:40.306 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T05:30:40.434 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T05:30:43.241 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T05:30:43.433 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T05:30:48.185 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T05:30:48.327 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T05:30:53.088 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T05:31:22.532 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T05:31:22.532 INFO:teuthology.orchestra.run.smithi076.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T05:31:22.533 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:31:22.615 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T05:31:23.168 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T05:31:23.168 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T05:31:23.168 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T05:31:23.168 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T05:31:23.556 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T05:31:23.556 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:31:23.556 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-12T05:31:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T05:31:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T05:31:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T05:31:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T05:31:23.557 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:31:23.557 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-12T05:31:24.068 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T05:31:24.244 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T05:31:24.244 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T05:31:24.245 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T05:31:24.245 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T05:31:24.245 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T05:31:24.245 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-12T05:31:24.245 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi076.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T05:31:24.245 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T05:31:24.365 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T05:31:24.365 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T05:31:24.365 DEBUG:teuthology.orchestra.run.smithi076:> sudo rpm -qi grub2-tools 2024-03-12T05:31:24.399 INFO:teuthology.orchestra.run.smithi076.stdout:Name : grub2-tools 2024-03-12T05:31:24.399 INFO:teuthology.orchestra.run.smithi076.stdout:Epoch : 1 2024-03-12T05:31:24.399 INFO:teuthology.orchestra.run.smithi076.stdout:Version : 2.06 2024-03-12T05:31:24.400 INFO:teuthology.orchestra.run.smithi076.stdout:Release : 68.el9 2024-03-12T05:31:24.400 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: x86_64 2024-03-12T05:31:24.400 INFO:teuthology.orchestra.run.smithi076.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T05:31:24.400 INFO:teuthology.orchestra.run.smithi076.stdout:Group : Unspecified 2024-03-12T05:31:24.400 INFO:teuthology.orchestra.run.smithi076.stdout:Size : 8264254 2024-03-12T05:31:24.400 INFO:teuthology.orchestra.run.smithi076.stdout:License : GPLv3+ 2024-03-12T05:31:24.400 INFO:teuthology.orchestra.run.smithi076.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T05:31:24.400 INFO:teuthology.orchestra.run.smithi076.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T05:31:24.400 INFO:teuthology.orchestra.run.smithi076.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T05:31:24.400 INFO:teuthology.orchestra.run.smithi076.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T05:31:24.400 INFO:teuthology.orchestra.run.smithi076.stdout:Packager : builder@centos.org 2024-03-12T05:31:24.400 INFO:teuthology.orchestra.run.smithi076.stdout:Vendor : CentOS 2024-03-12T05:31:24.400 INFO:teuthology.orchestra.run.smithi076.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T05:31:24.400 INFO:teuthology.orchestra.run.smithi076.stdout:Summary : Support tools for GRUB. 2024-03-12T05:31:24.401 INFO:teuthology.orchestra.run.smithi076.stdout:Description : 2024-03-12T05:31:24.401 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:31:24.401 INFO:teuthology.orchestra.run.smithi076.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T05:31:24.401 INFO:teuthology.orchestra.run.smithi076.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T05:31:24.401 INFO:teuthology.orchestra.run.smithi076.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T05:31:24.401 INFO:teuthology.orchestra.run.smithi076.stdout:hardware devices. 2024-03-12T05:31:24.401 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:31:24.401 INFO:teuthology.orchestra.run.smithi076.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T05:31:24.402 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T05:31:24.402 INFO:teuthology.task.kernel:Updating grub on smithi076 to boot 5.14.0-427.el9.x86_64 2024-03-12T05:31:24.402 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T05:31:24.993 INFO:teuthology.orchestra.run.smithi076.stderr:Generating grub configuration file ... 2024-03-12T05:31:26.407 INFO:teuthology.orchestra.run.smithi076.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T05:31:26.463 INFO:teuthology.orchestra.run.smithi076.stderr:done 2024-03-12T05:31:26.465 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2024-03-12T05:31:26.480 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.aOCmNSzWbT 2024-03-12T05:31:26.480 DEBUG:teuthology.orchestra.run.smithi076:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aOCmNSzWbT 2024-03-12T05:31:26.549 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.aOCmNSzWbT 2024-03-12T05:31:26.685 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.aOCmNSzWbT is 6KB 2024-03-12T05:31:26.733 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.aOCmNSzWbT 2024-03-12T05:31:26.747 DEBUG:teuthology.orchestra.run.smithi076:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T05:31:26.811 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T05:31:26.812 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T05:31:26.812 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T05:31:26.812 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T05:31:26.813 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T05:31:26.979 DEBUG:teuthology.orchestra.run.smithi076:> sudo shutdown -r now 2024-03-12T05:31:56.982 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-12T05:31:56.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-12T05:31:56.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T05:32:15.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-03-12T05:32:24.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-12T05:32:24.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T05:32:27.582 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-03-12T05:32:39.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-12T05:32:39.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T05:33:39.587 DEBUG:teuthology.orchestra.remote:timed out 2024-03-12T05:33:54.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-12T05:33:54.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T05:33:54.875 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-03-12T05:33:55.230 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-12T05:33:55.230 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-427.el9.x86_64"... 2024-03-12T05:33:55.230 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-03-12T05:33:55.244 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-427.el9.x86_64 2024-03-12T05:33:55.245 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T05:33:55.245 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T05:33:55.245 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T05:33:56.246 DEBUG:teuthology.parallel:result is None 2024-03-12T05:33:56.247 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-12T05:33:56.255 INFO:teuthology.task.internal:Creating test directory... 2024-03-12T05:33:56.256 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T05:33:56.259 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T05:33:56.261 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T05:33:56.279 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-12T05:33:56.286 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-12T05:33:56.295 INFO:teuthology.task.internal:Creating archive directory... 2024-03-12T05:33:56.296 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T05:33:56.315 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T05:33:56.319 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T05:33:56.595 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-12T05:33:56.602 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-12T05:33:56.602 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T05:33:56.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T05:33:56.619 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T05:33:56.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T05:33:56.636 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T05:33:56.638 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T05:33:56.661 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T05:33:56.703 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T05:33:56.712 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T05:33:56.725 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T05:33:56.729 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T05:33:56.732 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T05:33:56.764 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T05:33:56.766 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-12T05:33:56.773 INFO:teuthology.task.internal:Configuring sudo... 2024-03-12T05:33:56.773 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T05:33:56.775 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T05:33:56.777 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T05:33:56.807 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-12T05:33:56.816 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-12T05:33:56.816 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T05:33:56.847 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T05:33:56.849 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T05:33:56.865 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T05:33:56.948 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T05:33:57.030 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T05:33:57.070 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T05:33:57.119 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:33:57.120 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T05:33:57.190 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T05:33:57.232 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T05:33:57.292 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T05:33:57.375 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T05:33:57.422 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-12T05:33:57.422 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T05:33:57.491 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T05:33:57.534 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T05:33:57.594 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T05:33:57.679 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T05:33:57.722 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-12T05:33:57.722 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T05:33:57.790 DEBUG:teuthology.orchestra.run.smithi076:> sudo service rsyslog restart 2024-03-12T05:33:57.793 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2024-03-12T05:33:57.795 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2024-03-12T05:33:57.829 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T05:33:57.851 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T05:33:57.877 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T05:33:58.263 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-12T05:33:58.271 INFO:teuthology.task.internal:Starting timer... 2024-03-12T05:33:58.271 INFO:teuthology.run_tasks:Running task pcp... 2024-03-12T05:33:58.283 INFO:teuthology.run_tasks:Running task selinux... 2024-03-12T05:33:58.295 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-12T05:33:58.296 DEBUG:teuthology.orchestra.run.smithi076:> sudo service auditd rotate 2024-03-12T05:33:58.356 INFO:teuthology.orchestra.run.smithi076.stdout:Rotating logs: 2024-03-12T05:33:58.358 DEBUG:teuthology.orchestra.run.smithi082:> sudo service auditd rotate 2024-03-12T05:33:58.428 INFO:teuthology.orchestra.run.smithi082.stdout:Rotating logs: 2024-03-12T05:33:58.429 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2024-03-12T05:33:58.483 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2024-03-12T05:33:58.485 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-12T05:33:58.485 DEBUG:teuthology.orchestra.run.smithi076:> /usr/sbin/getenforce 2024-03-12T05:33:58.518 INFO:teuthology.orchestra.run.smithi076.stdout:Permissive 2024-03-12T05:33:58.518 DEBUG:teuthology.orchestra.run.smithi082:> /usr/sbin/getenforce 2024-03-12T05:33:58.543 INFO:teuthology.orchestra.run.smithi082.stdout:Permissive 2024-03-12T05:33:58.543 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2024-03-12T05:33:58.563 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2024-03-12T05:33:58.564 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi076.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi082.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi088.front.sepia.ceph.com': 'permissive'} 2024-03-12T05:33:58.564 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T05:33:58.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T05:33:58.592 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T05:33:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T05:33:58.621 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T05:33:58.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T05:33:58.649 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-12T05:33:58.649 DEBUG:teuthology.orchestra.run.smithi076:> sudo /usr/sbin/setenforce permissive 2024-03-12T05:33:58.686 DEBUG:teuthology.orchestra.run.smithi082:> sudo /usr/sbin/setenforce permissive 2024-03-12T05:33:58.719 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2024-03-12T05:33:58.757 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-12T05:33:58.767 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-12T05:33:58.773 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-12T05:33:58.925 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-12T05:33:58.941 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-12T05:33:58.943 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi076.front.sepia.ceph.com,smithi082.front.sepia.ceph.com,smithi088.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-12T05:38:50.038 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi076.front.sepia.ceph.com'), Remote(name='ubuntu@smithi082.front.sepia.ceph.com'), Remote(name='ubuntu@smithi088.front.sepia.ceph.com')] 2024-03-12T05:38:50.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-12T05:38:50.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T05:38:50.130 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-03-12T05:38:50.217 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-12T05:38:50.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-12T05:38:50.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T05:38:50.304 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-03-12T05:38:50.374 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-12T05:38:50.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-12T05:38:50.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T05:38:50.458 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-03-12T05:38:50.524 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-03-12T05:38:50.525 INFO:teuthology.run_tasks:Running task clock... 2024-03-12T05:38:50.534 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-12T05:38:50.534 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T05:38:50.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T05:38:50.537 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T05:38:50.537 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T05:38:50.539 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T05:38:50.539 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T05:38:50.566 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T05:38:50.570 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T05:38:50.582 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T05:38:50.588 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T05:38:50.595 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T05:38:50.611 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: ntpd: command not found 2024-03-12T05:38:50.612 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T05:38:50.618 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: ntpd: command not found 2024-03-12T05:38:50.625 INFO:teuthology.orchestra.run.smithi076.stdout:506 Cannot talk to daemon 2024-03-12T05:38:50.632 INFO:teuthology.orchestra.run.smithi082.stdout:506 Cannot talk to daemon 2024-03-12T05:38:50.641 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: ntpd: command not found 2024-03-12T05:38:50.642 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T05:38:50.649 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T05:38:50.655 INFO:teuthology.orchestra.run.smithi088.stdout:506 Cannot talk to daemon 2024-03-12T05:38:50.658 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T05:38:50.666 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T05:38:50.671 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T05:38:50.687 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T05:38:50.707 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-03-12T05:38:50.710 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-03-12T05:38:50.711 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T05:38:50.711 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-03-12T05:38:50.711 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:38:50.711 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:38:50.711 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:38:50.711 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:38:50.714 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T05:38:50.714 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-03-12T05:38:50.714 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:38:50.714 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:38:50.714 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:38:50.714 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:38:50.731 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2024-03-12T05:38:50.735 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T05:38:50.735 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-03-12T05:38:50.735 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:38:50.735 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:38:50.735 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:38:50.735 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T05:38:50.735 INFO:teuthology.run_tasks:Running task install... 2024-03-12T05:38:50.743 DEBUG:teuthology.task.install:project ceph 2024-03-12T05:38:50.743 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'}} 2024-03-12T05:38:50.743 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-12T05:38:50.743 INFO:teuthology.task.install:Using flavor: default 2024-03-12T05:38:50.750 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-12T05:38:50.750 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-12T05:38:50.751 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-12T05:38:50.751 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T05:38:50.752 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-12T05:38:50.753 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T05:38:50.753 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-12T05:38:50.754 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T05:38:50.899 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-12T05:38:50.899 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-12T05:38:50.912 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-12T05:38:50.912 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-12T05:38:50.978 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-12T05:38:50.979 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-12T05:38:51.029 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-12T05:38:51.029 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-12T05:38:51.029 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-12T05:38:51.035 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-12T05:38:51.035 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-12T05:38:51.035 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-12T05:38:51.057 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-12T05:38:51.057 DEBUG:teuthology.orchestra.run.smithi082:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-12T05:38:51.062 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-12T05:38:51.062 DEBUG:teuthology.orchestra.run.smithi088:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-12T05:38:51.090 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-12T05:38:51.090 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:38:51.090 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-12T05:38:51.119 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-12T05:38:51.119 DEBUG:teuthology.orchestra.run.smithi076:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-12T05:38:51.124 DEBUG:teuthology.orchestra.run.smithi082:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-12T05:38:51.129 DEBUG:teuthology.orchestra.run.smithi088:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-12T05:38:51.187 DEBUG:teuthology.orchestra.run.smithi076:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-12T05:38:51.203 DEBUG:teuthology.orchestra.run.smithi082:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-12T05:38:51.208 DEBUG:teuthology.orchestra.run.smithi088:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-12T05:38:51.267 DEBUG:teuthology.orchestra.run.smithi076:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-12T05:38:51.270 INFO:teuthology.orchestra.run.smithi082.stdout:check_obsoletes = 1 2024-03-12T05:38:51.271 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum clean all 2024-03-12T05:38:51.274 INFO:teuthology.orchestra.run.smithi088.stdout:check_obsoletes = 1 2024-03-12T05:38:51.276 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum clean all 2024-03-12T05:38:51.333 INFO:teuthology.orchestra.run.smithi076.stdout:check_obsoletes = 1 2024-03-12T05:38:51.334 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum clean all 2024-03-12T05:38:51.505 INFO:teuthology.orchestra.run.smithi082.stdout:74 files removed 2024-03-12T05:38:51.512 INFO:teuthology.orchestra.run.smithi088.stdout:74 files removed 2024-03-12T05:38:51.531 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-12T05:38:51.539 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-12T05:38:51.579 INFO:teuthology.orchestra.run.smithi076.stdout:74 files removed 2024-03-12T05:38:51.607 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-12T05:38:52.185 INFO:teuthology.orchestra.run.smithi082.stdout:ceph packages for x86_64 227 kB/s | 82 kB 00:00 2024-03-12T05:38:52.296 INFO:teuthology.orchestra.run.smithi088.stdout:ceph packages for x86_64 176 kB/s | 82 kB 00:00 2024-03-12T05:38:52.320 INFO:teuthology.orchestra.run.smithi076.stdout:ceph packages for x86_64 206 kB/s | 82 kB 00:00 2024-03-12T05:38:52.535 INFO:teuthology.orchestra.run.smithi082.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-12T05:38:52.671 INFO:teuthology.orchestra.run.smithi088.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-12T05:38:52.754 INFO:teuthology.orchestra.run.smithi076.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-12T05:38:52.877 INFO:teuthology.orchestra.run.smithi082.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-03-12T05:38:53.096 INFO:teuthology.orchestra.run.smithi076.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-03-12T05:38:53.097 INFO:teuthology.orchestra.run.smithi088.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-03-12T05:38:53.313 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-12T05:38:53.508 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-12T05:38:53.549 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-12T05:38:54.900 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-12T05:38:54.930 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-12T05:38:55.016 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-12T05:38:56.616 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-03-12T05:38:56.753 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.0 MB 00:01 2024-03-12T05:38:56.798 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-03-12T05:38:58.325 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-12T05:38:58.749 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-12T05:38:59.821 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 19 MB 00:02 2024-03-12T05:39:02.833 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 9.7 MB/s | 6.1 MB 00:00 2024-03-12T05:39:03.467 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.1 MB 00:01 2024-03-12T05:39:04.501 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 15 kB 00:00 2024-03-12T05:39:04.768 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 90 kB/s | 15 kB 00:00 2024-03-12T05:39:04.805 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.1 MB 00:01 2024-03-12T05:39:06.548 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 15 kB 00:00 2024-03-12T05:39:06.902 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 21 MB 00:02 2024-03-12T05:39:11.861 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-12T05:39:14.414 INFO:teuthology.orchestra.run.smithi076.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-12T05:39:14.415 INFO:teuthology.orchestra.run.smithi076.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-12T05:39:14.428 INFO:teuthology.orchestra.run.smithi076.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-12T05:39:14.429 INFO:teuthology.orchestra.run.smithi076.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-12T05:39:14.430 INFO:teuthology.orchestra.run.smithi076.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-12T05:39:14.430 INFO:teuthology.orchestra.run.smithi076.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-12T05:39:14.432 INFO:teuthology.orchestra.run.smithi076.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-12T05:39:14.435 INFO:teuthology.orchestra.run.smithi076.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-12T05:39:14.436 INFO:teuthology.orchestra.run.smithi076.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-12T05:39:14.519 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-12T05:39:14.546 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================ 2024-03-12T05:39:14.546 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-03-12T05:39:14.546 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================ 2024-03-12T05:39:14.546 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-12T05:39:14.546 INFO:teuthology.orchestra.run.smithi076.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-12T05:39:14.546 INFO:teuthology.orchestra.run.smithi076.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-12T05:39:14.546 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-12T05:39:14.546 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-12T05:39:14.546 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-12T05:39:14.546 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-12T05:39:14.546 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-12T05:39:14.546 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-12T05:39:14.546 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-12T05:39:14.547 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-12T05:39:14.547 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-12T05:39:14.547 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-12T05:39:14.547 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-12T05:39:14.547 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-12T05:39:14.547 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-12T05:39:14.547 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-12T05:39:14.547 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-12T05:39:14.547 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-12T05:39:14.547 INFO:teuthology.orchestra.run.smithi076.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-12T05:39:14.547 INFO:teuthology.orchestra.run.smithi076.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-12T05:39:14.547 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-12T05:39:14.547 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-12T05:39:14.547 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-12T05:39:14.548 INFO:teuthology.orchestra.run.smithi076.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-12T05:39:14.548 INFO:teuthology.orchestra.run.smithi076.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-12T05:39:14.548 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-12T05:39:14.548 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-12T05:39:14.548 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-12T05:39:14.548 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-12T05:39:14.548 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-12T05:39:14.548 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-12T05:39:14.548 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-12T05:39:14.548 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-12T05:39:14.548 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-12T05:39:14.548 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-12T05:39:14.548 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-12T05:39:14.548 INFO:teuthology.orchestra.run.smithi076.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrading: 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-12T05:39:14.549 INFO:teuthology.orchestra.run.smithi076.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-12T05:39:14.550 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-12T05:39:14.550 INFO:teuthology.orchestra.run.smithi076.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-12T05:39:14.550 INFO:teuthology.orchestra.run.smithi076.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-12T05:39:14.550 INFO:teuthology.orchestra.run.smithi076.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-12T05:39:14.550 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-12T05:39:14.550 INFO:teuthology.orchestra.run.smithi076.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-12T05:39:14.550 INFO:teuthology.orchestra.run.smithi076.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-12T05:39:14.550 INFO:teuthology.orchestra.run.smithi076.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-12T05:39:14.550 INFO:teuthology.orchestra.run.smithi076.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-12T05:39:14.550 INFO:teuthology.orchestra.run.smithi076.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-12T05:39:14.550 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-12T05:39:14.550 INFO:teuthology.orchestra.run.smithi076.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-12T05:39:14.551 INFO:teuthology.orchestra.run.smithi076.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-12T05:39:14.551 INFO:teuthology.orchestra.run.smithi076.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-12T05:39:14.551 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-12T05:39:14.551 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-12T05:39:14.551 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-12T05:39:14.551 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-12T05:39:14.551 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-12T05:39:14.551 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-12T05:39:14.551 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-12T05:39:14.551 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-12T05:39:14.551 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-12T05:39:14.551 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-12T05:39:14.551 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-12T05:39:14.552 INFO:teuthology.orchestra.run.smithi076.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-12T05:39:14.552 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-12T05:39:14.552 INFO:teuthology.orchestra.run.smithi076.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-12T05:39:14.552 INFO:teuthology.orchestra.run.smithi076.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-12T05:39:14.552 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-12T05:39:14.552 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-12T05:39:14.552 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-12T05:39:14.552 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-12T05:39:14.553 INFO:teuthology.orchestra.run.smithi076.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-12T05:39:14.553 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-12T05:39:14.553 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-12T05:39:14.553 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-12T05:39:14.553 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-12T05:39:14.553 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-12T05:39:14.553 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-12T05:39:14.553 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-12T05:39:14.553 INFO:teuthology.orchestra.run.smithi076.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-12T05:39:14.553 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-12T05:39:14.553 INFO:teuthology.orchestra.run.smithi076.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-12T05:39:14.553 INFO:teuthology.orchestra.run.smithi076.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-12T05:39:14.553 INFO:teuthology.orchestra.run.smithi076.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-12T05:39:14.554 INFO:teuthology.orchestra.run.smithi076.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-12T05:39:14.554 INFO:teuthology.orchestra.run.smithi076.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-12T05:39:14.554 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-12T05:39:14.554 INFO:teuthology.orchestra.run.smithi076.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-12T05:39:14.554 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-12T05:39:14.554 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-12T05:39:14.554 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-12T05:39:14.554 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-12T05:39:14.554 INFO:teuthology.orchestra.run.smithi076.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-12T05:39:14.554 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-12T05:39:14.554 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-12T05:39:14.554 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-12T05:39:14.554 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-12T05:39:14.554 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-12T05:39:14.555 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-12T05:39:14.556 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-12T05:39:14.557 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2024-03-12T05:39:14.558 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-12T05:39:14.559 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:39:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-12T05:39:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================ 2024-03-12T05:39:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:Install 150 Packages 2024-03-12T05:39:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrade 24 Packages 2024-03-12T05:39:14.559 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:39:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 223 M 2024-03-12T05:39:14.559 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-12T05:39:15.603 INFO:teuthology.orchestra.run.smithi076.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 30 kB/s | 6.6 kB 00:00 2024-03-12T05:39:16.187 INFO:teuthology.orchestra.run.smithi076.stdout:(2/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 1.5 MB/s | 886 kB 00:00 2024-03-12T05:39:16.362 INFO:teuthology.orchestra.run.smithi076.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 832 kB/s | 145 kB 00:00 2024-03-12T05:39:16.662 INFO:teuthology.orchestra.run.smithi076.stdout:(4/174): ceph-common-19.0.0-1607.g94a53970.el9. 16 MB/s | 20 MB 00:01 2024-03-12T05:39:16.787 INFO:teuthology.orchestra.run.smithi076.stdout:(5/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 12 MB/s | 1.5 MB 00:00 2024-03-12T05:39:16.863 INFO:teuthology.orchestra.run.smithi076.stdout:(6/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 3.6 MB/s | 5.3 MB 00:01 2024-03-12T05:39:17.088 INFO:teuthology.orchestra.run.smithi076.stdout:(7/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 16 MB/s | 4.7 MB 00:00 2024-03-12T05:39:17.172 INFO:teuthology.orchestra.run.smithi076.stdout:(8/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 2.6 MB/s | 2.1 MB 00:00 2024-03-12T05:39:17.247 INFO:teuthology.orchestra.run.smithi076.stdout:(9/174): ceph-selinux-19.0.0-1607.g94a53970.el9 336 kB/s | 25 kB 00:00 2024-03-12T05:39:17.897 INFO:teuthology.orchestra.run.smithi076.stdout:(10/174): ceph-radosgw-19.0.0-1607.g94a53970.el 12 MB/s | 9.8 MB 00:00 2024-03-12T05:39:17.956 INFO:teuthology.orchestra.run.smithi076.stdout:(11/174): cephfs-mirror-19.0.0-1607.g94a53970.e 3.6 MB/s | 217 kB 00:00 2024-03-12T05:39:18.006 INFO:teuthology.orchestra.run.smithi076.stdout:(12/174): cephfs-shell-19.0.0-1607.g94a53970.el 542 kB/s | 27 kB 00:00 2024-03-12T05:39:18.056 INFO:teuthology.orchestra.run.smithi076.stdout:(13/174): libcephfs-devel-19.0.0-1607.g94a53970 627 kB/s | 31 kB 00:00 2024-03-12T05:39:18.156 INFO:teuthology.orchestra.run.smithi076.stdout:(14/174): libcephfs2-19.0.0-1607.g94a53970.el9. 7.3 MB/s | 742 kB 00:00 2024-03-12T05:39:18.215 INFO:teuthology.orchestra.run.smithi076.stdout:(15/174): libcephsqlite-19.0.0-1607.g94a53970.e 2.7 MB/s | 163 kB 00:00 2024-03-12T05:39:18.273 INFO:teuthology.orchestra.run.smithi076.stdout:(16/174): librados-devel-19.0.0-1607.g94a53970. 2.1 MB/s | 127 kB 00:00 2024-03-12T05:39:18.349 INFO:teuthology.orchestra.run.smithi076.stdout:(17/174): libradosstriper1-19.0.0-1607.g94a5397 6.5 MB/s | 498 kB 00:00 2024-03-12T05:39:18.741 INFO:teuthology.orchestra.run.smithi076.stdout:(18/174): librgw2-19.0.0-1607.g94a53970.el9.x86 13 MB/s | 5.1 MB 00:00 2024-03-12T05:39:18.774 INFO:teuthology.orchestra.run.smithi076.stdout:(19/174): python3-ceph-argparse-19.0.0-1607.g94 1.3 MB/s | 45 kB 00:00 2024-03-12T05:39:18.833 INFO:teuthology.orchestra.run.smithi076.stdout:(20/174): python3-ceph-common-19.0.0-1607.g94a5 2.2 MB/s | 131 kB 00:00 2024-03-12T05:39:19.000 INFO:teuthology.orchestra.run.smithi076.stdout:(21/174): ceph-osd-19.0.0-1607.g94a53970.el9.x8 7.9 MB/s | 17 MB 00:02 2024-03-12T05:39:19.025 INFO:teuthology.orchestra.run.smithi076.stdout:(22/174): python3-cephfs-19.0.0-1607.g94a53970. 842 kB/s | 161 kB 00:00 2024-03-12T05:39:19.067 INFO:teuthology.orchestra.run.smithi076.stdout:(23/174): python3-rados-19.0.0-1607.g94a53970.e 4.7 MB/s | 320 kB 00:00 2024-03-12T05:39:19.092 INFO:teuthology.orchestra.run.smithi076.stdout:(24/174): python3-rbd-19.0.0-1607.g94a53970.el9 4.4 MB/s | 298 kB 00:00 2024-03-12T05:39:19.126 INFO:teuthology.orchestra.run.smithi076.stdout:(25/174): python3-rgw-19.0.0-1607.g94a53970.el9 1.7 MB/s | 99 kB 00:00 2024-03-12T05:39:19.151 INFO:teuthology.orchestra.run.smithi076.stdout:(26/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-12T05:39:19.201 INFO:teuthology.orchestra.run.smithi076.stdout:(27/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 3.3 MB/s | 171 kB 00:00 2024-03-12T05:39:19.260 INFO:teuthology.orchestra.run.smithi076.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1607.g 539 kB/s | 31 kB 00:00 2024-03-12T05:39:19.319 INFO:teuthology.orchestra.run.smithi076.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 2.3 MB/s | 138 kB 00:00 2024-03-12T05:39:19.511 INFO:teuthology.orchestra.run.smithi076.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 9.3 MB/s | 1.8 MB 00:00 2024-03-12T05:39:19.569 INFO:teuthology.orchestra.run.smithi076.stdout:(31/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 7.0 MB/s | 3.1 MB 00:00 2024-03-12T05:39:19.653 INFO:teuthology.orchestra.run.smithi076.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1607.g94 2.9 MB/s | 247 kB 00:00 2024-03-12T05:39:19.728 INFO:teuthology.orchestra.run.smithi076.stdout:(33/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 658 kB/s | 49 kB 00:00 2024-03-12T05:39:19.795 INFO:teuthology.orchestra.run.smithi076.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1607.g9 231 kB/s | 15 kB 00:00 2024-03-12T05:39:19.878 INFO:teuthology.orchestra.run.smithi076.stdout:(35/174): ceph-volume-19.0.0-1607.g94a53970.el9 3.1 MB/s | 264 kB 00:00 2024-03-12T05:39:19.996 INFO:teuthology.orchestra.run.smithi076.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 15 MB/s | 7.4 MB 00:00 2024-03-12T05:39:20.029 INFO:teuthology.orchestra.run.smithi076.stdout:(37/174): cephadm-19.0.0-1607.g94a53970.el9.noa 4.9 MB/s | 748 kB 00:00 2024-03-12T05:39:20.054 INFO:teuthology.orchestra.run.smithi076.stdout:(38/174): cephfs-top-19.0.0-1607.g94a53970.el9. 314 kB/s | 18 kB 00:00 2024-03-12T05:39:20.238 INFO:teuthology.orchestra.run.smithi076.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.1 MB/s | 662 kB 00:00 2024-03-12T05:39:20.280 INFO:teuthology.orchestra.run.smithi076.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.0 MB/s | 920 kB 00:00 2024-03-12T05:39:20.305 INFO:teuthology.orchestra.run.smithi076.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-12T05:39:20.330 INFO:teuthology.orchestra.run.smithi076.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-12T05:39:20.355 INFO:teuthology.orchestra.run.smithi076.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 638 kB/s | 32 kB 00:00 2024-03-12T05:39:20.431 INFO:teuthology.orchestra.run.smithi076.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-12T05:39:20.456 INFO:teuthology.orchestra.run.smithi076.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 173 kB/s | 17 kB 00:00 2024-03-12T05:39:20.498 INFO:teuthology.orchestra.run.smithi076.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 243 kB/s | 10 kB 00:00 2024-03-12T05:39:20.523 INFO:teuthology.orchestra.run.smithi076.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2024-03-12T05:39:20.548 INFO:teuthology.orchestra.run.smithi076.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-12T05:39:20.574 INFO:teuthology.orchestra.run.smithi076.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-12T05:39:20.599 INFO:teuthology.orchestra.run.smithi076.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-12T05:39:20.616 INFO:teuthology.orchestra.run.smithi076.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-12T05:39:20.642 INFO:teuthology.orchestra.run.smithi076.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-12T05:39:20.667 INFO:teuthology.orchestra.run.smithi076.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-12T05:39:20.692 INFO:teuthology.orchestra.run.smithi076.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-12T05:39:20.709 INFO:teuthology.orchestra.run.smithi076.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 830 kB/s | 34 kB 00:00 2024-03-12T05:39:20.785 INFO:teuthology.orchestra.run.smithi076.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 52 MB/s | 4.7 MB 00:00 2024-03-12T05:39:20.826 INFO:teuthology.orchestra.run.smithi076.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 13 MB/s | 1.5 MB 00:00 2024-03-12T05:39:20.859 INFO:teuthology.orchestra.run.smithi076.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-12T05:39:20.885 INFO:teuthology.orchestra.run.smithi076.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-12T05:39:20.977 INFO:teuthology.orchestra.run.smithi076.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 49 MB/s | 5.7 MB 00:00 2024-03-12T05:39:21.002 INFO:teuthology.orchestra.run.smithi076.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.8 MB/s | 1.0 MB 00:00 2024-03-12T05:39:21.027 INFO:teuthology.orchestra.run.smithi076.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 78 kB 00:00 2024-03-12T05:39:21.119 INFO:teuthology.orchestra.run.smithi076.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 50 MB/s | 5.8 MB 00:00 2024-03-12T05:39:21.144 INFO:teuthology.orchestra.run.smithi076.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 215 kB/s | 24 kB 00:00 2024-03-12T05:39:21.186 INFO:teuthology.orchestra.run.smithi076.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 437 kB/s | 18 kB 00:00 2024-03-12T05:39:21.211 INFO:teuthology.orchestra.run.smithi076.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-12T05:39:21.237 INFO:teuthology.orchestra.run.smithi076.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 632 kB/s | 32 kB 00:00 2024-03-12T05:39:21.321 INFO:teuthology.orchestra.run.smithi076.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 44 MB/s | 4.8 MB 00:00 2024-03-12T05:39:21.346 INFO:teuthology.orchestra.run.smithi076.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 217 kB 00:00 2024-03-12T05:39:21.371 INFO:teuthology.orchestra.run.smithi076.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-12T05:39:21.396 INFO:teuthology.orchestra.run.smithi076.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-12T05:39:21.421 INFO:teuthology.orchestra.run.smithi076.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 825 kB/s | 41 kB 00:00 2024-03-12T05:39:21.447 INFO:teuthology.orchestra.run.smithi076.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-12T05:39:21.472 INFO:teuthology.orchestra.run.smithi076.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-12T05:39:21.497 INFO:teuthology.orchestra.run.smithi076.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-12T05:39:21.523 INFO:teuthology.orchestra.run.smithi076.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 954 kB/s | 47 kB 00:00 2024-03-12T05:39:21.548 INFO:teuthology.orchestra.run.smithi076.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 856 kB/s | 43 kB 00:00 2024-03-12T05:39:21.798 INFO:teuthology.orchestra.run.smithi076.stdout:(78/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 57 MB/s | 16 MB 00:00 2024-03-12T05:39:21.823 INFO:teuthology.orchestra.run.smithi076.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 142 kB/s | 39 kB 00:00 2024-03-12T05:39:21.849 INFO:teuthology.orchestra.run.smithi076.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 823 kB/s | 41 kB 00:00 2024-03-12T05:39:21.874 INFO:teuthology.orchestra.run.smithi076.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-12T05:39:21.899 INFO:teuthology.orchestra.run.smithi076.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-12T05:39:21.924 INFO:teuthology.orchestra.run.smithi076.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-12T05:39:21.950 INFO:teuthology.orchestra.run.smithi076.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-12T05:39:21.975 INFO:teuthology.orchestra.run.smithi076.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-12T05:39:22.000 INFO:teuthology.orchestra.run.smithi076.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-12T05:39:22.042 INFO:teuthology.orchestra.run.smithi076.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-12T05:39:22.067 INFO:teuthology.orchestra.run.smithi076.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2024-03-12T05:39:22.093 INFO:teuthology.orchestra.run.smithi076.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-12T05:39:22.118 INFO:teuthology.orchestra.run.smithi076.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-12T05:39:22.244 INFO:teuthology.orchestra.run.smithi076.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-12T05:39:22.302 INFO:teuthology.orchestra.run.smithi076.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-03-12T05:39:22.327 INFO:teuthology.orchestra.run.smithi076.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-12T05:39:22.353 INFO:teuthology.orchestra.run.smithi076.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-12T05:39:22.395 INFO:teuthology.orchestra.run.smithi076.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 323 kB/s | 21 kB 00:00 2024-03-12T05:39:22.411 INFO:teuthology.orchestra.run.smithi076.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 436 kB/s | 25 kB 00:00 2024-03-12T05:39:22.487 INFO:teuthology.orchestra.run.smithi076.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-03-12T05:39:22.570 INFO:teuthology.orchestra.run.smithi076.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-03-12T05:39:22.645 INFO:teuthology.orchestra.run.smithi076.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 16 MB/s | 4.1 MB 00:00 2024-03-12T05:39:22.688 INFO:teuthology.orchestra.run.smithi076.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 840 kB/s | 35 kB 00:00 2024-03-12T05:39:22.713 INFO:teuthology.orchestra.run.smithi076.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.5 MB/s | 216 kB 00:00 2024-03-12T05:39:22.755 INFO:teuthology.orchestra.run.smithi076.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 668 kB/s | 44 kB 00:00 2024-03-12T05:39:22.788 INFO:teuthology.orchestra.run.smithi076.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-03-12T05:39:22.838 INFO:teuthology.orchestra.run.smithi076.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-12T05:39:22.955 INFO:teuthology.orchestra.run.smithi076.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 5.1 MB/s | 863 kB 00:00 2024-03-12T05:39:24.247 INFO:teuthology.orchestra.run.smithi076.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 239 kB/s | 309 kB 00:01 2024-03-12T05:39:24.373 INFO:teuthology.orchestra.run.smithi076.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-12T05:39:24.864 INFO:teuthology.orchestra.run.smithi076.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 271 kB/s | 548 kB 00:02 2024-03-12T05:39:24.990 INFO:teuthology.orchestra.run.smithi076.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-03-12T05:39:25.198 INFO:teuthology.orchestra.run.smithi076.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 233 kB/s | 49 kB 00:00 2024-03-12T05:39:25.498 INFO:teuthology.orchestra.run.smithi076.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 224 kB/s | 67 kB 00:00 2024-03-12T05:39:25.974 INFO:teuthology.orchestra.run.smithi076.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 318 kB/s | 151 kB 00:00 2024-03-12T05:39:26.666 INFO:teuthology.orchestra.run.smithi076.stdout:(113/174): ceph-test-19.0.0-1607.g94a53970.el9. 5.0 MB/s | 47 MB 00:09 2024-03-12T05:39:27.958 INFO:teuthology.orchestra.run.smithi076.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 422 kB/s | 838 kB 00:01 2024-03-12T05:39:28.075 INFO:teuthology.orchestra.run.smithi076.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 373 kB/s | 43 kB 00:00 2024-03-12T05:39:28.200 INFO:teuthology.orchestra.run.smithi076.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 258 kB/s | 32 kB 00:00 2024-03-12T05:39:28.317 INFO:teuthology.orchestra.run.smithi076.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-03-12T05:39:28.700 INFO:teuthology.orchestra.run.smithi076.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 448 kB/s | 172 kB 00:00 2024-03-12T05:39:28.725 INFO:teuthology.orchestra.run.smithi076.stdout:(119/174): python3-asyncssh-2.13.2-4.el9.noarch 266 kB/s | 548 kB 00:02 2024-03-12T05:39:29.409 INFO:teuthology.orchestra.run.smithi076.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 506 kB/s | 358 kB 00:00 2024-03-12T05:39:29.568 INFO:teuthology.orchestra.run.smithi076.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 223 kB/s | 35 kB 00:00 2024-03-12T05:39:29.743 INFO:teuthology.orchestra.run.smithi076.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 298 kB/s | 303 kB 00:01 2024-03-12T05:39:30.027 INFO:teuthology.orchestra.run.smithi076.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 442 kB/s | 203 kB 00:00 2024-03-12T05:39:30.377 INFO:teuthology.orchestra.run.smithi076.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 413 kB/s | 144 kB 00:00 2024-03-12T05:39:30.485 INFO:teuthology.orchestra.run.smithi076.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-03-12T05:39:30.594 INFO:teuthology.orchestra.run.smithi076.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-03-12T05:39:30.744 INFO:teuthology.orchestra.run.smithi076.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 154 kB/s | 23 kB 00:00 2024-03-12T05:39:30.853 INFO:teuthology.orchestra.run.smithi076.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-03-12T05:39:30.987 INFO:teuthology.orchestra.run.smithi076.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 147 kB/s | 20 kB 00:00 2024-03-12T05:39:32.654 INFO:teuthology.orchestra.run.smithi076.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 627 kB/s | 1.0 MB 00:01 2024-03-12T05:39:32.779 INFO:teuthology.orchestra.run.smithi076.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 371 kB/s | 46 kB 00:00 2024-03-12T05:39:32.996 INFO:teuthology.orchestra.run.smithi076.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 364 kB/s | 79 kB 00:00 2024-03-12T05:39:33.113 INFO:teuthology.orchestra.run.smithi076.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 495 kB/s | 58 kB 00:00 2024-03-12T05:39:33.496 INFO:teuthology.orchestra.run.smithi076.stdout:(134/174): python3-pecan-1.4.2-3.el9.noarch.rpm 711 kB/s | 272 kB 00:00 2024-03-12T05:39:33.622 INFO:teuthology.orchestra.run.smithi076.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-03-12T05:39:33.872 INFO:teuthology.orchestra.run.smithi076.stdout:(136/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 360 kB/s | 90 kB 00:00 2024-03-12T05:39:33.997 INFO:teuthology.orchestra.run.smithi076.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 226 kB/s | 28 kB 00:00 2024-03-12T05:39:34.114 INFO:teuthology.orchestra.run.smithi076.stdout:(138/174): python3-repoze-lru-0.7-16.el9.noarch 263 kB/s | 31 kB 00:00 2024-03-12T05:39:34.406 INFO:teuthology.orchestra.run.smithi076.stdout:(139/174): python3-routes-2.5.1-5.el9.noarch.rp 647 kB/s | 188 kB 00:00 2024-03-12T05:39:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 507 kB/s | 59 kB 00:00 2024-03-12T05:39:34.565 INFO:teuthology.orchestra.run.smithi076.stdout:(141/174): python3-grpcio-1.46.7-10.el9.x86_64. 433 kB/s | 2.0 MB 00:04 2024-03-12T05:39:34.649 INFO:teuthology.orchestra.run.smithi076.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 285 kB/s | 36 kB 00:00 2024-03-12T05:39:34.682 INFO:teuthology.orchestra.run.smithi076.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 436 kB/s | 51 kB 00:00 2024-03-12T05:39:34.891 INFO:teuthology.orchestra.run.smithi076.stdout:(144/174): python3-websocket-client-1.2.3-2.el9 430 kB/s | 90 kB 00:00 2024-03-12T05:39:35.033 INFO:teuthology.orchestra.run.smithi076.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 598 kB/s | 230 kB 00:00 2024-03-12T05:39:35.141 INFO:teuthology.orchestra.run.smithi076.stdout:(146/174): libarrow-9.0.0-10.el9.x86_64.rpm 420 kB/s | 4.4 MB 00:10 2024-03-12T05:39:35.167 INFO:teuthology.orchestra.run.smithi076.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 150 kB/s | 20 kB 00:00 2024-03-12T05:39:35.525 INFO:teuthology.orchestra.run.smithi076.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 500 kB/s | 192 kB 00:00 2024-03-12T05:39:35.592 INFO:teuthology.orchestra.run.smithi076.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 609 kB/s | 427 kB 00:00 2024-03-12T05:39:36.109 INFO:teuthology.orchestra.run.smithi076.stdout:(150/174): librbd1-19.0.0-1607.g94a53970.el9.x8 6.1 MB/s | 3.1 MB 00:00 2024-03-12T05:39:36.195 INFO:teuthology.orchestra.run.smithi076.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-12T05:39:36.243 INFO:teuthology.orchestra.run.smithi076.stdout:(152/174): librados2-19.0.0-1607.g94a53970.el9. 4.7 MB/s | 3.4 MB 00:00 2024-03-12T05:39:36.610 INFO:teuthology.orchestra.run.smithi076.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 5.0 MB/s | 1.8 MB 00:00 2024-03-12T05:39:36.643 INFO:teuthology.orchestra.run.smithi076.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 42 kB 00:00 2024-03-12T05:39:36.685 INFO:teuthology.orchestra.run.smithi076.stdout:(155/174): NetworkManager-1.46.0-1.el9.x86_64.r 4.3 MB/s | 2.3 MB 00:00 2024-03-12T05:39:36.736 INFO:teuthology.orchestra.run.smithi076.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-12T05:39:36.761 INFO:teuthology.orchestra.run.smithi076.stdout:(157/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.1 MB/s | 248 kB 00:00 2024-03-12T05:39:36.778 INFO:teuthology.orchestra.run.smithi076.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 873 kB/s | 36 kB 00:00 2024-03-12T05:39:36.836 INFO:teuthology.orchestra.run.smithi076.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.5 MB/s | 192 kB 00:00 2024-03-12T05:39:36.870 INFO:teuthology.orchestra.run.smithi076.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-03-12T05:39:36.887 INFO:teuthology.orchestra.run.smithi076.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 458 kB/s | 23 kB 00:00 2024-03-12T05:39:36.912 INFO:teuthology.orchestra.run.smithi076.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 108 kB 00:00 2024-03-12T05:39:36.954 INFO:teuthology.orchestra.run.smithi076.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 155 kB 00:00 2024-03-12T05:39:36.996 INFO:teuthology.orchestra.run.smithi076.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 136 kB 00:00 2024-03-12T05:39:37.013 INFO:teuthology.orchestra.run.smithi076.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-12T05:39:37.030 INFO:teuthology.orchestra.run.smithi076.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 860 kB/s | 29 kB 00:00 2024-03-12T05:39:37.255 INFO:teuthology.orchestra.run.smithi076.stdout:(167/174): thrift-0.15.0-2.el9.x86_64.rpm 776 kB/s | 1.6 MB 00:02 2024-03-12T05:39:37.397 INFO:teuthology.orchestra.run.smithi076.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.9 MB/s | 2.2 MB 00:00 2024-03-12T05:39:37.430 INFO:teuthology.orchestra.run.smithi076.stdout:(169/174): openssl-3.0.7-27.el9.x86_64.rpm 2.9 MB/s | 1.2 MB 00:00 2024-03-12T05:39:37.473 INFO:teuthology.orchestra.run.smithi076.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 747 kB/s | 31 kB 00:00 2024-03-12T05:39:37.498 INFO:teuthology.orchestra.run.smithi076.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 4.5 MB/s | 465 kB 00:00 2024-03-12T05:39:37.556 INFO:teuthology.orchestra.run.smithi076.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 74 kB 00:00 2024-03-12T05:39:37.606 INFO:teuthology.orchestra.run.smithi076.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 6.5 MB/s | 2.3 MB 00:00 2024-03-12T05:39:37.648 INFO:teuthology.orchestra.run.smithi076.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.1 MB/s | 1.1 MB 00:00 2024-03-12T05:39:37.652 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-12T05:39:37.653 INFO:teuthology.orchestra.run.smithi076.stdout:Total 9.6 MB/s | 223 MB 00:23 2024-03-12T05:39:38.129 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-12T05:39:38.441 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-12T05:39:38.441 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-12T05:39:40.489 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-12T05:39:40.489 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-12T05:39:47.808 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-12T05:39:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-12T05:39:47.946 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-12T05:39:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-12T05:39:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-12T05:39:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-12T05:39:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-12T05:39:48.116 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-12T05:39:48.142 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-12T05:39:48.200 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-12T05:39:48.520 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-12T05:39:48.594 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T05:39:48.698 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T05:39:48.736 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T05:39:48.832 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T05:39:48.873 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-12T05:39:48.911 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-12T05:39:49.097 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-12T05:39:49.147 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-12T05:39:49.174 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T05:39:49.278 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T05:39:49.309 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-12T05:39:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-12T05:39:49.421 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-12T05:39:49.452 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-12T05:39:49.468 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-12T05:39:49.687 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-12T05:39:49.717 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-12T05:39:49.756 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-12T05:39:49.936 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-12T05:39:50.069 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-12T05:39:50.129 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-12T05:39:50.247 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T05:39:50.305 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T05:39:50.503 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T05:39:50.610 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-12T05:39:50.704 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-12T05:39:50.744 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-12T05:39:50.770 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-12T05:39:50.806 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-12T05:39:50.879 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-12T05:39:50.933 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-12T05:39:50.972 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-12T05:39:51.036 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-12T05:39:51.070 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-12T05:39:51.114 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-12T05:39:51.148 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-12T05:39:51.175 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-12T05:39:51.244 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-12T05:39:51.281 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-12T05:39:51.340 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-12T05:39:51.368 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-12T05:39:51.437 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-12T05:39:51.483 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-12T05:39:51.556 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-12T05:39:51.689 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-12T05:39:51.743 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-12T05:39:52.274 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-12T05:39:52.356 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-12T05:39:52.446 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-12T05:39:52.509 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-12T05:39:52.518 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-12T05:39:52.556 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-12T05:39:52.598 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-12T05:39:52.636 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-12T05:39:52.667 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-12T05:39:52.951 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-12T05:39:53.156 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-12T05:39:53.243 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-12T05:39:53.296 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-12T05:39:53.446 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-12T05:39:53.499 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-12T05:39:53.553 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-12T05:39:53.626 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-12T05:39:53.658 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-12T05:39:53.694 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-12T05:39:53.741 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-12T05:39:53.777 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-12T05:39:53.820 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-12T05:39:53.862 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-12T05:39:53.996 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-12T05:39:54.051 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-12T05:39:54.082 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-12T05:39:54.123 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-12T05:39:54.217 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-12T05:39:54.257 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-12T05:39:54.289 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-12T05:39:54.314 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-12T05:39:54.338 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-12T05:39:54.427 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-12T05:39:54.555 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-12T05:39:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-12T05:39:54.705 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-12T05:39:54.753 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-12T05:39:54.817 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-12T05:39:55.025 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-12T05:39:55.067 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-12T05:39:55.194 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-12T05:39:55.233 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-12T05:39:55.265 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-12T05:39:55.295 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-12T05:39:55.323 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-12T05:39:55.358 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-12T05:39:55.397 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-12T05:39:55.428 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-12T05:39:55.467 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-12T05:39:55.550 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-12T05:39:55.586 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-12T05:39:55.619 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-12T05:39:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-12T05:39:55.695 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-12T05:39:55.783 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-12T05:39:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-12T05:39:55.909 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-12T05:39:55.941 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-12T05:39:55.987 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-12T05:39:56.084 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-12T05:39:56.142 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-12T05:39:56.862 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-12T05:39:56.906 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-12T05:39:56.940 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-12T05:39:56.988 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-12T05:39:57.068 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-12T05:39:57.101 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-12T05:39:57.271 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-12T05:39:57.307 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-12T05:39:57.395 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-12T05:39:58.028 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-12T05:39:58.146 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-12T05:39:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-12T05:39:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-12T05:39:59.465 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-12T05:39:59.495 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-12T05:39:59.523 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-12T05:39:59.831 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-12T05:39:59.861 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-12T05:39:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-12T05:39:59.983 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-12T05:40:00.020 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-12T05:40:00.475 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-12T05:40:00.515 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-12T05:40:00.585 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-12T05:40:00.608 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-12T05:40:02.784 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T05:40:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T05:40:03.366 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T05:40:03.401 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-12T05:40:03.475 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-12T05:40:03.475 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-12T05:40:03.475 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-12T05:40:03.475 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:40:03.519 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-12T05:40:15.020 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-12T05:40:15.020 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /sys 2024-03-12T05:40:15.020 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /proc 2024-03-12T05:40:15.020 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /mnt 2024-03-12T05:40:15.020 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /var/tmp 2024-03-12T05:40:15.020 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /home 2024-03-12T05:40:15.020 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /root 2024-03-12T05:40:15.021 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /tmp 2024-03-12T05:40:15.021 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:40:15.182 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-12T05:40:15.197 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-12T05:40:15.197 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:40:15.197 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-12T05:40:15.197 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:40:16.702 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-12T05:40:16.728 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-12T05:40:16.728 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:40:16.728 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-12T05:40:16.728 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T05:40:16.728 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T05:40:16.728 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:40:16.936 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-12T05:40:16.958 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-12T05:40:16.958 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:40:16.958 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-12T05:40:16.958 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T05:40:16.959 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T05:40:16.959 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:40:17.403 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-12T05:40:17.426 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-12T05:40:17.426 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:40:17.426 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-12T05:40:17.426 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T05:40:17.426 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T05:40:17.426 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:40:17.513 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-12T05:40:17.906 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-12T05:40:17.937 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-12T05:40:17.968 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-12T05:40:18.029 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-12T05:40:18.052 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-12T05:40:18.282 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-12T05:40:18.311 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-12T05:40:19.269 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-12T05:40:19.295 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-12T05:40:19.393 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-12T05:40:19.565 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-12T05:40:19.596 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-12T05:40:19.620 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-12T05:40:19.620 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:40:19.620 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-12T05:40:19.620 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T05:40:19.620 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T05:40:19.620 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:40:19.664 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-12T05:40:19.674 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-12T05:40:19.729 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-12T05:40:20.587 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-12T05:40:20.616 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-12T05:40:20.638 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-12T05:40:20.638 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:40:20.638 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-12T05:40:20.638 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T05:40:20.638 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T05:40:20.638 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:40:20.692 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-12T05:40:20.713 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-12T05:40:20.714 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:40:20.714 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-12T05:40:20.714 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:40:20.761 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T05:40:20.781 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T05:40:20.781 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:40:20.781 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-12T05:40:20.781 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-12T05:40:20.782 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-12T05:40:20.782 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:40:21.085 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-12T05:40:21.107 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-12T05:40:21.107 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:40:21.107 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-12T05:40:21.107 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T05:40:21.108 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T05:40:21.108 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:40:25.548 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-12T05:40:25.589 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-12T05:40:25.619 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-12T05:40:25.648 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-12T05:40:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-12T05:40:25.947 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-12T05:40:25.988 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-12T05:40:26.037 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-12T05:40:26.096 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-12T05:40:26.131 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-12T05:40:26.164 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-12T05:40:26.196 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-12T05:40:26.226 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-12T05:40:26.290 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-12T05:40:26.369 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-12T05:40:26.408 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-12T05:40:27.065 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T05:40:27.106 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T05:40:27.125 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T05:40:27.945 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-12T05:40:28.026 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-12T05:40:28.067 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-12T05:40:28.164 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-12T05:40:28.205 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-12T05:40:28.230 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-12T05:40:28.255 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-12T05:40:28.281 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-12T05:40:28.282 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-12T05:40:28.369 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-12T05:40:28.399 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-12T05:40:28.399 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-12T05:40:28.475 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-12T05:40:28.501 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-12T05:40:28.535 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-12T05:40:28.560 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-12T05:40:28.585 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-12T05:40:28.619 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-12T05:40:28.666 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-12T05:40:28.698 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-12T05:40:28.723 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-12T05:40:28.749 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-12T05:40:28.774 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-12T05:40:28.808 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-12T05:40:28.815 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T05:40:28.815 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T05:40:28.994 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T05:40:28.994 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-12T05:40:29.077 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-12T05:40:29.102 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-12T05:40:29.127 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-12T05:40:29.160 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-12T05:40:29.186 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-12T05:40:29.260 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-12T05:40:34.169 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-12T05:40:34.169 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-12T05:40:34.169 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-12T05:40:34.169 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-12T05:40:34.169 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-12T05:40:34.169 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-12T05:40:34.169 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-12T05:40:34.170 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-12T05:40:34.170 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-12T05:40:34.170 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-12T05:40:34.170 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T05:40:34.170 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T05:40:34.170 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-12T05:40:34.170 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-12T05:40:34.170 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-12T05:40:34.170 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-12T05:40:34.170 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T05:40:34.170 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-12T05:40:34.170 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-12T05:40:34.170 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-12T05:40:34.171 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-12T05:40:34.171 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-12T05:40:34.171 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-12T05:40:34.171 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-12T05:40:34.171 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-12T05:40:34.171 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-12T05:40:34.171 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-12T05:40:34.171 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-12T05:40:34.188 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-12T05:40:34.188 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-12T05:40:34.188 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-12T05:40:34.188 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-12T05:40:34.188 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-12T05:40:34.188 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-12T05:40:34.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-12T05:40:34.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-12T05:40:34.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-12T05:40:34.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-12T05:40:34.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-12T05:40:34.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-12T05:40:34.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-12T05:40:34.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-12T05:40:34.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-12T05:40:34.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-12T05:40:34.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-12T05:40:34.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-12T05:40:34.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-12T05:40:34.189 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-12T05:40:34.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-12T05:40:34.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-12T05:40:34.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-12T05:40:34.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-12T05:40:34.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-12T05:40:34.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-12T05:40:34.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-12T05:40:34.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-12T05:40:34.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-12T05:40:34.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-12T05:40:34.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-12T05:40:34.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-12T05:40:34.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-12T05:40:34.191 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-12T05:40:34.191 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-12T05:40:34.191 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-12T05:40:34.191 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-12T05:40:34.191 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-12T05:40:34.191 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-12T05:40:34.191 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-12T05:40:34.191 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-12T05:40:34.191 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-12T05:40:34.191 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-12T05:40:34.191 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-12T05:40:34.191 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-12T05:40:34.191 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-12T05:40:34.191 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-12T05:40:34.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-12T05:40:34.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-12T05:40:34.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-12T05:40:34.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-12T05:40:34.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-12T05:40:34.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-12T05:40:34.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-12T05:40:34.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-12T05:40:34.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-12T05:40:34.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-12T05:40:34.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-12T05:40:34.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-12T05:40:34.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-12T05:40:34.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-12T05:40:34.193 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-12T05:40:34.194 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-12T05:40:34.194 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-12T05:40:34.194 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-12T05:40:34.194 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-12T05:40:34.194 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-12T05:40:34.194 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-12T05:40:34.194 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-12T05:40:34.194 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-12T05:40:34.194 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-12T05:40:34.194 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-12T05:40:34.194 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-12T05:40:34.194 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-12T05:40:34.195 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-12T05:40:34.195 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-12T05:40:34.195 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-12T05:40:34.195 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-12T05:40:34.195 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-12T05:40:34.195 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-12T05:40:34.195 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-12T05:40:34.195 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-12T05:40:34.195 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-12T05:40:34.195 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-12T05:40:34.195 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-12T05:40:34.195 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-12T05:40:34.195 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-12T05:40:34.196 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-12T05:40:34.196 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-12T05:40:34.196 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-12T05:40:34.196 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-12T05:40:34.196 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-12T05:40:34.196 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-12T05:40:34.196 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-12T05:40:34.196 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-12T05:40:34.196 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-12T05:40:34.196 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-12T05:40:34.196 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-12T05:40:34.196 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-12T05:40:34.197 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-12T05:40:34.197 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-12T05:40:34.197 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-12T05:40:34.197 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-12T05:40:34.197 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-12T05:40:34.197 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-12T05:40:34.197 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-12T05:40:34.197 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-12T05:40:34.197 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-12T05:40:34.197 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-12T05:40:34.197 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T05:40:34.197 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-12T05:40:34.197 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-12T05:40:34.197 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-12T05:40:34.198 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-12T05:40:34.198 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-12T05:40:34.198 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-12T05:40:34.198 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-12T05:40:34.198 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-12T05:40:34.198 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-12T05:40:34.198 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-12T05:40:34.198 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-12T05:40:34.198 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-12T05:40:34.198 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-12T05:40:34.198 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-12T05:40:34.198 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-12T05:40:34.198 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-12T05:40:34.199 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-12T05:40:34.199 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-12T05:40:34.199 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-12T05:40:34.199 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-12T05:40:34.199 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-12T05:40:34.199 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-12T05:40:34.199 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-12T05:40:34.199 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-12T05:40:34.199 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-12T05:40:34.199 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-12T05:40:34.199 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-12T05:40:34.199 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-12T05:40:34.199 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-12T05:40:34.200 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-12T05:40:34.200 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-12T05:40:34.200 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-12T05:40:34.200 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-12T05:40:34.200 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-12T05:40:34.200 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-12T05:40:34.200 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-12T05:40:34.200 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-12T05:40:34.200 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-12T05:40:34.200 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-12T05:40:34.200 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-12T05:40:34.200 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-12T05:40:34.200 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-12T05:40:34.201 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-12T05:40:34.201 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-12T05:40:34.201 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-12T05:40:34.201 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-12T05:40:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-12T05:40:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:40:37.143 INFO:teuthology.orchestra.run.smithi076.stdout:Upgraded: 2024-03-12T05:40:37.143 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-12T05:40:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-12T05:40:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-12T05:40:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-12T05:40:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: acl-2.3.1-4.el9.x86_64 2024-03-12T05:40:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-12T05:40:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-12T05:40:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-12T05:40:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-12T05:40:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-12T05:40:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-12T05:40:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-12T05:40:37.144 INFO:teuthology.orchestra.run.smithi076.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-12T05:40:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-12T05:40:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-12T05:40:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-12T05:40:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-12T05:40:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-12T05:40:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-12T05:40:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-12T05:40:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-12T05:40:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-12T05:40:37.145 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-12T05:40:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-12T05:40:37.145 INFO:teuthology.orchestra.run.smithi076.stdout: automake-1.16.2-8.el9.noarch 2024-03-12T05:40:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: bison-3.7.4-5.el9.x86_64 2024-03-12T05:40:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-12T05:40:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:40:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:40:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:40:37.146 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:40:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:40:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:40:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:40:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:40:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:40:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.147 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:40:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-12T05:40:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: flex-2.6.4-9.el9.x86_64 2024-03-12T05:40:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-12T05:40:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-12T05:40:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-12T05:40:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-12T05:40:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-12T05:40:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-12T05:40:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: jq-1.6-16.el9.x86_64 2024-03-12T05:40:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-12T05:40:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-12T05:40:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-12T05:40:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.148 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-12T05:40:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-12T05:40:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-12T05:40:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-12T05:40:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-12T05:40:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-12T05:40:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-12T05:40:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-12T05:40:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-12T05:40:37.149 INFO:teuthology.orchestra.run.smithi076.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-12T05:40:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-12T05:40:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: lua-5.4.4-4.el9.x86_64 2024-03-12T05:40:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-12T05:40:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-12T05:40:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-12T05:40:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-12T05:40:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-12T05:40:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-12T05:40:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-12T05:40:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-13.11-1.el9.x86_64 2024-03-12T05:40:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-12T05:40:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-12T05:40:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-12T05:40:37.150 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-12T05:40:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-12T05:40:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-12T05:40:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-12T05:40:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-12T05:40:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-12T05:40:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-12T05:40:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-12T05:40:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-12T05:40:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-12T05:40:37.151 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-12T05:40:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-12T05:40:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-12T05:40:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-12T05:40:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-12T05:40:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-12T05:40:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-12T05:40:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-12T05:40:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-12T05:40:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-12T05:40:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-12T05:40:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-12T05:40:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-12T05:40:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-12T05:40:37.152 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-12T05:40:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-12T05:40:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-12T05:40:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-12T05:40:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-12T05:40:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-12T05:40:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-12T05:40:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-12T05:40:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-12T05:40:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-12T05:40:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-12T05:40:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-12T05:40:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-12T05:40:37.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply-3.11-14.el9.noarch 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-12T05:40:37.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-12T05:40:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-12T05:40:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-12T05:40:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-12T05:40:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-12T05:40:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-12T05:40:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-12T05:40:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-12T05:40:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-12T05:40:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-12T05:40:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-12T05:40:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-12T05:40:37.155 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-12T05:40:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-12T05:40:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-12T05:40:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:40:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: re2-1:20211101-3.el9.x86_64 2024-03-12T05:40:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-12T05:40:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-12T05:40:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-12T05:40:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-12T05:40:37.156 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-12T05:40:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-12T05:40:37.157 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:40:37.157 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-12T05:40:37.648 DEBUG:teuthology.parallel:result is None 2024-03-12T05:41:15.364 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 161 kB/s | 21 MB 02:10 2024-03-12T05:41:20.154 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-03-12T05:41:20.157 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 158 kB/s | 21 MB 02:13 2024-03-12T05:41:22.590 INFO:teuthology.orchestra.run.smithi088.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-12T05:41:22.591 INFO:teuthology.orchestra.run.smithi088.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-12T05:41:22.603 INFO:teuthology.orchestra.run.smithi088.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-12T05:41:22.604 INFO:teuthology.orchestra.run.smithi088.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-12T05:41:22.605 INFO:teuthology.orchestra.run.smithi088.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-12T05:41:22.606 INFO:teuthology.orchestra.run.smithi088.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-12T05:41:22.608 INFO:teuthology.orchestra.run.smithi088.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-12T05:41:22.610 INFO:teuthology.orchestra.run.smithi088.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-12T05:41:22.611 INFO:teuthology.orchestra.run.smithi088.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-12T05:41:22.691 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-03-12T05:41:22.718 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-03-12T05:41:22.718 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-03-12T05:41:22.718 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-03-12T05:41:22.718 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-03-12T05:41:22.718 INFO:teuthology.orchestra.run.smithi088.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-12T05:41:22.718 INFO:teuthology.orchestra.run.smithi088.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-12T05:41:22.718 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-12T05:41:22.719 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-12T05:41:22.719 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-12T05:41:22.719 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-12T05:41:22.719 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-12T05:41:22.719 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-12T05:41:22.719 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-12T05:41:22.719 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-12T05:41:22.719 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-12T05:41:22.719 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-12T05:41:22.719 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-12T05:41:22.719 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-12T05:41:22.719 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-12T05:41:22.719 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-12T05:41:22.720 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-12T05:41:22.720 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-12T05:41:22.720 INFO:teuthology.orchestra.run.smithi088.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-12T05:41:22.720 INFO:teuthology.orchestra.run.smithi088.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-12T05:41:22.720 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-12T05:41:22.720 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-12T05:41:22.720 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-12T05:41:22.720 INFO:teuthology.orchestra.run.smithi088.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-12T05:41:22.720 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-12T05:41:22.720 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-12T05:41:22.720 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-12T05:41:22.721 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-12T05:41:22.721 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-12T05:41:22.721 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-12T05:41:22.721 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-12T05:41:22.721 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-12T05:41:22.721 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-12T05:41:22.721 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-12T05:41:22.721 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-12T05:41:22.721 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-12T05:41:22.721 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-12T05:41:22.721 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrading: 2024-03-12T05:41:22.722 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-12T05:41:22.722 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-12T05:41:22.722 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-12T05:41:22.722 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-12T05:41:22.722 INFO:teuthology.orchestra.run.smithi088.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-12T05:41:22.722 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-12T05:41:22.722 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-12T05:41:22.722 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-12T05:41:22.722 INFO:teuthology.orchestra.run.smithi088.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-12T05:41:22.722 INFO:teuthology.orchestra.run.smithi088.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-12T05:41:22.722 INFO:teuthology.orchestra.run.smithi088.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-12T05:41:22.722 INFO:teuthology.orchestra.run.smithi088.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-12T05:41:22.723 INFO:teuthology.orchestra.run.smithi088.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-12T05:41:22.723 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-12T05:41:22.723 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-12T05:41:22.723 INFO:teuthology.orchestra.run.smithi088.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-12T05:41:22.723 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-12T05:41:22.723 INFO:teuthology.orchestra.run.smithi088.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-12T05:41:22.723 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-12T05:41:22.723 INFO:teuthology.orchestra.run.smithi088.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-12T05:41:22.723 INFO:teuthology.orchestra.run.smithi088.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-12T05:41:22.723 INFO:teuthology.orchestra.run.smithi088.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-12T05:41:22.723 INFO:teuthology.orchestra.run.smithi088.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-12T05:41:22.723 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-12T05:41:22.723 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-03-12T05:41:22.724 INFO:teuthology.orchestra.run.smithi088.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-12T05:41:22.724 INFO:teuthology.orchestra.run.smithi088.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-12T05:41:22.724 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-12T05:41:22.724 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-12T05:41:22.724 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-12T05:41:22.724 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-12T05:41:22.724 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-12T05:41:22.724 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-12T05:41:22.724 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-12T05:41:22.724 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-12T05:41:22.724 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-12T05:41:22.725 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-12T05:41:22.725 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-12T05:41:22.725 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-12T05:41:22.725 INFO:teuthology.orchestra.run.smithi088.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-12T05:41:22.725 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-12T05:41:22.725 INFO:teuthology.orchestra.run.smithi088.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-12T05:41:22.725 INFO:teuthology.orchestra.run.smithi088.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-12T05:41:22.725 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-12T05:41:22.725 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-12T05:41:22.725 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-12T05:41:22.725 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-12T05:41:22.725 INFO:teuthology.orchestra.run.smithi088.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-12T05:41:22.725 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-12T05:41:22.726 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-12T05:41:22.726 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-12T05:41:22.726 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-12T05:41:22.726 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-12T05:41:22.726 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-12T05:41:22.726 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-12T05:41:22.726 INFO:teuthology.orchestra.run.smithi088.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-12T05:41:22.726 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-12T05:41:22.726 INFO:teuthology.orchestra.run.smithi088.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-12T05:41:22.726 INFO:teuthology.orchestra.run.smithi088.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-12T05:41:22.726 INFO:teuthology.orchestra.run.smithi088.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-12T05:41:22.726 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-12T05:41:22.727 INFO:teuthology.orchestra.run.smithi088.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-12T05:41:22.727 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-12T05:41:22.727 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-12T05:41:22.727 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-12T05:41:22.727 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-12T05:41:22.727 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-12T05:41:22.727 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-12T05:41:22.727 INFO:teuthology.orchestra.run.smithi088.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-12T05:41:22.727 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-12T05:41:22.727 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-12T05:41:22.727 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-12T05:41:22.727 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-12T05:41:22.728 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-12T05:41:22.728 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-12T05:41:22.728 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-12T05:41:22.728 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-12T05:41:22.728 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-12T05:41:22.728 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-12T05:41:22.728 INFO:teuthology.orchestra.run.smithi088.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-12T05:41:22.728 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-12T05:41:22.728 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-12T05:41:22.728 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-12T05:41:22.728 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-12T05:41:22.728 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-12T05:41:22.728 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-12T05:41:22.729 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-12T05:41:22.729 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-12T05:41:22.729 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-12T05:41:22.729 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-12T05:41:22.729 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-12T05:41:22.729 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-12T05:41:22.729 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-12T05:41:22.729 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-12T05:41:22.729 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-12T05:41:22.729 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-12T05:41:22.729 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-12T05:41:22.729 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-12T05:41:22.729 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-12T05:41:22.730 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-12T05:41:22.730 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-12T05:41:22.730 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-12T05:41:22.730 INFO:teuthology.orchestra.run.smithi088.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-12T05:41:22.730 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-12T05:41:22.730 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-12T05:41:22.730 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-12T05:41:22.730 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-12T05:41:22.730 INFO:teuthology.orchestra.run.smithi088.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-12T05:41:22.730 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-12T05:41:22.730 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-12T05:41:22.730 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-12T05:41:22.730 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-12T05:41:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-12T05:41:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-12T05:41:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-12T05:41:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-12T05:41:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-12T05:41:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-12T05:41:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-12T05:41:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-12T05:41:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-12T05:41:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-12T05:41:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-12T05:41:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-12T05:41:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-12T05:41:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-12T05:41:22.732 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-12T05:41:22.732 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-12T05:41:22.732 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-12T05:41:22.732 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-12T05:41:22.732 INFO:teuthology.orchestra.run.smithi088.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-12T05:41:22.732 INFO:teuthology.orchestra.run.smithi088.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-12T05:41:22.732 INFO:teuthology.orchestra.run.smithi088.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-12T05:41:22.732 INFO:teuthology.orchestra.run.smithi088.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-12T05:41:22.732 INFO:teuthology.orchestra.run.smithi088.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-12T05:41:22.732 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-12T05:41:22.732 INFO:teuthology.orchestra.run.smithi088.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-12T05:41:22.732 INFO:teuthology.orchestra.run.smithi088.stdout:Installing weak dependencies: 2024-03-12T05:41:22.733 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-12T05:41:22.733 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:41:22.733 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-03-12T05:41:22.733 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================ 2024-03-12T05:41:22.733 INFO:teuthology.orchestra.run.smithi088.stdout:Install 150 Packages 2024-03-12T05:41:22.733 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade 24 Packages 2024-03-12T05:41:22.733 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:41:22.733 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 223 M 2024-03-12T05:41:22.734 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-03-12T05:41:23.414 INFO:teuthology.orchestra.run.smithi088.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-03-12T05:41:23.689 INFO:teuthology.orchestra.run.smithi088.stdout:(2/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 11 MB/s | 5.3 MB 00:00 2024-03-12T05:41:23.739 INFO:teuthology.orchestra.run.smithi088.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 2.8 MB/s | 145 kB 00:00 2024-03-12T05:41:23.898 INFO:teuthology.orchestra.run.smithi088.stdout:(4/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 13 MB/s | 2.1 MB 00:00 2024-03-12T05:41:24.022 INFO:teuthology.orchestra.run.smithi088.stdout:(5/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 13 MB/s | 1.5 MB 00:00 2024-03-12T05:41:24.090 INFO:teuthology.orchestra.run.smithi088.stdout:(6/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 1.3 MB/s | 886 kB 00:00 2024-03-12T05:41:24.290 INFO:teuthology.orchestra.run.smithi088.stdout:(7/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 17 MB/s | 4.7 MB 00:00 2024-03-12T05:41:24.851 INFO:teuthology.orchestra.run.smithi088.stdout:(8/174): ceph-radosgw-19.0.0-1607.g94a53970.el9 17 MB/s | 9.8 MB 00:00 2024-03-12T05:41:24.899 INFO:teuthology.orchestra.run.smithi088.stdout:(9/174): ceph-selinux-19.0.0-1607.g94a53970.el9 504 kB/s | 25 kB 00:00 2024-03-12T05:41:25.082 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-12T05:41:26.366 INFO:teuthology.orchestra.run.smithi088.stdout:(10/174): ceph-common-19.0.0-1607.g94a53970.el9 6.3 MB/s | 20 MB 00:03 2024-03-12T05:41:26.433 INFO:teuthology.orchestra.run.smithi088.stdout:(11/174): cephfs-mirror-19.0.0-1607.g94a53970.e 3.2 MB/s | 217 kB 00:00 2024-03-12T05:41:26.524 INFO:teuthology.orchestra.run.smithi088.stdout:(12/174): cephfs-shell-19.0.0-1607.g94a53970.el 296 kB/s | 27 kB 00:00 2024-03-12T05:41:26.616 INFO:teuthology.orchestra.run.smithi088.stdout:(13/174): libcephfs-devel-19.0.0-1607.g94a53970 340 kB/s | 31 kB 00:00 2024-03-12T05:41:26.784 INFO:teuthology.orchestra.run.smithi088.stdout:(14/174): libcephfs2-19.0.0-1607.g94a53970.el9. 4.3 MB/s | 742 kB 00:00 2024-03-12T05:41:26.875 INFO:teuthology.orchestra.run.smithi088.stdout:(15/174): libcephsqlite-19.0.0-1607.g94a53970.e 1.7 MB/s | 163 kB 00:00 2024-03-12T05:41:26.992 INFO:teuthology.orchestra.run.smithi088.stdout:(16/174): librados-devel-19.0.0-1607.g94a53970. 1.1 MB/s | 127 kB 00:00 2024-03-12T05:41:27.101 INFO:teuthology.orchestra.run.smithi088.stdout:(17/174): libradosstriper1-19.0.0-1607.g94a5397 4.5 MB/s | 498 kB 00:00 2024-03-12T05:41:27.526 INFO:teuthology.orchestra.run.smithi082.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-12T05:41:27.527 INFO:teuthology.orchestra.run.smithi082.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-12T05:41:27.539 INFO:teuthology.orchestra.run.smithi082.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-12T05:41:27.539 INFO:teuthology.orchestra.run.smithi082.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-12T05:41:27.540 INFO:teuthology.orchestra.run.smithi082.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-12T05:41:27.541 INFO:teuthology.orchestra.run.smithi082.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-12T05:41:27.543 INFO:teuthology.orchestra.run.smithi082.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-12T05:41:27.545 INFO:teuthology.orchestra.run.smithi082.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-12T05:41:27.546 INFO:teuthology.orchestra.run.smithi082.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-12T05:41:27.626 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-03-12T05:41:27.626 INFO:teuthology.orchestra.run.smithi088.stdout:(18/174): librgw2-19.0.0-1607.g94a53970.el9.x86 9.7 MB/s | 5.1 MB 00:00 2024-03-12T05:41:27.642 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================================ 2024-03-12T05:41:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-03-12T05:41:27.642 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================================ 2024-03-12T05:41:27.642 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-03-12T05:41:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-12T05:41:27.642 INFO:teuthology.orchestra.run.smithi082.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-12T05:41:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-12T05:41:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-12T05:41:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-12T05:41:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-12T05:41:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-12T05:41:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-12T05:41:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-12T05:41:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-12T05:41:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-12T05:41:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-12T05:41:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-12T05:41:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-12T05:41:27.643 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-12T05:41:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-12T05:41:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-12T05:41:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-12T05:41:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-12T05:41:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-12T05:41:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-12T05:41:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-12T05:41:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-12T05:41:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-12T05:41:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-12T05:41:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-12T05:41:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-12T05:41:27.644 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-12T05:41:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-12T05:41:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-12T05:41:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-12T05:41:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-12T05:41:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-12T05:41:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-12T05:41:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-12T05:41:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-12T05:41:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-12T05:41:27.645 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrading: 2024-03-12T05:41:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-12T05:41:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-12T05:41:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-12T05:41:27.645 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-12T05:41:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-12T05:41:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-12T05:41:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-12T05:41:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-12T05:41:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-12T05:41:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-12T05:41:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-12T05:41:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-12T05:41:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-12T05:41:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-12T05:41:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-12T05:41:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-12T05:41:27.646 INFO:teuthology.orchestra.run.smithi082.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-12T05:41:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-12T05:41:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-12T05:41:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-12T05:41:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-12T05:41:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-12T05:41:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-12T05:41:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-12T05:41:27.647 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-03-12T05:41:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-12T05:41:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-12T05:41:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-12T05:41:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-12T05:41:27.647 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-12T05:41:27.648 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-12T05:41:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-12T05:41:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-12T05:41:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-12T05:41:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-12T05:41:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-12T05:41:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-12T05:41:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-12T05:41:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-12T05:41:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-12T05:41:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-12T05:41:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-12T05:41:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-12T05:41:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-12T05:41:27.649 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-12T05:41:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-12T05:41:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-12T05:41:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-12T05:41:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-12T05:41:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-12T05:41:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-12T05:41:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-12T05:41:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-12T05:41:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-12T05:41:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-12T05:41:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-12T05:41:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-12T05:41:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-12T05:41:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-12T05:41:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-12T05:41:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-12T05:41:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-12T05:41:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-12T05:41:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-12T05:41:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-12T05:41:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-12T05:41:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-12T05:41:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-12T05:41:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-12T05:41:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-12T05:41:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-12T05:41:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-12T05:41:27.651 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-12T05:41:27.652 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-12T05:41:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-12T05:41:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-12T05:41:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-12T05:41:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-12T05:41:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-12T05:41:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-12T05:41:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-12T05:41:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-12T05:41:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-12T05:41:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-12T05:41:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-12T05:41:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-12T05:41:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-12T05:41:27.653 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-12T05:41:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-12T05:41:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-12T05:41:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-12T05:41:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-12T05:41:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-12T05:41:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-12T05:41:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-12T05:41:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-12T05:41:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-12T05:41:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-12T05:41:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-12T05:41:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-12T05:41:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-12T05:41:27.654 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout:Installing weak dependencies: 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================================ 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout:Install 150 Packages 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrade 24 Packages 2024-03-12T05:41:27.655 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:41:27.656 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 223 M 2024-03-12T05:41:27.656 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-03-12T05:41:28.043 INFO:teuthology.orchestra.run.smithi088.stdout:(19/174): ceph-test-19.0.0-1607.g94a53970.el9.x 15 MB/s | 47 MB 00:03 2024-03-12T05:41:28.123 INFO:teuthology.orchestra.run.smithi088.stdout:(20/174): python3-ceph-argparse-19.0.0-1607.g94 102 kB/s | 45 kB 00:00 2024-03-12T05:41:28.123 INFO:teuthology.orchestra.run.smithi088.stdout:(21/174): python3-ceph-common-19.0.0-1607.g94a5 2.6 MB/s | 131 kB 00:00 2024-03-12T05:41:28.128 INFO:teuthology.orchestra.run.smithi088.stdout:(22/174): python3-cephfs-19.0.0-1607.g94a53970. 2.7 MB/s | 161 kB 00:00 2024-03-12T05:41:28.178 INFO:teuthology.orchestra.run.smithi088.stdout:(23/174): python3-rados-19.0.0-1607.g94a53970.e 3.7 MB/s | 320 kB 00:00 2024-03-12T05:41:28.207 INFO:teuthology.orchestra.run.smithi082.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-03-12T05:41:28.211 INFO:teuthology.orchestra.run.smithi088.stdout:(24/174): python3-rbd-19.0.0-1607.g94a53970.el9 3.5 MB/s | 298 kB 00:00 2024-03-12T05:41:28.237 INFO:teuthology.orchestra.run.smithi088.stdout:(25/174): python3-rgw-19.0.0-1607.g94a53970.el9 1.7 MB/s | 99 kB 00:00 2024-03-12T05:41:28.270 INFO:teuthology.orchestra.run.smithi088.stdout:(26/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-12T05:41:28.395 INFO:teuthology.orchestra.run.smithi088.stdout:(27/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 1.3 MB/s | 171 kB 00:00 2024-03-12T05:41:28.466 INFO:teuthology.orchestra.run.smithi082.stdout:(2/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 3.4 MB/s | 886 kB 00:00 2024-03-12T05:41:28.498 INFO:teuthology.orchestra.run.smithi088.stdout:(28/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 12 MB/s | 3.1 MB 00:00 2024-03-12T05:41:28.516 INFO:teuthology.orchestra.run.smithi082.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 2.9 MB/s | 145 kB 00:00 2024-03-12T05:41:28.538 INFO:teuthology.orchestra.run.smithi088.stdout:(29/174): ceph-grafana-dashboards-19.0.0-1607.g 222 kB/s | 31 kB 00:00 2024-03-12T05:41:28.563 INFO:teuthology.orchestra.run.smithi088.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 2.1 MB/s | 138 kB 00:00 2024-03-12T05:41:28.908 INFO:teuthology.orchestra.run.smithi082.stdout:(4/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 5.4 MB/s | 2.1 MB 00:00 2024-03-12T05:41:29.088 INFO:teuthology.orchestra.run.smithi088.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 3.2 MB/s | 1.8 MB 00:00 2024-03-12T05:41:29.117 INFO:teuthology.orchestra.run.smithi082.stdout:(5/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 7.1 MB/s | 1.5 MB 00:00 2024-03-12T05:41:29.180 INFO:teuthology.orchestra.run.smithi088.stdout:(32/174): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-03-12T05:41:29.205 INFO:teuthology.orchestra.run.smithi088.stdout:(33/174): ceph-mgr-modules-core-19.0.0-1607.g94 2.1 MB/s | 247 kB 00:00 2024-03-12T05:41:29.231 INFO:teuthology.orchestra.run.smithi088.stdout:(34/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 985 kB/s | 49 kB 00:00 2024-03-12T05:41:29.273 INFO:teuthology.orchestra.run.smithi088.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1607.g9 229 kB/s | 15 kB 00:00 2024-03-12T05:41:29.298 INFO:teuthology.orchestra.run.smithi088.stdout:(36/174): ceph-volume-19.0.0-1607.g94a53970.el9 3.9 MB/s | 264 kB 00:00 2024-03-12T05:41:29.382 INFO:teuthology.orchestra.run.smithi088.stdout:(37/174): cephfs-top-19.0.0-1607.g94a53970.el9. 219 kB/s | 18 kB 00:00 2024-03-12T05:41:29.407 INFO:teuthology.orchestra.run.smithi088.stdout:(38/174): cephadm-19.0.0-1607.g94a53970.el9.noa 5.5 MB/s | 748 kB 00:00 2024-03-12T05:41:29.484 INFO:teuthology.orchestra.run.smithi082.stdout:(6/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 13 MB/s | 4.7 MB 00:00 2024-03-12T05:41:29.598 INFO:teuthology.orchestra.run.smithi088.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 4.7 MB/s | 920 kB 00:00 2024-03-12T05:41:29.624 INFO:teuthology.orchestra.run.smithi088.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 662 kB 00:00 2024-03-12T05:41:29.649 INFO:teuthology.orchestra.run.smithi088.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2024-03-12T05:41:29.675 INFO:teuthology.orchestra.run.smithi088.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-12T05:41:29.700 INFO:teuthology.orchestra.run.smithi088.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 634 kB/s | 32 kB 00:00 2024-03-12T05:41:29.784 INFO:teuthology.orchestra.run.smithi088.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-12T05:41:29.809 INFO:teuthology.orchestra.run.smithi088.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 159 kB/s | 17 kB 00:00 2024-03-12T05:41:29.851 INFO:teuthology.orchestra.run.smithi088.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 242 kB/s | 10 kB 00:00 2024-03-12T05:41:29.876 INFO:teuthology.orchestra.run.smithi088.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2024-03-12T05:41:29.901 INFO:teuthology.orchestra.run.smithi088.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-12T05:41:29.935 INFO:teuthology.orchestra.run.smithi088.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-12T05:41:29.960 INFO:teuthology.orchestra.run.smithi088.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-12T05:41:29.986 INFO:teuthology.orchestra.run.smithi088.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 46 kB 00:00 2024-03-12T05:41:29.992 INFO:teuthology.orchestra.run.smithi082.stdout:(7/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 2.7 MB/s | 5.3 MB 00:01 2024-03-12T05:41:30.011 INFO:teuthology.orchestra.run.smithi088.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-12T05:41:30.036 INFO:teuthology.orchestra.run.smithi088.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-12T05:41:30.061 INFO:teuthology.orchestra.run.smithi088.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-12T05:41:30.087 INFO:teuthology.orchestra.run.smithi088.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 687 kB/s | 34 kB 00:00 2024-03-12T05:41:30.171 INFO:teuthology.orchestra.run.smithi088.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 44 MB/s | 4.7 MB 00:00 2024-03-12T05:41:30.212 INFO:teuthology.orchestra.run.smithi088.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 12 MB/s | 1.5 MB 00:00 2024-03-12T05:41:30.256 INFO:teuthology.orchestra.run.smithi088.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 9.6 MB/s | 811 kB 00:00 2024-03-12T05:41:30.282 INFO:teuthology.orchestra.run.smithi088.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 1.9 MB/s | 134 kB 00:00 2024-03-12T05:41:30.374 INFO:teuthology.orchestra.run.smithi088.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 49 MB/s | 5.7 MB 00:00 2024-03-12T05:41:30.407 INFO:teuthology.orchestra.run.smithi088.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.2 MB/s | 1.0 MB 00:00 2024-03-12T05:41:30.433 INFO:teuthology.orchestra.run.smithi088.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-12T05:41:30.533 INFO:teuthology.orchestra.run.smithi088.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 46 MB/s | 5.8 MB 00:00 2024-03-12T05:41:30.550 INFO:teuthology.orchestra.run.smithi082.stdout:(8/174): ceph-osd-19.0.0-1607.g94a53970.el9.x86 16 MB/s | 17 MB 00:01 2024-03-12T05:41:30.558 INFO:teuthology.orchestra.run.smithi088.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 200 kB/s | 24 kB 00:00 2024-03-12T05:41:30.600 INFO:teuthology.orchestra.run.smithi088.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 437 kB/s | 18 kB 00:00 2024-03-12T05:41:30.617 INFO:teuthology.orchestra.run.smithi082.stdout:(9/174): ceph-selinux-19.0.0-1607.g94a53970.el9 379 kB/s | 25 kB 00:00 2024-03-12T05:41:30.626 INFO:teuthology.orchestra.run.smithi088.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-12T05:41:30.642 INFO:teuthology.orchestra.run.smithi088.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 758 kB/s | 32 kB 00:00 2024-03-12T05:41:30.726 INFO:teuthology.orchestra.run.smithi088.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 48 MB/s | 4.8 MB 00:00 2024-03-12T05:41:30.752 INFO:teuthology.orchestra.run.smithi088.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 217 kB 00:00 2024-03-12T05:41:30.777 INFO:teuthology.orchestra.run.smithi088.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-12T05:41:30.819 INFO:teuthology.orchestra.run.smithi088.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-12T05:41:30.836 INFO:teuthology.orchestra.run.smithi088.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 706 kB/s | 41 kB 00:00 2024-03-12T05:41:30.861 INFO:teuthology.orchestra.run.smithi088.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-12T05:41:30.886 INFO:teuthology.orchestra.run.smithi088.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-12T05:41:30.912 INFO:teuthology.orchestra.run.smithi088.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-12T05:41:30.929 INFO:teuthology.orchestra.run.smithi088.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-12T05:41:30.946 INFO:teuthology.orchestra.run.smithi088.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-12T05:41:31.063 INFO:teuthology.orchestra.run.smithi088.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 334 kB/s | 39 kB 00:00 2024-03-12T05:41:31.238 INFO:teuthology.orchestra.run.smithi088.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 51 MB/s | 16 MB 00:00 2024-03-12T05:41:31.255 INFO:teuthology.orchestra.run.smithi088.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 216 kB/s | 41 kB 00:00 2024-03-12T05:41:31.297 INFO:teuthology.orchestra.run.smithi088.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-12T05:41:31.322 INFO:teuthology.orchestra.run.smithi088.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 847 kB/s | 56 kB 00:00 2024-03-12T05:41:31.339 INFO:teuthology.orchestra.run.smithi088.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-12T05:41:31.356 INFO:teuthology.orchestra.run.smithi088.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-12T05:41:31.381 INFO:teuthology.orchestra.run.smithi088.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-12T05:41:31.406 INFO:teuthology.orchestra.run.smithi088.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-12T05:41:31.440 INFO:teuthology.orchestra.run.smithi088.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-12T05:41:31.465 INFO:teuthology.orchestra.run.smithi088.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-12T05:41:31.482 INFO:teuthology.orchestra.run.smithi088.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.9 MB/s | 124 kB 00:00 2024-03-12T05:41:31.508 INFO:teuthology.orchestra.run.smithi088.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 4.4 MB/s | 187 kB 00:00 2024-03-12T05:41:31.616 INFO:teuthology.orchestra.run.smithi088.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-12T05:41:31.675 INFO:teuthology.orchestra.run.smithi088.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 4.7 MB/s | 803 kB 00:00 2024-03-12T05:41:31.700 INFO:teuthology.orchestra.run.smithi088.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-12T05:41:31.725 INFO:teuthology.orchestra.run.smithi088.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-12T05:41:32.009 INFO:teuthology.orchestra.run.smithi088.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 90 kB/s | 25 kB 00:00 2024-03-12T05:41:32.168 INFO:teuthology.orchestra.run.smithi088.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 46 kB/s | 21 kB 00:00 2024-03-12T05:41:32.234 INFO:teuthology.orchestra.run.smithi082.stdout:(10/174): ceph-common-19.0.0-1607.g94a53970.el9 4.7 MB/s | 20 MB 00:04 2024-03-12T05:41:32.343 INFO:teuthology.orchestra.run.smithi082.stdout:(11/174): ceph-radosgw-19.0.0-1607.g94a53970.el 4.2 MB/s | 9.8 MB 00:02 2024-03-12T05:41:32.368 INFO:teuthology.orchestra.run.smithi082.stdout:(12/174): cephfs-mirror-19.0.0-1607.g94a53970.e 1.6 MB/s | 217 kB 00:00 2024-03-12T05:41:32.393 INFO:teuthology.orchestra.run.smithi082.stdout:(13/174): cephfs-shell-19.0.0-1607.g94a53970.el 542 kB/s | 27 kB 00:00 2024-03-12T05:41:32.418 INFO:teuthology.orchestra.run.smithi082.stdout:(14/174): libcephfs-devel-19.0.0-1607.g94a53970 620 kB/s | 31 kB 00:00 2024-03-12T05:41:32.510 INFO:teuthology.orchestra.run.smithi082.stdout:(15/174): libcephsqlite-19.0.0-1607.g94a53970.e 1.7 MB/s | 163 kB 00:00 2024-03-12T05:41:32.602 INFO:teuthology.orchestra.run.smithi082.stdout:(16/174): libcephfs2-19.0.0-1607.g94a53970.el9. 3.5 MB/s | 742 kB 00:00 2024-03-12T05:41:32.627 INFO:teuthology.orchestra.run.smithi082.stdout:(17/174): librados-devel-19.0.0-1607.g94a53970. 1.1 MB/s | 127 kB 00:00 2024-03-12T05:41:32.769 INFO:teuthology.orchestra.run.smithi082.stdout:(18/174): libradosstriper1-19.0.0-1607.g94a5397 2.9 MB/s | 498 kB 00:00 2024-03-12T05:41:32.862 INFO:teuthology.orchestra.run.smithi082.stdout:(19/174): python3-ceph-argparse-19.0.0-1607.g94 492 kB/s | 45 kB 00:00 2024-03-12T05:41:32.918 INFO:teuthology.orchestra.run.smithi088.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 327 kB/s | 245 kB 00:00 2024-03-12T05:41:32.961 INFO:teuthology.orchestra.run.smithi082.stdout:(20/174): python3-ceph-common-19.0.0-1607.g94a5 1.3 MB/s | 131 kB 00:00 2024-03-12T05:41:33.062 INFO:teuthology.orchestra.run.smithi082.stdout:(21/174): python3-cephfs-19.0.0-1607.g94a53970. 1.6 MB/s | 161 kB 00:00 2024-03-12T05:41:33.210 INFO:teuthology.orchestra.run.smithi088.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 851 kB/s | 248 kB 00:00 2024-03-12T05:41:33.229 INFO:teuthology.orchestra.run.smithi082.stdout:(22/174): python3-rados-19.0.0-1607.g94a53970.e 1.9 MB/s | 320 kB 00:00 2024-03-12T05:41:33.296 INFO:teuthology.orchestra.run.smithi082.stdout:(23/174): librgw2-19.0.0-1607.g94a53970.el9.x86 7.7 MB/s | 5.1 MB 00:00 2024-03-12T05:41:33.321 INFO:teuthology.orchestra.run.smithi082.stdout:(24/174): python3-rbd-19.0.0-1607.g94a53970.el9 3.2 MB/s | 298 kB 00:00 2024-03-12T05:41:33.371 INFO:teuthology.orchestra.run.smithi082.stdout:(25/174): python3-rgw-19.0.0-1607.g94a53970.el9 1.3 MB/s | 99 kB 00:00 2024-03-12T05:41:33.402 INFO:teuthology.orchestra.run.smithi088.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.1 MB/s | 216 kB 00:00 2024-03-12T05:41:33.405 INFO:teuthology.orchestra.run.smithi082.stdout:(26/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 1.0 MB/s | 85 kB 00:00 2024-03-12T05:41:33.469 INFO:teuthology.orchestra.run.smithi088.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 523 kB/s | 35 kB 00:00 2024-03-12T05:41:33.497 INFO:teuthology.orchestra.run.smithi082.stdout:(27/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 1.8 MB/s | 171 kB 00:00 2024-03-12T05:41:33.669 INFO:teuthology.orchestra.run.smithi088.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 223 kB/s | 44 kB 00:00 2024-03-12T05:41:33.981 INFO:teuthology.orchestra.run.smithi082.stdout:(28/174): ceph-test-19.0.0-1607.g94a53970.el9.x 14 MB/s | 47 MB 00:03 2024-03-12T05:41:33.997 INFO:teuthology.orchestra.run.smithi082.stdout:(29/174): ceph-grafana-dashboards-19.0.0-1607.g 63 kB/s | 31 kB 00:00 2024-03-12T05:41:34.031 INFO:teuthology.orchestra.run.smithi082.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 2.7 MB/s | 138 kB 00:00 2024-03-12T05:41:34.036 INFO:teuthology.orchestra.run.smithi088.stdout:(102/174): ceph-osd-19.0.0-1607.g94a53970.el9.x 1.7 MB/s | 17 MB 00:09 2024-03-12T05:41:34.078 INFO:teuthology.orchestra.run.smithi088.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 229 kB/s | 94 kB 00:00 2024-03-12T05:41:34.090 INFO:teuthology.orchestra.run.smithi082.stdout:(31/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 4.3 MB/s | 3.1 MB 00:00 2024-03-12T05:41:34.120 INFO:teuthology.orchestra.run.smithi088.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-12T05:41:34.190 INFO:teuthology.orchestra.run.smithi082.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1607.g94 2.4 MB/s | 247 kB 00:00 2024-03-12T05:41:34.316 INFO:teuthology.orchestra.run.smithi082.stdout:(33/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 395 kB/s | 49 kB 00:00 2024-03-12T05:41:34.328 INFO:teuthology.orchestra.run.smithi088.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.4 MB/s | 863 kB 00:00 2024-03-12T05:41:34.399 INFO:teuthology.orchestra.run.smithi082.stdout:(34/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 4.4 MB/s | 1.8 MB 00:00 2024-03-12T05:41:34.412 INFO:teuthology.orchestra.run.smithi088.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-03-12T05:41:34.424 INFO:teuthology.orchestra.run.smithi082.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1607.g9 141 kB/s | 15 kB 00:00 2024-03-12T05:41:34.516 INFO:teuthology.orchestra.run.smithi082.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 15 MB/s | 7.4 MB 00:00 2024-03-12T05:41:34.537 INFO:teuthology.orchestra.run.smithi088.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-12T05:41:34.541 INFO:teuthology.orchestra.run.smithi082.stdout:(37/174): ceph-volume-19.0.0-1607.g94a53970.el9 1.8 MB/s | 264 kB 00:00 2024-03-12T05:41:34.567 INFO:teuthology.orchestra.run.smithi082.stdout:(38/174): cephfs-top-19.0.0-1607.g94a53970.el9. 365 kB/s | 18 kB 00:00 2024-03-12T05:41:34.570 INFO:teuthology.orchestra.run.smithi088.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-03-12T05:41:34.600 INFO:teuthology.orchestra.run.smithi082.stdout:(39/174): cephadm-19.0.0-1607.g94a53970.el9.noa 4.2 MB/s | 748 kB 00:00 2024-03-12T05:41:34.704 INFO:teuthology.orchestra.run.smithi088.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 194 kB/s | 26 kB 00:00 2024-03-12T05:41:34.775 INFO:teuthology.orchestra.run.smithi082.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 857 kB/s | 106 kB 00:00 2024-03-12T05:41:34.800 INFO:teuthology.orchestra.run.smithi082.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 2.5 MB/s | 662 kB 00:00 2024-03-12T05:41:34.821 INFO:teuthology.orchestra.run.smithi088.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 417 kB/s | 49 kB 00:00 2024-03-12T05:41:34.834 INFO:teuthology.orchestra.run.smithi082.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 920 kB 00:00 2024-03-12T05:41:34.869 INFO:teuthology.orchestra.run.smithi082.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 307 kB 00:00 2024-03-12T05:41:34.886 INFO:teuthology.orchestra.run.smithi082.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 621 kB/s | 32 kB 00:00 2024-03-12T05:41:34.953 INFO:teuthology.orchestra.run.smithi082.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 25 MB/s | 3.0 MB 00:00 2024-03-12T05:41:34.970 INFO:teuthology.orchestra.run.smithi082.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 173 kB/s | 17 kB 00:00 2024-03-12T05:41:34.976 INFO:teuthology.orchestra.run.smithi088.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 435 kB/s | 67 kB 00:00 2024-03-12T05:41:34.986 INFO:teuthology.orchestra.run.smithi082.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 102 kB/s | 10 kB 00:00 2024-03-12T05:41:35.003 INFO:teuthology.orchestra.run.smithi082.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-12T05:41:35.029 INFO:teuthology.orchestra.run.smithi082.stdout:(49/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.6 MB/s | 159 kB 00:00 2024-03-12T05:41:35.062 INFO:teuthology.orchestra.run.smithi082.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-12T05:41:35.088 INFO:teuthology.orchestra.run.smithi082.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 6.9 MB/s | 578 kB 00:00 2024-03-12T05:41:35.105 INFO:teuthology.orchestra.run.smithi082.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-12T05:41:35.129 INFO:teuthology.orchestra.run.smithi082.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-12T05:41:35.146 INFO:teuthology.orchestra.run.smithi082.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-12T05:41:35.163 INFO:teuthology.orchestra.run.smithi082.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-12T05:41:35.188 INFO:teuthology.orchestra.run.smithi082.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-12T05:41:35.260 INFO:teuthology.orchestra.run.smithi088.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 4.4 MB 00:00 2024-03-12T05:41:35.264 INFO:teuthology.orchestra.run.smithi082.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 41 MB/s | 4.7 MB 00:00 2024-03-12T05:41:35.285 INFO:teuthology.orchestra.run.smithi088.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 489 kB/s | 151 kB 00:00 2024-03-12T05:41:35.307 INFO:teuthology.orchestra.run.smithi082.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-03-12T05:41:35.331 INFO:teuthology.orchestra.run.smithi082.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 5.7 MB/s | 811 kB 00:00 2024-03-12T05:41:35.356 INFO:teuthology.orchestra.run.smithi082.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.5 MB/s | 134 kB 00:00 2024-03-12T05:41:35.390 INFO:teuthology.orchestra.run.smithi082.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-12T05:41:35.474 INFO:teuthology.orchestra.run.smithi082.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-12T05:41:35.498 INFO:teuthology.orchestra.run.smithi082.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 547 kB/s | 78 kB 00:00 2024-03-12T05:41:35.561 INFO:teuthology.orchestra.run.smithi088.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 1.9 MB/s | 548 kB 00:00 2024-03-12T05:41:35.590 INFO:teuthology.orchestra.run.smithi082.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-12T05:41:35.607 INFO:teuthology.orchestra.run.smithi082.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 171 kB/s | 18 kB 00:00 2024-03-12T05:41:35.633 INFO:teuthology.orchestra.run.smithi082.stdout:(66/174): python3-jsonpatch-1.21-16.el9.noarch. 186 kB/s | 24 kB 00:00 2024-03-12T05:41:35.658 INFO:teuthology.orchestra.run.smithi082.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-12T05:41:35.661 INFO:teuthology.orchestra.run.smithi088.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 838 kB 00:00 2024-03-12T05:41:35.683 INFO:teuthology.orchestra.run.smithi082.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 421 kB/s | 32 kB 00:00 2024-03-12T05:41:35.686 INFO:teuthology.orchestra.run.smithi088.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 347 kB/s | 43 kB 00:00 2024-03-12T05:41:35.708 INFO:teuthology.orchestra.run.smithi082.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.3 MB/s | 217 kB 00:00 2024-03-12T05:41:35.784 INFO:teuthology.orchestra.run.smithi082.stdout:(70/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-12T05:41:35.786 INFO:teuthology.orchestra.run.smithi088.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-03-12T05:41:35.809 INFO:teuthology.orchestra.run.smithi082.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2024-03-12T05:41:35.812 INFO:teuthology.orchestra.run.smithi088.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 214 kB/s | 32 kB 00:00 2024-03-12T05:41:35.834 INFO:teuthology.orchestra.run.smithi082.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 560 kB/s | 69 kB 00:00 2024-03-12T05:41:35.876 INFO:teuthology.orchestra.run.smithi082.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 448 kB/s | 41 kB 00:00 2024-03-12T05:41:35.901 INFO:teuthology.orchestra.run.smithi082.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-12T05:41:35.927 INFO:teuthology.orchestra.run.smithi082.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-12T05:41:35.952 INFO:teuthology.orchestra.run.smithi082.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-12T05:41:35.977 INFO:teuthology.orchestra.run.smithi082.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 868 kB/s | 43 kB 00:00 2024-03-12T05:41:36.002 INFO:teuthology.orchestra.run.smithi082.stdout:(78/174): python3-pytz-2021.1-5.el9.noarch.rpm 632 kB/s | 47 kB 00:00 2024-03-12T05:41:36.070 INFO:teuthology.orchestra.run.smithi088.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 606 kB/s | 172 kB 00:00 2024-03-12T05:41:36.136 INFO:teuthology.orchestra.run.smithi082.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 248 kB/s | 39 kB 00:00 2024-03-12T05:41:36.146 INFO:teuthology.orchestra.run.smithi088.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 1.0 MB/s | 358 kB 00:00 2024-03-12T05:41:36.262 INFO:teuthology.orchestra.run.smithi088.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 303 kB/s | 35 kB 00:00 2024-03-12T05:41:36.288 INFO:teuthology.orchestra.run.smithi088.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.4 MB/s | 303 kB 00:00 2024-03-12T05:41:36.294 INFO:teuthology.orchestra.run.smithi082.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-12T05:41:36.322 INFO:teuthology.orchestra.run.smithi082.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 130 kB/s | 41 kB 00:00 2024-03-12T05:41:36.345 INFO:teuthology.orchestra.run.smithi082.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-12T05:41:36.370 INFO:teuthology.orchestra.run.smithi082.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-12T05:41:36.395 INFO:teuthology.orchestra.run.smithi082.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 843 kB/s | 63 kB 00:00 2024-03-12T05:41:36.421 INFO:teuthology.orchestra.run.smithi082.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 585 kB/s | 44 kB 00:00 2024-03-12T05:41:36.446 INFO:teuthology.orchestra.run.smithi082.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-12T05:41:36.471 INFO:teuthology.orchestra.run.smithi082.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-12T05:41:36.505 INFO:teuthology.orchestra.run.smithi082.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-12T05:41:36.530 INFO:teuthology.orchestra.run.smithi082.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-03-12T05:41:36.546 INFO:teuthology.orchestra.run.smithi088.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 714 kB/s | 203 kB 00:00 2024-03-12T05:41:36.563 INFO:teuthology.orchestra.run.smithi082.stdout:(90/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-12T05:41:36.617 INFO:teuthology.orchestra.run.smithi082.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.6 MB/s | 187 kB 00:00 2024-03-12T05:41:36.822 INFO:teuthology.orchestra.run.smithi088.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 526 kB/s | 144 kB 00:00 2024-03-12T05:41:36.905 INFO:teuthology.orchestra.run.smithi088.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 3.3 MB/s | 2.0 MB 00:00 2024-03-12T05:41:36.955 INFO:teuthology.orchestra.run.smithi088.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2024-03-12T05:41:37.031 INFO:teuthology.orchestra.run.smithi088.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 141 kB/s | 18 kB 00:00 2024-03-12T05:41:37.081 INFO:teuthology.orchestra.run.smithi088.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 184 kB/s | 23 kB 00:00 2024-03-12T05:41:37.173 INFO:teuthology.orchestra.run.smithi088.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 137 kB/s | 19 kB 00:00 2024-03-12T05:41:37.190 INFO:teuthology.orchestra.run.smithi088.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-03-12T05:41:37.357 INFO:teuthology.orchestra.run.smithi088.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 278 kB/s | 46 kB 00:00 2024-03-12T05:41:37.524 INFO:teuthology.orchestra.run.smithi088.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 3.1 MB/s | 1.0 MB 00:00 2024-03-12T05:41:37.591 INFO:teuthology.orchestra.run.smithi088.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 338 kB/s | 79 kB 00:00 2024-03-12T05:41:37.775 INFO:teuthology.orchestra.run.smithi088.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 230 kB/s | 58 kB 00:00 2024-03-12T05:41:37.916 INFO:teuthology.orchestra.run.smithi088.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 116 kB/s | 16 kB 00:00 2024-03-12T05:41:37.992 INFO:teuthology.orchestra.run.smithi088.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 680 kB/s | 272 kB 00:00 2024-03-12T05:41:38.092 INFO:teuthology.orchestra.run.smithi088.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 282 kB/s | 28 kB 00:00 2024-03-12T05:41:38.125 INFO:teuthology.orchestra.run.smithi088.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 431 kB/s | 90 kB 00:00 2024-03-12T05:41:38.201 INFO:teuthology.orchestra.run.smithi088.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-03-12T05:41:38.326 INFO:teuthology.orchestra.run.smithi088.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 943 kB/s | 188 kB 00:00 2024-03-12T05:41:38.368 INFO:teuthology.orchestra.run.smithi088.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 354 kB/s | 59 kB 00:00 2024-03-12T05:41:38.443 INFO:teuthology.orchestra.run.smithi088.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 305 kB/s | 36 kB 00:00 2024-03-12T05:41:38.552 INFO:teuthology.orchestra.run.smithi088.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 278 kB/s | 51 kB 00:00 2024-03-12T05:41:38.777 INFO:teuthology.orchestra.run.smithi088.stdout:(144/174): python3-websocket-client-1.2.3-2.el9 399 kB/s | 90 kB 00:00 2024-03-12T05:41:38.810 INFO:teuthology.orchestra.run.smithi088.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 626 kB/s | 230 kB 00:00 2024-03-12T05:41:38.927 INFO:teuthology.orchestra.run.smithi088.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 171 kB/s | 20 kB 00:00 2024-03-12T05:41:39.228 INFO:teuthology.orchestra.run.smithi088.stdout:(147/174): openssl-devel-3.0.7-27.el9.x86_64.rp 586 kB/s | 4.1 MB 00:07 2024-03-12T05:41:39.286 INFO:teuthology.orchestra.run.smithi088.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 534 kB/s | 192 kB 00:00 2024-03-12T05:41:39.478 INFO:teuthology.orchestra.run.smithi088.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 609 kB/s | 427 kB 00:00 2024-03-12T05:41:39.904 INFO:teuthology.orchestra.run.smithi088.stdout:(150/174): librbd1-19.0.0-1607.g94a53970.el9.x8 7.4 MB/s | 3.1 MB 00:00 2024-03-12T05:41:39.945 INFO:teuthology.orchestra.run.smithi088.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-12T05:41:40.129 INFO:teuthology.orchestra.run.smithi088.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 13 MB/s | 2.3 MB 00:00 2024-03-12T05:41:40.204 INFO:teuthology.orchestra.run.smithi088.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-03-12T05:41:40.238 INFO:teuthology.orchestra.run.smithi088.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 42 kB 00:00 2024-03-12T05:41:40.280 INFO:teuthology.orchestra.run.smithi088.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 248 kB 00:00 2024-03-12T05:41:40.322 INFO:teuthology.orchestra.run.smithi088.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-03-12T05:41:40.355 INFO:teuthology.orchestra.run.smithi088.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 1.1 MB/s | 36 kB 00:00 2024-03-12T05:41:40.397 INFO:teuthology.orchestra.run.smithi088.stdout:(158/174): thrift-0.15.0-2.el9.x86_64.rpm 1.4 MB/s | 1.6 MB 00:01 2024-03-12T05:41:40.422 INFO:teuthology.orchestra.run.smithi088.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-03-12T05:41:40.456 INFO:teuthology.orchestra.run.smithi088.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 691 kB/s | 23 kB 00:00 2024-03-12T05:41:40.498 INFO:teuthology.orchestra.run.smithi088.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 108 kB 00:00 2024-03-12T05:41:40.523 INFO:teuthology.orchestra.run.smithi088.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.0 MB/s | 256 kB 00:00 2024-03-12T05:41:40.580 INFO:teuthology.orchestra.run.smithi088.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.9 MB/s | 155 kB 00:00 2024-03-12T05:41:40.599 INFO:teuthology.orchestra.run.smithi088.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-12T05:41:40.616 INFO:teuthology.orchestra.run.smithi088.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.7 MB/s | 64 kB 00:00 2024-03-12T05:41:40.666 INFO:teuthology.orchestra.run.smithi088.stdout:(166/174): librados2-19.0.0-1607.g94a53970.el9. 2.4 MB/s | 3.4 MB 00:01 2024-03-12T05:41:40.683 INFO:teuthology.orchestra.run.smithi088.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 342 kB/s | 29 kB 00:00 2024-03-12T05:41:40.717 INFO:teuthology.orchestra.run.smithi088.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 12 MB/s | 1.2 MB 00:00 2024-03-12T05:41:40.775 INFO:teuthology.orchestra.run.smithi088.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 7.8 MB/s | 465 kB 00:00 2024-03-12T05:41:40.817 INFO:teuthology.orchestra.run.smithi088.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 17 MB/s | 2.3 MB 00:00 2024-03-12T05:41:40.834 INFO:teuthology.orchestra.run.smithi088.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 533 kB/s | 31 kB 00:00 2024-03-12T05:41:40.876 INFO:teuthology.orchestra.run.smithi088.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-12T05:41:40.918 INFO:teuthology.orchestra.run.smithi088.stdout:(173/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 8.6 MB/s | 2.2 MB 00:00 2024-03-12T05:41:41.102 INFO:teuthology.orchestra.run.smithi088.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 278 kB/s | 74 kB 00:00 2024-03-12T05:41:41.106 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-03-12T05:41:41.106 INFO:teuthology.orchestra.run.smithi088.stdout:Total 12 MB/s | 223 MB 00:18 2024-03-12T05:41:41.572 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-03-12T05:41:41.848 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-03-12T05:41:41.848 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-03-12T05:41:43.630 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-03-12T05:41:43.630 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-03-12T05:41:50.455 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-03-12T05:41:50.579 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-12T05:41:50.605 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-12T05:41:50.644 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-12T05:41:50.654 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-12T05:41:50.694 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-12T05:41:50.748 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-12T05:41:50.781 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-12T05:41:50.807 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-12T05:41:50.848 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-12T05:41:51.168 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-12T05:41:51.224 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T05:41:51.345 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T05:41:51.375 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T05:41:51.466 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T05:41:51.496 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-12T05:41:51.531 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-12T05:41:51.694 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-12T05:41:51.752 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-12T05:41:51.780 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T05:41:51.883 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T05:41:51.906 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-12T05:41:51.970 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-12T05:41:52.009 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-12T05:41:52.032 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-12T05:41:52.048 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-12T05:41:52.232 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-12T05:41:52.263 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-12T05:41:52.295 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-12T05:41:52.400 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-12T05:41:52.497 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-12T05:41:52.541 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-12T05:41:52.613 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T05:41:52.658 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T05:41:52.870 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T05:41:52.979 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-12T05:41:53.074 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-12T05:41:53.105 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-12T05:41:53.132 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-12T05:41:53.167 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-12T05:41:53.241 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-12T05:41:53.303 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-12T05:41:53.342 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-12T05:41:53.407 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-12T05:41:53.440 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-12T05:41:53.484 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-12T05:41:53.517 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-12T05:41:53.544 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-12T05:41:53.614 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-12T05:41:53.650 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-12T05:41:53.692 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-12T05:41:53.722 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-12T05:41:53.791 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-12T05:41:53.836 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-12T05:41:53.934 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-12T05:41:54.076 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-12T05:41:54.122 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-12T05:41:54.649 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-12T05:41:54.724 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-12T05:41:54.804 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-12T05:41:54.845 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-12T05:41:54.853 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-12T05:41:54.885 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-12T05:41:54.926 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-12T05:41:54.963 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-12T05:41:54.993 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-12T05:41:55.218 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-12T05:41:55.396 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-12T05:41:55.461 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-12T05:41:55.505 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-12T05:41:55.645 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-12T05:41:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-12T05:41:55.746 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-12T05:41:55.810 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-12T05:41:55.842 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-12T05:41:55.878 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-12T05:41:55.926 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-12T05:41:55.961 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-12T05:41:56.004 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-12T05:41:56.046 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-12T05:41:56.188 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-12T05:41:56.244 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-12T05:41:56.274 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-12T05:41:56.315 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-12T05:41:56.407 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-12T05:41:56.441 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-12T05:41:56.481 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-12T05:41:56.506 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-12T05:41:56.531 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-12T05:41:56.602 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-12T05:41:56.731 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-12T05:41:56.829 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-12T05:41:56.872 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-12T05:41:56.921 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-12T05:41:56.975 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-12T05:41:57.158 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-12T05:41:57.224 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-12T05:41:57.353 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-12T05:41:57.391 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-12T05:41:57.423 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-12T05:41:57.454 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-12T05:41:57.482 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-12T05:41:57.517 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-12T05:41:57.556 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-12T05:41:57.587 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-12T05:41:57.618 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-12T05:41:57.711 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-12T05:41:57.753 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-12T05:41:57.786 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-12T05:41:57.828 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-12T05:41:57.862 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-12T05:41:57.953 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-12T05:41:58.042 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-12T05:41:58.101 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-12T05:41:58.133 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-12T05:41:58.171 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-12T05:41:58.268 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-12T05:41:58.326 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-12T05:41:59.050 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-12T05:41:59.098 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-12T05:41:59.132 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-12T05:41:59.180 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-12T05:41:59.267 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-12T05:41:59.299 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-12T05:41:59.486 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-12T05:41:59.522 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-12T05:41:59.600 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-12T05:42:00.235 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-12T05:42:00.352 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-12T05:42:01.568 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-12T05:42:01.647 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-12T05:42:01.678 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-12T05:42:01.709 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-12T05:42:01.737 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-12T05:42:02.047 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-12T05:42:02.075 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-12T05:42:02.174 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-12T05:42:02.205 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-12T05:42:02.241 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-12T05:42:02.699 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-12T05:42:02.728 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-12T05:42:02.782 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-12T05:42:02.805 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-12T05:42:04.958 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T05:42:05.001 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T05:42:05.540 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T05:42:05.579 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-12T05:42:05.652 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-12T05:42:05.652 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-12T05:42:05.652 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-12T05:42:05.652 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:42:05.689 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-12T05:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-12T05:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /sys 2024-03-12T05:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /proc 2024-03-12T05:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /mnt 2024-03-12T05:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /var/tmp 2024-03-12T05:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /home 2024-03-12T05:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /root 2024-03-12T05:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /tmp 2024-03-12T05:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:42:17.549 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-12T05:42:17.564 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-12T05:42:17.565 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:42:17.565 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-12T05:42:17.565 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:42:19.043 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-12T05:42:19.070 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-12T05:42:19.071 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:42:19.071 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-12T05:42:19.071 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T05:42:19.071 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T05:42:19.071 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:42:19.278 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-12T05:42:19.301 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-12T05:42:19.301 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:42:19.301 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-12T05:42:19.301 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T05:42:19.301 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T05:42:19.301 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:42:19.757 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-12T05:42:19.779 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-12T05:42:19.779 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:42:19.779 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-12T05:42:19.779 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T05:42:19.779 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T05:42:19.779 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:42:19.875 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-12T05:42:20.326 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-12T05:42:20.356 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-12T05:42:20.388 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-12T05:42:20.449 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-12T05:42:20.479 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-12T05:42:20.709 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-12T05:42:20.755 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-12T05:42:21.723 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-12T05:42:21.747 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-12T05:42:21.847 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-12T05:42:22.010 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-12T05:42:22.041 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-12T05:42:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-12T05:42:22.064 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:42:22.064 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-12T05:42:22.064 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T05:42:22.064 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T05:42:22.064 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:42:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-12T05:42:22.119 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-12T05:42:22.165 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-12T05:42:23.018 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-12T05:42:23.044 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-12T05:42:23.067 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-12T05:42:23.067 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:42:23.067 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-12T05:42:23.067 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T05:42:23.067 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T05:42:23.068 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:42:23.111 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-12T05:42:23.133 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-12T05:42:23.133 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:42:23.133 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-12T05:42:23.133 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:42:23.187 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T05:42:23.209 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T05:42:23.209 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:42:23.209 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-12T05:42:23.209 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-12T05:42:23.209 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-12T05:42:23.209 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:42:23.521 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-12T05:42:23.543 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-12T05:42:23.543 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:42:23.543 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-12T05:42:23.543 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T05:42:23.543 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T05:42:23.543 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:42:27.966 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-12T05:42:28.008 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-12T05:42:28.038 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-12T05:42:28.065 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-12T05:42:28.250 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-12T05:42:28.364 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-12T05:42:28.405 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-12T05:42:28.444 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-12T05:42:28.496 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-12T05:42:28.531 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-12T05:42:28.555 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-12T05:42:28.587 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-12T05:42:28.617 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-12T05:42:28.671 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-12T05:42:28.732 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-12T05:42:28.766 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-12T05:42:29.360 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T05:42:29.405 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T05:42:29.425 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T05:42:30.247 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-12T05:42:30.333 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-12T05:42:30.374 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-12T05:42:30.455 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-12T05:42:30.512 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-12T05:42:30.537 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-12T05:42:30.562 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-12T05:42:30.588 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-12T05:42:30.588 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-12T05:42:30.668 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-12T05:42:30.697 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-12T05:42:30.698 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-12T05:42:30.782 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-12T05:42:30.809 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-12T05:42:30.842 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-12T05:42:30.867 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-12T05:42:30.892 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-12T05:42:30.925 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-12T05:42:30.964 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-12T05:42:30.997 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-12T05:42:31.022 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-12T05:42:31.047 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-12T05:42:31.072 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-12T05:42:31.089 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-12T05:42:31.096 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T05:42:31.096 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T05:42:31.278 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T05:42:31.279 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-12T05:42:31.361 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-12T05:42:31.383 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-12T05:42:31.408 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-12T05:42:31.442 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-12T05:42:31.477 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-12T05:42:31.541 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-12T05:42:36.396 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-12T05:42:36.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-12T05:42:36.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-12T05:42:36.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-12T05:42:36.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-12T05:42:36.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-12T05:42:36.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-12T05:42:36.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-12T05:42:36.396 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-12T05:42:36.397 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-12T05:42:36.397 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T05:42:36.397 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T05:42:36.397 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-12T05:42:36.397 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-12T05:42:36.397 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-12T05:42:36.397 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-12T05:42:36.397 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T05:42:36.397 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-12T05:42:36.397 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-12T05:42:36.397 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-12T05:42:36.397 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-12T05:42:36.397 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-12T05:42:36.397 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-12T05:42:36.398 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-12T05:42:36.398 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-12T05:42:36.398 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-12T05:42:36.398 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-12T05:42:36.398 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-12T05:42:36.398 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-12T05:42:36.398 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-12T05:42:36.398 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-12T05:42:36.414 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-12T05:42:36.414 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-12T05:42:36.415 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-12T05:42:36.415 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-12T05:42:36.415 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-12T05:42:36.415 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-12T05:42:36.415 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-12T05:42:36.415 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-12T05:42:36.415 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-12T05:42:36.415 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-12T05:42:36.415 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-12T05:42:36.415 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-12T05:42:36.415 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-12T05:42:36.415 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-12T05:42:36.416 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-12T05:42:36.416 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-12T05:42:36.416 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-12T05:42:36.416 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-12T05:42:36.416 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-12T05:42:36.416 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-12T05:42:36.416 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-12T05:42:36.416 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-12T05:42:36.416 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-12T05:42:36.416 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-12T05:42:36.416 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-12T05:42:36.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-12T05:42:36.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-12T05:42:36.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-12T05:42:36.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-12T05:42:36.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-12T05:42:36.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-12T05:42:36.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-12T05:42:36.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-12T05:42:36.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-12T05:42:36.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-12T05:42:36.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-12T05:42:36.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-12T05:42:36.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-12T05:42:36.417 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-12T05:42:36.418 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-12T05:42:36.418 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-12T05:42:36.418 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-12T05:42:36.418 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-12T05:42:36.418 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-12T05:42:36.418 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-12T05:42:36.418 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-12T05:42:36.418 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-12T05:42:36.418 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-12T05:42:36.418 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-12T05:42:36.418 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-12T05:42:36.419 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-12T05:42:36.419 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-12T05:42:36.419 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-12T05:42:36.419 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-12T05:42:36.419 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-12T05:42:36.419 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-12T05:42:36.419 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-12T05:42:36.419 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-12T05:42:36.419 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-12T05:42:36.419 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-12T05:42:36.419 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-12T05:42:36.419 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-12T05:42:36.420 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-12T05:42:36.420 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-12T05:42:36.420 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-12T05:42:36.420 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-12T05:42:36.420 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-12T05:42:36.420 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-12T05:42:36.420 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-12T05:42:36.420 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-12T05:42:36.420 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-12T05:42:36.420 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-12T05:42:36.420 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-12T05:42:36.420 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-12T05:42:36.420 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-12T05:42:36.421 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-12T05:42:36.421 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-12T05:42:36.421 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-12T05:42:36.421 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-12T05:42:36.421 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-12T05:42:36.421 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-12T05:42:36.421 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-12T05:42:36.421 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-12T05:42:36.421 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-12T05:42:36.421 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-12T05:42:36.421 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-12T05:42:36.421 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-12T05:42:36.421 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-12T05:42:36.421 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-12T05:42:36.422 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-12T05:42:36.422 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-12T05:42:36.422 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-12T05:42:36.422 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-12T05:42:36.422 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-12T05:42:36.422 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-12T05:42:36.422 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-12T05:42:36.422 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-12T05:42:36.422 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-12T05:42:36.422 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-12T05:42:36.422 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-12T05:42:36.422 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-12T05:42:36.422 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-12T05:42:36.423 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-12T05:42:36.423 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-12T05:42:36.423 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-12T05:42:36.423 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-12T05:42:36.423 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-12T05:42:36.423 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-12T05:42:36.423 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-12T05:42:36.423 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-12T05:42:36.423 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-12T05:42:36.423 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-12T05:42:36.423 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-12T05:42:36.423 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-12T05:42:36.423 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-12T05:42:36.424 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-12T05:42:36.424 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-12T05:42:36.424 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-12T05:42:36.424 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-12T05:42:36.424 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-12T05:42:36.424 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T05:42:36.424 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-12T05:42:36.424 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-12T05:42:36.424 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-12T05:42:36.424 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-12T05:42:36.424 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-12T05:42:36.424 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-12T05:42:36.424 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-12T05:42:36.425 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-12T05:42:36.425 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-12T05:42:36.425 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-12T05:42:36.425 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-12T05:42:36.425 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-12T05:42:36.425 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-12T05:42:36.425 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-12T05:42:36.425 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-12T05:42:36.425 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-12T05:42:36.425 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-12T05:42:36.425 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-12T05:42:36.425 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-12T05:42:36.425 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-12T05:42:36.425 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-12T05:42:36.426 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-12T05:42:36.426 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-12T05:42:36.426 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-12T05:42:36.426 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-12T05:42:36.426 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-12T05:42:36.426 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-12T05:42:36.426 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-12T05:42:36.426 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-12T05:42:36.426 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-12T05:42:36.426 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-12T05:42:36.426 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-12T05:42:36.426 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-12T05:42:36.426 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-12T05:42:36.427 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-12T05:42:36.427 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-12T05:42:36.427 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-12T05:42:36.427 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-12T05:42:36.427 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-12T05:42:36.427 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-12T05:42:36.427 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-12T05:42:36.427 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-12T05:42:36.427 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-12T05:42:36.427 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-12T05:42:36.427 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-12T05:42:36.427 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-12T05:42:39.424 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-12T05:42:39.425 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:42:39.425 INFO:teuthology.orchestra.run.smithi088.stdout:Upgraded: 2024-03-12T05:42:39.425 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-12T05:42:39.425 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-12T05:42:39.425 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-12T05:42:39.425 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-12T05:42:39.425 INFO:teuthology.orchestra.run.smithi088.stdout: acl-2.3.1-4.el9.x86_64 2024-03-12T05:42:39.425 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-12T05:42:39.425 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-12T05:42:39.425 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-12T05:42:39.425 INFO:teuthology.orchestra.run.smithi088.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-12T05:42:39.426 INFO:teuthology.orchestra.run.smithi088.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-12T05:42:39.426 INFO:teuthology.orchestra.run.smithi088.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-12T05:42:39.426 INFO:teuthology.orchestra.run.smithi088.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-12T05:42:39.426 INFO:teuthology.orchestra.run.smithi088.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-12T05:42:39.426 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.426 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.426 INFO:teuthology.orchestra.run.smithi088.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-12T05:42:39.426 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-12T05:42:39.426 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-12T05:42:39.426 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-12T05:42:39.426 INFO:teuthology.orchestra.run.smithi088.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-12T05:42:39.426 INFO:teuthology.orchestra.run.smithi088.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-12T05:42:39.426 INFO:teuthology.orchestra.run.smithi088.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-12T05:42:39.427 INFO:teuthology.orchestra.run.smithi088.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-12T05:42:39.427 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-12T05:42:39.427 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-03-12T05:42:39.427 INFO:teuthology.orchestra.run.smithi088.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-12T05:42:39.427 INFO:teuthology.orchestra.run.smithi088.stdout: automake-1.16.2-8.el9.noarch 2024-03-12T05:42:39.427 INFO:teuthology.orchestra.run.smithi088.stdout: bison-3.7.4-5.el9.x86_64 2024-03-12T05:42:39.427 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-12T05:42:39.427 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.427 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.427 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.427 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.427 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:42:39.427 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.428 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.428 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.428 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:42:39.428 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:42:39.428 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:42:39.428 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:42:39.428 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:42:39.428 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.428 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.428 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:42:39.428 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.428 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.429 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.429 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:42:39.429 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:42:39.429 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.429 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.429 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:42:39.429 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-12T05:42:39.429 INFO:teuthology.orchestra.run.smithi088.stdout: flex-2.6.4-9.el9.x86_64 2024-03-12T05:42:39.429 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-12T05:42:39.429 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-12T05:42:39.429 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-12T05:42:39.429 INFO:teuthology.orchestra.run.smithi088.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-12T05:42:39.429 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-12T05:42:39.429 INFO:teuthology.orchestra.run.smithi088.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-12T05:42:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: jq-1.6-16.el9.x86_64 2024-03-12T05:42:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-12T05:42:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-12T05:42:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-12T05:42:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-12T05:42:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-12T05:42:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-12T05:42:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-12T05:42:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-12T05:42:39.430 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: lua-5.4.4-4.el9.x86_64 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-12T05:42:39.431 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-12T05:42:39.432 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-12T05:42:39.432 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-13.11-1.el9.x86_64 2024-03-12T05:42:39.432 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-12T05:42:39.432 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-12T05:42:39.432 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-12T05:42:39.432 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-12T05:42:39.432 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-12T05:42:39.432 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-12T05:42:39.432 INFO:teuthology.orchestra.run.smithi088.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-12T05:42:39.432 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-12T05:42:39.432 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-12T05:42:39.432 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-12T05:42:39.432 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.433 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.433 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.433 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-12T05:42:39.433 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-12T05:42:39.433 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-12T05:42:39.433 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-12T05:42:39.433 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-12T05:42:39.433 INFO:teuthology.orchestra.run.smithi088.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-12T05:42:39.433 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-12T05:42:39.433 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-12T05:42:39.433 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-12T05:42:39.433 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-12T05:42:39.433 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-12T05:42:39.433 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-12T05:42:39.434 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-12T05:42:39.434 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-12T05:42:39.434 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-12T05:42:39.434 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-12T05:42:39.434 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-12T05:42:39.434 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-12T05:42:39.434 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-12T05:42:39.434 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-12T05:42:39.434 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-12T05:42:39.434 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-12T05:42:39.434 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-12T05:42:39.434 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-12T05:42:39.434 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply-3.11-14.el9.noarch 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-12T05:42:39.435 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-12T05:42:39.436 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.436 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.436 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-12T05:42:39.436 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-12T05:42:39.436 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-12T05:42:39.436 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.436 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-12T05:42:39.436 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-12T05:42:39.436 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-12T05:42:39.436 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-12T05:42:39.436 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-12T05:42:39.436 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-12T05:42:39.436 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-12T05:42:39.436 INFO:teuthology.orchestra.run.smithi088.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-12T05:42:39.437 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-12T05:42:39.437 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-12T05:42:39.437 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-12T05:42:39.437 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-12T05:42:39.437 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-12T05:42:39.437 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-12T05:42:39.437 INFO:teuthology.orchestra.run.smithi088.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-12T05:42:39.437 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.437 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.437 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:42:39.437 INFO:teuthology.orchestra.run.smithi088.stdout: re2-1:20211101-3.el9.x86_64 2024-03-12T05:42:39.437 INFO:teuthology.orchestra.run.smithi088.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-12T05:42:39.437 INFO:teuthology.orchestra.run.smithi088.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-12T05:42:39.437 INFO:teuthology.orchestra.run.smithi088.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-12T05:42:39.438 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-12T05:42:39.438 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-12T05:42:39.438 INFO:teuthology.orchestra.run.smithi088.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-12T05:42:39.438 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:42:39.438 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-03-12T05:42:39.878 DEBUG:teuthology.parallel:result is None 2024-03-12T05:42:42.400 INFO:teuthology.orchestra.run.smithi082.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 2.8 kB/s | 187 kB 01:05 2024-03-12T05:42:42.434 INFO:teuthology.orchestra.run.smithi082.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 3.0 kB/s | 195 kB 01:05 2024-03-12T05:42:42.484 INFO:teuthology.orchestra.run.smithi082.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 12 kB/s | 803 kB 01:05 2024-03-12T05:42:42.527 INFO:teuthology.orchestra.run.smithi082.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 1.0 MB/s | 125 kB 00:00 2024-03-12T05:42:42.568 INFO:teuthology.orchestra.run.smithi082.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 161 kB/s | 21 kB 00:00 2024-03-12T05:42:42.585 INFO:teuthology.orchestra.run.smithi082.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 254 kB/s | 25 kB 00:00 2024-03-12T05:42:42.702 INFO:teuthology.orchestra.run.smithi082.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-03-12T05:42:42.736 INFO:teuthology.orchestra.run.smithi082.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.6 MB/s | 248 kB 00:00 2024-03-12T05:42:42.786 INFO:teuthology.orchestra.run.smithi082.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.5 MB/s | 216 kB 00:00 2024-03-12T05:42:42.811 INFO:teuthology.orchestra.run.smithi082.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 465 kB/s | 35 kB 00:00 2024-03-12T05:42:42.836 INFO:teuthology.orchestra.run.smithi082.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-12T05:42:42.862 INFO:teuthology.orchestra.run.smithi082.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-12T05:42:42.928 INFO:teuthology.orchestra.run.smithi082.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-12T05:42:43.062 INFO:teuthology.orchestra.run.smithi082.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.2 MB/s | 863 kB 00:00 2024-03-12T05:42:43.512 INFO:teuthology.orchestra.run.smithi082.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.2 MB/s | 4.1 MB 00:00 2024-03-12T05:42:43.687 INFO:teuthology.orchestra.run.smithi082.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 111 kB/s | 19 kB 00:00 2024-03-12T05:42:46.537 INFO:teuthology.orchestra.run.smithi082.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 89 kB/s | 309 kB 00:03 2024-03-12T05:42:46.887 INFO:teuthology.orchestra.run.smithi082.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 74 kB/s | 26 kB 00:00 2024-03-12T05:42:47.438 INFO:teuthology.orchestra.run.smithi082.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 88 kB/s | 49 kB 00:00 2024-03-12T05:42:47.746 INFO:teuthology.orchestra.run.smithi082.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 218 kB/s | 67 kB 00:00 2024-03-12T05:42:47.980 INFO:teuthology.orchestra.run.smithi082.stdout:(112/174): abseil-cpp-20211102.0-3.el9.x86_64.r 108 kB/s | 548 kB 00:05 2024-03-12T05:42:48.038 INFO:teuthology.orchestra.run.smithi082.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 518 kB/s | 151 kB 00:00 2024-03-12T05:42:54.198 INFO:teuthology.orchestra.run.smithi082.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 89 kB/s | 548 kB 00:06 2024-03-12T05:42:54.762 INFO:teuthology.orchestra.run.smithi082.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 77 kB/s | 43 kB 00:00 2024-03-12T05:42:55.288 INFO:teuthology.orchestra.run.smithi082.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 61 kB/s | 32 kB 00:00 2024-03-12T05:42:55.738 INFO:teuthology.orchestra.run.smithi082.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 33 kB/s | 15 kB 00:00 2024-03-12T05:42:57.288 INFO:teuthology.orchestra.run.smithi082.stdout:(118/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 90 kB/s | 838 kB 00:09 2024-03-12T05:42:57.830 INFO:teuthology.orchestra.run.smithi082.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 82 kB/s | 172 kB 00:02 2024-03-12T05:42:58.996 INFO:teuthology.orchestra.run.smithi082.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 260 kB/s | 303 kB 00:01 2024-03-12T05:42:59.171 INFO:teuthology.orchestra.run.smithi082.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 190 kB/s | 358 kB 00:01 2024-03-12T05:42:59.455 INFO:teuthology.orchestra.run.smithi082.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 77 kB/s | 35 kB 00:00 2024-03-12T05:43:01.522 INFO:teuthology.orchestra.run.smithi082.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 86 kB/s | 203 kB 00:02 2024-03-12T05:43:03.263 INFO:teuthology.orchestra.run.smithi082.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 83 kB/s | 144 kB 00:01 2024-03-12T05:43:03.488 INFO:teuthology.orchestra.run.smithi082.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 47 kB/s | 11 kB 00:00 2024-03-12T05:43:03.764 INFO:teuthology.orchestra.run.smithi082.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 64 kB/s | 18 kB 00:00 2024-03-12T05:43:04.105 INFO:teuthology.orchestra.run.smithi082.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 68 kB/s | 23 kB 00:00 2024-03-12T05:43:04.389 INFO:teuthology.orchestra.run.smithi082.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 69 kB/s | 19 kB 00:00 2024-03-12T05:43:04.689 INFO:teuthology.orchestra.run.smithi082.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 66 kB/s | 20 kB 00:00 2024-03-12T05:43:16.113 INFO:teuthology.orchestra.run.smithi082.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 92 kB/s | 1.0 MB 00:11 2024-03-12T05:43:16.663 INFO:teuthology.orchestra.run.smithi082.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 84 kB/s | 46 kB 00:00 2024-03-12T05:43:17.654 INFO:teuthology.orchestra.run.smithi082.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 79 kB/s | 79 kB 00:00 2024-03-12T05:43:18.355 INFO:teuthology.orchestra.run.smithi082.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 82 kB/s | 58 kB 00:00 2024-03-12T05:43:19.621 INFO:teuthology.orchestra.run.smithi082.stdout:(134/174): libarrow-9.0.0-10.el9.x86_64.rpm 126 kB/s | 4.4 MB 00:35 2024-03-12T05:43:19.821 INFO:teuthology.orchestra.run.smithi082.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 186 kB/s | 272 kB 00:01 2024-03-12T05:43:20.080 INFO:teuthology.orchestra.run.smithi082.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 36 kB/s | 16 kB 00:00 2024-03-12T05:43:20.399 INFO:teuthology.orchestra.run.smithi082.stdout:(137/174): python3-pyperclip-1.8.2-2.el9.noarch 88 kB/s | 28 kB 00:00 2024-03-12T05:43:20.699 INFO:teuthology.orchestra.run.smithi082.stdout:(138/174): python3-repoze-lru-0.7-16.el9.noarch 102 kB/s | 31 kB 00:00 2024-03-12T05:43:20.733 INFO:teuthology.orchestra.run.smithi082.stdout:(139/174): python3-grpcio-1.46.7-10.el9.x86_64. 98 kB/s | 2.0 MB 00:21 2024-03-12T05:43:20.875 INFO:teuthology.orchestra.run.smithi082.stdout:(140/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 85 kB/s | 90 kB 00:01 2024-03-12T05:43:21.308 INFO:teuthology.orchestra.run.smithi082.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 103 kB/s | 59 kB 00:00 2024-03-12T05:43:21.334 INFO:teuthology.orchestra.run.smithi082.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 78 kB/s | 36 kB 00:00 2024-03-12T05:43:21.909 INFO:teuthology.orchestra.run.smithi082.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 85 kB/s | 51 kB 00:00 2024-03-12T05:43:22.834 INFO:teuthology.orchestra.run.smithi082.stdout:(144/174): python3-routes-2.5.1-5.el9.noarch.rp 88 kB/s | 188 kB 00:02 2024-03-12T05:43:22.976 INFO:teuthology.orchestra.run.smithi082.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 84 kB/s | 90 kB 00:01 2024-03-12T05:43:23.401 INFO:teuthology.orchestra.run.smithi082.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 111 kB/s | 230 kB 00:02 2024-03-12T05:43:23.502 INFO:teuthology.orchestra.run.smithi082.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 38 kB/s | 20 kB 00:00 2024-03-12T05:43:25.210 INFO:teuthology.orchestra.run.smithi082.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 106 kB/s | 192 kB 00:01 2024-03-12T05:43:25.801 INFO:teuthology.orchestra.run.smithi082.stdout:(149/174): librados2-19.0.0-1607.g94a53970.el9. 5.7 MB/s | 3.4 MB 00:00 2024-03-12T05:43:26.043 INFO:teuthology.orchestra.run.smithi082.stdout:(150/174): librbd1-19.0.0-1607.g94a53970.el9.x8 13 MB/s | 3.1 MB 00:00 2024-03-12T05:43:26.086 INFO:teuthology.orchestra.run.smithi082.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-12T05:43:26.602 INFO:teuthology.orchestra.run.smithi082.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 4.4 MB/s | 2.3 MB 00:00 2024-03-12T05:43:26.844 INFO:teuthology.orchestra.run.smithi082.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 7.5 MB/s | 1.8 MB 00:00 2024-03-12T05:43:26.936 INFO:teuthology.orchestra.run.smithi082.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 463 kB/s | 42 kB 00:00 2024-03-12T05:43:27.053 INFO:teuthology.orchestra.run.smithi082.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.1 MB/s | 248 kB 00:00 2024-03-12T05:43:27.128 INFO:teuthology.orchestra.run.smithi082.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 952 kB/s | 71 kB 00:00 2024-03-12T05:43:27.195 INFO:teuthology.orchestra.run.smithi082.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 547 kB/s | 36 kB 00:00 2024-03-12T05:43:27.287 INFO:teuthology.orchestra.run.smithi082.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-03-12T05:43:27.415 INFO:teuthology.orchestra.run.smithi082.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.0 MB/s | 256 kB 00:00 2024-03-12T05:43:27.470 INFO:teuthology.orchestra.run.smithi082.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 414 kB/s | 23 kB 00:00 2024-03-12T05:43:27.546 INFO:teuthology.orchestra.run.smithi082.stdout:(161/174): python3-werkzeug-2.0.3-3.el9.1.noarc 91 kB/s | 427 kB 00:04 2024-03-12T05:43:27.571 INFO:teuthology.orchestra.run.smithi082.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-03-12T05:43:27.655 INFO:teuthology.orchestra.run.smithi082.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-12T05:43:27.680 INFO:teuthology.orchestra.run.smithi082.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-12T05:43:27.730 INFO:teuthology.orchestra.run.smithi082.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 851 kB/s | 64 kB 00:00 2024-03-12T05:43:27.756 INFO:teuthology.orchestra.run.smithi082.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 380 kB/s | 29 kB 00:00 2024-03-12T05:43:27.973 INFO:teuthology.orchestra.run.smithi082.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 5.0 MB/s | 1.2 MB 00:00 2024-03-12T05:43:28.048 INFO:teuthology.orchestra.run.smithi082.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.4 MB/s | 2.2 MB 00:00 2024-03-12T05:43:28.190 INFO:teuthology.orchestra.run.smithi082.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.2 MB/s | 465 kB 00:00 2024-03-12T05:43:28.249 INFO:teuthology.orchestra.run.smithi082.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 535 kB/s | 31 kB 00:00 2024-03-12T05:43:28.382 INFO:teuthology.orchestra.run.smithi082.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 5.6 MB/s | 2.3 MB 00:00 2024-03-12T05:43:28.491 INFO:teuthology.orchestra.run.smithi082.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 4.4 MB/s | 1.1 MB 00:00 2024-03-12T05:43:28.524 INFO:teuthology.orchestra.run.smithi082.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 524 kB/s | 74 kB 00:00 2024-03-12T05:43:40.881 INFO:teuthology.orchestra.run.smithi082.stdout:(174/174): thrift-0.15.0-2.el9.x86_64.rpm 93 kB/s | 1.6 MB 00:17 2024-03-12T05:43:40.885 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-03-12T05:43:40.885 INFO:teuthology.orchestra.run.smithi082.stdout:Total 1.7 MB/s | 223 MB 02:13 2024-03-12T05:43:41.333 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-03-12T05:43:41.600 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-03-12T05:43:41.600 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-03-12T05:43:43.296 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-03-12T05:43:43.297 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-03-12T05:43:50.149 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-03-12T05:43:50.258 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-12T05:43:50.284 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-12T05:43:50.315 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-12T05:43:50.323 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-12T05:43:50.364 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-12T05:43:50.417 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-12T05:43:50.452 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-12T05:43:50.486 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-12T05:43:50.528 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-12T05:43:50.844 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-12T05:43:50.887 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T05:43:51.007 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T05:43:51.038 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T05:43:51.129 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T05:43:51.159 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-12T05:43:51.198 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-12T05:43:51.356 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-12T05:43:51.407 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-12T05:43:51.435 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T05:43:51.521 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T05:43:51.544 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-12T05:43:51.625 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-12T05:43:51.673 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-12T05:43:51.696 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-12T05:43:51.711 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-12T05:43:51.854 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-12T05:43:51.885 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-12T05:43:51.917 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-12T05:43:52.087 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-12T05:43:52.303 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-12T05:43:52.355 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-12T05:43:52.439 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T05:43:52.506 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T05:43:52.725 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-12T05:43:52.842 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-12T05:43:52.933 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-12T05:43:52.969 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-12T05:43:52.997 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-12T05:43:53.032 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-12T05:43:53.115 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-12T05:43:53.169 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-12T05:43:53.207 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-12T05:43:53.271 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-12T05:43:53.306 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-12T05:43:53.350 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-12T05:43:53.382 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-12T05:43:53.410 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-12T05:43:53.479 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-12T05:43:53.515 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-12T05:43:53.549 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-12T05:43:53.578 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-12T05:43:53.646 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-12T05:43:53.693 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-12T05:43:53.766 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-12T05:43:53.900 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-12T05:43:53.953 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-12T05:43:54.470 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-12T05:43:54.557 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-12T05:43:54.630 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-12T05:43:54.669 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-12T05:43:54.678 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-12T05:43:54.709 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-12T05:43:54.750 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-12T05:43:54.788 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-12T05:43:54.818 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-12T05:43:55.059 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-12T05:43:55.234 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-12T05:43:55.295 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-12T05:43:55.338 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-12T05:43:55.473 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-12T05:43:55.529 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-12T05:43:55.583 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-12T05:43:55.647 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-12T05:43:55.679 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-12T05:43:55.716 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-12T05:43:55.762 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-12T05:43:55.798 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-12T05:43:55.841 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-12T05:43:55.884 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-12T05:43:56.018 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-12T05:43:56.064 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-12T05:43:56.103 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-12T05:43:56.145 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-12T05:43:56.240 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-12T05:43:56.278 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-12T05:43:56.318 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-12T05:43:56.344 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-12T05:43:56.368 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-12T05:43:56.441 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-12T05:43:56.570 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-12T05:43:56.677 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-12T05:43:56.719 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-12T05:43:56.767 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-12T05:43:56.822 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-12T05:43:57.007 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-12T05:43:57.056 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-12T05:43:57.194 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-12T05:43:57.229 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-12T05:43:57.261 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-12T05:43:57.292 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-12T05:43:57.320 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-12T05:43:57.354 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-12T05:43:57.393 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-12T05:43:57.425 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-12T05:43:57.455 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-12T05:43:57.531 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-12T05:43:57.566 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-12T05:43:57.615 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-12T05:43:57.657 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-12T05:43:57.691 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-12T05:43:57.780 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-12T05:43:57.862 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-12T05:43:57.906 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-12T05:43:57.937 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-12T05:43:57.984 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-12T05:43:58.083 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-12T05:43:58.148 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-12T05:43:58.889 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-12T05:43:58.936 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-12T05:43:58.970 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-12T05:43:59.019 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-12T05:43:59.106 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-12T05:43:59.146 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-12T05:43:59.323 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-12T05:43:59.360 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-12T05:43:59.438 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-12T05:44:00.093 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-12T05:44:00.200 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-12T05:44:01.439 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-12T05:44:01.519 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-12T05:44:01.550 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-12T05:44:01.573 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-12T05:44:01.593 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-12T05:44:01.902 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-12T05:44:01.931 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-12T05:44:02.021 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-12T05:44:02.052 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-12T05:44:02.088 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-12T05:44:02.544 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-12T05:44:02.576 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-12T05:44:02.638 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-12T05:44:02.661 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-12T05:44:04.780 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T05:44:04.815 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T05:44:05.349 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-12T05:44:05.410 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-12T05:44:05.492 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-12T05:44:05.492 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-12T05:44:05.492 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-12T05:44:05.492 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:44:05.520 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-12T05:44:17.389 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-12T05:44:17.389 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /sys 2024-03-12T05:44:17.390 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /proc 2024-03-12T05:44:17.390 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /mnt 2024-03-12T05:44:17.390 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /var/tmp 2024-03-12T05:44:17.390 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /home 2024-03-12T05:44:17.390 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /root 2024-03-12T05:44:17.390 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /tmp 2024-03-12T05:44:17.390 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:44:17.557 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-12T05:44:17.572 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-12T05:44:17.572 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:44:17.572 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-12T05:44:17.572 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:44:19.038 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-12T05:44:19.063 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-12T05:44:19.063 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:44:19.064 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-12T05:44:19.064 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T05:44:19.064 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T05:44:19.064 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:44:19.272 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-12T05:44:19.294 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-12T05:44:19.294 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:44:19.294 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-12T05:44:19.294 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T05:44:19.294 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T05:44:19.294 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:44:19.732 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-12T05:44:19.753 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-12T05:44:19.754 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:44:19.754 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-12T05:44:19.754 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T05:44:19.754 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T05:44:19.754 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:44:19.850 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-12T05:44:20.251 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-12T05:44:20.282 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-12T05:44:20.313 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-12T05:44:20.372 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-12T05:44:20.405 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-12T05:44:20.634 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-12T05:44:20.664 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-12T05:44:21.620 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-12T05:44:21.647 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-12T05:44:21.748 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-12T05:44:21.909 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-12T05:44:21.933 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-12T05:44:21.955 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-12T05:44:21.955 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:44:21.955 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-12T05:44:21.955 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T05:44:21.956 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T05:44:21.956 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:44:22.000 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-12T05:44:22.011 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-12T05:44:22.056 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-12T05:44:22.917 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-12T05:44:22.944 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-12T05:44:22.966 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-12T05:44:22.966 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:44:22.966 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-12T05:44:22.966 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T05:44:22.966 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T05:44:22.966 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:44:23.003 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-12T05:44:23.025 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-12T05:44:23.025 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:44:23.025 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-12T05:44:23.025 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:44:23.071 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T05:44:23.093 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T05:44:23.093 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:44:23.093 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-12T05:44:23.093 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-12T05:44:23.093 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-12T05:44:23.093 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:44:23.421 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-12T05:44:23.443 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-12T05:44:23.443 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T05:44:23.443 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-12T05:44:23.444 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T05:44:23.444 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T05:44:23.444 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:44:27.850 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-12T05:44:27.891 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-12T05:44:27.930 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-12T05:44:27.958 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-12T05:44:28.140 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-12T05:44:28.256 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-12T05:44:28.297 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-12T05:44:28.345 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-12T05:44:28.389 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-12T05:44:28.432 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-12T05:44:28.457 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-12T05:44:28.489 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-12T05:44:28.519 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-12T05:44:28.565 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-12T05:44:28.617 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-12T05:44:28.651 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-12T05:44:29.278 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T05:44:29.324 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T05:44:29.342 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-12T05:44:30.162 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-12T05:44:30.252 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-12T05:44:30.293 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-12T05:44:30.374 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-12T05:44:30.406 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-12T05:44:30.431 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-12T05:44:30.457 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-12T05:44:30.483 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-12T05:44:30.483 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-12T05:44:30.563 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-12T05:44:30.592 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-12T05:44:30.593 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-12T05:44:30.677 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-12T05:44:30.703 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-12T05:44:30.728 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-12T05:44:30.753 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-12T05:44:30.778 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-12T05:44:30.811 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-12T05:44:30.851 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-12T05:44:30.883 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-12T05:44:30.909 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-12T05:44:30.934 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-12T05:44:30.959 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-12T05:44:30.985 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-12T05:44:30.990 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T05:44:30.991 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T05:44:31.171 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-12T05:44:31.171 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-12T05:44:31.254 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-12T05:44:31.278 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-12T05:44:31.304 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-12T05:44:31.337 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-12T05:44:31.363 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-12T05:44:31.427 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-12T05:44:36.070 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-12T05:44:36.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-12T05:44:36.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-12T05:44:36.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-12T05:44:36.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-12T05:44:36.070 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-12T05:44:36.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-12T05:44:36.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-12T05:44:36.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-12T05:44:36.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-12T05:44:36.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-12T05:44:36.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-12T05:44:36.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-12T05:44:36.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-12T05:44:36.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-12T05:44:36.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-12T05:44:36.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-12T05:44:36.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-12T05:44:36.071 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-12T05:44:36.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-12T05:44:36.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-12T05:44:36.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-12T05:44:36.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-12T05:44:36.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-12T05:44:36.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-12T05:44:36.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-12T05:44:36.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-12T05:44:36.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-12T05:44:36.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-12T05:44:36.072 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-12T05:44:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-12T05:44:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-12T05:44:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-12T05:44:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-12T05:44:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-12T05:44:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-12T05:44:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-12T05:44:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-12T05:44:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-12T05:44:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-12T05:44:36.089 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-12T05:44:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-12T05:44:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-12T05:44:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-12T05:44:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-12T05:44:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-12T05:44:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-12T05:44:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-12T05:44:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-12T05:44:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-12T05:44:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-12T05:44:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-12T05:44:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-12T05:44:36.090 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-12T05:44:36.091 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-12T05:44:36.091 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-12T05:44:36.091 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-12T05:44:36.091 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-12T05:44:36.091 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-12T05:44:36.091 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-12T05:44:36.091 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-12T05:44:36.091 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-12T05:44:36.091 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-12T05:44:36.091 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-12T05:44:36.091 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-12T05:44:36.091 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-12T05:44:36.091 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-12T05:44:36.092 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-12T05:44:36.092 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-12T05:44:36.092 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-12T05:44:36.092 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-12T05:44:36.092 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-12T05:44:36.092 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-12T05:44:36.092 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-12T05:44:36.092 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-12T05:44:36.092 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-12T05:44:36.092 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-12T05:44:36.092 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-12T05:44:36.092 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-12T05:44:36.092 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-12T05:44:36.093 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-12T05:44:36.093 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-12T05:44:36.093 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-12T05:44:36.093 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-12T05:44:36.093 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-12T05:44:36.093 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-12T05:44:36.093 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-12T05:44:36.093 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-12T05:44:36.093 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-12T05:44:36.093 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-12T05:44:36.093 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-12T05:44:36.093 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-12T05:44:36.094 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-12T05:44:36.095 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-12T05:44:36.095 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-12T05:44:36.095 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-12T05:44:36.095 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-12T05:44:36.095 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-12T05:44:36.095 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-12T05:44:36.095 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-12T05:44:36.095 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-12T05:44:36.095 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-12T05:44:36.095 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-12T05:44:36.095 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-12T05:44:36.095 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-12T05:44:36.095 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-12T05:44:36.095 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-12T05:44:36.096 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-12T05:44:36.096 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-12T05:44:36.096 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-12T05:44:36.096 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-12T05:44:36.096 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-12T05:44:36.096 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-12T05:44:36.096 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-12T05:44:36.096 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-12T05:44:36.096 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-12T05:44:36.096 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-12T05:44:36.096 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-12T05:44:36.096 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-12T05:44:36.096 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-12T05:44:36.096 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-12T05:44:36.097 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-12T05:44:36.097 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-12T05:44:36.097 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-12T05:44:36.097 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-12T05:44:36.097 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-12T05:44:36.097 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-12T05:44:36.097 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-12T05:44:36.097 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-12T05:44:36.097 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-12T05:44:36.097 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-12T05:44:36.097 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-12T05:44:36.097 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-12T05:44:36.097 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-12T05:44:36.097 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-12T05:44:36.098 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-12T05:44:36.098 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-12T05:44:36.098 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-12T05:44:36.098 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-12T05:44:36.098 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-12T05:44:36.098 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-12T05:44:36.098 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-12T05:44:36.098 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-12T05:44:36.098 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-12T05:44:36.098 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-12T05:44:36.098 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-12T05:44:36.098 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-12T05:44:36.098 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-12T05:44:36.098 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-12T05:44:36.099 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-12T05:44:36.100 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-12T05:44:36.100 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-12T05:44:36.100 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-12T05:44:36.100 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-12T05:44:36.100 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-12T05:44:36.100 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-12T05:44:36.100 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-12T05:44:36.100 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-12T05:44:36.100 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-12T05:44:36.100 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-12T05:44:36.100 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-12T05:44:36.100 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-12T05:44:36.100 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-12T05:44:36.100 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-12T05:44:36.101 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-12T05:44:36.101 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-12T05:44:36.101 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-12T05:44:36.101 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-12T05:44:36.101 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-12T05:44:36.101 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-12T05:44:39.223 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-12T05:44:39.223 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:44:39.223 INFO:teuthology.orchestra.run.smithi082.stdout:Upgraded: 2024-03-12T05:44:39.223 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-12T05:44:39.223 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-12T05:44:39.223 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-12T05:44:39.223 INFO:teuthology.orchestra.run.smithi082.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-12T05:44:39.223 INFO:teuthology.orchestra.run.smithi082.stdout: acl-2.3.1-4.el9.x86_64 2024-03-12T05:44:39.223 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-12T05:44:39.223 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-12T05:44:39.223 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-12T05:44:39.224 INFO:teuthology.orchestra.run.smithi082.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-12T05:44:39.224 INFO:teuthology.orchestra.run.smithi082.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-12T05:44:39.224 INFO:teuthology.orchestra.run.smithi082.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-12T05:44:39.224 INFO:teuthology.orchestra.run.smithi082.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-12T05:44:39.224 INFO:teuthology.orchestra.run.smithi082.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-12T05:44:39.224 INFO:teuthology.orchestra.run.smithi082.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.224 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.224 INFO:teuthology.orchestra.run.smithi082.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-12T05:44:39.224 INFO:teuthology.orchestra.run.smithi082.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-12T05:44:39.224 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-12T05:44:39.224 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-12T05:44:39.224 INFO:teuthology.orchestra.run.smithi082.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-12T05:44:39.224 INFO:teuthology.orchestra.run.smithi082.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-12T05:44:39.224 INFO:teuthology.orchestra.run.smithi082.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-12T05:44:39.225 INFO:teuthology.orchestra.run.smithi082.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-12T05:44:39.225 INFO:teuthology.orchestra.run.smithi082.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-12T05:44:39.225 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-03-12T05:44:39.225 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-12T05:44:39.225 INFO:teuthology.orchestra.run.smithi082.stdout: automake-1.16.2-8.el9.noarch 2024-03-12T05:44:39.225 INFO:teuthology.orchestra.run.smithi082.stdout: bison-3.7.4-5.el9.x86_64 2024-03-12T05:44:39.225 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-12T05:44:39.225 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.225 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.225 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.225 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.225 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:44:39.225 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.226 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.226 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.226 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:44:39.226 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:44:39.226 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:44:39.226 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:44:39.226 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:44:39.226 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.226 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.226 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:44:39.226 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.226 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.226 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.226 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:44:39.227 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:44:39.227 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.227 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.227 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-12T05:44:39.227 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-12T05:44:39.227 INFO:teuthology.orchestra.run.smithi082.stdout: flex-2.6.4-9.el9.x86_64 2024-03-12T05:44:39.227 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-12T05:44:39.227 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-12T05:44:39.227 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-12T05:44:39.227 INFO:teuthology.orchestra.run.smithi082.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-12T05:44:39.227 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-12T05:44:39.227 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-12T05:44:39.227 INFO:teuthology.orchestra.run.smithi082.stdout: jq-1.6-16.el9.x86_64 2024-03-12T05:44:39.227 INFO:teuthology.orchestra.run.smithi082.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-12T05:44:39.228 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-12T05:44:39.228 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-12T05:44:39.228 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.228 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.228 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.228 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-12T05:44:39.228 INFO:teuthology.orchestra.run.smithi082.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-12T05:44:39.228 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-12T05:44:39.228 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-12T05:44:39.228 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-12T05:44:39.228 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.228 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.228 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-12T05:44:39.228 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: lua-5.4.4-4.el9.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-13.11-1.el9.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-12T05:44:39.229 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-12T05:44:39.230 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-12T05:44:39.230 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-12T05:44:39.230 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-12T05:44:39.230 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-12T05:44:39.230 INFO:teuthology.orchestra.run.smithi082.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-12T05:44:39.230 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-12T05:44:39.230 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-12T05:44:39.230 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-12T05:44:39.230 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.230 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.230 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.230 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-12T05:44:39.230 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-12T05:44:39.231 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply-3.11-14.el9.noarch 2024-03-12T05:44:39.232 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-12T05:44:39.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-12T05:44:39.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-12T05:44:39.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-12T05:44:39.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-12T05:44:39.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-12T05:44:39.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-12T05:44:39.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-12T05:44:39.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-12T05:44:39.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-12T05:44:39.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-12T05:44:39.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-12T05:44:39.233 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-12T05:44:39.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-12T05:44:39.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-12T05:44:39.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-12T05:44:39.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-12T05:44:39.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-12T05:44:39.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-12T05:44:39.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-12T05:44:39.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-12T05:44:39.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-12T05:44:39.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-12T05:44:39.234 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-12T05:44:39.234 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-12T05:44:39.234 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-12T05:44:39.235 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-12T05:44:39.235 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.235 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.235 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-12T05:44:39.235 INFO:teuthology.orchestra.run.smithi082.stdout: re2-1:20211101-3.el9.x86_64 2024-03-12T05:44:39.235 INFO:teuthology.orchestra.run.smithi082.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-12T05:44:39.235 INFO:teuthology.orchestra.run.smithi082.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-12T05:44:39.235 INFO:teuthology.orchestra.run.smithi082.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-12T05:44:39.235 INFO:teuthology.orchestra.run.smithi082.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-12T05:44:39.235 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-12T05:44:39.236 INFO:teuthology.orchestra.run.smithi082.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-12T05:44:39.236 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-12T05:44:39.236 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-03-12T05:44:39.703 DEBUG:teuthology.parallel:result is None 2024-03-12T05:44:39.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T05:44:39.928 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-12T05:44:39.955 INFO:teuthology.orchestra.run.smithi076.stdout:19.0.0-1607.g94a53970.el9 2024-03-12T05:44:39.955 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-12T05:44:39.956 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-12T05:44:39.958 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T05:44:40.104 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-12T05:44:40.125 INFO:teuthology.orchestra.run.smithi082.stdout:19.0.0-1607.g94a53970.el9 2024-03-12T05:44:40.125 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-12T05:44:40.125 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-12T05:44:40.127 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T05:44:40.282 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-12T05:44:40.308 INFO:teuthology.orchestra.run.smithi088.stdout:19.0.0-1607.g94a53970.el9 2024-03-12T05:44:40.308 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-12T05:44:40.309 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-12T05:44:40.311 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-12T05:44:40.311 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:44:40.311 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-12T05:44:40.346 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-12T05:44:40.346 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-12T05:44:40.377 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-12T05:44:40.377 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-12T05:44:40.411 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-12T05:44:40.411 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:44:40.411 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/daemon-helper 2024-03-12T05:44:40.443 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-12T05:44:40.514 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-12T05:44:40.514 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2024-03-12T05:44:40.542 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-12T05:44:40.609 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-12T05:44:40.609 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2024-03-12T05:44:40.641 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-12T05:44:40.713 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-12T05:44:40.713 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:44:40.713 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-12T05:44:40.754 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-12T05:44:40.823 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-12T05:44:40.824 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-12T05:44:40.851 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-12T05:44:40.918 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-12T05:44:40.918 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-12T05:44:40.946 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-12T05:44:41.013 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-12T05:44:41.013 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:44:41.013 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/stdin-killer 2024-03-12T05:44:41.040 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-12T05:44:41.108 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-12T05:44:41.108 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2024-03-12T05:44:41.135 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-12T05:44:41.202 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-12T05:44:41.202 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2024-03-12T05:44:41.228 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-12T05:44:41.294 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-12T05:44:41.395 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'subvols': {'create': 2}} 2024-03-12T05:44:41.396 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T05:44:41.396 INFO:tasks.cephadm:Cluster fsid is 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:44:41.396 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-12T05:44:41.397 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.76', 'mon.b': '172.21.15.82', 'mon.c': '172.21.15.88'} 2024-03-12T05:44:41.397 INFO:tasks.cephadm:First mon is mon.a on smithi076 2024-03-12T05:44:41.397 INFO:tasks.cephadm:First mgr is x 2024-03-12T05:44:41.397 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-12T05:44:41.397 DEBUG:teuthology.orchestra.run.smithi076:> sudo hostname $(hostname -s) 2024-03-12T05:44:41.426 DEBUG:teuthology.orchestra.run.smithi082:> sudo hostname $(hostname -s) 2024-03-12T05:44:41.452 DEBUG:teuthology.orchestra.run.smithi088:> sudo hostname $(hostname -s) 2024-03-12T05:44:41.480 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-12T05:44:41.480 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T05:44:41.627 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77431/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:08:19.449750', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77429/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:06:59.696700', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-12T05:44:41.791 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 94a539707733325322964c6a187c6a1b41be1daf from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T05:44:41.792 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm 2024-03-12T05:44:41.792 DEBUG:teuthology.orchestra.run.smithi076:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T05:44:42.373 INFO:teuthology.orchestra.run.smithi076.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 12 05:44 /home/ubuntu/cephtest/cephadm 2024-03-12T05:44:42.374 DEBUG:teuthology.orchestra.run.smithi082:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T05:44:42.930 INFO:teuthology.orchestra.run.smithi082.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 12 05:44 /home/ubuntu/cephtest/cephadm 2024-03-12T05:44:42.930 DEBUG:teuthology.orchestra.run.smithi088:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T05:44:43.407 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 12 05:44 /home/ubuntu/cephtest/cephadm 2024-03-12T05:44:43.408 DEBUG:teuthology.orchestra.run.smithi076:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T05:44:43.426 DEBUG:teuthology.orchestra.run.smithi082:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T05:44:43.445 DEBUG:teuthology.orchestra.run.smithi088:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T05:44:43.481 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on all hosts... 2024-03-12T05:44:43.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-12T05:44:43.483 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-12T05:44:43.488 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-12T05:44:44.347 INFO:teuthology.orchestra.run.smithi088.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-12T05:44:44.368 INFO:teuthology.orchestra.run.smithi082.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-12T05:44:44.532 INFO:teuthology.orchestra.run.smithi076.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-12T05:45:19.199 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:45:19.200 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-12T05:45:19.200 INFO:teuthology.orchestra.run.smithi076.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-12T05:45:19.200 INFO:teuthology.orchestra.run.smithi076.stdout: "repo_digests": [ 2024-03-12T05:45:19.200 INFO:teuthology.orchestra.run.smithi076.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-12T05:45:19.200 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-12T05:45:19.200 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:45:31.930 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-03-12T05:45:31.930 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-12T05:45:31.930 INFO:teuthology.orchestra.run.smithi088.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-12T05:45:31.930 INFO:teuthology.orchestra.run.smithi088.stdout: "repo_digests": [ 2024-03-12T05:45:31.930 INFO:teuthology.orchestra.run.smithi088.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-12T05:45:31.930 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-03-12T05:45:31.931 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-03-12T05:45:36.881 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-12T05:45:36.881 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-12T05:45:36.881 INFO:teuthology.orchestra.run.smithi082.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-12T05:45:36.881 INFO:teuthology.orchestra.run.smithi082.stdout: "repo_digests": [ 2024-03-12T05:45:36.881 INFO:teuthology.orchestra.run.smithi082.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-12T05:45:36.881 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-12T05:45:36.881 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-12T05:45:36.905 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph 2024-03-12T05:45:36.939 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph 2024-03-12T05:45:36.968 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph 2024-03-12T05:45:37.002 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 777 /etc/ceph 2024-03-12T05:45:37.032 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /etc/ceph 2024-03-12T05:45:37.058 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /etc/ceph 2024-03-12T05:45:37.088 INFO:tasks.cephadm:Writing seed config... 2024-03-12T05:45:37.089 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-12T05:45:37.089 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-12T05:45:37.089 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-12T05:45:37.089 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-12T05:45:37.089 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-12T05:45:37.089 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-12T05:45:37.089 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-12T05:45:37.089 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-12T05:45:37.089 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-12T05:45:37.089 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-12T05:45:37.090 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-12T05:45:37.090 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-12T05:45:37.090 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-12T05:45:37.090 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-12T05:45:37.090 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-12T05:45:37.090 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-12T05:45:37.090 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-12T05:45:37.090 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-12T05:45:37.090 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-12T05:45:37.090 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-12T05:45:37.090 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-12T05:45:37.090 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-12T05:45:37.091 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-12T05:45:37.092 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-12T05:45:37.092 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-12T05:45:37.092 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-12T05:45:37.092 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-12T05:45:37.092 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-12T05:45:37.092 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-12T05:45:37.092 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-12T05:45:37.092 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-12T05:45:37.092 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-12T05:45:37.093 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:45:37.093 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-12T05:45:37.109 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9f336de8-e033-11ee-95c8-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-12T05:45:37.109 DEBUG:teuthology.orchestra.run.smithi076:mon.a> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mon.a.service 2024-03-12T05:45:37.154 DEBUG:teuthology.orchestra.run.smithi076:mgr.x> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mgr.x.service 2024-03-12T05:45:37.198 INFO:tasks.cephadm:Bootstrapping... 2024-03-12T05:45:37.199 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf -v bootstrap --fsid 9f336de8-e033-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.76 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-12T05:45:37.391 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-12T05:45:37.392 INFO:teuthology.orchestra.run.smithi076.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf', '-v', 'bootstrap', '--fsid', '9f336de8-e033-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.76', '--skip-admin-label'] 2024-03-12T05:45:37.414 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T05:45:37.415 INFO:teuthology.orchestra.run.smithi076.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-12T05:45:37.415 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying podman|docker is present... 2024-03-12T05:45:37.439 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T05:45:37.439 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying lvm2 is present... 2024-03-12T05:45:37.439 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying time synchronization is in place... 2024-03-12T05:45:37.446 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-12T05:45:37.446 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T05:45:37.452 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-12T05:45:37.452 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-03-12T05:45:37.458 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout enabled 2024-03-12T05:45:37.464 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout active 2024-03-12T05:45:37.464 INFO:teuthology.orchestra.run.smithi076.stdout:Unit chronyd.service is enabled and running 2024-03-12T05:45:37.464 INFO:teuthology.orchestra.run.smithi076.stdout:Repeating the final host check... 2024-03-12T05:45:37.487 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T05:45:37.487 INFO:teuthology.orchestra.run.smithi076.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-12T05:45:37.487 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl is present 2024-03-12T05:45:37.487 INFO:teuthology.orchestra.run.smithi076.stdout:lvcreate is present 2024-03-12T05:45:37.493 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-12T05:45:37.493 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T05:45:37.499 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-12T05:45:37.499 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-03-12T05:45:37.505 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout enabled 2024-03-12T05:45:37.512 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout active 2024-03-12T05:45:37.512 INFO:teuthology.orchestra.run.smithi076.stdout:Unit chronyd.service is enabled and running 2024-03-12T05:45:37.512 INFO:teuthology.orchestra.run.smithi076.stdout:Host looks OK 2024-03-12T05:45:37.512 INFO:teuthology.orchestra.run.smithi076.stdout:Cluster fsid: 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:45:37.512 INFO:teuthology.orchestra.run.smithi076.stdout:Acquiring lock 140498263840368 on /run/cephadm/9f336de8-e033-11ee-95c8-87774f69a715.lock 2024-03-12T05:45:37.512 INFO:teuthology.orchestra.run.smithi076.stdout:Lock 140498263840368 acquired on /run/cephadm/9f336de8-e033-11ee-95c8-87774f69a715.lock 2024-03-12T05:45:37.512 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying IP 172.21.15.76 port 3300 ... 2024-03-12T05:45:37.512 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying IP 172.21.15.76 port 6789 ... 2024-03-12T05:45:37.513 INFO:teuthology.orchestra.run.smithi076.stdout:Base mon IP(s) is [172.21.15.76:3300, 172.21.15.76:6789], mon addrv is [v2:172.21.15.76:3300,v1:172.21.15.76:6789] 2024-03-12T05:45:37.516 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.76 metric 100 2024-03-12T05:45:37.516 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.76 metric 100 2024-03-12T05:45:37.518 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-12T05:45:37.518 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-12T05:45:37.518 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1767sec hoplimit 64 pref medium 2024-03-12T05:45:37.520 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-12T05:45:37.521 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-12T05:45:37.521 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-12T05:45:37.521 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-12T05:45:37.521 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:47e1/64 scope link 2024-03-12T05:45:37.521 INFO:teuthology.orchestra.run.smithi076.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-12T05:45:37.521 INFO:teuthology.orchestra.run.smithi076.stdout:Mon IP `172.21.15.76` is in CIDR network `172.21.0.0/20` 2024-03-12T05:45:37.522 INFO:teuthology.orchestra.run.smithi076.stdout:Mon IP `172.21.15.76` is in CIDR network `172.21.0.0/20` 2024-03-12T05:45:37.522 INFO:teuthology.orchestra.run.smithi076.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-12T05:45:37.522 INFO:teuthology.orchestra.run.smithi076.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-12T05:45:37.523 INFO:teuthology.orchestra.run.smithi076.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-12T05:45:38.123 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stdout a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-12T05:45:38.123 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-12T05:45:38.123 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Getting image source signatures 2024-03-12T05:45:38.123 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Copying blob sha256:0b3c7bee4e66dab4b5376aaf0baf8ba5fc1c3e66c27dc601157782d617bd9dc6 2024-03-12T05:45:38.123 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-12T05:45:38.123 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Copying config sha256:a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-12T05:45:38.123 INFO:teuthology.orchestra.run.smithi076.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-12T05:45:39.077 INFO:teuthology.orchestra.run.smithi076.stdout:ceph: stdout ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-12T05:45:39.077 INFO:teuthology.orchestra.run.smithi076.stdout:Ceph version: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-12T05:45:39.077 INFO:teuthology.orchestra.run.smithi076.stdout:Extracting ceph user uid/gid from container image... 2024-03-12T05:45:40.082 INFO:teuthology.orchestra.run.smithi076.stdout:stat: stdout 167 167 2024-03-12T05:45:40.083 INFO:teuthology.orchestra.run.smithi076.stdout:Creating initial keys... 2024-03-12T05:45:41.063 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph-authtool: stdout AQAE7O9lCSPtFhAAkQcbLU9fxurmwBi4c5Oucw== 2024-03-12T05:45:42.012 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph-authtool: stdout AQAF7O9lyBApFhAAkjg/AValpmcAt6nksjOGgg== 2024-03-12T05:45:42.990 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph-authtool: stdout AQAG7O9lb90BFBAAo8sZAr9FCsit00P+DSPi6w== 2024-03-12T05:45:42.991 INFO:teuthology.orchestra.run.smithi076.stdout:Creating initial monmap... 2024-03-12T05:45:44.023 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T05:45:44.023 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-12T05:45:44.023 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:45:44.023 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T05:45:44.023 INFO:teuthology.orchestra.run.smithi076.stdout:monmaptool for a [v2:172.21.15.76:3300,v1:172.21.15.76:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T05:45:44.023 INFO:teuthology.orchestra.run.smithi076.stdout:setting min_mon_release = quincy 2024-03-12T05:45:44.023 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: set fsid to 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:45:44.023 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T05:45:44.024 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:45:44.024 INFO:teuthology.orchestra.run.smithi076.stdout:Creating mon... 2024-03-12T05:45:45.003 INFO:teuthology.orchestra.run.smithi076.stdout:create mon.a on 2024-03-12T05:45:45.212 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-12T05:45:45.356 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-12T05:45:45.522 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9f336de8-e033-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-9f336de8-e033-11ee-95c8-87774f69a715.target. 2024-03-12T05:45:45.522 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9f336de8-e033-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-9f336de8-e033-11ee-95c8-87774f69a715.target. 2024-03-12T05:45:45.756 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9f336de8-e033-11ee-95c8-87774f69a715@mon.a 2024-03-12T05:45:45.756 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9f336de8-e033-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-9f336de8-e033-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-12T05:45:45.917 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9f336de8-e033-11ee-95c8-87774f69a715.target.wants/ceph-9f336de8-e033-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-9f336de8-e033-11ee-95c8-87774f69a715@.service. 2024-03-12T05:45:46.485 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T05:45:46.485 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-03-12T05:45:46.492 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T05:45:46.492 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-03-12T05:45:46.492 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-03-12T05:45:46.492 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to enable service . firewalld.service is not available 2024-03-12T05:45:46.492 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mon to start... 2024-03-12T05:45:46.492 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mon... 2024-03-12T05:45:47.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:47 smithi076 ceph-mon[31714]: from='client.? 172.21.15.76:0/1116339436' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T05:45:47.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout cluster: 2024-03-12T05:45:47.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout id: 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:45:47.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-12T05:45:47.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-12T05:45:47.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout services: 2024-03-12T05:45:47.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.528381s) 2024-03-12T05:45:47.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-12T05:45:47.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-12T05:45:47.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-12T05:45:47.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout data: 2024-03-12T05:45:47.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-12T05:45:47.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-12T05:45:47.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-12T05:45:47.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout pgs: 2024-03-12T05:45:47.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-12T05:45:47.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.903+0000 7ff2fe2db700 1 Processor -- start 2024-03-12T05:45:47.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.903+0000 7ff2fe2db700 1 -- start start 2024-03-12T05:45:47.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.904+0000 7ff2fe2db700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f8108140 0x7ff2f810a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:45:47.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.904+0000 7ff2fe2db700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f80755e0 con 0x7ff2f8108140 2024-03-12T05:45:47.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.904+0000 7ff2f7fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f8108140 0x7ff2f810a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:45:47.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.904+0000 7ff2f7fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f8108140 0x7ff2f810a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44892/0 (socket says 172.21.15.76:44892) 2024-03-12T05:45:47.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.904+0000 7ff2f7fff700 1 -- 172.21.15.76:0/2830010666 learned_addr learned my addr 172.21.15.76:0/2830010666 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:45:47.547 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.904+0000 7ff2f7fff700 1 -- 172.21.15.76:0/2830010666 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f810aa70 con 0x7ff2f8108140 2024-03-12T05:45:47.547 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.904+0000 7ff2f7fff700 1 --2- 172.21.15.76:0/2830010666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f8108140 0x7ff2f810a530 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00098a0 tx=0x7ff2e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b09cb2813ca65ea3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:45:47.547 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.905+0000 7ff2f6ffd700 1 -- 172.21.15.76:0/2830010666 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff2e0004160 con 0x7ff2f8108140 2024-03-12T05:45:47.547 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.905+0000 7ff2f6ffd700 1 -- 172.21.15.76:0/2830010666 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e00042c0 con 0x7ff2f8108140 2024-03-12T05:45:47.547 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.905+0000 7ff2f6ffd700 1 -- 172.21.15.76:0/2830010666 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff2e00045a0 con 0x7ff2f8108140 2024-03-12T05:45:47.547 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.905+0000 7ff2fe2db700 1 -- 172.21.15.76:0/2830010666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f8108140 msgr2=0x7ff2f810a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:45:47.547 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.905+0000 7ff2fe2db700 1 --2- 172.21.15.76:0/2830010666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f8108140 0x7ff2f810a530 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00098a0 tx=0x7ff2e0009bb0 comp rx=0 tx=0).stop 2024-03-12T05:45:47.547 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.905+0000 7ff2fe2db700 1 -- 172.21.15.76:0/2830010666 shutdown_connections 2024-03-12T05:45:47.547 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.905+0000 7ff2fe2db700 1 --2- 172.21.15.76:0/2830010666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f8108140 0x7ff2f810a530 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:45:47.548 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.905+0000 7ff2fe2db700 1 -- 172.21.15.76:0/2830010666 >> 172.21.15.76:0/2830010666 conn(0x7ff2f8101b90 msgr2=0x7ff2f8103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:45:47.548 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.905+0000 7ff2fe2db700 1 -- 172.21.15.76:0/2830010666 shutdown_connections 2024-03-12T05:45:47.548 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.905+0000 7ff2fe2db700 1 -- 172.21.15.76:0/2830010666 wait complete. 2024-03-12T05:45:47.548 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.906+0000 7ff2fe2db700 1 Processor -- start 2024-03-12T05:45:47.548 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.906+0000 7ff2fe2db700 1 -- start start 2024-03-12T05:45:47.548 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.906+0000 7ff2fe2db700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f8108140 0x7ff2f819ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:45:47.548 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.906+0000 7ff2fe2db700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f819b0d0 con 0x7ff2f8108140 2024-03-12T05:45:47.548 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.906+0000 7ff2f7fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f8108140 0x7ff2f819ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:45:47.548 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.906+0000 7ff2f7fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f8108140 0x7ff2f819ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44908/0 (socket says 172.21.15.76:44908) 2024-03-12T05:45:47.548 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.906+0000 7ff2f7fff700 1 -- 172.21.15.76:0/1116339436 learned_addr learned my addr 172.21.15.76:0/1116339436 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:45:47.548 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.907+0000 7ff2f7fff700 1 -- 172.21.15.76:0/1116339436 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e0009550 con 0x7ff2f8108140 2024-03-12T05:45:47.548 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.907+0000 7ff2f7fff700 1 --2- 172.21.15.76:0/1116339436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f8108140 0x7ff2f819ab90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8074a20 tx=0x7ff2e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:45:47.549 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.907+0000 7ff2f57fa700 1 -- 172.21.15.76:0/1116339436 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff2e0011630 con 0x7ff2f8108140 2024-03-12T05:45:47.549 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.907+0000 7ff2f57fa700 1 -- 172.21.15.76:0/1116339436 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e0011790 con 0x7ff2f8108140 2024-03-12T05:45:47.549 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.907+0000 7ff2fe2db700 1 -- 172.21.15.76:0/1116339436 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f819b2d0 con 0x7ff2f8108140 2024-03-12T05:45:47.549 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.907+0000 7ff2f57fa700 1 -- 172.21.15.76:0/1116339436 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff2e0011a30 con 0x7ff2f8108140 2024-03-12T05:45:47.549 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.907+0000 7ff2fe2db700 1 -- 172.21.15.76:0/1116339436 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f819b770 con 0x7ff2f8108140 2024-03-12T05:45:47.549 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.907+0000 7ff2f57fa700 1 -- 172.21.15.76:0/1116339436 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff2e0022070 con 0x7ff2f8108140 2024-03-12T05:45:47.549 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.907+0000 7ff2f57fa700 1 -- 172.21.15.76:0/1116339436 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff2e001c450 con 0x7ff2f8108140 2024-03-12T05:45:47.549 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.908+0000 7ff2fe2db700 1 -- 172.21.15.76:0/1116339436 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f8194970 con 0x7ff2f8108140 2024-03-12T05:45:47.549 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.910+0000 7ff2f57fa700 1 -- 172.21.15.76:0/1116339436 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff2e0011b90 con 0x7ff2f8108140 2024-03-12T05:45:47.549 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.982+0000 7ff2fe2db700 1 -- 172.21.15.76:0/1116339436 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7ff2f802d800 con 0x7ff2f8108140 2024-03-12T05:45:47.549 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.982+0000 7ff2f57fa700 1 -- 172.21.15.76:0/1116339436 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7ff2e0011d30 con 0x7ff2f8108140 2024-03-12T05:45:47.549 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.983+0000 7ff2fe2db700 1 -- 172.21.15.76:0/1116339436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f8108140 msgr2=0x7ff2f819ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:45:47.549 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.983+0000 7ff2fe2db700 1 --2- 172.21.15.76:0/1116339436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f8108140 0x7ff2f819ab90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8074a20 tx=0x7ff2e0004750 comp rx=0 tx=0).stop 2024-03-12T05:45:47.549 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.983+0000 7ff2fe2db700 1 -- 172.21.15.76:0/1116339436 shutdown_connections 2024-03-12T05:45:47.550 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.983+0000 7ff2fe2db700 1 --2- 172.21.15.76:0/1116339436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff2f8108140 0x7ff2f819ab90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:45:47.550 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.983+0000 7ff2fe2db700 1 -- 172.21.15.76:0/1116339436 >> 172.21.15.76:0/1116339436 conn(0x7ff2f8101b90 msgr2=0x7ff2f8101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:45:47.550 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.983+0000 7ff2fe2db700 1 -- 172.21.15.76:0/1116339436 shutdown_connections 2024-03-12T05:45:47.550 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:46.983+0000 7ff2fe2db700 1 -- 172.21.15.76:0/1116339436 wait complete. 2024-03-12T05:45:47.550 INFO:teuthology.orchestra.run.smithi076.stdout:mon is available 2024-03-12T05:45:47.550 INFO:teuthology.orchestra.run.smithi076.stdout:Assimilating anything we can from ceph.conf... 2024-03-12T05:45:48.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:48 smithi076 ceph-mon[31714]: from='client.? 172.21.15.76:0/232611433' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-12T05:45:48.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:48 smithi076 ceph-mon[31714]: from='client.? 172.21.15.76:0/232611433' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-12T05:45:48.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-12T05:45:48.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [global] 2024-03-12T05:45:48.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout fsid = 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:45:48.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.76:3300,v1:172.21.15.76:6789] 2024-03-12T05:45:48.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-12T05:45:48.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-12T05:45:48.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-12T05:45:48.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-12T05:45:48.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-12T05:45:48.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-12T05:45:48.576 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-12T05:45:48.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-12T05:45:48.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [osd] 2024-03-12T05:45:48.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-12T05:45:48.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-12T05:45:48.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-12T05:45:48.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-12T05:45:48.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-12T05:45:48.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.960+0000 7f18e44b6700 1 Processor -- start 2024-03-12T05:45:48.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.960+0000 7f18e44b6700 1 -- start start 2024-03-12T05:45:48.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.960+0000 7f18e44b6700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18dc1059d0 0x7f18dc105df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:45:48.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.960+0000 7f18e44b6700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc1063c0 con 0x7f18dc1059d0 2024-03-12T05:45:48.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.960+0000 7f18e2252700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18dc1059d0 0x7f18dc105df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:45:48.577 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.960+0000 7f18e2252700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18dc1059d0 0x7f18dc105df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44910/0 (socket says 172.21.15.76:44910) 2024-03-12T05:45:48.578 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.960+0000 7f18e2252700 1 -- 172.21.15.76:0/438524702 learned_addr learned my addr 172.21.15.76:0/438524702 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:45:48.578 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.961+0000 7f18e2252700 1 -- 172.21.15.76:0/438524702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc106500 con 0x7f18dc1059d0 2024-03-12T05:45:48.578 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.961+0000 7f18e2252700 1 --2- 172.21.15.76:0/438524702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18dc1059d0 0x7f18dc105df0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f18d80098a0 tx=0x7f18d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=154382a6b15f90bd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:45:48.578 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.961+0000 7f18e1250700 1 -- 172.21.15.76:0/438524702 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18d8004160 con 0x7f18dc1059d0 2024-03-12T05:45:48.578 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.961+0000 7f18e1250700 1 -- 172.21.15.76:0/438524702 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d80042c0 con 0x7f18dc1059d0 2024-03-12T05:45:48.578 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.961+0000 7f18e1250700 1 -- 172.21.15.76:0/438524702 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18d80045a0 con 0x7f18dc1059d0 2024-03-12T05:45:48.578 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.962+0000 7f18e44b6700 1 -- 172.21.15.76:0/438524702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18dc1059d0 msgr2=0x7f18dc105df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:45:48.578 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.962+0000 7f18e44b6700 1 --2- 172.21.15.76:0/438524702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18dc1059d0 0x7f18dc105df0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f18d80098a0 tx=0x7f18d8009bb0 comp rx=0 tx=0).stop 2024-03-12T05:45:48.578 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.962+0000 7f18e44b6700 1 -- 172.21.15.76:0/438524702 shutdown_connections 2024-03-12T05:45:48.578 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.962+0000 7f18e44b6700 1 --2- 172.21.15.76:0/438524702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18dc1059d0 0x7f18dc105df0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:45:48.578 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.962+0000 7f18e44b6700 1 -- 172.21.15.76:0/438524702 >> 172.21.15.76:0/438524702 conn(0x7f18dc101430 msgr2=0x7f18dc103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:45:48.578 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.962+0000 7f18e44b6700 1 -- 172.21.15.76:0/438524702 shutdown_connections 2024-03-12T05:45:48.578 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.962+0000 7f18e44b6700 1 -- 172.21.15.76:0/438524702 wait complete. 2024-03-12T05:45:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.962+0000 7f18e44b6700 1 Processor -- start 2024-03-12T05:45:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.962+0000 7f18e44b6700 1 -- start start 2024-03-12T05:45:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.962+0000 7f18e44b6700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18dc1059d0 0x7f18dc19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:45:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.962+0000 7f18e44b6700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc19b120 con 0x7f18dc1059d0 2024-03-12T05:45:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.963+0000 7f18e2252700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18dc1059d0 0x7f18dc19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:45:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.963+0000 7f18e2252700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18dc1059d0 0x7f18dc19abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44922/0 (socket says 172.21.15.76:44922) 2024-03-12T05:45:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.963+0000 7f18e2252700 1 -- 172.21.15.76:0/232611433 learned_addr learned my addr 172.21.15.76:0/232611433 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:45:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.963+0000 7f18e2252700 1 -- 172.21.15.76:0/232611433 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d8009550 con 0x7f18dc1059d0 2024-03-12T05:45:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.963+0000 7f18e2252700 1 --2- 172.21.15.76:0/232611433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18dc1059d0 0x7f18dc19abe0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f18d8004000 tx=0x7f18d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:45:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.963+0000 7f18d37fe700 1 -- 172.21.15.76:0/232611433 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18d8011630 con 0x7f18dc1059d0 2024-03-12T05:45:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.963+0000 7f18d37fe700 1 -- 172.21.15.76:0/232611433 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d8011790 con 0x7f18dc1059d0 2024-03-12T05:45:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.963+0000 7f18e44b6700 1 -- 172.21.15.76:0/232611433 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18dc19b320 con 0x7f18dc1059d0 2024-03-12T05:45:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.963+0000 7f18d37fe700 1 -- 172.21.15.76:0/232611433 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18d8011a30 con 0x7f18dc1059d0 2024-03-12T05:45:48.579 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.963+0000 7f18e44b6700 1 -- 172.21.15.76:0/232611433 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18dc19b7c0 con 0x7f18dc1059d0 2024-03-12T05:45:48.580 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.963+0000 7f18d37fe700 1 -- 172.21.15.76:0/232611433 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f18d8021070 con 0x7f18dc1059d0 2024-03-12T05:45:48.580 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.964+0000 7f18d37fe700 1 -- 172.21.15.76:0/232611433 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f18d801b9d0 con 0x7f18dc1059d0 2024-03-12T05:45:48.580 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.965+0000 7f18e44b6700 1 -- 172.21.15.76:0/232611433 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18dc1948d0 con 0x7f18dc1059d0 2024-03-12T05:45:48.580 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:47.967+0000 7f18d37fe700 1 -- 172.21.15.76:0/232611433 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f18d801bb90 con 0x7f18dc1059d0 2024-03-12T05:45:48.580 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:48.033+0000 7f18e44b6700 1 -- 172.21.15.76:0/232611433 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f18dc108bd0 con 0x7f18dc1059d0 2024-03-12T05:45:48.580 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:48.036+0000 7f18d37fe700 1 -- 172.21.15.76:0/232611433 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f18d801bd30 con 0x7f18dc1059d0 2024-03-12T05:45:48.580 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:48.037+0000 7f18d37fe700 1 -- 172.21.15.76:0/232611433 <== mon.0 v2:172.21.15.76:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18d801a920 con 0x7f18dc1059d0 2024-03-12T05:45:48.580 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:48.038+0000 7f18e44b6700 1 -- 172.21.15.76:0/232611433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18dc1059d0 msgr2=0x7f18dc19abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:45:48.580 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:48.038+0000 7f18e44b6700 1 --2- 172.21.15.76:0/232611433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18dc1059d0 0x7f18dc19abe0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f18d8004000 tx=0x7f18d8004750 comp rx=0 tx=0).stop 2024-03-12T05:45:48.580 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:48.038+0000 7f18e44b6700 1 -- 172.21.15.76:0/232611433 shutdown_connections 2024-03-12T05:45:48.580 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:48.038+0000 7f18e44b6700 1 --2- 172.21.15.76:0/232611433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f18dc1059d0 0x7f18dc19abe0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:45:48.580 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:48.038+0000 7f18e44b6700 1 -- 172.21.15.76:0/232611433 >> 172.21.15.76:0/232611433 conn(0x7f18dc101430 msgr2=0x7f18dc191a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:45:48.580 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:48.038+0000 7f18e44b6700 1 -- 172.21.15.76:0/232611433 shutdown_connections 2024-03-12T05:45:48.580 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:48.038+0000 7f18e44b6700 1 -- 172.21.15.76:0/232611433 wait complete. 2024-03-12T05:45:48.581 INFO:teuthology.orchestra.run.smithi076.stdout:Generating new minimal ceph.conf... 2024-03-12T05:45:49.641 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:49 smithi076 ceph-mon[31714]: from='client.? 172.21.15.76:0/3174388194' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:45:49.646 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.000+0000 7f297ff65700 1 Processor -- start 2024-03-12T05:45:49.646 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.000+0000 7f297ff65700 1 -- start start 2024-03-12T05:45:49.646 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.000+0000 7f297ff65700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f297807ce00 0x7f297807b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:45:49.646 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.000+0000 7f297ff65700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297807b840 con 0x7f297807ce00 2024-03-12T05:45:49.647 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.000+0000 7f297dd01700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f297807ce00 0x7f297807b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:45:49.647 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.000+0000 7f297dd01700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f297807ce00 0x7f297807b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44938/0 (socket says 172.21.15.76:44938) 2024-03-12T05:45:49.647 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.000+0000 7f297dd01700 1 -- 172.21.15.76:0/2884939181 learned_addr learned my addr 172.21.15.76:0/2884939181 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:45:49.647 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.001+0000 7f297dd01700 1 -- 172.21.15.76:0/2884939181 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297807b980 con 0x7f297807ce00 2024-03-12T05:45:49.647 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.001+0000 7f297dd01700 1 --2- 172.21.15.76:0/2884939181 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f297807ce00 0x7f297807b300 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f29740098a0 tx=0x7f2974009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faf407b2ea7ea64b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:45:49.647 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.001+0000 7f297ccff700 1 -- 172.21.15.76:0/2884939181 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2974004160 con 0x7f297807ce00 2024-03-12T05:45:49.647 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.001+0000 7f297ccff700 1 -- 172.21.15.76:0/2884939181 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29740042c0 con 0x7f297807ce00 2024-03-12T05:45:49.647 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.001+0000 7f297ccff700 1 -- 172.21.15.76:0/2884939181 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2974011ac0 con 0x7f297807ce00 2024-03-12T05:45:49.647 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.002+0000 7f297ff65700 1 -- 172.21.15.76:0/2884939181 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f297807ce00 msgr2=0x7f297807b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:45:49.647 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.002+0000 7f297ff65700 1 --2- 172.21.15.76:0/2884939181 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f297807ce00 0x7f297807b300 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f29740098a0 tx=0x7f2974009bb0 comp rx=0 tx=0).stop 2024-03-12T05:45:49.647 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.002+0000 7f297ff65700 1 -- 172.21.15.76:0/2884939181 shutdown_connections 2024-03-12T05:45:49.647 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.002+0000 7f297ff65700 1 --2- 172.21.15.76:0/2884939181 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f297807ce00 0x7f297807b300 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:45:49.647 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.002+0000 7f297ff65700 1 -- 172.21.15.76:0/2884939181 >> 172.21.15.76:0/2884939181 conn(0x7f2978102d50 msgr2=0x7f2978105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:45:49.648 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.002+0000 7f297ff65700 1 -- 172.21.15.76:0/2884939181 shutdown_connections 2024-03-12T05:45:49.648 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.002+0000 7f297ff65700 1 -- 172.21.15.76:0/2884939181 wait complete. 2024-03-12T05:45:49.648 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.002+0000 7f297ff65700 1 Processor -- start 2024-03-12T05:45:49.648 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.002+0000 7f297ff65700 1 -- start start 2024-03-12T05:45:49.648 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.003+0000 7f297ff65700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f297807ce00 0x7f297819d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:45:49.648 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.003+0000 7f297ff65700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297819db70 con 0x7f297807ce00 2024-03-12T05:45:49.648 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.003+0000 7f297dd01700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f297807ce00 0x7f297819d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:45:49.648 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.003+0000 7f297dd01700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f297807ce00 0x7f297819d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44942/0 (socket says 172.21.15.76:44942) 2024-03-12T05:45:49.648 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.003+0000 7f297dd01700 1 -- 172.21.15.76:0/3174388194 learned_addr learned my addr 172.21.15.76:0/3174388194 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:45:49.648 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.003+0000 7f297dd01700 1 -- 172.21.15.76:0/3174388194 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2974009550 con 0x7f297807ce00 2024-03-12T05:45:49.648 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.003+0000 7f297dd01700 1 --2- 172.21.15.76:0/3174388194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f297807ce00 0x7f297819d630 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2974011a70 tx=0x7f2974011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:45:49.648 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.003+0000 7f296effd700 1 -- 172.21.15.76:0/3174388194 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29740039c0 con 0x7f297807ce00 2024-03-12T05:45:49.648 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.003+0000 7f296effd700 1 -- 172.21.15.76:0/3174388194 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2974003b20 con 0x7f297807ce00 2024-03-12T05:45:49.648 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.003+0000 7f297ff65700 1 -- 172.21.15.76:0/3174388194 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297819dd70 con 0x7f297807ce00 2024-03-12T05:45:49.649 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.003+0000 7f296effd700 1 -- 172.21.15.76:0/3174388194 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2974024610 con 0x7f297807ce00 2024-03-12T05:45:49.649 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.003+0000 7f297ff65700 1 -- 172.21.15.76:0/3174388194 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f297819e210 con 0x7f297807ce00 2024-03-12T05:45:49.649 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.003+0000 7f296effd700 1 -- 172.21.15.76:0/3174388194 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f2974024be0 con 0x7f297807ce00 2024-03-12T05:45:49.649 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.003+0000 7f296effd700 1 -- 172.21.15.76:0/3174388194 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2974019920 con 0x7f297807ce00 2024-03-12T05:45:49.649 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.004+0000 7f297ff65700 1 -- 172.21.15.76:0/3174388194 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f297810da60 con 0x7f297807ce00 2024-03-12T05:45:49.649 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.006+0000 7f296effd700 1 -- 172.21.15.76:0/3174388194 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f297401aa10 con 0x7f297807ce00 2024-03-12T05:45:49.649 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.072+0000 7f297ff65700 1 -- 172.21.15.76:0/3174388194 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f297810dc70 con 0x7f297807ce00 2024-03-12T05:45:49.649 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.073+0000 7f296effd700 1 -- 172.21.15.76:0/3174388194 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f297401abb0 con 0x7f297807ce00 2024-03-12T05:45:49.649 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.073+0000 7f297ff65700 1 -- 172.21.15.76:0/3174388194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f297807ce00 msgr2=0x7f297819d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:45:49.649 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.073+0000 7f297ff65700 1 --2- 172.21.15.76:0/3174388194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f297807ce00 0x7f297819d630 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2974011a70 tx=0x7f2974011b50 comp rx=0 tx=0).stop 2024-03-12T05:45:49.649 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.073+0000 7f297ff65700 1 -- 172.21.15.76:0/3174388194 shutdown_connections 2024-03-12T05:45:49.649 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.073+0000 7f297ff65700 1 --2- 172.21.15.76:0/3174388194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f297807ce00 0x7f297819d630 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:45:49.649 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.073+0000 7f297ff65700 1 -- 172.21.15.76:0/3174388194 >> 172.21.15.76:0/3174388194 conn(0x7f2978102d50 msgr2=0x7f2978103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:45:49.649 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.073+0000 7f297ff65700 1 -- 172.21.15.76:0/3174388194 shutdown_connections 2024-03-12T05:45:49.650 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:49.073+0000 7f297ff65700 1 -- 172.21.15.76:0/3174388194 wait complete. 2024-03-12T05:45:49.650 INFO:teuthology.orchestra.run.smithi076.stdout:Restarting the monitor... 2024-03-12T05:45:49.936 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:49 smithi076 systemd[1]: Stopping Ceph mon.a for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T05:45:49.938 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:49 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a[31710]: 2024-03-12T05:45:49.883+0000 7f5187c32700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T05:45:49.938 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:49 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a[31710]: 2024-03-12T05:45:49.883+0000 7f5187c32700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-12T05:45:49.938 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:49 smithi076 podman[31962]: 2024-03-12 05:45:49.935705084 +0000 UTC m=+0.129382198 container died faffbee47bf15f910b3834ba253de7282298efd649d5a3e34abead9ec8ca839d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T05:45:50.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:50 smithi076 podman[31962]: 2024-03-12 05:45:50.292257422 +0000 UTC m=+0.485934534 container cleanup faffbee47bf15f910b3834ba253de7282298efd649d5a3e34abead9ec8ca839d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-12T05:45:50.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:50 smithi076 bash[31962]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a 2024-03-12T05:45:50.911 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:50 smithi076 podman[31976]: 2024-03-12 05:45:50.576612293 +0000 UTC m=+0.645359301 container remove faffbee47bf15f910b3834ba253de7282298efd649d5a3e34abead9ec8ca839d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-12T05:45:50.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:50 smithi076 systemd[1]: ceph-9f336de8-e033-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-12T05:45:50.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:50 smithi076 systemd[1]: Stopped Ceph mon.a for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:45:50.912 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:50 smithi076 systemd[1]: Starting Ceph mon.a for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T05:45:51.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:50 smithi076 podman[32076]: 2024-03-12 05:45:50.814048568 +0000 UTC m=+0.017330670 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T05:45:51.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 podman[32076]: 2024-03-12 05:45:51.035819814 +0000 UTC m=+0.239101911 container create 9c347f27958bdc05fa1da6cd36fd0914a3ba9813bd5064b4b60d8b0d04bfed02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-12T05:45:51.350 INFO:teuthology.orchestra.run.smithi076.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-12T05:45:51.553 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 podman[32076]: 2024-03-12 05:45:51.238001331 +0000 UTC m=+0.441283436 container init 9c347f27958bdc05fa1da6cd36fd0914a3ba9813bd5064b4b60d8b0d04bfed02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-12T05:45:51.553 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 podman[32076]: 2024-03-12 05:45:51.240806835 +0000 UTC m=+0.444088939 container start 9c347f27958bdc05fa1da6cd36fd0914a3ba9813bd5064b4b60d8b0d04bfed02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-12T05:45:51.553 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T05:45:51.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-12T05:45:51.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: pidfile_write: ignore empty --pid-file 2024-03-12T05:45:51.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: load: jerasure load: lrc 2024-03-12T05:45:51.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: RocksDB version: 7.9.2 2024-03-12T05:45:51.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Git sha 0 2024-03-12T05:45:51.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-12T05:45:51.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: DB SUMMARY 2024-03-12T05:45:51.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: DB Session ID: KJ3X6JDLTN93EK9471IM 2024-03-12T05:45:51.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: CURRENT file: CURRENT 2024-03-12T05:45:51.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T05:45:51.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-12T05:45:51.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-12T05:45:51.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100536 ; 2024-03-12T05:45:51.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.error_if_exists: 0 2024-03-12T05:45:51.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.create_if_missing: 0 2024-03-12T05:45:51.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.paranoid_checks: 1 2024-03-12T05:45:51.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T05:45:51.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T05:45:51.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T05:45:51.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.env: 0x55fd47705ae0 2024-03-12T05:45:51.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T05:45:51.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.info_log: 0x55fd4859d8c0 2024-03-12T05:45:51.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T05:45:51.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.statistics: (nil) 2024-03-12T05:45:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.use_fsync: 0 2024-03-12T05:45:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_log_file_size: 0 2024-03-12T05:45:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T05:45:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T05:45:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T05:45:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T05:45:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.allow_fallocate: 1 2024-03-12T05:45:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T05:45:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T05:45:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.use_direct_reads: 0 2024-03-12T05:45:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T05:45:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T05:45:51.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.db_log_dir: 2024-03-12T05:45:51.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.wal_dir: 2024-03-12T05:45:51.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T05:45:51.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T05:45:51.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T05:45:51.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T05:45:51.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T05:45:51.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T05:45:51.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T05:45:51.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T05:45:51.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.write_buffer_manager: 0x55fd48571400 2024-03-12T05:45:51.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T05:45:51.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T05:45:51.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T05:45:51.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.unordered_write: 0 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.row_cache: None 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.wal_filter: None 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.two_write_queues: 0 2024-03-12T05:45:51.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.wal_compression: 0 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.atomic_flush: 0 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.log_readahead_size: 0 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_background_jobs: 2 2024-03-12T05:45:51.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_background_compactions: -1 2024-03-12T05:45:51.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_subcompactions: 1 2024-03-12T05:45:51.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T05:45:51.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T05:45:51.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T05:45:51.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T05:45:51.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T05:45:51.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T05:45:51.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T05:45:51.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T05:45:51.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_open_files: -1 2024-03-12T05:45:51.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T05:45:51.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T05:45:51.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T05:45:51.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T05:45:51.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_background_flushes: -1 2024-03-12T05:45:51.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Compression algorithms supported: 2024-03-12T05:45:51.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T05:45:51.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: kZSTD supported: 0 2024-03-12T05:45:51.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: kXpressCompression supported: 0 2024-03-12T05:45:51.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T05:45:51.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: kZlibCompression supported: 1 2024-03-12T05:45:51.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: kSnappyCompression supported: 1 2024-03-12T05:45:51.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: kLZ4Compression supported: 1 2024-03-12T05:45:51.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: kBZip2Compression supported: 0 2024-03-12T05:45:51.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T05:45:51.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T05:45:51.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-12T05:45:51.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.merge_operator: 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compaction_filter: None 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fd4859d9c0) 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout: cache_index_and_filter_blocks: 1 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout: pin_top_level_index_and_filter: 1 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout: index_type: 0 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout: data_block_index_type: 0 2024-03-12T05:45:51.563 INFO:journalctl@ceph.mon.a.smithi076.stdout: index_shortening: 1 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: checksum: 4 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: no_block_cache: 0 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache: 0x55fd485a1090 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache_name: BinnedLRUCache 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache_options: 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: capacity : 536870912 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: num_shard_bits : 4 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: strict_capacity_limit : 0 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: high_pri_pool_ratio: 0.000 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_cache_compressed: (nil) 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: persistent_cache: (nil) 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_size: 4096 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_size_deviation: 10 2024-03-12T05:45:51.564 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_restart_interval: 16 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: index_block_restart_interval: 1 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: metadata_block_size: 4096 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: partition_filters: 0 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: use_delta_encoding: 1 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: filter_policy: bloomfilter 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: whole_key_filtering: 1 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: verify_compression: 0 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: read_amp_bytes_per_bit: 0 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: format_version: 5 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: enable_index_compression: 1 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: block_align: 0 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: max_auto_readahead_size: 262144 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: prepopulate_block_cache: 0 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: initial_auto_readahead_size: 8192 2024-03-12T05:45:51.565 INFO:journalctl@ceph.mon.a.smithi076.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T05:45:51.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T05:45:51.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T05:45:51.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compression: NoCompression 2024-03-12T05:45:51.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T05:45:51.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T05:45:51.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T05:45:51.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.num_levels: 7 2024-03-12T05:45:51.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T05:45:51.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T05:45:51.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T05:45:51.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T05:45:51.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T05:45:51.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T05:45:51.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T05:45:51.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T05:45:51.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T05:45:51.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T05:45:51.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.table_properties_collectors: 2024-03-12T05:45:51.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.inplace_update_support: 0 2024-03-12T05:45:51.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T05:45:51.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T05:45:51.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T05:45:51.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T05:45:51.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.bloom_locality: 0 2024-03-12T05:45:51.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.max_successive_merges: 0 2024-03-12T05:45:51.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T05:45:51.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T05:45:51.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T05:45:51.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T05:45:51.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.ttl: 2592000 2024-03-12T05:45:51.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.enable_blob_files: false 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.min_blob_size: 0 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-12T05:45:51.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 96d54e93-9fb1-4765-8f6f-e496695cb09e 2024-03-12T05:45:51.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710222351269374, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-12T05:45:51.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-12T05:45:51.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710222351270352, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95597, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 320, "table_properties": {"data_size": 93587, "index_size": 292, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13989, "raw_average_key_size": 50, "raw_value_size": 86028, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710222351, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "96d54e93-9fb1-4765-8f6f-e496695cb09e", "db_session_id": "KJ3X6JDLTN93EK9471IM", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-12T05:45:51.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710222351270435, "job": 1, "event": "recovery_finished"} 2024-03-12T05:45:51.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-12T05:45:51.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T05:45:51.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fd48692000 2024-03-12T05:45:51.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: DB pointer 0x55fd4867c000 2024-03-12T05:45:51.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T05:45:51.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T05:45:51.572 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** DB Stats ** 2024-03-12T05:45:51.572 INFO:journalctl@ceph.mon.a.smithi076.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T05:45:51.572 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T05:45:51.572 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** Compaction Stats [default] ** 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: L0 2/0 95.19 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: Sum 2/0 95.19 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** Compaction Stats [default] ** 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T05:45:51.573 INFO:journalctl@ceph.mon.a.smithi076.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T05:45:51.574 INFO:journalctl@ceph.mon.a.smithi076.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T05:45:51.574 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-03-12T05:45:51.574 INFO:journalctl@ceph.mon.a.smithi076.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T05:45:51.574 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-03-12T05:45:51.574 INFO:journalctl@ceph.mon.a.smithi076.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T05:45:51.574 INFO:journalctl@ceph.mon.a.smithi076.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T05:45:51.574 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T05:45:51.574 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T05:45:51.574 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T05:45:51.574 INFO:journalctl@ceph.mon.a.smithi076.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T05:45:51.574 INFO:journalctl@ceph.mon.a.smithi076.stdout: Cumulative compaction: 0.00 GB write, 31.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T05:45:51.574 INFO:journalctl@ceph.mon.a.smithi076.stdout: Interval compaction: 0.00 GB write, 31.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T05:45:51.574 INFO:journalctl@ceph.mon.a.smithi076.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T05:45:51.574 INFO:journalctl@ceph.mon.a.smithi076.stdout: Block cache BinnedLRUCache@0x55fd485a1090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: starting mon.a rank 0 at public addrs [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] at bind addrs [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: mon.a@-1(???) e1 preinit fsid 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: mon.a@-1(???).mds e1 new map 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: mon.a@-1(???).mds e1 print_map 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout: e1 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout: legacy client fscid: -1 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout: 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout: No filesystems configured 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T05:45:51.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T05:45:51.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T05:45:51.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T05:45:51.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-12T05:45:51.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-12T05:45:51.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: monmap e1: 1 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:45:51.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: fsmap 2024-03-12T05:45:51.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: osdmap e1: 0 total, 0 up, 0 in 2024-03-12T05:45:51.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 ceph-mon[32095]: mgrmap e1: no daemons active 2024-03-12T05:45:51.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 bash[32076]: 9c347f27958bdc05fa1da6cd36fd0914a3ba9813bd5064b4b60d8b0d04bfed02 2024-03-12T05:45:51.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:51 smithi076 systemd[1]: Started Ceph mon.a for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:45:52.283 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.732+0000 7f5350398700 1 Processor -- start 2024-03-12T05:45:52.283 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.732+0000 7f5350398700 1 -- start start 2024-03-12T05:45:52.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.732+0000 7f5350398700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5348107760 0x7f5348107b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:45:52.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.732+0000 7f5350398700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5348108150 con 0x7f5348107760 2024-03-12T05:45:52.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.732+0000 7f534e134700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5348107760 0x7f5348107b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:45:52.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.732+0000 7f534e134700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5348107760 0x7f5348107b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58740/0 (socket says 172.21.15.76:58740) 2024-03-12T05:45:52.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.732+0000 7f534e134700 1 -- 172.21.15.76:0/3842883768 learned_addr learned my addr 172.21.15.76:0/3842883768 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:45:52.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.733+0000 7f534e134700 1 -- 172.21.15.76:0/3842883768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53481088c0 con 0x7f5348107760 2024-03-12T05:45:52.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.733+0000 7f534e134700 1 --2- 172.21.15.76:0/3842883768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5348107760 0x7f5348107b80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f53440098a0 tx=0x7f5344009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba3c6d814183b828 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:45:52.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.733+0000 7f534d132700 1 -- 172.21.15.76:0/3842883768 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5344004160 con 0x7f5348107760 2024-03-12T05:45:52.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.733+0000 7f534d132700 1 -- 172.21.15.76:0/3842883768 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53440042c0 con 0x7f5348107760 2024-03-12T05:45:52.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.733+0000 7f534d132700 1 -- 172.21.15.76:0/3842883768 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5344011ac0 con 0x7f5348107760 2024-03-12T05:45:52.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.733+0000 7f5350398700 1 -- 172.21.15.76:0/3842883768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5348107760 msgr2=0x7f5348107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:45:52.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.734+0000 7f5350398700 1 --2- 172.21.15.76:0/3842883768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5348107760 0x7f5348107b80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f53440098a0 tx=0x7f5344009bb0 comp rx=0 tx=0).stop 2024-03-12T05:45:52.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.734+0000 7f5350398700 1 -- 172.21.15.76:0/3842883768 shutdown_connections 2024-03-12T05:45:52.284 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.734+0000 7f5350398700 1 --2- 172.21.15.76:0/3842883768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5348107760 0x7f5348107b80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.734+0000 7f5350398700 1 -- 172.21.15.76:0/3842883768 >> 172.21.15.76:0/3842883768 conn(0x7f5348102de0 msgr2=0x7f5348105200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.734+0000 7f5350398700 1 -- 172.21.15.76:0/3842883768 shutdown_connections 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.734+0000 7f5350398700 1 -- 172.21.15.76:0/3842883768 wait complete. 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.734+0000 7f5350398700 1 Processor -- start 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.734+0000 7f5350398700 1 -- start start 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.734+0000 7f5350398700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5348107760 0x7f534819b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.734+0000 7f5350398700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534819b9c0 con 0x7f5348107760 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.735+0000 7f534e134700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5348107760 0x7f534819b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.735+0000 7f534e134700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5348107760 0x7f534819b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58754/0 (socket says 172.21.15.76:58754) 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.735+0000 7f534e134700 1 -- 172.21.15.76:0/2456874607 learned_addr learned my addr 172.21.15.76:0/2456874607 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.735+0000 7f534e134700 1 -- 172.21.15.76:0/2456874607 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5344009550 con 0x7f5348107760 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.735+0000 7f534e134700 1 --2- 172.21.15.76:0/2456874607 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5348107760 0x7f534819b480 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5344011780 tx=0x7f5344011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.735+0000 7f533f7fe700 1 -- 172.21.15.76:0/2456874607 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5344003af0 con 0x7f5348107760 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.735+0000 7f533f7fe700 1 -- 172.21.15.76:0/2456874607 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5344003c50 con 0x7f5348107760 2024-03-12T05:45:52.285 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.735+0000 7f5350398700 1 -- 172.21.15.76:0/2456874607 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534819bbc0 con 0x7f5348107760 2024-03-12T05:45:52.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.735+0000 7f533f7fe700 1 -- 172.21.15.76:0/2456874607 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f53440197e0 con 0x7f5348107760 2024-03-12T05:45:52.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.735+0000 7f5350398700 1 -- 172.21.15.76:0/2456874607 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f534819c060 con 0x7f5348107760 2024-03-12T05:45:52.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.735+0000 7f533f7fe700 1 -- 172.21.15.76:0/2456874607 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5344025460 con 0x7f5348107760 2024-03-12T05:45:52.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.735+0000 7f533f7fe700 1 -- 172.21.15.76:0/2456874607 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f534400bb20 con 0x7f5348107760 2024-03-12T05:45:52.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.736+0000 7f5350398700 1 -- 172.21.15.76:0/2456874607 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53480632a0 con 0x7f5348107760 2024-03-12T05:45:52.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.738+0000 7f533f7fe700 1 -- 172.21.15.76:0/2456874607 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f534402a060 con 0x7f5348107760 2024-03-12T05:45:52.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.805+0000 7f5350398700 1 -- 172.21.15.76:0/2456874607 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f534810baa0 con 0x7f5348107760 2024-03-12T05:45:52.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.807+0000 7f533f7fe700 1 -- 172.21.15.76:0/2456874607 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5344019d60 con 0x7f5348107760 2024-03-12T05:45:52.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.808+0000 7f5350398700 1 -- 172.21.15.76:0/2456874607 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5348107760 msgr2=0x7f534819b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:45:52.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.808+0000 7f5350398700 1 --2- 172.21.15.76:0/2456874607 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5348107760 0x7f534819b480 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5344011780 tx=0x7f5344011b40 comp rx=0 tx=0).stop 2024-03-12T05:45:52.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.808+0000 7f5350398700 1 -- 172.21.15.76:0/2456874607 shutdown_connections 2024-03-12T05:45:52.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.808+0000 7f5350398700 1 --2- 172.21.15.76:0/2456874607 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5348107760 0x7f534819b480 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:45:52.286 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.808+0000 7f5350398700 1 -- 172.21.15.76:0/2456874607 >> 172.21.15.76:0/2456874607 conn(0x7f5348102de0 msgr2=0x7f5348103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:45:52.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.808+0000 7f5350398700 1 -- 172.21.15.76:0/2456874607 shutdown_connections 2024-03-12T05:45:52.287 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:51.808+0000 7f5350398700 1 -- 172.21.15.76:0/2456874607 wait complete. 2024-03-12T05:45:52.287 INFO:teuthology.orchestra.run.smithi076.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-12T05:45:52.310 INFO:teuthology.orchestra.run.smithi076.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-12T05:45:52.310 INFO:teuthology.orchestra.run.smithi076.stdout:Creating mgr... 2024-03-12T05:45:52.310 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-12T05:45:52.311 INFO:teuthology.orchestra.run.smithi076.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-12T05:45:52.559 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9f336de8-e033-11ee-95c8-87774f69a715@mgr.x 2024-03-12T05:45:52.559 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9f336de8-e033-11ee-95c8-87774f69a715@mgr.x.service: Unit ceph-9f336de8-e033-11ee-95c8-87774f69a715@mgr.x.service not loaded. 2024-03-12T05:45:52.707 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9f336de8-e033-11ee-95c8-87774f69a715.target.wants/ceph-9f336de8-e033-11ee-95c8-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-9f336de8-e033-11ee-95c8-87774f69a715@.service. 2024-03-12T05:45:52.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:52 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2456874607' entity='client.admin' 2024-03-12T05:45:52.986 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:52 smithi076 podman[32313]: 2024-03-12 05:45:52.842922981 +0000 UTC m=+0.017906849 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-12T05:45:53.228 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T05:45:53.228 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-03-12T05:45:53.234 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T05:45:53.235 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-03-12T05:45:53.235 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-03-12T05:45:53.235 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to enable service . firewalld.service is not available 2024-03-12T05:45:53.243 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T05:45:53.243 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-03-12T05:45:53.250 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T05:45:53.250 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-03-12T05:45:53.250 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-03-12T05:45:53.250 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-12T05:45:53.250 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr to start... 2024-03-12T05:45:53.250 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr... 2024-03-12T05:45:53.357 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:53 smithi076 podman[32313]: 2024-03-12 05:45:53.148070748 +0000 UTC m=+0.323054629 container init 04d3f8c04590debd9ea085e82bc185eddc5ca97569d443fe9401f32c39e175a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-12T05:45:53.358 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:53 smithi076 podman[32313]: 2024-03-12 05:45:53.150448611 +0000 UTC m=+0.325432482 container start 04d3f8c04590debd9ea085e82bc185eddc5ca97569d443fe9401f32c39e175a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:45:53.358 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:53 smithi076 bash[32313]: 04d3f8c04590debd9ea085e82bc185eddc5ca97569d443fe9401f32c39e175a5 2024-03-12T05:45:53.358 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:53 smithi076 systemd[1]: Started Ceph mgr.x for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:45:53.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:53 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4192200694' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T05:45:53.986 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:53 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:53.704+0000 7fc394f74200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:45:53.986 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:53 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:53.801+0000 7fc394f74200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:45:54.307 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-12T05:45:54.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:45:54.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "9f336de8-e033-11ee-95c8-87774f69a715", 2024-03-12T05:45:54.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T05:45:54.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T05:45:54.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T05:45:54.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T05:45:54.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:45:54.308 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T05:45:54.309 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T05:45:54.320 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-03-12T05:45:54.320 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:45:54.320 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T05:45:54.320 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T05:45:54.320 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:45:54.320 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-12T05:45:54.320 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T05:45:54.320 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:45:54.320 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T05:45:54.320 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T05:45:54.320 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:45:54.320 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T05:45:54.320 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:45:54.321 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T05:45:54.321 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T05:45:54.321 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T05:45:54.321 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T05:45:54.321 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T05:45:54.321 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T05:45:54.321 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:45:54.321 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T05:45:54.321 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T05:45:54.321 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T05:45:54.321 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T05:45:54.321 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T05:45:54.321 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T05:45:54.322 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T05:45:54.322 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T05:45:54.322 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T05:45:54.322 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:45:54.322 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T05:45:54.322 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:45:54.322 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T05:45:54.322 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T05:45:54.322 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:45:54.322 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T05:45:54.322 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T05:45:54.322 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T05:45:54.323 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T05:45:54.323 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T05:45:54.323 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T05:45:54.323 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T05:45:54.323 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:45:54.323 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:45:54.323 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:45:54.323 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T05:45:54.323 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:45:54.323 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T05:45:46.456450+0000", 2024-03-12T05:45:54.323 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:45:54.323 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:45:54.323 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T05:45:54.323 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:45:54.324 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.650+0000 7f00a9caa700 1 Processor -- start 2024-03-12T05:45:54.324 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.650+0000 7f00a9caa700 1 -- start start 2024-03-12T05:45:54.324 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.650+0000 7f00a9caa700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00a4107b60 0x7f00a4107f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:45:54.324 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.650+0000 7f00a9caa700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a4108550 con 0x7f00a4107b60 2024-03-12T05:45:54.324 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.651+0000 7f00a37fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00a4107b60 0x7f00a4107f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:45:54.324 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.651+0000 7f00a37fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00a4107b60 0x7f00a4107f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58778/0 (socket says 172.21.15.76:58778) 2024-03-12T05:45:54.324 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.651+0000 7f00a37fe700 1 -- 172.21.15.76:0/305096331 learned_addr learned my addr 172.21.15.76:0/305096331 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:45:54.324 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.651+0000 7f00a37fe700 1 -- 172.21.15.76:0/305096331 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a41067c0 con 0x7f00a4107b60 2024-03-12T05:45:54.324 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.651+0000 7f00a37fe700 1 --2- 172.21.15.76:0/305096331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00a4107b60 0x7f00a4107f80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f00980098a0 tx=0x7f0098009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9dc0bc073a246b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:45:54.324 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.651+0000 7f00a27fc700 1 -- 172.21.15.76:0/305096331 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0098004160 con 0x7f00a4107b60 2024-03-12T05:45:54.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.651+0000 7f00a27fc700 1 -- 172.21.15.76:0/305096331 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00980042c0 con 0x7f00a4107b60 2024-03-12T05:45:54.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.651+0000 7f00a27fc700 1 -- 172.21.15.76:0/305096331 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0098011ac0 con 0x7f00a4107b60 2024-03-12T05:45:54.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.652+0000 7f00a9caa700 1 -- 172.21.15.76:0/305096331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00a4107b60 msgr2=0x7f00a4107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:45:54.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.652+0000 7f00a9caa700 1 --2- 172.21.15.76:0/305096331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00a4107b60 0x7f00a4107f80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f00980098a0 tx=0x7f0098009bb0 comp rx=0 tx=0).stop 2024-03-12T05:45:54.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.652+0000 7f00a9caa700 1 -- 172.21.15.76:0/305096331 shutdown_connections 2024-03-12T05:45:54.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.652+0000 7f00a9caa700 1 --2- 172.21.15.76:0/305096331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00a4107b60 0x7f00a4107f80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:45:54.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.652+0000 7f00a9caa700 1 -- 172.21.15.76:0/305096331 >> 172.21.15.76:0/305096331 conn(0x7f00a41013d0 msgr2=0x7f00a41037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:45:54.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.652+0000 7f00a9caa700 1 -- 172.21.15.76:0/305096331 shutdown_connections 2024-03-12T05:45:54.325 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.652+0000 7f00a9caa700 1 -- 172.21.15.76:0/305096331 wait complete. 2024-03-12T05:45:54.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.653+0000 7f00a9caa700 1 Processor -- start 2024-03-12T05:45:54.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.653+0000 7f00a9caa700 1 -- start start 2024-03-12T05:45:54.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.653+0000 7f00a9caa700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00a4107b60 0x7f00a4106130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:45:54.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.653+0000 7f00a9caa700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a4106670 con 0x7f00a4107b60 2024-03-12T05:45:54.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.653+0000 7f00a37fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00a4107b60 0x7f00a4106130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:45:54.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.653+0000 7f00a37fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00a4107b60 0x7f00a4106130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58792/0 (socket says 172.21.15.76:58792) 2024-03-12T05:45:54.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.653+0000 7f00a37fe700 1 -- 172.21.15.76:0/4192200694 learned_addr learned my addr 172.21.15.76:0/4192200694 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:45:54.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.653+0000 7f00a37fe700 1 -- 172.21.15.76:0/4192200694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0098009550 con 0x7f00a4107b60 2024-03-12T05:45:54.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.654+0000 7f00a37fe700 1 --2- 172.21.15.76:0/4192200694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00a4107b60 0x7f00a4106130 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0098011a70 tx=0x7f0098011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:45:54.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.654+0000 7f00a0ff9700 1 -- 172.21.15.76:0/4192200694 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00980039c0 con 0x7f00a4107b60 2024-03-12T05:45:54.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.654+0000 7f00a0ff9700 1 -- 172.21.15.76:0/4192200694 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0098003b20 con 0x7f00a4107b60 2024-03-12T05:45:54.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.654+0000 7f00a0ff9700 1 -- 172.21.15.76:0/4192200694 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0098024610 con 0x7f00a4107b60 2024-03-12T05:45:54.326 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.654+0000 7f00a9caa700 1 -- 172.21.15.76:0/4192200694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a419e1d0 con 0x7f00a4107b60 2024-03-12T05:45:54.327 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.654+0000 7f00a9caa700 1 -- 172.21.15.76:0/4192200694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a419e670 con 0x7f00a4107b60 2024-03-12T05:45:54.327 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.654+0000 7f00a0ff9700 1 -- 172.21.15.76:0/4192200694 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0098024be0 con 0x7f00a4107b60 2024-03-12T05:45:54.327 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.654+0000 7f00a0ff9700 1 -- 172.21.15.76:0/4192200694 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0098019960 con 0x7f00a4107b60 2024-03-12T05:45:54.327 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.655+0000 7f00a9caa700 1 -- 172.21.15.76:0/4192200694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a410aa00 con 0x7f00a4107b60 2024-03-12T05:45:54.327 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.659+0000 7f00a0ff9700 1 -- 172.21.15.76:0/4192200694 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f009801aa10 con 0x7f00a4107b60 2024-03-12T05:45:54.327 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.730+0000 7f00a9caa700 1 -- 172.21.15.76:0/4192200694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f00a419e8b0 con 0x7f00a4107b60 2024-03-12T05:45:54.327 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.731+0000 7f00a0ff9700 1 -- 172.21.15.76:0/4192200694 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f009801abb0 con 0x7f00a4107b60 2024-03-12T05:45:54.327 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.732+0000 7f00a9caa700 1 -- 172.21.15.76:0/4192200694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00a4107b60 msgr2=0x7f00a4106130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:45:54.327 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.732+0000 7f00a9caa700 1 --2- 172.21.15.76:0/4192200694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00a4107b60 0x7f00a4106130 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0098011a70 tx=0x7f0098011b50 comp rx=0 tx=0).stop 2024-03-12T05:45:54.327 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.732+0000 7f00a9caa700 1 -- 172.21.15.76:0/4192200694 shutdown_connections 2024-03-12T05:45:54.327 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.732+0000 7f00a9caa700 1 --2- 172.21.15.76:0/4192200694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00a4107b60 0x7f00a4106130 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:45:54.327 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.732+0000 7f00a9caa700 1 -- 172.21.15.76:0/4192200694 >> 172.21.15.76:0/4192200694 conn(0x7f00a41013d0 msgr2=0x7f00a4101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:45:54.327 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.732+0000 7f00a9caa700 1 -- 172.21.15.76:0/4192200694 shutdown_connections 2024-03-12T05:45:54.327 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:53.732+0000 7f00a9caa700 1 -- 172.21.15.76:0/4192200694 wait complete. 2024-03-12T05:45:54.328 INFO:teuthology.orchestra.run.smithi076.stdout:mgr not available, waiting (1/15)... 2024-03-12T05:45:54.735 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:54 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:54.441+0000 7fc394f74200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:45:54.735 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:54 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:54.531+0000 7fc394f74200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:45:54.735 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:54 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:54.668+0000 7fc394f74200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:45:55.109 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:54 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:54.761+0000 7fc394f74200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:45:55.110 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:54 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:54.840+0000 7fc394f74200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:45:55.110 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:55 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:55.108+0000 7fc394f74200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:45:55.395 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:55 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:55.216+0000 7fc394f74200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:45:55.395 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:55 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:55.301+0000 7fc394f74200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:45:55.395 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:55 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:55.394+0000 7fc394f74200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:45:55.735 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:55 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:55.567+0000 7fc394f74200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:45:56.585 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:56 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:56.546+0000 7fc394f74200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:45:56.860 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:56 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:56.678+0000 7fc394f74200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:45:57.112 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:56 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/211699660' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T05:45:57.112 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:57 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:57.022+0000 7fc394f74200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:45:57.112 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:57 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:57.111+0000 7fc394f74200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:45:57.427 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:57 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:57.303+0000 7fc394f74200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:45:57.431 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-12T05:45:57.431 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "9f336de8-e033-11ee-95c8-87774f69a715", 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T05:45:57.432 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:45:57.433 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T05:45:57.433 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T05:45:57.433 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:45:57.433 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T05:45:57.433 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:45:57.433 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T05:45:57.433 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T05:45:57.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T05:45:57.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T05:45:57.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T05:45:57.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T05:45:57.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:45:57.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T05:45:57.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T05:45:57.442 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T05:45:57.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T05:45:57.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T05:45:57.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T05:45:57.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T05:45:57.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T05:45:57.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T05:45:57.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:45:57.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T05:45:57.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:45:57.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T05:45:57.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T05:45:57.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:45:57.443 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T05:45:46.456450+0000", 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T05:45:57.444 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:45:57.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.779+0000 7f6e9b1e7700 1 Processor -- start 2024-03-12T05:45:57.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.779+0000 7f6e9b1e7700 1 -- start start 2024-03-12T05:45:57.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.780+0000 7f6e9b1e7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e94108160 0x7f6e9410a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:45:57.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.780+0000 7f6e9b1e7700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9407ae70 con 0x7f6e94108160 2024-03-12T05:45:57.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.780+0000 7f6e98f83700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e94108160 0x7f6e9410a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:45:57.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.780+0000 7f6e98f83700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e94108160 0x7f6e9410a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58794/0 (socket says 172.21.15.76:58794) 2024-03-12T05:45:57.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.780+0000 7f6e98f83700 1 -- 172.21.15.76:0/3871720363 learned_addr learned my addr 172.21.15.76:0/3871720363 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:45:57.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.780+0000 7f6e98f83700 1 -- 172.21.15.76:0/3871720363 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9410aa90 con 0x7f6e94108160 2024-03-12T05:45:57.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.780+0000 7f6e98f83700 1 --2- 172.21.15.76:0/3871720363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e94108160 0x7f6e9410a550 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6e880098a0 tx=0x7f6e88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71cbf2e8426c78c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:45:57.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.780+0000 7f6e937fe700 1 -- 172.21.15.76:0/3871720363 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e88004160 con 0x7f6e94108160 2024-03-12T05:45:57.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.780+0000 7f6e937fe700 1 -- 172.21.15.76:0/3871720363 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e880042c0 con 0x7f6e94108160 2024-03-12T05:45:57.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.780+0000 7f6e937fe700 1 -- 172.21.15.76:0/3871720363 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e88011ac0 con 0x7f6e94108160 2024-03-12T05:45:57.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.781+0000 7f6e9b1e7700 1 -- 172.21.15.76:0/3871720363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e94108160 msgr2=0x7f6e9410a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:45:57.445 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.781+0000 7f6e9b1e7700 1 --2- 172.21.15.76:0/3871720363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e94108160 0x7f6e9410a550 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6e880098a0 tx=0x7f6e88009bb0 comp rx=0 tx=0).stop 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.781+0000 7f6e9b1e7700 1 -- 172.21.15.76:0/3871720363 shutdown_connections 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.781+0000 7f6e9b1e7700 1 --2- 172.21.15.76:0/3871720363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e94108160 0x7f6e9410a550 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.781+0000 7f6e9b1e7700 1 -- 172.21.15.76:0/3871720363 >> 172.21.15.76:0/3871720363 conn(0x7f6e94101bb0 msgr2=0x7f6e94103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.781+0000 7f6e9b1e7700 1 -- 172.21.15.76:0/3871720363 shutdown_connections 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.781+0000 7f6e9b1e7700 1 -- 172.21.15.76:0/3871720363 wait complete. 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.782+0000 7f6e9b1e7700 1 Processor -- start 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.782+0000 7f6e9b1e7700 1 -- start start 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.782+0000 7f6e9b1e7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e94108160 0x7f6e9419b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.782+0000 7f6e9b1e7700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9419b9e0 con 0x7f6e94108160 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.782+0000 7f6e98f83700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e94108160 0x7f6e9419b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.782+0000 7f6e98f83700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e94108160 0x7f6e9419b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58802/0 (socket says 172.21.15.76:58802) 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.782+0000 7f6e98f83700 1 -- 172.21.15.76:0/211699660 learned_addr learned my addr 172.21.15.76:0/211699660 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.782+0000 7f6e98f83700 1 -- 172.21.15.76:0/211699660 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e88009550 con 0x7f6e94108160 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.782+0000 7f6e98f83700 1 --2- 172.21.15.76:0/211699660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e94108160 0x7f6e9419b4a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6e88011a70 tx=0x7f6e88011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:45:57.446 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.782+0000 7f6e91ffb700 1 -- 172.21.15.76:0/211699660 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e880039c0 con 0x7f6e94108160 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.782+0000 7f6e9b1e7700 1 -- 172.21.15.76:0/211699660 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9419bbe0 con 0x7f6e94108160 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.782+0000 7f6e91ffb700 1 -- 172.21.15.76:0/211699660 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e88003b20 con 0x7f6e94108160 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.782+0000 7f6e91ffb700 1 -- 172.21.15.76:0/211699660 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e88024610 con 0x7f6e94108160 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.783+0000 7f6e9b1e7700 1 -- 172.21.15.76:0/211699660 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9419c080 con 0x7f6e94108160 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.783+0000 7f6e91ffb700 1 -- 172.21.15.76:0/211699660 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6e88024c00 con 0x7f6e94108160 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.783+0000 7f6e91ffb700 1 -- 172.21.15.76:0/211699660 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6e88019920 con 0x7f6e94108160 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.784+0000 7f6e9b1e7700 1 -- 172.21.15.76:0/211699660 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e9410b890 con 0x7f6e94108160 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.786+0000 7f6e91ffb700 1 -- 172.21.15.76:0/211699660 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6e8801aa10 con 0x7f6e94108160 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.859+0000 7f6e9b1e7700 1 -- 172.21.15.76:0/211699660 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6e9419ebf0 con 0x7f6e94108160 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.859+0000 7f6e91ffb700 1 -- 172.21.15.76:0/211699660 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f6e8801abb0 con 0x7f6e94108160 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.861+0000 7f6e9b1e7700 1 -- 172.21.15.76:0/211699660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e94108160 msgr2=0x7f6e9419b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.861+0000 7f6e9b1e7700 1 --2- 172.21.15.76:0/211699660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e94108160 0x7f6e9419b4a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6e88011a70 tx=0x7f6e88011b50 comp rx=0 tx=0).stop 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.861+0000 7f6e9b1e7700 1 -- 172.21.15.76:0/211699660 shutdown_connections 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.861+0000 7f6e9b1e7700 1 --2- 172.21.15.76:0/211699660 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e94108160 0x7f6e9419b4a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:45:57.447 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.861+0000 7f6e9b1e7700 1 -- 172.21.15.76:0/211699660 >> 172.21.15.76:0/211699660 conn(0x7f6e94101bb0 msgr2=0x7f6e94102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:45:57.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.861+0000 7f6e9b1e7700 1 -- 172.21.15.76:0/211699660 shutdown_connections 2024-03-12T05:45:57.448 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:56.861+0000 7f6e9b1e7700 1 -- 172.21.15.76:0/211699660 wait complete. 2024-03-12T05:45:57.448 INFO:teuthology.orchestra.run.smithi076.stdout:mgr not available, waiting (2/15)... 2024-03-12T05:45:57.735 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:57 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:57.485+0000 7fc394f74200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:45:58.014 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:57 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:57.760+0000 7fc394f74200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:45:58.014 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:58 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:58.012+0000 7fc394f74200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:45:58.485 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:58 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:58.242+0000 7fc394f74200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:45:59.485 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:59.019+0000 7fc394f74200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:45:59.863 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:59.583+0000 7fc394f74200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:45:59.863 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:59.681+0000 7fc394f74200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:46:00.235 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:45:59.902+0000 7fc394f74200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:46:00.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-mon[32095]: Activating manager daemon x 2024-03-12T05:46:00.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-mon[32095]: mgrmap e2: x(active, starting, since 0.00316886s) 2024-03-12T05:46:00.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-mon[32095]: from='mgr.14100 172.21.15.76:0/1559665045' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T05:46:00.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-mon[32095]: from='mgr.14100 172.21.15.76:0/1559665045' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T05:46:00.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-mon[32095]: from='mgr.14100 172.21.15.76:0/1559665045' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T05:46:00.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-mon[32095]: from='mgr.14100 172.21.15.76:0/1559665045' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:46:00.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-mon[32095]: from='mgr.14100 172.21.15.76:0/1559665045' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T05:46:00.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-mon[32095]: Manager daemon x is now available 2024-03-12T05:46:00.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-mon[32095]: from='mgr.14100 172.21.15.76:0/1559665045' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T05:46:00.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-mon[32095]: from='mgr.14100 172.21.15.76:0/1559665045' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T05:46:00.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-mon[32095]: from='mgr.14100 172.21.15.76:0/1559665045' entity='mgr.x' 2024-03-12T05:46:00.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-mon[32095]: from='mgr.14100 172.21.15.76:0/1559665045' entity='mgr.x' 2024-03-12T05:46:00.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:45:59 smithi076 ceph-mon[32095]: from='mgr.14100 172.21.15.76:0/1559665045' entity='mgr.x' 2024-03-12T05:46:00.588 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-12T05:46:00.588 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:46:00.588 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "9f336de8-e033-11ee-95c8-87774f69a715", 2024-03-12T05:46:00.588 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T05:46:00.588 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T05:46:00.589 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T05:46:00.589 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T05:46:00.589 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:46:00.589 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T05:46:00.589 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T05:46:00.589 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-03-12T05:46:00.589 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:46:00.589 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T05:46:00.589 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T05:46:00.589 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:46:00.589 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-12T05:46:00.589 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T05:46:00.589 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:46:00.589 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T05:46:00.590 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T05:46:00.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T05:46:00.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T05:46:00.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T05:46:00.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:46:00.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T05:46:00.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T05:46:00.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T05:46:00.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T05:46:00.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T05:46:00.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T05:46:00.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T05:46:00.598 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T05:46:00.599 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:46:00.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:46:00.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:46:00.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T05:46:00.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:46:00.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T05:45:46.456450+0000", 2024-03-12T05:46:00.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:46:00.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:46:00.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T05:46:00.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:46:00.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.885+0000 7feaf27e6700 1 Processor -- start 2024-03-12T05:46:00.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.885+0000 7feaf27e6700 1 -- start start 2024-03-12T05:46:00.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.886+0000 7feaf27e6700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feaec108200 0x7feaec10a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:00.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.886+0000 7feaf27e6700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaec07add0 con 0x7feaec108200 2024-03-12T05:46:00.600 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.886+0000 7feaebfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feaec108200 0x7feaec10a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.886+0000 7feaebfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feaec108200 0x7feaec10a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58816/0 (socket says 172.21.15.76:58816) 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.886+0000 7feaebfff700 1 -- 172.21.15.76:0/3953951341 learned_addr learned my addr 172.21.15.76:0/3953951341 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.886+0000 7feaebfff700 1 -- 172.21.15.76:0/3953951341 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaec07af10 con 0x7feaec108200 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.886+0000 7feaebfff700 1 --2- 172.21.15.76:0/3953951341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feaec108200 0x7feaec10a5f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7feadc0098a0 tx=0x7feadc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c26c11178002216 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.886+0000 7feaeaffd700 1 -- 172.21.15.76:0/3953951341 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feadc004160 con 0x7feaec108200 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.886+0000 7feaeaffd700 1 -- 172.21.15.76:0/3953951341 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feadc0042c0 con 0x7feaec108200 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.886+0000 7feaeaffd700 1 -- 172.21.15.76:0/3953951341 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feadc011ac0 con 0x7feaec108200 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.887+0000 7feaf27e6700 1 -- 172.21.15.76:0/3953951341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feaec108200 msgr2=0x7feaec10a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.887+0000 7feaf27e6700 1 --2- 172.21.15.76:0/3953951341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feaec108200 0x7feaec10a5f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7feadc0098a0 tx=0x7feadc009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.887+0000 7feaf27e6700 1 -- 172.21.15.76:0/3953951341 shutdown_connections 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.887+0000 7feaf27e6700 1 --2- 172.21.15.76:0/3953951341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feaec108200 0x7feaec10a5f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.887+0000 7feaf27e6700 1 -- 172.21.15.76:0/3953951341 >> 172.21.15.76:0/3953951341 conn(0x7feaec101bb0 msgr2=0x7feaec103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.887+0000 7feaf27e6700 1 -- 172.21.15.76:0/3953951341 shutdown_connections 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.887+0000 7feaf27e6700 1 -- 172.21.15.76:0/3953951341 wait complete. 2024-03-12T05:46:00.601 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.888+0000 7feaf27e6700 1 Processor -- start 2024-03-12T05:46:00.602 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.888+0000 7feaf27e6700 1 -- start start 2024-03-12T05:46:00.602 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.888+0000 7feaf27e6700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feaec108200 0x7feaec19b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:00.602 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.888+0000 7feaf27e6700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaec19b9d0 con 0x7feaec108200 2024-03-12T05:46:00.602 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.888+0000 7feaebfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feaec108200 0x7feaec19b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:00.602 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.888+0000 7feaebfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feaec108200 0x7feaec19b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58826/0 (socket says 172.21.15.76:58826) 2024-03-12T05:46:00.602 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.888+0000 7feaebfff700 1 -- 172.21.15.76:0/1964831825 learned_addr learned my addr 172.21.15.76:0/1964831825 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:00.602 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.889+0000 7feaebfff700 1 -- 172.21.15.76:0/1964831825 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feadc009550 con 0x7feaec108200 2024-03-12T05:46:00.602 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.889+0000 7feaebfff700 1 --2- 172.21.15.76:0/1964831825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feaec108200 0x7feaec19b490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feadc011a70 tx=0x7feadc011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:00.602 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.889+0000 7feae97fa700 1 -- 172.21.15.76:0/1964831825 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feadc0039c0 con 0x7feaec108200 2024-03-12T05:46:00.602 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.889+0000 7feae97fa700 1 -- 172.21.15.76:0/1964831825 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feadc003b20 con 0x7feaec108200 2024-03-12T05:46:00.602 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.889+0000 7feaf27e6700 1 -- 172.21.15.76:0/1964831825 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaec19bbd0 con 0x7feaec108200 2024-03-12T05:46:00.602 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.889+0000 7feae97fa700 1 -- 172.21.15.76:0/1964831825 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feadc024610 con 0x7feaec108200 2024-03-12T05:46:00.602 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.889+0000 7feaf27e6700 1 -- 172.21.15.76:0/1964831825 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaec19c070 con 0x7feaec108200 2024-03-12T05:46:00.603 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.889+0000 7feae97fa700 1 -- 172.21.15.76:0/1964831825 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7feadc024c10 con 0x7feaec108200 2024-03-12T05:46:00.603 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.889+0000 7feae97fa700 1 -- 172.21.15.76:0/1964831825 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7feadc0198e0 con 0x7feaec108200 2024-03-12T05:46:00.603 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.890+0000 7feaf27e6700 1 -- 172.21.15.76:0/1964831825 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaec10b8c0 con 0x7feaec108200 2024-03-12T05:46:00.603 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.892+0000 7feae97fa700 1 -- 172.21.15.76:0/1964831825 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7feadc01aa10 con 0x7feaec108200 2024-03-12T05:46:00.603 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.917+0000 7feae97fa700 1 -- 172.21.15.76:0/1964831825 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 2) v1 ==== 45525+0+0 (secure 0 0 0) 0x7feadc019b40 con 0x7feaec108200 2024-03-12T05:46:00.603 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.968+0000 7feaf27e6700 1 -- 172.21.15.76:0/1964831825 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7feaec10bab0 con 0x7feaec108200 2024-03-12T05:46:00.603 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.968+0000 7feae97fa700 1 -- 172.21.15.76:0/1964831825 <== mon.0 v2:172.21.15.76:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7feadc0109d0 con 0x7feaec108200 2024-03-12T05:46:00.603 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.969+0000 7feaf27e6700 1 -- 172.21.15.76:0/1964831825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feaec108200 msgr2=0x7feaec19b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:00.603 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.969+0000 7feaf27e6700 1 --2- 172.21.15.76:0/1964831825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feaec108200 0x7feaec19b490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feadc011a70 tx=0x7feadc011b50 comp rx=0 tx=0).stop 2024-03-12T05:46:00.603 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.969+0000 7feaf27e6700 1 -- 172.21.15.76:0/1964831825 shutdown_connections 2024-03-12T05:46:00.603 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.969+0000 7feaf27e6700 1 --2- 172.21.15.76:0/1964831825 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feaec108200 0x7feaec19b490 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:00.603 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.969+0000 7feaf27e6700 1 -- 172.21.15.76:0/1964831825 >> 172.21.15.76:0/1964831825 conn(0x7feaec101bb0 msgr2=0x7feaec102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:00.603 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.970+0000 7feaf27e6700 1 -- 172.21.15.76:0/1964831825 shutdown_connections 2024-03-12T05:46:00.603 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:45:59.970+0000 7feaf27e6700 1 -- 172.21.15.76:0/1964831825 wait complete. 2024-03-12T05:46:00.604 INFO:teuthology.orchestra.run.smithi076.stdout:mgr not available, waiting (3/15)... 2024-03-12T05:46:01.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:00 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1964831825' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T05:46:01.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:00 smithi076 ceph-mon[32095]: mgrmap e3: x(active, since 1.0068s) 2024-03-12T05:46:03.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:02 smithi076 ceph-mon[32095]: mgrmap e4: x(active, since 2s) 2024-03-12T05:46:03.886 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-12T05:46:03.886 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:46:03.886 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsid": "9f336de8-e033-11ee-95c8-87774f69a715", 2024-03-12T05:46:03.886 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T05:46:03.886 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T05:46:03.886 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T05:46:03.886 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T05:46:03.886 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 0 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "a" 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T05:46:03.887 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:46:03.888 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T05:46:03.888 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T05:46:03.888 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T05:46:03.897 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T05:46:03.897 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T05:46:03.897 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T05:46:03.897 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:46:03.897 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T05:46:03.897 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T05:46:03.897 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T05:46:03.897 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T05:46:03.897 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T05:46:03.898 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T05:46:03.898 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T05:46:03.898 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T05:46:03.898 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T05:46:03.898 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:46:03.898 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T05:46:03.898 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:46:03.898 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T05:46:03.898 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T05:46:03.898 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:46:03.898 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T05:46:03.898 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T05:46:03.898 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T05:46:03.898 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T05:46:03.899 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T05:46:03.899 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T05:46:03.899 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T05:46:03.899 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ], 2024-03-12T05:46:03.899 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:46:03.899 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:46:03.899 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T05:46:03.899 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T05:46:03.899 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T05:45:46.456450+0000", 2024-03-12T05:46:03.899 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T05:46:03.899 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout }, 2024-03-12T05:46:03.899 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T05:46:03.899 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:46:03.900 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.022+0000 7f910481e700 1 Processor -- start 2024-03-12T05:46:03.900 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.022+0000 7f910481e700 1 -- start start 2024-03-12T05:46:03.900 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.022+0000 7f910481e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910007ce30 0x7f910007b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:03.900 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.022+0000 7f910481e700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910007b820 con 0x7f910007ce30 2024-03-12T05:46:03.900 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.022+0000 7f90fe59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910007ce30 0x7f910007b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:03.900 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.022+0000 7f90fe59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910007ce30 0x7f910007b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33806/0 (socket says 172.21.15.76:33806) 2024-03-12T05:46:03.900 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.022+0000 7f90fe59c700 1 -- 172.21.15.76:0/161696655 learned_addr learned my addr 172.21.15.76:0/161696655 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:03.900 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.023+0000 7f90fe59c700 1 -- 172.21.15.76:0/161696655 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910007b960 con 0x7f910007ce30 2024-03-12T05:46:03.900 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.023+0000 7f90fe59c700 1 --2- 172.21.15.76:0/161696655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910007ce30 0x7f910007b2e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f90f00098a0 tx=0x7f90f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac430b241a4ec66 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:03.900 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.023+0000 7f90fd59a700 1 -- 172.21.15.76:0/161696655 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90f0004160 con 0x7f910007ce30 2024-03-12T05:46:03.900 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.023+0000 7f90fd59a700 1 -- 172.21.15.76:0/161696655 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90f00042c0 con 0x7f910007ce30 2024-03-12T05:46:03.900 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.023+0000 7f90fd59a700 1 -- 172.21.15.76:0/161696655 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90f0011ac0 con 0x7f910007ce30 2024-03-12T05:46:03.901 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.024+0000 7f910481e700 1 -- 172.21.15.76:0/161696655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910007ce30 msgr2=0x7f910007b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:03.901 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.024+0000 7f910481e700 1 --2- 172.21.15.76:0/161696655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910007ce30 0x7f910007b2e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f90f00098a0 tx=0x7f90f0009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:03.901 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.024+0000 7f910481e700 1 -- 172.21.15.76:0/161696655 shutdown_connections 2024-03-12T05:46:03.901 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.024+0000 7f910481e700 1 --2- 172.21.15.76:0/161696655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910007ce30 0x7f910007b2e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:03.901 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.024+0000 7f910481e700 1 -- 172.21.15.76:0/161696655 >> 172.21.15.76:0/161696655 conn(0x7f9100102db0 msgr2=0x7f91001051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:03.901 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.024+0000 7f910481e700 1 -- 172.21.15.76:0/161696655 shutdown_connections 2024-03-12T05:46:03.901 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.024+0000 7f910481e700 1 -- 172.21.15.76:0/161696655 wait complete. 2024-03-12T05:46:03.901 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.024+0000 7f910481e700 1 Processor -- start 2024-03-12T05:46:03.901 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.025+0000 7f910481e700 1 -- start start 2024-03-12T05:46:03.901 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.025+0000 7f910481e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910007ce30 0x7f910019d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:03.901 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.025+0000 7f910481e700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910019db90 con 0x7f910007ce30 2024-03-12T05:46:03.901 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.025+0000 7f90fe59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910007ce30 0x7f910019d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:03.902 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.025+0000 7f90fe59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910007ce30 0x7f910019d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33818/0 (socket says 172.21.15.76:33818) 2024-03-12T05:46:03.902 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.025+0000 7f90fe59c700 1 -- 172.21.15.76:0/1837099778 learned_addr learned my addr 172.21.15.76:0/1837099778 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:03.902 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.025+0000 7f90fe59c700 1 -- 172.21.15.76:0/1837099778 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f0009550 con 0x7f910007ce30 2024-03-12T05:46:03.902 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.025+0000 7f90fe59c700 1 --2- 172.21.15.76:0/1837099778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910007ce30 0x7f910019d650 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009fb0 tx=0x7f90f0011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:03.902 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.025+0000 7f90ef7fe700 1 -- 172.21.15.76:0/1837099778 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90f00038b0 con 0x7f910007ce30 2024-03-12T05:46:03.902 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.025+0000 7f90ef7fe700 1 -- 172.21.15.76:0/1837099778 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90f0003a10 con 0x7f910007ce30 2024-03-12T05:46:03.902 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.025+0000 7f90ef7fe700 1 -- 172.21.15.76:0/1837099778 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90f00195e0 con 0x7f910007ce30 2024-03-12T05:46:03.902 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.025+0000 7f910481e700 1 -- 172.21.15.76:0/1837099778 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f910019dd90 con 0x7f910007ce30 2024-03-12T05:46:03.902 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.025+0000 7f910481e700 1 -- 172.21.15.76:0/1837099778 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910019e230 con 0x7f910007ce30 2024-03-12T05:46:03.902 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.026+0000 7f90ef7fe700 1 -- 172.21.15.76:0/1837099778 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7f90f0019850 con 0x7f910007ce30 2024-03-12T05:46:03.902 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.026+0000 7f910481e700 1 -- 172.21.15.76:0/1837099778 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f910010da60 con 0x7f910007ce30 2024-03-12T05:46:03.903 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.026+0000 7f90ef7fe700 1 --2- 172.21.15.76:0/1837099778 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f90e8038c60 0x7f90e803b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:03.903 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.026+0000 7f90ef7fe700 1 -- 172.21.15.76:0/1837099778 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f90f004cf80 con 0x7f910007ce30 2024-03-12T05:46:03.903 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.027+0000 7f90fdd9b700 1 --2- 172.21.15.76:0/1837099778 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f90e8038c60 0x7f90e803b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:03.903 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.027+0000 7f90fdd9b700 1 --2- 172.21.15.76:0/1837099778 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f90e8038c60 0x7f90e803b120 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f90f4006fd0 tx=0x7f90f4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:03.903 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.032+0000 7f90ef7fe700 1 -- 172.21.15.76:0/1837099778 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f90f0019b00 con 0x7f910007ce30 2024-03-12T05:46:03.903 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.302+0000 7f910481e700 1 -- 172.21.15.76:0/1837099778 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f91001a1000 con 0x7f910007ce30 2024-03-12T05:46:03.903 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.302+0000 7f90ef7fe700 1 -- 172.21.15.76:0/1837099778 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f90f00324f0 con 0x7f910007ce30 2024-03-12T05:46:03.903 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.305+0000 7f910481e700 1 -- 172.21.15.76:0/1837099778 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f90e8038c60 msgr2=0x7f90e803b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:03.903 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.305+0000 7f910481e700 1 --2- 172.21.15.76:0/1837099778 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f90e8038c60 0x7f90e803b120 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f90f4006fd0 tx=0x7f90f4006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:03.903 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.305+0000 7f910481e700 1 -- 172.21.15.76:0/1837099778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910007ce30 msgr2=0x7f910019d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:03.903 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.305+0000 7f910481e700 1 --2- 172.21.15.76:0/1837099778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910007ce30 0x7f910019d650 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009fb0 tx=0x7f90f0011b20 comp rx=0 tx=0).stop 2024-03-12T05:46:03.903 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.305+0000 7f910481e700 1 -- 172.21.15.76:0/1837099778 shutdown_connections 2024-03-12T05:46:03.903 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.305+0000 7f910481e700 1 --2- 172.21.15.76:0/1837099778 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f90e8038c60 0x7f90e803b120 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:03.904 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.305+0000 7f910481e700 1 --2- 172.21.15.76:0/1837099778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910007ce30 0x7f910019d650 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:03.904 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.305+0000 7f910481e700 1 -- 172.21.15.76:0/1837099778 >> 172.21.15.76:0/1837099778 conn(0x7f9100102db0 msgr2=0x7f9100103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:03.904 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.306+0000 7f910481e700 1 -- 172.21.15.76:0/1837099778 shutdown_connections 2024-03-12T05:46:03.904 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:03.306+0000 7f910481e700 1 -- 172.21.15.76:0/1837099778 wait complete. 2024-03-12T05:46:03.904 INFO:teuthology.orchestra.run.smithi076.stdout:mgr is available 2024-03-12T05:46:04.134 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:03 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1837099778' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T05:46:05.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-12T05:46:05.096 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [global] 2024-03-12T05:46:05.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout fsid = 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:46:05.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-12T05:46:05.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-12T05:46:05.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-12T05:46:05.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-12T05:46:05.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-12T05:46:05.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-12T05:46:05.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-12T05:46:05.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 2024-03-12T05:46:05.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout [osd] 2024-03-12T05:46:05.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-12T05:46:05.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-12T05:46:05.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-12T05:46:05.097 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-12T05:46:05.098 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-12T05:46:05.098 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.309+0000 7feff15c0700 1 Processor -- start 2024-03-12T05:46:05.098 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.309+0000 7feff15c0700 1 -- start start 2024-03-12T05:46:05.098 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.309+0000 7feff15c0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefec1076d0 0x7fefec107af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:05.098 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.309+0000 7feff15c0700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec1080c0 con 0x7fefec1076d0 2024-03-12T05:46:05.098 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.309+0000 7fefeaffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefec1076d0 0x7fefec107af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:05.098 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.309+0000 7fefeaffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefec1076d0 0x7fefec107af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33834/0 (socket says 172.21.15.76:33834) 2024-03-12T05:46:05.098 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.309+0000 7fefeaffd700 1 -- 172.21.15.76:0/3820949786 learned_addr learned my addr 172.21.15.76:0/3820949786 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:05.098 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.310+0000 7fefeaffd700 1 -- 172.21.15.76:0/3820949786 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec108880 con 0x7fefec1076d0 2024-03-12T05:46:05.098 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.310+0000 7fefeaffd700 1 --2- 172.21.15.76:0/3820949786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefec1076d0 0x7fefec107af0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fefdc0098a0 tx=0x7fefdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ed4c8c42c67eef server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:05.098 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.310+0000 7fefea7fc700 1 -- 172.21.15.76:0/3820949786 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefdc004160 con 0x7fefec1076d0 2024-03-12T05:46:05.098 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.310+0000 7fefea7fc700 1 -- 172.21.15.76:0/3820949786 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefdc0042c0 con 0x7fefec1076d0 2024-03-12T05:46:05.098 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.310+0000 7fefea7fc700 1 -- 172.21.15.76:0/3820949786 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefdc011ac0 con 0x7fefec1076d0 2024-03-12T05:46:05.099 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.310+0000 7feff15c0700 1 -- 172.21.15.76:0/3820949786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefec1076d0 msgr2=0x7fefec107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:05.099 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.310+0000 7feff15c0700 1 --2- 172.21.15.76:0/3820949786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefec1076d0 0x7fefec107af0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fefdc0098a0 tx=0x7fefdc009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:05.099 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.311+0000 7feff15c0700 1 -- 172.21.15.76:0/3820949786 shutdown_connections 2024-03-12T05:46:05.099 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.311+0000 7feff15c0700 1 --2- 172.21.15.76:0/3820949786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefec1076d0 0x7fefec107af0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:05.099 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.311+0000 7feff15c0700 1 -- 172.21.15.76:0/3820949786 >> 172.21.15.76:0/3820949786 conn(0x7fefec102d50 msgr2=0x7fefec105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:05.099 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.311+0000 7feff15c0700 1 -- 172.21.15.76:0/3820949786 shutdown_connections 2024-03-12T05:46:05.099 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.311+0000 7feff15c0700 1 -- 172.21.15.76:0/3820949786 wait complete. 2024-03-12T05:46:05.099 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.311+0000 7feff15c0700 1 Processor -- start 2024-03-12T05:46:05.099 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.311+0000 7feff15c0700 1 -- start start 2024-03-12T05:46:05.099 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.311+0000 7feff15c0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefec1076d0 0x7fefec19d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:05.099 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.312+0000 7feff15c0700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec19dc30 con 0x7fefec1076d0 2024-03-12T05:46:05.099 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.312+0000 7fefeaffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefec1076d0 0x7fefec19d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:05.099 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.312+0000 7fefeaffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefec1076d0 0x7fefec19d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33850/0 (socket says 172.21.15.76:33850) 2024-03-12T05:46:05.100 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.312+0000 7fefeaffd700 1 -- 172.21.15.76:0/385553776 learned_addr learned my addr 172.21.15.76:0/385553776 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:05.100 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.312+0000 7fefeaffd700 1 -- 172.21.15.76:0/385553776 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefdc009550 con 0x7fefec1076d0 2024-03-12T05:46:05.100 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.312+0000 7fefeaffd700 1 --2- 172.21.15.76:0/385553776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefec1076d0 0x7fefec19d6f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009fb0 tx=0x7fefdc011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:05.100 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.312+0000 7fefe8ff9700 1 -- 172.21.15.76:0/385553776 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefdc0038b0 con 0x7fefec1076d0 2024-03-12T05:46:05.100 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.312+0000 7fefe8ff9700 1 -- 172.21.15.76:0/385553776 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefdc003a10 con 0x7fefec1076d0 2024-03-12T05:46:05.100 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.312+0000 7feff15c0700 1 -- 172.21.15.76:0/385553776 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefec19de30 con 0x7fefec1076d0 2024-03-12T05:46:05.100 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.312+0000 7fefe8ff9700 1 -- 172.21.15.76:0/385553776 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefdc0195e0 con 0x7fefec1076d0 2024-03-12T05:46:05.100 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.312+0000 7feff15c0700 1 -- 172.21.15.76:0/385553776 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefec19e2d0 con 0x7fefec1076d0 2024-03-12T05:46:05.100 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.313+0000 7fefe8ff9700 1 -- 172.21.15.76:0/385553776 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7fefdc0198a0 con 0x7fefec1076d0 2024-03-12T05:46:05.100 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.313+0000 7fefe8ff9700 1 --2- 172.21.15.76:0/385553776 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fefd8038c10 0x7fefd803b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:05.100 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.313+0000 7fefe8ff9700 1 -- 172.21.15.76:0/385553776 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fefdc04cda0 con 0x7fefec1076d0 2024-03-12T05:46:05.100 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.313+0000 7feff15c0700 1 -- 172.21.15.76:0/385553776 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefec10bab0 con 0x7fefec1076d0 2024-03-12T05:46:05.100 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.313+0000 7fefe27fc700 1 --2- 172.21.15.76:0/385553776 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fefd8038c10 0x7fefd803b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:05.101 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.314+0000 7fefe27fc700 1 --2- 172.21.15.76:0/385553776 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fefd8038c10 0x7fefd803b0d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fefd4006fd0 tx=0x7fefd4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:05.101 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.318+0000 7fefe8ff9700 1 -- 172.21.15.76:0/385553776 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fefdc019b50 con 0x7fefec1076d0 2024-03-12T05:46:05.101 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.511+0000 7feff15c0700 1 -- 172.21.15.76:0/385553776 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fefec10bc70 con 0x7fefec1076d0 2024-03-12T05:46:05.101 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.512+0000 7fefe8ff9700 1 -- 172.21.15.76:0/385553776 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fefdc018be0 con 0x7fefec1076d0 2024-03-12T05:46:05.101 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.514+0000 7feff15c0700 1 -- 172.21.15.76:0/385553776 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fefd8038c10 msgr2=0x7fefd803b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:05.101 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.514+0000 7feff15c0700 1 --2- 172.21.15.76:0/385553776 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fefd8038c10 0x7fefd803b0d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fefd4006fd0 tx=0x7fefd4006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:05.101 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.514+0000 7feff15c0700 1 -- 172.21.15.76:0/385553776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefec1076d0 msgr2=0x7fefec19d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:05.101 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.514+0000 7feff15c0700 1 --2- 172.21.15.76:0/385553776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefec1076d0 0x7fefec19d6f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009fb0 tx=0x7fefdc011b20 comp rx=0 tx=0).stop 2024-03-12T05:46:05.101 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.514+0000 7feff15c0700 1 -- 172.21.15.76:0/385553776 shutdown_connections 2024-03-12T05:46:05.101 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.514+0000 7feff15c0700 1 --2- 172.21.15.76:0/385553776 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fefd8038c10 0x7fefd803b0d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:05.101 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.514+0000 7feff15c0700 1 --2- 172.21.15.76:0/385553776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fefec1076d0 0x7fefec19d6f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:05.101 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.514+0000 7feff15c0700 1 -- 172.21.15.76:0/385553776 >> 172.21.15.76:0/385553776 conn(0x7fefec102d50 msgr2=0x7fefec1037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:05.101 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.515+0000 7feff15c0700 1 -- 172.21.15.76:0/385553776 shutdown_connections 2024-03-12T05:46:05.102 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:04.515+0000 7feff15c0700 1 -- 172.21.15.76:0/385553776 wait complete. 2024-03-12T05:46:05.102 INFO:teuthology.orchestra.run.smithi076.stdout:Enabling cephadm module... 2024-03-12T05:46:05.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:04 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/385553776' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-12T05:46:06.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:05 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/544293443' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-12T05:46:06.276 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:06 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: ignoring --setuser ceph since I am not root 2024-03-12T05:46:06.276 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:06 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: ignoring --setgroup ceph since I am not root 2024-03-12T05:46:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.499+0000 7f4f94090700 1 Processor -- start 2024-03-12T05:46:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.499+0000 7f4f94090700 1 -- start start 2024-03-12T05:46:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.499+0000 7f4f94090700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4f8c108260 0x7f4f8c10a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.499+0000 7f4f94090700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c07ae10 con 0x7f4f8c108260 2024-03-12T05:46:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.500+0000 7f4f91e2c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4f8c108260 0x7f4f8c10a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.500+0000 7f4f91e2c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4f8c108260 0x7f4f8c10a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33854/0 (socket says 172.21.15.76:33854) 2024-03-12T05:46:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.500+0000 7f4f91e2c700 1 -- 172.21.15.76:0/3607919998 learned_addr learned my addr 172.21.15.76:0/3607919998 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.500+0000 7f4f91e2c700 1 -- 172.21.15.76:0/3607919998 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c07af50 con 0x7f4f8c108260 2024-03-12T05:46:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.500+0000 7f4f91e2c700 1 --2- 172.21.15.76:0/3607919998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4f8c108260 0x7f4f8c10a650 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4f880098a0 tx=0x7f4f88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280945e751741623 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.500+0000 7f4f90e2a700 1 -- 172.21.15.76:0/3607919998 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f88004160 con 0x7f4f8c108260 2024-03-12T05:46:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.500+0000 7f4f90e2a700 1 -- 172.21.15.76:0/3607919998 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f880042c0 con 0x7f4f8c108260 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.500+0000 7f4f90e2a700 1 -- 172.21.15.76:0/3607919998 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f88011ac0 con 0x7f4f8c108260 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.501+0000 7f4f94090700 1 -- 172.21.15.76:0/3607919998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4f8c108260 msgr2=0x7f4f8c10a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.501+0000 7f4f94090700 1 --2- 172.21.15.76:0/3607919998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4f8c108260 0x7f4f8c10a650 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4f880098a0 tx=0x7f4f88009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.501+0000 7f4f94090700 1 -- 172.21.15.76:0/3607919998 shutdown_connections 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.501+0000 7f4f94090700 1 --2- 172.21.15.76:0/3607919998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4f8c108260 0x7f4f8c10a650 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.501+0000 7f4f94090700 1 -- 172.21.15.76:0/3607919998 >> 172.21.15.76:0/3607919998 conn(0x7f4f8c101c90 msgr2=0x7f4f8c1040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.501+0000 7f4f94090700 1 -- 172.21.15.76:0/3607919998 shutdown_connections 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.501+0000 7f4f94090700 1 -- 172.21.15.76:0/3607919998 wait complete. 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.501+0000 7f4f94090700 1 Processor -- start 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.501+0000 7f4f94090700 1 -- start start 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.502+0000 7f4f94090700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4f8c108260 0x7f4f8c19b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.502+0000 7f4f94090700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c19ba70 con 0x7f4f8c108260 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.502+0000 7f4f91e2c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4f8c108260 0x7f4f8c19b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.502+0000 7f4f91e2c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4f8c108260 0x7f4f8c19b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33856/0 (socket says 172.21.15.76:33856) 2024-03-12T05:46:06.561 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.502+0000 7f4f91e2c700 1 -- 172.21.15.76:0/544293443 learned_addr learned my addr 172.21.15.76:0/544293443 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:06.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.502+0000 7f4f91e2c700 1 -- 172.21.15.76:0/544293443 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f88009550 con 0x7f4f8c108260 2024-03-12T05:46:06.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.502+0000 7f4f91e2c700 1 --2- 172.21.15.76:0/544293443 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4f8c108260 0x7f4f8c19b530 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009fb0 tx=0x7f4f88011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:06.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.502+0000 7f4f82ffd700 1 -- 172.21.15.76:0/544293443 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f880038b0 con 0x7f4f8c108260 2024-03-12T05:46:06.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.502+0000 7f4f82ffd700 1 -- 172.21.15.76:0/544293443 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f88003a10 con 0x7f4f8c108260 2024-03-12T05:46:06.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.502+0000 7f4f94090700 1 -- 172.21.15.76:0/544293443 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f8c19bc70 con 0x7f4f8c108260 2024-03-12T05:46:06.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.502+0000 7f4f82ffd700 1 -- 172.21.15.76:0/544293443 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f880195e0 con 0x7f4f8c108260 2024-03-12T05:46:06.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.502+0000 7f4f94090700 1 -- 172.21.15.76:0/544293443 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f8c19c110 con 0x7f4f8c108260 2024-03-12T05:46:06.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.503+0000 7f4f82ffd700 1 -- 172.21.15.76:0/544293443 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7f4f8801a430 con 0x7f4f8c108260 2024-03-12T05:46:06.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.503+0000 7f4f82ffd700 1 --2- 172.21.15.76:0/544293443 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f4f78038c60 0x7f4f7803b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:06.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.503+0000 7f4f94090700 1 -- 172.21.15.76:0/544293443 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f8c10b920 con 0x7f4f8c108260 2024-03-12T05:46:06.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.503+0000 7f4f82ffd700 1 -- 172.21.15.76:0/544293443 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4f8804cd70 con 0x7f4f8c108260 2024-03-12T05:46:06.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.503+0000 7f4f9162b700 1 --2- 172.21.15.76:0/544293443 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f4f78038c60 0x7f4f7803b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:06.562 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.504+0000 7f4f9162b700 1 --2- 172.21.15.76:0/544293443 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f4f78038c60 0x7f4f7803b120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c006fd0 tx=0x7f4f7c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.508+0000 7f4f82ffd700 1 -- 172.21.15.76:0/544293443 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4f88003b80 con 0x7f4f8c108260 2024-03-12T05:46:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.757+0000 7f4f94090700 1 -- 172.21.15.76:0/544293443 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f4f8c10bb30 con 0x7f4f8c108260 2024-03-12T05:46:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.975+0000 7f4f82ffd700 1 -- 172.21.15.76:0/544293443 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f4f8804b070 con 0x7f4f8c108260 2024-03-12T05:46:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.977+0000 7f4f82ffd700 1 -- 172.21.15.76:0/544293443 <== mon.0 v2:172.21.15.76:3300/0 8 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f4f880198c0 con 0x7f4f8c108260 2024-03-12T05:46:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.978+0000 7f4f94090700 1 -- 172.21.15.76:0/544293443 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f4f78038c60 msgr2=0x7f4f7803b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.978+0000 7f4f94090700 1 --2- 172.21.15.76:0/544293443 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f4f78038c60 0x7f4f7803b120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c006fd0 tx=0x7f4f7c006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.978+0000 7f4f94090700 1 -- 172.21.15.76:0/544293443 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4f8c108260 msgr2=0x7f4f8c19b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.978+0000 7f4f94090700 1 --2- 172.21.15.76:0/544293443 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4f8c108260 0x7f4f8c19b530 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009fb0 tx=0x7f4f88011b20 comp rx=0 tx=0).stop 2024-03-12T05:46:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.978+0000 7f4f94090700 1 -- 172.21.15.76:0/544293443 shutdown_connections 2024-03-12T05:46:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.978+0000 7f4f94090700 1 --2- 172.21.15.76:0/544293443 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f4f78038c60 0x7f4f7803b120 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.978+0000 7f4f94090700 1 --2- 172.21.15.76:0/544293443 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4f8c108260 0x7f4f8c19b530 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.978+0000 7f4f94090700 1 -- 172.21.15.76:0/544293443 >> 172.21.15.76:0/544293443 conn(0x7f4f8c101c90 msgr2=0x7f4f8c1027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.978+0000 7f4f94090700 1 -- 172.21.15.76:0/544293443 shutdown_connections 2024-03-12T05:46:06.563 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:05.978+0000 7f4f94090700 1 -- 172.21.15.76:0/544293443 wait complete. 2024-03-12T05:46:06.789 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:06 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:06.531+0000 7f72f4910200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:46:06.789 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:06 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:06.629+0000 7f72f4910200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:46:07.201 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:06 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/544293443' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-12T05:46:07.201 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:06 smithi076 ceph-mon[32095]: mgrmap e5: x(active, since 6s) 2024-03-12T05:46:07.485 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:07 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:07.300+0000 7f72f4910200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:46:07.485 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:07 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:07.393+0000 7f72f4910200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:46:07.849 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:07 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:07.538+0000 7f72f4910200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:46:07.850 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:07 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:07.635+0000 7f72f4910200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:46:07.850 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:07 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:07.718+0000 7f72f4910200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:46:07.858 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:46:07.858 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-12T05:46:07.858 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T05:46:07.858 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-12T05:46:07.858 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-12T05:46:07.858 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:46:07.858 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.994+0000 7f09725e0700 1 Processor -- start 2024-03-12T05:46:07.858 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.994+0000 7f09725e0700 1 -- start start 2024-03-12T05:46:07.859 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.995+0000 7f09725e0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f096c105f10 0x7f096c106330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:07.859 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.995+0000 7f09725e0700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c075600 con 0x7f096c105f10 2024-03-12T05:46:07.859 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.995+0000 7f096bfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f096c105f10 0x7f096c106330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:07.859 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.995+0000 7f096bfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f096c105f10 0x7f096c106330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33894/0 (socket says 172.21.15.76:33894) 2024-03-12T05:46:07.859 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.995+0000 7f096bfff700 1 -- 172.21.15.76:0/3122212680 learned_addr learned my addr 172.21.15.76:0/3122212680 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:07.859 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.995+0000 7f096bfff700 1 -- 172.21.15.76:0/3122212680 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096c106870 con 0x7f096c105f10 2024-03-12T05:46:07.859 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.995+0000 7f096bfff700 1 --2- 172.21.15.76:0/3122212680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f096c105f10 0x7f096c106330 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f095c0098a0 tx=0x7f095c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d88f9c8218ed93 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:07.859 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.995+0000 7f096affd700 1 -- 172.21.15.76:0/3122212680 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f095c004160 con 0x7f096c105f10 2024-03-12T05:46:07.859 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.995+0000 7f096affd700 1 -- 172.21.15.76:0/3122212680 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f095c0042c0 con 0x7f096c105f10 2024-03-12T05:46:07.859 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.995+0000 7f096affd700 1 -- 172.21.15.76:0/3122212680 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f095c011ac0 con 0x7f096c105f10 2024-03-12T05:46:07.859 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.996+0000 7f09725e0700 1 -- 172.21.15.76:0/3122212680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f096c105f10 msgr2=0x7f096c106330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:07.859 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.996+0000 7f09725e0700 1 --2- 172.21.15.76:0/3122212680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f096c105f10 0x7f096c106330 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f095c0098a0 tx=0x7f095c009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:07.859 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.996+0000 7f09725e0700 1 -- 172.21.15.76:0/3122212680 shutdown_connections 2024-03-12T05:46:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.996+0000 7f09725e0700 1 --2- 172.21.15.76:0/3122212680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f096c105f10 0x7f096c106330 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.996+0000 7f09725e0700 1 -- 172.21.15.76:0/3122212680 >> 172.21.15.76:0/3122212680 conn(0x7f096c101b90 msgr2=0x7f096c103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.996+0000 7f09725e0700 1 -- 172.21.15.76:0/3122212680 shutdown_connections 2024-03-12T05:46:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.996+0000 7f09725e0700 1 -- 172.21.15.76:0/3122212680 wait complete. 2024-03-12T05:46:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.997+0000 7f09725e0700 1 Processor -- start 2024-03-12T05:46:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.997+0000 7f09725e0700 1 -- start start 2024-03-12T05:46:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.997+0000 7f09725e0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f096c105f10 0x7f096c19d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.997+0000 7f09725e0700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c19db80 con 0x7f096c105f10 2024-03-12T05:46:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.998+0000 7f096bfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f096c105f10 0x7f096c19d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.998+0000 7f096bfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f096c105f10 0x7f096c19d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33898/0 (socket says 172.21.15.76:33898) 2024-03-12T05:46:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.998+0000 7f096bfff700 1 -- 172.21.15.76:0/3648012817 learned_addr learned my addr 172.21.15.76:0/3648012817 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.998+0000 7f096bfff700 1 -- 172.21.15.76:0/3648012817 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c009550 con 0x7f096c105f10 2024-03-12T05:46:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.998+0000 7f096bfff700 1 --2- 172.21.15.76:0/3648012817 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f096c105f10 0x7f096c19d640 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f095c009130 tx=0x7f095c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:07.860 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.998+0000 7f09697fa700 1 -- 172.21.15.76:0/3648012817 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f095c003aa0 con 0x7f096c105f10 2024-03-12T05:46:07.861 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.998+0000 7f09697fa700 1 -- 172.21.15.76:0/3648012817 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f095c003c00 con 0x7f096c105f10 2024-03-12T05:46:07.861 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.998+0000 7f09725e0700 1 -- 172.21.15.76:0/3648012817 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096c19dd80 con 0x7f096c105f10 2024-03-12T05:46:07.861 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.998+0000 7f09725e0700 1 -- 172.21.15.76:0/3648012817 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096c19e220 con 0x7f096c105f10 2024-03-12T05:46:07.861 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.998+0000 7f09697fa700 1 -- 172.21.15.76:0/3648012817 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f095c024800 con 0x7f096c105f10 2024-03-12T05:46:07.861 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.999+0000 7f09697fa700 1 -- 172.21.15.76:0/3648012817 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f095c024a90 con 0x7f096c105f10 2024-03-12T05:46:07.861 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.999+0000 7f09697fa700 1 --2- 172.21.15.76:0/3648012817 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f0954038d00 0x7f095403b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:07.861 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.999+0000 7f09697fa700 1 -- 172.21.15.76:0/3648012817 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f095c04ca10 con 0x7f096c105f10 2024-03-12T05:46:07.861 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.999+0000 7f096b7fe700 1 -- 172.21.15.76:0/3648012817 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f0954038d00 msgr2=0x7f095403b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/4265639592 2024-03-12T05:46:07.861 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.999+0000 7f096b7fe700 1 --2- 172.21.15.76:0/3648012817 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f0954038d00 0x7f095403b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:46:07.861 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:06.999+0000 7f09725e0700 1 -- 172.21.15.76:0/3648012817 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096c1972e0 con 0x7f096c105f10 2024-03-12T05:46:07.861 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.004+0000 7f09697fa700 1 -- 172.21.15.76:0/3648012817 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f095c024d40 con 0x7f096c105f10 2024-03-12T05:46:07.861 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.200+0000 7f096b7fe700 1 -- 172.21.15.76:0/3648012817 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f0954038d00 msgr2=0x7f095403b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/4265639592 2024-03-12T05:46:07.861 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.200+0000 7f096b7fe700 1 --2- 172.21.15.76:0/3648012817 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f0954038d00 0x7f095403b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T05:46:07.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.269+0000 7f09725e0700 1 -- 172.21.15.76:0/3648012817 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f096c02d800 con 0x7f096c105f10 2024-03-12T05:46:07.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.270+0000 7f09697fa700 1 -- 172.21.15.76:0/3648012817 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f095c02c350 con 0x7f096c105f10 2024-03-12T05:46:07.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.272+0000 7f09725e0700 1 -- 172.21.15.76:0/3648012817 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f0954038d00 msgr2=0x7f095403b1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:46:07.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.272+0000 7f09725e0700 1 --2- 172.21.15.76:0/3648012817 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f0954038d00 0x7f095403b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:07.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.272+0000 7f09725e0700 1 -- 172.21.15.76:0/3648012817 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f096c105f10 msgr2=0x7f096c19d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:07.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.272+0000 7f09725e0700 1 --2- 172.21.15.76:0/3648012817 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f096c105f10 0x7f096c19d640 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f095c009130 tx=0x7f095c011b40 comp rx=0 tx=0).stop 2024-03-12T05:46:07.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.272+0000 7f09725e0700 1 -- 172.21.15.76:0/3648012817 shutdown_connections 2024-03-12T05:46:07.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.272+0000 7f09725e0700 1 --2- 172.21.15.76:0/3648012817 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7f0954038d00 0x7f095403b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:07.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.272+0000 7f09725e0700 1 --2- 172.21.15.76:0/3648012817 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f096c105f10 0x7f096c19d640 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:07.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.272+0000 7f09725e0700 1 -- 172.21.15.76:0/3648012817 >> 172.21.15.76:0/3648012817 conn(0x7f096c101b90 msgr2=0x7f096c194420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:07.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.272+0000 7f09725e0700 1 -- 172.21.15.76:0/3648012817 shutdown_connections 2024-03-12T05:46:07.862 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:07.272+0000 7f09725e0700 1 -- 172.21.15.76:0/3648012817 wait complete. 2024-03-12T05:46:07.862 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for the mgr to restart... 2024-03-12T05:46:07.863 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr epoch 5... 2024-03-12T05:46:08.113 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:07 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3648012817' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T05:46:08.114 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:08 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:07.998+0000 7f72f4910200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:46:08.114 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:08 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:08.111+0000 7f72f4910200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:46:08.460 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:08 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:08.199+0000 7f72f4910200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:46:08.460 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:08 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:08.297+0000 7f72f4910200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:46:08.735 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:08 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:08.471+0000 7f72f4910200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:46:09.735 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:09 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:09.429+0000 7f72f4910200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:46:09.735 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:09 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:09.559+0000 7f72f4910200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:46:10.179 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:09 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:09.889+0000 7f72f4910200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:46:10.180 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:09 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:09.980+0000 7f72f4910200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:46:10.180 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:10 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:10.177+0000 7f72f4910200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:46:10.456 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:10 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:10.367+0000 7f72f4910200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:46:10.735 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:10 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:10.665+0000 7f72f4910200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:46:11.235 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:10 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:10.931+0000 7f72f4910200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:46:11.235 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:11 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:11.173+0000 7f72f4910200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:46:12.235 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:11 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:11.977+0000 7f72f4910200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:46:12.885 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:12.568+0000 7f72f4910200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:46:12.886 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:12.666+0000 7f72f4910200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:46:12.886 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:12.884+0000 7f72f4910200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:46:13.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-mon[32095]: Active manager daemon x restarted 2024-03-12T05:46:13.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-mon[32095]: Activating manager daemon x 2024-03-12T05:46:13.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-mon[32095]: osdmap e2: 0 total, 0 up, 0 in 2024-03-12T05:46:13.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-mon[32095]: mgrmap e6: x(active, starting, since 0.0034717s) 2024-03-12T05:46:13.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:46:13.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T05:46:13.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T05:46:13.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T05:46:13.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T05:46:13.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-mon[32095]: Manager daemon x is now available 2024-03-12T05:46:13.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' 2024-03-12T05:46:13.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' 2024-03-12T05:46:13.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:46:13.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:12 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:46:14.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:13 smithi076 ceph-mon[32095]: Found migration_current of "None". Setting to last migration. 2024-03-12T05:46:14.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:13 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:46:14.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:13 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T05:46:14.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:13 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T05:46:14.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:13 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' 2024-03-12T05:46:14.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:13 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' 2024-03-12T05:46:14.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:13 smithi076 ceph-mon[32095]: mgrmap e7: x(active, since 1.00727s) 2024-03-12T05:46:14.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:46:14.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-12T05:46:14.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-12T05:46:14.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:46:14.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.253+0000 7fcef9611700 1 Processor -- start 2024-03-12T05:46:14.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.253+0000 7fcef9611700 1 -- start start 2024-03-12T05:46:14.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.253+0000 7fcef9611700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcef41076f0 0x7fcef4107b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:14.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.253+0000 7fcef9611700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef41080e0 con 0x7fcef41076f0 2024-03-12T05:46:14.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.253+0000 7fcef2ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcef41076f0 0x7fcef4107b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:14.683 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.253+0000 7fcef2ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcef41076f0 0x7fcef4107b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33910/0 (socket says 172.21.15.76:33910) 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.253+0000 7fcef2ffd700 1 -- 172.21.15.76:0/3765530052 learned_addr learned my addr 172.21.15.76:0/3765530052 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.253+0000 7fcef2ffd700 1 -- 172.21.15.76:0/3765530052 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef4108850 con 0x7fcef41076f0 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.253+0000 7fcef2ffd700 1 --2- 172.21.15.76:0/3765530052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcef41076f0 0x7fcef4107b10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcee40098a0 tx=0x7fcee4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd0a2ec55ae6fbd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.254+0000 7fcef1ffb700 1 -- 172.21.15.76:0/3765530052 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcee4004160 con 0x7fcef41076f0 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.254+0000 7fcef1ffb700 1 -- 172.21.15.76:0/3765530052 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcee40042c0 con 0x7fcef41076f0 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.254+0000 7fcef1ffb700 1 -- 172.21.15.76:0/3765530052 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcee4011ac0 con 0x7fcef41076f0 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.254+0000 7fcef9611700 1 -- 172.21.15.76:0/3765530052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcef41076f0 msgr2=0x7fcef4107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.254+0000 7fcef9611700 1 --2- 172.21.15.76:0/3765530052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcef41076f0 0x7fcef4107b10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcee40098a0 tx=0x7fcee4009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.254+0000 7fcef9611700 1 -- 172.21.15.76:0/3765530052 shutdown_connections 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.254+0000 7fcef9611700 1 --2- 172.21.15.76:0/3765530052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcef41076f0 0x7fcef4107b10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.254+0000 7fcef9611700 1 -- 172.21.15.76:0/3765530052 >> 172.21.15.76:0/3765530052 conn(0x7fcef4102d50 msgr2=0x7fcef4105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.254+0000 7fcef9611700 1 -- 172.21.15.76:0/3765530052 shutdown_connections 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.254+0000 7fcef9611700 1 -- 172.21.15.76:0/3765530052 wait complete. 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.255+0000 7fcef9611700 1 Processor -- start 2024-03-12T05:46:14.684 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.255+0000 7fcef9611700 1 -- start start 2024-03-12T05:46:14.685 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.255+0000 7fcef9611700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcef41076f0 0x7fcef407cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:14.685 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.255+0000 7fcef9611700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef407d0b0 con 0x7fcef41076f0 2024-03-12T05:46:14.685 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.255+0000 7fcef2ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcef41076f0 0x7fcef407cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:14.685 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.255+0000 7fcef2ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcef41076f0 0x7fcef407cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33912/0 (socket says 172.21.15.76:33912) 2024-03-12T05:46:14.685 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.255+0000 7fcef2ffd700 1 -- 172.21.15.76:0/3956653572 learned_addr learned my addr 172.21.15.76:0/3956653572 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:14.685 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.256+0000 7fcef2ffd700 1 -- 172.21.15.76:0/3956653572 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee4009550 con 0x7fcef41076f0 2024-03-12T05:46:14.685 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.256+0000 7fcef2ffd700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcef41076f0 0x7fcef407cb70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcee4011780 tx=0x7fcee4011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:14.685 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.256+0000 7fcedbfff700 1 -- 172.21.15.76:0/3956653572 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcee4003af0 con 0x7fcef41076f0 2024-03-12T05:46:14.685 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.256+0000 7fcedbfff700 1 -- 172.21.15.76:0/3956653572 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcee4003c50 con 0x7fcef41076f0 2024-03-12T05:46:14.685 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.256+0000 7fcedbfff700 1 -- 172.21.15.76:0/3956653572 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcee40197e0 con 0x7fcef41076f0 2024-03-12T05:46:14.685 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.256+0000 7fcef9611700 1 -- 172.21.15.76:0/3956653572 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef407b2a0 con 0x7fcef41076f0 2024-03-12T05:46:14.685 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.256+0000 7fcef9611700 1 -- 172.21.15.76:0/3956653572 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef407b740 con 0x7fcef41076f0 2024-03-12T05:46:14.685 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.257+0000 7fcedbfff700 1 -- 172.21.15.76:0/3956653572 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fcee401aba0 con 0x7fcef41076f0 2024-03-12T05:46:14.685 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.257+0000 7fcedbfff700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fcedc038cb0 0x7fcedc03b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:14.686 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.257+0000 7fcedbfff700 1 -- 172.21.15.76:0/3956653572 --> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcedc03b880 con 0x7fcedc038cb0 2024-03-12T05:46:14.686 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.257+0000 7fcedbfff700 1 -- 172.21.15.76:0/3956653572 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcee404cff0 con 0x7fcef41076f0 2024-03-12T05:46:14.686 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.257+0000 7fcef27fc700 1 -- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fcedc038cb0 msgr2=0x7fcedc03b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/4265639592 2024-03-12T05:46:14.686 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.257+0000 7fcef27fc700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fcedc038cb0 0x7fcedc03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:46:14.686 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.458+0000 7fcef27fc700 1 -- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fcedc038cb0 msgr2=0x7fcedc03b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/4265639592 2024-03-12T05:46:14.686 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.458+0000 7fcef27fc700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fcedc038cb0 0x7fcedc03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T05:46:14.686 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.859+0000 7fcef27fc700 1 -- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fcedc038cb0 msgr2=0x7fcedc03b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/4265639592 2024-03-12T05:46:14.686 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:08.859+0000 7fcef27fc700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fcedc038cb0 0x7fcedc03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-12T05:46:14.686 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:09.660+0000 7fcef27fc700 1 -- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fcedc038cb0 msgr2=0x7fcedc03b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/4265639592 2024-03-12T05:46:14.686 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:09.660+0000 7fcef27fc700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fcedc038cb0 0x7fcedc03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-12T05:46:14.686 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:11.262+0000 7fcef27fc700 1 -- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fcedc038cb0 msgr2=0x7fcedc03b170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/4265639592 2024-03-12T05:46:14.686 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:11.262+0000 7fcef27fc700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fcedc038cb0 0x7fcedc03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-12T05:46:14.686 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:12.900+0000 7fcedbfff700 1 -- 172.21.15.76:0/3956653572 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mgrmap(e 6) v1 ==== 45536+0+0 (secure 0 0 0) 0x7fcee4019a20 con 0x7fcef41076f0 2024-03-12T05:46:14.686 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:12.900+0000 7fcedbfff700 1 -- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fcedc038cb0 msgr2=0x7fcedc03b170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:46:14.687 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:12.900+0000 7fcedbfff700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fcedc038cb0 0x7fcedc03b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:14.687 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.903+0000 7fcedbfff700 1 -- 172.21.15.76:0/3956653572 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 7) v1 ==== 45663+0+0 (secure 0 0 0) 0x7fcee404e9a0 con 0x7fcef41076f0 2024-03-12T05:46:14.687 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.903+0000 7fcedbfff700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7fcedc03c5a0 0x7fcedc03e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:14.687 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.903+0000 7fcedbfff700 1 -- 172.21.15.76:0/3956653572 --> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcedc03b880 con 0x7fcedc03c5a0 2024-03-12T05:46:14.687 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.904+0000 7fcef27fc700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7fcedc03c5a0 0x7fcedc03e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:14.687 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.904+0000 7fcef27fc700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7fcedc03c5a0 0x7fcedc03e990 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcee8003a10 tx=0x7fcee8009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:14.687 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.905+0000 7fcedbfff700 1 -- 172.21.15.76:0/3956653572 <== mgr.14120 v2:172.21.15.76:6800/306877437 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fcedc03b880 con 0x7fcedc03c5a0 2024-03-12T05:46:14.687 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.911+0000 7fcef9611700 1 -- 172.21.15.76:0/3956653572 --> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fcef4068d50 con 0x7fcedc03c5a0 2024-03-12T05:46:14.687 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.912+0000 7fcedbfff700 1 -- 172.21.15.76:0/3956653572 <== mgr.14120 v2:172.21.15.76:6800/306877437 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fcef4068d50 con 0x7fcedc03c5a0 2024-03-12T05:46:14.687 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.912+0000 7fcef9611700 1 -- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7fcedc03c5a0 msgr2=0x7fcedc03e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:14.687 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.912+0000 7fcef9611700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7fcedc03c5a0 0x7fcedc03e990 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcee8003a10 tx=0x7fcee8009290 comp rx=0 tx=0).stop 2024-03-12T05:46:14.687 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.912+0000 7fcef9611700 1 -- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcef41076f0 msgr2=0x7fcef407cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:14.687 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.912+0000 7fcef9611700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcef41076f0 0x7fcef407cb70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcee4011780 tx=0x7fcee4011b40 comp rx=0 tx=0).stop 2024-03-12T05:46:14.687 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.912+0000 7fcef9611700 1 -- 172.21.15.76:0/3956653572 shutdown_connections 2024-03-12T05:46:14.688 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.912+0000 7fcef9611700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7fcedc03c5a0 0x7fcedc03e990 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:14.688 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.912+0000 7fcef9611700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcef41076f0 0x7fcef407cb70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:14.688 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.912+0000 7fcef9611700 1 --2- 172.21.15.76:0/3956653572 >> [v2:172.21.15.76:6800/4265639592,v1:172.21.15.76:6801/4265639592] conn(0x7fcedc038cb0 0x7fcedc03b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:14.688 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.912+0000 7fcef9611700 1 -- 172.21.15.76:0/3956653572 >> 172.21.15.76:0/3956653572 conn(0x7fcef4102d50 msgr2=0x7fcef41038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:14.688 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.912+0000 7fcef9611700 1 -- 172.21.15.76:0/3956653572 shutdown_connections 2024-03-12T05:46:14.688 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:13.912+0000 7fcef9611700 1 -- 172.21.15.76:0/3956653572 wait complete. 2024-03-12T05:46:14.688 INFO:teuthology.orchestra.run.smithi076.stdout:mgr epoch 5 is available 2024-03-12T05:46:14.688 INFO:teuthology.orchestra.run.smithi076.stdout:Setting orchestrator backend to cephadm... 2024-03-12T05:46:15.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:14 smithi076 ceph-mon[32095]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T05:46:15.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:14 smithi076 ceph-mon[32095]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T05:46:15.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:14 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:46:15.919 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.102+0000 7f3feedf1700 1 Processor -- start 2024-03-12T05:46:15.920 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.102+0000 7f3feedf1700 1 -- start start 2024-03-12T05:46:15.920 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.102+0000 7f3feedf1700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fe81081b0 0x7f3fe810a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:15.920 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.102+0000 7f3feedf1700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe807ae20 con 0x7f3fe81081b0 2024-03-12T05:46:15.920 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.102+0000 7f3fecb8d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fe81081b0 0x7f3fe810a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:15.920 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.102+0000 7f3fecb8d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fe81081b0 0x7f3fe810a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47160/0 (socket says 172.21.15.76:47160) 2024-03-12T05:46:15.920 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.102+0000 7f3fecb8d700 1 -- 172.21.15.76:0/3124537196 learned_addr learned my addr 172.21.15.76:0/3124537196 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:15.920 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.103+0000 7f3fecb8d700 1 -- 172.21.15.76:0/3124537196 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe810aae0 con 0x7f3fe81081b0 2024-03-12T05:46:15.920 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.103+0000 7f3fecb8d700 1 --2- 172.21.15.76:0/3124537196 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fe81081b0 0x7f3fe810a5a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc0098a0 tx=0x7f3fdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e74105bd1ec818b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:15.920 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.103+0000 7f3fe77fe700 1 -- 172.21.15.76:0/3124537196 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fdc004160 con 0x7f3fe81081b0 2024-03-12T05:46:15.920 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.103+0000 7f3fe77fe700 1 -- 172.21.15.76:0/3124537196 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3fdc0042c0 con 0x7f3fe81081b0 2024-03-12T05:46:15.920 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.103+0000 7f3fe77fe700 1 -- 172.21.15.76:0/3124537196 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fdc011ac0 con 0x7f3fe81081b0 2024-03-12T05:46:15.920 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.103+0000 7f3feedf1700 1 -- 172.21.15.76:0/3124537196 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fe81081b0 msgr2=0x7f3fe810a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:15.920 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.103+0000 7f3feedf1700 1 --2- 172.21.15.76:0/3124537196 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fe81081b0 0x7f3fe810a5a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc0098a0 tx=0x7f3fdc009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:15.921 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.103+0000 7f3feedf1700 1 -- 172.21.15.76:0/3124537196 shutdown_connections 2024-03-12T05:46:15.921 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.103+0000 7f3feedf1700 1 --2- 172.21.15.76:0/3124537196 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fe81081b0 0x7f3fe810a5a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:15.921 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.103+0000 7f3feedf1700 1 -- 172.21.15.76:0/3124537196 >> 172.21.15.76:0/3124537196 conn(0x7f3fe8101c00 msgr2=0x7f3fe8104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:15.921 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.104+0000 7f3feedf1700 1 -- 172.21.15.76:0/3124537196 shutdown_connections 2024-03-12T05:46:15.921 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.104+0000 7f3feedf1700 1 -- 172.21.15.76:0/3124537196 wait complete. 2024-03-12T05:46:15.921 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.104+0000 7f3feedf1700 1 Processor -- start 2024-03-12T05:46:15.921 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.104+0000 7f3feedf1700 1 -- start start 2024-03-12T05:46:15.921 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.104+0000 7f3feedf1700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fe81081b0 0x7f3fe81992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:15.921 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.104+0000 7f3feedf1700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe81997f0 con 0x7f3fe81081b0 2024-03-12T05:46:15.921 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.104+0000 7f3fecb8d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fe81081b0 0x7f3fe81992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:15.921 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.104+0000 7f3fecb8d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fe81081b0 0x7f3fe81992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47164/0 (socket says 172.21.15.76:47164) 2024-03-12T05:46:15.921 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.104+0000 7f3fecb8d700 1 -- 172.21.15.76:0/1297639612 learned_addr learned my addr 172.21.15.76:0/1297639612 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:15.921 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.105+0000 7f3fecb8d700 1 -- 172.21.15.76:0/1297639612 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc009550 con 0x7f3fe81081b0 2024-03-12T05:46:15.922 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.105+0000 7f3fecb8d700 1 --2- 172.21.15.76:0/1297639612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fe81081b0 0x7f3fe81992b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc003f70 tx=0x7f3fdc011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:15.922 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.105+0000 7f3fe5ffb700 1 -- 172.21.15.76:0/1297639612 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fdc0039f0 con 0x7f3fe81081b0 2024-03-12T05:46:15.922 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.105+0000 7f3fe5ffb700 1 -- 172.21.15.76:0/1297639612 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3fdc003b50 con 0x7f3fe81081b0 2024-03-12T05:46:15.922 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.105+0000 7f3feedf1700 1 -- 172.21.15.76:0/1297639612 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe81999f0 con 0x7f3fe81081b0 2024-03-12T05:46:15.922 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.105+0000 7f3fe5ffb700 1 -- 172.21.15.76:0/1297639612 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fdc01a760 con 0x7f3fe81081b0 2024-03-12T05:46:15.922 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.105+0000 7f3feedf1700 1 -- 172.21.15.76:0/1297639612 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe8199e90 con 0x7f3fe81081b0 2024-03-12T05:46:15.922 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.106+0000 7f3fe5ffb700 1 -- 172.21.15.76:0/1297639612 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f3fdc01a9d0 con 0x7f3fe81081b0 2024-03-12T05:46:15.922 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.106+0000 7f3feedf1700 1 -- 172.21.15.76:0/1297639612 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe8193740 con 0x7f3fe81081b0 2024-03-12T05:46:15.922 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.106+0000 7f3fe5ffb700 1 --2- 172.21.15.76:0/1297639612 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f3fd803d070 0x7f3fd803f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:15.922 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.106+0000 7f3fe5ffb700 1 -- 172.21.15.76:0/1297639612 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3fdc04d030 con 0x7f3fe81081b0 2024-03-12T05:46:15.922 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.106+0000 7f3fe7fff700 1 --2- 172.21.15.76:0/1297639612 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f3fd803d070 0x7f3fd803f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:15.922 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.107+0000 7f3fe7fff700 1 --2- 172.21.15.76:0/1297639612 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f3fd803d070 0x7f3fd803f530 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4006fd0 tx=0x7f3fd4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:15.922 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.111+0000 7f3fe5ffb700 1 -- 172.21.15.76:0/1297639612 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3fdc080d20 con 0x7f3fe81081b0 2024-03-12T05:46:15.923 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.323+0000 7f3feedf1700 1 -- 172.21.15.76:0/1297639612 --> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fe802da30 con 0x7f3fd803d070 2024-03-12T05:46:15.923 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.329+0000 7f3fe5ffb700 1 -- 172.21.15.76:0/1297639612 <== mgr.14120 v2:172.21.15.76:6800/306877437 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3fe802da30 con 0x7f3fd803d070 2024-03-12T05:46:15.923 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.331+0000 7f3feedf1700 1 -- 172.21.15.76:0/1297639612 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f3fd803d070 msgr2=0x7f3fd803f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:15.923 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.331+0000 7f3feedf1700 1 --2- 172.21.15.76:0/1297639612 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f3fd803d070 0x7f3fd803f530 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4006fd0 tx=0x7f3fd4006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:15.923 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.331+0000 7f3feedf1700 1 -- 172.21.15.76:0/1297639612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fe81081b0 msgr2=0x7f3fe81992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:15.923 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.331+0000 7f3feedf1700 1 --2- 172.21.15.76:0/1297639612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fe81081b0 0x7f3fe81992b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc003f70 tx=0x7f3fdc011b40 comp rx=0 tx=0).stop 2024-03-12T05:46:15.923 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.331+0000 7f3feedf1700 1 -- 172.21.15.76:0/1297639612 shutdown_connections 2024-03-12T05:46:15.923 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.331+0000 7f3feedf1700 1 --2- 172.21.15.76:0/1297639612 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f3fd803d070 0x7f3fd803f530 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:15.923 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.331+0000 7f3feedf1700 1 --2- 172.21.15.76:0/1297639612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fe81081b0 0x7f3fe81992b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:15.923 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.331+0000 7f3feedf1700 1 -- 172.21.15.76:0/1297639612 >> 172.21.15.76:0/1297639612 conn(0x7f3fe8101c00 msgr2=0x7f3fe8102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:15.923 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.331+0000 7f3feedf1700 1 -- 172.21.15.76:0/1297639612 shutdown_connections 2024-03-12T05:46:15.923 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:15.332+0000 7f3feedf1700 1 -- 172.21.15.76:0/1297639612 wait complete. 2024-03-12T05:46:16.224 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:15 smithi076 ceph-mon[32095]: [12/Mar/2024:05:46:14] ENGINE Bus STARTING 2024-03-12T05:46:16.224 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:15 smithi076 ceph-mon[32095]: [12/Mar/2024:05:46:14] ENGINE Serving on https://172.21.15.76:7150 2024-03-12T05:46:16.224 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:15 smithi076 ceph-mon[32095]: [12/Mar/2024:05:46:14] ENGINE Serving on http://172.21.15.76:8765 2024-03-12T05:46:16.224 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:15 smithi076 ceph-mon[32095]: [12/Mar/2024:05:46:14] ENGINE Bus STARTED 2024-03-12T05:46:16.224 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:15 smithi076 ceph-mon[32095]: mgrmap e8: x(active, since 2s) 2024-03-12T05:46:16.224 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:15 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' 2024-03-12T05:46:16.224 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:15 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:46:17.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-12T05:46:17.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.326+0000 7f1eaa949700 1 Processor -- start 2024-03-12T05:46:17.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.326+0000 7f1eaa949700 1 -- start start 2024-03-12T05:46:17.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.326+0000 7f1eaa949700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ea4107c90 0x7f1ea41080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:17.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.326+0000 7f1eaa949700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea4108680 con 0x7f1ea4107c90 2024-03-12T05:46:17.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.326+0000 7f1ea3fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ea4107c90 0x7f1ea41080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:17.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.326+0000 7f1ea3fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ea4107c90 0x7f1ea41080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47166/0 (socket says 172.21.15.76:47166) 2024-03-12T05:46:17.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.326+0000 7f1ea3fff700 1 -- 172.21.15.76:0/2803499114 learned_addr learned my addr 172.21.15.76:0/2803499114 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:17.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.326+0000 7f1ea3fff700 1 -- 172.21.15.76:0/2803499114 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea41087c0 con 0x7f1ea4107c90 2024-03-12T05:46:17.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.327+0000 7f1ea3fff700 1 --2- 172.21.15.76:0/2803499114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ea4107c90 0x7f1ea41080b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e940098a0 tx=0x7f1e94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438df32a2a781408 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:17.127 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.327+0000 7f1ea2ffd700 1 -- 172.21.15.76:0/2803499114 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e94004160 con 0x7f1ea4107c90 2024-03-12T05:46:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.327+0000 7f1ea2ffd700 1 -- 172.21.15.76:0/2803499114 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e940042c0 con 0x7f1ea4107c90 2024-03-12T05:46:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.327+0000 7f1ea2ffd700 1 -- 172.21.15.76:0/2803499114 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e94011ac0 con 0x7f1ea4107c90 2024-03-12T05:46:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.327+0000 7f1eaa949700 1 -- 172.21.15.76:0/2803499114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ea4107c90 msgr2=0x7f1ea41080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.327+0000 7f1eaa949700 1 --2- 172.21.15.76:0/2803499114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ea4107c90 0x7f1ea41080b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e940098a0 tx=0x7f1e94009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.327+0000 7f1eaa949700 1 -- 172.21.15.76:0/2803499114 shutdown_connections 2024-03-12T05:46:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.327+0000 7f1eaa949700 1 --2- 172.21.15.76:0/2803499114 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ea4107c90 0x7f1ea41080b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.327+0000 7f1eaa949700 1 -- 172.21.15.76:0/2803499114 >> 172.21.15.76:0/2803499114 conn(0x7f1ea4101430 msgr2=0x7f1ea4103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.327+0000 7f1eaa949700 1 -- 172.21.15.76:0/2803499114 shutdown_connections 2024-03-12T05:46:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.327+0000 7f1eaa949700 1 -- 172.21.15.76:0/2803499114 wait complete. 2024-03-12T05:46:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.328+0000 7f1eaa949700 1 Processor -- start 2024-03-12T05:46:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.328+0000 7f1eaa949700 1 -- start start 2024-03-12T05:46:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.328+0000 7f1eaa949700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ea4107c90 0x7f1ea419d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.328+0000 7f1eaa949700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea419dbd0 con 0x7f1ea4107c90 2024-03-12T05:46:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.328+0000 7f1ea3fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ea4107c90 0x7f1ea419d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.328+0000 7f1ea3fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ea4107c90 0x7f1ea419d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47180/0 (socket says 172.21.15.76:47180) 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.328+0000 7f1ea3fff700 1 -- 172.21.15.76:0/3705636985 learned_addr learned my addr 172.21.15.76:0/3705636985 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.329+0000 7f1ea3fff700 1 -- 172.21.15.76:0/3705636985 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e94009550 con 0x7f1ea4107c90 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.329+0000 7f1ea3fff700 1 --2- 172.21.15.76:0/3705636985 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ea4107c90 0x7f1ea419d690 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009fb0 tx=0x7f1e94011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.329+0000 7f1ea17fa700 1 -- 172.21.15.76:0/3705636985 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e940038b0 con 0x7f1ea4107c90 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.329+0000 7f1ea17fa700 1 -- 172.21.15.76:0/3705636985 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e94003a10 con 0x7f1ea4107c90 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.329+0000 7f1eaa949700 1 -- 172.21.15.76:0/3705636985 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea419ddd0 con 0x7f1ea4107c90 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.329+0000 7f1ea17fa700 1 -- 172.21.15.76:0/3705636985 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e940195e0 con 0x7f1ea4107c90 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.329+0000 7f1eaa949700 1 -- 172.21.15.76:0/3705636985 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea419e270 con 0x7f1ea4107c90 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.330+0000 7f1ea17fa700 1 -- 172.21.15.76:0/3705636985 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f1e9401a430 con 0x7f1ea4107c90 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.330+0000 7f1eaa949700 1 -- 172.21.15.76:0/3705636985 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea41063f0 con 0x7f1ea4107c90 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.330+0000 7f1ea17fa700 1 --2- 172.21.15.76:0/3705636985 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f1e8c038c60 0x7f1e8c03b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.330+0000 7f1ea17fa700 1 -- 172.21.15.76:0/3705636985 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1e9404ce30 con 0x7f1ea4107c90 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.330+0000 7f1ea37fe700 1 --2- 172.21.15.76:0/3705636985 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f1e8c038c60 0x7f1e8c03b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:17.129 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.331+0000 7f1ea37fe700 1 --2- 172.21.15.76:0/3705636985 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f1e8c038c60 0x7f1e8c03b120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1e98006fd0 tx=0x7f1e98006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:17.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.335+0000 7f1ea17fa700 1 -- 172.21.15.76:0/3705636985 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1e94003b80 con 0x7f1ea4107c90 2024-03-12T05:46:17.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.529+0000 7f1eaa949700 1 -- 172.21.15.76:0/3705636985 --> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ea40620b0 con 0x7f1e8c038c60 2024-03-12T05:46:17.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.529+0000 7f1ea17fa700 1 -- 172.21.15.76:0/3705636985 <== mgr.14120 v2:172.21.15.76:6800/306877437 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f1ea40620b0 con 0x7f1e8c038c60 2024-03-12T05:46:17.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.531+0000 7f1eaa949700 1 -- 172.21.15.76:0/3705636985 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f1e8c038c60 msgr2=0x7f1e8c03b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:17.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.531+0000 7f1eaa949700 1 --2- 172.21.15.76:0/3705636985 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f1e8c038c60 0x7f1e8c03b120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1e98006fd0 tx=0x7f1e98006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:17.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.531+0000 7f1eaa949700 1 -- 172.21.15.76:0/3705636985 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ea4107c90 msgr2=0x7f1ea419d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:17.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.531+0000 7f1eaa949700 1 --2- 172.21.15.76:0/3705636985 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ea4107c90 0x7f1ea419d690 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009fb0 tx=0x7f1e94011b20 comp rx=0 tx=0).stop 2024-03-12T05:46:17.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.532+0000 7f1eaa949700 1 -- 172.21.15.76:0/3705636985 shutdown_connections 2024-03-12T05:46:17.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.532+0000 7f1eaa949700 1 --2- 172.21.15.76:0/3705636985 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f1e8c038c60 0x7f1e8c03b120 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:17.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.532+0000 7f1eaa949700 1 --2- 172.21.15.76:0/3705636985 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ea4107c90 0x7f1ea419d690 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:17.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.532+0000 7f1eaa949700 1 -- 172.21.15.76:0/3705636985 >> 172.21.15.76:0/3705636985 conn(0x7f1ea4101430 msgr2=0x7f1ea4193ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:17.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.532+0000 7f1eaa949700 1 -- 172.21.15.76:0/3705636985 shutdown_connections 2024-03-12T05:46:17.130 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:16.532+0000 7f1eaa949700 1 -- 172.21.15.76:0/3705636985 wait complete. 2024-03-12T05:46:17.130 INFO:teuthology.orchestra.run.smithi076.stdout:Generating ssh key... 2024-03-12T05:46:17.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:16 smithi076 ceph-mon[32095]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:46:18.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:17 smithi076 ceph-mon[32095]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:46:18.235 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: Generating public/private rsa key pair. 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: Your identification has been saved in /tmp/tmpkj466eda/key. 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: Your public key has been saved in /tmp/tmpkj466eda/key.pub. 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: The key fingerprint is: 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: SHA256:0cerjc/A95KU2NXrAykqsZbpS4M6mOXICL037uTHKB8 ceph-9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: The key's randomart image is: 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: +---[RSA 3072]----+ 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: | | 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: | . . | 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: | . . o . | 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: | . . .. .| 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: | . S o.o. .| 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: |. .. .. ..=+o . | 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: |+ *.E+ o= *.+.o | 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: |.=o==.+*.. =o. o | 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: | B*oooo o.. .| 2024-03-12T05:46:18.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: +----[SHA256]-----+ 2024-03-12T05:46:18.556 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.538+0000 7f3d430ee700 1 Processor -- start 2024-03-12T05:46:18.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.538+0000 7f3d430ee700 1 -- start start 2024-03-12T05:46:18.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.538+0000 7f3d430ee700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d3c107730 0x7f3d3c107b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:18.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.538+0000 7f3d430ee700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c108120 con 0x7f3d3c107730 2024-03-12T05:46:18.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.538+0000 7f3d40e8a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d3c107730 0x7f3d3c107b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:18.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.538+0000 7f3d40e8a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d3c107730 0x7f3d3c107b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47186/0 (socket says 172.21.15.76:47186) 2024-03-12T05:46:18.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.539+0000 7f3d40e8a700 1 -- 172.21.15.76:0/2549179429 learned_addr learned my addr 172.21.15.76:0/2549179429 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:18.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.539+0000 7f3d40e8a700 1 -- 172.21.15.76:0/2549179429 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c1088e0 con 0x7f3d3c107730 2024-03-12T05:46:18.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.539+0000 7f3d40e8a700 1 --2- 172.21.15.76:0/2549179429 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d3c107730 0x7f3d3c107b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3d380098a0 tx=0x7f3d38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84af2d4ffed55cc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:18.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.539+0000 7f3d337fe700 1 -- 172.21.15.76:0/2549179429 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d38004160 con 0x7f3d3c107730 2024-03-12T05:46:18.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.539+0000 7f3d337fe700 1 -- 172.21.15.76:0/2549179429 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d380042c0 con 0x7f3d3c107730 2024-03-12T05:46:18.557 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.539+0000 7f3d337fe700 1 -- 172.21.15.76:0/2549179429 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d38011ac0 con 0x7f3d3c107730 2024-03-12T05:46:18.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.539+0000 7f3d430ee700 1 -- 172.21.15.76:0/2549179429 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d3c107730 msgr2=0x7f3d3c107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:18.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.539+0000 7f3d430ee700 1 --2- 172.21.15.76:0/2549179429 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d3c107730 0x7f3d3c107b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3d380098a0 tx=0x7f3d38009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:18.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.540+0000 7f3d430ee700 1 -- 172.21.15.76:0/2549179429 shutdown_connections 2024-03-12T05:46:18.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.540+0000 7f3d430ee700 1 --2- 172.21.15.76:0/2549179429 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d3c107730 0x7f3d3c107b50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:18.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.540+0000 7f3d430ee700 1 -- 172.21.15.76:0/2549179429 >> 172.21.15.76:0/2549179429 conn(0x7f3d3c102db0 msgr2=0x7f3d3c1051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:18.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.540+0000 7f3d430ee700 1 -- 172.21.15.76:0/2549179429 shutdown_connections 2024-03-12T05:46:18.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.540+0000 7f3d430ee700 1 -- 172.21.15.76:0/2549179429 wait complete. 2024-03-12T05:46:18.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.540+0000 7f3d430ee700 1 Processor -- start 2024-03-12T05:46:18.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.540+0000 7f3d430ee700 1 -- start start 2024-03-12T05:46:18.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.540+0000 7f3d430ee700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d3c107730 0x7f3d3c19b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:18.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.540+0000 7f3d430ee700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c19b990 con 0x7f3d3c107730 2024-03-12T05:46:18.558 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.541+0000 7f3d40e8a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d3c107730 0x7f3d3c19b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:18.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.541+0000 7f3d40e8a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d3c107730 0x7f3d3c19b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47192/0 (socket says 172.21.15.76:47192) 2024-03-12T05:46:18.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.541+0000 7f3d40e8a700 1 -- 172.21.15.76:0/2585616102 learned_addr learned my addr 172.21.15.76:0/2585616102 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:18.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.541+0000 7f3d40e8a700 1 -- 172.21.15.76:0/2585616102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d38009550 con 0x7f3d3c107730 2024-03-12T05:46:18.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.541+0000 7f3d40e8a700 1 --2- 172.21.15.76:0/2585616102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d3c107730 0x7f3d3c19b450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3d38011780 tx=0x7f3d38011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:18.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.541+0000 7f3d31ffb700 1 -- 172.21.15.76:0/2585616102 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d38003af0 con 0x7f3d3c107730 2024-03-12T05:46:18.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.541+0000 7f3d430ee700 1 -- 172.21.15.76:0/2585616102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3c19bb90 con 0x7f3d3c107730 2024-03-12T05:46:18.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.541+0000 7f3d31ffb700 1 -- 172.21.15.76:0/2585616102 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d38003c50 con 0x7f3d3c107730 2024-03-12T05:46:18.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.541+0000 7f3d31ffb700 1 -- 172.21.15.76:0/2585616102 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d380197e0 con 0x7f3d3c107730 2024-03-12T05:46:18.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.541+0000 7f3d430ee700 1 -- 172.21.15.76:0/2585616102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3c19c030 con 0x7f3d3c107730 2024-03-12T05:46:18.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.542+0000 7f3d31ffb700 1 -- 172.21.15.76:0/2585616102 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f3d38019a20 con 0x7f3d3c107730 2024-03-12T05:46:18.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.542+0000 7f3d430ee700 1 -- 172.21.15.76:0/2585616102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3c10bb10 con 0x7f3d3c107730 2024-03-12T05:46:18.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.542+0000 7f3d31ffb700 1 --2- 172.21.15.76:0/2585616102 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f3d24038cb0 0x7f3d2403b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:18.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.542+0000 7f3d31ffb700 1 -- 172.21.15.76:0/2585616102 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3d3804d030 con 0x7f3d3c107730 2024-03-12T05:46:18.559 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.542+0000 7f3d33fff700 1 --2- 172.21.15.76:0/2585616102 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f3d24038cb0 0x7f3d2403b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:18.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.543+0000 7f3d33fff700 1 --2- 172.21.15.76:0/2585616102 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f3d24038cb0 0x7f3d2403b170 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c006fd0 tx=0x7f3d2c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:18.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.547+0000 7f3d31ffb700 1 -- 172.21.15.76:0/2585616102 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3d38018c20 con 0x7f3d3c107730 2024-03-12T05:46:18.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:17.739+0000 7f3d430ee700 1 -- 172.21.15.76:0/2585616102 --> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d3c104030 con 0x7f3d24038cb0 2024-03-12T05:46:18.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.071+0000 7f3d31ffb700 1 -- 172.21.15.76:0/2585616102 <== mgr.14120 v2:172.21.15.76:6800/306877437 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3d3c104030 con 0x7f3d24038cb0 2024-03-12T05:46:18.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.073+0000 7f3d430ee700 1 -- 172.21.15.76:0/2585616102 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f3d24038cb0 msgr2=0x7f3d2403b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:18.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.073+0000 7f3d430ee700 1 --2- 172.21.15.76:0/2585616102 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f3d24038cb0 0x7f3d2403b170 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c006fd0 tx=0x7f3d2c006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:18.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.073+0000 7f3d430ee700 1 -- 172.21.15.76:0/2585616102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d3c107730 msgr2=0x7f3d3c19b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:18.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.073+0000 7f3d430ee700 1 --2- 172.21.15.76:0/2585616102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d3c107730 0x7f3d3c19b450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3d38011780 tx=0x7f3d38011b40 comp rx=0 tx=0).stop 2024-03-12T05:46:18.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.073+0000 7f3d430ee700 1 -- 172.21.15.76:0/2585616102 shutdown_connections 2024-03-12T05:46:18.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.073+0000 7f3d430ee700 1 --2- 172.21.15.76:0/2585616102 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f3d24038cb0 0x7f3d2403b170 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:18.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.073+0000 7f3d430ee700 1 --2- 172.21.15.76:0/2585616102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d3c107730 0x7f3d3c19b450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:18.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.073+0000 7f3d430ee700 1 -- 172.21.15.76:0/2585616102 >> 172.21.15.76:0/2585616102 conn(0x7f3d3c102db0 msgr2=0x7f3d3c103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:18.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.073+0000 7f3d430ee700 1 -- 172.21.15.76:0/2585616102 shutdown_connections 2024-03-12T05:46:18.560 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.074+0000 7f3d430ee700 1 -- 172.21.15.76:0/2585616102 wait complete. 2024-03-12T05:46:19.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-mon[32095]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:46:19.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-mon[32095]: Generating ssh key... 2024-03-12T05:46:19.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' 2024-03-12T05:46:19.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:18 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' 2024-03-12T05:46:19.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtCA10uVSngFKztTlLDxuFH94mYrBGmPQLu8cao8jsx/9xAsLZKO+Ki5J6/I/CDZrGzhRNWJBlAI+DRjjOzVVjGXSlJVJm/Tpopcf/PK6tsEBt2qiDSDcI2mbcqLmjnuBhhqM0lojtNkaVk5YOp8htEU1QgEJDMFzPoey8yklPRsApny7D2xxLQj97V71apQKUrhFqdtERV6CY9nKC+zg0de6bq4AiC03Kf678RZmFM5tqaViGPQAXSwhe6QKC3wWaOE3qVWfWSMI+9CQDISvF1Pj7oYCbNr2xKprVAebl/afW6bhLDsV4Z58/FXX/ZsB3GZgkYdpSSdBfb8dFuXJDRKwWc/X8KjjmfttCXgAflEdkdyY8Uzu81aFJlOu5IEyyEG7eloxlnXJYL9FWK3Dzs7ssYQKh8PI4f0W80RdQzkiDgGvWpMZEHdKPE+DTTkaWmRiAe+9qp45AjXpoAt1d2rl4I/67uSMWKZrNUSDTi9NwSs9k5yWTvYQORSB/3EE= ceph-9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:46:19.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.967+0000 7f41e3861700 1 Processor -- start 2024-03-12T05:46:19.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.967+0000 7f41e3861700 1 -- start start 2024-03-12T05:46:19.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.967+0000 7f41e3861700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41dc107c30 0x7f41dc108050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:19.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.967+0000 7f41e3861700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc108620 con 0x7f41dc107c30 2024-03-12T05:46:19.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.967+0000 7f41e15fd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41dc107c30 0x7f41dc108050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:19.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.967+0000 7f41e15fd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41dc107c30 0x7f41dc108050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47198/0 (socket says 172.21.15.76:47198) 2024-03-12T05:46:19.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.967+0000 7f41e15fd700 1 -- 172.21.15.76:0/2332727652 learned_addr learned my addr 172.21.15.76:0/2332727652 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:19.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.968+0000 7f41e15fd700 1 -- 172.21.15.76:0/2332727652 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc106800 con 0x7f41dc107c30 2024-03-12T05:46:19.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.968+0000 7f41e15fd700 1 --2- 172.21.15.76:0/2332727652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41dc107c30 0x7f41dc108050 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f41d80098a0 tx=0x7f41d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f998920b38eef33 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:19.788 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.968+0000 7f41d3fff700 1 -- 172.21.15.76:0/2332727652 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41d8004160 con 0x7f41dc107c30 2024-03-12T05:46:19.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.968+0000 7f41d3fff700 1 -- 172.21.15.76:0/2332727652 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f41d80042c0 con 0x7f41dc107c30 2024-03-12T05:46:19.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.968+0000 7f41d3fff700 1 -- 172.21.15.76:0/2332727652 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41d8011ac0 con 0x7f41dc107c30 2024-03-12T05:46:19.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.968+0000 7f41e3861700 1 -- 172.21.15.76:0/2332727652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41dc107c30 msgr2=0x7f41dc108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:19.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.969+0000 7f41e3861700 1 --2- 172.21.15.76:0/2332727652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41dc107c30 0x7f41dc108050 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f41d80098a0 tx=0x7f41d8009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:19.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.969+0000 7f41e3861700 1 -- 172.21.15.76:0/2332727652 shutdown_connections 2024-03-12T05:46:19.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.969+0000 7f41e3861700 1 --2- 172.21.15.76:0/2332727652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41dc107c30 0x7f41dc108050 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:19.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.969+0000 7f41e3861700 1 -- 172.21.15.76:0/2332727652 >> 172.21.15.76:0/2332727652 conn(0x7f41dc101310 msgr2=0x7f41dc103730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:19.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.969+0000 7f41e3861700 1 -- 172.21.15.76:0/2332727652 shutdown_connections 2024-03-12T05:46:19.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.969+0000 7f41e3861700 1 -- 172.21.15.76:0/2332727652 wait complete. 2024-03-12T05:46:19.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.969+0000 7f41e3861700 1 Processor -- start 2024-03-12T05:46:19.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.969+0000 7f41e3861700 1 -- start start 2024-03-12T05:46:19.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.969+0000 7f41e3861700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41dc107c30 0x7f41dc199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:19.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.969+0000 7f41e3861700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc1997c0 con 0x7f41dc107c30 2024-03-12T05:46:19.789 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.970+0000 7f41e15fd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41dc107c30 0x7f41dc199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:19.790 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.970+0000 7f41e15fd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41dc107c30 0x7f41dc199280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47212/0 (socket says 172.21.15.76:47212) 2024-03-12T05:46:19.790 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.970+0000 7f41e15fd700 1 -- 172.21.15.76:0/1655849063 learned_addr learned my addr 172.21.15.76:0/1655849063 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:19.790 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.970+0000 7f41e15fd700 1 -- 172.21.15.76:0/1655849063 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d8009550 con 0x7f41dc107c30 2024-03-12T05:46:19.790 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.970+0000 7f41e15fd700 1 --2- 172.21.15.76:0/1655849063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41dc107c30 0x7f41dc199280 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f41d8011a70 tx=0x7f41d8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:19.790 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.970+0000 7f41d27fc700 1 -- 172.21.15.76:0/1655849063 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41d80039c0 con 0x7f41dc107c30 2024-03-12T05:46:19.790 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.970+0000 7f41d27fc700 1 -- 172.21.15.76:0/1655849063 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f41d8003b20 con 0x7f41dc107c30 2024-03-12T05:46:19.790 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.970+0000 7f41e3861700 1 -- 172.21.15.76:0/1655849063 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41dc1999c0 con 0x7f41dc107c30 2024-03-12T05:46:19.790 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.970+0000 7f41d27fc700 1 -- 172.21.15.76:0/1655849063 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41d8024610 con 0x7f41dc107c30 2024-03-12T05:46:19.790 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.970+0000 7f41e3861700 1 -- 172.21.15.76:0/1655849063 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc199e60 con 0x7f41dc107c30 2024-03-12T05:46:19.790 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.971+0000 7f41d27fc700 1 -- 172.21.15.76:0/1655849063 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f41d801a430 con 0x7f41dc107c30 2024-03-12T05:46:19.790 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.971+0000 7f41e3861700 1 -- 172.21.15.76:0/1655849063 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41dc1056f0 con 0x7f41dc107c30 2024-03-12T05:46:19.790 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.971+0000 7f41d27fc700 1 --2- 172.21.15.76:0/1655849063 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f41c803af10 0x7f41c803d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:19.790 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.971+0000 7f41d27fc700 1 -- 172.21.15.76:0/1655849063 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f41d804cc50 con 0x7f41dc107c30 2024-03-12T05:46:19.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.971+0000 7f41e0dfc700 1 --2- 172.21.15.76:0/1655849063 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f41c803af10 0x7f41c803d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:19.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.972+0000 7f41e0dfc700 1 --2- 172.21.15.76:0/1655849063 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f41c803af10 0x7f41c803d3d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f41cc006fd0 tx=0x7f41cc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:19.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:18.976+0000 7f41d27fc700 1 -- 172.21.15.76:0/1655849063 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f41d8052020 con 0x7f41dc107c30 2024-03-12T05:46:19.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:19.168+0000 7f41e3861700 1 -- 172.21.15.76:0/1655849063 --> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f41dc0620b0 con 0x7f41c803af10 2024-03-12T05:46:19.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:19.169+0000 7f41d27fc700 1 -- 172.21.15.76:0/1655849063 <== mgr.14120 v2:172.21.15.76:6800/306877437 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f41dc0620b0 con 0x7f41c803af10 2024-03-12T05:46:19.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:19.171+0000 7f41e3861700 1 -- 172.21.15.76:0/1655849063 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f41c803af10 msgr2=0x7f41c803d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:19.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:19.171+0000 7f41e3861700 1 --2- 172.21.15.76:0/1655849063 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f41c803af10 0x7f41c803d3d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f41cc006fd0 tx=0x7f41cc006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:19.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:19.171+0000 7f41e3861700 1 -- 172.21.15.76:0/1655849063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41dc107c30 msgr2=0x7f41dc199280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:19.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:19.171+0000 7f41e3861700 1 --2- 172.21.15.76:0/1655849063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41dc107c30 0x7f41dc199280 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f41d8011a70 tx=0x7f41d8011b50 comp rx=0 tx=0).stop 2024-03-12T05:46:19.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:19.171+0000 7f41e3861700 1 -- 172.21.15.76:0/1655849063 shutdown_connections 2024-03-12T05:46:19.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:19.171+0000 7f41e3861700 1 --2- 172.21.15.76:0/1655849063 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f41c803af10 0x7f41c803d3d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:19.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:19.171+0000 7f41e3861700 1 --2- 172.21.15.76:0/1655849063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41dc107c30 0x7f41dc199280 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:19.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:19.171+0000 7f41e3861700 1 -- 172.21.15.76:0/1655849063 >> 172.21.15.76:0/1655849063 conn(0x7f41dc101310 msgr2=0x7f41dc18fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:19.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:19.171+0000 7f41e3861700 1 -- 172.21.15.76:0/1655849063 shutdown_connections 2024-03-12T05:46:19.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:19.171+0000 7f41e3861700 1 -- 172.21.15.76:0/1655849063 wait complete. 2024-03-12T05:46:19.792 INFO:teuthology.orchestra.run.smithi076.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-12T05:46:19.792 INFO:teuthology.orchestra.run.smithi076.stdout:Adding key to root@localhost authorized_keys... 2024-03-12T05:46:19.792 INFO:teuthology.orchestra.run.smithi076.stdout:Adding host smithi076... 2024-03-12T05:46:20.379 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:20 smithi076 ceph-mon[32095]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:46:21.169 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:21 smithi076 ceph-mon[32095]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi076", "addr": "172.21.15.76", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:46:22.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:22 smithi076 ceph-mon[32095]: Deploying cephadm binary to smithi076 2024-03-12T05:46:25.229 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Added host 'smithi076' with addr '172.21.15.76' 2024-03-12T05:46:25.229 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.173+0000 7f9900723700 1 Processor -- start 2024-03-12T05:46:25.229 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.173+0000 7f9900723700 1 -- start start 2024-03-12T05:46:25.229 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.173+0000 7f9900723700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98f81059a0 0x7f98f8105dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:25.229 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.173+0000 7f9900723700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f8106390 con 0x7f98f81059a0 2024-03-12T05:46:25.229 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.173+0000 7f98fe4bf700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98f81059a0 0x7f98f8105dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:25.229 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.173+0000 7f98fe4bf700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98f81059a0 0x7f98f8105dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47228/0 (socket says 172.21.15.76:47228) 2024-03-12T05:46:25.229 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.173+0000 7f98fe4bf700 1 -- 172.21.15.76:0/397238319 learned_addr learned my addr 172.21.15.76:0/397238319 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:25.229 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.173+0000 7f98fe4bf700 1 -- 172.21.15.76:0/397238319 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f8108930 con 0x7f98f81059a0 2024-03-12T05:46:25.229 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.174+0000 7f98fe4bf700 1 --2- 172.21.15.76:0/397238319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98f81059a0 0x7f98f8105dc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f98f40098a0 tx=0x7f98f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52adf62ceb438b6a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:25.230 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.174+0000 7f98fd4bd700 1 -- 172.21.15.76:0/397238319 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f98f4004160 con 0x7f98f81059a0 2024-03-12T05:46:25.230 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.174+0000 7f98fd4bd700 1 -- 172.21.15.76:0/397238319 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98f40042c0 con 0x7f98f81059a0 2024-03-12T05:46:25.230 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.174+0000 7f98fd4bd700 1 -- 172.21.15.76:0/397238319 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f98f4011ac0 con 0x7f98f81059a0 2024-03-12T05:46:25.230 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.174+0000 7f9900723700 1 -- 172.21.15.76:0/397238319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98f81059a0 msgr2=0x7f98f8105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:25.230 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.174+0000 7f9900723700 1 --2- 172.21.15.76:0/397238319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98f81059a0 0x7f98f8105dc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f98f40098a0 tx=0x7f98f4009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:25.230 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.174+0000 7f9900723700 1 -- 172.21.15.76:0/397238319 shutdown_connections 2024-03-12T05:46:25.230 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.174+0000 7f9900723700 1 --2- 172.21.15.76:0/397238319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98f81059a0 0x7f98f8105dc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:25.230 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.174+0000 7f9900723700 1 -- 172.21.15.76:0/397238319 >> 172.21.15.76:0/397238319 conn(0x7f98f8101310 msgr2=0x7f98f8103730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:25.230 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.174+0000 7f9900723700 1 -- 172.21.15.76:0/397238319 shutdown_connections 2024-03-12T05:46:25.230 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.174+0000 7f9900723700 1 -- 172.21.15.76:0/397238319 wait complete. 2024-03-12T05:46:25.230 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.175+0000 7f9900723700 1 Processor -- start 2024-03-12T05:46:25.230 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.175+0000 7f9900723700 1 -- start start 2024-03-12T05:46:25.230 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.175+0000 7f9900723700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98f81059a0 0x7f98f81081b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:25.230 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.175+0000 7f9900723700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f81086f0 con 0x7f98f81059a0 2024-03-12T05:46:25.231 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.175+0000 7f98fe4bf700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98f81059a0 0x7f98f81081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:25.231 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.175+0000 7f98fe4bf700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98f81059a0 0x7f98f81081b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47230/0 (socket says 172.21.15.76:47230) 2024-03-12T05:46:25.231 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.175+0000 7f98fe4bf700 1 -- 172.21.15.76:0/511491488 learned_addr learned my addr 172.21.15.76:0/511491488 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:25.231 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.175+0000 7f98fe4bf700 1 -- 172.21.15.76:0/511491488 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f4009550 con 0x7f98f81059a0 2024-03-12T05:46:25.231 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.176+0000 7f98fe4bf700 1 --2- 172.21.15.76:0/511491488 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98f81059a0 0x7f98f81081b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f98f4000c00 tx=0x7f98f4011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:25.231 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.176+0000 7f98ef7fe700 1 -- 172.21.15.76:0/511491488 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f98f4003aa0 con 0x7f98f81059a0 2024-03-12T05:46:25.231 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.176+0000 7f98ef7fe700 1 -- 172.21.15.76:0/511491488 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98f4003c00 con 0x7f98f81059a0 2024-03-12T05:46:25.231 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.176+0000 7f9900723700 1 -- 172.21.15.76:0/511491488 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f81068e0 con 0x7f98f81059a0 2024-03-12T05:46:25.231 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.176+0000 7f98ef7fe700 1 -- 172.21.15.76:0/511491488 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f98f40248a0 con 0x7f98f81059a0 2024-03-12T05:46:25.231 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.176+0000 7f9900723700 1 -- 172.21.15.76:0/511491488 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f8106d80 con 0x7f98f81059a0 2024-03-12T05:46:25.231 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.177+0000 7f98ef7fe700 1 -- 172.21.15.76:0/511491488 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f98f401a430 con 0x7f98f81059a0 2024-03-12T05:46:25.231 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.177+0000 7f9900723700 1 -- 172.21.15.76:0/511491488 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f810bb60 con 0x7f98f81059a0 2024-03-12T05:46:25.231 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.177+0000 7f98ef7fe700 1 --2- 172.21.15.76:0/511491488 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f98e4038cb0 0x7f98e403b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:25.231 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.177+0000 7f98ef7fe700 1 -- 172.21.15.76:0/511491488 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f98f404cd80 con 0x7f98f81059a0 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.177+0000 7f98fdcbe700 1 --2- 172.21.15.76:0/511491488 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f98e4038cb0 0x7f98e403b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.177+0000 7f98fdcbe700 1 --2- 172.21.15.76:0/511491488 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f98e4038cb0 0x7f98e403b170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f98e8006fd0 tx=0x7f98e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.182+0000 7f98ef7fe700 1 -- 172.21.15.76:0/511491488 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f98f4018530 con 0x7f98f81059a0 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:20.378+0000 7f9900723700 1 -- 172.21.15.76:0/511491488 --> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi076", "addr": "172.21.15.76", "target": ["mon-mgr", ""]}) v1 -- 0x7f98f8107410 con 0x7f98e4038cb0 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:24.658+0000 7f98ef7fe700 1 -- 172.21.15.76:0/511491488 <== mgr.14120 v2:172.21.15.76:6800/306877437 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f98f8107410 con 0x7f98e4038cb0 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:24.661+0000 7f9900723700 1 -- 172.21.15.76:0/511491488 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f98e4038cb0 msgr2=0x7f98e403b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:24.661+0000 7f9900723700 1 --2- 172.21.15.76:0/511491488 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f98e4038cb0 0x7f98e403b170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f98e8006fd0 tx=0x7f98e8006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:24.661+0000 7f9900723700 1 -- 172.21.15.76:0/511491488 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98f81059a0 msgr2=0x7f98f81081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:24.661+0000 7f9900723700 1 --2- 172.21.15.76:0/511491488 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98f81059a0 0x7f98f81081b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f98f4000c00 tx=0x7f98f4011b40 comp rx=0 tx=0).stop 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:24.661+0000 7f9900723700 1 -- 172.21.15.76:0/511491488 shutdown_connections 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:24.661+0000 7f9900723700 1 --2- 172.21.15.76:0/511491488 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f98e4038cb0 0x7f98e403b170 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:24.661+0000 7f9900723700 1 --2- 172.21.15.76:0/511491488 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98f81059a0 0x7f98f81081b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:24.661+0000 7f9900723700 1 -- 172.21.15.76:0/511491488 >> 172.21.15.76:0/511491488 conn(0x7f98f8101310 msgr2=0x7f98f8101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:24.661+0000 7f9900723700 1 -- 172.21.15.76:0/511491488 shutdown_connections 2024-03-12T05:46:25.232 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:24.661+0000 7f9900723700 1 -- 172.21.15.76:0/511491488 wait complete. 2024-03-12T05:46:25.233 INFO:teuthology.orchestra.run.smithi076.stdout:Deploying unmanaged mon service... 2024-03-12T05:46:25.895 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:25 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' 2024-03-12T05:46:25.895 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:25 smithi076 ceph-mon[32095]: Added host smithi076 2024-03-12T05:46:25.895 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:25 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:46:27.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:26 smithi076 ceph-mon[32095]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:46:27.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:26 smithi076 ceph-mon[32095]: Saving service mon spec with placement count:5 2024-03-12T05:46:27.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:26 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' 2024-03-12T05:46:27.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:26 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' 2024-03-12T05:46:27.527 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-12T05:46:27.527 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.742+0000 7fc8dc3a3700 1 Processor -- start 2024-03-12T05:46:27.527 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.742+0000 7fc8dc3a3700 1 -- start start 2024-03-12T05:46:27.527 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.743+0000 7fc8dc3a3700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8d4105980 0x7fc8d4105da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:27.527 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.743+0000 7fc8dc3a3700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d4106370 con 0x7fc8d4105980 2024-03-12T05:46:27.527 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.743+0000 7fc8da13f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8d4105980 0x7fc8d4105da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:27.527 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.743+0000 7fc8da13f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8d4105980 0x7fc8d4105da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54192/0 (socket says 172.21.15.76:54192) 2024-03-12T05:46:27.527 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.743+0000 7fc8da13f700 1 -- 172.21.15.76:0/1656811439 learned_addr learned my addr 172.21.15.76:0/1656811439 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:27.527 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.743+0000 7fc8da13f700 1 -- 172.21.15.76:0/1656811439 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d41064b0 con 0x7fc8d4105980 2024-03-12T05:46:27.528 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.743+0000 7fc8da13f700 1 --2- 172.21.15.76:0/1656811439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8d4105980 0x7fc8d4105da0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc8c40098a0 tx=0x7fc8c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ecdcaed366cad6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:27.528 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.743+0000 7fc8d913d700 1 -- 172.21.15.76:0/1656811439 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8c4004160 con 0x7fc8d4105980 2024-03-12T05:46:27.528 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.743+0000 7fc8d913d700 1 -- 172.21.15.76:0/1656811439 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8c40042c0 con 0x7fc8d4105980 2024-03-12T05:46:27.528 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.743+0000 7fc8d913d700 1 -- 172.21.15.76:0/1656811439 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8c4011ac0 con 0x7fc8d4105980 2024-03-12T05:46:27.528 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.744+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/1656811439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8d4105980 msgr2=0x7fc8d4105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:27.528 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.744+0000 7fc8dc3a3700 1 --2- 172.21.15.76:0/1656811439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8d4105980 0x7fc8d4105da0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc8c40098a0 tx=0x7fc8c4009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:27.528 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.744+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/1656811439 shutdown_connections 2024-03-12T05:46:27.528 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.744+0000 7fc8dc3a3700 1 --2- 172.21.15.76:0/1656811439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8d4105980 0x7fc8d4105da0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:27.528 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.744+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/1656811439 >> 172.21.15.76:0/1656811439 conn(0x7fc8d4101370 msgr2=0x7fc8d4103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:27.528 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.744+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/1656811439 shutdown_connections 2024-03-12T05:46:27.528 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.744+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/1656811439 wait complete. 2024-03-12T05:46:27.528 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.745+0000 7fc8dc3a3700 1 Processor -- start 2024-03-12T05:46:27.528 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.745+0000 7fc8dc3a3700 1 -- start start 2024-03-12T05:46:27.528 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.745+0000 7fc8dc3a3700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8d4105980 0x7fc8d419b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.745+0000 7fc8dc3a3700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d419b980 con 0x7fc8d4105980 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.745+0000 7fc8da13f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8d4105980 0x7fc8d419b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.745+0000 7fc8da13f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8d4105980 0x7fc8d419b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54204/0 (socket says 172.21.15.76:54204) 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.745+0000 7fc8da13f700 1 -- 172.21.15.76:0/819401903 learned_addr learned my addr 172.21.15.76:0/819401903 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.745+0000 7fc8da13f700 1 -- 172.21.15.76:0/819401903 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c4009550 con 0x7fc8d4105980 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.745+0000 7fc8da13f700 1 --2- 172.21.15.76:0/819401903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8d4105980 0x7fc8d419b440 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4000c00 tx=0x7fc8c4011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.745+0000 7fc8cb7fe700 1 -- 172.21.15.76:0/819401903 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8c4003aa0 con 0x7fc8d4105980 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.745+0000 7fc8cb7fe700 1 -- 172.21.15.76:0/819401903 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8c4003c00 con 0x7fc8d4105980 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.745+0000 7fc8cb7fe700 1 -- 172.21.15.76:0/819401903 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8c40248a0 con 0x7fc8d4105980 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.745+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/819401903 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d419bb80 con 0x7fc8d4105980 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.746+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/819401903 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d419c020 con 0x7fc8d4105980 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.746+0000 7fc8cb7fe700 1 -- 172.21.15.76:0/819401903 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fc8c401ac60 con 0x7fc8d4105980 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.746+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/819401903 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d4109690 con 0x7fc8d4105980 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.746+0000 7fc8cb7fe700 1 --2- 172.21.15.76:0/819401903 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7fc8c0038d00 0x7fc8c003b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.747+0000 7fc8cb7fe700 1 -- 172.21.15.76:0/819401903 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc8c404cd10 con 0x7fc8d4105980 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.747+0000 7fc8d993e700 1 --2- 172.21.15.76:0/819401903 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7fc8c0038d00 0x7fc8c003b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:27.529 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.747+0000 7fc8d993e700 1 --2- 172.21.15.76:0/819401903 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7fc8c0038d00 0x7fc8c003b1c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0006fd0 tx=0x7fc8d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:27.530 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.751+0000 7fc8cb7fe700 1 -- 172.21.15.76:0/819401903 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc8c401a430 con 0x7fc8d4105980 2024-03-12T05:46:27.530 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.975+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/819401903 --> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fc8d41025c0 con 0x7fc8c0038d00 2024-03-12T05:46:27.530 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.979+0000 7fc8cb7fe700 1 -- 172.21.15.76:0/819401903 <== mgr.14120 v2:172.21.15.76:6800/306877437 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc8d41025c0 con 0x7fc8c0038d00 2024-03-12T05:46:27.530 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.981+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/819401903 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7fc8c0038d00 msgr2=0x7fc8c003b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:27.530 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.981+0000 7fc8dc3a3700 1 --2- 172.21.15.76:0/819401903 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7fc8c0038d00 0x7fc8c003b1c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0006fd0 tx=0x7fc8d0006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:27.530 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.981+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/819401903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8d4105980 msgr2=0x7fc8d419b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:27.530 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.981+0000 7fc8dc3a3700 1 --2- 172.21.15.76:0/819401903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8d4105980 0x7fc8d419b440 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4000c00 tx=0x7fc8c4011b40 comp rx=0 tx=0).stop 2024-03-12T05:46:27.530 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.981+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/819401903 shutdown_connections 2024-03-12T05:46:27.530 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.981+0000 7fc8dc3a3700 1 --2- 172.21.15.76:0/819401903 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7fc8c0038d00 0x7fc8c003b1c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:27.530 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.981+0000 7fc8dc3a3700 1 --2- 172.21.15.76:0/819401903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8d4105980 0x7fc8d419b440 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:27.530 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.981+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/819401903 >> 172.21.15.76:0/819401903 conn(0x7fc8d4101370 msgr2=0x7fc8d4101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:27.530 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.982+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/819401903 shutdown_connections 2024-03-12T05:46:27.530 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:25.982+0000 7fc8dc3a3700 1 -- 172.21.15.76:0/819401903 wait complete. 2024-03-12T05:46:27.530 INFO:teuthology.orchestra.run.smithi076.stdout:Deploying unmanaged mgr service... 2024-03-12T05:46:28.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:28 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' 2024-03-12T05:46:28.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:28 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' 2024-03-12T05:46:29.328 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-12T05:46:29.328 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.990+0000 7f544b59e700 1 Processor -- start 2024-03-12T05:46:29.328 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.991+0000 7f544b59e700 1 -- start start 2024-03-12T05:46:29.328 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.991+0000 7f544b59e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f544c108010 0x7f544c10a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:29.328 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.991+0000 7f544b59e700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c07add0 con 0x7f544c108010 2024-03-12T05:46:29.328 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.991+0000 7f544a59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f544c108010 0x7f544c10a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:29.328 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.991+0000 7f544a59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f544c108010 0x7f544c10a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54218/0 (socket says 172.21.15.76:54218) 2024-03-12T05:46:29.328 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.991+0000 7f544a59c700 1 -- 172.21.15.76:0/1130834969 learned_addr learned my addr 172.21.15.76:0/1130834969 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:29.328 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.991+0000 7f544a59c700 1 -- 172.21.15.76:0/1130834969 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544c07af10 con 0x7f544c108010 2024-03-12T05:46:29.328 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.991+0000 7f544a59c700 1 --2- 172.21.15.76:0/1130834969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f544c108010 0x7f544c10a400 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f543c0098a0 tx=0x7f543c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b950d5fdeeeadc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.992+0000 7f544959a700 1 -- 172.21.15.76:0/1130834969 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f543c004160 con 0x7f544c108010 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.992+0000 7f544959a700 1 -- 172.21.15.76:0/1130834969 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f543c0042c0 con 0x7f544c108010 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.992+0000 7f544959a700 1 -- 172.21.15.76:0/1130834969 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f543c011a80 con 0x7f544c108010 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.993+0000 7f544b59e700 1 -- 172.21.15.76:0/1130834969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f544c108010 msgr2=0x7f544c10a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.993+0000 7f544b59e700 1 --2- 172.21.15.76:0/1130834969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f544c108010 0x7f544c10a400 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f543c0098a0 tx=0x7f543c009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.993+0000 7f544b59e700 1 -- 172.21.15.76:0/1130834969 shutdown_connections 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.993+0000 7f544b59e700 1 --2- 172.21.15.76:0/1130834969 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f544c108010 0x7f544c10a400 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.993+0000 7f544b59e700 1 -- 172.21.15.76:0/1130834969 >> 172.21.15.76:0/1130834969 conn(0x7f544c1019c0 msgr2=0x7f544c103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.993+0000 7f544b59e700 1 -- 172.21.15.76:0/1130834969 shutdown_connections 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.993+0000 7f544b59e700 1 -- 172.21.15.76:0/1130834969 wait complete. 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.993+0000 7f544b59e700 1 Processor -- start 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.994+0000 7f544b59e700 1 -- start start 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.994+0000 7f544b59e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f544c108010 0x7f544c19b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.994+0000 7f544b59e700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c19b7a0 con 0x7f544c108010 2024-03-12T05:46:29.329 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.994+0000 7f544a59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f544c108010 0x7f544c19b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.994+0000 7f544a59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f544c108010 0x7f544c19b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54232/0 (socket says 172.21.15.76:54232) 2024-03-12T05:46:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.994+0000 7f544a59c700 1 -- 172.21.15.76:0/3268858607 learned_addr learned my addr 172.21.15.76:0/3268858607 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.994+0000 7f544a59c700 1 -- 172.21.15.76:0/3268858607 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c009550 con 0x7f544c108010 2024-03-12T05:46:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.994+0000 7f544a59c700 1 --2- 172.21.15.76:0/3268858607 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f544c108010 0x7f544c19b260 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f543c003f20 tx=0x7f543c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.994+0000 7f543b7fe700 1 -- 172.21.15.76:0/3268858607 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f543c0039f0 con 0x7f544c108010 2024-03-12T05:46:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.994+0000 7f543b7fe700 1 -- 172.21.15.76:0/3268858607 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f543c003b50 con 0x7f544c108010 2024-03-12T05:46:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.994+0000 7f544b59e700 1 -- 172.21.15.76:0/3268858607 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f544c19b9a0 con 0x7f544c108010 2024-03-12T05:46:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.995+0000 7f543b7fe700 1 -- 172.21.15.76:0/3268858607 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f543c01a6d0 con 0x7f544c108010 2024-03-12T05:46:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.995+0000 7f544b59e700 1 -- 172.21.15.76:0/3268858607 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544c19be40 con 0x7f544c108010 2024-03-12T05:46:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.995+0000 7f543b7fe700 1 -- 172.21.15.76:0/3268858607 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f543c01a940 con 0x7f544c108010 2024-03-12T05:46:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.996+0000 7f544b59e700 1 -- 172.21.15.76:0/3268858607 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544c10b710 con 0x7f544c108010 2024-03-12T05:46:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.996+0000 7f543b7fe700 1 --2- 172.21.15.76:0/3268858607 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f5434038cb0 0x7f543403b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.996+0000 7f543b7fe700 1 -- 172.21.15.76:0/3268858607 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f543c04d060 con 0x7f544c108010 2024-03-12T05:46:29.330 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.996+0000 7f5449d9b700 1 --2- 172.21.15.76:0/3268858607 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f5434038cb0 0x7f543403b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:29.331 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:27.996+0000 7f5449d9b700 1 --2- 172.21.15.76:0/3268858607 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f5434038cb0 0x7f543403b170 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5440006fd0 tx=0x7f5440006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:29.331 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:28.005+0000 7f543b7fe700 1 -- 172.21.15.76:0/3268858607 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f543c018dc0 con 0x7f544c108010 2024-03-12T05:46:29.331 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:28.230+0000 7f544b59e700 1 -- 172.21.15.76:0/3268858607 --> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f544c102ba0 con 0x7f5434038cb0 2024-03-12T05:46:29.331 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:28.234+0000 7f543b7fe700 1 -- 172.21.15.76:0/3268858607 <== mgr.14120 v2:172.21.15.76:6800/306877437 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f544c102ba0 con 0x7f5434038cb0 2024-03-12T05:46:29.331 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:28.236+0000 7f544b59e700 1 -- 172.21.15.76:0/3268858607 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f5434038cb0 msgr2=0x7f543403b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:29.331 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:28.236+0000 7f544b59e700 1 --2- 172.21.15.76:0/3268858607 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f5434038cb0 0x7f543403b170 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5440006fd0 tx=0x7f5440006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:29.331 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:28.236+0000 7f544b59e700 1 -- 172.21.15.76:0/3268858607 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f544c108010 msgr2=0x7f544c19b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:29.331 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:28.236+0000 7f544b59e700 1 --2- 172.21.15.76:0/3268858607 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f544c108010 0x7f544c19b260 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f543c003f20 tx=0x7f543c011b00 comp rx=0 tx=0).stop 2024-03-12T05:46:29.331 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:28.237+0000 7f544b59e700 1 -- 172.21.15.76:0/3268858607 shutdown_connections 2024-03-12T05:46:29.331 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:28.237+0000 7f544b59e700 1 --2- 172.21.15.76:0/3268858607 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f5434038cb0 0x7f543403b170 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:29.331 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:28.237+0000 7f544b59e700 1 --2- 172.21.15.76:0/3268858607 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f544c108010 0x7f544c19b260 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:29.331 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:28.237+0000 7f544b59e700 1 -- 172.21.15.76:0/3268858607 >> 172.21.15.76:0/3268858607 conn(0x7f544c1019c0 msgr2=0x7f544c102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:29.331 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:28.237+0000 7f544b59e700 1 -- 172.21.15.76:0/3268858607 shutdown_connections 2024-03-12T05:46:29.331 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:28.237+0000 7f544b59e700 1 -- 172.21.15.76:0/3268858607 wait complete. 2024-03-12T05:46:30.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:30 smithi076 ceph-mon[32095]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:46:30.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:30 smithi076 ceph-mon[32095]: Saving service mgr spec with placement count:2 2024-03-12T05:46:30.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:30 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' 2024-03-12T05:46:30.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:30 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/701826276' entity='client.admin' 2024-03-12T05:46:30.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.741+0000 7f7416f6c700 1 Processor -- start 2024-03-12T05:46:30.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.741+0000 7f7416f6c700 1 -- start start 2024-03-12T05:46:30.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.741+0000 7f7416f6c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7410107690 0x7f7410107ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:30.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.741+0000 7f7416f6c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7410108080 con 0x7f7410107690 2024-03-12T05:46:30.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.741+0000 7f7414d08700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7410107690 0x7f7410107ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:30.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.741+0000 7f7414d08700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7410107690 0x7f7410107ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54234/0 (socket says 172.21.15.76:54234) 2024-03-12T05:46:30.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.741+0000 7f7414d08700 1 -- 172.21.15.76:0/3515462834 learned_addr learned my addr 172.21.15.76:0/3515462834 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:30.543 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.742+0000 7f7414d08700 1 -- 172.21.15.76:0/3515462834 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7410108840 con 0x7f7410107690 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.742+0000 7f7414d08700 1 --2- 172.21.15.76:0/3515462834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7410107690 0x7f7410107ab0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f74040098a0 tx=0x7f7404009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15f43a5852b0810 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.742+0000 7f740f7fe700 1 -- 172.21.15.76:0/3515462834 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7404004160 con 0x7f7410107690 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.742+0000 7f740f7fe700 1 -- 172.21.15.76:0/3515462834 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74040042c0 con 0x7f7410107690 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.742+0000 7f740f7fe700 1 -- 172.21.15.76:0/3515462834 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7404011a80 con 0x7f7410107690 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.743+0000 7f7416f6c700 1 -- 172.21.15.76:0/3515462834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7410107690 msgr2=0x7f7410107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.743+0000 7f7416f6c700 1 --2- 172.21.15.76:0/3515462834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7410107690 0x7f7410107ab0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f74040098a0 tx=0x7f7404009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.743+0000 7f7416f6c700 1 -- 172.21.15.76:0/3515462834 shutdown_connections 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.743+0000 7f7416f6c700 1 --2- 172.21.15.76:0/3515462834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7410107690 0x7f7410107ab0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.743+0000 7f7416f6c700 1 -- 172.21.15.76:0/3515462834 >> 172.21.15.76:0/3515462834 conn(0x7f7410102cb0 msgr2=0x7f74101050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.743+0000 7f7416f6c700 1 -- 172.21.15.76:0/3515462834 shutdown_connections 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.743+0000 7f7416f6c700 1 -- 172.21.15.76:0/3515462834 wait complete. 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.744+0000 7f7416f6c700 1 Processor -- start 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.744+0000 7f7416f6c700 1 -- start start 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.744+0000 7f7416f6c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7410107690 0x7f741019d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.744+0000 7f7416f6c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741019db60 con 0x7f7410107690 2024-03-12T05:46:30.544 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.744+0000 7f7414d08700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7410107690 0x7f741019d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.744+0000 7f7414d08700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7410107690 0x7f741019d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54242/0 (socket says 172.21.15.76:54242) 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.744+0000 7f7414d08700 1 -- 172.21.15.76:0/701826276 learned_addr learned my addr 172.21.15.76:0/701826276 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.744+0000 7f7414d08700 1 -- 172.21.15.76:0/701826276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7404009550 con 0x7f7410107690 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.744+0000 7f7414d08700 1 --2- 172.21.15.76:0/701826276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7410107690 0x7f741019d620 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7404011740 tx=0x7f7404011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.745+0000 7f740dffb700 1 -- 172.21.15.76:0/701826276 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74040039e0 con 0x7f7410107690 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.745+0000 7f7416f6c700 1 -- 172.21.15.76:0/701826276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f741019dd60 con 0x7f7410107690 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.745+0000 7f740dffb700 1 -- 172.21.15.76:0/701826276 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7404003b40 con 0x7f7410107690 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.745+0000 7f740dffb700 1 -- 172.21.15.76:0/701826276 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7404024710 con 0x7f7410107690 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.745+0000 7f7416f6c700 1 -- 172.21.15.76:0/701826276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741019e200 con 0x7f7410107690 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.746+0000 7f7416f6c700 1 -- 172.21.15.76:0/701826276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741010ba70 con 0x7f7410107690 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.746+0000 7f740dffb700 1 -- 172.21.15.76:0/701826276 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f74040249d0 con 0x7f7410107690 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.746+0000 7f740dffb700 1 --2- 172.21.15.76:0/701826276 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f7400038c60 0x7f740003b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.746+0000 7f740dffb700 1 -- 172.21.15.76:0/701826276 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f740404cea0 con 0x7f7410107690 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.746+0000 7f740ffff700 1 --2- 172.21.15.76:0/701826276 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f7400038c60 0x7f740003b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:30.545 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.747+0000 7f740ffff700 1 --2- 172.21.15.76:0/701826276 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f7400038c60 0x7f740003b120 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f73fc006fd0 tx=0x7f73fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:30.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.751+0000 7f740dffb700 1 -- 172.21.15.76:0/701826276 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7404024c80 con 0x7f7410107690 2024-03-12T05:46:30.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.955+0000 7f7416f6c700 1 -- 172.21.15.76:0/701826276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f74101a0b40 con 0x7f7410107690 2024-03-12T05:46:30.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.958+0000 7f740dffb700 1 -- 172.21.15.76:0/701826276 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f740404c810 con 0x7f7410107690 2024-03-12T05:46:30.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.960+0000 7f7416f6c700 1 -- 172.21.15.76:0/701826276 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f7400038c60 msgr2=0x7f740003b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:30.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.960+0000 7f7416f6c700 1 --2- 172.21.15.76:0/701826276 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f7400038c60 0x7f740003b120 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f73fc006fd0 tx=0x7f73fc006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:30.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.960+0000 7f7416f6c700 1 -- 172.21.15.76:0/701826276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7410107690 msgr2=0x7f741019d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:30.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.960+0000 7f7416f6c700 1 --2- 172.21.15.76:0/701826276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7410107690 0x7f741019d620 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7404011740 tx=0x7f7404011b00 comp rx=0 tx=0).stop 2024-03-12T05:46:30.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.960+0000 7f7416f6c700 1 -- 172.21.15.76:0/701826276 shutdown_connections 2024-03-12T05:46:30.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.960+0000 7f7416f6c700 1 --2- 172.21.15.76:0/701826276 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f7400038c60 0x7f740003b120 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:30.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.960+0000 7f7416f6c700 1 --2- 172.21.15.76:0/701826276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7410107690 0x7f741019d620 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:30.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.960+0000 7f7416f6c700 1 -- 172.21.15.76:0/701826276 >> 172.21.15.76:0/701826276 conn(0x7f7410102cb0 msgr2=0x7f74101036f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:30.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.961+0000 7f7416f6c700 1 -- 172.21.15.76:0/701826276 shutdown_connections 2024-03-12T05:46:30.546 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:29.961+0000 7f7416f6c700 1 -- 172.21.15.76:0/701826276 wait complete. 2024-03-12T05:46:31.392 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:31 smithi076 ceph-mon[32095]: from='mgr.14120 172.21.15.76:0/3361815285' entity='mgr.x' 2024-03-12T05:46:31.392 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:31 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2225217322' entity='client.admin' 2024-03-12T05:46:32.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.921+0000 7f50aef67700 1 Processor -- start 2024-03-12T05:46:32.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.921+0000 7f50aef67700 1 -- start start 2024-03-12T05:46:32.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.922+0000 7f50aef67700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a8108c90 0x7f50a81090b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:32.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.922+0000 7f50aef67700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a8109680 con 0x7f50a8108c90 2024-03-12T05:46:32.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.922+0000 7f50acd03700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a8108c90 0x7f50a81090b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:32.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.922+0000 7f50acd03700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a8108c90 0x7f50a81090b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54244/0 (socket says 172.21.15.76:54244) 2024-03-12T05:46:32.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.922+0000 7f50acd03700 1 -- 172.21.15.76:0/2967902784 learned_addr learned my addr 172.21.15.76:0/2967902784 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:32.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.922+0000 7f50acd03700 1 -- 172.21.15.76:0/2967902784 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a81097c0 con 0x7f50a8108c90 2024-03-12T05:46:32.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.922+0000 7f50acd03700 1 --2- 172.21.15.76:0/2967902784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a8108c90 0x7f50a81090b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f509c0098a0 tx=0x7f509c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f8446092c4de4b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:32.389 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.922+0000 7f50a77fe700 1 -- 172.21.15.76:0/2967902784 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f509c004160 con 0x7f50a8108c90 2024-03-12T05:46:32.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.922+0000 7f50a77fe700 1 -- 172.21.15.76:0/2967902784 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f509c0042c0 con 0x7f50a8108c90 2024-03-12T05:46:32.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.922+0000 7f50a77fe700 1 -- 172.21.15.76:0/2967902784 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f509c011a80 con 0x7f50a8108c90 2024-03-12T05:46:32.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.923+0000 7f50aef67700 1 -- 172.21.15.76:0/2967902784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a8108c90 msgr2=0x7f50a81090b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:32.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.923+0000 7f50aef67700 1 --2- 172.21.15.76:0/2967902784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a8108c90 0x7f50a81090b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f509c0098a0 tx=0x7f509c009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:32.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.923+0000 7f50aef67700 1 -- 172.21.15.76:0/2967902784 shutdown_connections 2024-03-12T05:46:32.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.923+0000 7f50aef67700 1 --2- 172.21.15.76:0/2967902784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a8108c90 0x7f50a81090b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:32.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.923+0000 7f50aef67700 1 -- 172.21.15.76:0/2967902784 >> 172.21.15.76:0/2967902784 conn(0x7f50a8101370 msgr2=0x7f50a8103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:32.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.923+0000 7f50aef67700 1 -- 172.21.15.76:0/2967902784 shutdown_connections 2024-03-12T05:46:32.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.923+0000 7f50aef67700 1 -- 172.21.15.76:0/2967902784 wait complete. 2024-03-12T05:46:32.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.923+0000 7f50aef67700 1 Processor -- start 2024-03-12T05:46:32.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.924+0000 7f50aef67700 1 -- start start 2024-03-12T05:46:32.390 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.924+0000 7f50aef67700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a8108c90 0x7f50a819b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:32.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.924+0000 7f50aef67700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a819b940 con 0x7f50a8108c90 2024-03-12T05:46:32.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.924+0000 7f50acd03700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a8108c90 0x7f50a819b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:32.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.924+0000 7f50acd03700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a8108c90 0x7f50a819b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54254/0 (socket says 172.21.15.76:54254) 2024-03-12T05:46:32.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.924+0000 7f50acd03700 1 -- 172.21.15.76:0/2225217322 learned_addr learned my addr 172.21.15.76:0/2225217322 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:32.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.924+0000 7f50acd03700 1 -- 172.21.15.76:0/2225217322 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509c009550 con 0x7f50a8108c90 2024-03-12T05:46:32.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.924+0000 7f50acd03700 1 --2- 172.21.15.76:0/2225217322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a8108c90 0x7f50a819b400 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f509c011740 tx=0x7f509c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:32.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.924+0000 7f50a5ffb700 1 -- 172.21.15.76:0/2225217322 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f509c003af0 con 0x7f50a8108c90 2024-03-12T05:46:32.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.924+0000 7f50a5ffb700 1 -- 172.21.15.76:0/2225217322 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f509c003c50 con 0x7f50a8108c90 2024-03-12T05:46:32.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.924+0000 7f50aef67700 1 -- 172.21.15.76:0/2225217322 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a819bb40 con 0x7f50a8108c90 2024-03-12T05:46:32.391 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.924+0000 7f50a5ffb700 1 -- 172.21.15.76:0/2225217322 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f509c024710 con 0x7f50a8108c90 2024-03-12T05:46:32.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.925+0000 7f50aef67700 1 -- 172.21.15.76:0/2225217322 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a819bfe0 con 0x7f50a8108c90 2024-03-12T05:46:32.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.925+0000 7f50a5ffb700 1 -- 172.21.15.76:0/2225217322 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f509c01ab30 con 0x7f50a8108c90 2024-03-12T05:46:32.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.926+0000 7f50aef67700 1 -- 172.21.15.76:0/2225217322 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a81063c0 con 0x7f50a8108c90 2024-03-12T05:46:32.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.926+0000 7f50a5ffb700 1 --2- 172.21.15.76:0/2225217322 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f5098038c60 0x7f509803b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:32.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.926+0000 7f50a5ffb700 1 -- 172.21.15.76:0/2225217322 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f509c04ce40 con 0x7f50a8108c90 2024-03-12T05:46:32.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.926+0000 7f50a7fff700 1 --2- 172.21.15.76:0/2225217322 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f5098038c60 0x7f509803b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:32.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.926+0000 7f50a7fff700 1 --2- 172.21.15.76:0/2225217322 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f5098038c60 0x7f509803b120 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5094006fd0 tx=0x7f5094006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:32.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:30.931+0000 7f50a5ffb700 1 -- 172.21.15.76:0/2225217322 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f509c01a950 con 0x7f50a8108c90 2024-03-12T05:46:32.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:31.148+0000 7f50aef67700 1 -- 172.21.15.76:0/2225217322 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f50a819e8c0 con 0x7f50a8108c90 2024-03-12T05:46:32.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:31.150+0000 7f50a5ffb700 1 -- 172.21.15.76:0/2225217322 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f509c02c350 con 0x7f50a8108c90 2024-03-12T05:46:32.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:31.152+0000 7f50aef67700 1 -- 172.21.15.76:0/2225217322 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f5098038c60 msgr2=0x7f509803b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:32.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:31.152+0000 7f50aef67700 1 --2- 172.21.15.76:0/2225217322 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f5098038c60 0x7f509803b120 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5094006fd0 tx=0x7f5094006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:32.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:31.152+0000 7f50aef67700 1 -- 172.21.15.76:0/2225217322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a8108c90 msgr2=0x7f50a819b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:32.392 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:31.152+0000 7f50aef67700 1 --2- 172.21.15.76:0/2225217322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a8108c90 0x7f50a819b400 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f509c011740 tx=0x7f509c011a50 comp rx=0 tx=0).stop 2024-03-12T05:46:32.393 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:31.152+0000 7f50aef67700 1 -- 172.21.15.76:0/2225217322 shutdown_connections 2024-03-12T05:46:32.393 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:31.152+0000 7f50aef67700 1 --2- 172.21.15.76:0/2225217322 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f5098038c60 0x7f509803b120 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:32.393 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:31.152+0000 7f50aef67700 1 --2- 172.21.15.76:0/2225217322 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a8108c90 0x7f50a819b400 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:32.393 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:31.152+0000 7f50aef67700 1 -- 172.21.15.76:0/2225217322 >> 172.21.15.76:0/2225217322 conn(0x7f50a8101370 msgr2=0x7f50a8101dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:32.393 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:31.152+0000 7f50aef67700 1 -- 172.21.15.76:0/2225217322 shutdown_connections 2024-03-12T05:46:32.393 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:31.152+0000 7f50aef67700 1 -- 172.21.15.76:0/2225217322 wait complete. 2024-03-12T05:46:32.393 INFO:teuthology.orchestra.run.smithi076.stdout:Enabling the dashboard module... 2024-03-12T05:46:34.203 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:33 smithi076 ceph-mon[32095]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:46:34.204 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:33 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2454377823' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-12T05:46:34.204 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:33 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: ignoring --setuser ceph since I am not root 2024-03-12T05:46:34.204 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:33 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: ignoring --setgroup ceph since I am not root 2024-03-12T05:46:34.521 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.788+0000 7f5008016700 1 Processor -- start 2024-03-12T05:46:34.521 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.789+0000 7f5008016700 1 -- start start 2024-03-12T05:46:34.521 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.789+0000 7f5008016700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5000107760 0x7f5000107b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:34.521 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.789+0000 7f5008016700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5000108150 con 0x7f5000107760 2024-03-12T05:46:34.521 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.789+0000 7f5005db2700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5000107760 0x7f5000107b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:34.521 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.789+0000 7f5005db2700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5000107760 0x7f5000107b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50282/0 (socket says 172.21.15.76:50282) 2024-03-12T05:46:34.521 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.789+0000 7f5005db2700 1 -- 172.21.15.76:0/2352238594 learned_addr learned my addr 172.21.15.76:0/2352238594 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:34.521 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.789+0000 7f5005db2700 1 -- 172.21.15.76:0/2352238594 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50001088c0 con 0x7f5000107760 2024-03-12T05:46:34.521 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.790+0000 7f5005db2700 1 --2- 172.21.15.76:0/2352238594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5000107760 0x7f5000107b80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc0098a0 tx=0x7f4ffc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21266819bc5fc92c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:34.521 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.790+0000 7f5004db0700 1 -- 172.21.15.76:0/2352238594 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ffc004160 con 0x7f5000107760 2024-03-12T05:46:34.522 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.790+0000 7f5004db0700 1 -- 172.21.15.76:0/2352238594 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ffc0042c0 con 0x7f5000107760 2024-03-12T05:46:34.522 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.790+0000 7f5004db0700 1 -- 172.21.15.76:0/2352238594 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ffc011a80 con 0x7f5000107760 2024-03-12T05:46:34.522 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.790+0000 7f5008016700 1 -- 172.21.15.76:0/2352238594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5000107760 msgr2=0x7f5000107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:34.522 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.790+0000 7f5008016700 1 --2- 172.21.15.76:0/2352238594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5000107760 0x7f5000107b80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc0098a0 tx=0x7f4ffc009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:34.522 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.790+0000 7f5008016700 1 -- 172.21.15.76:0/2352238594 shutdown_connections 2024-03-12T05:46:34.522 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.790+0000 7f5008016700 1 --2- 172.21.15.76:0/2352238594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5000107760 0x7f5000107b80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:34.522 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.790+0000 7f5008016700 1 -- 172.21.15.76:0/2352238594 >> 172.21.15.76:0/2352238594 conn(0x7f5000102de0 msgr2=0x7f5000105200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:34.522 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.791+0000 7f5008016700 1 -- 172.21.15.76:0/2352238594 shutdown_connections 2024-03-12T05:46:34.522 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.791+0000 7f5008016700 1 -- 172.21.15.76:0/2352238594 wait complete. 2024-03-12T05:46:34.522 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.791+0000 7f5008016700 1 Processor -- start 2024-03-12T05:46:34.522 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.791+0000 7f5008016700 1 -- start start 2024-03-12T05:46:34.522 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.791+0000 7f5008016700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5000107760 0x7f500019b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:34.522 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.791+0000 7f5008016700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500019b950 con 0x7f5000107760 2024-03-12T05:46:34.522 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.791+0000 7f5005db2700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5000107760 0x7f500019b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.791+0000 7f5005db2700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5000107760 0x7f500019b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50296/0 (socket says 172.21.15.76:50296) 2024-03-12T05:46:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.791+0000 7f5005db2700 1 -- 172.21.15.76:0/2454377823 learned_addr learned my addr 172.21.15.76:0/2454377823 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.792+0000 7f5005db2700 1 -- 172.21.15.76:0/2454377823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc009550 con 0x7f5000107760 2024-03-12T05:46:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.792+0000 7f5005db2700 1 --2- 172.21.15.76:0/2454377823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5000107760 0x7f500019b410 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc003f20 tx=0x7f4ffc011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.792+0000 7f4ff6ffd700 1 -- 172.21.15.76:0/2454377823 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ffc0039f0 con 0x7f5000107760 2024-03-12T05:46:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.792+0000 7f4ff6ffd700 1 -- 172.21.15.76:0/2454377823 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ffc003b50 con 0x7f5000107760 2024-03-12T05:46:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.792+0000 7f5008016700 1 -- 172.21.15.76:0/2454377823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500019bb50 con 0x7f5000107760 2024-03-12T05:46:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.792+0000 7f4ff6ffd700 1 -- 172.21.15.76:0/2454377823 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ffc01a690 con 0x7f5000107760 2024-03-12T05:46:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.792+0000 7f5008016700 1 -- 172.21.15.76:0/2454377823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500019bff0 con 0x7f5000107760 2024-03-12T05:46:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.793+0000 7f4ff6ffd700 1 -- 172.21.15.76:0/2454377823 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f4ffc01a900 con 0x7f5000107760 2024-03-12T05:46:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.793+0000 7f5008016700 1 -- 172.21.15.76:0/2454377823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500010baf0 con 0x7f5000107760 2024-03-12T05:46:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.793+0000 7f4ff6ffd700 1 --2- 172.21.15.76:0/2454377823 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f4fec038cb0 0x7f4fec03b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.793+0000 7f4ff6ffd700 1 -- 172.21.15.76:0/2454377823 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4ffc04d080 con 0x7f5000107760 2024-03-12T05:46:34.523 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.793+0000 7f50055b1700 1 --2- 172.21.15.76:0/2454377823 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f4fec038cb0 0x7f4fec03b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.794+0000 7f50055b1700 1 --2- 172.21.15.76:0/2454377823 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f4fec038cb0 0x7f4fec03b170 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0006fd0 tx=0x7f4ff0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:32.798+0000 7f4ff6ffd700 1 -- 172.21.15.76:0/2454377823 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4ffc018a40 con 0x7f5000107760 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:33.048+0000 7f5008016700 1 -- 172.21.15.76:0/2454377823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f500019e8d0 con 0x7f5000107760 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:33.907+0000 7f4ff6ffd700 1 -- 172.21.15.76:0/2454377823 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f4ffc018350 con 0x7f5000107760 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:33.909+0000 7f4ff6ffd700 1 -- 172.21.15.76:0/2454377823 <== mon.0 v2:172.21.15.76:3300/0 8 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f4ffc02d960 con 0x7f5000107760 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:33.910+0000 7f5008016700 1 -- 172.21.15.76:0/2454377823 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f4fec038cb0 msgr2=0x7f4fec03b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:33.910+0000 7f5008016700 1 --2- 172.21.15.76:0/2454377823 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f4fec038cb0 0x7f4fec03b170 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0006fd0 tx=0x7f4ff0006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:33.910+0000 7f5008016700 1 -- 172.21.15.76:0/2454377823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5000107760 msgr2=0x7f500019b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:33.910+0000 7f5008016700 1 --2- 172.21.15.76:0/2454377823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5000107760 0x7f500019b410 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc003f20 tx=0x7f4ffc011b00 comp rx=0 tx=0).stop 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:33.910+0000 7f5008016700 1 -- 172.21.15.76:0/2454377823 shutdown_connections 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:33.911+0000 7f5008016700 1 --2- 172.21.15.76:0/2454377823 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f4fec038cb0 0x7f4fec03b170 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:33.911+0000 7f5008016700 1 --2- 172.21.15.76:0/2454377823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5000107760 0x7f500019b410 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:33.911+0000 7f5008016700 1 -- 172.21.15.76:0/2454377823 >> 172.21.15.76:0/2454377823 conn(0x7f5000102de0 msgr2=0x7f50001038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:33.911+0000 7f5008016700 1 -- 172.21.15.76:0/2454377823 shutdown_connections 2024-03-12T05:46:34.524 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:33.911+0000 7f5008016700 1 -- 172.21.15.76:0/2454377823 wait complete. 2024-03-12T05:46:34.752 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:34 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:34.485+0000 7f9b8030f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:46:34.752 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:34 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:34.592+0000 7f9b8030f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:46:35.155 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:34 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2454377823' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-12T05:46:35.155 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:34 smithi076 ceph-mon[32095]: mgrmap e9: x(active, since 21s) 2024-03-12T05:46:35.485 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:35.253+0000 7f9b8030f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:46:35.485 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:35.347+0000 7f9b8030f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:46:35.785 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:35.489+0000 7f9b8030f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:46:35.786 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:35.587+0000 7f9b8030f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:46:35.786 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:35.670+0000 7f9b8030f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:46:35.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:46:35.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-12T05:46:35.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T05:46:35.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-12T05:46:35.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-12T05:46:35.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:46:35.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.948+0000 7f7a5c430700 1 Processor -- start 2024-03-12T05:46:35.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.948+0000 7f7a5c430700 1 -- start start 2024-03-12T05:46:35.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.948+0000 7f7a5c430700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a54108120 0x7f7a5410a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:35.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.948+0000 7f7a5c430700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a54075600 con 0x7f7a54108120 2024-03-12T05:46:35.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.949+0000 7f7a5a1cc700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a54108120 0x7f7a5410a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:35.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.949+0000 7f7a5a1cc700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a54108120 0x7f7a5410a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50324/0 (socket says 172.21.15.76:50324) 2024-03-12T05:46:35.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.949+0000 7f7a5a1cc700 1 -- 172.21.15.76:0/1436972950 learned_addr learned my addr 172.21.15.76:0/1436972950 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:35.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.949+0000 7f7a5a1cc700 1 -- 172.21.15.76:0/1436972950 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5410aa50 con 0x7f7a54108120 2024-03-12T05:46:35.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.949+0000 7f7a5a1cc700 1 --2- 172.21.15.76:0/1436972950 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a54108120 0x7f7a5410a510 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7a500098a0 tx=0x7f7a50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77d4e8ce96e4949 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:35.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.949+0000 7f7a591ca700 1 -- 172.21.15.76:0/1436972950 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a50004160 con 0x7f7a54108120 2024-03-12T05:46:35.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.949+0000 7f7a591ca700 1 -- 172.21.15.76:0/1436972950 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a500042c0 con 0x7f7a54108120 2024-03-12T05:46:35.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.949+0000 7f7a591ca700 1 -- 172.21.15.76:0/1436972950 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a50011a80 con 0x7f7a54108120 2024-03-12T05:46:35.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.950+0000 7f7a5c430700 1 -- 172.21.15.76:0/1436972950 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a54108120 msgr2=0x7f7a5410a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:35.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.950+0000 7f7a5c430700 1 --2- 172.21.15.76:0/1436972950 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a54108120 0x7f7a5410a510 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7a500098a0 tx=0x7f7a50009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:35.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.950+0000 7f7a5c430700 1 -- 172.21.15.76:0/1436972950 shutdown_connections 2024-03-12T05:46:35.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.950+0000 7f7a5c430700 1 --2- 172.21.15.76:0/1436972950 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a54108120 0x7f7a5410a510 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:35.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.950+0000 7f7a5c430700 1 -- 172.21.15.76:0/1436972950 >> 172.21.15.76:0/1436972950 conn(0x7f7a54101b90 msgr2=0x7f7a54103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:35.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.950+0000 7f7a5c430700 1 -- 172.21.15.76:0/1436972950 shutdown_connections 2024-03-12T05:46:35.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.950+0000 7f7a5c430700 1 -- 172.21.15.76:0/1436972950 wait complete. 2024-03-12T05:46:35.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.951+0000 7f7a5c430700 1 Processor -- start 2024-03-12T05:46:35.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.951+0000 7f7a5c430700 1 -- start start 2024-03-12T05:46:35.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.951+0000 7f7a5c430700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a54108120 0x7f7a5419b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:35.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.951+0000 7f7a5c430700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5419b940 con 0x7f7a54108120 2024-03-12T05:46:35.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.951+0000 7f7a5a1cc700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a54108120 0x7f7a5419b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:35.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.951+0000 7f7a5a1cc700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a54108120 0x7f7a5419b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50330/0 (socket says 172.21.15.76:50330) 2024-03-12T05:46:35.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.951+0000 7f7a5a1cc700 1 -- 172.21.15.76:0/2297484253 learned_addr learned my addr 172.21.15.76:0/2297484253 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:35.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.951+0000 7f7a5a1cc700 1 -- 172.21.15.76:0/2297484253 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a50009550 con 0x7f7a54108120 2024-03-12T05:46:35.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.951+0000 7f7a5a1cc700 1 --2- 172.21.15.76:0/2297484253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a54108120 0x7f7a5419b400 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7a50011740 tx=0x7f7a50011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:35.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.951+0000 7f7a4b7fe700 1 -- 172.21.15.76:0/2297484253 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a50003b20 con 0x7f7a54108120 2024-03-12T05:46:35.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.952+0000 7f7a4b7fe700 1 -- 172.21.15.76:0/2297484253 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a5001a910 con 0x7f7a54108120 2024-03-12T05:46:35.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.952+0000 7f7a5c430700 1 -- 172.21.15.76:0/2297484253 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5419bb40 con 0x7f7a54108120 2024-03-12T05:46:35.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.952+0000 7f7a4b7fe700 1 -- 172.21.15.76:0/2297484253 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a50024900 con 0x7f7a54108120 2024-03-12T05:46:35.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.952+0000 7f7a5c430700 1 -- 172.21.15.76:0/2297484253 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5419bfe0 con 0x7f7a54108120 2024-03-12T05:46:35.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.952+0000 7f7a4b7fe700 1 -- 172.21.15.76:0/2297484253 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f7a50024b40 con 0x7f7a54108120 2024-03-12T05:46:35.797 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.953+0000 7f7a5c430700 1 -- 172.21.15.76:0/2297484253 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a5410b890 con 0x7f7a54108120 2024-03-12T05:46:35.797 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.953+0000 7f7a4b7fe700 1 --2- 172.21.15.76:0/2297484253 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f7a40038d10 0x7f7a4003b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:35.797 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.953+0000 7f7a4b7fe700 1 -- 172.21.15.76:0/2297484253 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7a5004d1d0 con 0x7f7a54108120 2024-03-12T05:46:35.797 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.953+0000 7f7a599cb700 1 -- 172.21.15.76:0/2297484253 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f7a40038d10 msgr2=0x7f7a4003b1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/306877437 2024-03-12T05:46:35.797 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.953+0000 7f7a599cb700 1 --2- 172.21.15.76:0/2297484253 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f7a40038d10 0x7f7a4003b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:46:35.797 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:34.957+0000 7f7a4b7fe700 1 -- 172.21.15.76:0/2297484253 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7a50022720 con 0x7f7a54108120 2024-03-12T05:46:35.797 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:35.154+0000 7f7a599cb700 1 -- 172.21.15.76:0/2297484253 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f7a40038d10 msgr2=0x7f7a4003b1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/306877437 2024-03-12T05:46:35.797 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:35.154+0000 7f7a599cb700 1 --2- 172.21.15.76:0/2297484253 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f7a40038d10 0x7f7a4003b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T05:46:35.797 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:35.217+0000 7f7a5c430700 1 -- 172.21.15.76:0/2297484253 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7a5419ec90 con 0x7f7a54108120 2024-03-12T05:46:35.797 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:35.217+0000 7f7a4b7fe700 1 -- 172.21.15.76:0/2297484253 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f7a50022310 con 0x7f7a54108120 2024-03-12T05:46:35.797 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:35.219+0000 7f7a5c430700 1 -- 172.21.15.76:0/2297484253 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f7a40038d10 msgr2=0x7f7a4003b1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:46:35.797 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:35.219+0000 7f7a5c430700 1 --2- 172.21.15.76:0/2297484253 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f7a40038d10 0x7f7a4003b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:35.797 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:35.220+0000 7f7a5c430700 1 -- 172.21.15.76:0/2297484253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a54108120 msgr2=0x7f7a5419b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:35.798 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:35.220+0000 7f7a5c430700 1 --2- 172.21.15.76:0/2297484253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a54108120 0x7f7a5419b400 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7a50011740 tx=0x7f7a50011a50 comp rx=0 tx=0).stop 2024-03-12T05:46:35.798 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:35.220+0000 7f7a5c430700 1 -- 172.21.15.76:0/2297484253 shutdown_connections 2024-03-12T05:46:35.798 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:35.220+0000 7f7a5c430700 1 --2- 172.21.15.76:0/2297484253 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f7a40038d10 0x7f7a4003b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:35.798 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:35.220+0000 7f7a5c430700 1 --2- 172.21.15.76:0/2297484253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a54108120 0x7f7a5419b400 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:35.798 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:35.220+0000 7f7a5c430700 1 -- 172.21.15.76:0/2297484253 >> 172.21.15.76:0/2297484253 conn(0x7f7a54101b90 msgr2=0x7f7a54102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:35.798 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:35.220+0000 7f7a5c430700 1 -- 172.21.15.76:0/2297484253 shutdown_connections 2024-03-12T05:46:35.798 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:35.220+0000 7f7a5c430700 1 -- 172.21.15.76:0/2297484253 wait complete. 2024-03-12T05:46:35.798 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for the mgr to restart... 2024-03-12T05:46:35.798 INFO:teuthology.orchestra.run.smithi076.stdout:Waiting for mgr epoch 9... 2024-03-12T05:46:36.055 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:35 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2297484253' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T05:46:36.056 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:35.941+0000 7f9b8030f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:46:36.056 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:36 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:36.054+0000 7f9b8030f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:46:36.419 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:36 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:36.142+0000 7f9b8030f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:46:36.420 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:36 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:36.242+0000 7f9b8030f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:46:36.735 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:36 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:36.418+0000 7f9b8030f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:46:37.648 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:37 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:37.382+0000 7f9b8030f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:46:37.648 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:37 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:37.513+0000 7f9b8030f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:46:37.937 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:37 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:37.846+0000 7f9b8030f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:46:37.938 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:37 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:37.936+0000 7f9b8030f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:46:38.235 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:38 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:38.132+0000 7f9b8030f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:46:38.601 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:38 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:38.316+0000 7f9b8030f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:46:38.859 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:38 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:38.600+0000 7f9b8030f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:46:39.235 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:38 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:38.858+0000 7f9b8030f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:46:39.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:39 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:39.092+0000 7f9b8030f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:46:40.235 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:39 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:39.888+0000 7f9b8030f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:46:40.554 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:40.454+0000 7f9b8030f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:46:40.834 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:40.551+0000 7f9b8030f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:46:40.834 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:46:40.766+0000 7f9b8030f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:46:41.132 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-mon[32095]: Active manager daemon x restarted 2024-03-12T05:46:41.132 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-mon[32095]: Activating manager daemon x 2024-03-12T05:46:41.132 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-mon[32095]: osdmap e3: 0 total, 0 up, 0 in 2024-03-12T05:46:41.133 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-mon[32095]: mgrmap e10: x(active, starting, since 0.00353197s) 2024-03-12T05:46:41.133 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:46:41.133 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T05:46:41.133 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T05:46:41.133 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T05:46:41.133 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T05:46:41.133 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-mon[32095]: Manager daemon x is now available 2024-03-12T05:46:41.133 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:46:41.133 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:46:41.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T05:46:41.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T05:46:41.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:41 smithi076 ceph-mon[32095]: mgrmap e11: x(active, since 1.00602s) 2024-03-12T05:46:42.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout { 2024-03-12T05:46:42.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-12T05:46:42.791 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-12T05:46:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout } 2024-03-12T05:46:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.239+0000 7f6c04e33700 1 Processor -- start 2024-03-12T05:46:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.239+0000 7f6c04e33700 1 -- start start 2024-03-12T05:46:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.239+0000 7f6c04e33700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c00108150 0x7f6c0010a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.239+0000 7f6c04e33700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0007ae70 con 0x7f6c00108150 2024-03-12T05:46:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.239+0000 7f6bfe59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c00108150 0x7f6c0010a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.239+0000 7f6bfe59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c00108150 0x7f6c0010a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50340/0 (socket says 172.21.15.76:50340) 2024-03-12T05:46:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.239+0000 7f6bfe59c700 1 -- 172.21.15.76:0/1100663235 learned_addr learned my addr 172.21.15.76:0/1100663235 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.240+0000 7f6bfe59c700 1 -- 172.21.15.76:0/1100663235 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0010aa80 con 0x7f6c00108150 2024-03-12T05:46:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.240+0000 7f6bfe59c700 1 --2- 172.21.15.76:0/1100663235 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c00108150 0x7f6c0010a540 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40098a0 tx=0x7f6bf4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82b1dc84cb43dd67 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.240+0000 7f6bfd59a700 1 -- 172.21.15.76:0/1100663235 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bf4004160 con 0x7f6c00108150 2024-03-12T05:46:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.240+0000 7f6bfd59a700 1 -- 172.21.15.76:0/1100663235 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6bf40042c0 con 0x7f6c00108150 2024-03-12T05:46:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.240+0000 7f6bfd59a700 1 -- 172.21.15.76:0/1100663235 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bf4011a80 con 0x7f6c00108150 2024-03-12T05:46:42.792 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.241+0000 7f6c04e33700 1 -- 172.21.15.76:0/1100663235 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c00108150 msgr2=0x7f6c0010a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:42.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.241+0000 7f6c04e33700 1 --2- 172.21.15.76:0/1100663235 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c00108150 0x7f6c0010a540 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40098a0 tx=0x7f6bf4009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:42.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.241+0000 7f6c04e33700 1 -- 172.21.15.76:0/1100663235 shutdown_connections 2024-03-12T05:46:42.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.241+0000 7f6c04e33700 1 --2- 172.21.15.76:0/1100663235 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c00108150 0x7f6c0010a540 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:42.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.241+0000 7f6c04e33700 1 -- 172.21.15.76:0/1100663235 >> 172.21.15.76:0/1100663235 conn(0x7f6c00101b80 msgr2=0x7f6c00103fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:42.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.241+0000 7f6c04e33700 1 -- 172.21.15.76:0/1100663235 shutdown_connections 2024-03-12T05:46:42.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.241+0000 7f6c04e33700 1 -- 172.21.15.76:0/1100663235 wait complete. 2024-03-12T05:46:42.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.242+0000 7f6c04e33700 1 Processor -- start 2024-03-12T05:46:42.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.242+0000 7f6c04e33700 1 -- start start 2024-03-12T05:46:42.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.242+0000 7f6c04e33700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c00108150 0x7f6c0019b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:42.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.242+0000 7f6c04e33700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0019b960 con 0x7f6c00108150 2024-03-12T05:46:42.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.242+0000 7f6bfe59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c00108150 0x7f6c0019b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:42.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.242+0000 7f6bfe59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c00108150 0x7f6c0019b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50350/0 (socket says 172.21.15.76:50350) 2024-03-12T05:46:42.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.242+0000 7f6bfe59c700 1 -- 172.21.15.76:0/1668102252 learned_addr learned my addr 172.21.15.76:0/1668102252 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:42.793 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.242+0000 7f6bfe59c700 1 -- 172.21.15.76:0/1668102252 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf4009550 con 0x7f6c00108150 2024-03-12T05:46:42.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.242+0000 7f6bfe59c700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c00108150 0x7f6c0019b420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4009130 tx=0x7f6bf4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:42.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.242+0000 7f6bef7fe700 1 -- 172.21.15.76:0/1668102252 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bf4003aa0 con 0x7f6c00108150 2024-03-12T05:46:42.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.242+0000 7f6bef7fe700 1 -- 172.21.15.76:0/1668102252 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6bf4003c00 con 0x7f6c00108150 2024-03-12T05:46:42.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.243+0000 7f6c04e33700 1 -- 172.21.15.76:0/1668102252 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0019bb60 con 0x7f6c00108150 2024-03-12T05:46:42.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.243+0000 7f6c04e33700 1 -- 172.21.15.76:0/1668102252 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0019c000 con 0x7f6c00108150 2024-03-12T05:46:42.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.243+0000 7f6bef7fe700 1 -- 172.21.15.76:0/1668102252 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bf40248a0 con 0x7f6c00108150 2024-03-12T05:46:42.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.243+0000 7f6bef7fe700 1 -- 172.21.15.76:0/1668102252 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f6bf4024b40 con 0x7f6c00108150 2024-03-12T05:46:42.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.244+0000 7f6bef7fe700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f6be8038d00 0x7f6be803b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:42.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.244+0000 7f6bef7fe700 1 -- 172.21.15.76:0/1668102252 --> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6be803b8d0 con 0x7f6be8038d00 2024-03-12T05:46:42.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.244+0000 7f6bef7fe700 1 -- 172.21.15.76:0/1668102252 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6bf404ce40 con 0x7f6c00108150 2024-03-12T05:46:42.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.244+0000 7f6bfdd9b700 1 -- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f6be8038d00 msgr2=0x7f6be803b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/306877437 2024-03-12T05:46:42.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.244+0000 7f6bfdd9b700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f6be8038d00 0x7f6be803b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:46:42.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.445+0000 7f6bfdd9b700 1 -- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f6be8038d00 msgr2=0x7f6be803b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/306877437 2024-03-12T05:46:42.794 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.445+0000 7f6bfdd9b700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f6be8038d00 0x7f6be803b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T05:46:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.846+0000 7f6bfdd9b700 1 -- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f6be8038d00 msgr2=0x7f6be803b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/306877437 2024-03-12T05:46:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:36.846+0000 7f6bfdd9b700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f6be8038d00 0x7f6be803b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-12T05:46:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:37.647+0000 7f6bfdd9b700 1 -- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f6be8038d00 msgr2=0x7f6be803b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/306877437 2024-03-12T05:46:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:37.647+0000 7f6bfdd9b700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f6be8038d00 0x7f6be803b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-12T05:46:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:39.247+0000 7f6bfdd9b700 1 -- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f6be8038d00 msgr2=0x7f6be803b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/306877437 2024-03-12T05:46:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:39.247+0000 7f6bfdd9b700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f6be8038d00 0x7f6be803b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-12T05:46:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:40.782+0000 7f6bef7fe700 1 -- 172.21.15.76:0/1668102252 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mgrmap(e 10) v1 ==== 45549+0+0 (secure 0 0 0) 0x7f6bf402d3f0 con 0x7f6c00108150 2024-03-12T05:46:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:40.782+0000 7f6bef7fe700 1 -- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f6be8038d00 msgr2=0x7f6be803b1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:46:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:40.782+0000 7f6bef7fe700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f6be8038d00 0x7f6be803b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.783+0000 7f6bef7fe700 1 -- 172.21.15.76:0/1668102252 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 11) v1 ==== 45676+0+0 (secure 0 0 0) 0x7f6bf404e380 con 0x7f6c00108150 2024-03-12T05:46:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.784+0000 7f6bef7fe700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6be803c780 0x7f6be803eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.784+0000 7f6bef7fe700 1 -- 172.21.15.76:0/1668102252 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6be803b8d0 con 0x7f6be803c780 2024-03-12T05:46:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.784+0000 7f6bfdd9b700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6be803c780 0x7f6be803eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:42.795 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.784+0000 7f6bfdd9b700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6be803c780 0x7f6be803eb70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0003a10 tx=0x7f6bf00092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.785+0000 7f6bef7fe700 1 -- 172.21.15.76:0/1668102252 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f6be803b8d0 con 0x7f6be803c780 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.792+0000 7f6c04e33700 1 -- 172.21.15.76:0/1668102252 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f6be0002630 con 0x7f6be803c780 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.792+0000 7f6bef7fe700 1 -- 172.21.15.76:0/1668102252 <== mgr.14152 v2:172.21.15.76:6800/1718825251 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f6be0002630 con 0x7f6be803c780 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.793+0000 7f6c04e33700 1 -- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6be803c780 msgr2=0x7f6be803eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.793+0000 7f6c04e33700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6be803c780 0x7f6be803eb70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0003a10 tx=0x7f6bf00092b0 comp rx=0 tx=0).stop 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.793+0000 7f6c04e33700 1 -- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c00108150 msgr2=0x7f6c0019b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.793+0000 7f6c04e33700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c00108150 0x7f6c0019b420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4009130 tx=0x7f6bf4011b00 comp rx=0 tx=0).stop 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.793+0000 7f6c04e33700 1 -- 172.21.15.76:0/1668102252 shutdown_connections 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.793+0000 7f6c04e33700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c00108150 0x7f6c0019b420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.793+0000 7f6c04e33700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6be803c780 0x7f6be803eb70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.793+0000 7f6c04e33700 1 --2- 172.21.15.76:0/1668102252 >> [v2:172.21.15.76:6800/306877437,v1:172.21.15.76:6801/306877437] conn(0x7f6be8038d00 0x7f6be803b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.793+0000 7f6c04e33700 1 -- 172.21.15.76:0/1668102252 >> 172.21.15.76:0/1668102252 conn(0x7f6c00101b80 msgr2=0x7f6c00102630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.793+0000 7f6c04e33700 1 -- 172.21.15.76:0/1668102252 shutdown_connections 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:41.793+0000 7f6c04e33700 1 -- 172.21.15.76:0/1668102252 wait complete. 2024-03-12T05:46:42.796 INFO:teuthology.orchestra.run.smithi076.stdout:mgr epoch 9 is available 2024-03-12T05:46:42.797 INFO:teuthology.orchestra.run.smithi076.stdout:Generating a dashboard self-signed certificate... 2024-03-12T05:46:43.044 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:42 smithi076 ceph-mon[32095]: [12/Mar/2024:05:46:41] ENGINE Bus STARTING 2024-03-12T05:46:43.044 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:42 smithi076 ceph-mon[32095]: [12/Mar/2024:05:46:41] ENGINE Serving on http://172.21.15.76:8765 2024-03-12T05:46:43.044 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:42 smithi076 ceph-mon[32095]: [12/Mar/2024:05:46:41] ENGINE Serving on https://172.21.15.76:7150 2024-03-12T05:46:43.044 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:42 smithi076 ceph-mon[32095]: [12/Mar/2024:05:46:41] ENGINE Bus STARTED 2024-03-12T05:46:43.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:42 smithi076 ceph-mon[32095]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T05:46:43.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:42 smithi076 ceph-mon[32095]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T05:46:44.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:43 smithi076 ceph-mon[32095]: mgrmap e12: x(active, since 2s) 2024-03-12T05:46:44.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:46:44.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:46:44.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:46:44.172 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-12T05:46:44.172 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.262+0000 7f8e94056700 1 Processor -- start 2024-03-12T05:46:44.172 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.263+0000 7f8e94056700 1 -- start start 2024-03-12T05:46:44.172 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.263+0000 7f8e94056700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e8c107b60 0x7f8e8c107f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:44.173 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.263+0000 7f8e94056700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c108550 con 0x7f8e8c107b60 2024-03-12T05:46:44.173 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.263+0000 7f8e91df2700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e8c107b60 0x7f8e8c107f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:44.173 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.263+0000 7f8e91df2700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e8c107b60 0x7f8e8c107f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45800/0 (socket says 172.21.15.76:45800) 2024-03-12T05:46:44.173 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.263+0000 7f8e91df2700 1 -- 172.21.15.76:0/3907296818 learned_addr learned my addr 172.21.15.76:0/3907296818 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:44.173 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.263+0000 7f8e91df2700 1 -- 172.21.15.76:0/3907296818 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c1067c0 con 0x7f8e8c107b60 2024-03-12T05:46:44.173 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.263+0000 7f8e91df2700 1 --2- 172.21.15.76:0/3907296818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e8c107b60 0x7f8e8c107f80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8e880098a0 tx=0x7f8e88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192c4cecf22bdefb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:44.173 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.264+0000 7f8e90df0700 1 -- 172.21.15.76:0/3907296818 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e88004160 con 0x7f8e8c107b60 2024-03-12T05:46:44.173 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.264+0000 7f8e90df0700 1 -- 172.21.15.76:0/3907296818 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e880042c0 con 0x7f8e8c107b60 2024-03-12T05:46:44.173 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.264+0000 7f8e90df0700 1 -- 172.21.15.76:0/3907296818 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e88011a80 con 0x7f8e8c107b60 2024-03-12T05:46:44.173 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.264+0000 7f8e94056700 1 -- 172.21.15.76:0/3907296818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e8c107b60 msgr2=0x7f8e8c107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:44.173 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.264+0000 7f8e94056700 1 --2- 172.21.15.76:0/3907296818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e8c107b60 0x7f8e8c107f80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8e880098a0 tx=0x7f8e88009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:44.173 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.264+0000 7f8e94056700 1 -- 172.21.15.76:0/3907296818 shutdown_connections 2024-03-12T05:46:44.173 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.264+0000 7f8e94056700 1 --2- 172.21.15.76:0/3907296818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e8c107b60 0x7f8e8c107f80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:44.174 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.264+0000 7f8e94056700 1 -- 172.21.15.76:0/3907296818 >> 172.21.15.76:0/3907296818 conn(0x7f8e8c1012d0 msgr2=0x7f8e8c103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:44.174 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.264+0000 7f8e94056700 1 -- 172.21.15.76:0/3907296818 shutdown_connections 2024-03-12T05:46:44.174 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.265+0000 7f8e94056700 1 -- 172.21.15.76:0/3907296818 wait complete. 2024-03-12T05:46:44.174 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.265+0000 7f8e94056700 1 Processor -- start 2024-03-12T05:46:44.174 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.265+0000 7f8e94056700 1 -- start start 2024-03-12T05:46:44.174 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.265+0000 7f8e94056700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e8c107b60 0x7f8e8c19d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:44.174 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.265+0000 7f8e94056700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c19db30 con 0x7f8e8c107b60 2024-03-12T05:46:44.174 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.265+0000 7f8e91df2700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e8c107b60 0x7f8e8c19d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:44.174 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.265+0000 7f8e91df2700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e8c107b60 0x7f8e8c19d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45814/0 (socket says 172.21.15.76:45814) 2024-03-12T05:46:44.174 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.265+0000 7f8e91df2700 1 -- 172.21.15.76:0/1594374048 learned_addr learned my addr 172.21.15.76:0/1594374048 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:44.174 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.266+0000 7f8e91df2700 1 -- 172.21.15.76:0/1594374048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e88009550 con 0x7f8e8c107b60 2024-03-12T05:46:44.174 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.266+0000 7f8e91df2700 1 --2- 172.21.15.76:0/1594374048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e8c107b60 0x7f8e8c19d5f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8e88003f20 tx=0x7f8e88011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:44.174 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.266+0000 7f8e82ffd700 1 -- 172.21.15.76:0/1594374048 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e880039f0 con 0x7f8e8c107b60 2024-03-12T05:46:44.175 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.266+0000 7f8e82ffd700 1 -- 172.21.15.76:0/1594374048 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e88003b50 con 0x7f8e8c107b60 2024-03-12T05:46:44.175 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.266+0000 7f8e94056700 1 -- 172.21.15.76:0/1594374048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8c19dd30 con 0x7f8e8c107b60 2024-03-12T05:46:44.175 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.266+0000 7f8e82ffd700 1 -- 172.21.15.76:0/1594374048 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e8801a6d0 con 0x7f8e8c107b60 2024-03-12T05:46:44.175 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.266+0000 7f8e94056700 1 -- 172.21.15.76:0/1594374048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8c19e1d0 con 0x7f8e8c107b60 2024-03-12T05:46:44.175 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.267+0000 7f8e82ffd700 1 -- 172.21.15.76:0/1594374048 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f8e8801aa10 con 0x7f8e8c107b60 2024-03-12T05:46:44.175 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.267+0000 7f8e82ffd700 1 --2- 172.21.15.76:0/1594374048 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8e78038cb0 0x7f8e7803b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:44.175 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.267+0000 7f8e82ffd700 1 -- 172.21.15.76:0/1594374048 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8e8804d1b0 con 0x7f8e8c107b60 2024-03-12T05:46:44.175 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.267+0000 7f8e915f1700 1 --2- 172.21.15.76:0/1594374048 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8e78038cb0 0x7f8e7803b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:44.175 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.268+0000 7f8e94056700 1 -- 172.21.15.76:0/1594374048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8c068d50 con 0x7f8e8c107b60 2024-03-12T05:46:44.175 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.268+0000 7f8e915f1700 1 --2- 172.21.15.76:0/1594374048 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8e78038cb0 0x7f8e7803b170 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c006fd0 tx=0x7f8e7c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:44.175 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.272+0000 7f8e82ffd700 1 -- 172.21.15.76:0/1594374048 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8e8801ad10 con 0x7f8e8c107b60 2024-03-12T05:46:44.175 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.477+0000 7f8e94056700 1 -- 172.21.15.76:0/1594374048 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e8c0620b0 con 0x7f8e78038cb0 2024-03-12T05:46:44.175 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.625+0000 7f8e82ffd700 1 -- 172.21.15.76:0/1594374048 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f8e8c0620b0 con 0x7f8e78038cb0 2024-03-12T05:46:44.175 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.628+0000 7f8e94056700 1 -- 172.21.15.76:0/1594374048 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8e78038cb0 msgr2=0x7f8e7803b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:44.176 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.628+0000 7f8e94056700 1 --2- 172.21.15.76:0/1594374048 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8e78038cb0 0x7f8e7803b170 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c006fd0 tx=0x7f8e7c006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:44.176 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.628+0000 7f8e94056700 1 -- 172.21.15.76:0/1594374048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e8c107b60 msgr2=0x7f8e8c19d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:44.176 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.628+0000 7f8e94056700 1 --2- 172.21.15.76:0/1594374048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e8c107b60 0x7f8e8c19d5f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8e88003f20 tx=0x7f8e88011b00 comp rx=0 tx=0).stop 2024-03-12T05:46:44.176 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.628+0000 7f8e94056700 1 -- 172.21.15.76:0/1594374048 shutdown_connections 2024-03-12T05:46:44.176 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.628+0000 7f8e94056700 1 --2- 172.21.15.76:0/1594374048 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8e78038cb0 0x7f8e7803b170 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:44.176 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.628+0000 7f8e94056700 1 --2- 172.21.15.76:0/1594374048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e8c107b60 0x7f8e8c19d5f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:44.176 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.628+0000 7f8e94056700 1 -- 172.21.15.76:0/1594374048 >> 172.21.15.76:0/1594374048 conn(0x7f8e8c1012d0 msgr2=0x7f8e8c18f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:44.176 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.628+0000 7f8e94056700 1 -- 172.21.15.76:0/1594374048 shutdown_connections 2024-03-12T05:46:44.176 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:43.628+0000 7f8e94056700 1 -- 172.21.15.76:0/1594374048 wait complete. 2024-03-12T05:46:44.176 INFO:teuthology.orchestra.run.smithi076.stdout:Creating initial admin user... 2024-03-12T05:46:45.094 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:44 smithi076 ceph-mon[32095]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:46:45.094 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:46:45.819 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$gv5hveFua3Xr1o5fFsw4ouoxnISDIDlzTGek/.2haVSFNf/ChRDr2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710222405, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-12T05:46:45.819 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.623+0000 7fb976e41700 1 Processor -- start 2024-03-12T05:46:45.819 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.623+0000 7fb976e41700 1 -- start start 2024-03-12T05:46:45.819 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.623+0000 7fb976e41700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb970107680 0x7fb970107aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:45.819 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.623+0000 7fb976e41700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb970108070 con 0x7fb970107680 2024-03-12T05:46:45.819 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.623+0000 7fb974bdd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb970107680 0x7fb970107aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:45.819 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.623+0000 7fb974bdd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb970107680 0x7fb970107aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45824/0 (socket says 172.21.15.76:45824) 2024-03-12T05:46:45.819 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.623+0000 7fb974bdd700 1 -- 172.21.15.76:0/1352225917 learned_addr learned my addr 172.21.15.76:0/1352225917 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:45.819 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.623+0000 7fb974bdd700 1 -- 172.21.15.76:0/1352225917 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb970108830 con 0x7fb970107680 2024-03-12T05:46:45.820 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.623+0000 7fb974bdd700 1 --2- 172.21.15.76:0/1352225917 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb970107680 0x7fb970107aa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb9640098a0 tx=0x7fb964009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1062a162b518606e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:45.820 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.624+0000 7fb96f7fe700 1 -- 172.21.15.76:0/1352225917 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb964004160 con 0x7fb970107680 2024-03-12T05:46:45.820 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.624+0000 7fb96f7fe700 1 -- 172.21.15.76:0/1352225917 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb9640042c0 con 0x7fb970107680 2024-03-12T05:46:45.820 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.624+0000 7fb96f7fe700 1 -- 172.21.15.76:0/1352225917 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb964011a80 con 0x7fb970107680 2024-03-12T05:46:45.820 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.624+0000 7fb976e41700 1 -- 172.21.15.76:0/1352225917 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb970107680 msgr2=0x7fb970107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:45.820 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.624+0000 7fb976e41700 1 --2- 172.21.15.76:0/1352225917 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb970107680 0x7fb970107aa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb9640098a0 tx=0x7fb964009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:45.820 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.624+0000 7fb976e41700 1 -- 172.21.15.76:0/1352225917 shutdown_connections 2024-03-12T05:46:45.820 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.624+0000 7fb976e41700 1 --2- 172.21.15.76:0/1352225917 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb970107680 0x7fb970107aa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:45.820 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.624+0000 7fb976e41700 1 -- 172.21.15.76:0/1352225917 >> 172.21.15.76:0/1352225917 conn(0x7fb970102c80 msgr2=0x7fb9701050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:45.820 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.624+0000 7fb976e41700 1 -- 172.21.15.76:0/1352225917 shutdown_connections 2024-03-12T05:46:45.820 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.624+0000 7fb976e41700 1 -- 172.21.15.76:0/1352225917 wait complete. 2024-03-12T05:46:45.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.625+0000 7fb976e41700 1 Processor -- start 2024-03-12T05:46:45.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.625+0000 7fb976e41700 1 -- start start 2024-03-12T05:46:45.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.625+0000 7fb976e41700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb970107680 0x7fb97019b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:45.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.625+0000 7fb976e41700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97019b920 con 0x7fb970107680 2024-03-12T05:46:45.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.625+0000 7fb974bdd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb970107680 0x7fb97019b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:45.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.625+0000 7fb974bdd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb970107680 0x7fb97019b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45826/0 (socket says 172.21.15.76:45826) 2024-03-12T05:46:45.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.625+0000 7fb974bdd700 1 -- 172.21.15.76:0/2494575696 learned_addr learned my addr 172.21.15.76:0/2494575696 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:45.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.625+0000 7fb974bdd700 1 -- 172.21.15.76:0/2494575696 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb964009550 con 0x7fb970107680 2024-03-12T05:46:45.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.626+0000 7fb974bdd700 1 --2- 172.21.15.76:0/2494575696 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb970107680 0x7fb97019b3e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb964011740 tx=0x7fb964011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:45.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.626+0000 7fb96dffb700 1 -- 172.21.15.76:0/2494575696 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9640039e0 con 0x7fb970107680 2024-03-12T05:46:45.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.626+0000 7fb96dffb700 1 -- 172.21.15.76:0/2494575696 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb964003b40 con 0x7fb970107680 2024-03-12T05:46:45.821 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.626+0000 7fb976e41700 1 -- 172.21.15.76:0/2494575696 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb97019bb20 con 0x7fb970107680 2024-03-12T05:46:45.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.626+0000 7fb96dffb700 1 -- 172.21.15.76:0/2494575696 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb964024740 con 0x7fb970107680 2024-03-12T05:46:45.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.626+0000 7fb976e41700 1 -- 172.21.15.76:0/2494575696 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97019bfc0 con 0x7fb970107680 2024-03-12T05:46:45.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.627+0000 7fb96dffb700 1 -- 172.21.15.76:0/2494575696 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7fb96402d3f0 con 0x7fb970107680 2024-03-12T05:46:45.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.627+0000 7fb96dffb700 1 --2- 172.21.15.76:0/2494575696 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb960038d60 0x7fb96003b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:45.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.627+0000 7fb96dffb700 1 -- 172.21.15.76:0/2494575696 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb96404cfe0 con 0x7fb970107680 2024-03-12T05:46:45.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.627+0000 7fb96ffff700 1 --2- 172.21.15.76:0/2494575696 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb960038d60 0x7fb96003b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:45.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.627+0000 7fb976e41700 1 -- 172.21.15.76:0/2494575696 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb97010ba60 con 0x7fb970107680 2024-03-12T05:46:45.822 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.628+0000 7fb96ffff700 1 --2- 172.21.15.76:0/2494575696 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb960038d60 0x7fb96003b220 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb95c006fd0 tx=0x7fb95c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:45.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.632+0000 7fb96dffb700 1 -- 172.21.15.76:0/2494575696 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb96402d740 con 0x7fb970107680 2024-03-12T05:46:45.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:44.852+0000 7fb976e41700 1 -- 172.21.15.76:0/2494575696 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb9700630c0 con 0x7fb960038d60 2024-03-12T05:46:45.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:45.078+0000 7fb96dffb700 1 -- 172.21.15.76:0/2494575696 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fb9700630c0 con 0x7fb960038d60 2024-03-12T05:46:45.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:45.081+0000 7fb976e41700 1 -- 172.21.15.76:0/2494575696 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb960038d60 msgr2=0x7fb96003b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:45.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:45.081+0000 7fb976e41700 1 --2- 172.21.15.76:0/2494575696 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb960038d60 0x7fb96003b220 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb95c006fd0 tx=0x7fb95c006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:45.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:45.081+0000 7fb976e41700 1 -- 172.21.15.76:0/2494575696 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb970107680 msgr2=0x7fb97019b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:45.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:45.081+0000 7fb976e41700 1 --2- 172.21.15.76:0/2494575696 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb970107680 0x7fb97019b3e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb964011740 tx=0x7fb964011b00 comp rx=0 tx=0).stop 2024-03-12T05:46:45.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:45.081+0000 7fb976e41700 1 -- 172.21.15.76:0/2494575696 shutdown_connections 2024-03-12T05:46:45.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:45.081+0000 7fb976e41700 1 --2- 172.21.15.76:0/2494575696 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb960038d60 0x7fb96003b220 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:45.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:45.081+0000 7fb976e41700 1 --2- 172.21.15.76:0/2494575696 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb970107680 0x7fb97019b3e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:45.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:45.081+0000 7fb976e41700 1 -- 172.21.15.76:0/2494575696 >> 172.21.15.76:0/2494575696 conn(0x7fb970102c80 msgr2=0x7fb970103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:45.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:45.081+0000 7fb976e41700 1 -- 172.21.15.76:0/2494575696 shutdown_connections 2024-03-12T05:46:45.823 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:45.081+0000 7fb976e41700 1 -- 172.21.15.76:0/2494575696 wait complete. 2024-03-12T05:46:45.823 INFO:teuthology.orchestra.run.smithi076.stdout:Fetching dashboard port number... 2024-03-12T05:46:46.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:45 smithi076 ceph-mon[32095]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:46:46.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:45 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:46:47.630 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stdout 8443 2024-03-12T05:46:47.630 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.701+0000 7f994ce87700 1 Processor -- start 2024-03-12T05:46:47.630 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.701+0000 7f994ce87700 1 -- start start 2024-03-12T05:46:47.630 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.702+0000 7f994ce87700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9948107700 0x7f9948107b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:47.630 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.702+0000 7f994ce87700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99481080f0 con 0x7f9948107700 2024-03-12T05:46:47.630 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.702+0000 7f994659c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9948107700 0x7f9948107b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:47.630 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.702+0000 7f994659c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9948107700 0x7f9948107b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45836/0 (socket says 172.21.15.76:45836) 2024-03-12T05:46:47.630 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.702+0000 7f994659c700 1 -- 172.21.15.76:0/4061497844 learned_addr learned my addr 172.21.15.76:0/4061497844 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:47.630 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.702+0000 7f994659c700 1 -- 172.21.15.76:0/4061497844 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9948108860 con 0x7f9948107700 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.702+0000 7f994659c700 1 --2- 172.21.15.76:0/4061497844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9948107700 0x7f9948107b20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9940009e80 tx=0x7f994000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220d3ea31ee22a58 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.702+0000 7f994559a700 1 -- 172.21.15.76:0/4061497844 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f994000e070 con 0x7f9948107700 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.703+0000 7f994559a700 1 -- 172.21.15.76:0/4061497844 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f994000b8c0 con 0x7f9948107700 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.703+0000 7f994559a700 1 -- 172.21.15.76:0/4061497844 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f99400044a0 con 0x7f9948107700 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.703+0000 7f994ce87700 1 -- 172.21.15.76:0/4061497844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9948107700 msgr2=0x7f9948107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.703+0000 7f994ce87700 1 --2- 172.21.15.76:0/4061497844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9948107700 0x7f9948107b20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9940009e80 tx=0x7f994000b330 comp rx=0 tx=0).stop 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.703+0000 7f994ce87700 1 -- 172.21.15.76:0/4061497844 shutdown_connections 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.703+0000 7f994ce87700 1 --2- 172.21.15.76:0/4061497844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9948107700 0x7f9948107b20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.703+0000 7f994ce87700 1 -- 172.21.15.76:0/4061497844 >> 172.21.15.76:0/4061497844 conn(0x7f9948102d80 msgr2=0x7f99481051a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.703+0000 7f994ce87700 1 -- 172.21.15.76:0/4061497844 shutdown_connections 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.703+0000 7f994ce87700 1 -- 172.21.15.76:0/4061497844 wait complete. 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.704+0000 7f994ce87700 1 Processor -- start 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.704+0000 7f994ce87700 1 -- start start 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.704+0000 7f994ce87700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9948107700 0x7f994819b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:47.631 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.704+0000 7f994ce87700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994819b960 con 0x7f9948107700 2024-03-12T05:46:47.632 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.704+0000 7f994659c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9948107700 0x7f994819b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:47.632 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.704+0000 7f994659c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9948107700 0x7f994819b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45846/0 (socket says 172.21.15.76:45846) 2024-03-12T05:46:47.632 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.704+0000 7f994659c700 1 -- 172.21.15.76:0/2024623827 learned_addr learned my addr 172.21.15.76:0/2024623827 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:47.632 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.705+0000 7f994659c700 1 -- 172.21.15.76:0/2024623827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9940009b30 con 0x7f9948107700 2024-03-12T05:46:47.632 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.705+0000 7f994659c700 1 --2- 172.21.15.76:0/2024623827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9948107700 0x7f994819b420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9940009fd0 tx=0x7f994000bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:47.632 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.705+0000 7f993f7fe700 1 -- 172.21.15.76:0/2024623827 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f994000e040 con 0x7f9948107700 2024-03-12T05:46:47.632 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.705+0000 7f994ce87700 1 -- 172.21.15.76:0/2024623827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994819bb60 con 0x7f9948107700 2024-03-12T05:46:47.632 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.705+0000 7f993f7fe700 1 -- 172.21.15.76:0/2024623827 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9940003d60 con 0x7f9948107700 2024-03-12T05:46:47.632 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.705+0000 7f993f7fe700 1 -- 172.21.15.76:0/2024623827 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f994001c670 con 0x7f9948107700 2024-03-12T05:46:47.632 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.705+0000 7f994ce87700 1 -- 172.21.15.76:0/2024623827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994819c060 con 0x7f9948107700 2024-03-12T05:46:47.632 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.706+0000 7f994ce87700 1 -- 172.21.15.76:0/2024623827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994810ba90 con 0x7f9948107700 2024-03-12T05:46:47.632 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.706+0000 7f993f7fe700 1 -- 172.21.15.76:0/2024623827 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f9940012430 con 0x7f9948107700 2024-03-12T05:46:47.632 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.706+0000 7f993f7fe700 1 --2- 172.21.15.76:0/2024623827 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9934038d00 0x7f993403b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:47.632 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.707+0000 7f993f7fe700 1 -- 172.21.15.76:0/2024623827 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f994004cc90 con 0x7f9948107700 2024-03-12T05:46:47.633 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.707+0000 7f9945d9b700 1 --2- 172.21.15.76:0/2024623827 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9934038d00 0x7f993403b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:47.633 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.707+0000 7f9945d9b700 1 --2- 172.21.15.76:0/2024623827 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9934038d00 0x7f993403b1c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9930006fd0 tx=0x7f9930006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:47.633 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.711+0000 7f993f7fe700 1 -- 172.21.15.76:0/2024623827 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9940011390 con 0x7f9948107700 2024-03-12T05:46:47.633 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.841+0000 7f993f7fe700 1 -- 172.21.15.76:0/2024623827 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f9940012ab0 con 0x7f9948107700 2024-03-12T05:46:47.633 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.908+0000 7f994ce87700 1 -- 172.21.15.76:0/2024623827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f994810bcd0 con 0x7f9948107700 2024-03-12T05:46:47.633 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.908+0000 7f993f7fe700 1 -- 172.21.15.76:0/2024623827 <== mon.0 v2:172.21.15.76:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f9940020020 con 0x7f9948107700 2024-03-12T05:46:47.633 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.910+0000 7f994ce87700 1 -- 172.21.15.76:0/2024623827 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9934038d00 msgr2=0x7f993403b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:47.633 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.910+0000 7f994ce87700 1 --2- 172.21.15.76:0/2024623827 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9934038d00 0x7f993403b1c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9930006fd0 tx=0x7f9930006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:47.633 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.910+0000 7f994ce87700 1 -- 172.21.15.76:0/2024623827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9948107700 msgr2=0x7f994819b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:47.633 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.910+0000 7f994ce87700 1 --2- 172.21.15.76:0/2024623827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9948107700 0x7f994819b420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9940009fd0 tx=0x7f994000bd10 comp rx=0 tx=0).stop 2024-03-12T05:46:47.633 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.910+0000 7f994ce87700 1 -- 172.21.15.76:0/2024623827 shutdown_connections 2024-03-12T05:46:47.633 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.910+0000 7f994ce87700 1 --2- 172.21.15.76:0/2024623827 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9934038d00 0x7f993403b1c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:47.633 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.910+0000 7f994ce87700 1 --2- 172.21.15.76:0/2024623827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9948107700 0x7f994819b420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:47.633 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.910+0000 7f994ce87700 1 -- 172.21.15.76:0/2024623827 >> 172.21.15.76:0/2024623827 conn(0x7f9948102d80 msgr2=0x7f9948103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:47.634 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.910+0000 7f994ce87700 1 -- 172.21.15.76:0/2024623827 shutdown_connections 2024-03-12T05:46:47.634 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:46.910+0000 7f994ce87700 1 -- 172.21.15.76:0/2024623827 wait complete. 2024-03-12T05:46:47.635 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T05:46:47.635 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout disabled 2024-03-12T05:46:47.642 INFO:teuthology.orchestra.run.smithi076.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T05:46:47.643 INFO:teuthology.orchestra.run.smithi076.stdout:systemctl: stdout inactive 2024-03-12T05:46:47.643 INFO:teuthology.orchestra.run.smithi076.stdout:firewalld.service is not enabled 2024-03-12T05:46:47.643 INFO:teuthology.orchestra.run.smithi076.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-12T05:46:47.644 INFO:teuthology.orchestra.run.smithi076.stdout:Ceph Dashboard is now available at: 2024-03-12T05:46:47.644 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:46:47.644 INFO:teuthology.orchestra.run.smithi076.stdout: URL: https://smithi076.front.sepia.ceph.com:8443/ 2024-03-12T05:46:47.644 INFO:teuthology.orchestra.run.smithi076.stdout: User: admin 2024-03-12T05:46:47.644 INFO:teuthology.orchestra.run.smithi076.stdout: Password: k38zactd5z 2024-03-12T05:46:47.644 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:46:47.645 INFO:teuthology.orchestra.run.smithi076.stdout:Saving cluster configuration to /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config directory 2024-03-12T05:46:47.645 INFO:teuthology.orchestra.run.smithi076.stdout:Enabling autotune for osd_memory_target 2024-03-12T05:46:47.873 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:47 smithi076 ceph-mon[32095]: mgrmap e13: x(active, since 6s) 2024-03-12T05:46:47.873 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:47 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2024623827' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-12T05:46:48.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.075+0000 7f9004837700 1 Processor -- start 2024-03-12T05:46:48.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.075+0000 7f9004837700 1 -- start start 2024-03-12T05:46:48.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.075+0000 7f9004837700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9000108150 0x7f900010a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:48.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.075+0000 7f9004837700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900007ae70 con 0x7f9000108150 2024-03-12T05:46:48.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.075+0000 7f8ffe59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9000108150 0x7f900010a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:48.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.075+0000 7f8ffe59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9000108150 0x7f900010a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45848/0 (socket says 172.21.15.76:45848) 2024-03-12T05:46:48.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.075+0000 7f8ffe59c700 1 -- 172.21.15.76:0/264542086 learned_addr learned my addr 172.21.15.76:0/264542086 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:48.754 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.075+0000 7f8ffe59c700 1 -- 172.21.15.76:0/264542086 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f900010aa80 con 0x7f9000108150 2024-03-12T05:46:48.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.076+0000 7f8ffe59c700 1 --2- 172.21.15.76:0/264542086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9000108150 0x7f900010a540 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40098a0 tx=0x7f8ff4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a771a8b0902fcd77 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:48.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.076+0000 7f8ffd59a700 1 -- 172.21.15.76:0/264542086 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ff4004160 con 0x7f9000108150 2024-03-12T05:46:48.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.076+0000 7f8ffd59a700 1 -- 172.21.15.76:0/264542086 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ff40042c0 con 0x7f9000108150 2024-03-12T05:46:48.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.076+0000 7f8ffd59a700 1 -- 172.21.15.76:0/264542086 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ff4011a80 con 0x7f9000108150 2024-03-12T05:46:48.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.076+0000 7f9004837700 1 -- 172.21.15.76:0/264542086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9000108150 msgr2=0x7f900010a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:48.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.076+0000 7f9004837700 1 --2- 172.21.15.76:0/264542086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9000108150 0x7f900010a540 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8ff40098a0 tx=0x7f8ff4009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:48.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.076+0000 7f9004837700 1 -- 172.21.15.76:0/264542086 shutdown_connections 2024-03-12T05:46:48.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.076+0000 7f9004837700 1 --2- 172.21.15.76:0/264542086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9000108150 0x7f900010a540 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:48.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.076+0000 7f9004837700 1 -- 172.21.15.76:0/264542086 >> 172.21.15.76:0/264542086 conn(0x7f9000101ba0 msgr2=0x7f9000103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:48.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.076+0000 7f9004837700 1 -- 172.21.15.76:0/264542086 shutdown_connections 2024-03-12T05:46:48.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.077+0000 7f9004837700 1 -- 172.21.15.76:0/264542086 wait complete. 2024-03-12T05:46:48.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.077+0000 7f9004837700 1 Processor -- start 2024-03-12T05:46:48.755 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.077+0000 7f9004837700 1 -- start start 2024-03-12T05:46:48.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.077+0000 7f9004837700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9000108150 0x7f900019b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:48.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.077+0000 7f9004837700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900019b930 con 0x7f9000108150 2024-03-12T05:46:48.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.077+0000 7f8ffe59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9000108150 0x7f900019b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:48.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.077+0000 7f8ffe59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9000108150 0x7f900019b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45860/0 (socket says 172.21.15.76:45860) 2024-03-12T05:46:48.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.078+0000 7f8ffe59c700 1 -- 172.21.15.76:0/2168529232 learned_addr learned my addr 172.21.15.76:0/2168529232 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:48.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.078+0000 7f8ffe59c700 1 -- 172.21.15.76:0/2168529232 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff4009550 con 0x7f9000108150 2024-03-12T05:46:48.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.078+0000 7f8ffe59c700 1 --2- 172.21.15.76:0/2168529232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9000108150 0x7f900019b3f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4011740 tx=0x7f8ff4011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:48.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.078+0000 7f8fef7fe700 1 -- 172.21.15.76:0/2168529232 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ff4003b20 con 0x7f9000108150 2024-03-12T05:46:48.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.078+0000 7f9004837700 1 -- 172.21.15.76:0/2168529232 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900019bb30 con 0x7f9000108150 2024-03-12T05:46:48.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.078+0000 7f8fef7fe700 1 -- 172.21.15.76:0/2168529232 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ff401a910 con 0x7f9000108150 2024-03-12T05:46:48.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.078+0000 7f8fef7fe700 1 -- 172.21.15.76:0/2168529232 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ff4024900 con 0x7f9000108150 2024-03-12T05:46:48.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.078+0000 7f9004837700 1 -- 172.21.15.76:0/2168529232 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900019bfd0 con 0x7f9000108150 2024-03-12T05:46:48.756 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.079+0000 7f8fef7fe700 1 -- 172.21.15.76:0/2168529232 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f8ff4024b70 con 0x7f9000108150 2024-03-12T05:46:48.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.079+0000 7f8fef7fe700 1 --2- 172.21.15.76:0/2168529232 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8fe8038d50 0x7f8fe803b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:48.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.079+0000 7f9004837700 1 -- 172.21.15.76:0/2168529232 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f900010b8c0 con 0x7f9000108150 2024-03-12T05:46:48.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.079+0000 7f8fef7fe700 1 -- 172.21.15.76:0/2168529232 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8ff404d170 con 0x7f9000108150 2024-03-12T05:46:48.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.079+0000 7f8ffdd9b700 1 --2- 172.21.15.76:0/2168529232 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8fe8038d50 0x7f8fe803b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:48.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.079+0000 7f8ffdd9b700 1 --2- 172.21.15.76:0/2168529232 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8fe8038d50 0x7f8fe803b210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0006fd0 tx=0x7f8ff0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:48.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.084+0000 7f8fef7fe700 1 -- 172.21.15.76:0/2168529232 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8ff402d3f0 con 0x7f9000108150 2024-03-12T05:46:48.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.276+0000 7f9004837700 1 -- 172.21.15.76:0/2168529232 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f900010baa0 con 0x7f9000108150 2024-03-12T05:46:48.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.276+0000 7f8fef7fe700 1 -- 172.21.15.76:0/2168529232 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f8ff40350c0 con 0x7f9000108150 2024-03-12T05:46:48.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.278+0000 7f9004837700 1 -- 172.21.15.76:0/2168529232 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8fe8038d50 msgr2=0x7f8fe803b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:48.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.278+0000 7f9004837700 1 --2- 172.21.15.76:0/2168529232 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8fe8038d50 0x7f8fe803b210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0006fd0 tx=0x7f8ff0006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:48.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.278+0000 7f9004837700 1 -- 172.21.15.76:0/2168529232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9000108150 msgr2=0x7f900019b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:48.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.278+0000 7f9004837700 1 --2- 172.21.15.76:0/2168529232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9000108150 0x7f900019b3f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4011740 tx=0x7f8ff4011a50 comp rx=0 tx=0).stop 2024-03-12T05:46:48.757 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.278+0000 7f9004837700 1 -- 172.21.15.76:0/2168529232 shutdown_connections 2024-03-12T05:46:48.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.278+0000 7f9004837700 1 --2- 172.21.15.76:0/2168529232 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8fe8038d50 0x7f8fe803b210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:48.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.278+0000 7f9004837700 1 --2- 172.21.15.76:0/2168529232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9000108150 0x7f900019b3f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:48.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.278+0000 7f9004837700 1 -- 172.21.15.76:0/2168529232 >> 172.21.15.76:0/2168529232 conn(0x7f9000101ba0 msgr2=0x7f9000102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:48.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.278+0000 7f9004837700 1 -- 172.21.15.76:0/2168529232 shutdown_connections 2024-03-12T05:46:48.758 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:48.278+0000 7f9004837700 1 -- 172.21.15.76:0/2168529232 wait complete. 2024-03-12T05:46:50.112 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.181+0000 7ff21a7c0700 1 Processor -- start 2024-03-12T05:46:50.113 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.181+0000 7ff21a7c0700 1 -- start start 2024-03-12T05:46:50.113 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.181+0000 7ff21a7c0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214107760 0x7ff214107b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:50.113 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.181+0000 7ff21a7c0700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff214108150 con 0x7ff214107760 2024-03-12T05:46:50.113 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.181+0000 7ff213fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214107760 0x7ff214107b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:50.113 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.181+0000 7ff213fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214107760 0x7ff214107b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45870/0 (socket says 172.21.15.76:45870) 2024-03-12T05:46:50.113 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.181+0000 7ff213fff700 1 -- 172.21.15.76:0/3454500891 learned_addr learned my addr 172.21.15.76:0/3454500891 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:50.113 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.181+0000 7ff213fff700 1 -- 172.21.15.76:0/3454500891 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2141088c0 con 0x7ff214107760 2024-03-12T05:46:50.113 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.182+0000 7ff213fff700 1 --2- 172.21.15.76:0/3454500891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214107760 0x7ff214107b80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff2040098a0 tx=0x7ff204009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2989871ae1e48cb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:50.113 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.182+0000 7ff212ffd700 1 -- 172.21.15.76:0/3454500891 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff204004160 con 0x7ff214107760 2024-03-12T05:46:50.113 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.182+0000 7ff212ffd700 1 -- 172.21.15.76:0/3454500891 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff2040042c0 con 0x7ff214107760 2024-03-12T05:46:50.113 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.182+0000 7ff212ffd700 1 -- 172.21.15.76:0/3454500891 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff204011a80 con 0x7ff214107760 2024-03-12T05:46:50.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.182+0000 7ff21a7c0700 1 -- 172.21.15.76:0/3454500891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214107760 msgr2=0x7ff214107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:50.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.182+0000 7ff21a7c0700 1 --2- 172.21.15.76:0/3454500891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214107760 0x7ff214107b80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff2040098a0 tx=0x7ff204009bb0 comp rx=0 tx=0).stop 2024-03-12T05:46:50.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.182+0000 7ff21a7c0700 1 -- 172.21.15.76:0/3454500891 shutdown_connections 2024-03-12T05:46:50.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.182+0000 7ff21a7c0700 1 --2- 172.21.15.76:0/3454500891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214107760 0x7ff214107b80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:50.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.182+0000 7ff21a7c0700 1 -- 172.21.15.76:0/3454500891 >> 172.21.15.76:0/3454500891 conn(0x7ff214102de0 msgr2=0x7ff214105200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:50.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.182+0000 7ff21a7c0700 1 -- 172.21.15.76:0/3454500891 shutdown_connections 2024-03-12T05:46:50.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.182+0000 7ff21a7c0700 1 -- 172.21.15.76:0/3454500891 wait complete. 2024-03-12T05:46:50.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.183+0000 7ff21a7c0700 1 Processor -- start 2024-03-12T05:46:50.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.183+0000 7ff21a7c0700 1 -- start start 2024-03-12T05:46:50.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.183+0000 7ff21a7c0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214107760 0x7ff21419b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:50.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.183+0000 7ff21a7c0700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21419b950 con 0x7ff214107760 2024-03-12T05:46:50.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.183+0000 7ff213fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214107760 0x7ff21419b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:50.114 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.183+0000 7ff213fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214107760 0x7ff21419b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45880/0 (socket says 172.21.15.76:45880) 2024-03-12T05:46:50.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.183+0000 7ff213fff700 1 -- 172.21.15.76:0/1454703720 learned_addr learned my addr 172.21.15.76:0/1454703720 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:50.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.184+0000 7ff213fff700 1 -- 172.21.15.76:0/1454703720 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff204009550 con 0x7ff214107760 2024-03-12T05:46:50.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.184+0000 7ff213fff700 1 --2- 172.21.15.76:0/1454703720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214107760 0x7ff21419b410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff204000c00 tx=0x7ff204011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:50.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.184+0000 7ff2117fa700 1 -- 172.21.15.76:0/1454703720 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff204003aa0 con 0x7ff214107760 2024-03-12T05:46:50.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.184+0000 7ff2117fa700 1 -- 172.21.15.76:0/1454703720 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff204003c00 con 0x7ff214107760 2024-03-12T05:46:50.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.184+0000 7ff21a7c0700 1 -- 172.21.15.76:0/1454703720 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff21419bb50 con 0x7ff214107760 2024-03-12T05:46:50.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.184+0000 7ff2117fa700 1 -- 172.21.15.76:0/1454703720 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff204024930 con 0x7ff214107760 2024-03-12T05:46:50.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.184+0000 7ff21a7c0700 1 -- 172.21.15.76:0/1454703720 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21419bff0 con 0x7ff214107760 2024-03-12T05:46:50.115 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.185+0000 7ff2117fa700 1 -- 172.21.15.76:0/1454703720 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7ff20402d3f0 con 0x7ff214107760 2024-03-12T05:46:50.116 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.185+0000 7ff21a7c0700 1 -- 172.21.15.76:0/1454703720 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21410baf0 con 0x7ff214107760 2024-03-12T05:46:50.116 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.185+0000 7ff2117fa700 1 --2- 172.21.15.76:0/1454703720 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff1fc038e00 0x7ff1fc03b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:50.116 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.185+0000 7ff2117fa700 1 -- 172.21.15.76:0/1454703720 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff20404cfc0 con 0x7ff214107760 2024-03-12T05:46:50.116 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.185+0000 7ff2137fe700 1 --2- 172.21.15.76:0/1454703720 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff1fc038e00 0x7ff1fc03b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:50.116 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.186+0000 7ff2137fe700 1 --2- 172.21.15.76:0/1454703720 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff1fc038e00 0x7ff1fc03b2c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff208006fd0 tx=0x7ff208006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:50.116 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.190+0000 7ff2117fa700 1 -- 172.21.15.76:0/1454703720 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff204010a70 con 0x7ff214107760 2024-03-12T05:46:50.116 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.488+0000 7ff21a7c0700 1 -- 172.21.15.76:0/1454703720 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7ff21419e8d0 con 0x7ff214107760 2024-03-12T05:46:50.116 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.491+0000 7ff2117fa700 1 -- 172.21.15.76:0/1454703720 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7ff204010ed0 con 0x7ff214107760 2024-03-12T05:46:50.116 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.493+0000 7ff21a7c0700 1 -- 172.21.15.76:0/1454703720 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff1fc038e00 msgr2=0x7ff1fc03b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:50.116 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.493+0000 7ff21a7c0700 1 --2- 172.21.15.76:0/1454703720 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff1fc038e00 0x7ff1fc03b2c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff208006fd0 tx=0x7ff208006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:50.116 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.493+0000 7ff21a7c0700 1 -- 172.21.15.76:0/1454703720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214107760 msgr2=0x7ff21419b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:50.116 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.493+0000 7ff21a7c0700 1 --2- 172.21.15.76:0/1454703720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214107760 0x7ff21419b410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff204000c00 tx=0x7ff204011b00 comp rx=0 tx=0).stop 2024-03-12T05:46:50.116 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.493+0000 7ff21a7c0700 1 -- 172.21.15.76:0/1454703720 shutdown_connections 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.493+0000 7ff21a7c0700 1 --2- 172.21.15.76:0/1454703720 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff1fc038e00 0x7ff1fc03b2c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.493+0000 7ff21a7c0700 1 --2- 172.21.15.76:0/1454703720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff214107760 0x7ff21419b410 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.493+0000 7ff21a7c0700 1 -- 172.21.15.76:0/1454703720 >> 172.21.15.76:0/1454703720 conn(0x7ff214102de0 msgr2=0x7ff2141038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.493+0000 7ff21a7c0700 1 -- 172.21.15.76:0/1454703720 shutdown_connections 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr 2024-03-12T05:46:49.493+0000 7ff21a7c0700 1 -- 172.21.15.76:0/1454703720 wait complete. 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout:Or, if you are only running a single cluster on this host: 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:46:50.117 INFO:teuthology.orchestra.run.smithi076.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-12T05:46:50.118 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:46:50.118 INFO:teuthology.orchestra.run.smithi076.stdout: ceph telemetry on 2024-03-12T05:46:50.118 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:46:50.118 INFO:teuthology.orchestra.run.smithi076.stdout:For more information see: 2024-03-12T05:46:50.118 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:46:50.118 INFO:teuthology.orchestra.run.smithi076.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-12T05:46:50.118 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:46:50.118 INFO:teuthology.orchestra.run.smithi076.stdout:Bootstrap complete. 2024-03-12T05:46:50.153 INFO:tasks.cephadm:Fetching config... 2024-03-12T05:46:50.153 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:46:50.153 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-12T05:46:50.169 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-12T05:46:50.169 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:46:50.169 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-12T05:46:50.224 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-12T05:46:50.224 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:46:50.224 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-12T05:46:50.292 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-12T05:46:50.292 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:46:50.292 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-12T05:46:50.347 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-12T05:46:50.347 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtCA10uVSngFKztTlLDxuFH94mYrBGmPQLu8cao8jsx/9xAsLZKO+Ki5J6/I/CDZrGzhRNWJBlAI+DRjjOzVVjGXSlJVJm/Tpopcf/PK6tsEBt2qiDSDcI2mbcqLmjnuBhhqM0lojtNkaVk5YOp8htEU1QgEJDMFzPoey8yklPRsApny7D2xxLQj97V71apQKUrhFqdtERV6CY9nKC+zg0de6bq4AiC03Kf678RZmFM5tqaViGPQAXSwhe6QKC3wWaOE3qVWfWSMI+9CQDISvF1Pj7oYCbNr2xKprVAebl/afW6bhLDsV4Z58/FXX/ZsB3GZgkYdpSSdBfb8dFuXJDRKwWc/X8KjjmfttCXgAflEdkdyY8Uzu81aFJlOu5IEyyEG7eloxlnXJYL9FWK3Dzs7ssYQKh8PI4f0W80RdQzkiDgGvWpMZEHdKPE+DTTkaWmRiAe+9qp45AjXpoAt1d2rl4I/67uSMWKZrNUSDTi9NwSs9k5yWTvYQORSB/3EE= ceph-9f336de8-e033-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T05:46:50.425 INFO:teuthology.orchestra.run.smithi076.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtCA10uVSngFKztTlLDxuFH94mYrBGmPQLu8cao8jsx/9xAsLZKO+Ki5J6/I/CDZrGzhRNWJBlAI+DRjjOzVVjGXSlJVJm/Tpopcf/PK6tsEBt2qiDSDcI2mbcqLmjnuBhhqM0lojtNkaVk5YOp8htEU1QgEJDMFzPoey8yklPRsApny7D2xxLQj97V71apQKUrhFqdtERV6CY9nKC+zg0de6bq4AiC03Kf678RZmFM5tqaViGPQAXSwhe6QKC3wWaOE3qVWfWSMI+9CQDISvF1Pj7oYCbNr2xKprVAebl/afW6bhLDsV4Z58/FXX/ZsB3GZgkYdpSSdBfb8dFuXJDRKwWc/X8KjjmfttCXgAflEdkdyY8Uzu81aFJlOu5IEyyEG7eloxlnXJYL9FWK3Dzs7ssYQKh8PI4f0W80RdQzkiDgGvWpMZEHdKPE+DTTkaWmRiAe+9qp45AjXpoAt1d2rl4I/67uSMWKZrNUSDTi9NwSs9k5yWTvYQORSB/3EE= ceph-9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:46:50.439 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtCA10uVSngFKztTlLDxuFH94mYrBGmPQLu8cao8jsx/9xAsLZKO+Ki5J6/I/CDZrGzhRNWJBlAI+DRjjOzVVjGXSlJVJm/Tpopcf/PK6tsEBt2qiDSDcI2mbcqLmjnuBhhqM0lojtNkaVk5YOp8htEU1QgEJDMFzPoey8yklPRsApny7D2xxLQj97V71apQKUrhFqdtERV6CY9nKC+zg0de6bq4AiC03Kf678RZmFM5tqaViGPQAXSwhe6QKC3wWaOE3qVWfWSMI+9CQDISvF1Pj7oYCbNr2xKprVAebl/afW6bhLDsV4Z58/FXX/ZsB3GZgkYdpSSdBfb8dFuXJDRKwWc/X8KjjmfttCXgAflEdkdyY8Uzu81aFJlOu5IEyyEG7eloxlnXJYL9FWK3Dzs7ssYQKh8PI4f0W80RdQzkiDgGvWpMZEHdKPE+DTTkaWmRiAe+9qp45AjXpoAt1d2rl4I/67uSMWKZrNUSDTi9NwSs9k5yWTvYQORSB/3EE= ceph-9f336de8-e033-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T05:46:50.485 INFO:teuthology.orchestra.run.smithi082.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtCA10uVSngFKztTlLDxuFH94mYrBGmPQLu8cao8jsx/9xAsLZKO+Ki5J6/I/CDZrGzhRNWJBlAI+DRjjOzVVjGXSlJVJm/Tpopcf/PK6tsEBt2qiDSDcI2mbcqLmjnuBhhqM0lojtNkaVk5YOp8htEU1QgEJDMFzPoey8yklPRsApny7D2xxLQj97V71apQKUrhFqdtERV6CY9nKC+zg0de6bq4AiC03Kf678RZmFM5tqaViGPQAXSwhe6QKC3wWaOE3qVWfWSMI+9CQDISvF1Pj7oYCbNr2xKprVAebl/afW6bhLDsV4Z58/FXX/ZsB3GZgkYdpSSdBfb8dFuXJDRKwWc/X8KjjmfttCXgAflEdkdyY8Uzu81aFJlOu5IEyyEG7eloxlnXJYL9FWK3Dzs7ssYQKh8PI4f0W80RdQzkiDgGvWpMZEHdKPE+DTTkaWmRiAe+9qp45AjXpoAt1d2rl4I/67uSMWKZrNUSDTi9NwSs9k5yWTvYQORSB/3EE= ceph-9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:46:50.500 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtCA10uVSngFKztTlLDxuFH94mYrBGmPQLu8cao8jsx/9xAsLZKO+Ki5J6/I/CDZrGzhRNWJBlAI+DRjjOzVVjGXSlJVJm/Tpopcf/PK6tsEBt2qiDSDcI2mbcqLmjnuBhhqM0lojtNkaVk5YOp8htEU1QgEJDMFzPoey8yklPRsApny7D2xxLQj97V71apQKUrhFqdtERV6CY9nKC+zg0de6bq4AiC03Kf678RZmFM5tqaViGPQAXSwhe6QKC3wWaOE3qVWfWSMI+9CQDISvF1Pj7oYCbNr2xKprVAebl/afW6bhLDsV4Z58/FXX/ZsB3GZgkYdpSSdBfb8dFuXJDRKwWc/X8KjjmfttCXgAflEdkdyY8Uzu81aFJlOu5IEyyEG7eloxlnXJYL9FWK3Dzs7ssYQKh8PI4f0W80RdQzkiDgGvWpMZEHdKPE+DTTkaWmRiAe+9qp45AjXpoAt1d2rl4I/67uSMWKZrNUSDTi9NwSs9k5yWTvYQORSB/3EE= ceph-9f336de8-e033-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T05:46:50.546 INFO:teuthology.orchestra.run.smithi088.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtCA10uVSngFKztTlLDxuFH94mYrBGmPQLu8cao8jsx/9xAsLZKO+Ki5J6/I/CDZrGzhRNWJBlAI+DRjjOzVVjGXSlJVJm/Tpopcf/PK6tsEBt2qiDSDcI2mbcqLmjnuBhhqM0lojtNkaVk5YOp8htEU1QgEJDMFzPoey8yklPRsApny7D2xxLQj97V71apQKUrhFqdtERV6CY9nKC+zg0de6bq4AiC03Kf678RZmFM5tqaViGPQAXSwhe6QKC3wWaOE3qVWfWSMI+9CQDISvF1Pj7oYCbNr2xKprVAebl/afW6bhLDsV4Z58/FXX/ZsB3GZgkYdpSSdBfb8dFuXJDRKwWc/X8KjjmfttCXgAflEdkdyY8Uzu81aFJlOu5IEyyEG7eloxlnXJYL9FWK3Dzs7ssYQKh8PI4f0W80RdQzkiDgGvWpMZEHdKPE+DTTkaWmRiAe+9qp45AjXpoAt1d2rl4I/67uSMWKZrNUSDTi9NwSs9k5yWTvYQORSB/3EE= ceph-9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:46:50.561 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-12T05:46:50.691 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:50 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1454703720' entity='client.admin' 2024-03-12T05:46:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:46:53.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.156+0000 7fa8e661d700 1 -- 172.21.15.76:0/3713617593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8e00fff40 msgr2=0x7fa8e0102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:53.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.156+0000 7fa8e661d700 1 --2- 172.21.15.76:0/3713617593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8e00fff40 0x7fa8e0102370 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0009910 tx=0x7fa8d0009c20 comp rx=0 tx=0).stop 2024-03-12T05:46:53.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.156+0000 7fa8e661d700 1 -- 172.21.15.76:0/3713617593 shutdown_connections 2024-03-12T05:46:53.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.156+0000 7fa8e661d700 1 --2- 172.21.15.76:0/3713617593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8e00fff40 0x7fa8e0102370 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:53.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.156+0000 7fa8e661d700 1 -- 172.21.15.76:0/3713617593 >> 172.21.15.76:0/3713617593 conn(0x7fa8e00fbb80 msgr2=0x7fa8e00fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:53.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.156+0000 7fa8e661d700 1 -- 172.21.15.76:0/3713617593 shutdown_connections 2024-03-12T05:46:53.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.156+0000 7fa8e661d700 1 -- 172.21.15.76:0/3713617593 wait complete. 2024-03-12T05:46:53.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.157+0000 7fa8e661d700 1 Processor -- start 2024-03-12T05:46:53.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.157+0000 7fa8e661d700 1 -- start start 2024-03-12T05:46:53.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.157+0000 7fa8e661d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8e00fff40 0x7fa8e019d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:53.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.157+0000 7fa8e661d700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e019d6f0 con 0x7fa8e00fff40 2024-03-12T05:46:53.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.157+0000 7fa8dffff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8e00fff40 0x7fa8e019d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:53.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.157+0000 7fa8dffff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8e00fff40 0x7fa8e019d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55996/0 (socket says 172.21.15.76:55996) 2024-03-12T05:46:53.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.157+0000 7fa8dffff700 1 -- 172.21.15.76:0/1815670553 learned_addr learned my addr 172.21.15.76:0/1815670553 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:53.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.158+0000 7fa8dffff700 1 -- 172.21.15.76:0/1815670553 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d00095f0 con 0x7fa8e00fff40 2024-03-12T05:46:53.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.158+0000 7fa8dffff700 1 --2- 172.21.15.76:0/1815670553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8e00fff40 0x7fa8e019d1b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0005950 tx=0x7fa8d0004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:53.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.158+0000 7fa8dd7fa700 1 -- 172.21.15.76:0/1815670553 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8d001d070 con 0x7fa8e00fff40 2024-03-12T05:46:53.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.158+0000 7fa8dd7fa700 1 -- 172.21.15.76:0/1815670553 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8d000bc30 con 0x7fa8e00fff40 2024-03-12T05:46:53.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.158+0000 7fa8e661d700 1 -- 172.21.15.76:0/1815670553 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e019d8f0 con 0x7fa8e00fff40 2024-03-12T05:46:53.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.158+0000 7fa8dd7fa700 1 -- 172.21.15.76:0/1815670553 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8d0003b20 con 0x7fa8e00fff40 2024-03-12T05:46:53.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.158+0000 7fa8e661d700 1 -- 172.21.15.76:0/1815670553 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e019dd90 con 0x7fa8e00fff40 2024-03-12T05:46:53.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.159+0000 7fa8dd7fa700 1 -- 172.21.15.76:0/1815670553 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fa8d0003c80 con 0x7fa8e00fff40 2024-03-12T05:46:53.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.159+0000 7fa8dd7fa700 1 --2- 172.21.15.76:0/1815670553 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa8c8038db0 0x7fa8c803b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:53.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.159+0000 7fa8dd7fa700 1 -- 172.21.15.76:0/1815670553 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa8d004cd20 con 0x7fa8e00fff40 2024-03-12T05:46:53.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.159+0000 7fa8e661d700 1 -- 172.21.15.76:0/1815670553 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e0197230 con 0x7fa8e00fff40 2024-03-12T05:46:53.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.160+0000 7fa8df7fe700 1 --2- 172.21.15.76:0/1815670553 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa8c8038db0 0x7fa8c803b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:53.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.160+0000 7fa8df7fe700 1 --2- 172.21.15.76:0/1815670553 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa8c8038db0 0x7fa8c803b270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4006fd0 tx=0x7fa8d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:53.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.165+0000 7fa8dd7fa700 1 -- 172.21.15.76:0/1815670553 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa8d0026020 con 0x7fa8e00fff40 2024-03-12T05:46:53.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.376+0000 7fa8e661d700 1 -- 172.21.15.76:0/1815670553 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fa8e0105680 con 0x7fa8e00fff40 2024-03-12T05:46:53.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.378+0000 7fa8dd7fa700 1 -- 172.21.15.76:0/1815670553 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa8d002b440 con 0x7fa8e00fff40 2024-03-12T05:46:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.381+0000 7fa8e661d700 1 -- 172.21.15.76:0/1815670553 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa8c8038db0 msgr2=0x7fa8c803b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.381+0000 7fa8e661d700 1 --2- 172.21.15.76:0/1815670553 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa8c8038db0 0x7fa8c803b270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4006fd0 tx=0x7fa8d4006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.381+0000 7fa8e661d700 1 -- 172.21.15.76:0/1815670553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8e00fff40 msgr2=0x7fa8e019d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.381+0000 7fa8e661d700 1 --2- 172.21.15.76:0/1815670553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8e00fff40 0x7fa8e019d1b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0005950 tx=0x7fa8d0004ff0 comp rx=0 tx=0).stop 2024-03-12T05:46:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.381+0000 7fa8e661d700 1 -- 172.21.15.76:0/1815670553 shutdown_connections 2024-03-12T05:46:53.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.381+0000 7fa8e661d700 1 --2- 172.21.15.76:0/1815670553 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa8c8038db0 0x7fa8c803b270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.381+0000 7fa8e661d700 1 --2- 172.21.15.76:0/1815670553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa8e00fff40 0x7fa8e019d1b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.381+0000 7fa8e661d700 1 -- 172.21.15.76:0/1815670553 >> 172.21.15.76:0/1815670553 conn(0x7fa8e00fbb80 msgr2=0x7fa8e00fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.381+0000 7fa8e661d700 1 -- 172.21.15.76:0/1815670553 shutdown_connections 2024-03-12T05:46:53.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:53.381+0000 7fa8e661d700 1 -- 172.21.15.76:0/1815670553 wait complete. 2024-03-12T05:46:53.978 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-12T05:46:53.978 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-12T05:46:54.584 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:46:54.705 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:54 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1815670553' entity='client.admin' 2024-03-12T05:46:55.765 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:46:55.766 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:46:55.766 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:46:55.766 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:46:55.766 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:46:55.766 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:46:56.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.002+0000 7f6b59487700 1 -- 172.21.15.76:0/1153428864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b541016c0 msgr2=0x7f6b54101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:56.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.002+0000 7f6b59487700 1 --2- 172.21.15.76:0/1153428864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b541016c0 0x7f6b54101ae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009910 tx=0x7f6b44009c20 comp rx=0 tx=0).stop 2024-03-12T05:46:56.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.002+0000 7f6b59487700 1 -- 172.21.15.76:0/1153428864 shutdown_connections 2024-03-12T05:46:56.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.002+0000 7f6b59487700 1 --2- 172.21.15.76:0/1153428864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b541016c0 0x7f6b54101ae0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:56.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.002+0000 7f6b59487700 1 -- 172.21.15.76:0/1153428864 >> 172.21.15.76:0/1153428864 conn(0x7f6b540fcc60 msgr2=0x7f6b540ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:56.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.002+0000 7f6b59487700 1 -- 172.21.15.76:0/1153428864 shutdown_connections 2024-03-12T05:46:56.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.002+0000 7f6b59487700 1 -- 172.21.15.76:0/1153428864 wait complete. 2024-03-12T05:46:56.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.003+0000 7f6b59487700 1 Processor -- start 2024-03-12T05:46:56.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.003+0000 7f6b59487700 1 -- start start 2024-03-12T05:46:56.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.003+0000 7f6b59487700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b541016c0 0x7f6b54196c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:56.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.003+0000 7f6b59487700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b541971a0 con 0x7f6b541016c0 2024-03-12T05:46:56.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.003+0000 7f6b52ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b541016c0 0x7f6b54196c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:56.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.003+0000 7f6b52ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b541016c0 0x7f6b54196c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56016/0 (socket says 172.21.15.76:56016) 2024-03-12T05:46:56.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.003+0000 7f6b52ffd700 1 -- 172.21.15.76:0/1360956068 learned_addr learned my addr 172.21.15.76:0/1360956068 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:56.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.004+0000 7f6b52ffd700 1 -- 172.21.15.76:0/1360956068 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b440095f0 con 0x7f6b541016c0 2024-03-12T05:46:56.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.004+0000 7f6b52ffd700 1 --2- 172.21.15.76:0/1360956068 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b541016c0 0x7f6b54196c60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6b44005850 tx=0x7f6b44004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:56.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.004+0000 7f6b3bfff700 1 -- 172.21.15.76:0/1360956068 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b4401d070 con 0x7f6b541016c0 2024-03-12T05:46:56.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.004+0000 7f6b3bfff700 1 -- 172.21.15.76:0/1360956068 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b4400be40 con 0x7f6b541016c0 2024-03-12T05:46:56.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.004+0000 7f6b3bfff700 1 -- 172.21.15.76:0/1360956068 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b44022ea0 con 0x7f6b541016c0 2024-03-12T05:46:56.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.004+0000 7f6b59487700 1 -- 172.21.15.76:0/1360956068 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b541973a0 con 0x7f6b541016c0 2024-03-12T05:46:56.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.004+0000 7f6b59487700 1 -- 172.21.15.76:0/1360956068 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b54197840 con 0x7f6b541016c0 2024-03-12T05:46:56.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.005+0000 7f6b3bfff700 1 -- 172.21.15.76:0/1360956068 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f6b4400f500 con 0x7f6b541016c0 2024-03-12T05:46:56.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.005+0000 7f6b3bfff700 1 --2- 172.21.15.76:0/1360956068 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b3c038d50 0x7f6b3c03b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:56.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.005+0000 7f6b3bfff700 1 -- 172.21.15.76:0/1360956068 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6b4404ce20 con 0x7f6b541016c0 2024-03-12T05:46:56.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.005+0000 7f6b527fc700 1 --2- 172.21.15.76:0/1360956068 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b3c038d50 0x7f6b3c03b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:56.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.005+0000 7f6b59487700 1 -- 172.21.15.76:0/1360956068 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b54190520 con 0x7f6b541016c0 2024-03-12T05:46:56.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.006+0000 7f6b527fc700 1 --2- 172.21.15.76:0/1360956068 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b3c038d50 0x7f6b3c03b210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6b48006fd0 tx=0x7f6b48006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:56.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.010+0000 7f6b3bfff700 1 -- 172.21.15.76:0/1360956068 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6b44027020 con 0x7f6b541016c0 2024-03-12T05:46:56.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.207+0000 7f6b59487700 1 -- 172.21.15.76:0/1360956068 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b540620b0 con 0x7f6b3c038d50 2024-03-12T05:46:56.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.210+0000 7f6b3bfff700 1 -- 172.21.15.76:0/1360956068 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6b540620b0 con 0x7f6b3c038d50 2024-03-12T05:46:56.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.212+0000 7f6b59487700 1 -- 172.21.15.76:0/1360956068 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b3c038d50 msgr2=0x7f6b3c03b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:56.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.212+0000 7f6b59487700 1 --2- 172.21.15.76:0/1360956068 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b3c038d50 0x7f6b3c03b210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6b48006fd0 tx=0x7f6b48006e40 comp rx=0 tx=0).stop 2024-03-12T05:46:56.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.212+0000 7f6b59487700 1 -- 172.21.15.76:0/1360956068 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b541016c0 msgr2=0x7f6b54196c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:56.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.212+0000 7f6b59487700 1 --2- 172.21.15.76:0/1360956068 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b541016c0 0x7f6b54196c60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6b44005850 tx=0x7f6b44004ff0 comp rx=0 tx=0).stop 2024-03-12T05:46:56.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.212+0000 7f6b59487700 1 -- 172.21.15.76:0/1360956068 shutdown_connections 2024-03-12T05:46:56.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.212+0000 7f6b59487700 1 --2- 172.21.15.76:0/1360956068 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b3c038d50 0x7f6b3c03b210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:56.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.212+0000 7f6b59487700 1 --2- 172.21.15.76:0/1360956068 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b541016c0 0x7f6b54196c60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:56.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.212+0000 7f6b59487700 1 -- 172.21.15.76:0/1360956068 >> 172.21.15.76:0/1360956068 conn(0x7f6b540fcc60 msgr2=0x7f6b540fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:56.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.212+0000 7f6b59487700 1 -- 172.21.15.76:0/1360956068 shutdown_connections 2024-03-12T05:46:56.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:56.212+0000 7f6b59487700 1 -- 172.21.15.76:0/1360956068 wait complete. 2024-03-12T05:46:57.062 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi082 2024-03-12T05:46:57.062 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-12T05:46:57.062 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/etc/ceph/ceph.conf 2024-03-12T05:46:57.081 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-12T05:46:57.081 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:46:57.138 INFO:tasks.cephadm:Adding host smithi082 to orchestrator... 2024-03-12T05:46:57.138 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch host add smithi082 2024-03-12T05:46:57.374 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:46:57.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:57 smithi076 ceph-mon[32095]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:46:57.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:46:57.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:46:57.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:46:57.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:46:57.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:57 smithi076 ceph-mon[32095]: Updating smithi076:/etc/ceph/ceph.conf 2024-03-12T05:46:57.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:57 smithi076 ceph-mon[32095]: Updating smithi076:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:46:57.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:57 smithi076 ceph-mon[32095]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:46:57.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:57 smithi076 ceph-mon[32095]: Updating smithi076:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:46:57.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:46:57.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:46:57.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:46:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:46:58.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.811+0000 7ff48d174700 1 -- 172.21.15.76:0/3664059625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff488106620 msgr2=0x7ff488106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:46:58.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.811+0000 7ff48d174700 1 --2- 172.21.15.76:0/3664059625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff488106620 0x7ff488106a00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff478009910 tx=0x7ff478009c20 comp rx=0 tx=0).stop 2024-03-12T05:46:58.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.811+0000 7ff48d174700 1 -- 172.21.15.76:0/3664059625 shutdown_connections 2024-03-12T05:46:58.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.811+0000 7ff48d174700 1 --2- 172.21.15.76:0/3664059625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff488106620 0x7ff488106a00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:46:58.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.811+0000 7ff48d174700 1 -- 172.21.15.76:0/3664059625 >> 172.21.15.76:0/3664059625 conn(0x7ff4880fc060 msgr2=0x7ff4880fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:46:58.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.812+0000 7ff48d174700 1 -- 172.21.15.76:0/3664059625 shutdown_connections 2024-03-12T05:46:58.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.812+0000 7ff48d174700 1 -- 172.21.15.76:0/3664059625 wait complete. 2024-03-12T05:46:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.812+0000 7ff48d174700 1 Processor -- start 2024-03-12T05:46:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.813+0000 7ff48d174700 1 -- start start 2024-03-12T05:46:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.813+0000 7ff48d174700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff488106620 0x7ff488197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.813+0000 7ff48d174700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48806aa80 con 0x7ff488106620 2024-03-12T05:46:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.813+0000 7ff486d9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff488106620 0x7ff488197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.813+0000 7ff486d9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff488106620 0x7ff488197ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56022/0 (socket says 172.21.15.76:56022) 2024-03-12T05:46:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.813+0000 7ff486d9d700 1 -- 172.21.15.76:0/479150118 learned_addr learned my addr 172.21.15.76:0/479150118 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:46:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.813+0000 7ff486d9d700 1 -- 172.21.15.76:0/479150118 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4780095f0 con 0x7ff488106620 2024-03-12T05:46:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.813+0000 7ff486d9d700 1 --2- 172.21.15.76:0/479150118 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff488106620 0x7ff488197ce0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff478004f40 tx=0x7ff478005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.813+0000 7ff46ffff700 1 -- 172.21.15.76:0/479150118 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff47801d070 con 0x7ff488106620 2024-03-12T05:46:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.813+0000 7ff48d174700 1 -- 172.21.15.76:0/479150118 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4881982e0 con 0x7ff488106620 2024-03-12T05:46:58.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.813+0000 7ff46ffff700 1 -- 172.21.15.76:0/479150118 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff47800bc30 con 0x7ff488106620 2024-03-12T05:46:58.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.813+0000 7ff46ffff700 1 -- 172.21.15.76:0/479150118 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff478003970 con 0x7ff488106620 2024-03-12T05:46:58.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.814+0000 7ff48d174700 1 -- 172.21.15.76:0/479150118 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff488198760 con 0x7ff488106620 2024-03-12T05:46:58.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.814+0000 7ff46ffff700 1 -- 172.21.15.76:0/479150118 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7ff478003c30 con 0x7ff488106620 2024-03-12T05:46:58.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.814+0000 7ff46ffff700 1 --2- 172.21.15.76:0/479150118 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff470038d50 0x7ff47003b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:46:58.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.814+0000 7ff48d174700 1 -- 172.21.15.76:0/479150118 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff488101c90 con 0x7ff488106620 2024-03-12T05:46:58.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.815+0000 7ff46ffff700 1 -- 172.21.15.76:0/479150118 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff47804cad0 con 0x7ff488106620 2024-03-12T05:46:58.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.815+0000 7ff48659c700 1 --2- 172.21.15.76:0/479150118 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff470038d50 0x7ff47003b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:46:58.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.815+0000 7ff48659c700 1 --2- 172.21.15.76:0/479150118 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff470038d50 0x7ff47003b210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff47c006fd0 tx=0x7ff47c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:46:58.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:58.819+0000 7ff46ffff700 1 -- 172.21.15.76:0/479150118 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff478029910 con 0x7ff488106620 2024-03-12T05:46:59.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:46:59.017+0000 7ff48d174700 1 -- 172.21.15.76:0/479150118 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}) v1 -- 0x7ff488198b20 con 0x7ff470038d50 2024-03-12T05:47:00.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:00 smithi076 ceph-mon[32095]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:47:00.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:00 smithi076 ceph-mon[32095]: Deploying cephadm binary to smithi082 2024-03-12T05:47:01.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:01 smithi076 ceph-mon[32095]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:03.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:03.122+0000 7ff46ffff700 1 -- 172.21.15.76:0/479150118 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7ff488198b20 con 0x7ff470038d50 2024-03-12T05:47:03.124 INFO:teuthology.orchestra.run.smithi076.stdout:Added host 'smithi082' with addr '172.21.15.82' 2024-03-12T05:47:03.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:03.124+0000 7ff48d174700 1 -- 172.21.15.76:0/479150118 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff470038d50 msgr2=0x7ff47003b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:03.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:03.124+0000 7ff48d174700 1 --2- 172.21.15.76:0/479150118 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff470038d50 0x7ff47003b210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff47c006fd0 tx=0x7ff47c006e40 comp rx=0 tx=0).stop 2024-03-12T05:47:03.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:03.124+0000 7ff48d174700 1 -- 172.21.15.76:0/479150118 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff488106620 msgr2=0x7ff488197ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:03.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:03.124+0000 7ff48d174700 1 --2- 172.21.15.76:0/479150118 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff488106620 0x7ff488197ce0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff478004f40 tx=0x7ff478005020 comp rx=0 tx=0).stop 2024-03-12T05:47:03.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:03.124+0000 7ff48d174700 1 -- 172.21.15.76:0/479150118 shutdown_connections 2024-03-12T05:47:03.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:03.124+0000 7ff48d174700 1 --2- 172.21.15.76:0/479150118 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff470038d50 0x7ff47003b210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:03.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:03.124+0000 7ff48d174700 1 --2- 172.21.15.76:0/479150118 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff488106620 0x7ff488197ce0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:03.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:03.124+0000 7ff48d174700 1 -- 172.21.15.76:0/479150118 >> 172.21.15.76:0/479150118 conn(0x7ff4880fc060 msgr2=0x7ff4880fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:03.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:03.125+0000 7ff48d174700 1 -- 172.21.15.76:0/479150118 shutdown_connections 2024-03-12T05:47:03.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:03.125+0000 7ff48d174700 1 -- 172.21.15.76:0/479150118 wait complete. 2024-03-12T05:47:03.670 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-12T05:47:03.905 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:47:03.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:03 smithi076 ceph-mon[32095]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:03.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:03.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:03.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:04.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:04 smithi076 ceph-mon[32095]: Added host smithi082 2024-03-12T05:47:04.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:04 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:05.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.449+0000 7f5684189700 1 -- 172.21.15.76:0/403443128 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f567c106610 msgr2=0x7f567c1069f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:05.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.449+0000 7f5684189700 1 --2- 172.21.15.76:0/403443128 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f567c106610 0x7f567c1069f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f566c009910 tx=0x7f566c009c20 comp rx=0 tx=0).stop 2024-03-12T05:47:05.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.449+0000 7f5684189700 1 -- 172.21.15.76:0/403443128 shutdown_connections 2024-03-12T05:47:05.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.450+0000 7f5684189700 1 --2- 172.21.15.76:0/403443128 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f567c106610 0x7f567c1069f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:05.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.450+0000 7f5684189700 1 -- 172.21.15.76:0/403443128 >> 172.21.15.76:0/403443128 conn(0x7f567c0fc010 msgr2=0x7f567c0fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:05.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.450+0000 7f5684189700 1 -- 172.21.15.76:0/403443128 shutdown_connections 2024-03-12T05:47:05.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.450+0000 7f5684189700 1 -- 172.21.15.76:0/403443128 wait complete. 2024-03-12T05:47:05.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.450+0000 7f5684189700 1 Processor -- start 2024-03-12T05:47:05.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.450+0000 7f5684189700 1 -- start start 2024-03-12T05:47:05.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.450+0000 7f5684189700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f567c106610 0x7f567c197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:05.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.450+0000 7f5684189700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c198190 con 0x7f567c106610 2024-03-12T05:47:05.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.451+0000 7f5681f25700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f567c106610 0x7f567c197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:05.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.451+0000 7f5681f25700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f567c106610 0x7f567c197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56840/0 (socket says 172.21.15.76:56840) 2024-03-12T05:47:05.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.451+0000 7f5681f25700 1 -- 172.21.15.76:0/641545403 learned_addr learned my addr 172.21.15.76:0/641545403 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:47:05.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.451+0000 7f5681f25700 1 -- 172.21.15.76:0/641545403 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c0095f0 con 0x7f567c106610 2024-03-12T05:47:05.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.451+0000 7f5681f25700 1 --2- 172.21.15.76:0/641545403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f567c106610 0x7f567c197c50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f566c004f40 tx=0x7f566c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:05.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.451+0000 7f5672ffd700 1 -- 172.21.15.76:0/641545403 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f566c01d070 con 0x7f567c106610 2024-03-12T05:47:05.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.451+0000 7f5684189700 1 -- 172.21.15.76:0/641545403 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567c1983f0 con 0x7f567c106610 2024-03-12T05:47:05.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.451+0000 7f5672ffd700 1 -- 172.21.15.76:0/641545403 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f566c00bc30 con 0x7f567c106610 2024-03-12T05:47:05.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.451+0000 7f5672ffd700 1 -- 172.21.15.76:0/641545403 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f566c003970 con 0x7f567c106610 2024-03-12T05:47:05.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.451+0000 7f5684189700 1 -- 172.21.15.76:0/641545403 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567c198870 con 0x7f567c106610 2024-03-12T05:47:05.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.452+0000 7f5672ffd700 1 -- 172.21.15.76:0/641545403 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f566c003be0 con 0x7f567c106610 2024-03-12T05:47:05.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.452+0000 7f5684189700 1 -- 172.21.15.76:0/641545403 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567c101c80 con 0x7f567c106610 2024-03-12T05:47:05.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.452+0000 7f5672ffd700 1 --2- 172.21.15.76:0/641545403 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5668038db0 0x7f566803b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:05.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.452+0000 7f5672ffd700 1 -- 172.21.15.76:0/641545403 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f566c04ca60 con 0x7f567c106610 2024-03-12T05:47:05.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.452+0000 7f5681724700 1 --2- 172.21.15.76:0/641545403 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5668038db0 0x7f566803b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:05.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.453+0000 7f5681724700 1 --2- 172.21.15.76:0/641545403 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5668038db0 0x7f566803b270 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5678006fd0 tx=0x7f5678006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:05.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.457+0000 7f5672ffd700 1 -- 172.21.15.76:0/641545403 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f566c029950 con 0x7f567c106610 2024-03-12T05:47:05.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.657+0000 7f5684189700 1 -- 172.21.15.76:0/641545403 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f567c06a1d0 con 0x7f5668038db0 2024-03-12T05:47:05.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.658+0000 7f5672ffd700 1 -- 172.21.15.76:0/641545403 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f567c06a1d0 con 0x7f5668038db0 2024-03-12T05:47:05.659 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:47:05.659 INFO:teuthology.orchestra.run.smithi076.stdout:[{"addr": "172.21.15.76", "hostname": "smithi076", "labels": [], "status": ""}, {"addr": "172.21.15.82", "hostname": "smithi082", "labels": [], "status": ""}] 2024-03-12T05:47:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.661+0000 7f5684189700 1 -- 172.21.15.76:0/641545403 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5668038db0 msgr2=0x7f566803b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.661+0000 7f5684189700 1 --2- 172.21.15.76:0/641545403 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5668038db0 0x7f566803b270 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5678006fd0 tx=0x7f5678006e40 comp rx=0 tx=0).stop 2024-03-12T05:47:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.661+0000 7f5684189700 1 -- 172.21.15.76:0/641545403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f567c106610 msgr2=0x7f567c197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.661+0000 7f5684189700 1 --2- 172.21.15.76:0/641545403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f567c106610 0x7f567c197c50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f566c004f40 tx=0x7f566c005020 comp rx=0 tx=0).stop 2024-03-12T05:47:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.661+0000 7f5684189700 1 -- 172.21.15.76:0/641545403 shutdown_connections 2024-03-12T05:47:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.661+0000 7f5684189700 1 --2- 172.21.15.76:0/641545403 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5668038db0 0x7f566803b270 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:05.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.661+0000 7f5684189700 1 --2- 172.21.15.76:0/641545403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f567c106610 0x7f567c197c50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.661+0000 7f5684189700 1 -- 172.21.15.76:0/641545403 >> 172.21.15.76:0/641545403 conn(0x7f567c0fc010 msgr2=0x7f567c0fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.661+0000 7f5684189700 1 -- 172.21.15.76:0/641545403 shutdown_connections 2024-03-12T05:47:05.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:05.661+0000 7f5684189700 1 -- 172.21.15.76:0/641545403 wait complete. 2024-03-12T05:47:05.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:05 smithi076 ceph-mon[32095]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:05.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:05 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:06.197 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi088 2024-03-12T05:47:06.198 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-12T05:47:06.198 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/etc/ceph/ceph.conf 2024-03-12T05:47:06.216 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-12T05:47:06.216 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:47:06.273 INFO:tasks.cephadm:Adding host smithi088 to orchestrator... 2024-03-12T05:47:06.274 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch host add smithi088 2024-03-12T05:47:06.510 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:47:06.907 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:06 smithi076 ceph-mon[32095]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:47:08.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.058+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3932579427 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5e0107680 msgr2=0x7fc5e0107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:08.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.058+0000 7fc5e4ada700 1 --2- 172.21.15.76:0/3932579427 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5e0107680 0x7fc5e0107a60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800b0a0 tx=0x7fc5d800b3b0 comp rx=0 tx=0).stop 2024-03-12T05:47:08.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.058+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3932579427 shutdown_connections 2024-03-12T05:47:08.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.058+0000 7fc5e4ada700 1 --2- 172.21.15.76:0/3932579427 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5e0107680 0x7fc5e0107a60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:08.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.058+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3932579427 >> 172.21.15.76:0/3932579427 conn(0x7fc5e00fd190 msgr2=0x7fc5e00ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:08.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.058+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3932579427 shutdown_connections 2024-03-12T05:47:08.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.058+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3932579427 wait complete. 2024-03-12T05:47:08.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.059+0000 7fc5e4ada700 1 Processor -- start 2024-03-12T05:47:08.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.059+0000 7fc5e4ada700 1 -- start start 2024-03-12T05:47:08.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.059+0000 7fc5e4ada700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5e0107680 0x7fc5e0073f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:08.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.059+0000 7fc5e4ada700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e0075e60 con 0x7fc5e0107680 2024-03-12T05:47:08.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.059+0000 7fc5de59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5e0107680 0x7fc5e0073f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.059+0000 7fc5de59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5e0107680 0x7fc5e0073f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56854/0 (socket says 172.21.15.76:56854) 2024-03-12T05:47:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.059+0000 7fc5de59c700 1 -- 172.21.15.76:0/3713167141 learned_addr learned my addr 172.21.15.76:0/3713167141 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:47:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.060+0000 7fc5de59c700 1 -- 172.21.15.76:0/3713167141 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8009d00 con 0x7fc5e0107680 2024-03-12T05:47:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.060+0000 7fc5de59c700 1 --2- 172.21.15.76:0/3713167141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5e0107680 0x7fc5e0073f70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc5d80062a0 tx=0x7fc5d800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.060+0000 7fc5cf7fe700 1 -- 172.21.15.76:0/3713167141 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5d800e040 con 0x7fc5e0107680 2024-03-12T05:47:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.060+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3713167141 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e00744b0 con 0x7fc5e0107680 2024-03-12T05:47:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.060+0000 7fc5cf7fe700 1 -- 172.21.15.76:0/3713167141 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5d8003c10 con 0x7fc5e0107680 2024-03-12T05:47:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.060+0000 7fc5cf7fe700 1 -- 172.21.15.76:0/3713167141 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5d801d600 con 0x7fc5e0107680 2024-03-12T05:47:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.060+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3713167141 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e0074930 con 0x7fc5e0107680 2024-03-12T05:47:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.061+0000 7fc5cf7fe700 1 -- 172.21.15.76:0/3713167141 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fc5d8019070 con 0x7fc5e0107680 2024-03-12T05:47:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.061+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3713167141 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e00632a0 con 0x7fc5e0107680 2024-03-12T05:47:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.061+0000 7fc5cf7fe700 1 --2- 172.21.15.76:0/3713167141 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc5c8038da0 0x7fc5c803b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.061+0000 7fc5cf7fe700 1 -- 172.21.15.76:0/3713167141 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc5d804ce60 con 0x7fc5e0107680 2024-03-12T05:47:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.061+0000 7fc5ddd9b700 1 --2- 172.21.15.76:0/3713167141 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc5c8038da0 0x7fc5c803b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.061+0000 7fc5ddd9b700 1 --2- 172.21.15.76:0/3713167141 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc5c8038da0 0x7fc5c803b260 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4006fd0 tx=0x7fc5d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:08.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.066+0000 7fc5cf7fe700 1 -- 172.21.15.76:0/3713167141 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc5d8003d80 con 0x7fc5e0107680 2024-03-12T05:47:08.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:07 smithi076 ceph-mon[32095]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:08.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:08.265+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3713167141 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi088", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5e0074bc0 con 0x7fc5c8038da0 2024-03-12T05:47:09.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:09.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:09.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:09.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:09.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:47:09.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:09.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:09.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: Updating smithi082:/etc/ceph/ceph.conf 2024-03-12T05:47:09.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi088", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:47:09.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: Updating smithi082:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:47:09.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:47:09.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:09.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: Updating smithi082:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:47:09.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: Deploying cephadm binary to smithi088 2024-03-12T05:47:09.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:09.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:09.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:09 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:12.189 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:11 smithi076 ceph-mon[32095]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:12.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:12.413+0000 7fc5cf7fe700 1 -- 172.21.15.76:0/3713167141 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fc5e0074bc0 con 0x7fc5c8038da0 2024-03-12T05:47:12.416 INFO:teuthology.orchestra.run.smithi076.stdout:Added host 'smithi088' with addr '172.21.15.88' 2024-03-12T05:47:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:12.415+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3713167141 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc5c8038da0 msgr2=0x7fc5c803b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:12.415+0000 7fc5e4ada700 1 --2- 172.21.15.76:0/3713167141 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc5c8038da0 0x7fc5c803b260 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4006fd0 tx=0x7fc5d4006e40 comp rx=0 tx=0).stop 2024-03-12T05:47:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:12.415+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3713167141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5e0107680 msgr2=0x7fc5e0073f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:12.415+0000 7fc5e4ada700 1 --2- 172.21.15.76:0/3713167141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5e0107680 0x7fc5e0073f70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc5d80062a0 tx=0x7fc5d800bd40 comp rx=0 tx=0).stop 2024-03-12T05:47:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:12.415+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3713167141 shutdown_connections 2024-03-12T05:47:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:12.415+0000 7fc5e4ada700 1 --2- 172.21.15.76:0/3713167141 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc5c8038da0 0x7fc5c803b260 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:12.415+0000 7fc5e4ada700 1 --2- 172.21.15.76:0/3713167141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc5e0107680 0x7fc5e0073f70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:12.415+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3713167141 >> 172.21.15.76:0/3713167141 conn(0x7fc5e00fd190 msgr2=0x7fc5e01091a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:12.415+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3713167141 shutdown_connections 2024-03-12T05:47:12.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:12.415+0000 7fc5e4ada700 1 -- 172.21.15.76:0/3713167141 wait complete. 2024-03-12T05:47:13.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-12T05:47:13.392 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:47:13.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:13.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:13 smithi076 ceph-mon[32095]: Added host smithi088 2024-03-12T05:47:13.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:13.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:13 smithi076 ceph-mon[32095]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:13.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:14.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:14 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.831+0000 7f55b2560700 1 -- 172.21.15.76:0/2625689223 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55ac102810 msgr2=0x7f55ac102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.831+0000 7f55b2560700 1 --2- 172.21.15.76:0/2625689223 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55ac102810 0x7f55ac102bf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f559c009910 tx=0x7f559c009c20 comp rx=0 tx=0).stop 2024-03-12T05:47:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.831+0000 7f55b2560700 1 -- 172.21.15.76:0/2625689223 shutdown_connections 2024-03-12T05:47:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.831+0000 7f55b2560700 1 --2- 172.21.15.76:0/2625689223 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55ac102810 0x7f55ac102bf0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.831+0000 7f55b2560700 1 -- 172.21.15.76:0/2625689223 >> 172.21.15.76:0/2625689223 conn(0x7f55ac0fc060 msgr2=0x7f55ac0fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.831+0000 7f55b2560700 1 -- 172.21.15.76:0/2625689223 shutdown_connections 2024-03-12T05:47:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.831+0000 7f55b2560700 1 -- 172.21.15.76:0/2625689223 wait complete. 2024-03-12T05:47:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.831+0000 7f55b2560700 1 Processor -- start 2024-03-12T05:47:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.831+0000 7f55b2560700 1 -- start start 2024-03-12T05:47:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.832+0000 7f55b2560700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55ac102810 0x7f55ac197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.832+0000 7f55b2560700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac1975b0 con 0x7f55ac102810 2024-03-12T05:47:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.832+0000 7f55abfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55ac102810 0x7f55ac197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.832+0000 7f55abfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55ac102810 0x7f55ac197070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54206/0 (socket says 172.21.15.76:54206) 2024-03-12T05:47:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.832+0000 7f55abfff700 1 -- 172.21.15.76:0/4283932702 learned_addr learned my addr 172.21.15.76:0/4283932702 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:47:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.832+0000 7f55abfff700 1 -- 172.21.15.76:0/4283932702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c0095f0 con 0x7f55ac102810 2024-03-12T05:47:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.832+0000 7f55abfff700 1 --2- 172.21.15.76:0/4283932702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55ac102810 0x7f55ac197070 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f559c005850 tx=0x7f559c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.832+0000 7f55a97fa700 1 -- 172.21.15.76:0/4283932702 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f559c01d070 con 0x7f55ac102810 2024-03-12T05:47:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.832+0000 7f55a97fa700 1 -- 172.21.15.76:0/4283932702 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f559c00bc30 con 0x7f55ac102810 2024-03-12T05:47:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.832+0000 7f55a97fa700 1 -- 172.21.15.76:0/4283932702 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f559c003a20 con 0x7f55ac102810 2024-03-12T05:47:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.832+0000 7f55b2560700 1 -- 172.21.15.76:0/4283932702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ac1977b0 con 0x7f55ac102810 2024-03-12T05:47:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.833+0000 7f55b2560700 1 -- 172.21.15.76:0/4283932702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ac197bd0 con 0x7f55ac102810 2024-03-12T05:47:14.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.833+0000 7f55a97fa700 1 -- 172.21.15.76:0/4283932702 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f559c003b80 con 0x7f55ac102810 2024-03-12T05:47:14.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.833+0000 7f55b2560700 1 -- 172.21.15.76:0/4283932702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ac06a3b0 con 0x7f55ac102810 2024-03-12T05:47:14.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.833+0000 7f55a97fa700 1 --2- 172.21.15.76:0/4283932702 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5594038db0 0x7f559403b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:14.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.834+0000 7f55a97fa700 1 -- 172.21.15.76:0/4283932702 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f559c04cbd0 con 0x7f55ac102810 2024-03-12T05:47:14.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.834+0000 7f55ab7fe700 1 --2- 172.21.15.76:0/4283932702 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5594038db0 0x7f559403b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:14.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.834+0000 7f55ab7fe700 1 --2- 172.21.15.76:0/4283932702 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5594038db0 0x7f559403b270 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f55a0006fd0 tx=0x7f55a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:14.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:14.838+0000 7f55a97fa700 1 -- 172.21.15.76:0/4283932702 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f559c02ae20 con 0x7f55ac102810 2024-03-12T05:47:15.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:15.036+0000 7f55b2560700 1 -- 172.21.15.76:0/4283932702 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f55ac10bb60 con 0x7f5594038db0 2024-03-12T05:47:15.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:15.037+0000 7f55a97fa700 1 -- 172.21.15.76:0/4283932702 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+238 (secure 0 0 0) 0x7f55ac10bb60 con 0x7f5594038db0 2024-03-12T05:47:15.039 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:47:15.039 INFO:teuthology.orchestra.run.smithi076.stdout:[{"addr": "172.21.15.76", "hostname": "smithi076", "labels": [], "status": ""}, {"addr": "172.21.15.82", "hostname": "smithi082", "labels": [], "status": ""}, {"addr": "172.21.15.88", "hostname": "smithi088", "labels": [], "status": ""}] 2024-03-12T05:47:15.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:15.039+0000 7f55b2560700 1 -- 172.21.15.76:0/4283932702 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5594038db0 msgr2=0x7f559403b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:15.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:15.039+0000 7f55b2560700 1 --2- 172.21.15.76:0/4283932702 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5594038db0 0x7f559403b270 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f55a0006fd0 tx=0x7f55a0006e40 comp rx=0 tx=0).stop 2024-03-12T05:47:15.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:15.039+0000 7f55b2560700 1 -- 172.21.15.76:0/4283932702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55ac102810 msgr2=0x7f55ac197070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:15.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:15.039+0000 7f55b2560700 1 --2- 172.21.15.76:0/4283932702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55ac102810 0x7f55ac197070 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f559c005850 tx=0x7f559c004ff0 comp rx=0 tx=0).stop 2024-03-12T05:47:15.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:15.039+0000 7f55b2560700 1 -- 172.21.15.76:0/4283932702 shutdown_connections 2024-03-12T05:47:15.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:15.039+0000 7f55b2560700 1 --2- 172.21.15.76:0/4283932702 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5594038db0 0x7f559403b270 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:15.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:15.039+0000 7f55b2560700 1 --2- 172.21.15.76:0/4283932702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55ac102810 0x7f55ac197070 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:15.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:15.039+0000 7f55b2560700 1 -- 172.21.15.76:0/4283932702 >> 172.21.15.76:0/4283932702 conn(0x7f55ac0fc060 msgr2=0x7f55ac1035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:15.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:15.040+0000 7f55b2560700 1 -- 172.21.15.76:0/4283932702 shutdown_connections 2024-03-12T05:47:15.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:15.040+0000 7f55b2560700 1 -- 172.21.15.76:0/4283932702 wait complete. 2024-03-12T05:47:15.602 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:15 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:15.602 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:15 smithi076 ceph-mon[32095]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:15.629 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-12T05:47:15.629 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-12T05:47:15.865 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:47:16.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:16 smithi076 ceph-mon[32095]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:47:17.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.400+0000 7fcc10b22700 1 -- 172.21.15.76:0/1005798115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0c106610 msgr2=0x7fcc0c1069f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:17.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.400+0000 7fcc10b22700 1 --2- 172.21.15.76:0/1005798115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0c106610 0x7fcc0c1069f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009910 tx=0x7fcbfc009c20 comp rx=0 tx=0).stop 2024-03-12T05:47:17.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.400+0000 7fcc10b22700 1 -- 172.21.15.76:0/1005798115 shutdown_connections 2024-03-12T05:47:17.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.400+0000 7fcc10b22700 1 --2- 172.21.15.76:0/1005798115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0c106610 0x7fcc0c1069f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:17.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.400+0000 7fcc10b22700 1 -- 172.21.15.76:0/1005798115 >> 172.21.15.76:0/1005798115 conn(0x7fcc0c0771b0 msgr2=0x7fcc0c0fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:17.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.400+0000 7fcc10b22700 1 -- 172.21.15.76:0/1005798115 shutdown_connections 2024-03-12T05:47:17.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.400+0000 7fcc10b22700 1 -- 172.21.15.76:0/1005798115 wait complete. 2024-03-12T05:47:17.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.401+0000 7fcc10b22700 1 Processor -- start 2024-03-12T05:47:17.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.401+0000 7fcc10b22700 1 -- start start 2024-03-12T05:47:17.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.401+0000 7fcc10b22700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0c106610 0x7fcc0c19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:17.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.401+0000 7fcc10b22700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c19ae00 con 0x7fcc0c106610 2024-03-12T05:47:17.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.401+0000 7fcc0a59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0c106610 0x7fcc0c19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:17.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.401+0000 7fcc0a59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0c106610 0x7fcc0c19a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54222/0 (socket says 172.21.15.76:54222) 2024-03-12T05:47:17.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.401+0000 7fcc0a59c700 1 -- 172.21.15.76:0/2510961529 learned_addr learned my addr 172.21.15.76:0/2510961529 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:47:17.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.402+0000 7fcc0a59c700 1 -- 172.21.15.76:0/2510961529 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbfc0095f0 con 0x7fcc0c106610 2024-03-12T05:47:17.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.402+0000 7fcc0a59c700 1 --2- 172.21.15.76:0/2510961529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0c106610 0x7fcc0c19a8c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc004f40 tx=0x7fcbfc005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:17.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.402+0000 7fcbfb7fe700 1 -- 172.21.15.76:0/2510961529 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcbfc01d070 con 0x7fcc0c106610 2024-03-12T05:47:17.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.402+0000 7fcbfb7fe700 1 -- 172.21.15.76:0/2510961529 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbfc00bc30 con 0x7fcc0c106610 2024-03-12T05:47:17.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.402+0000 7fcc10b22700 1 -- 172.21.15.76:0/2510961529 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0c19b060 con 0x7fcc0c106610 2024-03-12T05:47:17.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.402+0000 7fcbfb7fe700 1 -- 172.21.15.76:0/2510961529 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcbfc003970 con 0x7fcc0c106610 2024-03-12T05:47:17.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.402+0000 7fcc10b22700 1 -- 172.21.15.76:0/2510961529 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0c194a80 con 0x7fcc0c106610 2024-03-12T05:47:17.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.403+0000 7fcbfb7fe700 1 -- 172.21.15.76:0/2510961529 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fcbfc003be0 con 0x7fcc0c106610 2024-03-12T05:47:17.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.403+0000 7fcbfb7fe700 1 --2- 172.21.15.76:0/2510961529 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcbf403b000 0x7fcbf403d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:17.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.403+0000 7fcc10b22700 1 -- 172.21.15.76:0/2510961529 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0c101c80 con 0x7fcc0c106610 2024-03-12T05:47:17.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.403+0000 7fcbfb7fe700 1 -- 172.21.15.76:0/2510961529 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcbfc04cb80 con 0x7fcc0c106610 2024-03-12T05:47:17.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.403+0000 7fcc09d9b700 1 --2- 172.21.15.76:0/2510961529 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcbf403b000 0x7fcbf403d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:17.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.403+0000 7fcc09d9b700 1 --2- 172.21.15.76:0/2510961529 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcbf403b000 0x7fcbf403d4c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcc00006fd0 tx=0x7fcc00006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:17.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.408+0000 7fcbfb7fe700 1 -- 172.21.15.76:0/2510961529 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcbfc04de60 con 0x7fcc0c106610 2024-03-12T05:47:17.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:17.612+0000 7fcc10b22700 1 -- 172.21.15.76:0/2510961529 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fcc0c0632a0 con 0x7fcc0c106610 2024-03-12T05:47:17.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:17 smithi076 ceph-mon[32095]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:18.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:18.526+0000 7fcbfb7fe700 1 -- 172.21.15.76:0/2510961529 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fcbfc026070 con 0x7fcc0c106610 2024-03-12T05:47:18.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:18.528+0000 7fcc10b22700 1 -- 172.21.15.76:0/2510961529 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcbf403b000 msgr2=0x7fcbf403d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:18.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:18.528+0000 7fcc10b22700 1 --2- 172.21.15.76:0/2510961529 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcbf403b000 0x7fcbf403d4c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcc00006fd0 tx=0x7fcc00006e40 comp rx=0 tx=0).stop 2024-03-12T05:47:18.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:18.528+0000 7fcc10b22700 1 -- 172.21.15.76:0/2510961529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0c106610 msgr2=0x7fcc0c19a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:18.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:18.528+0000 7fcc10b22700 1 --2- 172.21.15.76:0/2510961529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0c106610 0x7fcc0c19a8c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc004f40 tx=0x7fcbfc005020 comp rx=0 tx=0).stop 2024-03-12T05:47:18.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:18.528+0000 7fcc10b22700 1 -- 172.21.15.76:0/2510961529 shutdown_connections 2024-03-12T05:47:18.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:18.528+0000 7fcc10b22700 1 --2- 172.21.15.76:0/2510961529 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcbf403b000 0x7fcbf403d4c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:18.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:18.528+0000 7fcc10b22700 1 --2- 172.21.15.76:0/2510961529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0c106610 0x7fcc0c19a8c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:18.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:18.528+0000 7fcc10b22700 1 -- 172.21.15.76:0/2510961529 >> 172.21.15.76:0/2510961529 conn(0x7fcc0c0771b0 msgr2=0x7fcc0c0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:18.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:18.528+0000 7fcc10b22700 1 -- 172.21.15.76:0/2510961529 shutdown_connections 2024-03-12T05:47:18.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:18.528+0000 7fcc10b22700 1 -- 172.21.15.76:0/2510961529 wait complete. 2024-03-12T05:47:18.531 INFO:teuthology.orchestra.run.smithi076.stderr:adjusted tunables profile to default 2024-03-12T05:47:18.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:18 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:18.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:18 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:18.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:18 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:18.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:18 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:18.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:18 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:47:18.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:18 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:18.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:18 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:18.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:18 smithi076 ceph-mon[32095]: Updating smithi088:/etc/ceph/ceph.conf 2024-03-12T05:47:18.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:18 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2510961529' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T05:47:18.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:18 smithi076 ceph-mon[32095]: Updating smithi088:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:47:18.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:18 smithi076 ceph-mon[32095]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:47:18.902 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:18 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:18.902 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:18 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:18.902 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:18 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:19.209 INFO:tasks.cephadm:Adding mon.a on smithi076 2024-03-12T05:47:19.209 INFO:tasks.cephadm:Adding mon.b on smithi082 2024-03-12T05:47:19.209 INFO:tasks.cephadm:Adding mon.c on smithi088 2024-03-12T05:47:19.209 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi076:172.21.15.76=a;smithi082:172.21.15.82=b;smithi088:172.21.15.88=c' 2024-03-12T05:47:19.445 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:47:19.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:19 smithi076 ceph-mon[32095]: Updating smithi088:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:47:19.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:19 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2510961529' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T05:47:19.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:19 smithi076 ceph-mon[32095]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:47:19.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:19 smithi076 ceph-mon[32095]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:20.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.765+0000 7f7016dc1700 1 -- 172.21.15.88:0/1928941968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7010106620 msgr2=0x7f7010106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:20.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.765+0000 7f7016dc1700 1 --2- 172.21.15.88:0/1928941968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7010106620 0x7f7010106a00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7004009910 tx=0x7f7004009c20 comp rx=0 tx=0).stop 2024-03-12T05:47:20.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.765+0000 7f7016dc1700 1 -- 172.21.15.88:0/1928941968 shutdown_connections 2024-03-12T05:47:20.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.765+0000 7f7016dc1700 1 --2- 172.21.15.88:0/1928941968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7010106620 0x7f7010106a00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:20.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.765+0000 7f7016dc1700 1 -- 172.21.15.88:0/1928941968 >> 172.21.15.88:0/1928941968 conn(0x7f70100fc060 msgr2=0x7f70100fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:20.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.765+0000 7f7016dc1700 1 -- 172.21.15.88:0/1928941968 shutdown_connections 2024-03-12T05:47:20.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.766+0000 7f7016dc1700 1 -- 172.21.15.88:0/1928941968 wait complete. 2024-03-12T05:47:20.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.766+0000 7f7016dc1700 1 Processor -- start 2024-03-12T05:47:20.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.766+0000 7f7016dc1700 1 -- start start 2024-03-12T05:47:20.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.766+0000 7f7016dc1700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7010106620 0x7f7010197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:20.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.766+0000 7f7016dc1700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701006aa80 con 0x7f7010106620 2024-03-12T05:47:20.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.766+0000 7f7014b5d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7010106620 0x7f7010197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:20.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.766+0000 7f7014b5d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7010106620 0x7f7010197ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.88:38450/0 (socket says 172.21.15.88:38450) 2024-03-12T05:47:20.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.766+0000 7f7014b5d700 1 -- 172.21.15.88:0/1963880088 learned_addr learned my addr 172.21.15.88:0/1963880088 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-03-12T05:47:20.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.767+0000 7f7014b5d700 1 -- 172.21.15.88:0/1963880088 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70040095f0 con 0x7f7010106620 2024-03-12T05:47:20.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.767+0000 7f7014b5d700 1 --2- 172.21.15.88:0/1963880088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7010106620 0x7f7010197ce0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7004004f40 tx=0x7f7004005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:20.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.767+0000 7f700dffb700 1 -- 172.21.15.88:0/1963880088 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f700401d070 con 0x7f7010106620 2024-03-12T05:47:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.767+0000 7f7016dc1700 1 -- 172.21.15.88:0/1963880088 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70101982e0 con 0x7f7010106620 2024-03-12T05:47:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.767+0000 7f700dffb700 1 -- 172.21.15.88:0/1963880088 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f700400bc30 con 0x7f7010106620 2024-03-12T05:47:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.767+0000 7f700dffb700 1 -- 172.21.15.88:0/1963880088 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7004003970 con 0x7f7010106620 2024-03-12T05:47:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.767+0000 7f7016dc1700 1 -- 172.21.15.88:0/1963880088 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7010198760 con 0x7f7010106620 2024-03-12T05:47:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.768+0000 7f7016dc1700 1 -- 172.21.15.88:0/1963880088 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7010101c90 con 0x7f7010106620 2024-03-12T05:47:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.768+0000 7f700dffb700 1 -- 172.21.15.88:0/1963880088 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f7004003c30 con 0x7f7010106620 2024-03-12T05:47:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.769+0000 7f700dffb700 1 --2- 172.21.15.88:0/1963880088 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f7000038d50 0x7f700003b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.769+0000 7f700dffb700 1 -- 172.21.15.88:0/1963880088 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f700404cb40 con 0x7f7010106620 2024-03-12T05:47:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.769+0000 7f700ffff700 1 --2- 172.21.15.88:0/1963880088 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f7000038d50 0x7f700003b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.769+0000 7f700ffff700 1 --2- 172.21.15.88:0/1963880088 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f7000038d50 0x7f700003b210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc006fd0 tx=0x7f6ffc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:20.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.774+0000 7f700dffb700 1 -- 172.21.15.88:0/1963880088 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7004029540 con 0x7f7010106620 2024-03-12T05:47:20.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.977+0000 7f7016dc1700 1 -- 172.21.15.88:0/1963880088 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi076:172.21.15.76=a;smithi082:172.21.15.82=b;smithi088:172.21.15.88=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f7010198b50 con 0x7f7000038d50 2024-03-12T05:47:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.985+0000 7f700dffb700 1 -- 172.21.15.88:0/1963880088 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7010198b50 con 0x7f7000038d50 2024-03-12T05:47:20.987 INFO:teuthology.orchestra.run.smithi088.stdout:Scheduled mon update... 2024-03-12T05:47:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.987+0000 7f7016dc1700 1 -- 172.21.15.88:0/1963880088 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f7000038d50 msgr2=0x7f700003b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.987+0000 7f7016dc1700 1 --2- 172.21.15.88:0/1963880088 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f7000038d50 0x7f700003b210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc006fd0 tx=0x7f6ffc006e40 comp rx=0 tx=0).stop 2024-03-12T05:47:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.987+0000 7f7016dc1700 1 -- 172.21.15.88:0/1963880088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7010106620 msgr2=0x7f7010197ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.987+0000 7f7016dc1700 1 --2- 172.21.15.88:0/1963880088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7010106620 0x7f7010197ce0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7004004f40 tx=0x7f7004005020 comp rx=0 tx=0).stop 2024-03-12T05:47:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.987+0000 7f7016dc1700 1 -- 172.21.15.88:0/1963880088 shutdown_connections 2024-03-12T05:47:20.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.987+0000 7f7016dc1700 1 --2- 172.21.15.88:0/1963880088 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f7000038d50 0x7f700003b210 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:20.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.987+0000 7f7016dc1700 1 --2- 172.21.15.88:0/1963880088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7010106620 0x7f7010197ce0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:20.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.987+0000 7f7016dc1700 1 -- 172.21.15.88:0/1963880088 >> 172.21.15.88:0/1963880088 conn(0x7f70100fc060 msgr2=0x7f70100fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:20.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.988+0000 7f7016dc1700 1 -- 172.21.15.88:0/1963880088 shutdown_connections 2024-03-12T05:47:20.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:20.988+0000 7f7016dc1700 1 -- 172.21.15.88:0/1963880088 wait complete. 2024-03-12T05:47:21.690 DEBUG:teuthology.orchestra.run.smithi082:mon.b> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mon.b.service 2024-03-12T05:47:21.693 DEBUG:teuthology.orchestra.run.smithi088:mon.c> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mon.c.service 2024-03-12T05:47:21.695 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T05:47:21.695 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T05:47:22.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:21 smithi076 ceph-mon[32095]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:22.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:22.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:22.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:22.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:22.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:22.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:47:22.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:23.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:22 smithi076 ceph-mon[32095]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi076:172.21.15.76=a;smithi082:172.21.15.82=b;smithi088:172.21.15.88=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:47:23.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:22 smithi076 ceph-mon[32095]: Saving service mon spec with placement smithi076:172.21.15.76=a;smithi082:172.21.15.82=b;smithi088:172.21.15.88=c;count:3 2024-03-12T05:47:23.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:22 smithi076 ceph-mon[32095]: Deploying daemon mon.c on smithi088 2024-03-12T05:47:23.149 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:47:23.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:23 smithi076 ceph-mon[32095]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.405+0000 7f5b11a9c700 1 -- 172.21.15.88:0/2964550626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b0c1076d0 msgr2=0x7f5b0c107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.405+0000 7f5b11a9c700 1 --2- 172.21.15.88:0/2964550626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b0c1076d0 0x7f5b0c107ab0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5afc009910 tx=0x7f5afc009c20 comp rx=0 tx=0).stop 2024-03-12T05:47:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.405+0000 7f5b11a9c700 1 -- 172.21.15.88:0/2964550626 shutdown_connections 2024-03-12T05:47:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.405+0000 7f5b11a9c700 1 --2- 172.21.15.88:0/2964550626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b0c1076d0 0x7f5b0c107ab0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.405+0000 7f5b11a9c700 1 -- 172.21.15.88:0/2964550626 >> 172.21.15.88:0/2964550626 conn(0x7f5b0c0fb7c0 msgr2=0x7f5b0c0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.405+0000 7f5b11a9c700 1 -- 172.21.15.88:0/2964550626 shutdown_connections 2024-03-12T05:47:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.405+0000 7f5b11a9c700 1 -- 172.21.15.88:0/2964550626 wait complete. 2024-03-12T05:47:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.405+0000 7f5b11a9c700 1 Processor -- start 2024-03-12T05:47:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.406+0000 7f5b11a9c700 1 -- start start 2024-03-12T05:47:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.406+0000 7f5b11a9c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b0c1076d0 0x7f5b0c196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.406+0000 7f5b11a9c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c197430 con 0x7f5b0c1076d0 2024-03-12T05:47:25.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.406+0000 7f5b0b7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b0c1076d0 0x7f5b0c196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:25.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.406+0000 7f5b0b7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b0c1076d0 0x7f5b0c196ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.88:43482/0 (socket says 172.21.15.88:43482) 2024-03-12T05:47:25.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.406+0000 7f5b0b7fe700 1 -- 172.21.15.88:0/3410499560 learned_addr learned my addr 172.21.15.88:0/3410499560 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-03-12T05:47:25.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.407+0000 7f5b0b7fe700 1 -- 172.21.15.88:0/3410499560 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc0095f0 con 0x7f5b0c1076d0 2024-03-12T05:47:25.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.407+0000 7f5b0b7fe700 1 --2- 172.21.15.88:0/3410499560 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b0c1076d0 0x7f5b0c196ef0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5afc004f40 tx=0x7f5afc005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:25.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.407+0000 7f5b08ff9700 1 -- 172.21.15.88:0/3410499560 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5afc01d070 con 0x7f5b0c1076d0 2024-03-12T05:47:25.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.407+0000 7f5b11a9c700 1 -- 172.21.15.88:0/3410499560 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b0c197690 con 0x7f5b0c1076d0 2024-03-12T05:47:25.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.407+0000 7f5b08ff9700 1 -- 172.21.15.88:0/3410499560 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5afc00bbd0 con 0x7f5b0c1076d0 2024-03-12T05:47:25.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.407+0000 7f5b08ff9700 1 -- 172.21.15.88:0/3410499560 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5afc003900 con 0x7f5b0c1076d0 2024-03-12T05:47:25.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.407+0000 7f5b11a9c700 1 -- 172.21.15.88:0/3410499560 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b0c197b90 con 0x7f5b0c1076d0 2024-03-12T05:47:25.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.408+0000 7f5b08ff9700 1 -- 172.21.15.88:0/3410499560 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f5afc003b70 con 0x7f5b0c1076d0 2024-03-12T05:47:25.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.408+0000 7f5b08ff9700 1 --2- 172.21.15.88:0/3410499560 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5af4038db0 0x7f5af403b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:25.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.408+0000 7f5b11a9c700 1 -- 172.21.15.88:0/3410499560 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b0c0632a0 con 0x7f5b0c1076d0 2024-03-12T05:47:25.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.408+0000 7f5b08ff9700 1 -- 172.21.15.88:0/3410499560 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5afc04c9a0 con 0x7f5b0c1076d0 2024-03-12T05:47:25.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.408+0000 7f5b0affd700 1 --2- 172.21.15.88:0/3410499560 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5af4038db0 0x7f5af403b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:25.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.409+0000 7f5b0affd700 1 --2- 172.21.15.88:0/3410499560 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5af4038db0 0x7f5af403b270 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5b00006fd0 tx=0x7f5b00006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:25.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.414+0000 7f5b08ff9700 1 -- 172.21.15.88:0/3410499560 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5afc026020 con 0x7f5b0c1076d0 2024-03-12T05:47:25.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:25 smithi088 systemd[1]: Starting Ceph mon.c for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T05:47:25.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.699+0000 7f5b11a9c700 1 -- 172.21.15.88:0/3410499560 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b0c10ba90 con 0x7f5b0c1076d0 2024-03-12T05:47:25.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.700+0000 7f5b08ff9700 1 -- 172.21.15.88:0/3410499560 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7f5afc029360 con 0x7f5b0c1076d0 2024-03-12T05:47:25.702 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:47:25.702 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","modified":"2024-03-12T05:45:43.279953Z","created":"2024-03-12T05:45:43.279953Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T05:47:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.702+0000 7f5b11a9c700 1 -- 172.21.15.88:0/3410499560 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5af4038db0 msgr2=0x7f5af403b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.702+0000 7f5b11a9c700 1 --2- 172.21.15.88:0/3410499560 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5af4038db0 0x7f5af403b270 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5b00006fd0 tx=0x7f5b00006e40 comp rx=0 tx=0).stop 2024-03-12T05:47:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.702+0000 7f5b11a9c700 1 -- 172.21.15.88:0/3410499560 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b0c1076d0 msgr2=0x7f5b0c196ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.702+0000 7f5b11a9c700 1 --2- 172.21.15.88:0/3410499560 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b0c1076d0 0x7f5b0c196ef0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5afc004f40 tx=0x7f5afc005020 comp rx=0 tx=0).stop 2024-03-12T05:47:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.702+0000 7f5b11a9c700 1 -- 172.21.15.88:0/3410499560 shutdown_connections 2024-03-12T05:47:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.702+0000 7f5b11a9c700 1 --2- 172.21.15.88:0/3410499560 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5af4038db0 0x7f5af403b270 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.702+0000 7f5b11a9c700 1 --2- 172.21.15.88:0/3410499560 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b0c1076d0 0x7f5b0c196ef0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.702+0000 7f5b11a9c700 1 -- 172.21.15.88:0/3410499560 >> 172.21.15.88:0/3410499560 conn(0x7f5b0c0fb7c0 msgr2=0x7f5b0c0fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.702+0000 7f5b11a9c700 1 -- 172.21.15.88:0/3410499560 shutdown_connections 2024-03-12T05:47:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:25.702+0000 7f5b11a9c700 1 -- 172.21.15.88:0/3410499560 wait complete. 2024-03-12T05:47:25.705 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 1 2024-03-12T05:47:25.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:25 smithi088 podman[36552]: 2024-03-12 05:47:25.627108539 +0000 UTC m=+0.112397434 container create 9ca03031be8eabda09b54d41d2a60fa443f1867a08ba32b52415c3bcb6fc9105 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-12T05:47:25.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:25 smithi088 podman[36552]: 2024-03-12 05:47:25.532536199 +0000 UTC m=+0.017825095 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:47:26.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:25 smithi076 ceph-mon[32095]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:26.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:25 smithi076 ceph-mon[32095]: from='client.? 172.21.15.88:0/3410499560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T05:47:26.434 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 podman[36552]: 2024-03-12 05:47:26.013976205 +0000 UTC m=+0.499265106 container init 9ca03031be8eabda09b54d41d2a60fa443f1867a08ba32b52415c3bcb6fc9105 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-12T05:47:26.434 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 podman[36552]: 2024-03-12 05:47:26.017149383 +0000 UTC m=+0.502438279 container start 9ca03031be8eabda09b54d41d2a60fa443f1867a08ba32b52415c3bcb6fc9105 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-12T05:47:26.434 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T05:47:26.434 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-12T05:47:26.435 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: pidfile_write: ignore empty --pid-file 2024-03-12T05:47:26.435 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: load: jerasure load: lrc 2024-03-12T05:47:26.435 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: RocksDB version: 7.9.2 2024-03-12T05:47:26.435 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Git sha 0 2024-03-12T05:47:26.435 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-12T05:47:26.435 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: DB SUMMARY 2024-03-12T05:47:26.435 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: DB Session ID: RGMVCR4GDJEBZA11IO0U 2024-03-12T05:47:26.435 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: CURRENT file: CURRENT 2024-03-12T05:47:26.435 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T05:47:26.435 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-12T05:47:26.435 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-12T05:47:26.435 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-12T05:47:26.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.error_if_exists: 0 2024-03-12T05:47:26.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.create_if_missing: 0 2024-03-12T05:47:26.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.paranoid_checks: 1 2024-03-12T05:47:26.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T05:47:26.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T05:47:26.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T05:47:26.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.env: 0x55b3338f1ae0 2024-03-12T05:47:26.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T05:47:26.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.info_log: 0x55b3344b38c0 2024-03-12T05:47:26.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T05:47:26.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.statistics: (nil) 2024-03-12T05:47:26.436 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.use_fsync: 0 2024-03-12T05:47:26.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_log_file_size: 0 2024-03-12T05:47:26.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T05:47:26.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T05:47:26.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T05:47:26.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T05:47:26.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.allow_fallocate: 1 2024-03-12T05:47:26.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T05:47:26.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T05:47:26.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.use_direct_reads: 0 2024-03-12T05:47:26.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T05:47:26.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T05:47:26.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.db_log_dir: 2024-03-12T05:47:26.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.wal_dir: 2024-03-12T05:47:26.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T05:47:26.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T05:47:26.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T05:47:26.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T05:47:26.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T05:47:26.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T05:47:26.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T05:47:26.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T05:47:26.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.write_buffer_manager: 0x55b334487400 2024-03-12T05:47:26.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T05:47:26.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T05:47:26.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T05:47:26.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T05:47:26.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T05:47:26.439 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T05:47:26.439 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T05:47:26.439 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T05:47:26.439 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.unordered_write: 0 2024-03-12T05:47:26.439 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T05:47:26.439 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T05:47:26.439 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T05:47:26.439 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T05:47:26.439 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.row_cache: None 2024-03-12T05:47:26.439 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.wal_filter: None 2024-03-12T05:47:26.441 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T05:47:26.441 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T05:47:26.441 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.two_write_queues: 0 2024-03-12T05:47:26.441 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T05:47:26.441 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.wal_compression: 0 2024-03-12T05:47:26.441 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.atomic_flush: 0 2024-03-12T05:47:26.441 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T05:47:26.441 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T05:47:26.441 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T05:47:26.441 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.log_readahead_size: 0 2024-03-12T05:47:26.442 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T05:47:26.442 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T05:47:26.442 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T05:47:26.442 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T05:47:26.442 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T05:47:26.442 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T05:47:26.442 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T05:47:26.442 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_background_jobs: 2 2024-03-12T05:47:26.442 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_background_compactions: -1 2024-03-12T05:47:26.442 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_subcompactions: 1 2024-03-12T05:47:26.442 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T05:47:26.442 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T05:47:26.443 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T05:47:26.443 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T05:47:26.443 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T05:47:26.443 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T05:47:26.443 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T05:47:26.443 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T05:47:26.443 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_open_files: -1 2024-03-12T05:47:26.443 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T05:47:26.443 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T05:47:26.443 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T05:47:26.443 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T05:47:26.443 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_background_flushes: -1 2024-03-12T05:47:26.443 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Compression algorithms supported: 2024-03-12T05:47:26.443 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: kZSTD supported: 0 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: kXpressCompression supported: 0 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: kZlibCompression supported: 1 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: kSnappyCompression supported: 1 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: kLZ4Compression supported: 1 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: kBZip2Compression supported: 0 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.merge_operator: 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compaction_filter: None 2024-03-12T05:47:26.444 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T05:47:26.445 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T05:47:26.445 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T05:47:26.445 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T05:47:26.445 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b3344b39c0) 2024-03-12T05:47:26.445 INFO:journalctl@ceph.mon.c.smithi088.stdout: cache_index_and_filter_blocks: 1 2024-03-12T05:47:26.445 INFO:journalctl@ceph.mon.c.smithi088.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T05:47:26.445 INFO:journalctl@ceph.mon.c.smithi088.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T05:47:26.445 INFO:journalctl@ceph.mon.c.smithi088.stdout: pin_top_level_index_and_filter: 1 2024-03-12T05:47:26.445 INFO:journalctl@ceph.mon.c.smithi088.stdout: index_type: 0 2024-03-12T05:47:26.445 INFO:journalctl@ceph.mon.c.smithi088.stdout: data_block_index_type: 0 2024-03-12T05:47:26.445 INFO:journalctl@ceph.mon.c.smithi088.stdout: index_shortening: 1 2024-03-12T05:47:26.445 INFO:journalctl@ceph.mon.c.smithi088.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T05:47:26.445 INFO:journalctl@ceph.mon.c.smithi088.stdout: checksum: 4 2024-03-12T05:47:26.445 INFO:journalctl@ceph.mon.c.smithi088.stdout: no_block_cache: 0 2024-03-12T05:47:26.446 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_cache: 0x55b3344b7090 2024-03-12T05:47:26.446 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_cache_name: BinnedLRUCache 2024-03-12T05:47:26.446 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_cache_options: 2024-03-12T05:47:26.446 INFO:journalctl@ceph.mon.c.smithi088.stdout: capacity : 536870912 2024-03-12T05:47:26.446 INFO:journalctl@ceph.mon.c.smithi088.stdout: num_shard_bits : 4 2024-03-12T05:47:26.446 INFO:journalctl@ceph.mon.c.smithi088.stdout: strict_capacity_limit : 0 2024-03-12T05:47:26.446 INFO:journalctl@ceph.mon.c.smithi088.stdout: high_pri_pool_ratio: 0.000 2024-03-12T05:47:26.446 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_cache_compressed: (nil) 2024-03-12T05:47:26.446 INFO:journalctl@ceph.mon.c.smithi088.stdout: persistent_cache: (nil) 2024-03-12T05:47:26.446 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_size: 4096 2024-03-12T05:47:26.446 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_size_deviation: 10 2024-03-12T05:47:26.446 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_restart_interval: 16 2024-03-12T05:47:26.446 INFO:journalctl@ceph.mon.c.smithi088.stdout: index_block_restart_interval: 1 2024-03-12T05:47:26.446 INFO:journalctl@ceph.mon.c.smithi088.stdout: metadata_block_size: 4096 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout: partition_filters: 0 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout: use_delta_encoding: 1 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout: filter_policy: bloomfilter 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout: whole_key_filtering: 1 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout: verify_compression: 0 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout: read_amp_bytes_per_bit: 0 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout: format_version: 5 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout: enable_index_compression: 1 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout: block_align: 0 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout: max_auto_readahead_size: 262144 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout: prepopulate_block_cache: 0 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout: initial_auto_readahead_size: 8192 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T05:47:26.447 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compression: NoCompression 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.num_levels: 7 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T05:47:26.448 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T05:47:26.449 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T05:47:26.449 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T05:47:26.449 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T05:47:26.449 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T05:47:26.449 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T05:47:26.449 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T05:47:26.449 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T05:47:26.449 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T05:47:26.449 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T05:47:26.449 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T05:47:26.449 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T05:47:26.449 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T05:47:26.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T05:47:26.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T05:47:26.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T05:47:26.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T05:47:26.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T05:47:26.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T05:47:26.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T05:47:26.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T05:47:26.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T05:47:26.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T05:47:26.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T05:47:26.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T05:47:26.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T05:47:26.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T05:47:26.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T05:47:26.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T05:47:26.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T05:47:26.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T05:47:26.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T05:47:26.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T05:47:26.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T05:47:26.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T05:47:26.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T05:47:26.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T05:47:26.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T05:47:26.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T05:47:26.452 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T05:47:26.452 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T05:47:26.452 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T05:47:26.452 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T05:47:26.452 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.table_properties_collectors: 2024-03-12T05:47:26.452 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.inplace_update_support: 0 2024-03-12T05:47:26.452 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T05:47:26.452 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T05:47:26.452 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T05:47:26.452 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T05:47:26.452 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.bloom_locality: 0 2024-03-12T05:47:26.452 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.max_successive_merges: 0 2024-03-12T05:47:26.452 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T05:47:26.453 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T05:47:26.453 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T05:47:26.453 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T05:47:26.453 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.ttl: 2592000 2024-03-12T05:47:26.453 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T05:47:26.453 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T05:47:26.453 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T05:47:26.453 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.enable_blob_files: false 2024-03-12T05:47:26.453 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.min_blob_size: 0 2024-03-12T05:47:26.453 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T05:47:26.453 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T05:47:26.453 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T05:47:26.453 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T05:47:26.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T05:47:26.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T05:47:26.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T05:47:26.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T05:47:26.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-12T05:47:26.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-12T05:47:26.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 628a0dd3-bd6b-42cb-bd91-06fe02017ec1 2024-03-12T05:47:26.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710222446047081, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-12T05:47:26.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-12T05:47:26.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710222446047491, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710222446, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "628a0dd3-bd6b-42cb-bd91-06fe02017ec1", "db_session_id": "RGMVCR4GDJEBZA11IO0U", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-12T05:47:26.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710222446047569, "job": 1, "event": "recovery_finished"} 2024-03-12T05:47:26.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-12T05:47:26.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T05:47:26.455 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b3345a8000 2024-03-12T05:47:26.455 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: DB pointer 0x55b334592000 2024-03-12T05:47:26.455 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T05:47:26.455 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T05:47:26.455 INFO:journalctl@ceph.mon.c.smithi088.stdout: ** DB Stats ** 2024-03-12T05:47:26.455 INFO:journalctl@ceph.mon.c.smithi088.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T05:47:26.455 INFO:journalctl@ceph.mon.c.smithi088.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T05:47:26.455 INFO:journalctl@ceph.mon.c.smithi088.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T05:47:26.455 INFO:journalctl@ceph.mon.c.smithi088.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T05:47:26.455 INFO:journalctl@ceph.mon.c.smithi088.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T05:47:26.455 INFO:journalctl@ceph.mon.c.smithi088.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T05:47:26.455 INFO:journalctl@ceph.mon.c.smithi088.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T05:47:26.455 INFO:journalctl@ceph.mon.c.smithi088.stdout: 2024-03-12T05:47:26.455 INFO:journalctl@ceph.mon.c.smithi088.stdout: ** Compaction Stats [default] ** 2024-03-12T05:47:26.456 INFO:journalctl@ceph.mon.c.smithi088.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T05:47:26.456 INFO:journalctl@ceph.mon.c.smithi088.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T05:47:26.456 INFO:journalctl@ceph.mon.c.smithi088.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T05:47:26.456 INFO:journalctl@ceph.mon.c.smithi088.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T05:47:26.456 INFO:journalctl@ceph.mon.c.smithi088.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T05:47:26.456 INFO:journalctl@ceph.mon.c.smithi088.stdout: 2024-03-12T05:47:26.456 INFO:journalctl@ceph.mon.c.smithi088.stdout: ** Compaction Stats [default] ** 2024-03-12T05:47:26.456 INFO:journalctl@ceph.mon.c.smithi088.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T05:47:26.456 INFO:journalctl@ceph.mon.c.smithi088.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T05:47:26.456 INFO:journalctl@ceph.mon.c.smithi088.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-12T05:47:26.456 INFO:journalctl@ceph.mon.c.smithi088.stdout: 2024-03-12T05:47:26.456 INFO:journalctl@ceph.mon.c.smithi088.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T05:47:26.456 INFO:journalctl@ceph.mon.c.smithi088.stdout: 2024-03-12T05:47:26.456 INFO:journalctl@ceph.mon.c.smithi088.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T05:47:26.457 INFO:journalctl@ceph.mon.c.smithi088.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T05:47:26.457 INFO:journalctl@ceph.mon.c.smithi088.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T05:47:26.457 INFO:journalctl@ceph.mon.c.smithi088.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T05:47:26.457 INFO:journalctl@ceph.mon.c.smithi088.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T05:47:26.457 INFO:journalctl@ceph.mon.c.smithi088.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T05:47:26.457 INFO:journalctl@ceph.mon.c.smithi088.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T05:47:26.457 INFO:journalctl@ceph.mon.c.smithi088.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T05:47:26.457 INFO:journalctl@ceph.mon.c.smithi088.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T05:47:26.457 INFO:journalctl@ceph.mon.c.smithi088.stdout: Block cache BinnedLRUCache@0x55b3344b7090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-12T05:47:26.457 INFO:journalctl@ceph.mon.c.smithi088.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-12T05:47:26.457 INFO:journalctl@ceph.mon.c.smithi088.stdout: 2024-03-12T05:47:26.457 INFO:journalctl@ceph.mon.c.smithi088.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T05:47:26.457 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-12T05:47:26.457 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: using public_addr v2:172.21.15.88:0/0 -> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-03-12T05:47:26.458 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: starting mon.c rank -1 at public addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] at bind addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:47:26.458 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c@-1(???) e0 preinit fsid 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:47:26.458 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c@-1(synchronizing).mds e1 new map 2024-03-12T05:47:26.458 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-12T05:47:26.458 INFO:journalctl@ceph.mon.c.smithi088.stdout: e1 2024-03-12T05:47:26.458 INFO:journalctl@ceph.mon.c.smithi088.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T05:47:26.458 INFO:journalctl@ceph.mon.c.smithi088.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T05:47:26.458 INFO:journalctl@ceph.mon.c.smithi088.stdout: legacy client fscid: -1 2024-03-12T05:47:26.458 INFO:journalctl@ceph.mon.c.smithi088.stdout: 2024-03-12T05:47:26.458 INFO:journalctl@ceph.mon.c.smithi088.stdout: No filesystems configured 2024-03-12T05:47:26.458 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-12T05:47:26.458 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-12T05:47:26.458 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-12T05:47:26.459 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-12T05:47:26.459 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-12T05:47:26.459 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-12T05:47:26.459 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T05:47:26.459 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T05:47:26.459 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T05:47:26.459 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T05:47:26.459 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:47:26.459 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:26.459 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.459 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.459 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:47:26.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:26.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:26.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: Updating smithi082:/etc/ceph/ceph.conf 2024-03-12T05:47:26.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi088", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:47:26.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: Updating smithi082:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:47:26.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:47:26.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:26.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: Updating smithi082:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:47:26.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: Deploying cephadm binary to smithi088 2024-03-12T05:47:26.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:26.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: Added host smithi088 2024-03-12T05:47:26.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:26.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:26.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:26.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:47:26.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:26.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.462 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.462 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.462 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:47:26.462 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:26.462 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:26.462 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: Updating smithi088:/etc/ceph/ceph.conf 2024-03-12T05:47:26.462 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2510961529' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T05:47:26.462 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: Updating smithi088:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:47:26.462 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:47:26.462 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.462 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.462 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.462 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: Updating smithi088:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2510961529' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi076:172.21.15.76=a;smithi082:172.21.15.82=b;smithi088:172.21.15.88=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: Saving service mon spec with placement smithi076:172.21.15.76=a;smithi082:172.21.15.82=b;smithi088:172.21.15.88=c;count:3 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: Deploying daemon mon.c on smithi088 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:26.463 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:26.464 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: from='client.? 172.21.15.88:0/3410499560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T05:47:26.464 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 ceph-mon[36585]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-12T05:47:26.464 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 bash[36552]: 9ca03031be8eabda09b54d41d2a60fa443f1867a08ba32b52415c3bcb6fc9105 2024-03-12T05:47:26.464 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:26 smithi088 systemd[1]: Started Ceph mon.c for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:47:27.466 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T05:47:27.466 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T05:47:27.709 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:47:31.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:47:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: mon.a calling monitor election 2024-03-12T05:47:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: mon.c calling monitor election 2024-03-12T05:47:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:31.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T05:47:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: monmap e2: 2 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],c=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:47:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: fsmap 2024-03-12T05:47:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:47:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: mgrmap e13: x(active, since 50s) 2024-03-12T05:47:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: overall HEALTH_OK 2024-03-12T05:47:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:47:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:31.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:47:31.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: mon.a calling monitor election 2024-03-12T05:47:31.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:31.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:31.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: mon.c calling monitor election 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: monmap e2: 2 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],c=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: fsmap 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: mgrmap e13: x(active, since 50s) 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: overall HEALTH_OK 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:31.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:31.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:31.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:47:31.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:32.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.023+0000 7f853bec3700 1 -- 172.21.15.88:0/119583524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f85200035c0 msgr2=0x7f8520005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:32.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.023+0000 7f853bec3700 1 --2- 172.21.15.88:0/119583524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f85200035c0 0x7f8520005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8534101680 tx=0x7f852400feb0 comp rx=0 tx=0).stop 2024-03-12T05:47:32.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.023+0000 7f853bec3700 1 -- 172.21.15.88:0/119583524 shutdown_connections 2024-03-12T05:47:32.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.023+0000 7f853bec3700 1 --2- 172.21.15.88:0/119583524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f85200035c0 0x7f8520005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:32.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.023+0000 7f853bec3700 1 -- 172.21.15.88:0/119583524 >> 172.21.15.88:0/119583524 conn(0x7f85340fd240 msgr2=0x7f85340ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:32.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.024+0000 7f853bec3700 1 -- 172.21.15.88:0/119583524 shutdown_connections 2024-03-12T05:47:32.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.024+0000 7f853bec3700 1 -- 172.21.15.88:0/119583524 wait complete. 2024-03-12T05:47:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.024+0000 7f853bec3700 1 Processor -- start 2024-03-12T05:47:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.024+0000 7f853bec3700 1 -- start start 2024-03-12T05:47:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f853bec3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85200035c0 0x7f85341970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f853bec3700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f85341975e0 0x7f853419b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f853bec3700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8534197b70 con 0x7f85341975e0 2024-03-12T05:47:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f853bec3700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8534197ce0 con 0x7f85200035c0 2024-03-12T05:47:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f8539c5f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85200035c0 0x7f85341970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f8539c5f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85200035c0 0x7f85341970a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52342/0 (socket says 172.21.15.88:52342) 2024-03-12T05:47:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f8539c5f700 1 -- 172.21.15.88:0/914038118 learned_addr learned my addr 172.21.15.88:0/914038118 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-03-12T05:47:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f8539c5f700 1 -- 172.21.15.88:0/914038118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85200035c0 msgr2=0x7f85341970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T05:47:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f8539c5f700 1 -- 172.21.15.88:0/914038118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85200035c0 msgr2=0x7f85341970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T05:47:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f8539c5f700 1 --2- 172.21.15.88:0/914038118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85200035c0 0x7f85341970a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T05:47:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f853945e700 1 --2- 172.21.15.88:0/914038118 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f85341975e0 0x7f853419b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f8539c5f700 1 --2- 172.21.15.88:0/914038118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85200035c0 0x7f85341970a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:47:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f853945e700 1 -- 172.21.15.88:0/914038118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85200035c0 msgr2=0x7f85341970a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:47:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f853945e700 1 --2- 172.21.15.88:0/914038118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85200035c0 0x7f85341970a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.025+0000 7f853945e700 1 -- 172.21.15.88:0/914038118 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852400f900 con 0x7f85341975e0 2024-03-12T05:47:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.026+0000 7f853945e700 1 --2- 172.21.15.88:0/914038118 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f85341975e0 0x7f853419b970 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f853000b4d0 tx=0x7f853000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.026+0000 7f852affd700 1 -- 172.21.15.88:0/914038118 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f8530011630 con 0x7f85341975e0 2024-03-12T05:47:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.026+0000 7f852affd700 1 -- 172.21.15.88:0/914038118 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8530011c70 con 0x7f85341975e0 2024-03-12T05:47:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.026+0000 7f853bec3700 1 -- 172.21.15.88:0/914038118 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853419bf10 con 0x7f85341975e0 2024-03-12T05:47:32.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.026+0000 7f852affd700 1 -- 172.21.15.88:0/914038118 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f8530010de0 con 0x7f85341975e0 2024-03-12T05:47:32.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.026+0000 7f853bec3700 1 -- 172.21.15.88:0/914038118 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853419c460 con 0x7f85341975e0 2024-03-12T05:47:32.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.027+0000 7f852affd700 1 -- 172.21.15.88:0/914038118 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f853001a400 con 0x7f85341975e0 2024-03-12T05:47:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.027+0000 7f852affd700 1 --2- 172.21.15.88:0/914038118 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8518038af0 0x7f851803afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.027+0000 7f853bec3700 1 -- 172.21.15.88:0/914038118 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8534190ff0 con 0x7f85341975e0 2024-03-12T05:47:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.027+0000 7f852affd700 1 -- 172.21.15.88:0/914038118 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f853004b5b0 con 0x7f85341975e0 2024-03-12T05:47:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.027+0000 7f8539c5f700 1 --2- 172.21.15.88:0/914038118 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8518038af0 0x7f851803afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.028+0000 7f8539c5f700 1 --2- 172.21.15.88:0/914038118 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8518038af0 0x7f851803afb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f852400f800 tx=0x7f8524005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:32.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.033+0000 7f852affd700 1 -- 172.21.15.88:0/914038118 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f85300193b0 con 0x7f85341975e0 2024-03-12T05:47:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.310+0000 7f853bec3700 1 -- 172.21.15.88:0/914038118 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8534198400 con 0x7f85341975e0 2024-03-12T05:47:32.312 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:32 smithi088 ceph-mon[36585]: Deploying daemon mon.b on smithi082 2024-03-12T05:47:32.312 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:32 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:32.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.310+0000 7f852affd700 1 -- 172.21.15.88:0/914038118 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1011 (secure 0 0 0) 0x7f853000fb70 con 0x7f85341975e0 2024-03-12T05:47:32.312 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:47:32.313 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":2,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","modified":"2024-03-12T05:47:26.069359Z","created":"2024-03-12T05:45:43.279953Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-12T05:47:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.313+0000 7f853bec3700 1 -- 172.21.15.88:0/914038118 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8518038af0 msgr2=0x7f851803afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.313+0000 7f853bec3700 1 --2- 172.21.15.88:0/914038118 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8518038af0 0x7f851803afb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f852400f800 tx=0x7f8524005c20 comp rx=0 tx=0).stop 2024-03-12T05:47:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.313+0000 7f853bec3700 1 -- 172.21.15.88:0/914038118 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f85341975e0 msgr2=0x7f853419b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.313+0000 7f853bec3700 1 --2- 172.21.15.88:0/914038118 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f85341975e0 0x7f853419b970 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f853000b4d0 tx=0x7f853000b890 comp rx=0 tx=0).stop 2024-03-12T05:47:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.314+0000 7f853bec3700 1 -- 172.21.15.88:0/914038118 shutdown_connections 2024-03-12T05:47:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.314+0000 7f853bec3700 1 --2- 172.21.15.88:0/914038118 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8518038af0 0x7f851803afb0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.314+0000 7f853bec3700 1 --2- 172.21.15.88:0/914038118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85200035c0 0x7f85341970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.314+0000 7f853bec3700 1 --2- 172.21.15.88:0/914038118 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f85341975e0 0x7f853419b970 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.314+0000 7f853bec3700 1 -- 172.21.15.88:0/914038118 >> 172.21.15.88:0/914038118 conn(0x7f85340fd240 msgr2=0x7f85340ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.314+0000 7f853bec3700 1 -- 172.21.15.88:0/914038118 shutdown_connections 2024-03-12T05:47:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:32.314+0000 7f853bec3700 1 -- 172.21.15.88:0/914038118 wait complete. 2024-03-12T05:47:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 2 2024-03-12T05:47:32.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:32 smithi076 ceph-mon[32095]: Deploying daemon mon.b on smithi082 2024-03-12T05:47:32.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:32 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:33.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:33 smithi088 ceph-mon[36585]: from='client.? 172.21.15.88:0/914038118' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T05:47:33.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:33 smithi088 ceph-mon[36585]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:33.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:33 smithi076 ceph-mon[32095]: from='client.? 172.21.15.88:0/914038118' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T05:47:33.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:33 smithi076 ceph-mon[32095]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:33.846 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T05:47:33.847 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T05:47:34.088 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:47:35.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:35 smithi082 systemd[1]: Started Ceph mon.b for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:47:40.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:47:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: mon.a calling monitor election 2024-03-12T05:47:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: mon.c calling monitor election 2024-03-12T05:47:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T05:47:40.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: monmap e3: 3 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0],c=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:47:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: fsmap 2024-03-12T05:47:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:47:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: mgrmap e13: x(active, since 59s) 2024-03-12T05:47:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-12T05:47:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-12T05:47:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-12T05:47:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: mon.b (rank 2) addr [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] is down (out of quorum) 2024-03-12T05:47:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:40 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:40.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:47:40.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: mon.a calling monitor election 2024-03-12T05:47:40.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: mon.c calling monitor election 2024-03-12T05:47:40.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:40.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:40.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:40.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:40.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:40.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:40.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:40.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:40.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:40.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T05:47:40.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: monmap e3: 3 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0],c=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:47:40.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: fsmap 2024-03-12T05:47:40.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:47:40.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: mgrmap e13: x(active, since 59s) 2024-03-12T05:47:40.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-12T05:47:40.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-12T05:47:40.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-12T05:47:40.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: mon.b (rank 2) addr [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] is down (out of quorum) 2024-03-12T05:47:40.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:40.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:40.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:40.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:41.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:41.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:41 smithi088 ceph-mon[36585]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:41.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:41.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:41.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.410+0000 7f21e435c700 1 -- 172.21.15.88:0/797036999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c8006240 msgr2=0x7f21c80086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:41.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.410+0000 7f21e435c700 1 --2- 172.21.15.88:0/797036999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c8006240 0x7f21c80086b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0695c0 tx=0x7f21d800fe90 comp rx=0 tx=0).stop 2024-03-12T05:47:41.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.410+0000 7f21e435c700 1 -- 172.21.15.88:0/797036999 shutdown_connections 2024-03-12T05:47:41.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.410+0000 7f21e435c700 1 --2- 172.21.15.88:0/797036999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c8006240 0x7f21c80086b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:41.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.410+0000 7f21e435c700 1 -- 172.21.15.88:0/797036999 >> 172.21.15.88:0/797036999 conn(0x7f21dc076c60 msgr2=0x7f21dc077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:41.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.410+0000 7f21e435c700 1 -- 172.21.15.88:0/797036999 shutdown_connections 2024-03-12T05:47:41.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.410+0000 7f21e435c700 1 -- 172.21.15.88:0/797036999 wait complete. 2024-03-12T05:47:41.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.411+0000 7f21e435c700 1 Processor -- start 2024-03-12T05:47:41.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.411+0000 7f21e435c700 1 -- start start 2024-03-12T05:47:41.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.411+0000 7f21e435c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c8006240 0x7f21dc199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:41.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.411+0000 7f21e435c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21dc199a50 0x7f21dc19dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:41.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.411+0000 7f21e435c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21dc19e320 0x7f21dc19e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.411+0000 7f21e435c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc19a140 con 0x7f21c8006240 2024-03-12T05:47:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.411+0000 7f21e435c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc1a6090 con 0x7f21dc19e320 2024-03-12T05:47:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.411+0000 7f21e435c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc1a61d0 con 0x7f21dc199a50 2024-03-12T05:47:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.411+0000 7f21e18f7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21dc199a50 0x7f21dc19dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.411+0000 7f21e18f7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21dc199a50 0x7f21dc19dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43800/0 (socket says 172.21.15.88:43800) 2024-03-12T05:47:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.411+0000 7f21e18f7700 1 -- 172.21.15.88:0/3501062944 learned_addr learned my addr 172.21.15.88:0/3501062944 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-03-12T05:47:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.411+0000 7f21e20f8700 1 --2- 172.21.15.88:0/3501062944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c8006240 0x7f21dc199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.412+0000 7f21e18f7700 1 -- 172.21.15.88:0/3501062944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21dc19e320 msgr2=0x7f21dc19e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.412+0000 7f21e18f7700 1 --2- 172.21.15.88:0/3501062944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21dc19e320 0x7f21dc19e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.412+0000 7f21e18f7700 1 -- 172.21.15.88:0/3501062944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c8006240 msgr2=0x7f21dc199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.412+0000 7f21e18f7700 1 --2- 172.21.15.88:0/3501062944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c8006240 0x7f21dc199510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.412+0000 7f21e18f7700 1 -- 172.21.15.88:0/3501062944 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d800f8e0 con 0x7f21dc199a50 2024-03-12T05:47:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.412+0000 7f21e18f7700 1 --2- 172.21.15.88:0/3501062944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21dc199a50 0x7f21dc19dde0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f21cc00b780 tx=0x7f21cc00ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:41.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.412+0000 7f21d37fe700 1 -- 172.21.15.88:0/3501062944 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc004230 con 0x7f21dc199a50 2024-03-12T05:47:41.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.412+0000 7f21e435c700 1 -- 172.21.15.88:0/3501062944 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21dc1a64b0 con 0x7f21dc199a50 2024-03-12T05:47:41.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.412+0000 7f21e435c700 1 -- 172.21.15.88:0/3501062944 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21dc1a6a00 con 0x7f21dc199a50 2024-03-12T05:47:41.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.412+0000 7f21d37fe700 1 -- 172.21.15.88:0/3501062944 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21cc004390 con 0x7f21dc199a50 2024-03-12T05:47:41.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.412+0000 7f21d37fe700 1 -- 172.21.15.88:0/3501062944 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc011b90 con 0x7f21dc199a50 2024-03-12T05:47:41.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.413+0000 7f21e435c700 1 -- 172.21.15.88:0/3501062944 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c80057b0 con 0x7f21dc199a50 2024-03-12T05:47:41.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.415+0000 7f21d37fe700 1 -- 172.21.15.88:0/3501062944 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f21cc0036a0 con 0x7f21dc199a50 2024-03-12T05:47:41.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.416+0000 7f21d37fe700 1 --2- 172.21.15.88:0/3501062944 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21c00391e0 0x7f21c003b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:41.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.416+0000 7f21d37fe700 1 -- 172.21.15.88:0/3501062944 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f21cc04b8b0 con 0x7f21dc199a50 2024-03-12T05:47:41.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.416+0000 7f21e20f8700 1 --2- 172.21.15.88:0/3501062944 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21c00391e0 0x7f21c003b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:41.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.417+0000 7f21e20f8700 1 --2- 172.21.15.88:0/3501062944 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21c00391e0 0x7f21c003b6a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f21d8005b40 tx=0x7f21d8005ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:41.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.420+0000 7f21d37fe700 1 -- 172.21.15.88:0/3501062944 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f21cc018940 con 0x7f21dc199a50 2024-03-12T05:47:41.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:41.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:41 smithi076 ceph-mon[32095]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:41.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:41.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:41.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.707+0000 7f21e435c700 1 -- 172.21.15.88:0/3501062944 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21c8005a10 con 0x7f21dc199a50 2024-03-12T05:47:41.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.707+0000 7f21d37fe700 1 -- 172.21.15.88:0/3501062944 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1275 (secure 0 0 0) 0x7f21cc018350 con 0x7f21dc199a50 2024-03-12T05:47:41.709 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-03-12T05:47:41.709 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":3,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","modified":"2024-03-12T05:47:34.981872Z","created":"2024-03-12T05:45:43.279953Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-12T05:47:41.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.710+0000 7f21e435c700 1 -- 172.21.15.88:0/3501062944 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21c00391e0 msgr2=0x7f21c003b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:41.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.710+0000 7f21e435c700 1 --2- 172.21.15.88:0/3501062944 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21c00391e0 0x7f21c003b6a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f21d8005b40 tx=0x7f21d8005ad0 comp rx=0 tx=0).stop 2024-03-12T05:47:41.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.710+0000 7f21e435c700 1 -- 172.21.15.88:0/3501062944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21dc199a50 msgr2=0x7f21dc19dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:41.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.710+0000 7f21e435c700 1 --2- 172.21.15.88:0/3501062944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21dc199a50 0x7f21dc19dde0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f21cc00b780 tx=0x7f21cc00ba90 comp rx=0 tx=0).stop 2024-03-12T05:47:41.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.711+0000 7f21e435c700 1 -- 172.21.15.88:0/3501062944 shutdown_connections 2024-03-12T05:47:41.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.711+0000 7f21e435c700 1 --2- 172.21.15.88:0/3501062944 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21c00391e0 0x7f21c003b6a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:41.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.711+0000 7f21e435c700 1 --2- 172.21.15.88:0/3501062944 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21dc19e320 0x7f21dc19e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:41.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.711+0000 7f21e435c700 1 --2- 172.21.15.88:0/3501062944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c8006240 0x7f21dc199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:41.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.711+0000 7f21e435c700 1 --2- 172.21.15.88:0/3501062944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21dc199a50 0x7f21dc19dde0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:41.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.711+0000 7f21e435c700 1 -- 172.21.15.88:0/3501062944 >> 172.21.15.88:0/3501062944 conn(0x7f21dc076c60 msgr2=0x7f21dc0ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:41.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.711+0000 7f21e435c700 1 -- 172.21.15.88:0/3501062944 shutdown_connections 2024-03-12T05:47:41.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:41.711+0000 7f21e435c700 1 -- 172.21.15.88:0/3501062944 wait complete. 2024-03-12T05:47:41.713 INFO:teuthology.orchestra.run.smithi088.stderr:dumped monmap epoch 3 2024-03-12T05:47:42.336 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-12T05:47:42.337 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-12T05:47:42.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:47:42.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: mon.a calling monitor election 2024-03-12T05:47:42.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: mon.c calling monitor election 2024-03-12T05:47:42.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:42.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:47:42.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: monmap e3: 3 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0],c=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: fsmap 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: mgrmap e13: x(active, since 59s) 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: mon.b (rank 2) addr [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] is down (out of quorum) 2024-03-12T05:47:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:42.580 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:47:43.379 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: mon.b calling monitor election 2024-03-12T05:47:43.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: mon.b calling monitor election 2024-03-12T05:47:43.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: mon.c calling monitor election 2024-03-12T05:47:43.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: mon.a calling monitor election 2024-03-12T05:47:43.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T05:47:43.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: monmap e3: 3 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0],c=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:47:43.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: fsmap 2024-03-12T05:47:43.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:47:43.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: mgrmap e13: x(active, since 61s) 2024-03-12T05:47:43.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-12T05:47:43.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: Cluster is now healthy 2024-03-12T05:47:43.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: overall HEALTH_OK 2024-03-12T05:47:43.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: Reconfiguring daemon mon.a on smithi076 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:43.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: mon.b calling monitor election 2024-03-12T05:47:43.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: mon.b calling monitor election 2024-03-12T05:47:43.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: mon.c calling monitor election 2024-03-12T05:47:43.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: mon.a calling monitor election 2024-03-12T05:47:43.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: monmap e3: 3 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0],c=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: fsmap 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: mgrmap e13: x(active, since 61s) 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: Cluster is now healthy 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: overall HEALTH_OK 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T05:47:43.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:47:43.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:47:43.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:43.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: Reconfiguring daemon mon.a on smithi076 2024-03-12T05:47:43.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:43.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:43.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: mon.b calling monitor election 2024-03-12T05:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: mon.b calling monitor election 2024-03-12T05:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: mon.c calling monitor election 2024-03-12T05:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: mon.a calling monitor election 2024-03-12T05:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T05:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: monmap e3: 3 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0],c=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T05:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: fsmap 2024-03-12T05:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T05:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: mgrmap e13: x(active, since 61s) 2024-03-12T05:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-12T05:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: Cluster is now healthy 2024-03-12T05:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: overall HEALTH_OK 2024-03-12T05:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T05:47:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:47:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:47:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: Reconfiguring daemon mon.a on smithi076 2024-03-12T05:47:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:47:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.763+0000 7fe5c5f96700 1 -- 172.21.15.76:0/4280844968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5c0107650 msgr2=0x7fe5c0107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.763+0000 7fe5c5f96700 1 --2- 172.21.15.76:0/4280844968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5c0107650 0x7fe5c0107a30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800b0a0 tx=0x7fe5b800b3b0 comp rx=0 tx=0).stop 2024-03-12T05:47:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.763+0000 7fe5c5f96700 1 -- 172.21.15.76:0/4280844968 shutdown_connections 2024-03-12T05:47:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.763+0000 7fe5c5f96700 1 --2- 172.21.15.76:0/4280844968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5c0107650 0x7fe5c0107a30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.763+0000 7fe5c5f96700 1 -- 172.21.15.76:0/4280844968 >> 172.21.15.76:0/4280844968 conn(0x7fe5c00fd160 msgr2=0x7fe5c00ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.763+0000 7fe5c5f96700 1 -- 172.21.15.76:0/4280844968 shutdown_connections 2024-03-12T05:47:44.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.763+0000 7fe5c5f96700 1 -- 172.21.15.76:0/4280844968 wait complete. 2024-03-12T05:47:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.763+0000 7fe5c5f96700 1 Processor -- start 2024-03-12T05:47:44.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5c5f96700 1 -- start start 2024-03-12T05:47:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5c5f96700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe5c0107650 0x7fe5c01993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5c5f96700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5c0199930 0x7fe5c019dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5c5f96700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5c019e200 0x7fe5c019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5c5f96700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c019a020 con 0x7fe5c019e200 2024-03-12T05:47:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5c5f96700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c01a5f70 con 0x7fe5c0107650 2024-03-12T05:47:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5c5f96700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c01a60b0 con 0x7fe5c0199930 2024-03-12T05:47:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5bffff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5c019e200 0x7fe5c019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5bffff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5c019e200 0x7fe5c019e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51456/0 (socket says 172.21.15.76:51456) 2024-03-12T05:47:44.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5bffff700 1 -- 172.21.15.76:0/3287166845 learned_addr learned my addr 172.21.15.76:0/3287166845 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:47:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5beffd700 1 --2- 172.21.15.76:0/3287166845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5c0199930 0x7fe5c019dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5bf7fe700 1 --2- 172.21.15.76:0/3287166845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe5c0107650 0x7fe5c01993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5bffff700 1 -- 172.21.15.76:0/3287166845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5c0199930 msgr2=0x7fe5c019dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5bffff700 1 --2- 172.21.15.76:0/3287166845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5c0199930 0x7fe5c019dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5bffff700 1 -- 172.21.15.76:0/3287166845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe5c0107650 msgr2=0x7fe5c01993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5bffff700 1 --2- 172.21.15.76:0/3287166845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe5c0107650 0x7fe5c01993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5bffff700 1 -- 172.21.15.76:0/3287166845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b8009d00 con 0x7fe5c019e200 2024-03-12T05:47:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.764+0000 7fe5bffff700 1 --2- 172.21.15.76:0/3287166845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5c019e200 0x7fe5c019e6b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe5b000b550 tx=0x7fe5b000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.765+0000 7fe5bcff9700 1 -- 172.21.15.76:0/3287166845 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0011830 con 0x7fe5c019e200 2024-03-12T05:47:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.765+0000 7fe5bcff9700 1 -- 172.21.15.76:0/3287166845 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5b0011e70 con 0x7fe5c019e200 2024-03-12T05:47:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.765+0000 7fe5bcff9700 1 -- 172.21.15.76:0/3287166845 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b000f2e0 con 0x7fe5c019e200 2024-03-12T05:47:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.765+0000 7fe5c5f96700 1 -- 172.21.15.76:0/3287166845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c01a6390 con 0x7fe5c019e200 2024-03-12T05:47:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.765+0000 7fe5c5f96700 1 -- 172.21.15.76:0/3287166845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c01a6840 con 0x7fe5c019e200 2024-03-12T05:47:44.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.766+0000 7fe5bcff9700 1 -- 172.21.15.76:0/3287166845 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7fe5b000f440 con 0x7fe5c019e200 2024-03-12T05:47:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.766+0000 7fe5c5f96700 1 -- 172.21.15.76:0/3287166845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c00632a0 con 0x7fe5c019e200 2024-03-12T05:47:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.766+0000 7fe5bcff9700 1 --2- 172.21.15.76:0/3287166845 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe5ac0391e0 0x7fe5ac03b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.766+0000 7fe5bcff9700 1 -- 172.21.15.76:0/3287166845 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe5b004b690 con 0x7fe5c019e200 2024-03-12T05:47:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.766+0000 7fe5bf7fe700 1 --2- 172.21.15.76:0/3287166845 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe5ac0391e0 0x7fe5ac03b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:44.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.766+0000 7fe5bf7fe700 1 --2- 172.21.15.76:0/3287166845 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe5ac0391e0 0x7fe5ac03b6a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800b830 tx=0x7fe5b8005040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:44.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.771+0000 7fe5bcff9700 1 -- 172.21.15.76:0/3287166845 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe5b0017890 con 0x7fe5c019e200 2024-03-12T05:47:44.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.967+0000 7fe5c5f96700 1 -- 172.21.15.76:0/3287166845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fe5c010ba40 con 0x7fe5c019e200 2024-03-12T05:47:44.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.968+0000 7fe5bcff9700 1 -- 172.21.15.76:0/3287166845 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+271 (secure 0 0 0) 0x7fe5b0015030 con 0x7fe5c019e200 2024-03-12T05:47:44.970 INFO:teuthology.orchestra.run.smithi076.stdout:# minimal ceph.conf for 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:47:44.970 INFO:teuthology.orchestra.run.smithi076.stdout:[global] 2024-03-12T05:47:44.970 INFO:teuthology.orchestra.run.smithi076.stdout: fsid = 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:47:44.970 INFO:teuthology.orchestra.run.smithi076.stdout: mon_host = [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2024-03-12T05:47:44.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.969+0000 7fe5c5f96700 1 -- 172.21.15.76:0/3287166845 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe5ac0391e0 msgr2=0x7fe5ac03b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:44.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.969+0000 7fe5c5f96700 1 --2- 172.21.15.76:0/3287166845 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe5ac0391e0 0x7fe5ac03b6a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800b830 tx=0x7fe5b8005040 comp rx=0 tx=0).stop 2024-03-12T05:47:44.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.969+0000 7fe5c5f96700 1 -- 172.21.15.76:0/3287166845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5c019e200 msgr2=0x7fe5c019e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:44.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.969+0000 7fe5c5f96700 1 --2- 172.21.15.76:0/3287166845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5c019e200 0x7fe5c019e6b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe5b000b550 tx=0x7fe5b000b910 comp rx=0 tx=0).stop 2024-03-12T05:47:44.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.969+0000 7fe5c5f96700 1 -- 172.21.15.76:0/3287166845 shutdown_connections 2024-03-12T05:47:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.969+0000 7fe5c5f96700 1 --2- 172.21.15.76:0/3287166845 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe5ac0391e0 0x7fe5ac03b6a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.969+0000 7fe5c5f96700 1 --2- 172.21.15.76:0/3287166845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe5c019e200 0x7fe5c019e6b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.969+0000 7fe5c5f96700 1 --2- 172.21.15.76:0/3287166845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe5c0107650 0x7fe5c01993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.969+0000 7fe5c5f96700 1 --2- 172.21.15.76:0/3287166845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5c0199930 0x7fe5c019dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.970+0000 7fe5c5f96700 1 -- 172.21.15.76:0/3287166845 >> 172.21.15.76:0/3287166845 conn(0x7fe5c00fd160 msgr2=0x7fe5c0109190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.970+0000 7fe5c5f96700 1 -- 172.21.15.76:0/3287166845 shutdown_connections 2024-03-12T05:47:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:44.970+0000 7fe5c5f96700 1 -- 172.21.15.76:0/3287166845 wait complete. 2024-03-12T05:47:45.713 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-12T05:47:45.713 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:47:45.713 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T05:47:45.742 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:47:45.743 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:47:45.809 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-12T05:47:45.810 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T05:47:45.838 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-12T05:47:45.838 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:47:45.905 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-12T05:47:45.905 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T05:47:45.940 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-12T05:47:45.940 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:47:45.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:45 smithi076 ceph-mon[32095]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:45.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:45 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3287166845' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:45.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:45 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:45.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:45 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:45.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:45 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:47:45.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:45 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:47:45.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:45 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:46.011 INFO:tasks.cephadm:Adding mgr.x on smithi076 2024-03-12T05:47:46.011 INFO:tasks.cephadm:Adding mgr.y on smithi082 2024-03-12T05:47:46.011 INFO:tasks.cephadm:Adding mgr.z on smithi088 2024-03-12T05:47:46.011 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch apply mgr '3;smithi076=x;smithi082=y;smithi088=z' 2024-03-12T05:47:46.057 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:45 smithi082 ceph-mon[36382]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:46.057 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:45 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3287166845' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:46.057 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:45 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:46.058 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:45 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:46.058 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:45 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:47:46.058 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:45 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:47:46.058 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:45 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:46.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:45 smithi088 ceph-mon[36585]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:46.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:45 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3287166845' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:46.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:45 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:46.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:45 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:46.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:45 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:47:46.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:45 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:47:46.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:45 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:46.284 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:47:46.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:46 smithi082 ceph-mon[36382]: Reconfiguring mon.b (monmap changed)... 2024-03-12T05:47:46.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:46 smithi082 ceph-mon[36382]: Reconfiguring daemon mon.b on smithi082 2024-03-12T05:47:46.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:46 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:46.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:46 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:46.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:46 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:47:46.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:46 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:47:46.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:46 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:46 smithi088 ceph-mon[36585]: Reconfiguring mon.b (monmap changed)... 2024-03-12T05:47:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:46 smithi088 ceph-mon[36585]: Reconfiguring daemon mon.b on smithi082 2024-03-12T05:47:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:46 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:46 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:46 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:47:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:46 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:47:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:46 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:47.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:46 smithi076 ceph-mon[32095]: Reconfiguring mon.b (monmap changed)... 2024-03-12T05:47:47.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:46 smithi076 ceph-mon[32095]: Reconfiguring daemon mon.b on smithi082 2024-03-12T05:47:47.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:46 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:47.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:46 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:47.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:46 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T05:47:47.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:46 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T05:47:47.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:46 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:47.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:47 smithi088 ceph-mon[36585]: Reconfiguring mon.c (monmap changed)... 2024-03-12T05:47:47.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:47 smithi088 ceph-mon[36585]: Reconfiguring daemon mon.c on smithi088 2024-03-12T05:47:47.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:47 smithi088 ceph-mon[36585]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:48.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:47 smithi082 ceph-mon[36382]: Reconfiguring mon.c (monmap changed)... 2024-03-12T05:47:48.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:47 smithi082 ceph-mon[36382]: Reconfiguring daemon mon.c on smithi088 2024-03-12T05:47:48.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:47 smithi082 ceph-mon[36382]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:48.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:47 smithi076 ceph-mon[32095]: Reconfiguring mon.c (monmap changed)... 2024-03-12T05:47:48.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:47 smithi076 ceph-mon[32095]: Reconfiguring daemon mon.c on smithi088 2024-03-12T05:47:48.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:47 smithi076 ceph-mon[32095]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:48.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.288+0000 7ff62e9ad700 1 -- 172.21.15.88:0/1485444698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff628100690 msgr2=0x7ff628100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:48.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.288+0000 7ff62e9ad700 1 --2- 172.21.15.88:0/1485444698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff628100690 0x7ff628100ab0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff618009910 tx=0x7ff618009c20 comp rx=0 tx=0).stop 2024-03-12T05:47:48.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.288+0000 7ff62e9ad700 1 -- 172.21.15.88:0/1485444698 shutdown_connections 2024-03-12T05:47:48.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.288+0000 7ff62e9ad700 1 --2- 172.21.15.88:0/1485444698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff628100690 0x7ff628100ab0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:48.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.288+0000 7ff62e9ad700 1 -- 172.21.15.88:0/1485444698 >> 172.21.15.88:0/1485444698 conn(0x7ff6280fbc50 msgr2=0x7ff6280fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:48.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.288+0000 7ff62e9ad700 1 -- 172.21.15.88:0/1485444698 shutdown_connections 2024-03-12T05:47:48.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.288+0000 7ff62e9ad700 1 -- 172.21.15.88:0/1485444698 wait complete. 2024-03-12T05:47:48.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.289+0000 7ff62e9ad700 1 Processor -- start 2024-03-12T05:47:48.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.289+0000 7ff62e9ad700 1 -- start start 2024-03-12T05:47:48.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.289+0000 7ff62e9ad700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff628100690 0x7ff628195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:48.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.289+0000 7ff62e9ad700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff628196260 0x7ff62819b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:48.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.289+0000 7ff62e9ad700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6281966e0 0x7ff62819d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:48.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.289+0000 7ff62e9ad700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62819e010 con 0x7ff628196260 2024-03-12T05:47:48.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.289+0000 7ff62e9ad700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62819e180 con 0x7ff6281966e0 2024-03-12T05:47:48.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.289+0000 7ff62e9ad700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62819e340 con 0x7ff628100690 2024-03-12T05:47:48.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.289+0000 7ff627fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff628100690 0x7ff628195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:48.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.289+0000 7ff627fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff628100690 0x7ff628195d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56718/0 (socket says 172.21.15.88:56718) 2024-03-12T05:47:48.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.289+0000 7ff627fff700 1 -- 172.21.15.88:0/218102525 learned_addr learned my addr 172.21.15.88:0/218102525 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-03-12T05:47:48.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.289+0000 7ff6277fe700 1 --2- 172.21.15.88:0/218102525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff628196260 0x7ff62819b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:48.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.290+0000 7ff627fff700 1 -- 172.21.15.88:0/218102525 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6281966e0 msgr2=0x7ff62819d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:48.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.290+0000 7ff62cf4a700 1 --2- 172.21.15.88:0/218102525 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6281966e0 0x7ff62819d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:48.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.290+0000 7ff627fff700 1 --2- 172.21.15.88:0/218102525 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6281966e0 0x7ff62819d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:48.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.290+0000 7ff627fff700 1 -- 172.21.15.88:0/218102525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff628196260 msgr2=0x7ff62819b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:48.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.290+0000 7ff627fff700 1 --2- 172.21.15.88:0/218102525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff628196260 0x7ff62819b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:48.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.290+0000 7ff627fff700 1 -- 172.21.15.88:0/218102525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6180095f0 con 0x7ff628100690 2024-03-12T05:47:48.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.290+0000 7ff627fff700 1 --2- 172.21.15.88:0/218102525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff628100690 0x7ff628195d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff61800bcf0 tx=0x7ff61800bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:48.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.290+0000 7ff6257fa700 1 -- 172.21.15.88:0/218102525 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61801d070 con 0x7ff628100690 2024-03-12T05:47:48.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.290+0000 7ff62e9ad700 1 -- 172.21.15.88:0/218102525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6280695c0 con 0x7ff628100690 2024-03-12T05:47:48.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.290+0000 7ff6257fa700 1 -- 172.21.15.88:0/218102525 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff618004ec0 con 0x7ff628100690 2024-03-12T05:47:48.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.290+0000 7ff6257fa700 1 -- 172.21.15.88:0/218102525 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff618022e30 con 0x7ff628100690 2024-03-12T05:47:48.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.290+0000 7ff62e9ad700 1 -- 172.21.15.88:0/218102525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff628069ae0 con 0x7ff628100690 2024-03-12T05:47:48.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.291+0000 7ff6257fa700 1 -- 172.21.15.88:0/218102525 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7ff61800f460 con 0x7ff628100690 2024-03-12T05:47:48.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.291+0000 7ff62e9ad700 1 -- 172.21.15.88:0/218102525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff628106dd0 con 0x7ff628100690 2024-03-12T05:47:48.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.291+0000 7ff6257fa700 1 --2- 172.21.15.88:0/218102525 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff6100391e0 0x7ff61003b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:48.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.291+0000 7ff6257fa700 1 -- 172.21.15.88:0/218102525 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff61804ccc0 con 0x7ff628100690 2024-03-12T05:47:48.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.291+0000 7ff6277fe700 1 --2- 172.21.15.88:0/218102525 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff6100391e0 0x7ff61003b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:48.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.292+0000 7ff6277fe700 1 --2- 172.21.15.88:0/218102525 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff6100391e0 0x7ff61003b6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009ad0 tx=0x7ff61c009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:48.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.297+0000 7ff6257fa700 1 -- 172.21.15.88:0/218102525 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff618020960 con 0x7ff628100690 2024-03-12T05:47:48.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.505+0000 7ff62e9ad700 1 -- 172.21.15.88:0/218102525 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi076=x;smithi082=y;smithi088=z", "target": ["mon-mgr", ""]}) v1 -- 0x7ff62818eff0 con 0x7ff6100391e0 2024-03-12T05:47:48.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.511+0000 7ff6257fa700 1 -- 172.21.15.88:0/218102525 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff62818eff0 con 0x7ff6100391e0 2024-03-12T05:47:48.513 INFO:teuthology.orchestra.run.smithi088.stdout:Scheduled mgr update... 2024-03-12T05:47:48.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.513+0000 7ff62e9ad700 1 -- 172.21.15.88:0/218102525 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff6100391e0 msgr2=0x7ff61003b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:48.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.513+0000 7ff62e9ad700 1 --2- 172.21.15.88:0/218102525 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff6100391e0 0x7ff61003b6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009ad0 tx=0x7ff61c009210 comp rx=0 tx=0).stop 2024-03-12T05:47:48.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.513+0000 7ff62e9ad700 1 -- 172.21.15.88:0/218102525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff628100690 msgr2=0x7ff628195d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:48.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.513+0000 7ff62e9ad700 1 --2- 172.21.15.88:0/218102525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff628100690 0x7ff628195d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff61800bcf0 tx=0x7ff61800bd20 comp rx=0 tx=0).stop 2024-03-12T05:47:48.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.513+0000 7ff62e9ad700 1 -- 172.21.15.88:0/218102525 shutdown_connections 2024-03-12T05:47:48.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.513+0000 7ff62e9ad700 1 --2- 172.21.15.88:0/218102525 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff6100391e0 0x7ff61003b6a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:48.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.513+0000 7ff62e9ad700 1 --2- 172.21.15.88:0/218102525 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6281966e0 0x7ff62819d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:48.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.513+0000 7ff62e9ad700 1 --2- 172.21.15.88:0/218102525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff628100690 0x7ff628195d20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:48.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.513+0000 7ff62e9ad700 1 --2- 172.21.15.88:0/218102525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff628196260 0x7ff62819b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:48.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.513+0000 7ff62e9ad700 1 -- 172.21.15.88:0/218102525 >> 172.21.15.88:0/218102525 conn(0x7ff6280fbc50 msgr2=0x7ff6280fc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:48.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.513+0000 7ff62e9ad700 1 -- 172.21.15.88:0/218102525 shutdown_connections 2024-03-12T05:47:48.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:47:48.513+0000 7ff62e9ad700 1 -- 172.21.15.88:0/218102525 wait complete. 2024-03-12T05:47:49.356 DEBUG:teuthology.orchestra.run.smithi082:mgr.y> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mgr.y.service 2024-03-12T05:47:49.358 DEBUG:teuthology.orchestra.run.smithi088:mgr.z> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mgr.z.service 2024-03-12T05:47:49.361 INFO:tasks.cephadm:Deploying OSDs... 2024-03-12T05:47:49.361 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:47:49.361 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T05:47:49.376 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T05:47:49.376 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_1 2024-03-12T05:47:49.431 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T05:47:49.431 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:47:49.431 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-12T05:47:49.432 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:47:49.432 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:47:49.432 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-03-12 05:46:53.395553811 +0000 2024-03-12T05:47:49.432 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-03-12 05:37:45.501334884 +0000 2024-03-12T05:47:49.432 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-03-12 05:37:45.501334884 +0000 2024-03-12T05:47:49.432 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-03-12 05:37:45.501334884 +0000 2024-03-12T05:47:49.432 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T05:47:49.497 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-03-12T05:47:49.497 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-03-12T05:47:49.497 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000192183 s, 2.7 MB/s 2024-03-12T05:47:49.498 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T05:47:49.554 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_2 2024-03-12T05:47:49.609 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T05:47:49.609 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:47:49.609 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-12T05:47:49.609 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:47:49.609 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:47:49.609 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-03-12 05:46:53.425552869 +0000 2024-03-12T05:47:49.609 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-03-12 05:37:45.856323859 +0000 2024-03-12T05:47:49.609 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-03-12 05:37:45.856323859 +0000 2024-03-12T05:47:49.609 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-03-12 05:37:45.856323859 +0000 2024-03-12T05:47:49.609 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T05:47:49.675 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-03-12T05:47:49.675 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-03-12T05:47:49.675 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000220326 s, 2.3 MB/s 2024-03-12T05:47:49.676 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T05:47:49.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:49 smithi082 ceph-mon[36382]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi076=x;smithi082=y;smithi088=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:47:49.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:49 smithi082 ceph-mon[36382]: Saving service mgr spec with placement smithi076=x;smithi082=y;smithi088=z;count:3 2024-03-12T05:47:49.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:49.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:49 smithi082 ceph-mon[36382]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:49.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:49.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:49.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:49.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:49.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:49.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:49.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:47:49.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T05:47:49.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:47:49.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:49.731 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_3 2024-03-12T05:47:49.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:49 smithi088 ceph-mon[36585]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi076=x;smithi082=y;smithi088=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:47:49.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:49 smithi088 ceph-mon[36585]: Saving service mgr spec with placement smithi076=x;smithi082=y;smithi088=z;count:3 2024-03-12T05:47:49.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:49.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:49 smithi088 ceph-mon[36585]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:49.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:49.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:49.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:49.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:49.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:49.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:49.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:47:49.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T05:47:49.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:47:49.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:49.786 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T05:47:49.786 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:47:49.786 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-12T05:47:49.786 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:47:49.786 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:47:49.786 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-03-12 05:46:53.453551991 +0000 2024-03-12T05:47:49.786 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-03-12 05:37:46.172314045 +0000 2024-03-12T05:47:49.786 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-03-12 05:37:46.172314045 +0000 2024-03-12T05:47:49.786 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-03-12 05:37:46.172314045 +0000 2024-03-12T05:47:49.787 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T05:47:49.849 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:49 smithi076 ceph-mon[32095]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi076=x;smithi082=y;smithi088=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:47:49.849 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:49 smithi076 ceph-mon[32095]: Saving service mgr spec with placement smithi076=x;smithi082=y;smithi088=z;count:3 2024-03-12T05:47:49.850 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:49.850 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:49 smithi076 ceph-mon[32095]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:49.850 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:49.850 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:49.850 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:49.850 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:49.850 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:49.850 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:49.850 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:47:49.850 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T05:47:49.850 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:47:49.850 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:49.851 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-03-12T05:47:49.851 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-03-12T05:47:49.851 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000216158 s, 2.4 MB/s 2024-03-12T05:47:49.853 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T05:47:49.909 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_4 2024-03-12T05:47:49.963 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T05:47:49.964 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:47:49.964 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-12T05:47:49.964 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:47:49.964 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:47:49.964 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-03-12 05:46:53.480551143 +0000 2024-03-12T05:47:49.964 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-03-12 05:37:46.521303206 +0000 2024-03-12T05:47:49.964 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-03-12 05:37:46.521303206 +0000 2024-03-12T05:47:49.964 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-03-12 05:37:46.521303206 +0000 2024-03-12T05:47:49.964 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T05:47:50.029 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-03-12T05:47:50.029 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-03-12T05:47:50.030 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000207827 s, 2.5 MB/s 2024-03-12T05:47:50.031 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T05:47:50.086 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-12T05:47:50.087 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T05:47:50.105 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T05:47:50.105 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2024-03-12T05:47:50.163 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T05:47:50.163 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:47:50.163 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 660 Links: 1 2024-03-12T05:47:50.163 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:47:50.163 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:47:50.163 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-12 05:47:34.438168857 +0000 2024-03-12T05:47:50.163 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-12 05:38:12.477050069 +0000 2024-03-12T05:47:50.163 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-12 05:38:12.477050069 +0000 2024-03-12T05:47:50.163 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-12 05:38:12.477050069 +0000 2024-03-12T05:47:50.163 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T05:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-12T05:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-12T05:47:50.233 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000277863 s, 1.8 MB/s 2024-03-12T05:47:50.235 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T05:47:50.294 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2024-03-12T05:47:50.350 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T05:47:50.350 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:47:50.350 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-12T05:47:50.350 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:47:50.350 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:47:50.350 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-12 05:47:34.439168838 +0000 2024-03-12T05:47:50.351 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-12 05:38:12.806043536 +0000 2024-03-12T05:47:50.351 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-12 05:38:12.806043536 +0000 2024-03-12T05:47:50.351 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-12 05:38:12.806043536 +0000 2024-03-12T05:47:50.351 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T05:47:50.419 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-12T05:47:50.419 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-12T05:47:50.419 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000234406 s, 2.2 MB/s 2024-03-12T05:47:50.420 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T05:47:50.477 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2024-03-12T05:47:50.534 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T05:47:50.534 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:47:50.534 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 698 Links: 1 2024-03-12T05:47:50.534 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:47:50.534 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:47:50.534 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-12 05:47:34.439168838 +0000 2024-03-12T05:47:50.534 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-12 05:38:13.141036883 +0000 2024-03-12T05:47:50.534 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-12 05:38:13.141036883 +0000 2024-03-12T05:47:50.534 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-12 05:38:13.141036883 +0000 2024-03-12T05:47:50.534 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T05:47:50.602 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-12T05:47:50.602 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-12T05:47:50.602 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000223142 s, 2.3 MB/s 2024-03-12T05:47:50.603 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T05:47:50.660 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2024-03-12T05:47:50.716 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T05:47:50.716 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:47:50.716 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-12T05:47:50.716 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:47:50.716 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:47:50.716 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-12 05:47:34.439168838 +0000 2024-03-12T05:47:50.716 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-12 05:38:13.496029833 +0000 2024-03-12T05:47:50.716 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-12 05:38:13.496029833 +0000 2024-03-12T05:47:50.716 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-03-12 05:38:13.496029833 +0000 2024-03-12T05:47:50.716 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T05:47:50.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:50 smithi088 ceph-mon[36585]: Deploying daemon mgr.z on smithi088 2024-03-12T05:47:50.780 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:50 smithi082 ceph-mon[36382]: Deploying daemon mgr.z on smithi088 2024-03-12T05:47:50.783 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-12T05:47:50.783 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-12T05:47:50.783 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000210692 s, 2.4 MB/s 2024-03-12T05:47:50.784 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T05:47:50.840 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-03-12T05:47:50.840 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T05:47:50.855 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T05:47:50.855 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2024-03-12T05:47:50.910 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T05:47:50.910 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:47:50.910 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 660 Links: 1 2024-03-12T05:47:50.910 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:47:50.910 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:47:50.910 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-03-12 05:47:25.338438088 +0000 2024-03-12T05:47:50.910 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-03-12 05:38:28.932539334 +0000 2024-03-12T05:47:50.910 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-03-12 05:38:28.932539334 +0000 2024-03-12T05:47:50.910 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-03-12 05:38:28.932539334 +0000 2024-03-12T05:47:50.911 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T05:47:50.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:50 smithi076 ceph-mon[32095]: Deploying daemon mgr.z on smithi088 2024-03-12T05:47:51.087 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-03-12T05:47:51.087 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-03-12T05:47:51.087 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000269121 s, 1.9 MB/s 2024-03-12T05:47:51.088 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T05:47:51.103 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2024-03-12T05:47:51.159 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T05:47:51.159 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:47:51.159 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-12T05:47:51.159 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:47:51.159 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:47:51.159 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-03-12 05:47:51.026965537 +0000 2024-03-12T05:47:51.159 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-03-12 05:38:29.283532746 +0000 2024-03-12T05:47:51.159 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-03-12 05:38:29.283532746 +0000 2024-03-12T05:47:51.159 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-03-12 05:38:29.283532746 +0000 2024-03-12T05:47:51.159 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T05:47:51.273 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-03-12T05:47:51.273 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-03-12T05:47:51.274 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000203016 s, 2.5 MB/s 2024-03-12T05:47:51.275 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T05:47:51.290 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2024-03-12T05:47:51.344 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T05:47:51.345 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:47:51.345 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-12T05:47:51.345 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:47:51.345 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:47:51.345 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-03-12 05:47:51.216962039 +0000 2024-03-12T05:47:51.345 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-03-12 05:38:29.626526309 +0000 2024-03-12T05:47:51.345 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-03-12 05:38:29.626526309 +0000 2024-03-12T05:47:51.345 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-03-12 05:38:29.626526309 +0000 2024-03-12T05:47:51.345 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T05:47:51.409 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:51 smithi088 systemd[1]: Starting Ceph mgr.z for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T05:47:51.412 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-03-12T05:47:51.412 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-03-12T05:47:51.412 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000171162 s, 3.0 MB/s 2024-03-12T05:47:51.413 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T05:47:51.468 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2024-03-12T05:47:51.523 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T05:47:51.523 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T05:47:51.523 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-12T05:47:51.523 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:47:51.523 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T05:47:51.523 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-03-12 05:47:51.216962039 +0000 2024-03-12T05:47:51.523 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-03-12 05:38:29.959520059 +0000 2024-03-12T05:47:51.523 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-03-12 05:38:29.959520059 +0000 2024-03-12T05:47:51.523 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-03-12 05:38:29.959520059 +0000 2024-03-12T05:47:51.523 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T05:47:51.589 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-03-12T05:47:51.589 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-03-12T05:47:51.589 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000213369 s, 2.4 MB/s 2024-03-12T05:47:51.591 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T05:47:51.646 INFO:tasks.cephadm:Deploying osd.0 on smithi076 with /dev/vg_nvme/lv_4... 2024-03-12T05:47:51.646 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-12T05:47:51.663 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:51 smithi088 ceph-mon[36585]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:51.664 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:51 smithi088 podman[38323]: 2024-03-12 05:47:51.397354555 +0000 UTC m=+0.018182036 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:47:51.664 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:51 smithi088 podman[38323]: 2024-03-12 05:47:51.529116557 +0000 UTC m=+0.149944037 container create 89c69d18a0a82cae00bac66580d89f934463b0e73b4397d1f9e230acd919bec3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-12T05:47:51.899 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:47:51.942 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:51 smithi082 ceph-mon[36382]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:51.982 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:51 smithi088 podman[38323]: 2024-03-12 05:47:51.723402118 +0000 UTC m=+0.344229589 container init 89c69d18a0a82cae00bac66580d89f934463b0e73b4397d1f9e230acd919bec3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-12T05:47:51.982 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:51 smithi088 podman[38323]: 2024-03-12 05:47:51.725990817 +0000 UTC m=+0.346818288 container start 89c69d18a0a82cae00bac66580d89f934463b0e73b4397d1f9e230acd919bec3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-12T05:47:51.982 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:51 smithi088 ceph-mgr[38431]: -- 172.21.15.88:0/3277846356 <== mon.1 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b907c781a0 con 0x55b907c4ec00 2024-03-12T05:47:51.982 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:51 smithi088 bash[38323]: 89c69d18a0a82cae00bac66580d89f934463b0e73b4397d1f9e230acd919bec3 2024-03-12T05:47:51.982 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:51 smithi088 systemd[1]: Started Ceph mgr.z for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:47:51.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:51 smithi076 ceph-mon[32095]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:52.732 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:52 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:52.269+0000 7efe382e5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:47:52.732 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:52 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:52.363+0000 7efe382e5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:47:53.056 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:52 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:53.056 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:52 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:53.056 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:52 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:53.056 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:52 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:47:53.056 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:52 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T05:47:53.056 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:52 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:47:53.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:52 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:53.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:52 smithi076 ceph-mon[32095]: Deploying daemon mgr.y on smithi082 2024-03-12T05:47:53.143 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:53.009+0000 7efe382e5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:47:53.143 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:53.105+0000 7efe382e5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:47:53.144 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:52 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:53.144 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:52 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:53.144 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:52 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:53.144 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:52 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:47:53.144 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:52 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T05:47:53.144 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:52 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:47:53.144 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:52 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:53.144 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:52 smithi088 ceph-mon[36585]: Deploying daemon mgr.y on smithi082 2024-03-12T05:47:53.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:52 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:53.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:52 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:53.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:52 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:53.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:52 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:47:53.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:52 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T05:47:53.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:52 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:47:53.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:52 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:53.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:52 smithi082 ceph-mon[36382]: Deploying daemon mgr.y on smithi082 2024-03-12T05:47:53.441 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:53.254+0000 7efe382e5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:47:53.441 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:53.353+0000 7efe382e5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:47:53.441 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:53.439+0000 7efe382e5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:47:53.690 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-12T05:47:53.691 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-12T05:47:53.691 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-03-12T05:47:53.691 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-03-12T05:47:53.691 INFO:teuthology.orchestra.run.smithi076.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113867 s, 921 MB/s 2024-03-12T05:47:53.691 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 2024-03-12T05:47:53.691 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-03-12T05:47:53.902 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:53 smithi082 systemd[1]: Starting Ceph mgr.y for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T05:47:53.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:53 smithi088 ceph-mon[36585]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:53.982 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:53.710+0000 7efe382e5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:47:53.982 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:53.822+0000 7efe382e5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:47:53.982 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:53.910+0000 7efe382e5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:47:53.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:53 smithi076 ceph-mon[32095]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:54.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:53 smithi082 ceph-mon[36382]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:54.193 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:53 smithi082 podman[37744]: 2024-03-12 05:47:53.816812929 +0000 UTC m=+0.017429551 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:47:54.193 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:53 smithi082 podman[37744]: 2024-03-12 05:47:53.933492886 +0000 UTC m=+0.134109493 container create 60fa6851228615ba2ec618fdcfd123a583a03cd05ade0ada32358ffee74e2c43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-12T05:47:54.193 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:54 smithi082 podman[37744]: 2024-03-12 05:47:54.119026144 +0000 UTC m=+0.319642752 container init 60fa6851228615ba2ec618fdcfd123a583a03cd05ade0ada32358ffee74e2c43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-12T05:47:54.193 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:54 smithi082 podman[37744]: 2024-03-12 05:47:54.121956722 +0000 UTC m=+0.322573340 container start 60fa6851228615ba2ec618fdcfd123a583a03cd05ade0ada32358ffee74e2c43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-12T05:47:54.193 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:54 smithi082 ceph-mgr[37764]: -- 172.21.15.82:0/556661826 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e2affb21a0 con 0x55e2aff88c00 2024-03-12T05:47:54.318 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi076:vg_nvme/lv_4 2024-03-12T05:47:54.468 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:54 smithi082 bash[37744]: 60fa6851228615ba2ec618fdcfd123a583a03cd05ade0ada32358ffee74e2c43 2024-03-12T05:47:54.468 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:54 smithi082 systemd[1]: Started Ceph mgr.y for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:47:54.482 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:54 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:54.007+0000 7efe382e5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:47:54.482 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:54 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:54.188+0000 7efe382e5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:47:54.557 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:47:54.806 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:54 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:54.707+0000 7f730213d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:47:54.806 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:54 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:54.804+0000 7f730213d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:47:55.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:55.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:55.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:55.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:55.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:55.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:55 smithi082 ceph-mon[36382]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:55.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:55.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:55.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:55.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:55.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:55 smithi088 ceph-mon[36585]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:55.483 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:55 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:55.198+0000 7efe382e5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:47:55.483 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:55 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:55.335+0000 7efe382e5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:47:55.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:55.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:55.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:55.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:55.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:55.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:55 smithi076 ceph-mon[32095]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:55.827 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:55 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:55.487+0000 7f730213d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:47:55.827 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:55 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:55.578+0000 7f730213d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:47:55.827 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:55 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:55.726+0000 7f730213d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:47:55.827 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:55 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:55.825+0000 7f730213d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:47:55.972 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:55 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:55.680+0000 7efe382e5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:47:55.972 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:55 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:55.775+0000 7efe382e5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:47:55.972 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:55 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:55.970+0000 7efe382e5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:47:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.092+0000 7f65d5f67700 1 -- 172.21.15.76:0/2931307088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f65d0102770 msgr2=0x7f65d0109020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.092+0000 7f65d5f67700 1 --2- 172.21.15.76:0/2931307088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f65d0102770 0x7f65d0109020 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f65c800b0d0 tx=0x7f65c800b3e0 comp rx=0 tx=0).stop 2024-03-12T05:47:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.092+0000 7f65d5f67700 1 -- 172.21.15.76:0/2931307088 shutdown_connections 2024-03-12T05:47:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.092+0000 7f65d5f67700 1 --2- 172.21.15.76:0/2931307088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f65d0102770 0x7f65d0109020 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.092+0000 7f65d5f67700 1 --2- 172.21.15.76:0/2931307088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65d0100bd0 0x7f65d0100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.092+0000 7f65d5f67700 1 --2- 172.21.15.76:0/2931307088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65d0101dd0 0x7f65d0102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.092+0000 7f65d5f67700 1 -- 172.21.15.76:0/2931307088 >> 172.21.15.76:0/2931307088 conn(0x7f65d00fc170 msgr2=0x7f65d00fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:47:56.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.092+0000 7f65d5f67700 1 -- 172.21.15.76:0/2931307088 shutdown_connections 2024-03-12T05:47:56.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.092+0000 7f65d5f67700 1 -- 172.21.15.76:0/2931307088 wait complete. 2024-03-12T05:47:56.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.093+0000 7f65d5f67700 1 Processor -- start 2024-03-12T05:47:56.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.093+0000 7f65d5f67700 1 -- start start 2024-03-12T05:47:56.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.093+0000 7f65d5f67700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65d0100bd0 0x7f65d006ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:56.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.093+0000 7f65d5f67700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f65d0101dd0 0x7f65d019ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:56.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.093+0000 7f65d5f67700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65d0102770 0x7f65d019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:56.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.093+0000 7f65d5f67700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d019f800 con 0x7f65d0101dd0 2024-03-12T05:47:56.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.093+0000 7f65d5f67700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d019f970 con 0x7f65d0100bd0 2024-03-12T05:47:56.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.093+0000 7f65d5f67700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d019fb30 con 0x7f65d0102770 2024-03-12T05:47:56.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.093+0000 7f65ceffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f65d0101dd0 0x7f65d019ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:56.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.093+0000 7f65ceffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f65d0101dd0 0x7f65d019ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57730/0 (socket says 172.21.15.76:57730) 2024-03-12T05:47:56.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.093+0000 7f65cf7fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65d0100bd0 0x7f65d006ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:56.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.093+0000 7f65ceffd700 1 -- 172.21.15.76:0/1269462343 learned_addr learned my addr 172.21.15.76:0/1269462343 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:47:56.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.094+0000 7f65cffff700 1 --2- 172.21.15.76:0/1269462343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65d0102770 0x7f65d019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:56.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.094+0000 7f65cf7fe700 1 -- 172.21.15.76:0/1269462343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65d0102770 msgr2=0x7f65d019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:56.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.094+0000 7f65cf7fe700 1 --2- 172.21.15.76:0/1269462343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65d0102770 0x7f65d019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:56.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.094+0000 7f65cf7fe700 1 -- 172.21.15.76:0/1269462343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f65d0101dd0 msgr2=0x7f65d019ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:47:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.094+0000 7f65cf7fe700 1 --2- 172.21.15.76:0/1269462343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f65d0101dd0 0x7f65d019ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:47:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.094+0000 7f65cf7fe700 1 -- 172.21.15.76:0/1269462343 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c4009520 con 0x7f65d0100bd0 2024-03-12T05:47:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.094+0000 7f65cf7fe700 1 --2- 172.21.15.76:0/1269462343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65d0100bd0 0x7f65d006ae70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f65c400e9c0 tx=0x7f65c400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.094+0000 7f65ccff9700 1 -- 172.21.15.76:0/1269462343 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c400cb70 con 0x7f65d0100bd0 2024-03-12T05:47:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.094+0000 7f65ccff9700 1 -- 172.21.15.76:0/1269462343 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65c400ee60 con 0x7f65d0100bd0 2024-03-12T05:47:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.094+0000 7f65d5f67700 1 -- 172.21.15.76:0/1269462343 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c8009d00 con 0x7f65d0100bd0 2024-03-12T05:47:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.094+0000 7f65ccff9700 1 -- 172.21.15.76:0/1269462343 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4018680 con 0x7f65d0100bd0 2024-03-12T05:47:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.094+0000 7f65d5f67700 1 -- 172.21.15.76:0/1269462343 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d01a0180 con 0x7f65d0100bd0 2024-03-12T05:47:56.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.095+0000 7f65ccff9700 1 -- 172.21.15.76:0/1269462343 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) v1 ==== 45825+0+0 (secure 0 0 0) 0x7f65c40187e0 con 0x7f65d0100bd0 2024-03-12T05:47:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.095+0000 7f65d5f67700 1 -- 172.21.15.76:0/1269462343 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d0107ec0 con 0x7f65d0100bd0 2024-03-12T05:47:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.095+0000 7f65ccff9700 1 --2- 172.21.15.76:0/1269462343 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f65b8038f90 0x7f65b803b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:47:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.095+0000 7f65ccff9700 1 -- 172.21.15.76:0/1269462343 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f65c4014070 con 0x7f65d0100bd0 2024-03-12T05:47:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.095+0000 7f65ceffd700 1 --2- 172.21.15.76:0/1269462343 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f65b8038f90 0x7f65b803b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:47:56.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.096+0000 7f65ceffd700 1 --2- 172.21.15.76:0/1269462343 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f65b8038f90 0x7f65b803b450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f65d006b3f0 tx=0x7f65c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:47:56.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.101+0000 7f65ccff9700 1 -- 172.21.15.76:0/1269462343 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f65c4016da0 con 0x7f65d0100bd0 2024-03-12T05:47:56.171 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:55 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:55.908+0000 7f730213d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:47:56.171 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:56.169+0000 7f730213d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:47:56.232 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:56.155+0000 7efe382e5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:47:56.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:56.314+0000 7f65d5f67700 1 -- 172.21.15.76:0/1269462343 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f65d002d8d0 con 0x7f65b8038f90 2024-03-12T05:47:56.443 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:56.277+0000 7f730213d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:47:56.443 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:56.361+0000 7f730213d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:47:56.695 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:56.438+0000 7efe382e5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:47:56.943 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:56.453+0000 7f730213d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:47:56.943 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:56.627+0000 7f730213d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:47:56.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:56.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:56.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: Reconfiguring mgr.x (unknown last config time)... 2024-03-12T05:47:56.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:47:56.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:47:56.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:56.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: Reconfiguring daemon mgr.x on smithi076 2024-03-12T05:47:56.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:47:56.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:47:56.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:56.982 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:56.696+0000 7efe382e5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:47:56.982 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:56.929+0000 7efe382e5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:47:56.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: Reconfiguring mgr.x (unknown last config time)... 2024-03-12T05:47:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:47:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:47:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: Reconfiguring daemon mgr.x on smithi076 2024-03-12T05:47:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:47:56.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:47:56.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:56.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:56.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:56.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: Reconfiguring mgr.x (unknown last config time)... 2024-03-12T05:47:56.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T05:47:56.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T05:47:56.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:56.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: Reconfiguring daemon mgr.x on smithi076 2024-03-12T05:47:56.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:56.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:47:56.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:47:56.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:57.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:57 smithi076 ceph-mon[32095]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:47:57.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:57 smithi076 ceph-mon[32095]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:57.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:57.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:57.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:57.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:57.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:57.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:57.943 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:57 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:57.587+0000 7f730213d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:47:57.943 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:57 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:57.716+0000 7f730213d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:47:57.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:57 smithi082 ceph-mon[36382]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:47:57.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:57 smithi082 ceph-mon[36382]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:57.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:57 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:57.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:57 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:57.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:57 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:57.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:57 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:57.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:57 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:57.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:57 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:57.982 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:57 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:57.722+0000 7efe382e5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:47:57.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:57 smithi088 ceph-mon[36585]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:47:57.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:57 smithi088 ceph-mon[36585]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:57.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:57 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:57.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:57 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:57.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:57 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:47:57.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:57 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:47:57.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:57 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:47:57.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:57 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:47:58.324 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:58 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:58.043+0000 7f730213d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:47:58.325 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:58 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:58.130+0000 7f730213d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:47:58.325 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:58 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:58.322+0000 7f730213d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:47:58.584 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:58 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:58.278+0000 7efe382e5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:47:58.584 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:58 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:58.373+0000 7efe382e5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:47:58.584 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:47:58 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:47:58.582+0000 7efe382e5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:47:58.693 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:58 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:58.503+0000 7f730213d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:47:58.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:47:58.707+0000 7f65ccff9700 1 -- 172.21.15.76:0/1269462343 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mgrmap(e 14) v1 ==== 91024+0+0 (secure 0 0 0) 0x7f65c4017cb0 con 0x7f65d0100bd0 2024-03-12T05:47:58.949 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:58 smithi076 ceph-mon[32095]: Standby manager daemon z started 2024-03-12T05:47:58.949 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:58 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.88:0/901910802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T05:47:58.950 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:58 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.88:0/901910802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:47:58.950 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:58 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.88:0/901910802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T05:47:58.950 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:58 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.88:0/901910802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:47:58.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:58 smithi088 ceph-mon[36585]: Standby manager daemon z started 2024-03-12T05:47:58.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:58 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.88:0/901910802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T05:47:58.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:58 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.88:0/901910802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:47:58.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:58 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.88:0/901910802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T05:47:58.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:58 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.88:0/901910802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:47:59.037 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:58 smithi082 ceph-mon[36382]: Standby manager daemon z started 2024-03-12T05:47:59.037 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:58 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.88:0/901910802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T05:47:59.037 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:58 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.88:0/901910802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:47:59.037 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:58 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.88:0/901910802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T05:47:59.038 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:58 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.88:0/901910802' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:47:59.038 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:58 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:58.780+0000 7f730213d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:47:59.038 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:59 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:59.035+0000 7f730213d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:47:59.443 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:47:59 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:47:59.266+0000 7f730213d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:47:59.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:59 smithi088 ceph-mon[36585]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-12T05:47:59.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:59 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T05:47:59.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:59 smithi088 ceph-mon[36585]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:59.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:59 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/486680148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5faf2b7d-916a-4077-8da3-06bba2a002ab"}]: dispatch 2024-03-12T05:47:59.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:59 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/486680148' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5faf2b7d-916a-4077-8da3-06bba2a002ab"}]': finished 2024-03-12T05:47:59.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:59 smithi088 ceph-mon[36585]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T05:47:59.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:47:59 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:47:59.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:59 smithi076 ceph-mon[32095]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-12T05:47:59.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:59 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T05:47:59.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:59 smithi076 ceph-mon[32095]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:47:59.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:59 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/486680148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5faf2b7d-916a-4077-8da3-06bba2a002ab"}]: dispatch 2024-03-12T05:47:59.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:59 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/486680148' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5faf2b7d-916a-4077-8da3-06bba2a002ab"}]': finished 2024-03-12T05:47:59.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:59 smithi076 ceph-mon[32095]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T05:47:59.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:47:59 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:00.056 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:59 smithi082 ceph-mon[36382]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-12T05:48:00.056 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:59 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T05:48:00.056 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:59 smithi082 ceph-mon[36382]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:00.056 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:59 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/486680148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5faf2b7d-916a-4077-8da3-06bba2a002ab"}]: dispatch 2024-03-12T05:48:00.056 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:59 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/486680148' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5faf2b7d-916a-4077-8da3-06bba2a002ab"}]': finished 2024-03-12T05:48:00.056 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:59 smithi082 ceph-mon[36382]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T05:48:00.057 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:47:59 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:00.443 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:48:00 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:48:00.054+0000 7f730213d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:48:00.926 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:00 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3835510842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:48:00.927 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:48:00 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:48:00.618+0000 7f730213d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:48:00.927 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:48:00 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:48:00.714+0000 7f730213d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:48:00.927 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:48:00 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:48:00.924+0000 7f730213d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:48:00.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:00 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3835510842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:48:00.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:00 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3835510842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:48:01.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:48:01.724+0000 7f65ccff9700 1 -- 172.21.15.76:0/1269462343 <== mon.2 v2:172.21.15.82:3300/0 8 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7f65c4016330 con 0x7f65d0100bd0 2024-03-12T05:48:01.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:01 smithi088 ceph-mon[36585]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:01.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:01 smithi088 ceph-mon[36585]: Standby manager daemon y started 2024-03-12T05:48:01.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:01 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.82:0/2360445886' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T05:48:01.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:01 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.82:0/2360445886' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:48:01.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:01 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.82:0/2360445886' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T05:48:01.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:01 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.82:0/2360445886' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:48:01.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:01 smithi076 ceph-mon[32095]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:01.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:01 smithi076 ceph-mon[32095]: Standby manager daemon y started 2024-03-12T05:48:01.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:01 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.82:0/2360445886' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T05:48:01.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:01 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.82:0/2360445886' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:48:01.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:01 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.82:0/2360445886' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T05:48:01.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:01 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.82:0/2360445886' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:48:02.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:01 smithi082 ceph-mon[36382]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:02.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:01 smithi082 ceph-mon[36382]: Standby manager daemon y started 2024-03-12T05:48:02.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:01 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.82:0/2360445886' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T05:48:02.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:01 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.82:0/2360445886' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:48:02.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:01 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.82:0/2360445886' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T05:48:02.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:01 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.82:0/2360445886' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:48:02.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:02 smithi088 ceph-mon[36585]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-12T05:48:02.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:02 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T05:48:02.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:02 smithi076 ceph-mon[32095]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-12T05:48:02.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:02 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T05:48:03.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:02 smithi082 ceph-mon[36382]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-12T05:48:03.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:02 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T05:48:03.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:03 smithi076 ceph-mon[32095]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:03.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:03 smithi088 ceph-mon[36585]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:04.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:03 smithi082 ceph-mon[36382]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:05.916 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:05 smithi076 ceph-mon[32095]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:06.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:05 smithi082 ceph-mon[36382]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:06.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:05 smithi088 ceph-mon[36585]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:08.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:07 smithi082 ceph-mon[36382]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:08.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T05:48:08.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:08.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:07 smithi082 ceph-mon[36382]: Deploying daemon osd.0 on smithi076 2024-03-12T05:48:08.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:07 smithi088 ceph-mon[36585]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:08.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T05:48:08.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:08.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:07 smithi088 ceph-mon[36585]: Deploying daemon osd.0 on smithi076 2024-03-12T05:48:08.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:07 smithi076 ceph-mon[32095]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:08.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T05:48:08.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:08.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:07 smithi076 ceph-mon[32095]: Deploying daemon osd.0 on smithi076 2024-03-12T05:48:10.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:09 smithi076 ceph-mon[32095]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:10.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:09 smithi082 ceph-mon[36382]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:10.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:09 smithi088 ceph-mon[36585]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:11.982 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:11 smithi076 ceph-mon[32095]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:12.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:11 smithi082 ceph-mon[36382]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:12.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:11 smithi088 ceph-mon[36585]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:13.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:13.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:13.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:48:13.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:13 smithi076 ceph-mon[32095]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:13.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:13.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:13.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:48:13.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:13 smithi082 ceph-mon[36382]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:13.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:13.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:48:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:13 smithi088 ceph-mon[36585]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:15.956 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:15 smithi076 ceph-mon[32095]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:15.956 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:15 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:15.956 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:15 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:15.956 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:15 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:15.957 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:15 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:48:15.957 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:15 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:16.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:15 smithi082 ceph-mon[36382]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:16.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:15 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:16.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:15 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:16.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:15 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:16.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:15 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:48:16.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:15 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:16.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:15 smithi088 ceph-mon[36585]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:16.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:15 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:16.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:15 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:16.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:15 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:16.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:15 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:48:16.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:15 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:16.273 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 0 on host 'smithi076' 2024-03-12T05:48:16.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:48:16.270+0000 7f65ccff9700 1 -- 172.21.15.76:0/1269462343 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f65d002d8d0 con 0x7f65b8038f90 2024-03-12T05:48:16.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:48:16.272+0000 7f65d5f67700 1 -- 172.21.15.76:0/1269462343 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f65b8038f90 msgr2=0x7f65b803b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:48:16.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:48:16.272+0000 7f65d5f67700 1 --2- 172.21.15.76:0/1269462343 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f65b8038f90 0x7f65b803b450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f65d006b3f0 tx=0x7f65c0009260 comp rx=0 tx=0).stop 2024-03-12T05:48:16.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:48:16.272+0000 7f65d5f67700 1 -- 172.21.15.76:0/1269462343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65d0100bd0 msgr2=0x7f65d006ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:48:16.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:48:16.272+0000 7f65d5f67700 1 --2- 172.21.15.76:0/1269462343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65d0100bd0 0x7f65d006ae70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f65c400e9c0 tx=0x7f65c400ecd0 comp rx=0 tx=0).stop 2024-03-12T05:48:16.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:48:16.272+0000 7f65d5f67700 1 -- 172.21.15.76:0/1269462343 shutdown_connections 2024-03-12T05:48:16.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:48:16.272+0000 7f65d5f67700 1 --2- 172.21.15.76:0/1269462343 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f65b8038f90 0x7f65b803b450 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:16.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:48:16.272+0000 7f65d5f67700 1 --2- 172.21.15.76:0/1269462343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65d0102770 0x7f65d019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:16.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:48:16.272+0000 7f65d5f67700 1 --2- 172.21.15.76:0/1269462343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65d0100bd0 0x7f65d006ae70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:16.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:48:16.272+0000 7f65d5f67700 1 --2- 172.21.15.76:0/1269462343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f65d0101dd0 0x7f65d019ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:16.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:48:16.272+0000 7f65d5f67700 1 -- 172.21.15.76:0/1269462343 >> 172.21.15.76:0/1269462343 conn(0x7f65d00fc170 msgr2=0x7f65d0105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:48:16.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:48:16.272+0000 7f65d5f67700 1 -- 172.21.15.76:0/1269462343 shutdown_connections 2024-03-12T05:48:16.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:48:16.272+0000 7f65d5f67700 1 -- 172.21.15.76:0/1269462343 wait complete. 2024-03-12T05:48:16.871 DEBUG:teuthology.orchestra.run.smithi076:osd.0> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.0.service 2024-03-12T05:48:16.873 INFO:tasks.cephadm:Deploying osd.1 on smithi082 with /dev/vg_nvme/lv_4... 2024-03-12T05:48:16.873 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-12T05:48:17.116 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:48:17.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:17 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:17.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:17 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:17.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:17 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:48:17.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:17 smithi076 ceph-mon[32095]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T05:48:17.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:17 smithi076 ceph-mon[32095]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:17.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:17 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:17.522 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:17 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:17.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:17 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:48:17.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:17 smithi082 ceph-mon[36382]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T05:48:17.523 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:17 smithi082 ceph-mon[36382]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:17.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:17 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:17.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:17 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:17.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:17 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:48:17.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:17 smithi088 ceph-mon[36585]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T05:48:17.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:17 smithi088 ceph-mon[36585]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:18.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:18 smithi082 ceph-mon[36382]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T05:48:18.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:18 smithi082 ceph-mon[36382]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T05:48:18.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:18 smithi082 ceph-mon[36382]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-12T05:48:18.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:18 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:18.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:18 smithi088 ceph-mon[36585]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T05:48:18.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:18 smithi088 ceph-mon[36585]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T05:48:18.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:18 smithi088 ceph-mon[36585]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-12T05:48:18.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:18 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:18.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:18 smithi076 ceph-mon[32095]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T05:48:18.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:18 smithi076 ceph-mon[32095]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T05:48:18.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:18 smithi076 ceph-mon[32095]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-12T05:48:18.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:18 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:18.891 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-12T05:48:18.891 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-12T05:48:18.891 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-03-12T05:48:18.891 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-03-12T05:48:18.891 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128045 s, 819 MB/s 2024-03-12T05:48:18.891 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-03-12T05:48:19.482 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi082:vg_nvme/lv_4 2024-03-12T05:48:19.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:19 smithi076 ceph-mon[32095]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-12T05:48:19.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:19 smithi076 ceph-mon[32095]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T05:48:19.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:19.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:19.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:19 smithi076 ceph-mon[32095]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:19.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:19 smithi076 ceph-mon[32095]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' 2024-03-12T05:48:19.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:19.487 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 05:48:19 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-0[46549]: 2024-03-12T05:48:19.105+0000 7f367650b700 -1 osd.0 0 waiting for initial osdmap 2024-03-12T05:48:19.487 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 05:48:19 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-0[46549]: 2024-03-12T05:48:19.114+0000 7f366eaf7700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:48:19.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:19 smithi082 ceph-mon[36382]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-12T05:48:19.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:19 smithi082 ceph-mon[36382]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T05:48:19.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:19.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:19.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:19 smithi082 ceph-mon[36382]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:19.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:19 smithi082 ceph-mon[36382]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' 2024-03-12T05:48:19.493 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:19.724 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:48:19.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:19 smithi088 ceph-mon[36585]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-12T05:48:19.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:19 smithi088 ceph-mon[36585]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T05:48:19.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:19.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:19.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:19 smithi088 ceph-mon[36585]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T05:48:19.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:19 smithi088 ceph-mon[36585]: from='osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805]' entity='osd.0' 2024-03-12T05:48:19.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:20.634 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:20 smithi088 ceph-mon[36585]: purged_snaps scrub starts 2024-03-12T05:48:20.634 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:20 smithi088 ceph-mon[36585]: purged_snaps scrub ok 2024-03-12T05:48:20.634 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:20 smithi088 ceph-mon[36585]: Detected new or changed devices on smithi076 2024-03-12T05:48:20.634 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:20.634 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:48:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:20 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi076 to 17088M 2024-03-12T05:48:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:48:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:20 smithi088 ceph-mon[36585]: osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805] boot 2024-03-12T05:48:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:20 smithi088 ceph-mon[36585]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T05:48:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:20.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:20 smithi082 ceph-mon[36382]: purged_snaps scrub starts 2024-03-12T05:48:20.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:20 smithi082 ceph-mon[36382]: purged_snaps scrub ok 2024-03-12T05:48:20.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:20 smithi082 ceph-mon[36382]: Detected new or changed devices on smithi076 2024-03-12T05:48:20.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:48:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:20 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi076 to 17088M 2024-03-12T05:48:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:48:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:20 smithi082 ceph-mon[36382]: osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805] boot 2024-03-12T05:48:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:20 smithi082 ceph-mon[36382]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T05:48:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:20.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:20 smithi076 ceph-mon[32095]: purged_snaps scrub starts 2024-03-12T05:48:20.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:20 smithi076 ceph-mon[32095]: purged_snaps scrub ok 2024-03-12T05:48:20.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:20 smithi076 ceph-mon[32095]: Detected new or changed devices on smithi076 2024-03-12T05:48:20.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:20.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:20.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:48:20.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:20 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi076 to 17088M 2024-03-12T05:48:20.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:20.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:20.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:48:20.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:20.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:20 smithi076 ceph-mon[32095]: osd.0 [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805] boot 2024-03-12T05:48:20.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:20 smithi076 ceph-mon[32095]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T05:48:20.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:48:21.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.250+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/1016252785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e14107a20 msgr2=0x7f9e14109e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:48:21.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.250+0000 7f9e1a0e0700 1 --2- 172.21.15.82:0/1016252785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e14107a20 0x7f9e14109e30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00b0a0 tx=0x7f9e0c00b3b0 comp rx=0 tx=0).stop 2024-03-12T05:48:21.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.250+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/1016252785 shutdown_connections 2024-03-12T05:48:21.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.250+0000 7f9e1a0e0700 1 --2- 172.21.15.82:0/1016252785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e14107a20 0x7f9e14109e30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:21.254 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.250+0000 7f9e1a0e0700 1 --2- 172.21.15.82:0/1016252785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e141027c0 0x7f9e14104bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.250+0000 7f9e1a0e0700 1 --2- 172.21.15.82:0/1016252785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e141050f0 0x7f9e141074e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.250+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/1016252785 >> 172.21.15.82:0/1016252785 conn(0x7f9e140fc190 msgr2=0x7f9e140fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.250+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/1016252785 shutdown_connections 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.250+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/1016252785 wait complete. 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.250+0000 7f9e1a0e0700 1 Processor -- start 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.250+0000 7f9e1a0e0700 1 -- start start 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e1a0e0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e141027c0 0x7f9e14102120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e1a0e0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e141050f0 0x7f9e14100770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e1a0e0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e14107a20 0x7f9e14100cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e1a0e0700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e14101430 con 0x7f9e14107a20 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e1a0e0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e141015a0 con 0x7f9e141050f0 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e1a0e0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e141a1dc0 con 0x7f9e141027c0 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e12ffd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e141050f0 0x7f9e14100770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e137fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e141027c0 0x7f9e14102120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e137fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e141027c0 0x7f9e14102120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.82:44440/0 (socket says 172.21.15.82:44440) 2024-03-12T05:48:21.255 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e137fe700 1 -- 172.21.15.82:0/2665192428 learned_addr learned my addr 172.21.15.82:0/2665192428 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-03-12T05:48:21.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e12ffd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e141050f0 0x7f9e14100770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:51990/0 (socket says 172.21.15.82:51990) 2024-03-12T05:48:21.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e13fff700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e14107a20 0x7f9e14100cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:48:21.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e137fe700 1 -- 172.21.15.82:0/2665192428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e141050f0 msgr2=0x7f9e14100770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:48:21.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e137fe700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e141050f0 0x7f9e14100770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:21.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e137fe700 1 -- 172.21.15.82:0/2665192428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e14107a20 msgr2=0x7f9e14100cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:48:21.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e137fe700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e14107a20 0x7f9e14100cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:21.256 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e137fe700 1 -- 172.21.15.82:0/2665192428 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c009d00 con 0x7f9e141027c0 2024-03-12T05:48:21.257 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e12ffd700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e141050f0 0x7f9e14100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:48:21.257 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e13fff700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e14107a20 0x7f9e14100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:48:21.257 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e137fe700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e141027c0 0x7f9e14102120 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400d980 tx=0x7f9e0400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:48:21.257 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e10ff9700 1 -- 172.21.15.82:0/2665192428 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04004a30 con 0x7f9e141027c0 2024-03-12T05:48:21.257 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e10ff9700 1 -- 172.21.15.82:0/2665192428 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e04005500 con 0x7f9e141027c0 2024-03-12T05:48:21.257 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/2665192428 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e141a20a0 con 0x7f9e141027c0 2024-03-12T05:48:21.257 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.251+0000 7f9e10ff9700 1 -- 172.21.15.82:0/2665192428 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04009d90 con 0x7f9e141027c0 2024-03-12T05:48:21.258 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.252+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/2665192428 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e141a25a0 con 0x7f9e141027c0 2024-03-12T05:48:21.258 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.252+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/2665192428 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e140fdd10 con 0x7f9e141027c0 2024-03-12T05:48:21.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.254+0000 7f9e10ff9700 1 -- 172.21.15.82:0/2665192428 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7f9e04004cb0 con 0x7f9e141027c0 2024-03-12T05:48:21.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.254+0000 7f9e10ff9700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9dfc0a1ab0 0x7f9dfc0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:48:21.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.254+0000 7f9e10ff9700 1 -- 172.21.15.82:0/2665192428 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f9e040cbba0 con 0x7f9e141027c0 2024-03-12T05:48:21.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.254+0000 7f9e12ffd700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9dfc0a1ab0 0x7f9dfc0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:48:21.259 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.255+0000 7f9e12ffd700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9dfc0a1ab0 0x7f9dfc0a3f70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9e08005950 tx=0x7f9e0800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:48:21.260 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.258+0000 7f9e10ff9700 1 -- 172.21.15.82:0/2665192428 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9e040968a0 con 0x7f9e141027c0 2024-03-12T05:48:21.463 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:21.461+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/2665192428 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e1402d870 con 0x7f9dfc0a1ab0 2024-03-12T05:48:21.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:21 smithi082 ceph-mon[36382]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:21.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:21 smithi088 ceph-mon[36585]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:21.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:21 smithi076 ceph-mon[32095]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:22.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:22 smithi082 ceph-mon[36382]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T05:48:22.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:22 smithi082 ceph-mon[36382]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:48:22.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:22 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:48:22.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:22 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:48:22.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:22 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:22.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:22 smithi088 ceph-mon[36585]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T05:48:22.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:22 smithi088 ceph-mon[36585]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:48:22.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:22 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:48:22.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:22 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:48:22.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:22 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:22.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:22 smithi076 ceph-mon[32095]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T05:48:22.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:22 smithi076 ceph-mon[32095]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:48:22.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:22 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:48:22.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:22 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:48:22.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:22 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:23.424 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:23 smithi082 ceph-mon[36382]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:23.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:23 smithi088 ceph-mon[36585]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:23.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:23 smithi076 ceph-mon[32095]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:24.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:24 smithi082 ceph-mon[36382]: from='client.? 172.21.15.82:0/2584452996' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a64f23f-579e-4114-8250-228b150a753b"}]: dispatch 2024-03-12T05:48:24.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:24 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a64f23f-579e-4114-8250-228b150a753b"}]: dispatch 2024-03-12T05:48:24.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:24 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a64f23f-579e-4114-8250-228b150a753b"}]': finished 2024-03-12T05:48:24.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:24 smithi082 ceph-mon[36382]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T05:48:24.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:24.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:24 smithi088 ceph-mon[36585]: from='client.? 172.21.15.82:0/2584452996' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a64f23f-579e-4114-8250-228b150a753b"}]: dispatch 2024-03-12T05:48:24.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:24 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a64f23f-579e-4114-8250-228b150a753b"}]: dispatch 2024-03-12T05:48:24.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:24 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a64f23f-579e-4114-8250-228b150a753b"}]': finished 2024-03-12T05:48:24.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:24 smithi088 ceph-mon[36585]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T05:48:24.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:24.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:24 smithi076 ceph-mon[32095]: from='client.? 172.21.15.82:0/2584452996' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a64f23f-579e-4114-8250-228b150a753b"}]: dispatch 2024-03-12T05:48:24.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:24 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a64f23f-579e-4114-8250-228b150a753b"}]: dispatch 2024-03-12T05:48:24.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:24 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a64f23f-579e-4114-8250-228b150a753b"}]': finished 2024-03-12T05:48:24.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:24 smithi076 ceph-mon[32095]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T05:48:24.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:25.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:25 smithi082 ceph-mon[36382]: from='client.? 172.21.15.82:0/991574388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:48:25.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:25 smithi082 ceph-mon[36382]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:25.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:25 smithi088 ceph-mon[36585]: from='client.? 172.21.15.82:0/991574388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:48:25.732 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:25 smithi088 ceph-mon[36585]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:25.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:25 smithi076 ceph-mon[32095]: from='client.? 172.21.15.82:0/991574388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:48:25.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:25 smithi076 ceph-mon[32095]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:28.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:27 smithi082 ceph-mon[36382]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:28.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:27 smithi088 ceph-mon[36585]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:28.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:27 smithi076 ceph-mon[32095]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:30.158 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:29 smithi082 ceph-mon[36382]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:30.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:29 smithi088 ceph-mon[36585]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:30.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:29 smithi076 ceph-mon[32095]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:32.143 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:31 smithi082 ceph-mon[36382]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:32.143 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T05:48:32.143 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:32.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:31 smithi088 ceph-mon[36585]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T05:48:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:32.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:31 smithi076 ceph-mon[32095]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:32.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T05:48:32.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:33.132 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:32 smithi082 ceph-mon[36382]: Deploying daemon osd.1 on smithi082 2024-03-12T05:48:33.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:32 smithi088 ceph-mon[36585]: Deploying daemon osd.1 on smithi082 2024-03-12T05:48:33.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:32 smithi076 ceph-mon[32095]: Deploying daemon osd.1 on smithi082 2024-03-12T05:48:33.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:33 smithi082 ceph-mon[36382]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:34.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:33 smithi088 ceph-mon[36585]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:34.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:33 smithi076 ceph-mon[32095]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:35.985 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:35 smithi082 ceph-mon[36382]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:36.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:35 smithi088 ceph-mon[36585]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:36.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:35 smithi076 ceph-mon[32095]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:37.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:37 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:37.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:37 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:37.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:37 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:48:37.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:37 smithi082 ceph-mon[36382]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:37.982 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:37 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:37.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:37 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:37.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:37 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:48:37.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:37 smithi088 ceph-mon[36585]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:37.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:37 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:37.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:37 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:37.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:37 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:48:37.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:37 smithi076 ceph-mon[32095]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:39.777 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 1 on host 'smithi082' 2024-03-12T05:48:39.778 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:39.775+0000 7f9e10ff9700 1 -- 172.21.15.82:0/2665192428 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9e1402d870 con 0x7f9dfc0a1ab0 2024-03-12T05:48:39.779 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:39.777+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/2665192428 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9dfc0a1ab0 msgr2=0x7f9dfc0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:48:39.780 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:39.777+0000 7f9e1a0e0700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9dfc0a1ab0 0x7f9dfc0a3f70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9e08005950 tx=0x7f9e0800a300 comp rx=0 tx=0).stop 2024-03-12T05:48:39.780 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:39.777+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/2665192428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e141027c0 msgr2=0x7f9e14102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:48:39.780 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:39.777+0000 7f9e1a0e0700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e141027c0 0x7f9e14102120 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400d980 tx=0x7f9e0400dc90 comp rx=0 tx=0).stop 2024-03-12T05:48:39.780 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:39.777+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/2665192428 shutdown_connections 2024-03-12T05:48:39.780 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:39.777+0000 7f9e1a0e0700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9dfc0a1ab0 0x7f9dfc0a3f70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:39.780 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:39.777+0000 7f9e1a0e0700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e14107a20 0x7f9e14100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:39.780 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:39.777+0000 7f9e1a0e0700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e141027c0 0x7f9e14102120 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:39.780 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:39.777+0000 7f9e1a0e0700 1 --2- 172.21.15.82:0/2665192428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e141050f0 0x7f9e14100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:39.780 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:39.777+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/2665192428 >> 172.21.15.82:0/2665192428 conn(0x7f9e140fc190 msgr2=0x7f9e14103100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:48:39.780 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:39.777+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/2665192428 shutdown_connections 2024-03-12T05:48:39.780 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:48:39.777+0000 7f9e1a0e0700 1 -- 172.21.15.82:0/2665192428 wait complete. 2024-03-12T05:48:40.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:39 smithi082 ceph-mon[36382]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:40.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:39 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:40.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:39 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:40.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:39 smithi088 ceph-mon[36585]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:40.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:39 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:40.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:39 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:40.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:39 smithi076 ceph-mon[32095]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:40.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:39 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:40.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:39 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:40.448 DEBUG:teuthology.orchestra.run.smithi082:osd.1> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.1.service 2024-03-12T05:48:40.490 INFO:tasks.cephadm:Deploying osd.2 on smithi088 with /dev/vg_nvme/lv_4... 2024-03-12T05:48:40.490 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-12T05:48:40.734 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:48:41.138 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:40 smithi082 ceph-mon[36382]: from='osd.1 [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T05:48:41.138 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:40 smithi082 ceph-mon[36382]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T05:48:41.138 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:40 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:41.138 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:40 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:41.173 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:40 smithi088 ceph-mon[36585]: from='osd.1 [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T05:48:41.173 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:40 smithi088 ceph-mon[36585]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T05:48:41.173 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:40 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:41.173 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:40 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:41.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:40 smithi076 ceph-mon[32095]: from='osd.1 [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T05:48:41.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:40 smithi076 ceph-mon[32095]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T05:48:41.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:41.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:40 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:41.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:41 smithi082 ceph-mon[36382]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:41.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:41 smithi082 ceph-mon[36382]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T05:48:41.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:41 smithi082 ceph-mon[36382]: from='osd.1 [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:48:41.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:41 smithi082 ceph-mon[36382]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T05:48:41.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:41.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:41 smithi082 ceph-mon[36382]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:48:41.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:41 smithi082 ceph-mon[36382]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-12T05:48:42.115 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:41 smithi088 ceph-mon[36585]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:42.115 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:41 smithi088 ceph-mon[36585]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T05:48:42.115 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:41 smithi088 ceph-mon[36585]: from='osd.1 [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:48:42.115 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:41 smithi088 ceph-mon[36585]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T05:48:42.115 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:42.115 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:41 smithi088 ceph-mon[36585]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:48:42.115 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:41 smithi088 ceph-mon[36585]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-12T05:48:42.138 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:41 smithi076 ceph-mon[32095]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:42.138 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:41 smithi076 ceph-mon[32095]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T05:48:42.138 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:41 smithi076 ceph-mon[32095]: from='osd.1 [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:48:42.138 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:41 smithi076 ceph-mon[32095]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T05:48:42.138 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:42.138 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:41 smithi076 ceph-mon[32095]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:48:42.138 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:41 smithi076 ceph-mon[32095]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-12T05:48:42.676 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-12T05:48:42.676 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-12T05:48:42.676 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2024-03-12T05:48:42.676 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2024-03-12T05:48:42.676 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126758 s, 827 MB/s 2024-03-12T05:48:42.676 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2024-03-12T05:48:42.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:42 smithi082 ceph-mon[36382]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T05:48:42.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:42 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:42.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:42 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:42.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:42 smithi082 ceph-mon[36382]: from='osd.1 ' entity='osd.1' 2024-03-12T05:48:42.944 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 05:48:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1[40981]: 2024-03-12T05:48:42.679+0000 7f98e2f77700 -1 osd.1 0 waiting for initial osdmap 2024-03-12T05:48:42.944 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 05:48:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1[40981]: 2024-03-12T05:48:42.688+0000 7f98db563700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:48:43.154 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:42 smithi076 ceph-mon[32095]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T05:48:43.154 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:42 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:43.154 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:42 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:43.154 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:42 smithi076 ceph-mon[32095]: from='osd.1 ' entity='osd.1' 2024-03-12T05:48:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:42 smithi088 ceph-mon[36585]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T05:48:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:42 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:42 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:43.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:42 smithi088 ceph-mon[36585]: from='osd.1 ' entity='osd.1' 2024-03-12T05:48:43.302 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi088:vg_nvme/lv_4 2024-03-12T05:48:43.544 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:48:44.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:43 smithi082 ceph-mon[36382]: purged_snaps scrub starts 2024-03-12T05:48:44.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:43 smithi082 ceph-mon[36382]: purged_snaps scrub ok 2024-03-12T05:48:44.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:43 smithi082 ceph-mon[36382]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:44.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:44.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:43 smithi082 ceph-mon[36382]: osd.1 [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488] boot 2024-03-12T05:48:44.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:43 smithi082 ceph-mon[36382]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T05:48:44.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:44.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:44.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:44.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:48:44.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:44.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:44.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:48:44.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:44.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:43 smithi088 ceph-mon[36585]: purged_snaps scrub starts 2024-03-12T05:48:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:43 smithi088 ceph-mon[36585]: purged_snaps scrub ok 2024-03-12T05:48:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:43 smithi088 ceph-mon[36585]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:43 smithi088 ceph-mon[36585]: osd.1 [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488] boot 2024-03-12T05:48:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:43 smithi088 ceph-mon[36585]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T05:48:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:48:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:48:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:44.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:43 smithi076 ceph-mon[32095]: purged_snaps scrub starts 2024-03-12T05:48:44.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:43 smithi076 ceph-mon[32095]: purged_snaps scrub ok 2024-03-12T05:48:44.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:43 smithi076 ceph-mon[32095]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-12T05:48:44.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:44.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:43 smithi076 ceph-mon[32095]: osd.1 [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488] boot 2024-03-12T05:48:44.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:43 smithi076 ceph-mon[32095]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T05:48:44.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:48:44.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:44.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:44.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:48:44.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:44.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:44.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:48:44.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:44.910 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:44 smithi088 ceph-mon[36585]: Detected new or changed devices on smithi082 2024-03-12T05:48:44.910 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:44 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi082 to 17080M 2024-03-12T05:48:44.910 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:48:44.910 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:44.910 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:48:44.910 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.127+0000 7f64013da700 1 -- 172.21.15.88:0/2479227327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63fc102740 msgr2=0x7f63fc108ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:48:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.127+0000 7f64013da700 1 --2- 172.21.15.88:0/2479227327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63fc102740 0x7f63fc108ff0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f63f400b0a0 tx=0x7f63f400b3b0 comp rx=0 tx=0).stop 2024-03-12T05:48:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.127+0000 7f64013da700 1 -- 172.21.15.88:0/2479227327 shutdown_connections 2024-03-12T05:48:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.127+0000 7f64013da700 1 --2- 172.21.15.88:0/2479227327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63fc102740 0x7f63fc108ff0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.127+0000 7f64013da700 1 --2- 172.21.15.88:0/2479227327 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63fc100ba0 0x7f63fc100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.127+0000 7f64013da700 1 --2- 172.21.15.88:0/2479227327 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f63fc101da0 0x7f63fc102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.127+0000 7f64013da700 1 -- 172.21.15.88:0/2479227327 >> 172.21.15.88:0/2479227327 conn(0x7f63fc0fc140 msgr2=0x7f63fc0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:48:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.127+0000 7f64013da700 1 -- 172.21.15.88:0/2479227327 shutdown_connections 2024-03-12T05:48:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.127+0000 7f64013da700 1 -- 172.21.15.88:0/2479227327 wait complete. 2024-03-12T05:48:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f64013da700 1 Processor -- start 2024-03-12T05:48:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f64013da700 1 -- start start 2024-03-12T05:48:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f64013da700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63fc100ba0 0x7f63fc06ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:48:45.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f64013da700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f63fc101da0 0x7f63fc19caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f64013da700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63fc102740 0x7f63fc19f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f64013da700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc19f8d0 con 0x7f63fc102740 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f64013da700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc19fa40 con 0x7f63fc101da0 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f64013da700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc19fc00 con 0x7f63fc100ba0 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f63faffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63fc100ba0 0x7f63fc06ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f63faffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63fc100ba0 0x7f63fc06ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:45218/0 (socket says 172.21.15.88:45218) 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f63faffd700 1 -- 172.21.15.88:0/4178466383 learned_addr learned my addr 172.21.15.88:0/4178466383 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f63fb7fe700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63fc102740 0x7f63fc19f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f63fa7fc700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f63fc101da0 0x7f63fc19caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f63faffd700 1 -- 172.21.15.88:0/4178466383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f63fc101da0 msgr2=0x7f63fc19caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f63faffd700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f63fc101da0 0x7f63fc19caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f63faffd700 1 -- 172.21.15.88:0/4178466383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63fc102740 msgr2=0x7f63fc19f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.128+0000 7f63faffd700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63fc102740 0x7f63fc19f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.129+0000 7f63faffd700 1 -- 172.21.15.88:0/4178466383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f4009d00 con 0x7f63fc100ba0 2024-03-12T05:48:45.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.129+0000 7f63fa7fc700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f63fc101da0 0x7f63fc19caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:48:45.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.129+0000 7f63faffd700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63fc100ba0 0x7f63fc06ae70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00d980 tx=0x7f63ec00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:48:45.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.129+0000 7f63f3fff700 1 -- 172.21.15.88:0/4178466383 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec00b950 con 0x7f63fc100ba0 2024-03-12T05:48:45.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.129+0000 7f63f3fff700 1 -- 172.21.15.88:0/4178466383 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63ec007620 con 0x7f63fc100ba0 2024-03-12T05:48:45.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.129+0000 7f63f3fff700 1 -- 172.21.15.88:0/4178466383 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec00f620 con 0x7f63fc100ba0 2024-03-12T05:48:45.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.129+0000 7f64013da700 1 -- 172.21.15.88:0/4178466383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63fc19fee0 con 0x7f63fc100ba0 2024-03-12T05:48:45.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.129+0000 7f64013da700 1 -- 172.21.15.88:0/4178466383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63fc1a03a0 con 0x7f63fc100ba0 2024-03-12T05:48:45.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.130+0000 7f64013da700 1 -- 172.21.15.88:0/4178466383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63fc107f00 con 0x7f63fc100ba0 2024-03-12T05:48:45.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.131+0000 7f63f3fff700 1 -- 172.21.15.88:0/4178466383 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7f63ec0106f0 con 0x7f63fc100ba0 2024-03-12T05:48:45.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.131+0000 7f63f3fff700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f63e80a1ab0 0x7f63e80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:48:45.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.132+0000 7f63fa7fc700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f63e80a1ab0 0x7f63e80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:48:45.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.132+0000 7f63f3fff700 1 -- 172.21.15.88:0/4178466383 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f63ec0cc240 con 0x7f63fc100ba0 2024-03-12T05:48:45.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.132+0000 7f63fa7fc700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f63e80a1ab0 0x7f63e80a3f70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f63fc06b3f0 tx=0x7f63e4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:48:45.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.135+0000 7f63f3fff700 1 -- 172.21.15.88:0/4178466383 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f63ec096ca0 con 0x7f63fc100ba0 2024-03-12T05:48:45.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:44 smithi082 ceph-mon[36382]: Detected new or changed devices on smithi082 2024-03-12T05:48:45.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:44 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi082 to 17080M 2024-03-12T05:48:45.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:48:45.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:45.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:48:45.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:45.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:44 smithi076 ceph-mon[32095]: Detected new or changed devices on smithi082 2024-03-12T05:48:45.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:44 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi082 to 17080M 2024-03-12T05:48:45.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:48:45.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:45.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:48:45.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:48:45.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:48:45.338+0000 7f64013da700 1 -- 172.21.15.88:0/4178466383 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f63fc02d960 con 0x7f63e80a1ab0 2024-03-12T05:48:46.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:45 smithi088 ceph-mon[36585]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:46.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:45 smithi088 ceph-mon[36585]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T05:48:46.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:45 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:48:46.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:45 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:48:46.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:45 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:46.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:45 smithi076 ceph-mon[32095]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:46.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:45 smithi076 ceph-mon[32095]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T05:48:46.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:45 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:48:46.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:45 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:48:46.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:45 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:46.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:45 smithi082 ceph-mon[36382]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:46.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:45 smithi082 ceph-mon[36382]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T05:48:46.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:45 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:48:46.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:45 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:48:46.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:45 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:47.076 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:46 smithi088 ceph-mon[36585]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:48:47.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:46 smithi076 ceph-mon[32095]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:48:47.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:46 smithi082 ceph-mon[36382]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:48:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:47 smithi088 ceph-mon[36585]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:47 smithi088 ceph-mon[36585]: from='client.? 172.21.15.88:0/1065462214' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "621a410c-75b7-42c3-aa04-f4a3e188586b"}]: dispatch 2024-03-12T05:48:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:47 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "621a410c-75b7-42c3-aa04-f4a3e188586b"}]: dispatch 2024-03-12T05:48:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:47 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "621a410c-75b7-42c3-aa04-f4a3e188586b"}]': finished 2024-03-12T05:48:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:47 smithi088 ceph-mon[36585]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T05:48:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:47 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:48:48.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:47 smithi076 ceph-mon[32095]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:48.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:47 smithi076 ceph-mon[32095]: from='client.? 172.21.15.88:0/1065462214' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "621a410c-75b7-42c3-aa04-f4a3e188586b"}]: dispatch 2024-03-12T05:48:48.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:47 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "621a410c-75b7-42c3-aa04-f4a3e188586b"}]: dispatch 2024-03-12T05:48:48.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:47 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "621a410c-75b7-42c3-aa04-f4a3e188586b"}]': finished 2024-03-12T05:48:48.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:47 smithi076 ceph-mon[32095]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T05:48:48.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:47 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:48:48.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:47 smithi082 ceph-mon[36382]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:48.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:47 smithi082 ceph-mon[36382]: from='client.? 172.21.15.88:0/1065462214' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "621a410c-75b7-42c3-aa04-f4a3e188586b"}]: dispatch 2024-03-12T05:48:48.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:47 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "621a410c-75b7-42c3-aa04-f4a3e188586b"}]: dispatch 2024-03-12T05:48:48.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:47 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "621a410c-75b7-42c3-aa04-f4a3e188586b"}]': finished 2024-03-12T05:48:48.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:47 smithi082 ceph-mon[36382]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T05:48:48.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:47 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:48:49.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:48 smithi088 ceph-mon[36585]: from='client.? 172.21.15.88:0/3451663643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:48:49.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:48 smithi076 ceph-mon[32095]: from='client.? 172.21.15.88:0/3451663643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:48:49.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:48 smithi082 ceph-mon[36382]: from='client.? 172.21.15.88:0/3451663643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:48:50.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:49 smithi088 ceph-mon[36585]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:50.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:49 smithi076 ceph-mon[32095]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:50.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:49 smithi082 ceph-mon[36382]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:52.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:51 smithi088 ceph-mon[36585]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:52.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:51 smithi076 ceph-mon[32095]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:52.245 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:51 smithi082 ceph-mon[36382]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:54.059 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:53 smithi088 ceph-mon[36585]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:54.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:53 smithi076 ceph-mon[32095]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:54.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:53 smithi082 ceph-mon[36382]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:55.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:54 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T05:48:55.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:54 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:55.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:54 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T05:48:55.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:54 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:55.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:54 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T05:48:55.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:54 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:48:56.124 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:55 smithi088 ceph-mon[36585]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:56.124 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:55 smithi088 ceph-mon[36585]: Deploying daemon osd.2 on smithi088 2024-03-12T05:48:56.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:55 smithi082 ceph-mon[36382]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:56.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:55 smithi082 ceph-mon[36382]: Deploying daemon osd.2 on smithi088 2024-03-12T05:48:56.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:55 smithi076 ceph-mon[32095]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:56.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:55 smithi076 ceph-mon[32095]: Deploying daemon osd.2 on smithi088 2024-03-12T05:48:57.230 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:56 smithi088 ceph-mon[36585]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:57.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:57 smithi082 ceph-mon[36382]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:57.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:56 smithi076 ceph-mon[32095]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:48:59.942 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:48:59 smithi088 ceph-mon[36585]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:00.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:48:59 smithi082 ceph-mon[36382]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:00.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:48:59 smithi076 ceph-mon[32095]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:01.407 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:00 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:01.407 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:00 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:01.407 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:00 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:01.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:00 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:01.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:00 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:01.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:00 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:01.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:00 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:01.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:00 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:01.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:00 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:02.288 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:01 smithi088 ceph-mon[36585]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:02.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:01 smithi082 ceph-mon[36382]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:02.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:01 smithi076 ceph-mon[32095]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:02.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:02.898+0000 7f63f3fff700 1 -- 172.21.15.88:0/4178466383 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f63fc02d960 con 0x7f63e80a1ab0 2024-03-12T05:49:02.901 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 2 on host 'smithi088' 2024-03-12T05:49:02.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:02.900+0000 7f64013da700 1 -- 172.21.15.88:0/4178466383 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f63e80a1ab0 msgr2=0x7f63e80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:02.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:02.900+0000 7f64013da700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f63e80a1ab0 0x7f63e80a3f70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f63fc06b3f0 tx=0x7f63e4008040 comp rx=0 tx=0).stop 2024-03-12T05:49:02.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:02.900+0000 7f64013da700 1 -- 172.21.15.88:0/4178466383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63fc100ba0 msgr2=0x7f63fc06ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:02.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:02.900+0000 7f64013da700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63fc100ba0 0x7f63fc06ae70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00d980 tx=0x7f63ec00dc90 comp rx=0 tx=0).stop 2024-03-12T05:49:02.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:02.901+0000 7f64013da700 1 -- 172.21.15.88:0/4178466383 shutdown_connections 2024-03-12T05:49:02.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:02.901+0000 7f64013da700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f63e80a1ab0 0x7f63e80a3f70 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:02.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:02.901+0000 7f64013da700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63fc102740 0x7f63fc19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:02.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:02.901+0000 7f64013da700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63fc100ba0 0x7f63fc06ae70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:02.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:02.901+0000 7f64013da700 1 --2- 172.21.15.88:0/4178466383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f63fc101da0 0x7f63fc19caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:02.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:02.901+0000 7f64013da700 1 -- 172.21.15.88:0/4178466383 >> 172.21.15.88:0/4178466383 conn(0x7f63fc0fc140 msgr2=0x7f63fc105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:49:02.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:02.901+0000 7f64013da700 1 -- 172.21.15.88:0/4178466383 shutdown_connections 2024-03-12T05:49:02.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:02.901+0000 7f64013da700 1 -- 172.21.15.88:0/4178466383 wait complete. 2024-03-12T05:49:03.704 DEBUG:teuthology.orchestra.run.smithi088:osd.2> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.2.service 2024-03-12T05:49:03.707 INFO:tasks.cephadm:Deploying osd.3 on smithi076 with /dev/vg_nvme/lv_3... 2024-03-12T05:49:03.707 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-12T05:49:03.962 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:49:03.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:03 smithi088 ceph-mon[36585]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:03.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:03 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:03.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:03 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:03.983 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 05:49:03 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2[41708]: 2024-03-12T05:49:03.890+0000 7f3f559c17c0 -1 osd.2 0 log_to_monitors true 2024-03-12T05:49:04.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:03 smithi076 ceph-mon[32095]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:04.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:04.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:04.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:03 smithi082 ceph-mon[36382]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:04.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:03 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:04.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:03 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:04.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:04 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:04.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:04 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:04.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:04 smithi088 ceph-mon[36585]: from='osd.2 [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T05:49:04.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:04 smithi088 ceph-mon[36585]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T05:49:04.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:04 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:04.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:04 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:04.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:04 smithi076 ceph-mon[32095]: from='osd.2 [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T05:49:04.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:04 smithi076 ceph-mon[32095]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T05:49:05.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:04 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:05.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:04 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:05.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:04 smithi082 ceph-mon[36382]: from='osd.2 [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T05:49:05.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:04 smithi082 ceph-mon[36382]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T05:49:05.894 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-12T05:49:05.895 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-12T05:49:05.895 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-03-12T05:49:05.895 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-03-12T05:49:05.895 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116805 s, 898 MB/s 2024-03-12T05:49:05.895 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-03-12T05:49:06.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:05 smithi082 ceph-mon[36382]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:06.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:05 smithi082 ceph-mon[36382]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T05:49:06.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:05 smithi082 ceph-mon[36382]: from='osd.2 [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:49:06.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:05 smithi082 ceph-mon[36382]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T05:49:06.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:05 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:06.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:05 smithi082 ceph-mon[36382]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:49:06.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:05 smithi076 ceph-mon[32095]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:06.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:05 smithi076 ceph-mon[32095]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T05:49:06.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:05 smithi076 ceph-mon[32095]: from='osd.2 [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:49:06.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:05 smithi076 ceph-mon[32095]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T05:49:06.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:05 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:06.223 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:05 smithi076 ceph-mon[32095]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:49:06.233 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 05:49:05 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2[41708]: 2024-03-12T05:49:05.892+0000 7f3f4a825700 -1 osd.2 0 waiting for initial osdmap 2024-03-12T05:49:06.233 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 05:49:05 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2[41708]: 2024-03-12T05:49:05.897+0000 7f3f46e19700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:49:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:05 smithi088 ceph-mon[36585]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:05 smithi088 ceph-mon[36585]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T05:49:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:05 smithi088 ceph-mon[36585]: from='osd.2 [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:49:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:05 smithi088 ceph-mon[36585]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T05:49:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:05 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:06.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:05 smithi088 ceph-mon[36585]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:49:06.528 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi076:vg_nvme/lv_3 2024-03-12T05:49:06.776 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:49:07.172 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:06 smithi076 ceph-mon[32095]: purged_snaps scrub starts 2024-03-12T05:49:07.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:06 smithi076 ceph-mon[32095]: purged_snaps scrub ok 2024-03-12T05:49:07.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:06 smithi076 ceph-mon[32095]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-12T05:49:07.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:06 smithi076 ceph-mon[32095]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T05:49:07.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:06 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:07.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:06 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:07.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:06 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:07.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:06 smithi082 ceph-mon[36382]: purged_snaps scrub starts 2024-03-12T05:49:07.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:06 smithi082 ceph-mon[36382]: purged_snaps scrub ok 2024-03-12T05:49:07.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:06 smithi082 ceph-mon[36382]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-12T05:49:07.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:06 smithi082 ceph-mon[36382]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T05:49:07.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:06 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:07.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:06 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:07.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:06 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:07.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:06 smithi088 ceph-mon[36585]: purged_snaps scrub starts 2024-03-12T05:49:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:06 smithi088 ceph-mon[36585]: purged_snaps scrub ok 2024-03-12T05:49:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:06 smithi088 ceph-mon[36585]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-12T05:49:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:06 smithi088 ceph-mon[36585]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T05:49:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:06 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:06 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:07.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:06 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:08.049 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:08.049 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: osd.2 [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838] boot 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: Detected new or changed devices on smithi088 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi088 to 17080M 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:08.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: osd.2 [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838] boot 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: Detected new or changed devices on smithi088 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi088 to 17080M 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:08.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-12T05:49:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: osd.2 [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838] boot 2024-03-12T05:49:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T05:49:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:49:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: Detected new or changed devices on smithi088 2024-03-12T05:49:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi088 to 17080M 2024-03-12T05:49:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:08.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:08.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.273+0000 7fde7e598700 1 -- 172.21.15.76:0/1873790297 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde78100e90 msgr2=0x7fde781012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:08.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.273+0000 7fde7e598700 1 --2- 172.21.15.76:0/1873790297 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde78100e90 0x7fde781012f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009990 tx=0x7fde6c009ca0 comp rx=0 tx=0).stop 2024-03-12T05:49:08.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.273+0000 7fde7e598700 1 -- 172.21.15.76:0/1873790297 shutdown_connections 2024-03-12T05:49:08.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.273+0000 7fde7e598700 1 --2- 172.21.15.76:0/1873790297 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde78101830 0x7fde781090f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:08.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.273+0000 7fde7e598700 1 --2- 172.21.15.76:0/1873790297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde78102150 0x7fde78102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:08.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.273+0000 7fde7e598700 1 --2- 172.21.15.76:0/1873790297 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde78100e90 0x7fde781012f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:08.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.273+0000 7fde7e598700 1 -- 172.21.15.76:0/1873790297 >> 172.21.15.76:0/1873790297 conn(0x7fde780fb8c0 msgr2=0x7fde780fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:49:08.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.273+0000 7fde7e598700 1 -- 172.21.15.76:0/1873790297 shutdown_connections 2024-03-12T05:49:08.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.273+0000 7fde7e598700 1 -- 172.21.15.76:0/1873790297 wait complete. 2024-03-12T05:49:08.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.274+0000 7fde7e598700 1 Processor -- start 2024-03-12T05:49:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.274+0000 7fde7e598700 1 -- start start 2024-03-12T05:49:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.274+0000 7fde7e598700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde78100e90 0x7fde78197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:49:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.274+0000 7fde7e598700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde78101830 0x7fde78198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:49:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.274+0000 7fde7e598700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde78102150 0x7fde7819f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:49:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.274+0000 7fde7e598700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7819f850 con 0x7fde78102150 2024-03-12T05:49:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.274+0000 7fde7e598700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7819f990 con 0x7fde78100e90 2024-03-12T05:49:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.274+0000 7fde7e598700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7819fb50 con 0x7fde78101830 2024-03-12T05:49:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.274+0000 7fde7cb35700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde78102150 0x7fde7819f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:49:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.274+0000 7fde7cb35700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde78102150 0x7fde7819f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44142/0 (socket says 172.21.15.76:44142) 2024-03-12T05:49:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.274+0000 7fde7cb35700 1 -- 172.21.15.76:0/1099806891 learned_addr learned my addr 172.21.15.76:0/1099806891 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:49:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.275+0000 7fde77fff700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde78100e90 0x7fde78197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:49:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.275+0000 7fde777fe700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde78101830 0x7fde78198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:49:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.275+0000 7fde7cb35700 1 -- 172.21.15.76:0/1099806891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde78101830 msgr2=0x7fde78198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:08.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.275+0000 7fde7cb35700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde78101830 0x7fde78198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:08.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.275+0000 7fde7cb35700 1 -- 172.21.15.76:0/1099806891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde78100e90 msgr2=0x7fde78197d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:08.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.275+0000 7fde7cb35700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde78100e90 0x7fde78197d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:08.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.275+0000 7fde7cb35700 1 -- 172.21.15.76:0/1099806891 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c0095f0 con 0x7fde78102150 2024-03-12T05:49:08.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.275+0000 7fde777fe700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde78101830 0x7fde78198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:49:08.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.275+0000 7fde7cb35700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde78102150 0x7fde7819f280 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fde7000ada0 tx=0x7fde7000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:49:08.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.275+0000 7fde757fa700 1 -- 172.21.15.76:0/1099806891 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7000f070 con 0x7fde78102150 2024-03-12T05:49:08.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.275+0000 7fde757fa700 1 -- 172.21.15.76:0/1099806891 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde70007b30 con 0x7fde78102150 2024-03-12T05:49:08.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.275+0000 7fde7e598700 1 -- 172.21.15.76:0/1099806891 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7819fe30 con 0x7fde78102150 2024-03-12T05:49:08.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.275+0000 7fde757fa700 1 -- 172.21.15.76:0/1099806891 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde700053c0 con 0x7fde78102150 2024-03-12T05:49:08.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.275+0000 7fde7e598700 1 -- 172.21.15.76:0/1099806891 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde781a01f0 con 0x7fde78102150 2024-03-12T05:49:08.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.276+0000 7fde7e598700 1 -- 172.21.15.76:0/1099806891 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde78108060 con 0x7fde78102150 2024-03-12T05:49:08.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.277+0000 7fde757fa700 1 -- 172.21.15.76:0/1099806891 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 15) v1 ==== 136223+0+0 (secure 0 0 0) 0x7fde7000c980 con 0x7fde78102150 2024-03-12T05:49:08.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.278+0000 7fde757fa700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fde600a1a30 0x7fde600a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:49:08.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.278+0000 7fde77fff700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fde600a1a30 0x7fde600a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:49:08.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.278+0000 7fde757fa700 1 -- 172.21.15.76:0/1099806891 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 2851+0+0 (secure 0 0 0) 0x7fde700cb3a0 con 0x7fde78102150 2024-03-12T05:49:08.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.278+0000 7fde77fff700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fde600a1a30 0x7fde600a3ef0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fde781991e0 tx=0x7fde68009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:49:08.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.282+0000 7fde757fa700 1 -- 172.21.15.76:0/1099806891 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fde70095c70 con 0x7fde78102150 2024-03-12T05:49:08.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:08.489+0000 7fde7e598700 1 -- 172.21.15.76:0/1099806891 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fde7802d950 con 0x7fde600a1a30 2024-03-12T05:49:09.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:09 smithi076 ceph-mon[32095]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T05:49:09.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:09 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:49:09.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:09 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:49:09.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:09 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:09.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:09 smithi082 ceph-mon[36382]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T05:49:09.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:09 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:49:09.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:09 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:49:09.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:09 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:09.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:09 smithi088 ceph-mon[36585]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T05:49:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:09 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:49:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:09 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:49:09.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:09 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:10.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:10 smithi076 ceph-mon[32095]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:49:10.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:10 smithi076 ceph-mon[32095]: pgmap v83: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:10.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:10 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:49:10.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:10 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T05:49:10.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:10 smithi076 ceph-mon[32095]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T05:49:10.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:10 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:49:10.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:10 smithi082 ceph-mon[36382]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:49:10.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:10 smithi082 ceph-mon[36382]: pgmap v83: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:10.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:10 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:49:10.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:10 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T05:49:10.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:10 smithi082 ceph-mon[36382]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T05:49:10.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:10 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:49:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:10 smithi088 ceph-mon[36585]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:49:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:10 smithi088 ceph-mon[36585]: pgmap v83: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:10 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:49:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:10 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T05:49:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:10 smithi088 ceph-mon[36585]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T05:49:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:10 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T05:49:10.736 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 05:49:10 smithi076 sudo[51003]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T05:49:10.737 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 05:49:10 smithi076 sudo[51003]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:49:10.737 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 05:49:10 smithi076 sudo[51003]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:49:10.737 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 05:49:10 smithi076 sudo[51003]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:49:11.184 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 05:49:10 smithi076 sudo[51026]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T05:49:11.184 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 05:49:10 smithi076 sudo[51026]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:49:11.184 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 05:49:10 smithi076 sudo[51026]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:49:11.184 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 05:49:10 smithi076 sudo[51026]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:49:11.328 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:11 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T05:49:11.328 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:11 smithi082 ceph-mon[36382]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T05:49:11.329 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:11 smithi082 ceph-mon[36382]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:11.329 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:11 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:11.329 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:11 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2999778049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e843f131-6b3e-4148-90d2-0e44ea1b5511"}]: dispatch 2024-03-12T05:49:11.329 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:11 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2999778049' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e843f131-6b3e-4148-90d2-0e44ea1b5511"}]': finished 2024-03-12T05:49:11.329 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:11 smithi082 ceph-mon[36382]: osdmap e22: 4 total, 3 up, 4 in 2024-03-12T05:49:11.329 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:11 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:11.329 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 05:49:11 smithi082 sudo[44882]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T05:49:11.329 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 05:49:11 smithi082 sudo[44882]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:49:11.329 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 05:49:11 smithi082 sudo[44882]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:49:11.329 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 05:49:11 smithi082 sudo[44882]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:49:11.467 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:11 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T05:49:11.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:11 smithi076 ceph-mon[32095]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T05:49:11.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:11 smithi076 ceph-mon[32095]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:11.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:11 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:11.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:11 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2999778049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e843f131-6b3e-4148-90d2-0e44ea1b5511"}]: dispatch 2024-03-12T05:49:11.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:11 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2999778049' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e843f131-6b3e-4148-90d2-0e44ea1b5511"}]': finished 2024-03-12T05:49:11.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:11 smithi076 ceph-mon[32095]: osdmap e22: 4 total, 3 up, 4 in 2024-03-12T05:49:11.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:11 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:11.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:11 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T05:49:11.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:11 smithi088 ceph-mon[36585]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T05:49:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:11 smithi088 ceph-mon[36585]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:11 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:11 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2999778049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e843f131-6b3e-4148-90d2-0e44ea1b5511"}]: dispatch 2024-03-12T05:49:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:11 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2999778049' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e843f131-6b3e-4148-90d2-0e44ea1b5511"}]': finished 2024-03-12T05:49:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:11 smithi088 ceph-mon[36585]: osdmap e22: 4 total, 3 up, 4 in 2024-03-12T05:49:11.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:11 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:11.693 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 05:49:11 smithi082 sudo[44885]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T05:49:11.693 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 05:49:11 smithi082 sudo[44885]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:49:11.693 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 05:49:11 smithi082 sudo[44885]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:49:11.693 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 05:49:11 smithi082 sudo[44885]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:49:11.855 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 05:49:11 smithi088 sudo[45781]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T05:49:11.855 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 05:49:11 smithi088 sudo[45781]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:49:11.855 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 05:49:11 smithi088 sudo[45781]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:49:11.855 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 05:49:11 smithi088 sudo[45781]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:49:12.232 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 05:49:11 smithi088 sudo[45784]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T05:49:12.232 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 05:49:11 smithi088 sudo[45784]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:49:12.233 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 05:49:11 smithi088 sudo[45784]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:49:12.233 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 05:49:11 smithi088 sudo[45784]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:49:12.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:12 smithi076 sudo[51132]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T05:49:12.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:12 smithi076 sudo[51132]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:49:12.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:12 smithi076 sudo[51132]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:49:12.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:12 smithi076 sudo[51132]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:49:12.928 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:12 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:12.928 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:12 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/5001929' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:49:12.928 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:12 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:12.928 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:12 smithi088 ceph-mon[36585]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:49:12.928 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:12 smithi088 sudo[45787]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T05:49:12.928 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:12 smithi088 sudo[45787]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:49:12.929 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:12 smithi088 sudo[45787]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:49:12.929 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:12 smithi088 sudo[45787]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:49:12.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:12 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:12.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:12 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/5001929' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:49:12.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:12 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:12.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:12 smithi082 ceph-mon[36382]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:49:12.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:12 smithi076 sudo[51322]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T05:49:12.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:12 smithi076 sudo[51322]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:49:12.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:12 smithi076 sudo[51322]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:49:12.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:12 smithi076 sudo[51322]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:49:12.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:12 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:12.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:12 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/5001929' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:49:12.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:12 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:12.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:12 smithi076 ceph-mon[32095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:49:13.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:13.171+0000 7fde757fa700 1 -- 172.21.15.76:0/1099806891 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fde70096040 con 0x7fde78102150 2024-03-12T05:49:13.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 sudo[45790]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T05:49:13.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 sudo[45790]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:49:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 sudo[45790]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:49:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 sudo[45790]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:49:13.659 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 sudo[44888]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T05:49:13.659 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 sudo[44888]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:49:13.659 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 sudo[44888]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:49:13.659 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 sudo[44888]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:49:13.659 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 sudo[44891]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T05:49:13.659 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 sudo[44891]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T05:49:13.659 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 sudo[44891]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T05:49:13.659 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 sudo[44891]: pam_unix(sudo:session): session closed for user root 2024-03-12T05:49:13.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:49:13.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:49:13.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:49:13.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:49:13.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:49:13.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:49:13.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:49:13.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:49:13.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:13.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:49:13.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:49:13.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-12T05:49:13.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:49:13.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:49:13.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:49:13.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:13 smithi082 ceph-mon[36382]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:49:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:49:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:49:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:49:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:49:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:49:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:49:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:49:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:49:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:49:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:49:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-12T05:49:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:49:13.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:49:13.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:49:13.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:13 smithi088 ceph-mon[36585]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:49:13.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:49:13.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:49:13.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:49:13.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:49:13.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:49:13.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:49:13.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:49:13.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:49:13.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:13.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:49:13.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T05:49:13.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-12T05:49:13.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:49:13.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:49:13.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:49:13.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:13 smithi076 ceph-mon[32095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T05:49:15.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:15 smithi076 ceph-mon[32095]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:16.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:15 smithi082 ceph-mon[36382]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:16.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:15 smithi088 ceph-mon[36585]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:18.159 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:17 smithi076 ceph-mon[32095]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:18.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:17 smithi082 ceph-mon[36382]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:18.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:17 smithi088 ceph-mon[36585]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:18.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:18 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T05:49:18.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:18 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:19.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:18 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T05:49:19.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:18 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:19.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:18 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T05:49:19.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:18 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:19.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:19 smithi076 ceph-mon[32095]: Deploying daemon osd.3 on smithi076 2024-03-12T05:49:19.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:19 smithi076 ceph-mon[32095]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:20.134 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:19 smithi088 ceph-mon[36585]: Deploying daemon osd.3 on smithi076 2024-03-12T05:49:20.134 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:19 smithi088 ceph-mon[36585]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:20.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:19 smithi082 ceph-mon[36382]: Deploying daemon osd.3 on smithi076 2024-03-12T05:49:20.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:19 smithi082 ceph-mon[36382]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:22.112 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:21 smithi076 ceph-mon[32095]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:22.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:21 smithi082 ceph-mon[36382]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:21 smithi088 ceph-mon[36585]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:23.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:23 smithi076 ceph-mon[32095]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:24.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:23 smithi082 ceph-mon[36382]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:24.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:23 smithi088 ceph-mon[36585]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:25.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:25 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:25.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:25 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:25.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:25 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:25.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:25 smithi076 ceph-mon[32095]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:25.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:25 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:25.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:25 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:25.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:25 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:25.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:25 smithi082 ceph-mon[36382]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:25.482 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:25 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:25 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:25 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:25.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:25 smithi088 ceph-mon[36585]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:27.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:27.758+0000 7fde757fa700 1 -- 172.21.15.76:0/1099806891 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fde7802d950 con 0x7fde600a1a30 2024-03-12T05:49:27.760 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 3 on host 'smithi076' 2024-03-12T05:49:27.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:27.761+0000 7fde7e598700 1 -- 172.21.15.76:0/1099806891 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fde600a1a30 msgr2=0x7fde600a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:27.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:27.761+0000 7fde7e598700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fde600a1a30 0x7fde600a3ef0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fde781991e0 tx=0x7fde68009190 comp rx=0 tx=0).stop 2024-03-12T05:49:27.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:27.761+0000 7fde7e598700 1 -- 172.21.15.76:0/1099806891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde78102150 msgr2=0x7fde7819f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:27.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:27.761+0000 7fde7e598700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde78102150 0x7fde7819f280 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fde7000ada0 tx=0x7fde7000c330 comp rx=0 tx=0).stop 2024-03-12T05:49:27.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:27.761+0000 7fde7e598700 1 -- 172.21.15.76:0/1099806891 shutdown_connections 2024-03-12T05:49:27.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:27.761+0000 7fde7e598700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fde600a1a30 0x7fde600a3ef0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:27.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:27.761+0000 7fde7e598700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde78102150 0x7fde7819f280 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:27.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:27.761+0000 7fde7e598700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde78100e90 0x7fde78197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:27.761+0000 7fde7e598700 1 --2- 172.21.15.76:0/1099806891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde78101830 0x7fde78198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:27.761+0000 7fde7e598700 1 -- 172.21.15.76:0/1099806891 >> 172.21.15.76:0/1099806891 conn(0x7fde780fb8c0 msgr2=0x7fde781051a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:49:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:27.761+0000 7fde7e598700 1 -- 172.21.15.76:0/1099806891 shutdown_connections 2024-03-12T05:49:27.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:49:27.761+0000 7fde7e598700 1 -- 172.21.15.76:0/1099806891 wait complete. 2024-03-12T05:49:28.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:27 smithi076 ceph-mon[32095]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:28.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:28.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:28.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:27 smithi088 ceph-mon[36585]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:28.425 DEBUG:teuthology.orchestra.run.smithi076:osd.3> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.3.service 2024-03-12T05:49:28.427 INFO:tasks.cephadm:Deploying osd.4 on smithi082 with /dev/vg_nvme/lv_3... 2024-03-12T05:49:28.427 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-12T05:49:28.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:27 smithi082 ceph-mon[36382]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:28.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:28.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:28.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:28.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:28.672 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:49:29.114 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:28 smithi082 ceph-mon[36382]: from='osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T05:49:29.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:28 smithi088 ceph-mon[36585]: from='osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T05:49:29.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:28 smithi076 ceph-mon[32095]: from='osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T05:49:30.129 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:29 smithi082 ceph-mon[36382]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:30.129 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:29 smithi082 ceph-mon[36382]: from='osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T05:49:30.129 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:29 smithi082 ceph-mon[36382]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T05:49:30.129 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:29 smithi082 ceph-mon[36382]: from='osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-12T05:49:30.129 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:29 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:30.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:29 smithi076 ceph-mon[32095]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:30.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:29 smithi076 ceph-mon[32095]: from='osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T05:49:30.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:29 smithi076 ceph-mon[32095]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T05:49:30.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:29 smithi076 ceph-mon[32095]: from='osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-12T05:49:30.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:29 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:30.237 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 05:49:29 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3[53101]: 2024-03-12T05:49:29.985+0000 7f39cf497700 -1 osd.3 0 waiting for initial osdmap 2024-03-12T05:49:30.237 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 05:49:29 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3[53101]: 2024-03-12T05:49:29.991+0000 7f39cba8b700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:49:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:29 smithi088 ceph-mon[36585]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:29 smithi088 ceph-mon[36585]: from='osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T05:49:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:29 smithi088 ceph-mon[36585]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T05:49:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:29 smithi088 ceph-mon[36585]: from='osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-12T05:49:30.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:29 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:30.565 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-12T05:49:30.565 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-12T05:49:30.565 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-03-12T05:49:30.565 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-03-12T05:49:30.565 INFO:teuthology.orchestra.run.smithi082.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112486 s, 932 MB/s 2024-03-12T05:49:30.565 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 2024-03-12T05:49:30.565 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-03-12T05:49:31.157 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:30 smithi082 ceph-mon[36382]: from='osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-12T05:49:31.157 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:30 smithi082 ceph-mon[36382]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T05:49:31.158 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:30 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:31.158 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:30 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:31.183 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi082:vg_nvme/lv_3 2024-03-12T05:49:31.232 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:30 smithi088 ceph-mon[36585]: from='osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-12T05:49:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:30 smithi088 ceph-mon[36585]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T05:49:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:30 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:31.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:30 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:31.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:30 smithi076 ceph-mon[32095]: from='osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-12T05:49:31.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:30 smithi076 ceph-mon[32095]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T05:49:31.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:30 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:31.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:30 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:31.465 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:49:32.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: purged_snaps scrub starts 2024-03-12T05:49:32.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: purged_snaps scrub ok 2024-03-12T05:49:32.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618] boot 2024-03-12T05:49:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: osdmap e25: 4 total, 4 up, 4 in 2024-03-12T05:49:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: osdmap e26: 4 total, 4 up, 4 in 2024-03-12T05:49:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:32.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: purged_snaps scrub starts 2024-03-12T05:49:32.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: purged_snaps scrub ok 2024-03-12T05:49:32.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:32.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:32.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618] boot 2024-03-12T05:49:32.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: osdmap e25: 4 total, 4 up, 4 in 2024-03-12T05:49:32.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:32.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: osdmap e26: 4 total, 4 up, 4 in 2024-03-12T05:49:32.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:32.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: purged_snaps scrub starts 2024-03-12T05:49:32.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: purged_snaps scrub ok 2024-03-12T05:49:32.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-12T05:49:32.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:32.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: osd.3 [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618] boot 2024-03-12T05:49:32.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: osdmap e25: 4 total, 4 up, 4 in 2024-03-12T05:49:32.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:49:32.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: osdmap e26: 4 total, 4 up, 4 in 2024-03-12T05:49:32.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:32.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:32.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:32.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:32.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:32.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:32.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:32.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:33.045 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.043+0000 7f1e2d126700 1 -- 172.21.15.82:0/813673417 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e28102200 msgr2=0x7f1e28102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:33.046 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.043+0000 7f1e2d126700 1 --2- 172.21.15.82:0/813673417 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e28102200 0x7f1e28102620 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1e20009990 tx=0x7f1e20009ca0 comp rx=0 tx=0).stop 2024-03-12T05:49:33.046 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.043+0000 7f1e2d126700 1 -- 172.21.15.82:0/813673417 shutdown_connections 2024-03-12T05:49:33.046 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.043+0000 7f1e2d126700 1 --2- 172.21.15.82:0/813673417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e28100730 0x7f1e28100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:33.046 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.043+0000 7f1e2d126700 1 --2- 172.21.15.82:0/813673417 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e28102200 0x7f1e28102620 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:33.046 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.043+0000 7f1e2d126700 1 --2- 172.21.15.82:0/813673417 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e280ffd90 0x7f1e281001f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:33.046 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.043+0000 7f1e2d126700 1 -- 172.21.15.82:0/813673417 >> 172.21.15.82:0/813673417 conn(0x7f1e280fb910 msgr2=0x7f1e280fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:49:33.046 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.043+0000 7f1e2d126700 1 -- 172.21.15.82:0/813673417 shutdown_connections 2024-03-12T05:49:33.046 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.043+0000 7f1e2d126700 1 -- 172.21.15.82:0/813673417 wait complete. 2024-03-12T05:49:33.046 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.043+0000 7f1e2d126700 1 Processor -- start 2024-03-12T05:49:33.048 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2d126700 1 -- start start 2024-03-12T05:49:33.048 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2d126700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e280ffd90 0x7f1e28199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:49:33.048 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2d126700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e28100730 0x7f1e2819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:49:33.048 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2d126700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e28102200 0x7f1e281a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:49:33.048 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2d126700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e281a1900 con 0x7f1e280ffd90 2024-03-12T05:49:33.048 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2d126700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e281a1a40 con 0x7f1e28102200 2024-03-12T05:49:33.048 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2d126700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e281a1c00 con 0x7f1e28100730 2024-03-12T05:49:33.048 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2759e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e28102200 0x7f1e281a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:49:33.049 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2759e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e28102200 0x7f1e281a1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:42258/0 (socket says 172.21.15.82:42258) 2024-03-12T05:49:33.049 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e26d9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e280ffd90 0x7f1e28199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:49:33.049 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2759e700 1 -- 172.21.15.82:0/4085819536 learned_addr learned my addr 172.21.15.82:0/4085819536 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-03-12T05:49:33.049 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2659c700 1 --2- 172.21.15.82:0/4085819536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e28100730 0x7f1e2819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:49:33.049 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2759e700 1 -- 172.21.15.82:0/4085819536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e28100730 msgr2=0x7f1e2819a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:33.049 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2759e700 1 --2- 172.21.15.82:0/4085819536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e28100730 0x7f1e2819a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:33.049 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2759e700 1 -- 172.21.15.82:0/4085819536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e280ffd90 msgr2=0x7f1e28199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:33.049 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2759e700 1 --2- 172.21.15.82:0/4085819536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e280ffd90 0x7f1e28199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:33.049 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.044+0000 7f1e2759e700 1 -- 172.21.15.82:0/4085819536 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e200095f0 con 0x7f1e28102200 2024-03-12T05:49:33.049 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.045+0000 7f1e2759e700 1 --2- 172.21.15.82:0/4085819536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e28102200 0x7f1e281a1330 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1e18007d70 tx=0x7f1e1800a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:49:33.049 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.045+0000 7f1e1ffff700 1 -- 172.21.15.82:0/4085819536 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1800ad40 con 0x7f1e28102200 2024-03-12T05:49:33.049 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.045+0000 7f1e2d126700 1 -- 172.21.15.82:0/4085819536 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e281a1ee0 con 0x7f1e28102200 2024-03-12T05:49:33.049 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.045+0000 7f1e1ffff700 1 -- 172.21.15.82:0/4085819536 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e18012900 con 0x7f1e28102200 2024-03-12T05:49:33.049 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.045+0000 7f1e1ffff700 1 -- 172.21.15.82:0/4085819536 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18011780 con 0x7f1e28102200 2024-03-12T05:49:33.050 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.045+0000 7f1e2d126700 1 -- 172.21.15.82:0/4085819536 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e281a22a0 con 0x7f1e28102200 2024-03-12T05:49:33.050 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.046+0000 7f1e2d126700 1 -- 172.21.15.82:0/4085819536 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2810bc40 con 0x7f1e28102200 2024-03-12T05:49:33.050 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.047+0000 7f1e1ffff700 1 -- 172.21.15.82:0/4085819536 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1e18012a70 con 0x7f1e28102200 2024-03-12T05:49:33.050 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.047+0000 7f1e1ffff700 1 --2- 172.21.15.82:0/4085819536 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1e140a1ac0 0x7f1e140a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:49:33.050 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.047+0000 7f1e1ffff700 1 -- 172.21.15.82:0/4085819536 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f1e180cc8b0 con 0x7f1e28102200 2024-03-12T05:49:33.051 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.048+0000 7f1e26d9d700 1 --2- 172.21.15.82:0/4085819536 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1e140a1ac0 0x7f1e140a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:49:33.051 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.048+0000 7f1e26d9d700 1 --2- 172.21.15.82:0/4085819536 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1e140a1ac0 0x7f1e140a3f80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1e2819b290 tx=0x7f1e20005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:49:33.053 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.051+0000 7f1e1ffff700 1 -- 172.21.15.82:0/4085819536 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1e18096cc0 con 0x7f1e28102200 2024-03-12T05:49:33.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:32 smithi082 ceph-mon[36382]: Detected new or changed devices on smithi076 2024-03-12T05:49:33.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:32 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi076 to 8544M 2024-03-12T05:49:33.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:32 smithi082 ceph-mon[36382]: osdmap e27: 4 total, 4 up, 4 in 2024-03-12T05:49:33.253 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:33.251+0000 7f1e2d126700 1 -- 172.21.15.82:0/4085819536 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e2802d950 con 0x7f1e140a1ac0 2024-03-12T05:49:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:32 smithi088 ceph-mon[36585]: Detected new or changed devices on smithi076 2024-03-12T05:49:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:32 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi076 to 8544M 2024-03-12T05:49:33.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:32 smithi088 ceph-mon[36585]: osdmap e27: 4 total, 4 up, 4 in 2024-03-12T05:49:33.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:32 smithi076 ceph-mon[32095]: Detected new or changed devices on smithi076 2024-03-12T05:49:33.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:32 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi076 to 8544M 2024-03-12T05:49:33.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:32 smithi076 ceph-mon[32095]: osdmap e27: 4 total, 4 up, 4 in 2024-03-12T05:49:34.177 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:33 smithi082 ceph-mon[36382]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:34.177 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:33 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:49:34.177 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:33 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:49:34.177 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:33 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:33 smithi088 ceph-mon[36585]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:33 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:49:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:33 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:49:34.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:33 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:34.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:33 smithi076 ceph-mon[32095]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:34.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:33 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:49:34.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:33 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:49:34.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:33 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:35.090 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:35 smithi082 ceph-mon[36382]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:49:35.090 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:35 smithi082 ceph-mon[36382]: osdmap e28: 4 total, 4 up, 4 in 2024-03-12T05:49:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:35 smithi088 ceph-mon[36585]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:49:35.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:35 smithi088 ceph-mon[36585]: osdmap e28: 4 total, 4 up, 4 in 2024-03-12T05:49:35.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:35 smithi076 ceph-mon[32095]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:49:35.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:35 smithi076 ceph-mon[32095]: osdmap e28: 4 total, 4 up, 4 in 2024-03-12T05:49:36.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:36 smithi082 ceph-mon[36382]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T05:49:36.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:36 smithi082 ceph-mon[36382]: osdmap e29: 4 total, 4 up, 4 in 2024-03-12T05:49:36.292 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:36 smithi082 ceph-mon[36382]: from='client.? 172.21.15.82:0/1401871613' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f20ff3d5-82fd-4d8c-aac9-a3b538e12dff"}]: dispatch 2024-03-12T05:49:36.293 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:36 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f20ff3d5-82fd-4d8c-aac9-a3b538e12dff"}]: dispatch 2024-03-12T05:49:36.293 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:36 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f20ff3d5-82fd-4d8c-aac9-a3b538e12dff"}]': finished 2024-03-12T05:49:36.293 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:36 smithi082 ceph-mon[36382]: osdmap e30: 5 total, 4 up, 5 in 2024-03-12T05:49:36.293 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:36 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:36 smithi088 ceph-mon[36585]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T05:49:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:36 smithi088 ceph-mon[36585]: osdmap e29: 4 total, 4 up, 4 in 2024-03-12T05:49:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:36 smithi088 ceph-mon[36585]: from='client.? 172.21.15.82:0/1401871613' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f20ff3d5-82fd-4d8c-aac9-a3b538e12dff"}]: dispatch 2024-03-12T05:49:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:36 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f20ff3d5-82fd-4d8c-aac9-a3b538e12dff"}]: dispatch 2024-03-12T05:49:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:36 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f20ff3d5-82fd-4d8c-aac9-a3b538e12dff"}]': finished 2024-03-12T05:49:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:36 smithi088 ceph-mon[36585]: osdmap e30: 5 total, 4 up, 5 in 2024-03-12T05:49:36.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:36 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:36.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:36 smithi076 ceph-mon[32095]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T05:49:36.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:36 smithi076 ceph-mon[32095]: osdmap e29: 4 total, 4 up, 4 in 2024-03-12T05:49:36.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:36 smithi076 ceph-mon[32095]: from='client.? 172.21.15.82:0/1401871613' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f20ff3d5-82fd-4d8c-aac9-a3b538e12dff"}]: dispatch 2024-03-12T05:49:36.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:36 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f20ff3d5-82fd-4d8c-aac9-a3b538e12dff"}]: dispatch 2024-03-12T05:49:36.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:36 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f20ff3d5-82fd-4d8c-aac9-a3b538e12dff"}]': finished 2024-03-12T05:49:36.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:36 smithi076 ceph-mon[32095]: osdmap e30: 5 total, 4 up, 5 in 2024-03-12T05:49:36.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:36 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:37.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:37 smithi082 ceph-mon[36382]: from='client.? 172.21.15.82:0/2701687718' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:49:37.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:37 smithi082 ceph-mon[36382]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-12T05:49:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:37 smithi088 ceph-mon[36585]: from='client.? 172.21.15.82:0/2701687718' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:49:37.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:37 smithi088 ceph-mon[36585]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-12T05:49:37.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:37 smithi076 ceph-mon[32095]: from='client.? 172.21.15.82:0/2701687718' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:49:37.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:37 smithi076 ceph-mon[32095]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-12T05:49:40.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:39 smithi082 ceph-mon[36382]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T05:49:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:39 smithi088 ceph-mon[36585]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T05:49:40.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:39 smithi076 ceph-mon[32095]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T05:49:41.236 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:49:40 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:49:40.810+0000 7f9b0fcb5700 -1 calc_pg_upmaps abort due to max <= 0 2024-03-12T05:49:42.138 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:41 smithi076 ceph-mon[32095]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:42.141 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:41 smithi082 ceph-mon[36382]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:41 smithi088 ceph-mon[36585]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:44.171 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:43 smithi082 ceph-mon[36382]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:44.172 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T05:49:44.172 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:43 smithi088 ceph-mon[36585]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T05:49:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:44.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:43 smithi076 ceph-mon[32095]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:44.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T05:49:44.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:45.170 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:44 smithi082 ceph-mon[36382]: Deploying daemon osd.4 on smithi082 2024-03-12T05:49:45.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:44 smithi088 ceph-mon[36585]: Deploying daemon osd.4 on smithi082 2024-03-12T05:49:45.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:44 smithi076 ceph-mon[32095]: Deploying daemon osd.4 on smithi082 2024-03-12T05:49:46.133 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:45 smithi082 ceph-mon[36382]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:46.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:45 smithi088 ceph-mon[36585]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:46.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:45 smithi076 ceph-mon[32095]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:48.163 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:47 smithi082 ceph-mon[36382]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:47 smithi088 ceph-mon[36585]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:48.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:47 smithi076 ceph-mon[32095]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:49.943 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:49.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:49.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:49 smithi082 ceph-mon[36382]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:49.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:50.035 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:50.035 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:50.035 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:49 smithi088 ceph-mon[36585]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:50.035 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:50.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:50.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:50.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:49 smithi076 ceph-mon[32095]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:50.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:52.133 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:51 smithi082 ceph-mon[36382]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:52.133 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:51 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:52.133 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:51 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:52.133 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:51 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:52.133 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:51 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:52.133 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:51 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:52.218 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:52.215+0000 7f1e1ffff700 1 -- 172.21.15.82:0/4085819536 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1e2802d950 con 0x7f1e140a1ac0 2024-03-12T05:49:52.218 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 4 on host 'smithi082' 2024-03-12T05:49:52.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:52.217+0000 7f1e2d126700 1 -- 172.21.15.82:0/4085819536 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1e140a1ac0 msgr2=0x7f1e140a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:52.219 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:52.217+0000 7f1e2d126700 1 --2- 172.21.15.82:0/4085819536 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1e140a1ac0 0x7f1e140a3f80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1e2819b290 tx=0x7f1e20005fb0 comp rx=0 tx=0).stop 2024-03-12T05:49:52.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:52.217+0000 7f1e2d126700 1 -- 172.21.15.82:0/4085819536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e28102200 msgr2=0x7f1e281a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:52.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:52.217+0000 7f1e2d126700 1 --2- 172.21.15.82:0/4085819536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e28102200 0x7f1e281a1330 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1e18007d70 tx=0x7f1e1800a340 comp rx=0 tx=0).stop 2024-03-12T05:49:52.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:52.217+0000 7f1e2d126700 1 -- 172.21.15.82:0/4085819536 shutdown_connections 2024-03-12T05:49:52.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:52.217+0000 7f1e2d126700 1 --2- 172.21.15.82:0/4085819536 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1e140a1ac0 0x7f1e140a3f80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:52.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:52.217+0000 7f1e2d126700 1 --2- 172.21.15.82:0/4085819536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e28102200 0x7f1e281a1330 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:52.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:52.217+0000 7f1e2d126700 1 --2- 172.21.15.82:0/4085819536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e280ffd90 0x7f1e28199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:52.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:52.217+0000 7f1e2d126700 1 --2- 172.21.15.82:0/4085819536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e28100730 0x7f1e2819a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:52.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:52.217+0000 7f1e2d126700 1 -- 172.21.15.82:0/4085819536 >> 172.21.15.82:0/4085819536 conn(0x7f1e280fb910 msgr2=0x7f1e280fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:49:52.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:52.217+0000 7f1e2d126700 1 -- 172.21.15.82:0/4085819536 shutdown_connections 2024-03-12T05:49:52.220 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:49:52.217+0000 7f1e2d126700 1 -- 172.21.15.82:0/4085819536 wait complete. 2024-03-12T05:49:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:51 smithi088 ceph-mon[36585]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:51 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:51 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:51 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:51 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:51 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:52.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:51 smithi076 ceph-mon[32095]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:52.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:51 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:52.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:51 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:52.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:51 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:52.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:51 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:52.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:51 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:52.769 DEBUG:teuthology.orchestra.run.smithi082:osd.4> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.4.service 2024-03-12T05:49:52.771 INFO:tasks.cephadm:Deploying osd.5 on smithi088 with /dev/vg_nvme/lv_3... 2024-03-12T05:49:52.771 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-12T05:49:53.013 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:49:53.423 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:53 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:53.423 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:53 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:53.423 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:53 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:53.423 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:53 smithi088 ceph-mon[36585]: from='osd.4 [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T05:49:53.423 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:53 smithi088 ceph-mon[36585]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T05:49:53.423 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:53 smithi088 ceph-mon[36585]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:53.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:53 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:53.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:53 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:53.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:53 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:53.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:53 smithi082 ceph-mon[36382]: from='osd.4 [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T05:49:53.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:53 smithi082 ceph-mon[36382]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T05:49:53.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:53 smithi082 ceph-mon[36382]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:53.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:53 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:53.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:53 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:53.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:53 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:49:53.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:53 smithi076 ceph-mon[32095]: from='osd.4 [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T05:49:53.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:53 smithi076 ceph-mon[32095]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T05:49:53.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:53 smithi076 ceph-mon[32095]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:54 smithi088 ceph-mon[36585]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T05:49:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:54 smithi088 ceph-mon[36585]: osdmap e31: 5 total, 4 up, 5 in 2024-03-12T05:49:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:54 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:54 smithi088 ceph-mon[36585]: from='osd.4 [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:49:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:54 smithi088 ceph-mon[36585]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:49:54.363 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:54 smithi082 ceph-mon[36382]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T05:49:54.363 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:54 smithi082 ceph-mon[36382]: osdmap e31: 5 total, 4 up, 5 in 2024-03-12T05:49:54.363 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:54 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:54.363 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:54 smithi082 ceph-mon[36382]: from='osd.4 [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:49:54.363 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:54 smithi082 ceph-mon[36382]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:49:54.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:54 smithi076 ceph-mon[32095]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T05:49:54.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:54 smithi076 ceph-mon[32095]: osdmap e31: 5 total, 4 up, 5 in 2024-03-12T05:49:54.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:54 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:54.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:54 smithi076 ceph-mon[32095]: from='osd.4 [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:49:54.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:54 smithi076 ceph-mon[32095]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:49:54.858 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-12T05:49:54.858 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-12T05:49:54.858 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2024-03-12T05:49:54.858 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2024-03-12T05:49:54.858 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120756 s, 868 MB/s 2024-03-12T05:49:54.858 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2024-03-12T05:49:55.332 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:55 smithi082 ceph-mon[36382]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-12T05:49:55.332 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:55 smithi082 ceph-mon[36382]: osdmap e32: 5 total, 4 up, 5 in 2024-03-12T05:49:55.332 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:55.332 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:55.332 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:55 smithi082 ceph-mon[36382]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:55.332 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:55 smithi082 ceph-mon[36382]: from='osd.4 ' entity='osd.4' 2024-03-12T05:49:55.332 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:55.332 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 05:49:55 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4[47711]: 2024-03-12T05:49:55.130+0000 7f407e4d6700 -1 osd.4 0 waiting for initial osdmap 2024-03-12T05:49:55.333 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 05:49:55 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4[47711]: 2024-03-12T05:49:55.143+0000 7f4076ac2700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:49:55.415 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:55 smithi088 ceph-mon[36585]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-12T05:49:55.415 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:55 smithi088 ceph-mon[36585]: osdmap e32: 5 total, 4 up, 5 in 2024-03-12T05:49:55.416 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:55.416 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:55.416 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:55 smithi088 ceph-mon[36585]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:55.416 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:55 smithi088 ceph-mon[36585]: from='osd.4 ' entity='osd.4' 2024-03-12T05:49:55.416 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:55.442 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi088:vg_nvme/lv_3 2024-03-12T05:49:55.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:55 smithi076 ceph-mon[32095]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-12T05:49:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:55 smithi076 ceph-mon[32095]: osdmap e32: 5 total, 4 up, 5 in 2024-03-12T05:49:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:55 smithi076 ceph-mon[32095]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-12T05:49:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:55 smithi076 ceph-mon[32095]: from='osd.4 ' entity='osd.4' 2024-03-12T05:49:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:55.687 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:49:56.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: purged_snaps scrub starts 2024-03-12T05:49:56.437 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: purged_snaps scrub ok 2024-03-12T05:49:56.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: Detected new or changed devices on smithi082 2024-03-12T05:49:56.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:56.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:56.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:56.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:56.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi082 to 8540M 2024-03-12T05:49:56.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:56.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:56.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:56.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:56.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: osd.4 [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628] boot 2024-03-12T05:49:56.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T05:49:56.438 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:56.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: purged_snaps scrub starts 2024-03-12T05:49:56.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: purged_snaps scrub ok 2024-03-12T05:49:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: Detected new or changed devices on smithi082 2024-03-12T05:49:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi082 to 8540M 2024-03-12T05:49:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: osd.4 [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628] boot 2024-03-12T05:49:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T05:49:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: purged_snaps scrub starts 2024-03-12T05:49:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: purged_snaps scrub ok 2024-03-12T05:49:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: Detected new or changed devices on smithi082 2024-03-12T05:49:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:49:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi082 to 8540M 2024-03-12T05:49:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:49:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:49:56.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: osd.4 [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628] boot 2024-03-12T05:49:56.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T05:49:56.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:49:57.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.199+0000 7f5dc803d700 1 -- 172.21.15.88:0/1921279673 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc0103860 msgr2=0x7f5dc010a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:57.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.199+0000 7f5dc803d700 1 --2- 172.21.15.88:0/1921279673 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc0103860 0x7f5dc010a110 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00b0a0 tx=0x7f5dbc00b3b0 comp rx=0 tx=0).stop 2024-03-12T05:49:57.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.199+0000 7f5dc803d700 1 -- 172.21.15.88:0/1921279673 shutdown_connections 2024-03-12T05:49:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.199+0000 7f5dc803d700 1 --2- 172.21.15.88:0/1921279673 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc0103860 0x7f5dc010a110 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.199+0000 7f5dc803d700 1 --2- 172.21.15.88:0/1921279673 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dc0101cc0 0x7f5dc01020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.199+0000 7f5dc803d700 1 --2- 172.21.15.88:0/1921279673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dc0102ec0 0x7f5dc0103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.199+0000 7f5dc803d700 1 -- 172.21.15.88:0/1921279673 >> 172.21.15.88:0/1921279673 conn(0x7f5dc00fd220 msgr2=0x7f5dc00ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:49:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.199+0000 7f5dc803d700 1 -- 172.21.15.88:0/1921279673 shutdown_connections 2024-03-12T05:49:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.199+0000 7f5dc803d700 1 -- 172.21.15.88:0/1921279673 wait complete. 2024-03-12T05:49:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc803d700 1 Processor -- start 2024-03-12T05:49:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc803d700 1 -- start start 2024-03-12T05:49:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc803d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dc0101cc0 0x7f5dc0075a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:49:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc803d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dc0102ec0 0x7f5dc0074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:49:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc803d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc0103860 0x7f5dc00745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:49:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc803d700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0074d20 con 0x7f5dc0102ec0 2024-03-12T05:49:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc803d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0074e90 con 0x7f5dc0101cc0 2024-03-12T05:49:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc803d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc01a1d90 con 0x7f5dc0103860 2024-03-12T05:49:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc65da700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc0103860 0x7f5dc00745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:49:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc65da700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc0103860 0x7f5dc00745a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42376/0 (socket says 172.21.15.88:42376) 2024-03-12T05:49:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc65da700 1 -- 172.21.15.88:0/2168643347 learned_addr learned my addr 172.21.15.88:0/2168643347 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-03-12T05:49:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc55d8700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dc0102ec0 0x7f5dc0074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:49:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc5dd9700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dc0101cc0 0x7f5dc0075a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:49:57.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc65da700 1 -- 172.21.15.88:0/2168643347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dc0101cc0 msgr2=0x7f5dc0075a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc65da700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dc0101cc0 0x7f5dc0075a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc65da700 1 -- 172.21.15.88:0/2168643347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dc0102ec0 msgr2=0x7f5dc0074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:49:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc65da700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dc0102ec0 0x7f5dc0074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:49:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.200+0000 7f5dc65da700 1 -- 172.21.15.88:0/2168643347 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc009d00 con 0x7f5dc0103860 2024-03-12T05:49:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.201+0000 7f5dc5dd9700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dc0101cc0 0x7f5dc0075a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:49:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.201+0000 7f5dc65da700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc0103860 0x7f5dc00745a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00bd60 tx=0x7f5dbc00be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:49:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.201+0000 7f5db6ffd700 1 -- 172.21.15.88:0/2168643347 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc00e040 con 0x7f5dc0103860 2024-03-12T05:49:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.201+0000 7f5db6ffd700 1 -- 172.21.15.88:0/2168643347 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5dbc0094c0 con 0x7f5dc0103860 2024-03-12T05:49:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.201+0000 7f5dc803d700 1 -- 172.21.15.88:0/2168643347 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc01a2010 con 0x7f5dc0103860 2024-03-12T05:49:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.201+0000 7f5db6ffd700 1 -- 172.21.15.88:0/2168643347 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc0053e0 con 0x7f5dc0103860 2024-03-12T05:49:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.201+0000 7f5dc803d700 1 -- 172.21.15.88:0/2168643347 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc01a2480 con 0x7f5dc0103860 2024-03-12T05:49:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.202+0000 7f5dc803d700 1 -- 172.21.15.88:0/2168643347 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc0109020 con 0x7f5dc0103860 2024-03-12T05:49:57.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.203+0000 7f5db6ffd700 1 -- 172.21.15.88:0/2168643347 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5dbc01da70 con 0x7f5dc0103860 2024-03-12T05:49:57.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.203+0000 7f5db6ffd700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5dac0a1b40 0x7f5dac0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:49:57.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.203+0000 7f5db6ffd700 1 -- 172.21.15.88:0/2168643347 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f5dbc0cdad0 con 0x7f5dc0103860 2024-03-12T05:49:57.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.204+0000 7f5dc5dd9700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5dac0a1b40 0x7f5dac0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:49:57.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.204+0000 7f5dc5dd9700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5dac0a1b40 0x7f5dac0a4000 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0102d20 tx=0x7f5db8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:49:57.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.208+0000 7f5db6ffd700 1 -- 172.21.15.88:0/2168643347 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5dbc097de0 con 0x7f5dc0103860 2024-03-12T05:49:57.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:49:57.410+0000 7f5dc803d700 1 -- 172.21.15.88:0/2168643347 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5dc002d7d0 con 0x7f5dac0a1b40 2024-03-12T05:49:57.412 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:57 smithi088 ceph-mon[36585]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:49:57.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:57 smithi076 ceph-mon[32095]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:49:57.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:57 smithi082 ceph-mon[36382]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:49:58.335 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:58 smithi088 ceph-mon[36585]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T05:49:58.335 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:58 smithi088 ceph-mon[36585]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:49:58.335 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:49:58.335 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:49:58.335 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:58.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:58 smithi076 ceph-mon[32095]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T05:49:58.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:58 smithi076 ceph-mon[32095]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:49:58.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:49:58.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:49:58.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:58.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:58 smithi082 ceph-mon[36382]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T05:49:58.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:58 smithi082 ceph-mon[36382]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:49:58.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:49:58.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:49:58.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:49:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:49:59 smithi088 ceph-mon[36585]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:49:59.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:49:59 smithi076 ceph-mon[32095]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:49:59.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:49:59 smithi082 ceph-mon[36382]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:00 smithi088 ceph-mon[36585]: overall HEALTH_OK 2024-03-12T05:50:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:00 smithi088 ceph-mon[36585]: from='client.? 172.21.15.88:0/3491263571' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9499144-0f35-47f2-89bc-ce4af1aa1b58"}]: dispatch 2024-03-12T05:50:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:00 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9499144-0f35-47f2-89bc-ce4af1aa1b58"}]: dispatch 2024-03-12T05:50:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:00 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9499144-0f35-47f2-89bc-ce4af1aa1b58"}]': finished 2024-03-12T05:50:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:00 smithi088 ceph-mon[36585]: osdmap e35: 6 total, 5 up, 6 in 2024-03-12T05:50:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:00 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:00.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:00 smithi076 ceph-mon[32095]: overall HEALTH_OK 2024-03-12T05:50:00.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:00 smithi076 ceph-mon[32095]: from='client.? 172.21.15.88:0/3491263571' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9499144-0f35-47f2-89bc-ce4af1aa1b58"}]: dispatch 2024-03-12T05:50:00.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:00 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9499144-0f35-47f2-89bc-ce4af1aa1b58"}]: dispatch 2024-03-12T05:50:00.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:00 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9499144-0f35-47f2-89bc-ce4af1aa1b58"}]': finished 2024-03-12T05:50:00.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:00 smithi076 ceph-mon[32095]: osdmap e35: 6 total, 5 up, 6 in 2024-03-12T05:50:00.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:00 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:00.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:00 smithi082 ceph-mon[36382]: overall HEALTH_OK 2024-03-12T05:50:00.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:00 smithi082 ceph-mon[36382]: from='client.? 172.21.15.88:0/3491263571' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9499144-0f35-47f2-89bc-ce4af1aa1b58"}]: dispatch 2024-03-12T05:50:00.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:00 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9499144-0f35-47f2-89bc-ce4af1aa1b58"}]: dispatch 2024-03-12T05:50:00.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:00 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9499144-0f35-47f2-89bc-ce4af1aa1b58"}]': finished 2024-03-12T05:50:00.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:00 smithi082 ceph-mon[36382]: osdmap e35: 6 total, 5 up, 6 in 2024-03-12T05:50:00.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:00 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:01 smithi088 ceph-mon[36585]: from='client.? 172.21.15.88:0/1726677693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:50:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:01 smithi088 ceph-mon[36585]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:01.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:01 smithi076 ceph-mon[32095]: from='client.? 172.21.15.88:0/1726677693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:50:01.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:01 smithi076 ceph-mon[32095]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:01.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:01 smithi082 ceph-mon[36382]: from='client.? 172.21.15.88:0/1726677693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:50:01.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:01 smithi082 ceph-mon[36382]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:04.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:03 smithi082 ceph-mon[36382]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:03 smithi088 ceph-mon[36585]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:04.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:03 smithi076 ceph-mon[32095]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:06.162 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:05 smithi088 ceph-mon[36585]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:06.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:05 smithi082 ceph-mon[36382]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:06.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:05 smithi076 ceph-mon[32095]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:08.077 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:07 smithi088 ceph-mon[36585]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:08.077 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T05:50:08.077 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:07 smithi082 ceph-mon[36382]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T05:50:08.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:08.439 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:07 smithi076 ceph-mon[32095]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:08.439 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T05:50:08.439 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:08.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:08 smithi088 ceph-mon[36585]: Deploying daemon osd.5 on smithi088 2024-03-12T05:50:09.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:08 smithi082 ceph-mon[36382]: Deploying daemon osd.5 on smithi088 2024-03-12T05:50:09.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:08 smithi076 ceph-mon[32095]: Deploying daemon osd.5 on smithi088 2024-03-12T05:50:10.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:09 smithi082 ceph-mon[36382]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:10.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:09 smithi088 ceph-mon[36585]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:10.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:09 smithi076 ceph-mon[32095]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:11.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:11 smithi088 ceph-mon[36585]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:12.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:11 smithi082 ceph-mon[36382]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:12.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:11 smithi076 ceph-mon[32095]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:13.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:13.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:13.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:50:13.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:13 smithi082 ceph-mon[36382]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:13.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:13.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:13.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:50:13.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:13 smithi088 ceph-mon[36585]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:13.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:13.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:13.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:50:13.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:13 smithi076 ceph-mon[32095]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:15.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:50:15.522+0000 7f5db6ffd700 1 -- 172.21.15.88:0/2168643347 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5dc002d7d0 con 0x7f5dac0a1b40 2024-03-12T05:50:15.525 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 5 on host 'smithi088' 2024-03-12T05:50:15.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:50:15.525+0000 7f5dc803d700 1 -- 172.21.15.88:0/2168643347 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5dac0a1b40 msgr2=0x7f5dac0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:50:15.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:50:15.525+0000 7f5dc803d700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5dac0a1b40 0x7f5dac0a4000 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0102d20 tx=0x7f5db8009260 comp rx=0 tx=0).stop 2024-03-12T05:50:15.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:50:15.525+0000 7f5dc803d700 1 -- 172.21.15.88:0/2168643347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc0103860 msgr2=0x7f5dc00745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:50:15.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:50:15.525+0000 7f5dc803d700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc0103860 0x7f5dc00745a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00bd60 tx=0x7f5dbc00be40 comp rx=0 tx=0).stop 2024-03-12T05:50:15.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:50:15.525+0000 7f5dc803d700 1 -- 172.21.15.88:0/2168643347 shutdown_connections 2024-03-12T05:50:15.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:50:15.525+0000 7f5dc803d700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5dac0a1b40 0x7f5dac0a4000 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:15.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:50:15.525+0000 7f5dc803d700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5dc0103860 0x7f5dc00745a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:15.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:50:15.525+0000 7f5dc803d700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5dc0101cc0 0x7f5dc0075a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:15.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:50:15.525+0000 7f5dc803d700 1 --2- 172.21.15.88:0/2168643347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5dc0102ec0 0x7f5dc0074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:15.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:50:15.525+0000 7f5dc803d700 1 -- 172.21.15.88:0/2168643347 >> 172.21.15.88:0/2168643347 conn(0x7f5dc00fd220 msgr2=0x7f5dc0106160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:50:15.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:50:15.525+0000 7f5dc803d700 1 -- 172.21.15.88:0/2168643347 shutdown_connections 2024-03-12T05:50:15.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:50:15.525+0000 7f5dc803d700 1 -- 172.21.15.88:0/2168643347 wait complete. 2024-03-12T05:50:16.157 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:15 smithi088 ceph-mon[36585]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:16.157 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:15 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:16.157 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:15 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:16.183 DEBUG:teuthology.orchestra.run.smithi088:osd.5> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.5.service 2024-03-12T05:50:16.185 INFO:tasks.cephadm:Deploying osd.6 on smithi076 with /dev/vg_nvme/lv_2... 2024-03-12T05:50:16.185 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-12T05:50:16.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:15 smithi082 ceph-mon[36382]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:16.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:15 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:16.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:15 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:16.215 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:15 smithi076 ceph-mon[32095]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:16.215 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:15 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:16.215 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:15 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:16.433 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:50:16.483 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 05:50:16 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T05:50:16.284+0000 7fd7359567c0 -1 osd.5 0 log_to_monitors true 2024-03-12T05:50:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:17 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:17 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:17 smithi088 ceph-mon[36585]: from='osd.5 [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T05:50:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:17 smithi088 ceph-mon[36585]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T05:50:17.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:17 smithi088 ceph-mon[36585]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:17.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:17 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:17.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:17 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:17.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:17 smithi076 ceph-mon[32095]: from='osd.5 [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T05:50:17.577 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:17 smithi076 ceph-mon[32095]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T05:50:17.577 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:17 smithi076 ceph-mon[32095]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:17.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:17 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:17.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:17 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:17.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:17 smithi082 ceph-mon[36382]: from='osd.5 [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T05:50:17.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:17 smithi082 ceph-mon[36382]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T05:50:17.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:17 smithi082 ceph-mon[36382]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:18.466 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-12T05:50:18.467 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-12T05:50:18.467 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-03-12T05:50:18.467 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-03-12T05:50:18.467 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011491 s, 913 MB/s 2024-03-12T05:50:18.467 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-03-12T05:50:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:18 smithi088 ceph-mon[36585]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T05:50:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:18 smithi088 ceph-mon[36585]: from='osd.5 [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:50:18.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:18 smithi088 ceph-mon[36585]: osdmap e36: 6 total, 5 up, 6 in 2024-03-12T05:50:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:18 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:18 smithi088 ceph-mon[36585]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:50:18.484 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 05:50:18 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T05:50:18.282+0000 7fd72a7ba700 -1 osd.5 0 waiting for initial osdmap 2024-03-12T05:50:18.484 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 05:50:18 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T05:50:18.289+0000 7fd726dae700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:50:18.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:18 smithi076 ceph-mon[32095]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T05:50:18.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:18 smithi076 ceph-mon[32095]: from='osd.5 [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:50:18.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:18 smithi076 ceph-mon[32095]: osdmap e36: 6 total, 5 up, 6 in 2024-03-12T05:50:18.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:18 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:18.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:18 smithi076 ceph-mon[32095]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:50:18.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:18 smithi082 ceph-mon[36382]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T05:50:18.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:18 smithi082 ceph-mon[36382]: from='osd.5 [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:50:18.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:18 smithi082 ceph-mon[36382]: osdmap e36: 6 total, 5 up, 6 in 2024-03-12T05:50:18.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:18 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:18.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:18 smithi082 ceph-mon[36382]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:50:19.132 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi076:vg_nvme/lv_2 2024-03-12T05:50:19.362 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:19 smithi088 ceph-mon[36585]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-12T05:50:19.362 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:19 smithi088 ceph-mon[36585]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T05:50:19.362 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:19.362 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:19.362 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:19 smithi088 ceph-mon[36585]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:19.362 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:19.410 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:50:19.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:19 smithi076 ceph-mon[32095]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-12T05:50:19.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:19 smithi076 ceph-mon[32095]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T05:50:19.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:19.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:19.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:19 smithi076 ceph-mon[32095]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:19.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:19.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:19 smithi082 ceph-mon[36382]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-12T05:50:19.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:19 smithi082 ceph-mon[36382]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T05:50:19.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:19.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:19.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:19 smithi082 ceph-mon[36382]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-12T05:50:19.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:20.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: purged_snaps scrub starts 2024-03-12T05:50:20.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: purged_snaps scrub ok 2024-03-12T05:50:20.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: osd.5 [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264] boot 2024-03-12T05:50:20.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: osdmap e38: 6 total, 6 up, 6 in 2024-03-12T05:50:20.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:20.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: Detected new or changed devices on smithi088 2024-03-12T05:50:20.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:20.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:20.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi088 to 8540M 2024-03-12T05:50:20.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:20.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:50:20.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:50:20.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:20.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:50:20.466 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:20 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: purged_snaps scrub starts 2024-03-12T05:50:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: purged_snaps scrub ok 2024-03-12T05:50:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: osd.5 [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264] boot 2024-03-12T05:50:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: osdmap e38: 6 total, 6 up, 6 in 2024-03-12T05:50:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: Detected new or changed devices on smithi088 2024-03-12T05:50:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi088 to 8540M 2024-03-12T05:50:20.635 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.636 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:20.636 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:50:20.636 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.636 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:50:20.636 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:20.636 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:50:20.636 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:20 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: purged_snaps scrub starts 2024-03-12T05:50:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: purged_snaps scrub ok 2024-03-12T05:50:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: osd.5 [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264] boot 2024-03-12T05:50:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: osdmap e38: 6 total, 6 up, 6 in 2024-03-12T05:50:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:50:20.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: Detected new or changed devices on smithi088 2024-03-12T05:50:20.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:20.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:20.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi088 to 8540M 2024-03-12T05:50:20.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:20.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:50:20.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:50:20.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:20.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:50:20.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:20 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:20.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.944+0000 7f564dee5700 1 -- 172.21.15.76:0/2494754571 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5648101ca0 msgr2=0x7f56481020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:50:20.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.944+0000 7f564dee5700 1 --2- 172.21.15.76:0/2494754571 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5648101ca0 0x7f56481020c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f56380099c0 tx=0x7f5638009cd0 comp rx=0 tx=0).stop 2024-03-12T05:50:20.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.944+0000 7f564dee5700 1 -- 172.21.15.76:0/2494754571 shutdown_connections 2024-03-12T05:50:20.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.944+0000 7f564dee5700 1 --2- 172.21.15.76:0/2494754571 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5648103840 0x7f564810a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:20.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.944+0000 7f564dee5700 1 --2- 172.21.15.76:0/2494754571 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5648101ca0 0x7f56481020c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:20.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.944+0000 7f564dee5700 1 --2- 172.21.15.76:0/2494754571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5648102ea0 0x7f5648103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:20.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.944+0000 7f564dee5700 1 -- 172.21.15.76:0/2494754571 >> 172.21.15.76:0/2494754571 conn(0x7f56480fd280 msgr2=0x7f56480ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:50:20.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.944+0000 7f564dee5700 1 -- 172.21.15.76:0/2494754571 shutdown_connections 2024-03-12T05:50:20.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.944+0000 7f564dee5700 1 -- 172.21.15.76:0/2494754571 wait complete. 2024-03-12T05:50:20.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.944+0000 7f564dee5700 1 Processor -- start 2024-03-12T05:50:20.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.944+0000 7f564dee5700 1 -- start start 2024-03-12T05:50:20.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f564dee5700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5648101ca0 0x7f5648199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:50:20.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f564dee5700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5648102ea0 0x7f564819a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:50:20.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f564dee5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5648103840 0x7f56481a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:50:20.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f564dee5700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56481a1a40 con 0x7f5648102ea0 2024-03-12T05:50:20.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f564dee5700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56481a1b80 con 0x7f5648101ca0 2024-03-12T05:50:20.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f564dee5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56481a1d10 con 0x7f5648103840 2024-03-12T05:50:20.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f5646ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5648102ea0 0x7f564819a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:50:20.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f5646ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5648102ea0 0x7f564819a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46636/0 (socket says 172.21.15.76:46636) 2024-03-12T05:50:20.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f5646ffd700 1 -- 172.21.15.76:0/1802839131 learned_addr learned my addr 172.21.15.76:0/1802839131 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:50:20.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f5647fff700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5648103840 0x7f56481a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:50:20.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f56477fe700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5648101ca0 0x7f5648199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:50:20.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f5646ffd700 1 -- 172.21.15.76:0/1802839131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5648103840 msgr2=0x7f56481a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:50:20.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f5646ffd700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5648103840 0x7f56481a1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:20.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f5646ffd700 1 -- 172.21.15.76:0/1802839131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5648101ca0 msgr2=0x7f5648199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:50:20.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f5646ffd700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5648101ca0 0x7f5648199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:20.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f5646ffd700 1 -- 172.21.15.76:0/1802839131 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56380095f0 con 0x7f5648102ea0 2024-03-12T05:50:20.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f56477fe700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5648101ca0 0x7f5648199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:50:20.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f5646ffd700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5648102ea0 0x7f564819a3f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f563c00c950 tx=0x7f563c00cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:50:20.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f5644ff9700 1 -- 172.21.15.76:0/1802839131 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c007ac0 con 0x7f5648102ea0 2024-03-12T05:50:20.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f564dee5700 1 -- 172.21.15.76:0/1802839131 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56481a1ff0 con 0x7f5648102ea0 2024-03-12T05:50:20.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f5644ff9700 1 -- 172.21.15.76:0/1802839131 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f563c004d10 con 0x7f5648102ea0 2024-03-12T05:50:20.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f5644ff9700 1 -- 172.21.15.76:0/1802839131 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c005610 con 0x7f5648102ea0 2024-03-12T05:50:20.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.945+0000 7f564dee5700 1 -- 172.21.15.76:0/1802839131 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56481a24b0 con 0x7f5648102ea0 2024-03-12T05:50:20.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.946+0000 7f564dee5700 1 -- 172.21.15.76:0/1802839131 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5648068930 con 0x7f5648102ea0 2024-03-12T05:50:20.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.947+0000 7f5644ff9700 1 -- 172.21.15.76:0/1802839131 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f563c004750 con 0x7f5648102ea0 2024-03-12T05:50:20.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.948+0000 7f5644ff9700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f56300a1ac0 0x7f56300a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:50:20.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.948+0000 7f56477fe700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f56300a1ac0 0x7f56300a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:50:20.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.948+0000 7f5644ff9700 1 -- 172.21.15.76:0/1802839131 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f563c0cc630 con 0x7f5648102ea0 2024-03-12T05:50:20.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.949+0000 7f56477fe700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f56300a1ac0 0x7f56300a3f80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5638006010 tx=0x7f5638009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:50:20.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:20.952+0000 7f5644ff9700 1 -- 172.21.15.76:0/1802839131 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f563c096640 con 0x7f5648102ea0 2024-03-12T05:50:21.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:21.153+0000 7f564dee5700 1 -- 172.21.15.76:0/1802839131 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f56480755c0 con 0x7f56300a1ac0 2024-03-12T05:50:21.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:21 smithi076 ceph-mon[32095]: osdmap e39: 6 total, 6 up, 6 in 2024-03-12T05:50:21.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:21 smithi076 ceph-mon[32095]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:21.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:50:21.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:50:21.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:21.651 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:21 smithi088 ceph-mon[36585]: osdmap e39: 6 total, 6 up, 6 in 2024-03-12T05:50:21.651 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:21 smithi088 ceph-mon[36585]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:21.651 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:21 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:50:21.651 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:21 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:50:21.651 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:21 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:21.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:21 smithi082 ceph-mon[36382]: osdmap e39: 6 total, 6 up, 6 in 2024-03-12T05:50:21.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:21 smithi082 ceph-mon[36382]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:21.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:21 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:50:21.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:21 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:50:21.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:21 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:22.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:22 smithi076 ceph-mon[32095]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:50:22.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:22 smithi076 ceph-mon[32095]: osdmap e40: 6 total, 6 up, 6 in 2024-03-12T05:50:22.654 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:22 smithi088 ceph-mon[36585]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:50:22.655 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:22 smithi088 ceph-mon[36585]: osdmap e40: 6 total, 6 up, 6 in 2024-03-12T05:50:22.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:22 smithi082 ceph-mon[36382]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:50:22.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:22 smithi082 ceph-mon[36382]: osdmap e40: 6 total, 6 up, 6 in 2024-03-12T05:50:23.595 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:23 smithi082 ceph-mon[36382]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T05:50:23.595 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:23 smithi082 ceph-mon[36382]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:23.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:23 smithi088 ceph-mon[36585]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T05:50:23.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:23 smithi088 ceph-mon[36585]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:23.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:23 smithi076 ceph-mon[32095]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T05:50:23.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:23 smithi076 ceph-mon[32095]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:24.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:24 smithi076 ceph-mon[32095]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T05:50:24.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:24 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3588662014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4"}]: dispatch 2024-03-12T05:50:24.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:24 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3588662014' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4"}]': finished 2024-03-12T05:50:24.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:24 smithi076 ceph-mon[32095]: osdmap e43: 7 total, 6 up, 7 in 2024-03-12T05:50:24.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:24.612 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:24 smithi082 ceph-mon[36382]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T05:50:24.612 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:24 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3588662014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4"}]: dispatch 2024-03-12T05:50:24.612 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:24 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3588662014' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4"}]': finished 2024-03-12T05:50:24.612 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:24 smithi082 ceph-mon[36382]: osdmap e43: 7 total, 6 up, 7 in 2024-03-12T05:50:24.612 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:24.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:24 smithi088 ceph-mon[36585]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T05:50:24.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:24 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3588662014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4"}]: dispatch 2024-03-12T05:50:24.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:24 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3588662014' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4"}]': finished 2024-03-12T05:50:24.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:24 smithi088 ceph-mon[36585]: osdmap e43: 7 total, 6 up, 7 in 2024-03-12T05:50:24.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:25.693 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:25 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4131433108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:50:25.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:25 smithi082 ceph-mon[36382]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T05:50:25.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:25 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4131433108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:50:25.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:25 smithi088 ceph-mon[36585]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T05:50:25.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:25 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4131433108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:50:25.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:25 smithi076 ceph-mon[32095]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T05:50:28.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:27 smithi082 ceph-mon[36382]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-12T05:50:28.229 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:27 smithi076 ceph-mon[32095]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-12T05:50:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:27 smithi088 ceph-mon[36585]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-12T05:50:30.150 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:29 smithi076 ceph-mon[32095]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-12T05:50:30.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:29 smithi082 ceph-mon[36382]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-12T05:50:30.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:29 smithi088 ceph-mon[36585]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-12T05:50:32.144 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:31 smithi076 ceph-mon[32095]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:50:32.144 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T05:50:32.144 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:31 smithi082 ceph-mon[36382]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:50:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T05:50:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:31 smithi088 ceph-mon[36585]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:50:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T05:50:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:33.179 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:32 smithi076 ceph-mon[32095]: Deploying daemon osd.6 on smithi076 2024-03-12T05:50:33.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:32 smithi082 ceph-mon[36382]: Deploying daemon osd.6 on smithi076 2024-03-12T05:50:33.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:32 smithi088 ceph-mon[36585]: Deploying daemon osd.6 on smithi076 2024-03-12T05:50:34.108 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:33 smithi076 ceph-mon[32095]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-12T05:50:34.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:33 smithi082 ceph-mon[36382]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-12T05:50:34.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:33 smithi088 ceph-mon[36585]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-12T05:50:36.133 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:35 smithi076 ceph-mon[32095]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-12T05:50:36.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:35 smithi082 ceph-mon[36382]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-12T05:50:36.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:35 smithi088 ceph-mon[36585]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-12T05:50:37.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:37 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:37.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:37 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:37.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:37 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:50:37.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:37 smithi088 ceph-mon[36585]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:37.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:37 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:37.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:37 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:37.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:37 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:50:37.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:37 smithi076 ceph-mon[32095]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:38.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:37 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:38.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:37 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:38.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:37 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:50:38.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:37 smithi082 ceph-mon[36382]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:40.076 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:39 smithi076 ceph-mon[32095]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:40.077 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:39 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:40.077 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:39 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:40.077 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:39 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:40.077 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:39 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:50:40.077 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:39 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:40.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:39 smithi082 ceph-mon[36382]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:40.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:39 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:40.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:39 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:40.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:39 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:40.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:39 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:50:40.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:39 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:39 smithi088 ceph-mon[36585]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:39 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:39 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:39 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:39 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:50:40.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:39 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:40.515 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 6 on host 'smithi076' 2024-03-12T05:50:40.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:40.511+0000 7f5644ff9700 1 -- 172.21.15.76:0/1802839131 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f56480755c0 con 0x7f56300a1ac0 2024-03-12T05:50:40.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:40.513+0000 7f564dee5700 1 -- 172.21.15.76:0/1802839131 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f56300a1ac0 msgr2=0x7f56300a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:50:40.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:40.513+0000 7f564dee5700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f56300a1ac0 0x7f56300a3f80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5638006010 tx=0x7f5638009f10 comp rx=0 tx=0).stop 2024-03-12T05:50:40.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:40.513+0000 7f564dee5700 1 -- 172.21.15.76:0/1802839131 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5648102ea0 msgr2=0x7f564819a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:50:40.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:40.513+0000 7f564dee5700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5648102ea0 0x7f564819a3f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f563c00c950 tx=0x7f563c00cd10 comp rx=0 tx=0).stop 2024-03-12T05:50:40.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:40.513+0000 7f564dee5700 1 -- 172.21.15.76:0/1802839131 shutdown_connections 2024-03-12T05:50:40.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:40.513+0000 7f564dee5700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f56300a1ac0 0x7f56300a3f80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:40.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:40.513+0000 7f564dee5700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5648103840 0x7f56481a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:40.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:40.513+0000 7f564dee5700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5648101ca0 0x7f5648199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:40.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:40.513+0000 7f564dee5700 1 --2- 172.21.15.76:0/1802839131 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5648102ea0 0x7f564819a3f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:40.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:40.513+0000 7f564dee5700 1 -- 172.21.15.76:0/1802839131 >> 172.21.15.76:0/1802839131 conn(0x7f56480fd280 msgr2=0x7f5648073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:50:40.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:40.513+0000 7f564dee5700 1 -- 172.21.15.76:0/1802839131 shutdown_connections 2024-03-12T05:50:40.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:50:40.513+0000 7f564dee5700 1 -- 172.21.15.76:0/1802839131 wait complete. 2024-03-12T05:50:41.131 DEBUG:teuthology.orchestra.run.smithi076:osd.6> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.6.service 2024-03-12T05:50:41.133 INFO:tasks.cephadm:Deploying osd.7 on smithi082 with /dev/vg_nvme/lv_2... 2024-03-12T05:50:41.133 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-12T05:50:41.380 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:50:41.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:41.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:41.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:50:41.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:41 smithi076 ceph-mon[32095]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T05:50:41.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:41 smithi076 ceph-mon[32095]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:41.804 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:41.804 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:41.804 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:50:41.804 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:41 smithi082 ceph-mon[36382]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T05:50:41.804 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:41 smithi082 ceph-mon[36382]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:41.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:41.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:41.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:50:41.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:41 smithi088 ceph-mon[36585]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T05:50:41.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:41 smithi088 ceph-mon[36585]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:42.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:42 smithi076 ceph-mon[32095]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T05:50:42.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:42 smithi076 ceph-mon[32095]: osdmap e44: 7 total, 6 up, 7 in 2024-03-12T05:50:42.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:42 smithi076 ceph-mon[32095]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-12T05:50:42.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:42 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:42.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:42 smithi082 ceph-mon[36382]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T05:50:42.785 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:42 smithi082 ceph-mon[36382]: osdmap e44: 7 total, 6 up, 7 in 2024-03-12T05:50:42.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:42 smithi082 ceph-mon[36382]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-12T05:50:42.786 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:42 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:42.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:42 smithi088 ceph-mon[36585]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T05:50:42.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:42 smithi088 ceph-mon[36585]: osdmap e44: 7 total, 6 up, 7 in 2024-03-12T05:50:42.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:42 smithi088 ceph-mon[36585]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-12T05:50:42.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:42 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:43.334 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-12T05:50:43.335 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-12T05:50:43.335 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-03-12T05:50:43.335 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-03-12T05:50:43.335 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0119624 s, 877 MB/s 2024-03-12T05:50:43.335 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-03-12T05:50:43.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:43 smithi076 ceph-mon[32095]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-12T05:50:43.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:43 smithi076 ceph-mon[32095]: osdmap e45: 7 total, 6 up, 7 in 2024-03-12T05:50:43.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:43.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:43 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:43.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:43 smithi076 ceph-mon[32095]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:43.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:43 smithi076 ceph-mon[32095]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' 2024-03-12T05:50:43.737 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 05:50:43 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T05:50:43.360+0000 7fa732462700 -1 osd.6 0 waiting for initial osdmap 2024-03-12T05:50:43.738 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 05:50:43 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T05:50:43.374+0000 7fa72924b700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:50:43.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:43 smithi082 ceph-mon[36382]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-12T05:50:43.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:43 smithi082 ceph-mon[36382]: osdmap e45: 7 total, 6 up, 7 in 2024-03-12T05:50:43.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:43.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:43 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:43.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:43 smithi082 ceph-mon[36382]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:43.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:43 smithi082 ceph-mon[36382]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' 2024-03-12T05:50:43.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:43 smithi088 ceph-mon[36585]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-12T05:50:43.987 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:43 smithi088 ceph-mon[36585]: osdmap e45: 7 total, 6 up, 7 in 2024-03-12T05:50:43.987 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:43.987 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:43 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:43.987 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:43 smithi088 ceph-mon[36585]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-12T05:50:43.987 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:43 smithi088 ceph-mon[36585]: from='osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935]' entity='osd.6' 2024-03-12T05:50:43.998 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi082:vg_nvme/lv_2 2024-03-12T05:50:44.241 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:50:44.634 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: purged_snaps scrub starts 2024-03-12T05:50:44.634 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: purged_snaps scrub ok 2024-03-12T05:50:44.634 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:44.634 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:44.635 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:44.635 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:44.635 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:44.635 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:44.635 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:44.635 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:44.635 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:50:44.635 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:44.635 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935] boot 2024-03-12T05:50:44.635 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: osdmap e46: 7 total, 7 up, 7 in 2024-03-12T05:50:44.635 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:44.804 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: purged_snaps scrub starts 2024-03-12T05:50:44.804 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: purged_snaps scrub ok 2024-03-12T05:50:44.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:44.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:44.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:44.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:44.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:44.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:44.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:44.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:44.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:50:44.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:44.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935] boot 2024-03-12T05:50:44.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: osdmap e46: 7 total, 7 up, 7 in 2024-03-12T05:50:44.806 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:44.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: purged_snaps scrub starts 2024-03-12T05:50:44.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: purged_snaps scrub ok 2024-03-12T05:50:44.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:44.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:44.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:44.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:44.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:44.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:50:44.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:44.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:44.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:50:44.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:50:44.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: osd.6 [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935] boot 2024-03-12T05:50:44.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: osdmap e46: 7 total, 7 up, 7 in 2024-03-12T05:50:44.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:50:45.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.792+0000 7f1df01dd700 1 -- 172.21.15.82:0/4278528937 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de8101da0 msgr2=0x7f1de8102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:50:45.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.792+0000 7f1df01dd700 1 --2- 172.21.15.82:0/4278528937 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de8101da0 0x7f1de8102200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80099c0 tx=0x7f1dd8009cd0 comp rx=0 tx=0).stop 2024-03-12T05:50:45.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.792+0000 7f1df01dd700 1 -- 172.21.15.82:0/4278528937 shutdown_connections 2024-03-12T05:50:45.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.792+0000 7f1df01dd700 1 --2- 172.21.15.82:0/4278528937 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1de8102740 0x7f1de8108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:45.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.792+0000 7f1df01dd700 1 --2- 172.21.15.82:0/4278528937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1de8100ba0 0x7f1de8100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:45.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.792+0000 7f1df01dd700 1 --2- 172.21.15.82:0/4278528937 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de8101da0 0x7f1de8102200 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:45.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.792+0000 7f1df01dd700 1 -- 172.21.15.82:0/4278528937 >> 172.21.15.82:0/4278528937 conn(0x7f1de80fc160 msgr2=0x7f1de80fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:50:45.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.792+0000 7f1df01dd700 1 -- 172.21.15.82:0/4278528937 shutdown_connections 2024-03-12T05:50:45.795 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.792+0000 7f1df01dd700 1 -- 172.21.15.82:0/4278528937 wait complete. 2024-03-12T05:50:45.796 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:45 smithi082 ceph-mon[36382]: Detected new or changed devices on smithi076 2024-03-12T05:50:45.796 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:45 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi076 to 5696M 2024-03-12T05:50:45.796 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:45 smithi082 ceph-mon[36382]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:45.796 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:45 smithi082 ceph-mon[36382]: osdmap e47: 7 total, 7 up, 7 in 2024-03-12T05:50:45.797 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.792+0000 7f1df01dd700 1 Processor -- start 2024-03-12T05:50:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.792+0000 7f1df01dd700 1 -- start start 2024-03-12T05:50:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1df01dd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1de8100ba0 0x7f1de8195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:50:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1df01dd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1de8101da0 0x7f1de8195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:50:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1df01dd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de8102740 0x7f1de819d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:50:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1df01dd700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de819d620 con 0x7f1de8101da0 2024-03-12T05:50:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1df01dd700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de819d790 con 0x7f1de8102740 2024-03-12T05:50:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1df01dd700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de819d950 con 0x7f1de8100ba0 2024-03-12T05:50:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1dee77a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de8102740 0x7f1de819d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:50:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1dee77a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de8102740 0x7f1de819d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:54400/0 (socket says 172.21.15.82:54400) 2024-03-12T05:50:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1dee77a700 1 -- 172.21.15.82:0/4070327961 learned_addr learned my addr 172.21.15.82:0/4070327961 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-03-12T05:50:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1ded778700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1de8101da0 0x7f1de8195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:50:45.798 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1dedf79700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1de8100ba0 0x7f1de8195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:50:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1dee77a700 1 -- 172.21.15.82:0/4070327961 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1de8100ba0 msgr2=0x7f1de8195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:50:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1dee77a700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1de8100ba0 0x7f1de8195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1dee77a700 1 -- 172.21.15.82:0/4070327961 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1de8101da0 msgr2=0x7f1de8195fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:50:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1dee77a700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1de8101da0 0x7f1de8195fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:50:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1dee77a700 1 -- 172.21.15.82:0/4070327961 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd80095f0 con 0x7f1de8102740 2024-03-12T05:50:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1ded778700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1de8101da0 0x7f1de8195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:50:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1dee77a700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de8102740 0x7f1de819d050 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1de0007ee0 tx=0x7f1de000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:50:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1ddeffd700 1 -- 172.21.15.82:0/4070327961 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de000dd20 con 0x7f1de8102740 2024-03-12T05:50:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1ddeffd700 1 -- 172.21.15.82:0/4070327961 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1de000f040 con 0x7f1de8102740 2024-03-12T05:50:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1ddeffd700 1 -- 172.21.15.82:0/4070327961 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de00048e0 con 0x7f1de8102740 2024-03-12T05:50:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.793+0000 7f1df01dd700 1 -- 172.21.15.82:0/4070327961 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de819dc30 con 0x7f1de8102740 2024-03-12T05:50:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.794+0000 7f1df01dd700 1 -- 172.21.15.82:0/4070327961 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de819e0e0 con 0x7f1de8102740 2024-03-12T05:50:45.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.794+0000 7f1df01dd700 1 -- 172.21.15.82:0/4070327961 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de8068930 con 0x7f1de8102740 2024-03-12T05:50:45.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.796+0000 7f1ddeffd700 1 -- 172.21.15.82:0/4070327961 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1de0003a00 con 0x7f1de8102740 2024-03-12T05:50:45.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.797+0000 7f1ddeffd700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1dd40a1ac0 0x7f1dd40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:50:45.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.797+0000 7f1ddeffd700 1 -- 172.21.15.82:0/4070327961 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f1de00cbf30 con 0x7f1de8102740 2024-03-12T05:50:45.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.797+0000 7f1dedf79700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1dd40a1ac0 0x7f1dd40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:50:45.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.797+0000 7f1dedf79700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1dd40a1ac0 0x7f1dd40a3f80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1de8101c00 tx=0x7f1de400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:50:45.804 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:45.801+0000 7f1ddeffd700 1 -- 172.21.15.82:0/4070327961 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1de0095e50 con 0x7f1de8102740 2024-03-12T05:50:45.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:45 smithi088 ceph-mon[36585]: Detected new or changed devices on smithi076 2024-03-12T05:50:45.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:45 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi076 to 5696M 2024-03-12T05:50:45.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:45 smithi088 ceph-mon[36585]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:45.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:45 smithi088 ceph-mon[36585]: osdmap e47: 7 total, 7 up, 7 in 2024-03-12T05:50:45.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:45 smithi076 ceph-mon[32095]: Detected new or changed devices on smithi076 2024-03-12T05:50:45.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:45 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi076 to 5696M 2024-03-12T05:50:45.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:45 smithi076 ceph-mon[32095]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:45.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:45 smithi076 ceph-mon[32095]: osdmap e47: 7 total, 7 up, 7 in 2024-03-12T05:50:46.004 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:50:46.001+0000 7f1df01dd700 1 -- 172.21.15.82:0/4070327961 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1de819e530 con 0x7f1dd40a1ac0 2024-03-12T05:50:46.831 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:46 smithi082 ceph-mon[36382]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:50:46.831 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:46 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:50:46.831 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:46 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:50:46.832 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:46 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:46 smithi088 ceph-mon[36585]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:50:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:46 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:50:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:46 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:50:46.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:46 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:46.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:46 smithi076 ceph-mon[32095]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:50:46.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:46 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:50:46.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:46 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:50:46.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:46 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:47.867 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:47 smithi082 ceph-mon[36382]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:47.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:47 smithi088 ceph-mon[36585]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:47.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:47 smithi076 ceph-mon[32095]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:48.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:48 smithi088 ceph-mon[36585]: from='client.? 172.21.15.82:0/715464940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "233f8b14-0d82-43d6-96f7-bc9a4ed524a9"}]: dispatch 2024-03-12T05:50:48.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:48 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "233f8b14-0d82-43d6-96f7-bc9a4ed524a9"}]: dispatch 2024-03-12T05:50:48.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:48 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "233f8b14-0d82-43d6-96f7-bc9a4ed524a9"}]': finished 2024-03-12T05:50:48.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:48 smithi088 ceph-mon[36585]: osdmap e48: 8 total, 7 up, 8 in 2024-03-12T05:50:48.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:48 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:50:48.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:48 smithi076 ceph-mon[32095]: from='client.? 172.21.15.82:0/715464940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "233f8b14-0d82-43d6-96f7-bc9a4ed524a9"}]: dispatch 2024-03-12T05:50:48.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:48 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "233f8b14-0d82-43d6-96f7-bc9a4ed524a9"}]: dispatch 2024-03-12T05:50:48.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:48 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "233f8b14-0d82-43d6-96f7-bc9a4ed524a9"}]': finished 2024-03-12T05:50:48.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:48 smithi076 ceph-mon[32095]: osdmap e48: 8 total, 7 up, 8 in 2024-03-12T05:50:48.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:48 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:50:49.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:48 smithi082 ceph-mon[36382]: from='client.? 172.21.15.82:0/715464940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "233f8b14-0d82-43d6-96f7-bc9a4ed524a9"}]: dispatch 2024-03-12T05:50:49.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:48 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "233f8b14-0d82-43d6-96f7-bc9a4ed524a9"}]: dispatch 2024-03-12T05:50:49.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:48 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "233f8b14-0d82-43d6-96f7-bc9a4ed524a9"}]': finished 2024-03-12T05:50:49.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:48 smithi082 ceph-mon[36382]: osdmap e48: 8 total, 7 up, 8 in 2024-03-12T05:50:49.031 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:48 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:50:49.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:49 smithi076 ceph-mon[32095]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:49.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:49 smithi076 ceph-mon[32095]: from='client.? 172.21.15.82:0/3466703181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:50:50.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:49 smithi082 ceph-mon[36382]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:50.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:49 smithi082 ceph-mon[36382]: from='client.? 172.21.15.82:0/3466703181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:50:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:49 smithi088 ceph-mon[36585]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:49 smithi088 ceph-mon[36585]: from='client.? 172.21.15.82:0/3466703181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:50:52.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:51 smithi082 ceph-mon[36382]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:51 smithi088 ceph-mon[36585]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:52.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:51 smithi076 ceph-mon[32095]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:54.074 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:53 smithi082 ceph-mon[36382]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:53 smithi088 ceph-mon[36585]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:54.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:53 smithi076 ceph-mon[32095]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:56.193 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:55 smithi082 ceph-mon[36382]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:56.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:55 smithi088 ceph-mon[36585]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:56.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:55 smithi076 ceph-mon[32095]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:57.188 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T05:50:57.188 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:57.189 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:56 smithi082 ceph-mon[36382]: Deploying daemon osd.7 on smithi082 2024-03-12T05:50:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T05:50:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:56 smithi088 ceph-mon[36585]: Deploying daemon osd.7 on smithi082 2024-03-12T05:50:57.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T05:50:57.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:50:57.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:56 smithi076 ceph-mon[32095]: Deploying daemon osd.7 on smithi082 2024-03-12T05:50:58.083 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:57 smithi082 ceph-mon[36382]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:57 smithi088 ceph-mon[36585]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:50:58.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:57 smithi076 ceph-mon[32095]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:00.088 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:50:59 smithi082 ceph-mon[36382]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:00.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:50:59 smithi088 ceph-mon[36585]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:00.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:50:59 smithi076 ceph-mon[32095]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:02.151 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:01 smithi082 ceph-mon[36382]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:02.151 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:01 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:02.151 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:01 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:02.151 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:01 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:01 smithi088 ceph-mon[36585]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:02.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:01 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:01 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:01 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:02.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:01 smithi076 ceph-mon[32095]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:02.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:01 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:02.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:01 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:02.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:01 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:04.077 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:03 smithi082 ceph-mon[36382]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:04.077 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:03 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:04.077 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:03 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:04.078 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:03 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:04.078 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:03 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:03 smithi088 ceph-mon[36585]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:03 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:03 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:04.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:03 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:04.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:03 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:04.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:03 smithi076 ceph-mon[32095]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:04.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:04.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:04.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:04.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:04.719 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:51:04.717+0000 7f1ddeffd700 1 -- 172.21.15.82:0/4070327961 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1de819e530 con 0x7f1dd40a1ac0 2024-03-12T05:51:04.719 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 7 on host 'smithi082' 2024-03-12T05:51:04.721 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:51:04.718+0000 7f1df01dd700 1 -- 172.21.15.82:0/4070327961 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1dd40a1ac0 msgr2=0x7f1dd40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:51:04.721 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:51:04.718+0000 7f1df01dd700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1dd40a1ac0 0x7f1dd40a3f80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1de8101c00 tx=0x7f1de400b410 comp rx=0 tx=0).stop 2024-03-12T05:51:04.721 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:51:04.718+0000 7f1df01dd700 1 -- 172.21.15.82:0/4070327961 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de8102740 msgr2=0x7f1de819d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:51:04.721 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:51:04.718+0000 7f1df01dd700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de8102740 0x7f1de819d050 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1de0007ee0 tx=0x7f1de000d3b0 comp rx=0 tx=0).stop 2024-03-12T05:51:04.721 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:51:04.718+0000 7f1df01dd700 1 -- 172.21.15.82:0/4070327961 shutdown_connections 2024-03-12T05:51:04.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:51:04.718+0000 7f1df01dd700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1dd40a1ac0 0x7f1dd40a3f80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:04.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:51:04.718+0000 7f1df01dd700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1de8102740 0x7f1de819d050 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:04.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:51:04.718+0000 7f1df01dd700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1de8100ba0 0x7f1de8195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:04.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:51:04.718+0000 7f1df01dd700 1 --2- 172.21.15.82:0/4070327961 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1de8101da0 0x7f1de8195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:04.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:51:04.718+0000 7f1df01dd700 1 -- 172.21.15.82:0/4070327961 >> 172.21.15.82:0/4070327961 conn(0x7f1de80fc160 msgr2=0x7f1de80fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:51:04.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:51:04.719+0000 7f1df01dd700 1 -- 172.21.15.82:0/4070327961 shutdown_connections 2024-03-12T05:51:04.722 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:51:04.719+0000 7f1df01dd700 1 -- 172.21.15.82:0/4070327961 wait complete. 2024-03-12T05:51:05.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:04 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:05.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:04 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:05.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:04 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:05.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:04 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:04 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:04 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:04 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:05.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:04 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:05.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:04 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:05.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:04 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:05.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:04 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:05.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:04 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:05.404 DEBUG:teuthology.orchestra.run.smithi082:osd.7> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.7.service 2024-03-12T05:51:05.407 INFO:tasks.cephadm:Deploying osd.8 on smithi088 with /dev/vg_nvme/lv_2... 2024-03-12T05:51:05.407 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-12T05:51:05.653 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:51:05.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:05 smithi082 ceph-mon[36382]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:05.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:05 smithi082 ceph-mon[36382]: from='osd.7 [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T05:51:05.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:05 smithi082 ceph-mon[36382]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T05:51:06.125 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:05 smithi088 ceph-mon[36585]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:06.125 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:05 smithi088 ceph-mon[36585]: from='osd.7 [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T05:51:06.125 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:05 smithi088 ceph-mon[36585]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T05:51:06.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:05 smithi076 ceph-mon[32095]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:06.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:05 smithi076 ceph-mon[32095]: from='osd.7 [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T05:51:06.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:05 smithi076 ceph-mon[32095]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T05:51:06.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:06 smithi088 ceph-mon[36585]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T05:51:06.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:06 smithi088 ceph-mon[36585]: osdmap e49: 8 total, 7 up, 8 in 2024-03-12T05:51:06.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:06 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:51:06.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:06 smithi088 ceph-mon[36585]: from='osd.7 [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:51:06.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:06 smithi088 ceph-mon[36585]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:51:07.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:06 smithi082 ceph-mon[36382]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T05:51:07.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:06 smithi082 ceph-mon[36382]: osdmap e49: 8 total, 7 up, 8 in 2024-03-12T05:51:07.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:06 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:51:07.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:06 smithi082 ceph-mon[36382]: from='osd.7 [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:51:07.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:06 smithi082 ceph-mon[36382]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:51:07.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:06 smithi076 ceph-mon[32095]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T05:51:07.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:06 smithi076 ceph-mon[32095]: osdmap e49: 8 total, 7 up, 8 in 2024-03-12T05:51:07.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:06 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:51:07.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:06 smithi076 ceph-mon[32095]: from='osd.7 [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:51:07.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:06 smithi076 ceph-mon[32095]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:51:07.564 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-12T05:51:07.564 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-12T05:51:07.564 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2024-03-12T05:51:07.564 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2024-03-12T05:51:07.564 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109299 s, 959 MB/s 2024-03-12T05:51:07.564 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2024-03-12T05:51:08.013 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:07 smithi082 ceph-mon[36382]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:08.013 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:07 smithi082 ceph-mon[36382]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-12T05:51:08.013 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:07 smithi082 ceph-mon[36382]: osdmap e50: 8 total, 7 up, 8 in 2024-03-12T05:51:08.013 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:51:08.013 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:51:08.013 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:07 smithi082 ceph-mon[36382]: from='osd.7 ' entity='osd.7' 2024-03-12T05:51:08.013 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:07 smithi082 ceph-mon[36382]: osd.7 [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522] boot 2024-03-12T05:51:08.014 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:07 smithi082 ceph-mon[36382]: osdmap e51: 8 total, 8 up, 8 in 2024-03-12T05:51:08.014 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:07 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:51:08.014 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 05:51:07 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T05:51:07.756+0000 7fd14d1e1700 -1 osd.7 0 waiting for initial osdmap 2024-03-12T05:51:08.014 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 05:51:07 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T05:51:07.769+0000 7fd1457cd700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:51:08.168 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:07 smithi088 ceph-mon[36585]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:08.168 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:07 smithi088 ceph-mon[36585]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-12T05:51:08.168 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:07 smithi088 ceph-mon[36585]: osdmap e50: 8 total, 7 up, 8 in 2024-03-12T05:51:08.168 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:51:08.169 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:51:08.169 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:07 smithi088 ceph-mon[36585]: from='osd.7 ' entity='osd.7' 2024-03-12T05:51:08.169 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:07 smithi088 ceph-mon[36585]: osd.7 [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522] boot 2024-03-12T05:51:08.169 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:07 smithi088 ceph-mon[36585]: osdmap e51: 8 total, 8 up, 8 in 2024-03-12T05:51:08.169 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:07 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:51:08.195 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi088:vg_nvme/lv_2 2024-03-12T05:51:08.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:07 smithi076 ceph-mon[32095]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-12T05:51:08.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:07 smithi076 ceph-mon[32095]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-12T05:51:08.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:07 smithi076 ceph-mon[32095]: osdmap e50: 8 total, 7 up, 8 in 2024-03-12T05:51:08.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:51:08.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:51:08.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:07 smithi076 ceph-mon[32095]: from='osd.7 ' entity='osd.7' 2024-03-12T05:51:08.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:07 smithi076 ceph-mon[32095]: osd.7 [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522] boot 2024-03-12T05:51:08.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:07 smithi076 ceph-mon[32095]: osdmap e51: 8 total, 8 up, 8 in 2024-03-12T05:51:08.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:07 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:51:08.427 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:51:09.182 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:08 smithi088 ceph-mon[36585]: purged_snaps scrub starts 2024-03-12T05:51:09.182 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:08 smithi088 ceph-mon[36585]: purged_snaps scrub ok 2024-03-12T05:51:09.183 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:09.183 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:09.183 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:09.183 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:09.183 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:09.183 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:09.183 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:09.183 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:09.183 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:08 smithi082 ceph-mon[36382]: purged_snaps scrub starts 2024-03-12T05:51:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:08 smithi082 ceph-mon[36382]: purged_snaps scrub ok 2024-03-12T05:51:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:09.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:09.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:08 smithi076 ceph-mon[32095]: purged_snaps scrub starts 2024-03-12T05:51:09.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:08 smithi076 ceph-mon[32095]: purged_snaps scrub ok 2024-03-12T05:51:09.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:09.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:09.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:09.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:09.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:09.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:09.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:09.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:09.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:09.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.903+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2855726630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bd0105060 msgr2=0x7f3bd0107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:51:09.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.903+0000 7f3bd4df7700 1 --2- 172.21.15.88:0/2855726630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bd0105060 0x7f3bd0107450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8009990 tx=0x7f3bb8009ca0 comp rx=0 tx=0).stop 2024-03-12T05:51:09.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.903+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2855726630 shutdown_connections 2024-03-12T05:51:09.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.903+0000 7f3bd4df7700 1 --2- 172.21.15.88:0/2855726630 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bd0107990 0x7f3bd0109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:09.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.903+0000 7f3bd4df7700 1 --2- 172.21.15.88:0/2855726630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd0102730 0x7f3bd0104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:09.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.903+0000 7f3bd4df7700 1 --2- 172.21.15.88:0/2855726630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bd0105060 0x7f3bd0107450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:09.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.903+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2855726630 >> 172.21.15.88:0/2855726630 conn(0x7f3bd00fc140 msgr2=0x7f3bd00fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:51:09.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.903+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2855726630 shutdown_connections 2024-03-12T05:51:09.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.903+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2855726630 wait complete. 2024-03-12T05:51:09.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.904+0000 7f3bd4df7700 1 Processor -- start 2024-03-12T05:51:09.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.904+0000 7f3bd4df7700 1 -- start start 2024-03-12T05:51:09.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.904+0000 7f3bd4df7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bd0102730 0x7f3bd0197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:51:09.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.904+0000 7f3bd4df7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bd0105060 0x7f3bd0198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:51:09.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.904+0000 7f3bd4df7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd0107990 0x7f3bd019f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:51:09.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.904+0000 7f3bd4df7700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd019f850 con 0x7f3bd0102730 2024-03-12T05:51:09.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.904+0000 7f3bd4df7700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd019f990 con 0x7f3bd0107990 2024-03-12T05:51:09.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.904+0000 7f3bd4df7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd019fb50 con 0x7f3bd0105060 2024-03-12T05:51:09.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.904+0000 7f3bc5d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bd0105060 0x7f3bd0198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:51:09.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.904+0000 7f3bc5d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bd0105060 0x7f3bd0198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54292/0 (socket says 172.21.15.88:54292) 2024-03-12T05:51:09.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.904+0000 7f3bc5d9b700 1 -- 172.21.15.88:0/2207627389 learned_addr learned my addr 172.21.15.88:0/2207627389 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-03-12T05:51:09.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.904+0000 7f3bce59c700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bd0102730 0x7f3bd0197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:51:09.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.905+0000 7f3bced9d700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd0107990 0x7f3bd019f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:51:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.905+0000 7f3bc5d9b700 1 -- 172.21.15.88:0/2207627389 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd0107990 msgr2=0x7f3bd019f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:51:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.905+0000 7f3bc5d9b700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd0107990 0x7f3bd019f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.905+0000 7f3bc5d9b700 1 -- 172.21.15.88:0/2207627389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bd0102730 msgr2=0x7f3bd0197d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:51:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.905+0000 7f3bc5d9b700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bd0102730 0x7f3bd0197d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.905+0000 7f3bc5d9b700 1 -- 172.21.15.88:0/2207627389 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb80095f0 con 0x7f3bd0105060 2024-03-12T05:51:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.905+0000 7f3bce59c700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bd0102730 0x7f3bd0197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:51:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.905+0000 7f3bc5d9b700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bd0105060 0x7f3bd0198240 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3bb800b5c0 tx=0x7f3bb800f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:51:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.905+0000 7f3bc7fff700 1 -- 172.21.15.88:0/2207627389 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb801d070 con 0x7f3bd0105060 2024-03-12T05:51:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.905+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2207627389 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd019fdd0 con 0x7f3bd0105060 2024-03-12T05:51:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.905+0000 7f3bc7fff700 1 -- 172.21.15.88:0/2207627389 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bb800fc30 con 0x7f3bd0105060 2024-03-12T05:51:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.905+0000 7f3bc7fff700 1 -- 172.21.15.88:0/2207627389 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8017650 con 0x7f3bd0105060 2024-03-12T05:51:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.905+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2207627389 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd01a0190 con 0x7f3bd0105060 2024-03-12T05:51:09.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.906+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2207627389 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd00fdcc0 con 0x7f3bd0105060 2024-03-12T05:51:09.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.907+0000 7f3bc7fff700 1 -- 172.21.15.88:0/2207627389 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3bb80177b0 con 0x7f3bd0105060 2024-03-12T05:51:09.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.907+0000 7f3bc7fff700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3bbc0a1ac0 0x7f3bbc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:51:09.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.908+0000 7f3bc7fff700 1 -- 172.21.15.88:0/2207627389 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f3bb80ce470 con 0x7f3bd0105060 2024-03-12T05:51:09.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.908+0000 7f3bce59c700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3bbc0a1ac0 0x7f3bbc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:51:09.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.908+0000 7f3bce59c700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3bbc0a1ac0 0x7f3bbc0a3f80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3bd01991e0 tx=0x7f3bc0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:51:09.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:09.911+0000 7f3bc7fff700 1 -- 172.21.15.88:0/2207627389 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3bb8098250 con 0x7f3bd0105060 2024-03-12T05:51:09.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:09 smithi088 ceph-mon[36585]: Detected new or changed devices on smithi082 2024-03-12T05:51:09.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:09 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi082 to 5693M 2024-03-12T05:51:09.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:09 smithi088 ceph-mon[36585]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 484 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:09.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:09 smithi088 ceph-mon[36585]: osdmap e52: 8 total, 8 up, 8 in 2024-03-12T05:51:09.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:09 smithi088 ceph-mon[36585]: osdmap e53: 8 total, 8 up, 8 in 2024-03-12T05:51:10.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:10.119+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2207627389 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bd002d8d0 con 0x7f3bbc0a1ac0 2024-03-12T05:51:10.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:09 smithi082 ceph-mon[36382]: Detected new or changed devices on smithi082 2024-03-12T05:51:10.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:09 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi082 to 5693M 2024-03-12T05:51:10.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:09 smithi082 ceph-mon[36382]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 484 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:10.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:09 smithi082 ceph-mon[36382]: osdmap e52: 8 total, 8 up, 8 in 2024-03-12T05:51:10.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:09 smithi082 ceph-mon[36382]: osdmap e53: 8 total, 8 up, 8 in 2024-03-12T05:51:10.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:09 smithi076 ceph-mon[32095]: Detected new or changed devices on smithi082 2024-03-12T05:51:10.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:09 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi082 to 5693M 2024-03-12T05:51:10.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:09 smithi076 ceph-mon[32095]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 484 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:10.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:09 smithi076 ceph-mon[32095]: osdmap e52: 8 total, 8 up, 8 in 2024-03-12T05:51:10.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:09 smithi076 ceph-mon[32095]: osdmap e53: 8 total, 8 up, 8 in 2024-03-12T05:51:10.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:10 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:51:11.739 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:10 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:51:11.739 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:10 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:11.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:10 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:51:11.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:10 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:51:11.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:10 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:11.740 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:10 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:51:11.740 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:10 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:51:11.740 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:10 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:12.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:11 smithi082 ceph-mon[36382]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:51:12.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:11 smithi082 ceph-mon[36382]: pgmap v178: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 484 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:12.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:11 smithi082 ceph-mon[36382]: osdmap e54: 8 total, 8 up, 8 in 2024-03-12T05:51:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:11 smithi088 ceph-mon[36585]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:51:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:11 smithi088 ceph-mon[36585]: pgmap v178: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 484 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:12.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:11 smithi088 ceph-mon[36585]: osdmap e54: 8 total, 8 up, 8 in 2024-03-12T05:51:12.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:11 smithi076 ceph-mon[32095]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:51:12.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:11 smithi076 ceph-mon[32095]: pgmap v178: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 484 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:12.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:11 smithi076 ceph-mon[32095]: osdmap e54: 8 total, 8 up, 8 in 2024-03-12T05:51:13.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:12 smithi082 ceph-mon[36382]: osdmap e55: 8 total, 8 up, 8 in 2024-03-12T05:51:13.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:12 smithi082 ceph-mon[36382]: from='client.? 172.21.15.88:0/3381182944' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "959d820e-13bb-4a2b-a9fc-11122a0be243"}]: dispatch 2024-03-12T05:51:13.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:12 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "959d820e-13bb-4a2b-a9fc-11122a0be243"}]: dispatch 2024-03-12T05:51:13.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:12 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "959d820e-13bb-4a2b-a9fc-11122a0be243"}]': finished 2024-03-12T05:51:13.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:12 smithi082 ceph-mon[36382]: osdmap e56: 9 total, 8 up, 9 in 2024-03-12T05:51:13.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:12 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:12 smithi088 ceph-mon[36585]: osdmap e55: 8 total, 8 up, 8 in 2024-03-12T05:51:13.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:12 smithi088 ceph-mon[36585]: from='client.? 172.21.15.88:0/3381182944' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "959d820e-13bb-4a2b-a9fc-11122a0be243"}]: dispatch 2024-03-12T05:51:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:12 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "959d820e-13bb-4a2b-a9fc-11122a0be243"}]: dispatch 2024-03-12T05:51:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:12 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "959d820e-13bb-4a2b-a9fc-11122a0be243"}]': finished 2024-03-12T05:51:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:12 smithi088 ceph-mon[36585]: osdmap e56: 9 total, 8 up, 9 in 2024-03-12T05:51:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:12 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:13.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:12 smithi076 ceph-mon[32095]: osdmap e55: 8 total, 8 up, 8 in 2024-03-12T05:51:13.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:12 smithi076 ceph-mon[32095]: from='client.? 172.21.15.88:0/3381182944' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "959d820e-13bb-4a2b-a9fc-11122a0be243"}]: dispatch 2024-03-12T05:51:13.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:12 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "959d820e-13bb-4a2b-a9fc-11122a0be243"}]: dispatch 2024-03-12T05:51:13.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:12 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "959d820e-13bb-4a2b-a9fc-11122a0be243"}]': finished 2024-03-12T05:51:13.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:12 smithi076 ceph-mon[32095]: osdmap e56: 9 total, 8 up, 9 in 2024-03-12T05:51:13.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:12 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:14.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:13 smithi082 ceph-mon[36382]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 485 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:14.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:13 smithi082 ceph-mon[36382]: from='client.? 172.21.15.88:0/3991938581' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:51:14.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:13 smithi088 ceph-mon[36585]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 485 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:14.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:13 smithi088 ceph-mon[36585]: from='client.? 172.21.15.88:0/3991938581' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:51:14.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:13 smithi076 ceph-mon[32095]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 485 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:14.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:13 smithi076 ceph-mon[32095]: from='client.? 172.21.15.88:0/3991938581' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:51:16.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:15 smithi088 ceph-mon[36585]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:16.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:15 smithi076 ceph-mon[32095]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:16.443 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:15 smithi082 ceph-mon[36382]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:18.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:17 smithi088 ceph-mon[36585]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:18.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:17 smithi076 ceph-mon[32095]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:18.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:17 smithi082 ceph-mon[36382]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:20.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:19 smithi088 ceph-mon[36585]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:20.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-12T05:51:20.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:20.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:19 smithi076 ceph-mon[32095]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:20.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-12T05:51:20.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:20.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:19 smithi082 ceph-mon[36382]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:20.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-12T05:51:20.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:21.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:20 smithi088 ceph-mon[36585]: Deploying daemon osd.8 on smithi088 2024-03-12T05:51:21.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:20 smithi076 ceph-mon[32095]: Deploying daemon osd.8 on smithi088 2024-03-12T05:51:21.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:20 smithi082 ceph-mon[36382]: Deploying daemon osd.8 on smithi088 2024-03-12T05:51:22.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:21 smithi088 ceph-mon[36585]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:22.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:21 smithi076 ceph-mon[32095]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:22.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:21 smithi082 ceph-mon[36382]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:24.188 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:23 smithi088 ceph-mon[36585]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:24.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:23 smithi076 ceph-mon[32095]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:24.246 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:23 smithi082 ceph-mon[36382]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:26.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:25 smithi088 ceph-mon[36585]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:26.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:25 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:26.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:25 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:26.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:25 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:26.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:25 smithi076 ceph-mon[32095]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:26.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:25 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:26.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:25 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:26.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:25 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:26.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:25 smithi082 ceph-mon[36382]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:26.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:25 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:26.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:25 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:26.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:25 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:27 smithi088 ceph-mon[36585]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:28.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:28.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:27 smithi076 ceph-mon[32095]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:28.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:28.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:28.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:28.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:28.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:28.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:27 smithi082 ceph-mon[36382]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:28.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:28.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:28.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:28.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:28.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:28.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:28.730+0000 7f3bc7fff700 1 -- 172.21.15.88:0/2207627389 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3bd002d8d0 con 0x7f3bbc0a1ac0 2024-03-12T05:51:28.733 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 8 on host 'smithi088' 2024-03-12T05:51:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:28.733+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2207627389 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3bbc0a1ac0 msgr2=0x7f3bbc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:51:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:28.733+0000 7f3bd4df7700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3bbc0a1ac0 0x7f3bbc0a3f80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3bd01991e0 tx=0x7f3bc0008040 comp rx=0 tx=0).stop 2024-03-12T05:51:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:28.733+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2207627389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bd0105060 msgr2=0x7f3bd0198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:51:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:28.733+0000 7f3bd4df7700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bd0105060 0x7f3bd0198240 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3bb800b5c0 tx=0x7f3bb800f740 comp rx=0 tx=0).stop 2024-03-12T05:51:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:28.733+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2207627389 shutdown_connections 2024-03-12T05:51:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:28.733+0000 7f3bd4df7700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3bbc0a1ac0 0x7f3bbc0a3f80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:28.733+0000 7f3bd4df7700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bd0107990 0x7f3bd019f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:28.733+0000 7f3bd4df7700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bd0102730 0x7f3bd0197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:28.733+0000 7f3bd4df7700 1 --2- 172.21.15.88:0/2207627389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bd0105060 0x7f3bd0198240 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:28.733+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2207627389 >> 172.21.15.88:0/2207627389 conn(0x7f3bd00fc140 msgr2=0x7f3bd0103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:51:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:28.733+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2207627389 shutdown_connections 2024-03-12T05:51:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:51:28.733+0000 7f3bd4df7700 1 -- 172.21.15.88:0/2207627389 wait complete. 2024-03-12T05:51:29.312 DEBUG:teuthology.orchestra.run.smithi088:osd.8> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.8.service 2024-03-12T05:51:29.314 INFO:tasks.cephadm:Deploying osd.9 on smithi076 with /dev/vg_nvme/lv_1... 2024-03-12T05:51:29.315 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-12T05:51:29.569 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:51:29.943 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:29 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:29.943 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:29 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:29.943 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:29 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:29.943 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:29 smithi076 ceph-mon[32095]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:29.943 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:29 smithi076 ceph-mon[32095]: from='osd.8 [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-12T05:51:29.943 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:29 smithi076 ceph-mon[32095]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-12T05:51:29.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:29 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:29.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:29 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:29.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:29 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:29.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:29 smithi088 ceph-mon[36585]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:29.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:29 smithi088 ceph-mon[36585]: from='osd.8 [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-12T05:51:29.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:29 smithi088 ceph-mon[36585]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-12T05:51:30.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:29 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:30.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:29 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:30.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:29 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:30.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:29 smithi082 ceph-mon[36382]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:30.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:29 smithi082 ceph-mon[36382]: from='osd.8 [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-12T05:51:30.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:29 smithi082 ceph-mon[36382]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-12T05:51:30.927 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:30 smithi076 ceph-mon[32095]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-12T05:51:30.927 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:30 smithi076 ceph-mon[32095]: from='osd.8 [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:51:30.927 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:30 smithi076 ceph-mon[32095]: osdmap e57: 9 total, 8 up, 9 in 2024-03-12T05:51:30.927 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:30 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:30.927 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:30 smithi076 ceph-mon[32095]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:51:30.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:30 smithi088 ceph-mon[36585]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-12T05:51:30.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:30 smithi088 ceph-mon[36585]: from='osd.8 [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:51:30.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:30 smithi088 ceph-mon[36585]: osdmap e57: 9 total, 8 up, 9 in 2024-03-12T05:51:30.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:30 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:30.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:30 smithi088 ceph-mon[36585]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:51:31.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:30 smithi082 ceph-mon[36382]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-12T05:51:31.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:30 smithi082 ceph-mon[36382]: from='osd.8 [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:51:31.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:30 smithi082 ceph-mon[36382]: osdmap e57: 9 total, 8 up, 9 in 2024-03-12T05:51:31.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:30 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:31.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:30 smithi082 ceph-mon[36382]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:51:31.598 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-12T05:51:31.598 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-12T05:51:31.598 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-03-12T05:51:31.598 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-03-12T05:51:31.598 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114096 s, 919 MB/s 2024-03-12T05:51:31.598 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-03-12T05:51:31.890 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:31 smithi088 ceph-mon[36585]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-12T05:51:31.890 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:31 smithi088 ceph-mon[36585]: osdmap e58: 9 total, 8 up, 9 in 2024-03-12T05:51:31.891 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:31.891 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:31.891 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:31 smithi088 ceph-mon[36585]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:31.891 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:31 smithi088 ceph-mon[36585]: osdmap e59: 9 total, 8 up, 9 in 2024-03-12T05:51:31.891 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:31 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:31.891 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:31 smithi088 ceph-mon[36585]: from='osd.8 ' entity='osd.8' 2024-03-12T05:51:31.891 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 05:51:31 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T05:51:31.535+0000 7f55288da700 -1 osd.8 0 waiting for initial osdmap 2024-03-12T05:51:31.891 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 05:51:31 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T05:51:31.554+0000 7f5520ec6700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:51:31.892 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:31 smithi076 ceph-mon[32095]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-12T05:51:31.892 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:31 smithi076 ceph-mon[32095]: osdmap e58: 9 total, 8 up, 9 in 2024-03-12T05:51:31.892 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:31.892 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:31.892 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:31 smithi076 ceph-mon[32095]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:31.892 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:31 smithi076 ceph-mon[32095]: osdmap e59: 9 total, 8 up, 9 in 2024-03-12T05:51:31.892 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:31 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:31.892 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:31 smithi076 ceph-mon[32095]: from='osd.8 ' entity='osd.8' 2024-03-12T05:51:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:31 smithi082 ceph-mon[36382]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-12T05:51:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:31 smithi082 ceph-mon[36382]: osdmap e58: 9 total, 8 up, 9 in 2024-03-12T05:51:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:31 smithi082 ceph-mon[36382]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-12T05:51:32.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:31 smithi082 ceph-mon[36382]: osdmap e59: 9 total, 8 up, 9 in 2024-03-12T05:51:32.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:31 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:32.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:31 smithi082 ceph-mon[36382]: from='osd.8 ' entity='osd.8' 2024-03-12T05:51:32.306 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi076:vg_nvme/lv_1 2024-03-12T05:51:32.553 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:51:32.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: purged_snaps scrub starts 2024-03-12T05:51:32.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: purged_snaps scrub ok 2024-03-12T05:51:32.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:32.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:32.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:32.946 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:32.946 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:32.946 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:32.946 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:32.946 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:32.946 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:32.946 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:32.946 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: osd.8 [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247] boot 2024-03-12T05:51:32.946 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: osdmap e60: 9 total, 9 up, 9 in 2024-03-12T05:51:32.946 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:32 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:32.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: purged_snaps scrub starts 2024-03-12T05:51:32.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: purged_snaps scrub ok 2024-03-12T05:51:32.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:32.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:32.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:32.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:32.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:32.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:32.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:32.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:32.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:32.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:32.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: osd.8 [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247] boot 2024-03-12T05:51:32.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: osdmap e60: 9 total, 9 up, 9 in 2024-03-12T05:51:32.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:32 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:33.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: purged_snaps scrub starts 2024-03-12T05:51:33.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: purged_snaps scrub ok 2024-03-12T05:51:33.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:33.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:33.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:33.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:33.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:33.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:33.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:33.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:33.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:33.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:33.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: osd.8 [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247] boot 2024-03-12T05:51:33.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: osdmap e60: 9 total, 9 up, 9 in 2024-03-12T05:51:33.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:32 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:51:33.977 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:33 smithi076 ceph-mon[32095]: Detected new or changed devices on smithi088 2024-03-12T05:51:33.977 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:33 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi088 to 5693M 2024-03-12T05:51:33.977 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:33 smithi076 ceph-mon[32095]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:33.977 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:33 smithi076 ceph-mon[32095]: osdmap e61: 9 total, 9 up, 9 in 2024-03-12T05:51:33.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:33 smithi088 ceph-mon[36585]: Detected new or changed devices on smithi088 2024-03-12T05:51:33.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:33 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi088 to 5693M 2024-03-12T05:51:33.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:33 smithi088 ceph-mon[36585]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:33.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:33 smithi088 ceph-mon[36585]: osdmap e61: 9 total, 9 up, 9 in 2024-03-12T05:51:34.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:33 smithi082 ceph-mon[36382]: Detected new or changed devices on smithi088 2024-03-12T05:51:34.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:33 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi088 to 5693M 2024-03-12T05:51:34.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:33 smithi082 ceph-mon[36382]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:34.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:33 smithi082 ceph-mon[36382]: osdmap e61: 9 total, 9 up, 9 in 2024-03-12T05:51:34.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.195+0000 7fb40303d700 1 -- 172.21.15.76:0/3572892477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc107aa0 msgr2=0x7fb3fc109e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:51:34.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.195+0000 7fb40303d700 1 --2- 172.21.15.76:0/3572892477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc107aa0 0x7fb3fc109e90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400b0a0 tx=0x7fb3f400b3b0 comp rx=0 tx=0).stop 2024-03-12T05:51:34.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.195+0000 7fb40303d700 1 -- 172.21.15.76:0/3572892477 shutdown_connections 2024-03-12T05:51:34.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.195+0000 7fb40303d700 1 --2- 172.21.15.76:0/3572892477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc107aa0 0x7fb3fc109e90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:34.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.195+0000 7fb40303d700 1 --2- 172.21.15.76:0/3572892477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3fc102840 0x7fb3fc104c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:34.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.195+0000 7fb40303d700 1 --2- 172.21.15.76:0/3572892477 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3fc105170 0x7fb3fc107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:34.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.195+0000 7fb40303d700 1 -- 172.21.15.76:0/3572892477 >> 172.21.15.76:0/3572892477 conn(0x7fb3fc0fc1d0 msgr2=0x7fb3fc0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:51:34.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.196+0000 7fb40303d700 1 -- 172.21.15.76:0/3572892477 shutdown_connections 2024-03-12T05:51:34.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.196+0000 7fb40303d700 1 -- 172.21.15.76:0/3572892477 wait complete. 2024-03-12T05:51:34.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.196+0000 7fb40303d700 1 Processor -- start 2024-03-12T05:51:34.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.196+0000 7fb40303d700 1 -- start start 2024-03-12T05:51:34.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.196+0000 7fb40303d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc102840 0x7fb3fc197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:51:34.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.196+0000 7fb40303d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3fc105170 0x7fb3fc1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:51:34.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.196+0000 7fb40303d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3fc107aa0 0x7fb3fc19f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:51:34.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.196+0000 7fb40303d700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc19f810 con 0x7fb3fc102840 2024-03-12T05:51:34.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.196+0000 7fb40303d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc19f950 con 0x7fb3fc105170 2024-03-12T05:51:34.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.196+0000 7fb40303d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc19fb10 con 0x7fb3fc107aa0 2024-03-12T05:51:34.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.196+0000 7fb400dd9700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc102840 0x7fb3fc197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:51:34.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.196+0000 7fb400dd9700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc102840 0x7fb3fc197c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49758/0 (socket says 172.21.15.76:49758) 2024-03-12T05:51:34.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.196+0000 7fb400dd9700 1 -- 172.21.15.76:0/3737877190 learned_addr learned my addr 172.21.15.76:0/3737877190 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:51:34.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.197+0000 7fb4015da700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3fc107aa0 0x7fb3fc19f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:51:34.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.197+0000 7fb3fbfff700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3fc105170 0x7fb3fc1981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:51:34.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.197+0000 7fb400dd9700 1 -- 172.21.15.76:0/3737877190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3fc107aa0 msgr2=0x7fb3fc19f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:51:34.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.197+0000 7fb400dd9700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3fc107aa0 0x7fb3fc19f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:34.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.197+0000 7fb400dd9700 1 -- 172.21.15.76:0/3737877190 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3fc105170 msgr2=0x7fb3fc1981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:51:34.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.197+0000 7fb400dd9700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3fc105170 0x7fb3fc1981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:34.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.197+0000 7fb400dd9700 1 -- 172.21.15.76:0/3737877190 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f4009d00 con 0x7fb3fc102840 2024-03-12T05:51:34.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.197+0000 7fb3fbfff700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3fc105170 0x7fb3fc1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:51:34.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.197+0000 7fb400dd9700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc102840 0x7fb3fc197c90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000c9b0 tx=0x7fb3f000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:51:34.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.197+0000 7fb3f9ffb700 1 -- 172.21.15.76:0/3737877190 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0007b00 con 0x7fb3fc102840 2024-03-12T05:51:34.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.197+0000 7fb40303d700 1 -- 172.21.15.76:0/3737877190 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc100690 con 0x7fb3fc102840 2024-03-12T05:51:34.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.197+0000 7fb3f9ffb700 1 -- 172.21.15.76:0/3737877190 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3f0007c60 con 0x7fb3fc102840 2024-03-12T05:51:34.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.197+0000 7fb3f9ffb700 1 -- 172.21.15.76:0/3737877190 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0018660 con 0x7fb3fc102840 2024-03-12T05:51:34.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.197+0000 7fb40303d700 1 -- 172.21.15.76:0/3737877190 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc100b90 con 0x7fb3fc102840 2024-03-12T05:51:34.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.198+0000 7fb40303d700 1 -- 172.21.15.76:0/3737877190 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc0fdd50 con 0x7fb3fc102840 2024-03-12T05:51:34.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.199+0000 7fb3f9ffb700 1 -- 172.21.15.76:0/3737877190 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb3f000f450 con 0x7fb3fc102840 2024-03-12T05:51:34.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.200+0000 7fb3f9ffb700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb3ec0a1ac0 0x7fb3ec0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:51:34.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.200+0000 7fb3fbfff700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb3ec0a1ac0 0x7fb3ec0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:51:34.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.200+0000 7fb3f9ffb700 1 -- 172.21.15.76:0/3737877190 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5858+0+0 (secure 0 0 0) 0x7fb3f00cca40 con 0x7fb3fc102840 2024-03-12T05:51:34.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.200+0000 7fb3fbfff700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb3ec0a1ac0 0x7fb3ec0a3f80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8005950 tx=0x7fb3e80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:51:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.205+0000 7fb3f9ffb700 1 -- 172.21.15.76:0/3737877190 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb3f00cee60 con 0x7fb3fc102840 2024-03-12T05:51:34.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:34.413+0000 7fb40303d700 1 -- 172.21.15.76:0/3737877190 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3fc02d870 con 0x7fb3ec0a1ac0 2024-03-12T05:51:35.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:35 smithi082 ceph-mon[36382]: osdmap e62: 9 total, 9 up, 9 in 2024-03-12T05:51:35.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:35 smithi082 ceph-mon[36382]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:51:35.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:35 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:51:35.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:35 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:51:35.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:35 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:35.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:35 smithi082 ceph-mon[36382]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:35 smithi088 ceph-mon[36585]: osdmap e62: 9 total, 9 up, 9 in 2024-03-12T05:51:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:35 smithi088 ceph-mon[36585]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:51:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:35 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:51:35.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:35 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:51:35.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:35 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:35.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:35 smithi088 ceph-mon[36585]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:35.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:35 smithi076 ceph-mon[32095]: osdmap e62: 9 total, 9 up, 9 in 2024-03-12T05:51:35.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:35 smithi076 ceph-mon[32095]: from='client.14439 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:51:35.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:35 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:51:35.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:35 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:51:35.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:35 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:35.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:35 smithi076 ceph-mon[32095]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:36.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:36 smithi076 ceph-mon[32095]: osdmap e63: 9 total, 9 up, 9 in 2024-03-12T05:51:36.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:36 smithi082 ceph-mon[36382]: osdmap e63: 9 total, 9 up, 9 in 2024-03-12T05:51:36.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:36 smithi088 ceph-mon[36585]: osdmap e63: 9 total, 9 up, 9 in 2024-03-12T05:51:37.637 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:37 smithi076 ceph-mon[32095]: osdmap e64: 9 total, 9 up, 9 in 2024-03-12T05:51:37.637 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:37 smithi076 ceph-mon[32095]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:37.637 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:37 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3306353790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bcb44da2-47f9-4ff3-bd96-f026626faea2"}]: dispatch 2024-03-12T05:51:37.637 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:37 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3306353790' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bcb44da2-47f9-4ff3-bd96-f026626faea2"}]': finished 2024-03-12T05:51:37.637 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:37 smithi076 ceph-mon[32095]: osdmap e65: 10 total, 9 up, 10 in 2024-03-12T05:51:37.637 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:37 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:37.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:37 smithi082 ceph-mon[36382]: osdmap e64: 9 total, 9 up, 9 in 2024-03-12T05:51:37.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:37 smithi082 ceph-mon[36382]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:37.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:37 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3306353790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bcb44da2-47f9-4ff3-bd96-f026626faea2"}]: dispatch 2024-03-12T05:51:37.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:37 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3306353790' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bcb44da2-47f9-4ff3-bd96-f026626faea2"}]': finished 2024-03-12T05:51:37.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:37 smithi082 ceph-mon[36382]: osdmap e65: 10 total, 9 up, 10 in 2024-03-12T05:51:37.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:37 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:37.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:37 smithi088 ceph-mon[36585]: osdmap e64: 9 total, 9 up, 9 in 2024-03-12T05:51:37.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:37 smithi088 ceph-mon[36585]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:37.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:37 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3306353790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bcb44da2-47f9-4ff3-bd96-f026626faea2"}]: dispatch 2024-03-12T05:51:37.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:37 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3306353790' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bcb44da2-47f9-4ff3-bd96-f026626faea2"}]': finished 2024-03-12T05:51:37.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:37 smithi088 ceph-mon[36585]: osdmap e65: 10 total, 9 up, 10 in 2024-03-12T05:51:37.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:37 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:38.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:38 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1728498003' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:51:38.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:38 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1728498003' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:51:38.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:38 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1728498003' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:51:39.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:39 smithi082 ceph-mon[36382]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-12T05:51:39.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:39 smithi088 ceph-mon[36585]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-12T05:51:39.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:39 smithi076 ceph-mon[32095]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-12T05:51:42.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:41 smithi082 ceph-mon[36382]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T05:51:42.297 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:41 smithi076 ceph-mon[32095]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T05:51:42.297 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:41 smithi088 ceph-mon[36585]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-12T05:51:43.968 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:43 smithi076 ceph-mon[32095]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-12T05:51:44.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:43 smithi082 ceph-mon[36382]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-12T05:51:44.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:43 smithi088 ceph-mon[36585]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-12T05:51:46.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:45 smithi076 ceph-mon[32095]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-12T05:51:46.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:45 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-12T05:51:46.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:45 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:46.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:45 smithi082 ceph-mon[36382]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-12T05:51:46.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:45 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-12T05:51:46.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:45 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:46.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:45 smithi088 ceph-mon[36585]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-12T05:51:46.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:45 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-12T05:51:46.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:45 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:47.106 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:46 smithi076 ceph-mon[32095]: Deploying daemon osd.9 on smithi076 2024-03-12T05:51:47.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:46 smithi082 ceph-mon[36382]: Deploying daemon osd.9 on smithi076 2024-03-12T05:51:47.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:46 smithi088 ceph-mon[36585]: Deploying daemon osd.9 on smithi076 2024-03-12T05:51:48.129 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:47 smithi076 ceph-mon[32095]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T05:51:48.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:47 smithi082 ceph-mon[36382]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T05:51:48.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:47 smithi088 ceph-mon[36585]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-12T05:51:50.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:49 smithi082 ceph-mon[36382]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-12T05:51:50.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:49 smithi088 ceph-mon[36585]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-12T05:51:50.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:49 smithi076 ceph-mon[32095]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-12T05:51:52.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:51 smithi082 ceph-mon[36382]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:52.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:51 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:52.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:51 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:52.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:51 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:52.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:51 smithi088 ceph-mon[36585]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:52.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:51 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:52.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:51 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:52.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:51 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:52.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:51 smithi076 ceph-mon[32095]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:52.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:51 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:52.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:51 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:52.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:51 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:54.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:53 smithi082 ceph-mon[36382]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:54.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:53 smithi088 ceph-mon[36585]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:54.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:53 smithi076 ceph-mon[32095]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:54.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:54.968+0000 7fb3f9ffb700 1 -- 172.21.15.76:0/3737877190 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb3fc02d870 con 0x7fb3ec0a1ac0 2024-03-12T05:51:54.972 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 9 on host 'smithi076' 2024-03-12T05:51:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:54.971+0000 7fb40303d700 1 -- 172.21.15.76:0/3737877190 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb3ec0a1ac0 msgr2=0x7fb3ec0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:51:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:54.971+0000 7fb40303d700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb3ec0a1ac0 0x7fb3ec0a3f80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8005950 tx=0x7fb3e80058e0 comp rx=0 tx=0).stop 2024-03-12T05:51:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:54.971+0000 7fb40303d700 1 -- 172.21.15.76:0/3737877190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc102840 msgr2=0x7fb3fc197c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:51:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:54.971+0000 7fb40303d700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc102840 0x7fb3fc197c90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000c9b0 tx=0x7fb3f000ccc0 comp rx=0 tx=0).stop 2024-03-12T05:51:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:54.971+0000 7fb40303d700 1 -- 172.21.15.76:0/3737877190 shutdown_connections 2024-03-12T05:51:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:54.971+0000 7fb40303d700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb3ec0a1ac0 0x7fb3ec0a3f80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:54.971+0000 7fb40303d700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3fc107aa0 0x7fb3fc19f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:54.971+0000 7fb40303d700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3fc102840 0x7fb3fc197c90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:54.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:54.971+0000 7fb40303d700 1 --2- 172.21.15.76:0/3737877190 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3fc105170 0x7fb3fc1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:51:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:54.971+0000 7fb40303d700 1 -- 172.21.15.76:0/3737877190 >> 172.21.15.76:0/3737877190 conn(0x7fb3fc0fc1d0 msgr2=0x7fb3fc1031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:51:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:54.971+0000 7fb40303d700 1 -- 172.21.15.76:0/3737877190 shutdown_connections 2024-03-12T05:51:54.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:51:54.971+0000 7fb40303d700 1 -- 172.21.15.76:0/3737877190 wait complete. 2024-03-12T05:51:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:55 smithi088 ceph-mon[36585]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:55 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:55.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:55 smithi088 ceph-mon[36585]: from='osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-12T05:51:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:55 smithi076 ceph-mon[32095]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:55 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:55.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:55 smithi076 ceph-mon[32095]: from='osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-12T05:51:55.619 DEBUG:teuthology.orchestra.run.smithi076:osd.9> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.9.service 2024-03-12T05:51:55.621 INFO:tasks.cephadm:Deploying osd.10 on smithi082 with /dev/vg_nvme/lv_1... 2024-03-12T05:51:55.621 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-12T05:51:55.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:55.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:55.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:55 smithi082 ceph-mon[36382]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:55.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:55.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:55 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:51:55.652 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:55 smithi082 ceph-mon[36382]: from='osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-12T05:51:55.867 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:51:57.070 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:56 smithi076 ceph-mon[32095]: from='osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-12T05:51:57.070 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:56 smithi076 ceph-mon[32095]: osdmap e66: 10 total, 9 up, 10 in 2024-03-12T05:51:57.071 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:56 smithi076 ceph-mon[32095]: from='osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-12T05:51:57.071 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:56 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:57.071 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 05:51:56 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T05:51:56.976+0000 7f27cf026700 -1 osd.9 0 waiting for initial osdmap 2024-03-12T05:51:57.071 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 05:51:56 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T05:51:56.991+0000 7f27cb61a700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:51:57.158 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:56 smithi082 ceph-mon[36382]: from='osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-12T05:51:57.158 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:56 smithi082 ceph-mon[36382]: osdmap e66: 10 total, 9 up, 10 in 2024-03-12T05:51:57.158 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:56 smithi082 ceph-mon[36382]: from='osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-12T05:51:57.158 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:56 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:56 smithi088 ceph-mon[36585]: from='osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-12T05:51:57.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:56 smithi088 ceph-mon[36585]: osdmap e66: 10 total, 9 up, 10 in 2024-03-12T05:51:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:56 smithi088 ceph-mon[36585]: from='osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-03-12T05:51:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:56 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:57.723 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-12T05:51:57.724 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-12T05:51:57.724 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-03-12T05:51:57.724 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-03-12T05:51:57.724 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110969 s, 945 MB/s 2024-03-12T05:51:57.724 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-03-12T05:51:58.210 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:57 smithi076 ceph-mon[32095]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:58.210 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:57 smithi076 ceph-mon[32095]: from='osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-12T05:51:58.210 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:57 smithi076 ceph-mon[32095]: osdmap e67: 10 total, 9 up, 10 in 2024-03-12T05:51:58.211 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:58.211 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:57 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:58.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:57 smithi082 ceph-mon[36382]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:58.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:57 smithi082 ceph-mon[36382]: from='osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-12T05:51:58.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:57 smithi082 ceph-mon[36382]: osdmap e67: 10 total, 9 up, 10 in 2024-03-12T05:51:58.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:57 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:58.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:57 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:57 smithi088 ceph-mon[36585]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:51:58.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:57 smithi088 ceph-mon[36585]: from='osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-03-12T05:51:58.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:57 smithi088 ceph-mon[36585]: osdmap e67: 10 total, 9 up, 10 in 2024-03-12T05:51:58.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:57 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:58.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:57 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:58.508 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi082:vg_nvme/lv_1 2024-03-12T05:51:58.805 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T05:51:59.068 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: purged_snaps scrub starts 2024-03-12T05:51:59.068 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: purged_snaps scrub ok 2024-03-12T05:51:59.068 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191] boot 2024-03-12T05:51:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: osdmap e68: 10 total, 10 up, 10 in 2024-03-12T05:51:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:59.069 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:58 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: purged_snaps scrub starts 2024-03-12T05:51:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: purged_snaps scrub ok 2024-03-12T05:51:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191] boot 2024-03-12T05:51:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: osdmap e68: 10 total, 10 up, 10 in 2024-03-12T05:51:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:59.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:59.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:59.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:59.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:59.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:59.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:59.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:51:58 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:59.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: purged_snaps scrub starts 2024-03-12T05:51:59.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: purged_snaps scrub ok 2024-03-12T05:51:59.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:59.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: osd.9 [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191] boot 2024-03-12T05:51:59.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: osdmap e68: 10 total, 10 up, 10 in 2024-03-12T05:51:59.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:51:59.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:59.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:59.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:59.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:59.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:59.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:51:59.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:51:59.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:51:59.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:51:59.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:58 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:00.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:59 smithi082 ceph-mon[36382]: Detected new or changed devices on smithi076 2024-03-12T05:52:00.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:59 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi076 to 4272M 2024-03-12T05:52:00.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:59 smithi082 ceph-mon[36382]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:52:00.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:51:59 smithi082 ceph-mon[36382]: osdmap e69: 10 total, 10 up, 10 in 2024-03-12T05:52:00.447 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.439+0000 7f9edbce9700 1 -- 172.21.15.82:0/3963477125 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ed4105060 msgr2=0x7f9ed4107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:00.447 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.439+0000 7f9edbce9700 1 --2- 172.21.15.82:0/3963477125 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ed4105060 0x7f9ed4107450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009990 tx=0x7f9ec8009ca0 comp rx=0 tx=0).stop 2024-03-12T05:52:00.447 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.439+0000 7f9edbce9700 1 -- 172.21.15.82:0/3963477125 shutdown_connections 2024-03-12T05:52:00.447 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.439+0000 7f9edbce9700 1 --2- 172.21.15.82:0/3963477125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed4107990 0x7f9ed4109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:00.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.439+0000 7f9edbce9700 1 --2- 172.21.15.82:0/3963477125 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ed4102730 0x7f9ed4104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:00.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.439+0000 7f9edbce9700 1 --2- 172.21.15.82:0/3963477125 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ed4105060 0x7f9ed4107450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:00.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.439+0000 7f9edbce9700 1 -- 172.21.15.82:0/3963477125 >> 172.21.15.82:0/3963477125 conn(0x7f9ed40fc140 msgr2=0x7f9ed40fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:52:00.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.439+0000 7f9edbce9700 1 -- 172.21.15.82:0/3963477125 shutdown_connections 2024-03-12T05:52:00.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.439+0000 7f9edbce9700 1 -- 172.21.15.82:0/3963477125 wait complete. 2024-03-12T05:52:00.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9edbce9700 1 Processor -- start 2024-03-12T05:52:00.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9edbce9700 1 -- start start 2024-03-12T05:52:00.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9edbce9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed4102730 0x7f9ed4197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:00.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9edbce9700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ed4105060 0x7f9ed4198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:00.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9edbce9700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ed4107990 0x7f9ed419f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:00.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9edbce9700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed419f770 con 0x7f9ed4105060 2024-03-12T05:52:00.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9edbce9700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed419f8b0 con 0x7f9ed4107990 2024-03-12T05:52:00.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9edbce9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed419fa70 con 0x7f9ed4102730 2024-03-12T05:52:00.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9eda286700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ed4107990 0x7f9ed419f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9eda286700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ed4107990 0x7f9ed419f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:45832/0 (socket says 172.21.15.82:45832) 2024-03-12T05:52:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9eda286700 1 -- 172.21.15.82:0/55507509 learned_addr learned my addr 172.21.15.82:0/55507509 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-03-12T05:52:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9ed9a85700 1 --2- 172.21.15.82:0/55507509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed4102730 0x7f9ed4197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9eda286700 1 -- 172.21.15.82:0/55507509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed4102730 msgr2=0x7f9ed4197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9eda286700 1 --2- 172.21.15.82:0/55507509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed4102730 0x7f9ed4197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9eda286700 1 -- 172.21.15.82:0/55507509 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ed4105060 msgr2=0x7f9ed4198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9eda286700 1 --2- 172.21.15.82:0/55507509 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ed4105060 0x7f9ed4198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9eda286700 1 -- 172.21.15.82:0/55507509 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec80095f0 con 0x7f9ed4107990 2024-03-12T05:52:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.440+0000 7f9ed9a85700 1 --2- 172.21.15.82:0/55507509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed4102730 0x7f9ed4197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:52:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.441+0000 7f9eda286700 1 --2- 172.21.15.82:0/55507509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ed4107990 0x7f9ed419f1a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc0094f0 tx=0x7f9ecc007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:52:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.441+0000 7f9ed2ffd700 1 -- 172.21.15.82:0/55507509 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc004db0 con 0x7f9ed4107990 2024-03-12T05:52:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.441+0000 7f9ed2ffd700 1 -- 172.21.15.82:0/55507509 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ecc00f040 con 0x7f9ed4107990 2024-03-12T05:52:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.441+0000 7f9edbce9700 1 -- 172.21.15.82:0/55507509 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed419fd50 con 0x7f9ed4107990 2024-03-12T05:52:00.449 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.441+0000 7f9ed2ffd700 1 -- 172.21.15.82:0/55507509 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc005aa0 con 0x7f9ed4107990 2024-03-12T05:52:00.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.441+0000 7f9edbce9700 1 -- 172.21.15.82:0/55507509 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed41a01a0 con 0x7f9ed4107990 2024-03-12T05:52:00.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.442+0000 7f9edbce9700 1 -- 172.21.15.82:0/55507509 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed40fdcc0 con 0x7f9ed4107990 2024-03-12T05:52:00.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.443+0000 7f9ed2ffd700 1 -- 172.21.15.82:0/55507509 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9ecc00dbf0 con 0x7f9ed4107990 2024-03-12T05:52:00.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.443+0000 7f9ed2ffd700 1 --2- 172.21.15.82:0/55507509 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9ebc0a1ac0 0x7f9ebc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:00.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.443+0000 7f9ed2ffd700 1 -- 172.21.15.82:0/55507509 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f9ecc0cce30 con 0x7f9ed4107990 2024-03-12T05:52:00.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.443+0000 7f9ed9a85700 1 --2- 172.21.15.82:0/55507509 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9ebc0a1ac0 0x7f9ebc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:00.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.444+0000 7f9ed9a85700 1 --2- 172.21.15.82:0/55507509 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9ebc0a1ac0 0x7f9ebc0a3f80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4005fd0 tx=0x7f9ec4005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:52:00.450 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.447+0000 7f9ed2ffd700 1 -- 172.21.15.82:0/55507509 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9ecc096840 con 0x7f9ed4107990 2024-03-12T05:52:00.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:00 smithi088 ceph-mon[36585]: Detected new or changed devices on smithi076 2024-03-12T05:52:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:00 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi076 to 4272M 2024-03-12T05:52:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:00 smithi088 ceph-mon[36585]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:52:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:00 smithi088 ceph-mon[36585]: osdmap e69: 10 total, 10 up, 10 in 2024-03-12T05:52:00.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:59 smithi076 ceph-mon[32095]: Detected new or changed devices on smithi076 2024-03-12T05:52:00.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:59 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi076 to 4272M 2024-03-12T05:52:00.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:59 smithi076 ceph-mon[32095]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-03-12T05:52:00.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:51:59 smithi076 ceph-mon[32095]: osdmap e69: 10 total, 10 up, 10 in 2024-03-12T05:52:00.650 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:00.647+0000 7f9edbce9700 1 -- 172.21.15.82:0/55507509 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ed402da10 con 0x7f9ebc0a1ac0 2024-03-12T05:52:01.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:01 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:52:01.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:01 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:52:01.281 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:01 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:01 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:52:01.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:01 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:52:01.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:01 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:01.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:01 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:52:01.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:01 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:52:01.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:01 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:02.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:02 smithi082 ceph-mon[36382]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:52:02.284 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:02 smithi082 ceph-mon[36382]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:02.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:02 smithi088 ceph-mon[36585]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:52:02.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:02 smithi088 ceph-mon[36585]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:02.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:02 smithi076 ceph-mon[32095]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:52:02.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:02 smithi076 ceph-mon[32095]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:04.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:04 smithi082 ceph-mon[36382]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:04.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:04 smithi082 ceph-mon[36382]: from='client.? 172.21.15.82:0/1697383542' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c565e2eb-11bf-4889-b19e-11473f9ed6eb"}]: dispatch 2024-03-12T05:52:04.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:04 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c565e2eb-11bf-4889-b19e-11473f9ed6eb"}]: dispatch 2024-03-12T05:52:04.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:04 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c565e2eb-11bf-4889-b19e-11473f9ed6eb"}]': finished 2024-03-12T05:52:04.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:04 smithi082 ceph-mon[36382]: osdmap e70: 11 total, 10 up, 11 in 2024-03-12T05:52:04.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:04 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:04.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:04 smithi088 ceph-mon[36585]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:04.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:04 smithi088 ceph-mon[36585]: from='client.? 172.21.15.82:0/1697383542' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c565e2eb-11bf-4889-b19e-11473f9ed6eb"}]: dispatch 2024-03-12T05:52:04.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:04 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c565e2eb-11bf-4889-b19e-11473f9ed6eb"}]: dispatch 2024-03-12T05:52:04.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:04 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c565e2eb-11bf-4889-b19e-11473f9ed6eb"}]': finished 2024-03-12T05:52:04.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:04 smithi088 ceph-mon[36585]: osdmap e70: 11 total, 10 up, 11 in 2024-03-12T05:52:04.322 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:04 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:04.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:04 smithi076 ceph-mon[32095]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:04.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:04 smithi076 ceph-mon[32095]: from='client.? 172.21.15.82:0/1697383542' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c565e2eb-11bf-4889-b19e-11473f9ed6eb"}]: dispatch 2024-03-12T05:52:04.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:04 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c565e2eb-11bf-4889-b19e-11473f9ed6eb"}]: dispatch 2024-03-12T05:52:04.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:04 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c565e2eb-11bf-4889-b19e-11473f9ed6eb"}]': finished 2024-03-12T05:52:04.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:04 smithi076 ceph-mon[32095]: osdmap e70: 11 total, 10 up, 11 in 2024-03-12T05:52:04.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:04 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:05.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:05 smithi082 ceph-mon[36382]: from='client.? 172.21.15.82:0/2838590342' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:52:05.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:05 smithi088 ceph-mon[36585]: from='client.? 172.21.15.82:0/2838590342' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:52:05.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:05 smithi076 ceph-mon[32095]: from='client.? 172.21.15.82:0/2838590342' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:52:06.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:06 smithi082 ceph-mon[36382]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:06.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:06 smithi088 ceph-mon[36585]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:06.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:06 smithi076 ceph-mon[32095]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:08.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:08 smithi082 ceph-mon[36382]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:08.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:08 smithi088 ceph-mon[36585]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:08.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:08 smithi076 ceph-mon[32095]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:10.425 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:10 smithi082 ceph-mon[36382]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:10.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:10 smithi088 ceph-mon[36585]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:10.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:10 smithi076 ceph-mon[32095]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:12.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:12 smithi082 ceph-mon[36382]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:12.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:12 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-12T05:52:12.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:12 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:12.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:12 smithi076 ceph-mon[32095]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:12.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:12 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-12T05:52:12.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:12 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:12.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:12 smithi088 ceph-mon[36585]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:12.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:12 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-12T05:52:12.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:12 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:13.305 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:13 smithi082 ceph-mon[36382]: Deploying daemon osd.10 on smithi082 2024-03-12T05:52:13.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:13 smithi088 ceph-mon[36585]: Deploying daemon osd.10 on smithi082 2024-03-12T05:52:13.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:13 smithi076 ceph-mon[32095]: Deploying daemon osd.10 on smithi082 2024-03-12T05:52:14.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:14 smithi082 ceph-mon[36382]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:14.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:14 smithi088 ceph-mon[36585]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:14.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:14 smithi076 ceph-mon[32095]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:16.297 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:16 smithi082 ceph-mon[36382]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:16.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:16 smithi088 ceph-mon[36585]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:16.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:16 smithi076 ceph-mon[32095]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:17.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:17 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:17.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:17 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:17.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:17 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:52:17.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:17 smithi082 ceph-mon[36382]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:17.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:17 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:17.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:17 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:17.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:17 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:52:17.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:17 smithi088 ceph-mon[36585]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:17.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:17 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:17.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:17 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:17.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:17 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:52:17.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:17 smithi076 ceph-mon[32095]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:20.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:19 smithi082 ceph-mon[36382]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:20.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:20.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:20.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:20.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:52:20.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:20.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:19 smithi088 ceph-mon[36585]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:20.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:20.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:52:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:20.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:19 smithi076 ceph-mon[32095]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:20.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:20.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:20.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:20.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:52:20.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:20.266 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 10 on host 'smithi082' 2024-03-12T05:52:20.266 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:20.262+0000 7f9ed2ffd700 1 -- 172.21.15.82:0/55507509 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f9ed402da10 con 0x7f9ebc0a1ac0 2024-03-12T05:52:20.267 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:20.264+0000 7f9edbce9700 1 -- 172.21.15.82:0/55507509 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9ebc0a1ac0 msgr2=0x7f9ebc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:20.267 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:20.264+0000 7f9edbce9700 1 --2- 172.21.15.82:0/55507509 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9ebc0a1ac0 0x7f9ebc0a3f80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4005fd0 tx=0x7f9ec4005e40 comp rx=0 tx=0).stop 2024-03-12T05:52:20.267 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:20.264+0000 7f9edbce9700 1 -- 172.21.15.82:0/55507509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ed4107990 msgr2=0x7f9ed419f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:20.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:20.264+0000 7f9edbce9700 1 --2- 172.21.15.82:0/55507509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ed4107990 0x7f9ed419f1a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc0094f0 tx=0x7f9ecc007f90 comp rx=0 tx=0).stop 2024-03-12T05:52:20.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:20.264+0000 7f9edbce9700 1 -- 172.21.15.82:0/55507509 shutdown_connections 2024-03-12T05:52:20.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:20.264+0000 7f9edbce9700 1 --2- 172.21.15.82:0/55507509 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9ebc0a1ac0 0x7f9ebc0a3f80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:20.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:20.264+0000 7f9edbce9700 1 --2- 172.21.15.82:0/55507509 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ed4107990 0x7f9ed419f1a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:20.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:20.264+0000 7f9edbce9700 1 --2- 172.21.15.82:0/55507509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed4102730 0x7f9ed4197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:20.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:20.264+0000 7f9edbce9700 1 --2- 172.21.15.82:0/55507509 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ed4105060 0x7f9ed4198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:20.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:20.264+0000 7f9edbce9700 1 -- 172.21.15.82:0/55507509 >> 172.21.15.82:0/55507509 conn(0x7f9ed40fc140 msgr2=0x7f9ed4103170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:52:20.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:20.264+0000 7f9edbce9700 1 -- 172.21.15.82:0/55507509 shutdown_connections 2024-03-12T05:52:20.268 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-12T05:52:20.264+0000 7f9edbce9700 1 -- 172.21.15.82:0/55507509 wait complete. 2024-03-12T05:52:20.837 DEBUG:teuthology.orchestra.run.smithi082:osd.10> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.10.service 2024-03-12T05:52:20.839 INFO:tasks.cephadm:Deploying osd.11 on smithi088 with /dev/vg_nvme/lv_1... 2024-03-12T05:52:20.840 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-12T05:52:21.088 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:52:21.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:21 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:21.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:21 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:21.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:21 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:52:21.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:21 smithi082 ceph-mon[36382]: from='osd.10 [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-12T05:52:21.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:21 smithi082 ceph-mon[36382]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-12T05:52:21.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:21 smithi082 ceph-mon[36382]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:21.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:21 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:21.460 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:21 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:21.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:21 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:52:21.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:21 smithi088 ceph-mon[36585]: from='osd.10 [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-12T05:52:21.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:21 smithi088 ceph-mon[36585]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-12T05:52:21.461 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:21 smithi088 ceph-mon[36585]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:21.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:21.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:21.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:52:21.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:21 smithi076 ceph-mon[32095]: from='osd.10 [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-12T05:52:21.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:21 smithi076 ceph-mon[32095]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-12T05:52:21.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:21 smithi076 ceph-mon[32095]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:22.537 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:22 smithi088 ceph-mon[36585]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-12T05:52:22.537 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:22 smithi088 ceph-mon[36585]: osdmap e71: 11 total, 10 up, 11 in 2024-03-12T05:52:22.538 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:22 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:22.538 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:22 smithi088 ceph-mon[36585]: from='osd.10 [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:52:22.538 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:22 smithi088 ceph-mon[36585]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:52:22.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:22 smithi082 ceph-mon[36382]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-12T05:52:22.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:22 smithi082 ceph-mon[36382]: osdmap e71: 11 total, 10 up, 11 in 2024-03-12T05:52:22.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:22 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:22.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:22 smithi082 ceph-mon[36382]: from='osd.10 [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:52:22.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:22 smithi082 ceph-mon[36382]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:52:22.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:22 smithi076 ceph-mon[32095]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-12T05:52:22.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:22 smithi076 ceph-mon[32095]: osdmap e71: 11 total, 10 up, 11 in 2024-03-12T05:52:22.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:22 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:22.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:22 smithi076 ceph-mon[32095]: from='osd.10 [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:52:22.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:22 smithi076 ceph-mon[32095]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-03-12T05:52:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-12T05:52:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-12T05:52:23.103 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2024-03-12T05:52:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2024-03-12T05:52:23.103 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144284 s, 727 MB/s 2024-03-12T05:52:23.104 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2024-03-12T05:52:23.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:23 smithi082 ceph-mon[36382]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-12T05:52:23.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:23 smithi082 ceph-mon[36382]: osdmap e72: 11 total, 10 up, 11 in 2024-03-12T05:52:23.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:23 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:23.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:23 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:23.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:23 smithi082 ceph-mon[36382]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:23.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:23 smithi082 ceph-mon[36382]: from='osd.10 ' entity='osd.10' 2024-03-12T05:52:23.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:23 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:23.445 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 05:52:23 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T05:52:23.080+0000 7f228e501700 -1 osd.10 0 waiting for initial osdmap 2024-03-12T05:52:23.445 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 05:52:23 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T05:52:23.101+0000 7f2286aed700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:52:23.564 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:23 smithi088 ceph-mon[36585]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-12T05:52:23.564 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:23 smithi088 ceph-mon[36585]: osdmap e72: 11 total, 10 up, 11 in 2024-03-12T05:52:23.564 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:23 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:23.564 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:23 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:23.564 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:23 smithi088 ceph-mon[36585]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:23.564 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:23 smithi088 ceph-mon[36585]: from='osd.10 ' entity='osd.10' 2024-03-12T05:52:23.564 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:23 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:23.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:23 smithi076 ceph-mon[32095]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-03-12T05:52:23.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:23 smithi076 ceph-mon[32095]: osdmap e72: 11 total, 10 up, 11 in 2024-03-12T05:52:23.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:23 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:23.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:23 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:23.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:23 smithi076 ceph-mon[32095]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-03-12T05:52:23.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:23 smithi076 ceph-mon[32095]: from='osd.10 ' entity='osd.10' 2024-03-12T05:52:23.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:23 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:23.887 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi088:vg_nvme/lv_1 2024-03-12T05:52:24.126 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:52:24.534 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: purged_snaps scrub starts 2024-03-12T05:52:24.534 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: purged_snaps scrub ok 2024-03-12T05:52:24.534 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: osd.10 [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327] boot 2024-03-12T05:52:24.534 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: osdmap e73: 11 total, 11 up, 11 in 2024-03-12T05:52:24.534 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:24.534 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: Detected new or changed devices on smithi082 2024-03-12T05:52:24.534 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:24.534 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:24.535 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:24.535 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:24.535 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:24.535 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:24.535 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi082 to 4270M 2024-03-12T05:52:24.535 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:24.535 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:24.535 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:52:24.535 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:24.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: purged_snaps scrub starts 2024-03-12T05:52:24.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: purged_snaps scrub ok 2024-03-12T05:52:24.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: osd.10 [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327] boot 2024-03-12T05:52:24.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: osdmap e73: 11 total, 11 up, 11 in 2024-03-12T05:52:24.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:24.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: Detected new or changed devices on smithi082 2024-03-12T05:52:24.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:24.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:24.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:24.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:24.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:24.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:24.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi082 to 4270M 2024-03-12T05:52:24.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:24.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:24.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:52:24.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:24.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: purged_snaps scrub starts 2024-03-12T05:52:24.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: purged_snaps scrub ok 2024-03-12T05:52:24.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: osd.10 [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327] boot 2024-03-12T05:52:24.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: osdmap e73: 11 total, 11 up, 11 in 2024-03-12T05:52:24.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:52:24.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: Detected new or changed devices on smithi082 2024-03-12T05:52:24.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:24.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:24.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:24.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:24.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:24.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:24.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi082 to 4270M 2024-03-12T05:52:24.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:24.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:24.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:52:24.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:25.539 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:25 smithi088 ceph-mon[36585]: osdmap e74: 11 total, 11 up, 11 in 2024-03-12T05:52:25.539 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:25 smithi088 ceph-mon[36585]: pgmap v236: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:25.596 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:25 smithi082 ceph-mon[36382]: osdmap e74: 11 total, 11 up, 11 in 2024-03-12T05:52:25.597 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:25 smithi082 ceph-mon[36382]: pgmap v236: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:25.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.638+0000 7f16c04dd700 1 -- 172.21.15.88:0/1080628452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b8102ea0 msgr2=0x7f16b8103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:25.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.638+0000 7f16c04dd700 1 --2- 172.21.15.88:0/1080628452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b8102ea0 0x7f16b8103300 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009990 tx=0x7f16b4009ca0 comp rx=0 tx=0).stop 2024-03-12T05:52:25.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.638+0000 7f16c04dd700 1 -- 172.21.15.88:0/1080628452 shutdown_connections 2024-03-12T05:52:25.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.638+0000 7f16c04dd700 1 --2- 172.21.15.88:0/1080628452 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16b8103840 0x7f16b810a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:25.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.638+0000 7f16c04dd700 1 --2- 172.21.15.88:0/1080628452 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16b8101ca0 0x7f16b81020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:25.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.638+0000 7f16c04dd700 1 --2- 172.21.15.88:0/1080628452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b8102ea0 0x7f16b8103300 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:25.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.638+0000 7f16c04dd700 1 -- 172.21.15.88:0/1080628452 >> 172.21.15.88:0/1080628452 conn(0x7f16b80fd280 msgr2=0x7f16b80ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:52:25.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.638+0000 7f16c04dd700 1 -- 172.21.15.88:0/1080628452 shutdown_connections 2024-03-12T05:52:25.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.638+0000 7f16c04dd700 1 -- 172.21.15.88:0/1080628452 wait complete. 2024-03-12T05:52:25.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16c04dd700 1 Processor -- start 2024-03-12T05:52:25.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16c04dd700 1 -- start start 2024-03-12T05:52:25.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16c04dd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16b8101ca0 0x7f16b8197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:25.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16c04dd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b8102ea0 0x7f16b8198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:25.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16c04dd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16b8103840 0x7f16b819f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:25.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16c04dd700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b819f7a0 con 0x7f16b8101ca0 2024-03-12T05:52:25.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16c04dd700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b819f8e0 con 0x7f16b8103840 2024-03-12T05:52:25.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16c04dd700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b819faa0 con 0x7f16b8102ea0 2024-03-12T05:52:25.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16bda78700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b8102ea0 0x7f16b8198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:25.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16bda78700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b8102ea0 0x7f16b8198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35190/0 (socket says 172.21.15.88:35190) 2024-03-12T05:52:25.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16bda78700 1 -- 172.21.15.88:0/242554133 learned_addr learned my addr 172.21.15.88:0/242554133 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-03-12T05:52:25.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16bea7a700 1 --2- 172.21.15.88:0/242554133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16b8103840 0x7f16b819f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:25.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16be279700 1 --2- 172.21.15.88:0/242554133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16b8101ca0 0x7f16b8197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:25.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16bda78700 1 -- 172.21.15.88:0/242554133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16b8103840 msgr2=0x7f16b819f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:25.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16bda78700 1 --2- 172.21.15.88:0/242554133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16b8103840 0x7f16b819f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:25.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16bda78700 1 -- 172.21.15.88:0/242554133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16b8101ca0 msgr2=0x7f16b8197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:25.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16bda78700 1 --2- 172.21.15.88:0/242554133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16b8101ca0 0x7f16b8197c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:25.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.639+0000 7f16bda78700 1 -- 172.21.15.88:0/242554133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b40095f0 con 0x7f16b8102ea0 2024-03-12T05:52:25.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.640+0000 7f16bda78700 1 --2- 172.21.15.88:0/242554133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b8102ea0 0x7f16b8198190 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f16b40095c0 tx=0x7f16b400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:52:25.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.640+0000 7f16af7fe700 1 -- 172.21.15.88:0/242554133 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b401d070 con 0x7f16b8102ea0 2024-03-12T05:52:25.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.640+0000 7f16c04dd700 1 -- 172.21.15.88:0/242554133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b819fd20 con 0x7f16b8102ea0 2024-03-12T05:52:25.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.640+0000 7f16af7fe700 1 -- 172.21.15.88:0/242554133 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16b400fb40 con 0x7f16b8102ea0 2024-03-12T05:52:25.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.640+0000 7f16af7fe700 1 -- 172.21.15.88:0/242554133 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b40173f0 con 0x7f16b8102ea0 2024-03-12T05:52:25.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.640+0000 7f16c04dd700 1 -- 172.21.15.88:0/242554133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b81a0130 con 0x7f16b8102ea0 2024-03-12T05:52:25.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.641+0000 7f16c04dd700 1 -- 172.21.15.88:0/242554133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b8068930 con 0x7f16b8102ea0 2024-03-12T05:52:25.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.642+0000 7f16af7fe700 1 -- 172.21.15.88:0/242554133 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f16b400fcb0 con 0x7f16b8102ea0 2024-03-12T05:52:25.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.642+0000 7f16af7fe700 1 --2- 172.21.15.88:0/242554133 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f16a40a1ac0 0x7f16a40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:25.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.643+0000 7f16af7fe700 1 -- 172.21.15.88:0/242554133 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f16b40ce200 con 0x7f16b8102ea0 2024-03-12T05:52:25.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.643+0000 7f16be279700 1 --2- 172.21.15.88:0/242554133 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f16a40a1ac0 0x7f16a40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:25.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.643+0000 7f16be279700 1 --2- 172.21.15.88:0/242554133 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f16a40a1ac0 0x7f16a40a3f80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f16a8005fd0 tx=0x7f16a8005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:52:25.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.646+0000 7f16af7fe700 1 -- 172.21.15.88:0/242554133 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f16b4097ad0 con 0x7f16b8102ea0 2024-03-12T05:52:25.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:25 smithi076 ceph-mon[32095]: osdmap e74: 11 total, 11 up, 11 in 2024-03-12T05:52:25.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:25 smithi076 ceph-mon[32095]: pgmap v236: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:25.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:25.844+0000 7f16c04dd700 1 -- 172.21.15.88:0/242554133 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f16b802d910 con 0x7f16a40a1ac0 2024-03-12T05:52:26.587 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:26 smithi088 ceph-mon[36585]: osdmap e75: 11 total, 11 up, 11 in 2024-03-12T05:52:26.587 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:26 smithi088 ceph-mon[36585]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:52:26.587 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:52:26.588 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:52:26.588 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:26 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:26.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:26 smithi082 ceph-mon[36382]: osdmap e75: 11 total, 11 up, 11 in 2024-03-12T05:52:26.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:26 smithi082 ceph-mon[36382]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:52:26.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:26 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:52:26.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:26 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:52:26.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:26 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:26.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:26 smithi076 ceph-mon[32095]: osdmap e75: 11 total, 11 up, 11 in 2024-03-12T05:52:26.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:26 smithi076 ceph-mon[32095]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:52:26.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:26 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T05:52:26.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:26 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T05:52:26.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:26 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:27.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:27 smithi088 ceph-mon[36585]: osdmap e76: 11 total, 11 up, 11 in 2024-03-12T05:52:27.454 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:27 smithi088 ceph-mon[36585]: pgmap v239: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:27.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:27 smithi082 ceph-mon[36382]: osdmap e76: 11 total, 11 up, 11 in 2024-03-12T05:52:27.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:27 smithi082 ceph-mon[36382]: pgmap v239: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:27.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:27 smithi076 ceph-mon[32095]: osdmap e76: 11 total, 11 up, 11 in 2024-03-12T05:52:27.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:27 smithi076 ceph-mon[32095]: pgmap v239: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:28.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:28 smithi082 ceph-mon[36382]: osdmap e77: 11 total, 11 up, 11 in 2024-03-12T05:52:28.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:28 smithi088 ceph-mon[36585]: osdmap e77: 11 total, 11 up, 11 in 2024-03-12T05:52:28.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:28 smithi076 ceph-mon[32095]: osdmap e77: 11 total, 11 up, 11 in 2024-03-12T05:52:29.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:29 smithi082 ceph-mon[36382]: from='client.? 172.21.15.88:0/1164875428' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd7c7012-e16e-4a96-b9e0-fb8d7a5af114"}]: dispatch 2024-03-12T05:52:29.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:29 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd7c7012-e16e-4a96-b9e0-fb8d7a5af114"}]: dispatch 2024-03-12T05:52:29.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:29 smithi082 ceph-mon[36382]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd7c7012-e16e-4a96-b9e0-fb8d7a5af114"}]': finished 2024-03-12T05:52:29.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:29 smithi082 ceph-mon[36382]: osdmap e78: 12 total, 11 up, 12 in 2024-03-12T05:52:29.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:29 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:29.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:29 smithi082 ceph-mon[36382]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T05:52:29.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:29 smithi082 ceph-mon[36382]: from='client.? 172.21.15.88:0/2504113880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:52:29.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:29 smithi088 ceph-mon[36585]: from='client.? 172.21.15.88:0/1164875428' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd7c7012-e16e-4a96-b9e0-fb8d7a5af114"}]: dispatch 2024-03-12T05:52:29.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:29 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd7c7012-e16e-4a96-b9e0-fb8d7a5af114"}]: dispatch 2024-03-12T05:52:29.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:29 smithi088 ceph-mon[36585]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd7c7012-e16e-4a96-b9e0-fb8d7a5af114"}]': finished 2024-03-12T05:52:29.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:29 smithi088 ceph-mon[36585]: osdmap e78: 12 total, 11 up, 12 in 2024-03-12T05:52:29.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:29 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:29.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:29 smithi088 ceph-mon[36585]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T05:52:29.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:29 smithi088 ceph-mon[36585]: from='client.? 172.21.15.88:0/2504113880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:52:29.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:29 smithi076 ceph-mon[32095]: from='client.? 172.21.15.88:0/1164875428' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd7c7012-e16e-4a96-b9e0-fb8d7a5af114"}]: dispatch 2024-03-12T05:52:29.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:29 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd7c7012-e16e-4a96-b9e0-fb8d7a5af114"}]: dispatch 2024-03-12T05:52:29.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:29 smithi076 ceph-mon[32095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd7c7012-e16e-4a96-b9e0-fb8d7a5af114"}]': finished 2024-03-12T05:52:29.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:29 smithi076 ceph-mon[32095]: osdmap e78: 12 total, 11 up, 12 in 2024-03-12T05:52:29.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:29 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:29.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:29 smithi076 ceph-mon[32095]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T05:52:29.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:29 smithi076 ceph-mon[32095]: from='client.? 172.21.15.88:0/2504113880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T05:52:32.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:31 smithi082 ceph-mon[36382]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-12T05:52:32.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:31 smithi088 ceph-mon[36585]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-12T05:52:32.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:31 smithi076 ceph-mon[32095]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-12T05:52:33.966 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:33 smithi088 ceph-mon[36585]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-12T05:52:34.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:33 smithi082 ceph-mon[36382]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-12T05:52:34.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:33 smithi076 ceph-mon[32095]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-12T05:52:35.977 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:35 smithi088 ceph-mon[36585]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:52:36.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:35 smithi082 ceph-mon[36382]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:52:36.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:35 smithi076 ceph-mon[32095]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T05:52:37.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:36 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-12T05:52:37.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:36 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:37.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:36 smithi082 ceph-mon[36382]: Deploying daemon osd.11 on smithi088 2024-03-12T05:52:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:36 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-12T05:52:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:36 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:37.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:36 smithi088 ceph-mon[36585]: Deploying daemon osd.11 on smithi088 2024-03-12T05:52:37.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:36 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-12T05:52:37.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:36 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:37.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:36 smithi076 ceph-mon[32095]: Deploying daemon osd.11 on smithi088 2024-03-12T05:52:38.130 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:37 smithi088 ceph-mon[36585]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-12T05:52:38.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:37 smithi082 ceph-mon[36382]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-12T05:52:38.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:37 smithi076 ceph-mon[32095]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-12T05:52:40.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:39 smithi082 ceph-mon[36382]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:40.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:39 smithi088 ceph-mon[36585]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:40.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:39 smithi076 ceph-mon[32095]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:42.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:41 smithi082 ceph-mon[36382]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:42.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:42.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:42.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:52:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:41 smithi088 ceph-mon[36585]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:42.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:52:42.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:41 smithi076 ceph-mon[32095]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:42.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:42.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:42.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:52:44.139 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:43 smithi088 ceph-mon[36585]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:44.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:43 smithi082 ceph-mon[36382]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:44.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:43 smithi076 ceph-mon[32095]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:44.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:44.757+0000 7f16af7fe700 1 -- 172.21.15.88:0/242554133 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f16b802d910 con 0x7f16a40a1ac0 2024-03-12T05:52:44.760 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 11 on host 'smithi088' 2024-03-12T05:52:44.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:44.759+0000 7f16c04dd700 1 -- 172.21.15.88:0/242554133 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f16a40a1ac0 msgr2=0x7f16a40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:44.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:44.759+0000 7f16c04dd700 1 --2- 172.21.15.88:0/242554133 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f16a40a1ac0 0x7f16a40a3f80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f16a8005fd0 tx=0x7f16a8005dd0 comp rx=0 tx=0).stop 2024-03-12T05:52:44.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:44.759+0000 7f16c04dd700 1 -- 172.21.15.88:0/242554133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b8102ea0 msgr2=0x7f16b8198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:44.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:44.759+0000 7f16c04dd700 1 --2- 172.21.15.88:0/242554133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b8102ea0 0x7f16b8198190 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f16b40095c0 tx=0x7f16b400f690 comp rx=0 tx=0).stop 2024-03-12T05:52:44.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:44.759+0000 7f16c04dd700 1 -- 172.21.15.88:0/242554133 shutdown_connections 2024-03-12T05:52:44.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:44.759+0000 7f16c04dd700 1 --2- 172.21.15.88:0/242554133 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f16a40a1ac0 0x7f16a40a3f80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:44.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:44.759+0000 7f16c04dd700 1 --2- 172.21.15.88:0/242554133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f16b8103840 0x7f16b819f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:44.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:44.759+0000 7f16c04dd700 1 --2- 172.21.15.88:0/242554133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16b8101ca0 0x7f16b8197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:44.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:44.759+0000 7f16c04dd700 1 --2- 172.21.15.88:0/242554133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b8102ea0 0x7f16b8198190 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:44.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:44.759+0000 7f16c04dd700 1 -- 172.21.15.88:0/242554133 >> 172.21.15.88:0/242554133 conn(0x7f16b80fd280 msgr2=0x7f16b8106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:52:44.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:44.759+0000 7f16c04dd700 1 -- 172.21.15.88:0/242554133 shutdown_connections 2024-03-12T05:52:44.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:44.759+0000 7f16c04dd700 1 -- 172.21.15.88:0/242554133 wait complete. 2024-03-12T05:52:45.294 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.294 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.294 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:45.294 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:52:45.294 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.295 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.295 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.295 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:44 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:52:45.321 DEBUG:teuthology.orchestra.run.smithi088:osd.11> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.11.service 2024-03-12T05:52:45.323 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-12T05:52:45.324 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-12T05:52:45.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:45.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:52:45.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:44 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:52:45.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:45.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:52:45.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:45.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:44 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:52:45.564 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:52:46.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:46 smithi082 ceph-mon[36382]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:46.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:46 smithi082 ceph-mon[36382]: from='osd.11 [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-12T05:52:46.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:46 smithi082 ceph-mon[36382]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-12T05:52:46.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:46 smithi088 ceph-mon[36585]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:46.450 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:46 smithi088 ceph-mon[36585]: from='osd.11 [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-12T05:52:46.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:46 smithi088 ceph-mon[36585]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-12T05:52:46.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:46 smithi076 ceph-mon[32095]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:46.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:46 smithi076 ceph-mon[32095]: from='osd.11 [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-12T05:52:46.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:46 smithi076 ceph-mon[32095]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-12T05:52:47.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.214+0000 7ff3d6daf700 1 -- 172.21.15.76:0/4269441584 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3d0103870 msgr2=0x7ff3d010a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:47.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.214+0000 7ff3d6daf700 1 --2- 172.21.15.76:0/4269441584 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3d0103870 0x7ff3d010a120 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800b0d0 tx=0x7ff3c800b3e0 comp rx=0 tx=0).stop 2024-03-12T05:52:47.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.214+0000 7ff3d6daf700 1 -- 172.21.15.76:0/4269441584 shutdown_connections 2024-03-12T05:52:47.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.214+0000 7ff3d6daf700 1 --2- 172.21.15.76:0/4269441584 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3d0103870 0x7ff3d010a120 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:47.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.214+0000 7ff3d6daf700 1 --2- 172.21.15.76:0/4269441584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3d0101cd0 0x7ff3d01020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:47.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.214+0000 7ff3d6daf700 1 --2- 172.21.15.76:0/4269441584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0102ed0 0x7ff3d0103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:47.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.214+0000 7ff3d6daf700 1 -- 172.21.15.76:0/4269441584 >> 172.21.15.76:0/4269441584 conn(0x7ff3d00fd270 msgr2=0x7ff3d00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:52:47.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.214+0000 7ff3d6daf700 1 -- 172.21.15.76:0/4269441584 shutdown_connections 2024-03-12T05:52:47.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.214+0000 7ff3d6daf700 1 -- 172.21.15.76:0/4269441584 wait complete. 2024-03-12T05:52:47.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.214+0000 7ff3d6daf700 1 Processor -- start 2024-03-12T05:52:47.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.214+0000 7ff3d6daf700 1 -- start start 2024-03-12T05:52:47.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d6daf700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0101cd0 0x7ff3d0197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d6daf700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3d0102ed0 0x7ff3d0198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d6daf700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3d0103870 0x7ff3d019f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d6daf700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d019f860 con 0x7ff3d0103870 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d6daf700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d019f9a0 con 0x7ff3d0102ed0 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d6daf700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d019fb60 con 0x7ff3d0101cd0 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d534c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3d0103870 0x7ff3d019f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d4b4b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0101cd0 0x7ff3d0197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d534c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3d0103870 0x7ff3d019f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53190/0 (socket says 172.21.15.76:53190) 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d4b4b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0101cd0 0x7ff3d0197d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.76:49672/0 (socket says 172.21.15.76:49672) 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d534c700 1 -- 172.21.15.76:0/723945056 learned_addr learned my addr 172.21.15.76:0/723945056 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3cffff700 1 --2- 172.21.15.76:0/723945056 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3d0102ed0 0x7ff3d0198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d534c700 1 -- 172.21.15.76:0/723945056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0101cd0 msgr2=0x7ff3d0197d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d534c700 1 --2- 172.21.15.76:0/723945056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0101cd0 0x7ff3d0197d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d534c700 1 -- 172.21.15.76:0/723945056 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3d0102ed0 msgr2=0x7ff3d0198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d534c700 1 --2- 172.21.15.76:0/723945056 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3d0102ed0 0x7ff3d0198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d534c700 1 -- 172.21.15.76:0/723945056 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c8009d00 con 0x7ff3d0103870 2024-03-12T05:52:47.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d4b4b700 1 --2- 172.21.15.76:0/723945056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0101cd0 0x7ff3d0197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:52:47.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3d534c700 1 --2- 172.21.15.76:0/723945056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3d0103870 0x7ff3d019f290 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8007c90 tx=0x7ff3c8009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:52:47.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.215+0000 7ff3cdffb700 1 -- 172.21.15.76:0/723945056 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c800e040 con 0x7ff3d0103870 2024-03-12T05:52:47.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.216+0000 7ff3d6daf700 1 -- 172.21.15.76:0/723945056 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d019fde0 con 0x7ff3d0103870 2024-03-12T05:52:47.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.216+0000 7ff3cdffb700 1 -- 172.21.15.76:0/723945056 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3c801dd70 con 0x7ff3d0103870 2024-03-12T05:52:47.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.216+0000 7ff3cdffb700 1 -- 172.21.15.76:0/723945056 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c801be20 con 0x7ff3d0103870 2024-03-12T05:52:47.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.216+0000 7ff3d6daf700 1 -- 172.21.15.76:0/723945056 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d01a01a0 con 0x7ff3d0103870 2024-03-12T05:52:47.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.217+0000 7ff3d6daf700 1 -- 172.21.15.76:0/723945056 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d0109030 con 0x7ff3d0103870 2024-03-12T05:52:47.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.219+0000 7ff3cdffb700 1 -- 172.21.15.76:0/723945056 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff3c801d890 con 0x7ff3d0103870 2024-03-12T05:52:47.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.219+0000 7ff3cdffb700 1 --2- 172.21.15.76:0/723945056 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff3c00a1ac0 0x7ff3c00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:47.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:47 smithi076 ceph-mon[32095]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-12T05:52:47.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:47 smithi076 ceph-mon[32095]: osdmap e79: 12 total, 11 up, 12 in 2024-03-12T05:52:47.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:47 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:47.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:47 smithi076 ceph-mon[32095]: from='osd.11 [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:52:47.223 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:47 smithi076 ceph-mon[32095]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:52:47.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.219+0000 7ff3d4b4b700 1 --2- 172.21.15.76:0/723945056 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff3c00a1ac0 0x7ff3c00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:47.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.219+0000 7ff3cdffb700 1 -- 172.21.15.76:0/723945056 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6847+0+0 (secure 0 0 0) 0x7ff3c80cf220 con 0x7ff3d0103870 2024-03-12T05:52:47.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.220+0000 7ff3d4b4b700 1 --2- 172.21.15.76:0/723945056 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff3c00a1ac0 0x7ff3c00a3f80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff3d01991f0 tx=0x7ff3c400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:52:47.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.223+0000 7ff3cdffb700 1 -- 172.21.15.76:0/723945056 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff3c8060d20 con 0x7ff3d0103870 2024-03-12T05:52:47.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.416+0000 7ff3d6daf700 1 -- 172.21.15.76:0/723945056 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7ff3d010d6e0 con 0x7ff3d0103870 2024-03-12T05:52:47.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.416+0000 7ff3cdffb700 1 -- 172.21.15.76:0/723945056 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) v1 ==== 74+0+133 (secure 0 0 0) 0x7ff3c8098a80 con 0x7ff3d0103870 2024-03-12T05:52:47.420 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:52:47.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.419+0000 7ff3d6daf700 1 -- 172.21.15.76:0/723945056 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff3c00a1ac0 msgr2=0x7ff3c00a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:47.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.419+0000 7ff3d6daf700 1 --2- 172.21.15.76:0/723945056 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff3c00a1ac0 0x7ff3c00a3f80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff3d01991f0 tx=0x7ff3c400a380 comp rx=0 tx=0).stop 2024-03-12T05:52:47.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.419+0000 7ff3d6daf700 1 -- 172.21.15.76:0/723945056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3d0103870 msgr2=0x7ff3d019f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:47.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.419+0000 7ff3d6daf700 1 --2- 172.21.15.76:0/723945056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3d0103870 0x7ff3d019f290 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8007c90 tx=0x7ff3c8009380 comp rx=0 tx=0).stop 2024-03-12T05:52:47.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.419+0000 7ff3d6daf700 1 -- 172.21.15.76:0/723945056 shutdown_connections 2024-03-12T05:52:47.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.419+0000 7ff3d6daf700 1 --2- 172.21.15.76:0/723945056 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff3c00a1ac0 0x7ff3c00a3f80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:47.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.419+0000 7ff3d6daf700 1 --2- 172.21.15.76:0/723945056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3d0103870 0x7ff3d019f290 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:47.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.419+0000 7ff3d6daf700 1 --2- 172.21.15.76:0/723945056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3d0101cd0 0x7ff3d0197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:47.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.419+0000 7ff3d6daf700 1 --2- 172.21.15.76:0/723945056 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3d0102ed0 0x7ff3d0198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:47.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.419+0000 7ff3d6daf700 1 -- 172.21.15.76:0/723945056 >> 172.21.15.76:0/723945056 conn(0x7ff3d00fd270 msgr2=0x7ff3d0106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:52:47.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.419+0000 7ff3d6daf700 1 -- 172.21.15.76:0/723945056 shutdown_connections 2024-03-12T05:52:47.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:47.419+0000 7ff3d6daf700 1 -- 172.21.15.76:0/723945056 wait complete. 2024-03-12T05:52:47.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:47 smithi082 ceph-mon[36382]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-12T05:52:47.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:47 smithi082 ceph-mon[36382]: osdmap e79: 12 total, 11 up, 12 in 2024-03-12T05:52:47.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:47 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:47.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:47 smithi082 ceph-mon[36382]: from='osd.11 [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:52:47.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:47 smithi082 ceph-mon[36382]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:52:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:47 smithi088 ceph-mon[36585]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-12T05:52:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:47 smithi088 ceph-mon[36585]: osdmap e79: 12 total, 11 up, 12 in 2024-03-12T05:52:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:47 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:47 smithi088 ceph-mon[36585]: from='osd.11 [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:52:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:47 smithi088 ceph-mon[36585]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2024-03-12T05:52:47.984 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 05:52:47 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T05:52:47.868+0000 7f4987cc3700 -1 osd.11 0 waiting for initial osdmap 2024-03-12T05:52:47.984 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 05:52:47 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T05:52:47.891+0000 7f497eaac700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T05:52:48.087 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":80,"num_osds":12,"num_up_osds":11,"osd_up_since":1710222743,"num_in_osds":12,"osd_in_since":1710222748,"num_remapped_pgs":0} 2024-03-12T05:52:48.424 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:48 smithi088 ceph-mon[36585]: purged_snaps scrub starts 2024-03-12T05:52:48.424 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:48 smithi088 ceph-mon[36585]: purged_snaps scrub ok 2024-03-12T05:52:48.424 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:48 smithi088 ceph-mon[36585]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:48.424 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:48 smithi088 ceph-mon[36585]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-12T05:52:48.424 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:48 smithi088 ceph-mon[36585]: osdmap e80: 12 total, 11 up, 12 in 2024-03-12T05:52:48.424 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:48 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:48.424 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:48 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:48.424 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:48 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/723945056' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T05:52:48.425 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:48 smithi088 ceph-mon[36585]: from='osd.11 ' entity='osd.11' 2024-03-12T05:52:48.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:48 smithi082 ceph-mon[36382]: purged_snaps scrub starts 2024-03-12T05:52:48.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:48 smithi082 ceph-mon[36382]: purged_snaps scrub ok 2024-03-12T05:52:48.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:48 smithi082 ceph-mon[36382]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:48.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:48 smithi082 ceph-mon[36382]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-12T05:52:48.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:48 smithi082 ceph-mon[36382]: osdmap e80: 12 total, 11 up, 12 in 2024-03-12T05:52:48.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:48 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:48.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:48 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:48.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:48 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/723945056' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T05:52:48.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:48 smithi082 ceph-mon[36382]: from='osd.11 ' entity='osd.11' 2024-03-12T05:52:48.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:48 smithi076 ceph-mon[32095]: purged_snaps scrub starts 2024-03-12T05:52:48.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:48 smithi076 ceph-mon[32095]: purged_snaps scrub ok 2024-03-12T05:52:48.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:48 smithi076 ceph-mon[32095]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:48.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:48 smithi076 ceph-mon[32095]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2024-03-12T05:52:48.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:48 smithi076 ceph-mon[32095]: osdmap e80: 12 total, 11 up, 12 in 2024-03-12T05:52:48.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:48 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:48.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:48 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:48.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:48 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/723945056' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T05:52:48.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:48 smithi076 ceph-mon[32095]: from='osd.11 ' entity='osd.11' 2024-03-12T05:52:49.088 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-12T05:52:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:52:49.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:49.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:49.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:52:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:49 smithi082 ceph-mon[36382]: osd.11 [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649] boot 2024-03-12T05:52:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:49 smithi082 ceph-mon[36382]: osdmap e81: 12 total, 12 up, 12 in 2024-03-12T05:52:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:49 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:49.449 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:49.449 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:49.449 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:49.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:49.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:49.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:49.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:49.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:49.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:49.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:52:49.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:49.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:49 smithi076 ceph-mon[32095]: osd.11 [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649] boot 2024-03-12T05:52:49.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:49 smithi076 ceph-mon[32095]: osdmap e81: 12 total, 12 up, 12 in 2024-03-12T05:52:49.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:49 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:52:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:52:49.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:52:49.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:52:49.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:49 smithi088 ceph-mon[36585]: osd.11 [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649] boot 2024-03-12T05:52:49.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:49 smithi088 ceph-mon[36585]: osdmap e81: 12 total, 12 up, 12 in 2024-03-12T05:52:49.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:49 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:52:50.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:50 smithi076 ceph-mon[32095]: Detected new or changed devices on smithi088 2024-03-12T05:52:50.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:50 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi088 to 4270M 2024-03-12T05:52:50.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:50 smithi076 ceph-mon[32095]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:50.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:50 smithi076 ceph-mon[32095]: osdmap e82: 12 total, 12 up, 12 in 2024-03-12T05:52:50.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:50 smithi082 ceph-mon[36382]: Detected new or changed devices on smithi088 2024-03-12T05:52:50.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:50 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi088 to 4270M 2024-03-12T05:52:50.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:50 smithi082 ceph-mon[36382]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:50.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:50 smithi082 ceph-mon[36382]: osdmap e82: 12 total, 12 up, 12 in 2024-03-12T05:52:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:50 smithi088 ceph-mon[36585]: Detected new or changed devices on smithi088 2024-03-12T05:52:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:50 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi088 to 4270M 2024-03-12T05:52:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:50 smithi088 ceph-mon[36585]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-12T05:52:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:50 smithi088 ceph-mon[36585]: osdmap e82: 12 total, 12 up, 12 in 2024-03-12T05:52:50.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.977+0000 7f7799abe700 1 -- 172.21.15.76:0/3327098540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794101da0 msgr2=0x7f7794102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:50.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.977+0000 7f7799abe700 1 --2- 172.21.15.76:0/3327098540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794101da0 0x7f7794102200 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f777c0099c0 tx=0x7f777c009cd0 comp rx=0 tx=0).stop 2024-03-12T05:52:50.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.977+0000 7f7799abe700 1 -- 172.21.15.76:0/3327098540 shutdown_connections 2024-03-12T05:52:50.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.977+0000 7f7799abe700 1 --2- 172.21.15.76:0/3327098540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7794102740 0x7f7794108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:50.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.977+0000 7f7799abe700 1 --2- 172.21.15.76:0/3327098540 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7794100ba0 0x7f7794100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:50.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.977+0000 7f7799abe700 1 --2- 172.21.15.76:0/3327098540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794101da0 0x7f7794102200 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:50.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.977+0000 7f7799abe700 1 -- 172.21.15.76:0/3327098540 >> 172.21.15.76:0/3327098540 conn(0x7f77940fc160 msgr2=0x7f77940fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:52:50.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.977+0000 7f7799abe700 1 -- 172.21.15.76:0/3327098540 shutdown_connections 2024-03-12T05:52:50.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.977+0000 7f7799abe700 1 -- 172.21.15.76:0/3327098540 wait complete. 2024-03-12T05:52:50.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.977+0000 7f7799abe700 1 Processor -- start 2024-03-12T05:52:50.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.977+0000 7f7799abe700 1 -- start start 2024-03-12T05:52:50.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f7799abe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7794100ba0 0x7f779406af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:50.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f7799abe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794101da0 0x7f779419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:50.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f7799abe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7794102740 0x7f779419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:50.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f7799abe700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779419f6e0 con 0x7f7794101da0 2024-03-12T05:52:50.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f7799abe700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779419f850 con 0x7f7794100ba0 2024-03-12T05:52:50.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f7799abe700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779419fa10 con 0x7f7794102740 2024-03-12T05:52:50.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f778affd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794101da0 0x7f779419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:50.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f778affd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794101da0 0x7f779419c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53208/0 (socket says 172.21.15.76:53208) 2024-03-12T05:52:50.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f778affd700 1 -- 172.21.15.76:0/4482641 learned_addr learned my addr 172.21.15.76:0/4482641 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:52:50.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f77937fe700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7794100ba0 0x7f779406af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:50.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f7793fff700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7794102740 0x7f779419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:50.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f778affd700 1 -- 172.21.15.76:0/4482641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7794102740 msgr2=0x7f779419ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:50.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f778affd700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7794102740 0x7f779419ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:50.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f778affd700 1 -- 172.21.15.76:0/4482641 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7794100ba0 msgr2=0x7f779406af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:50.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f778affd700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7794100ba0 0x7f779406af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:50.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f778affd700 1 -- 172.21.15.76:0/4482641 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c0095f0 con 0x7f7794101da0 2024-03-12T05:52:50.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f7793fff700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7794102740 0x7f779419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:52:50.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f778affd700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794101da0 0x7f779419c8b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f777c00b580 tx=0x7f777c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:52:50.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.978+0000 7f77917fa700 1 -- 172.21.15.76:0/4482641 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c01d070 con 0x7f7794101da0 2024-03-12T05:52:50.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.979+0000 7f7799abe700 1 -- 172.21.15.76:0/4482641 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779419fc90 con 0x7f7794101da0 2024-03-12T05:52:50.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.979+0000 7f77917fa700 1 -- 172.21.15.76:0/4482641 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f777c0041d0 con 0x7f7794101da0 2024-03-12T05:52:50.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.979+0000 7f77917fa700 1 -- 172.21.15.76:0/4482641 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c00f460 con 0x7f7794101da0 2024-03-12T05:52:50.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.979+0000 7f7799abe700 1 -- 172.21.15.76:0/4482641 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77941a0180 con 0x7f7794101da0 2024-03-12T05:52:50.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.980+0000 7f7799abe700 1 -- 172.21.15.76:0/4482641 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7794107e90 con 0x7f7794101da0 2024-03-12T05:52:50.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.981+0000 7f77917fa700 1 -- 172.21.15.76:0/4482641 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f777c022670 con 0x7f7794101da0 2024-03-12T05:52:50.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.981+0000 7f77917fa700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f77800a1ac0 0x7f77800a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:50.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.981+0000 7f77937fe700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f77800a1ac0 0x7f77800a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:50.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.981+0000 7f77917fa700 1 -- 172.21.15.76:0/4482641 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f777c0ce430 con 0x7f7794101da0 2024-03-12T05:52:50.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.982+0000 7f77937fe700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f77800a1ac0 0x7f77800a3f80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7794101c00 tx=0x7f778400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:52:50.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:50.985+0000 7f77917fa700 1 -- 172.21.15.76:0/4482641 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f777c097b70 con 0x7f7794101da0 2024-03-12T05:52:51.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:51.179+0000 7f7799abe700 1 -- 172.21.15.76:0/4482641 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f779410c600 con 0x7f7794101da0 2024-03-12T05:52:51.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:51.179+0000 7f77917fa700 1 -- 172.21.15.76:0/4482641 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) v1 ==== 74+0+133 (secure 0 0 0) 0x7f777c09b230 con 0x7f7794101da0 2024-03-12T05:52:51.182 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:52:51.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:51.181+0000 7f7799abe700 1 -- 172.21.15.76:0/4482641 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f77800a1ac0 msgr2=0x7f77800a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:51.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:51.181+0000 7f7799abe700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f77800a1ac0 0x7f77800a3f80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7794101c00 tx=0x7f778400a3b0 comp rx=0 tx=0).stop 2024-03-12T05:52:51.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:51.181+0000 7f7799abe700 1 -- 172.21.15.76:0/4482641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794101da0 msgr2=0x7f779419c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:51.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:51.181+0000 7f7799abe700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794101da0 0x7f779419c8b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f777c00b580 tx=0x7f777c0057c0 comp rx=0 tx=0).stop 2024-03-12T05:52:51.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:51.181+0000 7f7799abe700 1 -- 172.21.15.76:0/4482641 shutdown_connections 2024-03-12T05:52:51.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:51.181+0000 7f7799abe700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f77800a1ac0 0x7f77800a3f80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:51.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:51.181+0000 7f7799abe700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7794102740 0x7f779419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:51.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:51.181+0000 7f7799abe700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7794100ba0 0x7f779406af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:51.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:51.181+0000 7f7799abe700 1 --2- 172.21.15.76:0/4482641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7794101da0 0x7f779419c8b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:51.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:51.181+0000 7f7799abe700 1 -- 172.21.15.76:0/4482641 >> 172.21.15.76:0/4482641 conn(0x7f77940fc160 msgr2=0x7f7794104fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:52:51.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:51.181+0000 7f7799abe700 1 -- 172.21.15.76:0/4482641 shutdown_connections 2024-03-12T05:52:51.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:51.181+0000 7f7799abe700 1 -- 172.21.15.76:0/4482641 wait complete. 2024-03-12T05:52:51.751 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1710222768,"num_in_osds":12,"osd_in_since":1710222748,"num_remapped_pgs":0} 2024-03-12T05:52:51.752 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T05:52:51.986 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:52:52.301 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:52 smithi088 ceph-mon[36585]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:52.302 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:52 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4482641' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T05:52:52.361 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:52 smithi076 ceph-mon[32095]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:52.362 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:52 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4482641' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T05:52:52.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:52 smithi082 ceph-mon[36382]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:52.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:52 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4482641' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T05:52:53.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.635+0000 7f755470a700 1 -- 172.21.15.76:0/2637606921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f754c106d40 msgr2=0x7f754c107120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:53.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.635+0000 7f755470a700 1 --2- 172.21.15.76:0/2637606921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f754c106d40 0x7f754c107120 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f75480099c0 tx=0x7f7548009cd0 comp rx=0 tx=0).stop 2024-03-12T05:52:53.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.635+0000 7f755470a700 1 -- 172.21.15.76:0/2637606921 shutdown_connections 2024-03-12T05:52:53.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.635+0000 7f755470a700 1 --2- 172.21.15.76:0/2637606921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f754c101030 0x7f754c10ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:53.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.635+0000 7f755470a700 1 --2- 172.21.15.76:0/2637606921 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f754c106d40 0x7f754c107120 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:53.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.635+0000 7f755470a700 1 --2- 172.21.15.76:0/2637606921 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f754c100670 0x7f754c100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:53.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.635+0000 7f755470a700 1 -- 172.21.15.76:0/2637606921 >> 172.21.15.76:0/2637606921 conn(0x7f754c075ec0 msgr2=0x7f754c0762d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:52:53.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.635+0000 7f755470a700 1 -- 172.21.15.76:0/2637606921 shutdown_connections 2024-03-12T05:52:53.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.635+0000 7f755470a700 1 -- 172.21.15.76:0/2637606921 wait complete. 2024-03-12T05:52:53.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.635+0000 7f755470a700 1 Processor -- start 2024-03-12T05:52:53.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.635+0000 7f755470a700 1 -- start start 2024-03-12T05:52:53.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.635+0000 7f755470a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f754c100670 0x7f754c1981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:53.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.635+0000 7f755470a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f754c101030 0x7f754c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:53.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f755470a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f754c106d40 0x7f754c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:53.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f755470a700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c19d030 con 0x7f754c101030 2024-03-12T05:52:53.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f755470a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c19d1a0 con 0x7f754c100670 2024-03-12T05:52:53.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f755470a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c19d360 con 0x7f754c106d40 2024-03-12T05:52:53.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f7551ca5700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f754c101030 0x7f754c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:53.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f7551ca5700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f754c101030 0x7f754c1986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47654/0 (socket says 172.21.15.76:47654) 2024-03-12T05:52:53.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f7551ca5700 1 -- 172.21.15.76:0/3886948272 learned_addr learned my addr 172.21.15.76:0/3886948272 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:52:53.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f7552ca7700 1 --2- 172.21.15.76:0/3886948272 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f754c106d40 0x7f754c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:53.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f75524a6700 1 --2- 172.21.15.76:0/3886948272 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f754c100670 0x7f754c1981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:53.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f7551ca5700 1 -- 172.21.15.76:0/3886948272 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f754c106d40 msgr2=0x7f754c19ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:53.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f7551ca5700 1 --2- 172.21.15.76:0/3886948272 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f754c106d40 0x7f754c19ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:53.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f7551ca5700 1 -- 172.21.15.76:0/3886948272 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f754c100670 msgr2=0x7f754c1981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:53.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f7551ca5700 1 --2- 172.21.15.76:0/3886948272 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f754c100670 0x7f754c1981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:53.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f7551ca5700 1 -- 172.21.15.76:0/3886948272 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75480095f0 con 0x7f754c101030 2024-03-12T05:52:53.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f7551ca5700 1 --2- 172.21.15.76:0/3886948272 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f754c101030 0x7f754c1986e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f753c00b530 tx=0x7f753c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:52:53.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f75437fe700 1 -- 172.21.15.76:0/3886948272 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c011830 con 0x7f754c101030 2024-03-12T05:52:53.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f75437fe700 1 -- 172.21.15.76:0/3886948272 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f753c011e70 con 0x7f754c101030 2024-03-12T05:52:53.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f755470a700 1 -- 172.21.15.76:0/3886948272 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754c19d640 con 0x7f754c101030 2024-03-12T05:52:53.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f75437fe700 1 -- 172.21.15.76:0/3886948272 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c00f340 con 0x7f754c101030 2024-03-12T05:52:53.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.636+0000 7f755470a700 1 -- 172.21.15.76:0/3886948272 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754c1a4ea0 con 0x7f754c101030 2024-03-12T05:52:53.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.637+0000 7f755470a700 1 -- 172.21.15.76:0/3886948272 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f754c10a2f0 con 0x7f754c101030 2024-03-12T05:52:53.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.638+0000 7f75437fe700 1 -- 172.21.15.76:0/3886948272 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f753c0106d0 con 0x7f754c101030 2024-03-12T05:52:53.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.639+0000 7f75437fe700 1 --2- 172.21.15.76:0/3886948272 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75380a1ac0 0x7f75380a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:53.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.639+0000 7f75524a6700 1 --2- 172.21.15.76:0/3886948272 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75380a1ac0 0x7f75380a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:53.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.639+0000 7f75437fe700 1 -- 172.21.15.76:0/3886948272 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f753c0cca90 con 0x7f754c101030 2024-03-12T05:52:53.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.639+0000 7f75524a6700 1 --2- 172.21.15.76:0/3886948272 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75380a1ac0 0x7f75380a3f80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7548003800 tx=0x7f7548005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:52:53.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.643+0000 7f75437fe700 1 -- 172.21.15.76:0/3886948272 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f753c096140 con 0x7f754c101030 2024-03-12T05:52:53.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.841+0000 7f755470a700 1 -- 172.21.15.76:0/3886948272 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f754c068930 con 0x7f754c101030 2024-03-12T05:52:53.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.842+0000 7f75437fe700 1 -- 172.21.15.76:0/3886948272 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18347 (secure 0 0 0) 0x7f753c099800 con 0x7f754c101030 2024-03-12T05:52:53.845 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:52:53.846 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":82,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:52:49.872922+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:52:53.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.844+0000 7f755470a700 1 -- 172.21.15.76:0/3886948272 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75380a1ac0 msgr2=0x7f75380a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:53.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.844+0000 7f755470a700 1 --2- 172.21.15.76:0/3886948272 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75380a1ac0 0x7f75380a3f80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7548003800 tx=0x7f7548005fb0 comp rx=0 tx=0).stop 2024-03-12T05:52:53.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.844+0000 7f755470a700 1 -- 172.21.15.76:0/3886948272 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f754c101030 msgr2=0x7f754c1986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:53.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.844+0000 7f755470a700 1 --2- 172.21.15.76:0/3886948272 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f754c101030 0x7f754c1986e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f753c00b530 tx=0x7f753c00b840 comp rx=0 tx=0).stop 2024-03-12T05:52:53.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.844+0000 7f755470a700 1 -- 172.21.15.76:0/3886948272 shutdown_connections 2024-03-12T05:52:53.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.844+0000 7f755470a700 1 --2- 172.21.15.76:0/3886948272 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75380a1ac0 0x7f75380a3f80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:53.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.844+0000 7f755470a700 1 --2- 172.21.15.76:0/3886948272 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f754c106d40 0x7f754c19ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:53.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.844+0000 7f755470a700 1 --2- 172.21.15.76:0/3886948272 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f754c100670 0x7f754c1981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:53.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.844+0000 7f755470a700 1 --2- 172.21.15.76:0/3886948272 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f754c101030 0x7f754c1986e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:53.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.844+0000 7f755470a700 1 -- 172.21.15.76:0/3886948272 >> 172.21.15.76:0/3886948272 conn(0x7f754c075ec0 msgr2=0x7f754c0fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:52:53.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.844+0000 7f755470a700 1 -- 172.21.15.76:0/3886948272 shutdown_connections 2024-03-12T05:52:53.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:53.844+0000 7f755470a700 1 -- 172.21.15.76:0/3886948272 wait complete. 2024-03-12T05:52:54.317 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:54 smithi076 ceph-mon[32095]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:54.317 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:54 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3886948272' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:52:54.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:54 smithi082 ceph-mon[36382]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:54.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:54 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3886948272' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:52:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:54 smithi088 ceph-mon[36585]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:54 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3886948272' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:52:54.627 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:49:09.125536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:52:54.627 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-12T05:52:54.865 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:52:56.223 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:56 smithi076 ceph-mon[32095]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:56 smithi082 ceph-mon[36382]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:56.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.478+0000 7f246bb2e700 1 -- 172.21.15.76:0/2266977788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2464101530 msgr2=0x7f246410d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:56.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.478+0000 7f246bb2e700 1 --2- 172.21.15.76:0/2266977788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2464101530 0x7f246410d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f245c00b0a0 tx=0x7f245c00b3b0 comp rx=0 tx=0).stop 2024-03-12T05:52:56.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.478+0000 7f246bb2e700 1 -- 172.21.15.76:0/2266977788 shutdown_connections 2024-03-12T05:52:56.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.478+0000 7f246bb2e700 1 --2- 172.21.15.76:0/2266977788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2464101530 0x7f246410d9c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:56.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.478+0000 7f246bb2e700 1 --2- 172.21.15.76:0/2266977788 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2464106b80 0x7f2464106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:56.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.478+0000 7f246bb2e700 1 --2- 172.21.15.76:0/2266977788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2464100b70 0x7f2464100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:56.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.478+0000 7f246bb2e700 1 -- 172.21.15.76:0/2266977788 >> 172.21.15.76:0/2266977788 conn(0x7f2464077aa0 msgr2=0x7f2464077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:52:56.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.479+0000 7f246bb2e700 1 -- 172.21.15.76:0/2266977788 shutdown_connections 2024-03-12T05:52:56.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.479+0000 7f246bb2e700 1 -- 172.21.15.76:0/2266977788 wait complete. 2024-03-12T05:52:56.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.479+0000 7f246bb2e700 1 Processor -- start 2024-03-12T05:52:56.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.479+0000 7f246bb2e700 1 -- start start 2024-03-12T05:52:56.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.479+0000 7f246bb2e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2464100b70 0x7f2464197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.479+0000 7f246bb2e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2464101530 0x7f24641976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.479+0000 7f246bb2e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2464106b80 0x7f246419b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.479+0000 7f246bb2e700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246419c000 con 0x7f2464106b80 2024-03-12T05:52:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.479+0000 7f246bb2e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246419c170 con 0x7f2464100b70 2024-03-12T05:52:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.479+0000 7f246bb2e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246419c330 con 0x7f2464101530 2024-03-12T05:52:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f246a0cb700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2464106b80 0x7f246419b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f24698ca700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2464100b70 0x7f2464197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f246a0cb700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2464106b80 0x7f246419b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47668/0 (socket says 172.21.15.76:47668) 2024-03-12T05:52:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f24698ca700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2464100b70 0x7f2464197170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:49616/0 (socket says 172.21.15.76:49616) 2024-03-12T05:52:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f24698ca700 1 -- 172.21.15.76:0/1382781397 learned_addr learned my addr 172.21.15.76:0/1382781397 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:52:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f24690c9700 1 --2- 172.21.15.76:0/1382781397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2464101530 0x7f24641976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f246a0cb700 1 -- 172.21.15.76:0/1382781397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2464101530 msgr2=0x7f24641976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:56.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f246a0cb700 1 --2- 172.21.15.76:0/1382781397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2464101530 0x7f24641976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:56.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f246a0cb700 1 -- 172.21.15.76:0/1382781397 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2464100b70 msgr2=0x7f2464197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:56.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f246a0cb700 1 --2- 172.21.15.76:0/1382781397 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2464100b70 0x7f2464197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:56.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f246a0cb700 1 -- 172.21.15.76:0/1382781397 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245c009d00 con 0x7f2464106b80 2024-03-12T05:52:56.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f246a0cb700 1 --2- 172.21.15.76:0/1382781397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2464106b80 0x7f246419b9f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f245c00b0a0 tx=0x7f245c004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:52:56.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f245affd700 1 -- 172.21.15.76:0/1382781397 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c00e040 con 0x7f2464106b80 2024-03-12T05:52:56.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f245affd700 1 -- 172.21.15.76:0/1382781397 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f245c007a50 con 0x7f2464106b80 2024-03-12T05:52:56.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f246bb2e700 1 -- 172.21.15.76:0/1382781397 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246419c5b0 con 0x7f2464106b80 2024-03-12T05:52:56.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f245affd700 1 -- 172.21.15.76:0/1382781397 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c01b8c0 con 0x7f2464106b80 2024-03-12T05:52:56.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.480+0000 7f246bb2e700 1 -- 172.21.15.76:0/1382781397 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2464069af0 con 0x7f2464106b80 2024-03-12T05:52:56.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.481+0000 7f246bb2e700 1 -- 172.21.15.76:0/1382781397 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24641021f0 con 0x7f2464106b80 2024-03-12T05:52:56.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:56 smithi088 ceph-mon[36585]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:56.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.482+0000 7f245affd700 1 -- 172.21.15.76:0/1382781397 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f245c01daf0 con 0x7f2464106b80 2024-03-12T05:52:56.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.483+0000 7f245affd700 1 --2- 172.21.15.76:0/1382781397 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f24500a1ac0 0x7f24500a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:56.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.483+0000 7f24698ca700 1 --2- 172.21.15.76:0/1382781397 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f24500a1ac0 0x7f24500a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:56.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.483+0000 7f245affd700 1 -- 172.21.15.76:0/1382781397 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f245c0cf370 con 0x7f2464106b80 2024-03-12T05:52:56.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.484+0000 7f24698ca700 1 --2- 172.21.15.76:0/1382781397 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f24500a1ac0 0x7f24500a3f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2460005950 tx=0x7f246000b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:52:56.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.487+0000 7f245affd700 1 -- 172.21.15.76:0/1382781397 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f245c098a10 con 0x7f2464106b80 2024-03-12T05:52:56.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.676+0000 7f246bb2e700 1 -- 172.21.15.76:0/1382781397 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2464110b80 con 0x7f2464106b80 2024-03-12T05:52:56.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.676+0000 7f245affd700 1 -- 172.21.15.76:0/1382781397 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f245c09c0d0 con 0x7f2464106b80 2024-03-12T05:52:56.679 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-12T05:52:56.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.678+0000 7f246bb2e700 1 -- 172.21.15.76:0/1382781397 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f24500a1ac0 msgr2=0x7f24500a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:56.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.678+0000 7f246bb2e700 1 --2- 172.21.15.76:0/1382781397 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f24500a1ac0 0x7f24500a3f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2460005950 tx=0x7f246000b500 comp rx=0 tx=0).stop 2024-03-12T05:52:56.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.678+0000 7f246bb2e700 1 -- 172.21.15.76:0/1382781397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2464106b80 msgr2=0x7f246419b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:56.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.678+0000 7f246bb2e700 1 --2- 172.21.15.76:0/1382781397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2464106b80 0x7f246419b9f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f245c00b0a0 tx=0x7f245c004e20 comp rx=0 tx=0).stop 2024-03-12T05:52:56.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.678+0000 7f246bb2e700 1 -- 172.21.15.76:0/1382781397 shutdown_connections 2024-03-12T05:52:56.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.678+0000 7f246bb2e700 1 --2- 172.21.15.76:0/1382781397 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f24500a1ac0 0x7f24500a3f80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:56.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.678+0000 7f246bb2e700 1 --2- 172.21.15.76:0/1382781397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2464106b80 0x7f246419b9f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:56.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.678+0000 7f246bb2e700 1 --2- 172.21.15.76:0/1382781397 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2464100b70 0x7f2464197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:56.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.678+0000 7f246bb2e700 1 --2- 172.21.15.76:0/1382781397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2464101530 0x7f24641976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:56.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.678+0000 7f246bb2e700 1 -- 172.21.15.76:0/1382781397 >> 172.21.15.76:0/1382781397 conn(0x7f2464077aa0 msgr2=0x7f24640ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:52:56.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.678+0000 7f246bb2e700 1 -- 172.21.15.76:0/1382781397 shutdown_connections 2024-03-12T05:52:56.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:52:56.678+0000 7f246bb2e700 1 -- 172.21.15.76:0/1382781397 wait complete. 2024-03-12T05:52:57.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:57 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1382781397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:52:57.334 INFO:tasks.cephadm:Adding mds.a on smithi076 2024-03-12T05:52:57.334 INFO:tasks.cephadm:Adding mds.d on smithi076 2024-03-12T05:52:57.334 INFO:tasks.cephadm:Adding mds.g on smithi076 2024-03-12T05:52:57.334 INFO:tasks.cephadm:Adding mds.j on smithi076 2024-03-12T05:52:57.335 INFO:tasks.cephadm:Adding mds.b on smithi082 2024-03-12T05:52:57.335 INFO:tasks.cephadm:Adding mds.e on smithi082 2024-03-12T05:52:57.335 INFO:tasks.cephadm:Adding mds.h on smithi082 2024-03-12T05:52:57.335 INFO:tasks.cephadm:Adding mds.k on smithi082 2024-03-12T05:52:57.335 INFO:tasks.cephadm:Adding mds.c on smithi088 2024-03-12T05:52:57.335 INFO:tasks.cephadm:Adding mds.f on smithi088 2024-03-12T05:52:57.335 INFO:tasks.cephadm:Adding mds.i on smithi088 2024-03-12T05:52:57.335 INFO:tasks.cephadm:Adding mds.l on smithi088 2024-03-12T05:52:57.335 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph orch apply mds all '12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi088=c;smithi088=f;smithi088=i;smithi088=l' 2024-03-12T05:52:57.366 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:57 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1382781397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:52:57.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:57 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1382781397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:52:57.585 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T05:52:58.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:58 smithi088 ceph-mon[36585]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:58.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:58 smithi082 ceph-mon[36382]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:58.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:58 smithi076 ceph-mon[32095]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:58.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.982+0000 7f4421a10700 1 -- 172.21.15.88:0/2941376129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f441c101810 msgr2=0x7f441c1090d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:58.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.982+0000 7f4421a10700 1 --2- 172.21.15.88:0/2941376129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f441c101810 0x7f441c1090d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f441400b0a0 tx=0x7f441400b3b0 comp rx=0 tx=0).stop 2024-03-12T05:52:58.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.982+0000 7f4421a10700 1 -- 172.21.15.88:0/2941376129 shutdown_connections 2024-03-12T05:52:58.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.982+0000 7f4421a10700 1 --2- 172.21.15.88:0/2941376129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f441c101810 0x7f441c1090d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:58.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.982+0000 7f4421a10700 1 --2- 172.21.15.88:0/2941376129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f441c102130 0x7f441c102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:58.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.982+0000 7f4421a10700 1 --2- 172.21.15.88:0/2941376129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f441c100e70 0x7f441c1012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:58.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.982+0000 7f4421a10700 1 -- 172.21.15.88:0/2941376129 >> 172.21.15.88:0/2941376129 conn(0x7f441c0fb8e0 msgr2=0x7f441c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:52:58.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.982+0000 7f4421a10700 1 -- 172.21.15.88:0/2941376129 shutdown_connections 2024-03-12T05:52:58.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.982+0000 7f4421a10700 1 -- 172.21.15.88:0/2941376129 wait complete. 2024-03-12T05:52:58.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f4421a10700 1 Processor -- start 2024-03-12T05:52:58.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f4421a10700 1 -- start start 2024-03-12T05:52:58.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f4421a10700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f441c100e70 0x7f441c197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:58.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f4421a10700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f441c101810 0x7f441c1982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:58.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f4421a10700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f441c102130 0x7f441c19f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:58.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f4421a10700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c19f940 con 0x7f441c102130 2024-03-12T05:52:58.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f4421a10700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c19fa80 con 0x7f441c100e70 2024-03-12T05:52:58.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f4421a10700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c19fc10 con 0x7f441c101810 2024-03-12T05:52:58.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f44127fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f441c101810 0x7f441c1982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:58.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f44127fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f441c101810 0x7f441c1982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50706/0 (socket says 172.21.15.88:50706) 2024-03-12T05:52:58.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f44127fc700 1 -- 172.21.15.88:0/798640164 learned_addr learned my addr 172.21.15.88:0/798640164 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-03-12T05:52:58.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f441affd700 1 --2- 172.21.15.88:0/798640164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f441c100e70 0x7f441c197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:58.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f441b7fe700 1 --2- 172.21.15.88:0/798640164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f441c102130 0x7f441c19f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:58.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f44127fc700 1 -- 172.21.15.88:0/798640164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f441c100e70 msgr2=0x7f441c197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:58.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f44127fc700 1 --2- 172.21.15.88:0/798640164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f441c100e70 0x7f441c197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:58.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f44127fc700 1 -- 172.21.15.88:0/798640164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f441c102130 msgr2=0x7f441c19f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:58.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f44127fc700 1 --2- 172.21.15.88:0/798640164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f441c102130 0x7f441c19f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:58.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.983+0000 7f44127fc700 1 -- 172.21.15.88:0/798640164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4414009d00 con 0x7f441c101810 2024-03-12T05:52:58.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.984+0000 7f44127fc700 1 --2- 172.21.15.88:0/798640164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f441c101810 0x7f441c1982f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f440400d980 tx=0x7f440400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:52:58.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.984+0000 7f4420a0e700 1 -- 172.21.15.88:0/798640164 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4404009b40 con 0x7f441c101810 2024-03-12T05:52:58.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.984+0000 7f4420a0e700 1 -- 172.21.15.88:0/798640164 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f440400b6e0 con 0x7f441c101810 2024-03-12T05:52:58.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.984+0000 7f4421a10700 1 -- 172.21.15.88:0/798640164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441c19fef0 con 0x7f441c101810 2024-03-12T05:52:58.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.984+0000 7f4420a0e700 1 -- 172.21.15.88:0/798640164 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440400f620 con 0x7f441c101810 2024-03-12T05:52:58.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.984+0000 7f4421a10700 1 -- 172.21.15.88:0/798640164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441c1a03b0 con 0x7f441c101810 2024-03-12T05:52:58.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.985+0000 7f4421a10700 1 -- 172.21.15.88:0/798640164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f441c068930 con 0x7f441c101810 2024-03-12T05:52:58.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.986+0000 7f4420a0e700 1 -- 172.21.15.88:0/798640164 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4404010460 con 0x7f441c101810 2024-03-12T05:52:58.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.987+0000 7f4420a0e700 1 --2- 172.21.15.88:0/798640164 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f44080a1b40 0x7f44080a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:52:58.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.987+0000 7f4420a0e700 1 -- 172.21.15.88:0/798640164 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f44040cccf0 con 0x7f441c101810 2024-03-12T05:52:58.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.987+0000 7f441affd700 1 --2- 172.21.15.88:0/798640164 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f44080a1b40 0x7f44080a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:52:58.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.987+0000 7f441affd700 1 --2- 172.21.15.88:0/798640164 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f44080a1b40 0x7f44080a4000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f441c0696a0 tx=0x7f440c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:52:58.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:58.990+0000 7f4420a0e700 1 -- 172.21.15.88:0/798640164 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f44040963a0 con 0x7f441c101810 2024-03-12T05:52:59.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:59.195+0000 7f4421a10700 1 -- 172.21.15.88:0/798640164 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi088=c;smithi088=f;smithi088=i;smithi088=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f441c100570 con 0x7f44080a1b40 2024-03-12T05:52:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:59.205+0000 7f4420a0e700 1 -- 172.21.15.88:0/798640164 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f441c100570 con 0x7f44080a1b40 2024-03-12T05:52:59.208 INFO:teuthology.orchestra.run.smithi088.stdout:Scheduled mds.all update... 2024-03-12T05:52:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:59.207+0000 7f4421a10700 1 -- 172.21.15.88:0/798640164 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f44080a1b40 msgr2=0x7f44080a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:59.207+0000 7f4421a10700 1 --2- 172.21.15.88:0/798640164 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f44080a1b40 0x7f44080a4000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f441c0696a0 tx=0x7f440c00b410 comp rx=0 tx=0).stop 2024-03-12T05:52:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:59.207+0000 7f4421a10700 1 -- 172.21.15.88:0/798640164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f441c101810 msgr2=0x7f441c1982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:52:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:59.207+0000 7f4421a10700 1 --2- 172.21.15.88:0/798640164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f441c101810 0x7f441c1982f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f440400d980 tx=0x7f440400dc90 comp rx=0 tx=0).stop 2024-03-12T05:52:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:59.207+0000 7f4421a10700 1 -- 172.21.15.88:0/798640164 shutdown_connections 2024-03-12T05:52:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:59.207+0000 7f4421a10700 1 --2- 172.21.15.88:0/798640164 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f44080a1b40 0x7f44080a4000 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:59.207+0000 7f4421a10700 1 --2- 172.21.15.88:0/798640164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f441c102130 0x7f441c19f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:59.207+0000 7f4421a10700 1 --2- 172.21.15.88:0/798640164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f441c100e70 0x7f441c197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:59.207+0000 7f4421a10700 1 --2- 172.21.15.88:0/798640164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f441c101810 0x7f441c1982f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:52:59.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:59.207+0000 7f4421a10700 1 -- 172.21.15.88:0/798640164 >> 172.21.15.88:0/798640164 conn(0x7f441c0fb8e0 msgr2=0x7f441c0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:52:59.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:59.207+0000 7f4421a10700 1 -- 172.21.15.88:0/798640164 shutdown_connections 2024-03-12T05:52:59.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-03-12T05:52:59.207+0000 7f4421a10700 1 -- 172.21.15.88:0/798640164 wait complete. 2024-03-12T05:52:59.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:52:59 smithi082 ceph-mon[36382]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:59.483 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:52:59 smithi088 ceph-mon[36585]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:59.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:52:59 smithi076 ceph-mon[32095]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:52:59.769 DEBUG:teuthology.orchestra.run.smithi076:mds.a> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.a.service 2024-03-12T05:52:59.772 DEBUG:teuthology.orchestra.run.smithi076:mds.d> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.d.service 2024-03-12T05:52:59.816 DEBUG:teuthology.orchestra.run.smithi076:mds.g> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.g.service 2024-03-12T05:52:59.859 DEBUG:teuthology.orchestra.run.smithi076:mds.j> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.j.service 2024-03-12T05:52:59.902 DEBUG:teuthology.orchestra.run.smithi082:mds.b> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.b.service 2024-03-12T05:52:59.904 DEBUG:teuthology.orchestra.run.smithi082:mds.e> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.e.service 2024-03-12T05:52:59.947 DEBUG:teuthology.orchestra.run.smithi082:mds.h> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.h.service 2024-03-12T05:52:59.990 DEBUG:teuthology.orchestra.run.smithi082:mds.k> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.k.service 2024-03-12T05:53:00.033 DEBUG:teuthology.orchestra.run.smithi088:mds.c> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.c.service 2024-03-12T05:53:00.035 DEBUG:teuthology.orchestra.run.smithi088:mds.f> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.f.service 2024-03-12T05:53:00.079 DEBUG:teuthology.orchestra.run.smithi088:mds.i> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.i.service 2024-03-12T05:53:00.122 DEBUG:teuthology.orchestra.run.smithi088:mds.l> sudo journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.l.service 2024-03-12T05:53:00.165 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-12T05:53:00.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:53:00.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 -- 172.21.15.76:0/2512419140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98e00ff230 msgr2=0x7f98e00ff6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:00.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 --2- 172.21.15.76:0/2512419140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98e00ff230 0x7f98e00ff6b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f98c8009900 tx=0x7f98c802bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:00.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 -- 172.21.15.76:0/2512419140 shutdown_connections 2024-03-12T05:53:00.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 --2- 172.21.15.76:0/2512419140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e00ffbf0 0x7f98e01097a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 --2- 172.21.15.76:0/2512419140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98e00ff230 0x7f98e00ff6b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 --2- 172.21.15.76:0/2512419140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98e01027c0 0x7f98e0102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 -- 172.21.15.76:0/2512419140 >> 172.21.15.76:0/2512419140 conn(0x7f98e00fadb0 msgr2=0x7f98e00fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:00.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 -- 172.21.15.76:0/2512419140 shutdown_connections 2024-03-12T05:53:00.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 -- 172.21.15.76:0/2512419140 wait complete. 2024-03-12T05:53:00.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 Processor -- start 2024-03-12T05:53:00.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 -- start start 2024-03-12T05:53:00.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98e00ff230 0x7f98e01972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:00.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98e00ffbf0 0x7f98e01977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:00.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e01027c0 0x7f98e019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:00.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e019ed90 con 0x7f98e00ff230 2024-03-12T05:53:00.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e019ef00 con 0x7f98e01027c0 2024-03-12T05:53:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.279+0000 7f98e530d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e019f0c0 con 0x7f98e00ffbf0 2024-03-12T05:53:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98deffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98e00ff230 0x7f98e01972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98deffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98e00ff230 0x7f98e01972a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47694/0 (socket says 172.21.15.76:47694) 2024-03-12T05:53:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98deffd640 1 -- 172.21.15.76:0/2668580423 learned_addr learned my addr 172.21.15.76:0/2668580423 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98d7fff640 1 --2- 172.21.15.76:0/2668580423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98e00ffbf0 0x7f98e01977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98df7fe640 1 --2- 172.21.15.76:0/2668580423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e01027c0 0x7f98e019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:00.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98deffd640 1 -- 172.21.15.76:0/2668580423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98e00ffbf0 msgr2=0x7f98e01977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:00.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98deffd640 1 --2- 172.21.15.76:0/2668580423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98e00ffbf0 0x7f98e01977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98deffd640 1 -- 172.21.15.76:0/2668580423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e01027c0 msgr2=0x7f98e019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:00.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98deffd640 1 --2- 172.21.15.76:0/2668580423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e01027c0 0x7f98e019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98deffd640 1 -- 172.21.15.76:0/2668580423 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d8009840 con 0x7f98e00ff230 2024-03-12T05:53:00.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98deffd640 1 --2- 172.21.15.76:0/2668580423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98e00ff230 0x7f98e01972a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f98d000e9c0 tx=0x7f98d000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:00.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98dcff9640 1 -- 172.21.15.76:0/2668580423 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d00096e0 con 0x7f98e00ff230 2024-03-12T05:53:00.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98e530d640 1 -- 172.21.15.76:0/2668580423 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c8009560 con 0x7f98e00ff230 2024-03-12T05:53:00.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98dcff9640 1 -- 172.21.15.76:0/2668580423 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98d000ccc0 con 0x7f98e00ff230 2024-03-12T05:53:00.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98dcff9640 1 -- 172.21.15.76:0/2668580423 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d0010430 con 0x7f98e00ff230 2024-03-12T05:53:00.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.280+0000 7f98e530d640 1 -- 172.21.15.76:0/2668580423 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e019f6f0 con 0x7f98e00ff230 2024-03-12T05:53:00.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.281+0000 7f98e530d640 1 -- 172.21.15.76:0/2668580423 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e0108680 con 0x7f98e00ff230 2024-03-12T05:53:00.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.282+0000 7f98dcff9640 1 -- 172.21.15.76:0/2668580423 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f98d0003f60 con 0x7f98e00ff230 2024-03-12T05:53:00.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.283+0000 7f98dcff9640 1 --2- 172.21.15.76:0/2668580423 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f98cc0a1a30 0x7f98cc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:00.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.283+0000 7f98d7fff640 1 --2- 172.21.15.76:0/2668580423 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f98cc0a1a30 0x7f98cc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:00.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.283+0000 7f98dcff9640 1 -- 172.21.15.76:0/2668580423 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f98d0014070 con 0x7f98e00ff230 2024-03-12T05:53:00.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.283+0000 7f98d7fff640 1 --2- 172.21.15.76:0/2668580423 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f98cc0a1a30 0x7f98cc0a3ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f98c8009230 tx=0x7f98c8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:00.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.286+0000 7f98dcff9640 1 -- 172.21.15.76:0/2668580423 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f98d0096ab0 con 0x7f98e00ff230 2024-03-12T05:53:00.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.460+0000 7f98e530d640 1 -- 172.21.15.76:0/2668580423 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98e002cda0 con 0x7f98e00ff230 2024-03-12T05:53:00.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.461+0000 7f98dcff9640 1 -- 172.21.15.76:0/2668580423 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18347 (secure 0 0 0) 0x7f98d009a170 con 0x7f98e00ff230 2024-03-12T05:53:00.465 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:00.465 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":82,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:52:49.872922+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:53:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.463+0000 7f98e530d640 1 -- 172.21.15.76:0/2668580423 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f98cc0a1a30 msgr2=0x7f98cc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.463+0000 7f98e530d640 1 --2- 172.21.15.76:0/2668580423 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f98cc0a1a30 0x7f98cc0a3ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f98c8009230 tx=0x7f98c8005fd0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.463+0000 7f98e530d640 1 -- 172.21.15.76:0/2668580423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98e00ff230 msgr2=0x7f98e01972a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.463+0000 7f98e530d640 1 --2- 172.21.15.76:0/2668580423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98e00ff230 0x7f98e01972a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f98d000e9c0 tx=0x7f98d000edc0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.463+0000 7f98e530d640 1 -- 172.21.15.76:0/2668580423 shutdown_connections 2024-03-12T05:53:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.463+0000 7f98e530d640 1 --2- 172.21.15.76:0/2668580423 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f98cc0a1a30 0x7f98cc0a3ef0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.463+0000 7f98e530d640 1 --2- 172.21.15.76:0/2668580423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f98e01027c0 0x7f98e019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.463+0000 7f98e530d640 1 --2- 172.21.15.76:0/2668580423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98e00ffbf0 0x7f98e01977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.463+0000 7f98e530d640 1 --2- 172.21.15.76:0/2668580423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98e00ff230 0x7f98e01972a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.463+0000 7f98e530d640 1 -- 172.21.15.76:0/2668580423 >> 172.21.15.76:0/2668580423 conn(0x7f98e00fadb0 msgr2=0x7f98e0105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.463+0000 7f98e530d640 1 -- 172.21.15.76:0/2668580423 shutdown_connections 2024-03-12T05:53:00.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.463+0000 7f98e530d640 1 -- 172.21.15.76:0/2668580423 wait complete. 2024-03-12T05:53:00.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:00 smithi076 ceph-mon[32095]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi088=c;smithi088=f;smithi088=i;smithi088=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:00.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:00 smithi076 ceph-mon[32095]: Saving service mds.all spec with placement smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi088=c;smithi088=f;smithi088=i;smithi088=l;count:12 2024-03-12T05:53:00.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:00 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:00.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:00 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:53:00.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:00 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:00.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:00 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:53:00.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:00 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:00.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:00 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:00.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:00 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:00.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:00 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:00.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:00 smithi076 ceph-mon[32095]: Deploying daemon mds.l on smithi088 2024-03-12T05:53:00.479 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:49:09.125536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:53:00.479 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:53:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:00 smithi088 ceph-mon[36585]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi088=c;smithi088=f;smithi088=i;smithi088=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:00 smithi088 ceph-mon[36585]: Saving service mds.all spec with placement smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi088=c;smithi088=f;smithi088=i;smithi088=l;count:12 2024-03-12T05:53:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:00 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:00 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:53:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:00 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:00 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:53:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:00 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:00.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:00 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:00.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:00 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:00.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:00 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:00.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:00 smithi088 ceph-mon[36585]: Deploying daemon mds.l on smithi088 2024-03-12T05:53:00.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.580+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1954331287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9381000a0 msgr2=0x7fb9381004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:00.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.580+0000 7fb93fc4d640 1 --2- 172.21.15.76:0/1954331287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9381000a0 0x7fb9381004c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb930009900 tx=0x7fb93002bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:00.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.580+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1954331287 shutdown_connections 2024-03-12T05:53:00.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.580+0000 7fb93fc4d640 1 --2- 172.21.15.76:0/1954331287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb938101c50 0x7fb9381084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.580+0000 7fb93fc4d640 1 --2- 172.21.15.76:0/1954331287 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb938101290 0x7fb938101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.580+0000 7fb93fc4d640 1 --2- 172.21.15.76:0/1954331287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9381000a0 0x7fb9381004c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.580+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1954331287 >> 172.21.15.76:0/1954331287 conn(0x7fb9380fb610 msgr2=0x7fb9380fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:00.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.580+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1954331287 shutdown_connections 2024-03-12T05:53:00.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.580+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1954331287 wait complete. 2024-03-12T05:53:00.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.580+0000 7fb93fc4d640 1 Processor -- start 2024-03-12T05:53:00.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.580+0000 7fb93fc4d640 1 -- start start 2024-03-12T05:53:00.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93fc4d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9381000a0 0x7fb938196150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:00.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93fc4d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb938101290 0x7fb938196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93fc4d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb938101c50 0x7fb93819d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93fc4d640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93819dc40 con 0x7fb9381000a0 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93fc4d640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93819ddb0 con 0x7fb938101290 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93fc4d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93819df70 con 0x7fb938101c50 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93d9e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9381000a0 0x7fb938196150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93d9e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9381000a0 0x7fb938196150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47706/0 (socket says 172.21.15.76:47706) 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93d9e9640 1 -- 172.21.15.76:0/1026090047 learned_addr learned my addr 172.21.15.76:0/1026090047 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93d1e8640 1 --2- 172.21.15.76:0/1026090047 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb938101290 0x7fb938196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93e1ea640 1 --2- 172.21.15.76:0/1026090047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb938101c50 0x7fb93819d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93d9e9640 1 -- 172.21.15.76:0/1026090047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb938101c50 msgr2=0x7fb93819d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93d9e9640 1 --2- 172.21.15.76:0/1026090047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb938101c50 0x7fb93819d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93d9e9640 1 -- 172.21.15.76:0/1026090047 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb938101290 msgr2=0x7fb938196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93d9e9640 1 --2- 172.21.15.76:0/1026090047 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb938101290 0x7fb938196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93d9e9640 1 -- 172.21.15.76:0/1026090047 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb930009560 con 0x7fb9381000a0 2024-03-12T05:53:00.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93d9e9640 1 --2- 172.21.15.76:0/1026090047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9381000a0 0x7fb938196150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb930005ab0 tx=0x7fb930005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:00.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb92effd640 1 -- 172.21.15.76:0/1026090047 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb930004df0 con 0x7fb9381000a0 2024-03-12T05:53:00.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb92effd640 1 -- 172.21.15.76:0/1026090047 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb930040070 con 0x7fb9381000a0 2024-03-12T05:53:00.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1026090047 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb938068450 con 0x7fb9381000a0 2024-03-12T05:53:00.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb92effd640 1 -- 172.21.15.76:0/1026090047 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb930003e20 con 0x7fb9381000a0 2024-03-12T05:53:00.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.581+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1026090047 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb938068960 con 0x7fb9381000a0 2024-03-12T05:53:00.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.582+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1026090047 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb938068000 con 0x7fb9381000a0 2024-03-12T05:53:00.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.583+0000 7fb92effd640 1 -- 172.21.15.76:0/1026090047 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb9300035f0 con 0x7fb9381000a0 2024-03-12T05:53:00.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.584+0000 7fb92effd640 1 --2- 172.21.15.76:0/1026090047 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb9240a1a30 0x7fb9240a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:00.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.584+0000 7fb93d1e8640 1 --2- 172.21.15.76:0/1026090047 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb9240a1a30 0x7fb9240a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:00.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.584+0000 7fb92effd640 1 -- 172.21.15.76:0/1026090047 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb930042070 con 0x7fb9381000a0 2024-03-12T05:53:00.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.584+0000 7fb93d1e8640 1 --2- 172.21.15.76:0/1026090047 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb9240a1a30 0x7fb9240a3ef0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb938197660 tx=0x7fb928005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:00.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.588+0000 7fb92effd640 1 -- 172.21.15.76:0/1026090047 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb9300ba950 con 0x7fb9381000a0 2024-03-12T05:53:00.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:00 smithi082 ceph-mon[36382]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi088=c;smithi088=f;smithi088=i;smithi088=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:00.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:00 smithi082 ceph-mon[36382]: Saving service mds.all spec with placement smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi088=c;smithi088=f;smithi088=i;smithi088=l;count:12 2024-03-12T05:53:00.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:00 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:00.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:00 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:53:00.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:00 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:00.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:00 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:53:00.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:00 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:00.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:00 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:00.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:00 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:00.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:00 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:00.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:00 smithi082 ceph-mon[36382]: Deploying daemon mds.l on smithi088 2024-03-12T05:53:00.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.760+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1026090047 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb93810bab0 con 0x7fb9381000a0 2024-03-12T05:53:00.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.760+0000 7fb92effd640 1 -- 172.21.15.76:0/1026090047 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb9300be010 con 0x7fb9381000a0 2024-03-12T05:53:00.763 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-12T05:53:00.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.762+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1026090047 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb9240a1a30 msgr2=0x7fb9240a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:00.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.762+0000 7fb93fc4d640 1 --2- 172.21.15.76:0/1026090047 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb9240a1a30 0x7fb9240a3ef0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb938197660 tx=0x7fb928005f90 comp rx=0 tx=0).stop 2024-03-12T05:53:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.762+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1026090047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9381000a0 msgr2=0x7fb938196150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.762+0000 7fb93fc4d640 1 --2- 172.21.15.76:0/1026090047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9381000a0 0x7fb938196150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb930005ab0 tx=0x7fb930005e50 comp rx=0 tx=0).stop 2024-03-12T05:53:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.762+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1026090047 shutdown_connections 2024-03-12T05:53:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.762+0000 7fb93fc4d640 1 --2- 172.21.15.76:0/1026090047 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb9240a1a30 0x7fb9240a3ef0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.762+0000 7fb93fc4d640 1 --2- 172.21.15.76:0/1026090047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb938101c50 0x7fb93819d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.762+0000 7fb93fc4d640 1 --2- 172.21.15.76:0/1026090047 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb938101290 0x7fb938196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.762+0000 7fb93fc4d640 1 --2- 172.21.15.76:0/1026090047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9381000a0 0x7fb938196150 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.762+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1026090047 >> 172.21.15.76:0/1026090047 conn(0x7fb9380fb610 msgr2=0x7fb938104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.762+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1026090047 shutdown_connections 2024-03-12T05:53:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.762+0000 7fb93fc4d640 1 -- 172.21.15.76:0/1026090047 wait complete. 2024-03-12T05:53:00.776 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:00.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.874+0000 7f0d917a0640 1 -- 172.21.15.76:0/316407856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d8c102270 msgr2=0x7f0d8c1026f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:00.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.874+0000 7f0d917a0640 1 --2- 172.21.15.76:0/316407856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d8c102270 0x7f0d8c1026f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009900 tx=0x7f0d8002bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:00.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.874+0000 7f0d917a0640 1 -- 172.21.15.76:0/316407856 shutdown_connections 2024-03-12T05:53:00.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.874+0000 7f0d917a0640 1 --2- 172.21.15.76:0/316407856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c102c30 0x7f0d8c1094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.874+0000 7f0d917a0640 1 --2- 172.21.15.76:0/316407856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d8c102270 0x7f0d8c1026f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.874+0000 7f0d917a0640 1 --2- 172.21.15.76:0/316407856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d8c101080 0x7f0d8c1014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.874+0000 7f0d917a0640 1 -- 172.21.15.76:0/316407856 >> 172.21.15.76:0/316407856 conn(0x7f0d8c0fc610 msgr2=0x7f0d8c0fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:00.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.874+0000 7f0d917a0640 1 -- 172.21.15.76:0/316407856 shutdown_connections 2024-03-12T05:53:00.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.874+0000 7f0d917a0640 1 -- 172.21.15.76:0/316407856 wait complete. 2024-03-12T05:53:00.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.875+0000 7f0d917a0640 1 Processor -- start 2024-03-12T05:53:00.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.875+0000 7f0d917a0640 1 -- start start 2024-03-12T05:53:00.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.875+0000 7f0d917a0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d8c101080 0x7f0d8c1970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:00.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.875+0000 7f0d917a0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c102270 0x7f0d8c197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:00.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.875+0000 7f0d917a0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d8c102c30 0x7f0d8c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:00.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.875+0000 7f0d917a0640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c19ebe0 con 0x7f0d8c102c30 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.875+0000 7f0d917a0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c19ed50 con 0x7f0d8c101080 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.875+0000 7f0d917a0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c19ef10 con 0x7f0d8c102270 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.875+0000 7f0d90f9f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d8c102c30 0x7f0d8c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.875+0000 7f0d90f9f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d8c102c30 0x7f0d8c19e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47726/0 (socket says 172.21.15.76:47726) 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.875+0000 7f0d90f9f640 1 -- 172.21.15.76:0/2528601390 learned_addr learned my addr 172.21.15.76:0/2528601390 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.875+0000 7f0d8bfff640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d8c101080 0x7f0d8c1970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.876+0000 7f0d90f9f640 1 -- 172.21.15.76:0/2528601390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c102270 msgr2=0x7f0d8c197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.876+0000 7f0d8b7fe640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c102270 0x7f0d8c197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.876+0000 7f0d90f9f640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c102270 0x7f0d8c197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.876+0000 7f0d90f9f640 1 -- 172.21.15.76:0/2528601390 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d8c101080 msgr2=0x7f0d8c1970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.876+0000 7f0d90f9f640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d8c101080 0x7f0d8c1970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.876+0000 7f0d90f9f640 1 -- 172.21.15.76:0/2528601390 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d80009560 con 0x7f0d8c102c30 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.876+0000 7f0d8bfff640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d8c101080 0x7f0d8c1970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.876+0000 7f0d8b7fe640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c102270 0x7f0d8c197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:00.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.876+0000 7f0d90f9f640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d8c102c30 0x7f0d8c19e6a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400b970 tx=0x7f0d8400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:00.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.876+0000 7f0d897fa640 1 -- 172.21.15.76:0/2528601390 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8400cb40 con 0x7f0d8c102c30 2024-03-12T05:53:00.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.876+0000 7f0d917a0640 1 -- 172.21.15.76:0/2528601390 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d8c19f1f0 con 0x7f0d8c102c30 2024-03-12T05:53:00.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.876+0000 7f0d897fa640 1 -- 172.21.15.76:0/2528601390 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d8400cca0 con 0x7f0d8c102c30 2024-03-12T05:53:00.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.876+0000 7f0d897fa640 1 -- 172.21.15.76:0/2528601390 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d840077d0 con 0x7f0d8c102c30 2024-03-12T05:53:00.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.876+0000 7f0d917a0640 1 -- 172.21.15.76:0/2528601390 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d8c19f7d0 con 0x7f0d8c102c30 2024-03-12T05:53:00.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.877+0000 7f0d917a0640 1 -- 172.21.15.76:0/2528601390 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8c108340 con 0x7f0d8c102c30 2024-03-12T05:53:00.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.878+0000 7f0d897fa640 1 -- 172.21.15.76:0/2528601390 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0d84014440 con 0x7f0d8c102c30 2024-03-12T05:53:00.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.879+0000 7f0d897fa640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0d740a1a20 0x7f0d740a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:00.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.879+0000 7f0d8bfff640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0d740a1a20 0x7f0d740a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:00.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.879+0000 7f0d897fa640 1 -- 172.21.15.76:0/2528601390 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0d840cc7a0 con 0x7f0d8c102c30 2024-03-12T05:53:00.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.879+0000 7f0d8bfff640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0d740a1a20 0x7f0d740a3ee0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c1020d0 tx=0x7f0d7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:00.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:00.883+0000 7f0d897fa640 1 -- 172.21.15.76:0/2528601390 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0d84095f00 con 0x7f0d8c102c30 2024-03-12T05:53:01.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:01.094+0000 7f0d917a0640 1 -- 172.21.15.76:0/2528601390 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d8c19f380 con 0x7f0d8c102c30 2024-03-12T05:53:01.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:01.094+0000 7f0d897fa640 1 -- 172.21.15.76:0/2528601390 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f0d8c19f380 con 0x7f0d8c102c30 2024-03-12T05:53:01.099 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:01.099 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-12T05:53:01.099 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 1 2024-03-12T05:53:01.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:01.096+0000 7f0d917a0640 1 -- 172.21.15.76:0/2528601390 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0d740a1a20 msgr2=0x7f0d740a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:01.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:01.096+0000 7f0d917a0640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0d740a1a20 0x7f0d740a3ee0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c1020d0 tx=0x7f0d7c005f90 comp rx=0 tx=0).stop 2024-03-12T05:53:01.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:01.096+0000 7f0d917a0640 1 -- 172.21.15.76:0/2528601390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d8c102c30 msgr2=0x7f0d8c19e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:01.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:01.096+0000 7f0d917a0640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d8c102c30 0x7f0d8c19e6a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400b970 tx=0x7f0d8400be30 comp rx=0 tx=0).stop 2024-03-12T05:53:01.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:01.096+0000 7f0d917a0640 1 -- 172.21.15.76:0/2528601390 shutdown_connections 2024-03-12T05:53:01.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:01.096+0000 7f0d917a0640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0d740a1a20 0x7f0d740a3ee0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:01.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:01.096+0000 7f0d917a0640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d8c102c30 0x7f0d8c19e6a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:01.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:01.096+0000 7f0d917a0640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d8c102270 0x7f0d8c197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:01.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:01.096+0000 7f0d917a0640 1 --2- 172.21.15.76:0/2528601390 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0d8c101080 0x7f0d8c1970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:01.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:01.096+0000 7f0d917a0640 1 -- 172.21.15.76:0/2528601390 >> 172.21.15.76:0/2528601390 conn(0x7f0d8c0fc610 msgr2=0x7f0d8c105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:01.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:01.097+0000 7f0d917a0640 1 -- 172.21.15.76:0/2528601390 shutdown_connections 2024-03-12T05:53:01.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:01.097+0000 7f0d917a0640 1 -- 172.21.15.76:0/2528601390 wait complete. 2024-03-12T05:53:01.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:01 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2668580423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:01.467 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:01 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1026090047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:53:01.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:01 smithi088 ceph-mon[36585]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:01.468 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:01 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2528601390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:01.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:01 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2668580423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:01.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:01 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1026090047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:53:01.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:01 smithi076 ceph-mon[32095]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:01.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:01 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2528601390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:01.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:01 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2668580423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:01.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:01 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1026090047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:53:01.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:01 smithi082 ceph-mon[36382]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:01.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:01 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2528601390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:02.137 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 05:53:01 smithi088 systemd[1]: Starting Ceph mds.l for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T05:53:02.405 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 05:53:02 smithi088 podman[67373]: 2024-03-12 05:53:02.13418498 +0000 UTC m=+0.105214151 container create 5caf516650fad2c35100cf448d6e82f59b26ca35738d393a6fcd1529dc22325c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-l, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-12T05:53:02.405 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 05:53:02 smithi088 podman[67373]: 2024-03-12 05:53:02.04675812 +0000 UTC m=+0.017787244 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:53:02.406 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 05:53:02 smithi088 podman[67373]: 2024-03-12 05:53:02.344877973 +0000 UTC m=+0.315907111 container init 5caf516650fad2c35100cf448d6e82f59b26ca35738d393a6fcd1529dc22325c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-l, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:53:02.406 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 05:53:02 smithi088 podman[67373]: 2024-03-12 05:53:02.347497517 +0000 UTC m=+0.318526648 container start 5caf516650fad2c35100cf448d6e82f59b26ca35738d393a6fcd1529dc22325c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-l, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-12T05:53:02.406 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 05:53:02 smithi088 ceph-mds[67392]: -- 172.21.15.88:0/723857661 <== mon.1 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e46db521a0 con 0x55e46db28c00 2024-03-12T05:53:02.406 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 05:53:02 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-l[67388]: starting mds.l at 2024-03-12T05:53:02.406 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 05:53:02 smithi088 bash[67373]: 5caf516650fad2c35100cf448d6e82f59b26ca35738d393a6fcd1529dc22325c 2024-03-12T05:53:02.734 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 05:53:02 smithi088 systemd[1]: Started Ceph mds.l for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:53:03.112 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:03.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 -- 172.21.15.76:0/102329415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47d0106ef0 msgr2=0x7f47d01092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:03.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 --2- 172.21.15.76:0/102329415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47d0106ef0 0x7f47d01092e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009ca0 tx=0x7f47cc02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:53:03.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 -- 172.21.15.76:0/102329415 shutdown_connections 2024-03-12T05:53:03.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 --2- 172.21.15.76:0/102329415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47d0106ef0 0x7f47d01092e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:03.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 --2- 172.21.15.76:0/102329415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d01045c0 0x7f47d01069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:03.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 --2- 172.21.15.76:0/102329415 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47d0101c90 0x7f47d0104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:03.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 -- 172.21.15.76:0/102329415 >> 172.21.15.76:0/102329415 conn(0x7f47d00fb620 msgr2=0x7f47d00fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:03.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 -- 172.21.15.76:0/102329415 shutdown_connections 2024-03-12T05:53:03.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 -- 172.21.15.76:0/102329415 wait complete. 2024-03-12T05:53:03.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 Processor -- start 2024-03-12T05:53:03.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 -- start start 2024-03-12T05:53:03.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47d0101c90 0x7f47d01971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:03.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d01045c0 0x7f47d0197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:03.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47d0106ef0 0x7f47d019e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:03.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d019ecd0 con 0x7f47d0101c90 2024-03-12T05:53:03.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.222+0000 7f47d75e2640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d019ee40 con 0x7f47d0106ef0 2024-03-12T05:53:03.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d75e2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d019f000 con 0x7f47d01045c0 2024-03-12T05:53:03.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d537e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47d0101c90 0x7f47d01971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:03.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d537e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47d0101c90 0x7f47d01971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40040/0 (socket says 172.21.15.76:40040) 2024-03-12T05:53:03.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d537e640 1 -- 172.21.15.76:0/3941703970 learned_addr learned my addr 172.21.15.76:0/3941703970 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:03.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d5b7f640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47d0106ef0 0x7f47d019e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:03.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d4b7d640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d01045c0 0x7f47d0197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:03.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d537e640 1 -- 172.21.15.76:0/3941703970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d01045c0 msgr2=0x7f47d0197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:03.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d537e640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d01045c0 0x7f47d0197720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:03.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d537e640 1 -- 172.21.15.76:0/3941703970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47d0106ef0 msgr2=0x7f47d019e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:03.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d537e640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47d0106ef0 0x7f47d019e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:03.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d537e640 1 -- 172.21.15.76:0/3941703970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc009900 con 0x7f47d0101c90 2024-03-12T05:53:03.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d4b7d640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d01045c0 0x7f47d0197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:53:03.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d5b7f640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47d0106ef0 0x7f47d019e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:53:03.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d537e640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47d0101c90 0x7f47d01971e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f47c800e9f0 tx=0x7f47c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:03.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47c67fc640 1 -- 172.21.15.76:0/3941703970 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c8009700 con 0x7f47d0101c90 2024-03-12T05:53:03.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d75e2640 1 -- 172.21.15.76:0/3941703970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d019f2e0 con 0x7f47d0101c90 2024-03-12T05:53:03.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47c67fc640 1 -- 172.21.15.76:0/3941703970 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47c800cd80 con 0x7f47d0101c90 2024-03-12T05:53:03.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47c67fc640 1 -- 172.21.15.76:0/3941703970 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c8018920 con 0x7f47d0101c90 2024-03-12T05:53:03.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.223+0000 7f47d75e2640 1 -- 172.21.15.76:0/3941703970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d019f7f0 con 0x7f47d0101c90 2024-03-12T05:53:03.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.224+0000 7f47d75e2640 1 -- 172.21.15.76:0/3941703970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d00fd110 con 0x7f47d0101c90 2024-03-12T05:53:03.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.225+0000 7f47c67fc640 1 -- 172.21.15.76:0/3941703970 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f47c8018a80 con 0x7f47d0101c90 2024-03-12T05:53:03.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.226+0000 7f47c67fc640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f47bc0a1a30 0x7f47bc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:03.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.226+0000 7f47d4b7d640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f47bc0a1a30 0x7f47bc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:03.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.226+0000 7f47c67fc640 1 -- 172.21.15.76:0/3941703970 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f47c8014070 con 0x7f47d0101c90 2024-03-12T05:53:03.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.226+0000 7f47d4b7d640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f47bc0a1a30 0x7f47bc0a3ef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f47c00058c0 tx=0x7f47c0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:03.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.229+0000 7f47c67fc640 1 -- 172.21.15.76:0/3941703970 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f47c8096ed0 con 0x7f47d0101c90 2024-03-12T05:53:03.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.440+0000 7f47d75e2640 1 -- 172.21.15.76:0/3941703970 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47d010cbc0 con 0x7f47d0101c90 2024-03-12T05:53:03.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.440+0000 7f47c67fc640 1 -- 172.21.15.76:0/3941703970 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1325 (secure 0 0 0) 0x7f47c809a590 con 0x7f47d0101c90 2024-03-12T05:53:03.445 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:03.445 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-12T05:53:03.445 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 2 2024-03-12T05:53:03.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.442+0000 7f47d75e2640 1 -- 172.21.15.76:0/3941703970 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f47bc0a1a30 msgr2=0x7f47bc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:03.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.442+0000 7f47d75e2640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f47bc0a1a30 0x7f47bc0a3ef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f47c00058c0 tx=0x7f47c0005850 comp rx=0 tx=0).stop 2024-03-12T05:53:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.442+0000 7f47d75e2640 1 -- 172.21.15.76:0/3941703970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47d0101c90 msgr2=0x7f47d01971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.442+0000 7f47d75e2640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47d0101c90 0x7f47d01971e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f47c800e9f0 tx=0x7f47c800eeb0 comp rx=0 tx=0).stop 2024-03-12T05:53:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.442+0000 7f47d75e2640 1 -- 172.21.15.76:0/3941703970 shutdown_connections 2024-03-12T05:53:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.442+0000 7f47d75e2640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f47bc0a1a30 0x7f47bc0a3ef0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.442+0000 7f47d75e2640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47d0106ef0 0x7f47d019e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.442+0000 7f47d75e2640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47d01045c0 0x7f47d0197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.442+0000 7f47d75e2640 1 --2- 172.21.15.76:0/3941703970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47d0101c90 0x7f47d01971e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.442+0000 7f47d75e2640 1 -- 172.21.15.76:0/3941703970 >> 172.21.15.76:0/3941703970 conn(0x7f47d00fb620 msgr2=0x7f47d01025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.442+0000 7f47d75e2640 1 -- 172.21.15.76:0/3941703970 shutdown_connections 2024-03-12T05:53:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:03.442+0000 7f47d75e2640 1 -- 172.21.15.76:0/3941703970 wait complete. 2024-03-12T05:53:03.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:03 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.88:6832/3297889286,v1:172.21.15.88:6833/3297889286] up:boot 2024-03-12T05:53:03.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:03 smithi082 ceph-mon[36382]: fsmap 1 up:standby 2024-03-12T05:53:03.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:03 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:53:03.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:03 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:03.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:03 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:03.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:03 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:03.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:03 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:03.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:03 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:03.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:03 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:03.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:03 smithi082 ceph-mon[36382]: Deploying daemon mds.k on smithi082 2024-03-12T05:53:03.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:03 smithi082 ceph-mon[36382]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:03.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:03 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.88:6832/3297889286,v1:172.21.15.88:6833/3297889286] up:boot 2024-03-12T05:53:03.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:03 smithi088 ceph-mon[36585]: fsmap 1 up:standby 2024-03-12T05:53:03.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:03 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:53:03.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:03 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:03.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:03 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:03.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:03 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:03.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:03 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:03.739 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:03 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:03.739 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:03 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:03.739 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:03 smithi088 ceph-mon[36585]: Deploying daemon mds.k on smithi082 2024-03-12T05:53:03.739 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:03 smithi088 ceph-mon[36585]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:03.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:03 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.88:6832/3297889286,v1:172.21.15.88:6833/3297889286] up:boot 2024-03-12T05:53:03.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:03 smithi076 ceph-mon[32095]: fsmap 1 up:standby 2024-03-12T05:53:03.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:53:03.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:03.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:03.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:03.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:03.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:03.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:03 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:03.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:03 smithi076 ceph-mon[32095]: Deploying daemon mds.k on smithi082 2024-03-12T05:53:03.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:03 smithi076 ceph-mon[32095]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:04.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:04 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3941703970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:04.695 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 05:53:04 smithi082 systemd[1]: Starting Ceph mds.k for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T05:53:04.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:04 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3941703970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:04.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:04 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3941703970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:04.972 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 05:53:04 smithi082 podman[65580]: 2024-03-12 05:53:04.612948559 +0000 UTC m=+0.017704102 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:53:04.973 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 05:53:04 smithi082 podman[65580]: 2024-03-12 05:53:04.711732824 +0000 UTC m=+0.116488423 container create 3120ec94420a57965e107cddfc53d37c427cc42b93c71f906cde1836570598d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-k, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-12T05:53:04.973 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 05:53:04 smithi082 podman[65580]: 2024-03-12 05:53:04.970515867 +0000 UTC m=+0.375271419 container init 3120ec94420a57965e107cddfc53d37c427cc42b93c71f906cde1836570598d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-k, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:53:05.403 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 05:53:04 smithi082 podman[65580]: 2024-03-12 05:53:04.973404293 +0000 UTC m=+0.378159847 container start 3120ec94420a57965e107cddfc53d37c427cc42b93c71f906cde1836570598d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-k, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-12T05:53:05.403 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 05:53:04 smithi082 ceph-mds[65599]: -- 172.21.15.82:0/1445497307 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d0fec6a1a0 con 0x55d0fec41000 2024-03-12T05:53:05.403 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 05:53:04 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-k[65595]: starting mds.k at 2024-03-12T05:53:05.404 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 05:53:05 smithi082 bash[65580]: 3120ec94420a57965e107cddfc53d37c427cc42b93c71f906cde1836570598d4 2024-03-12T05:53:05.404 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 05:53:05 smithi082 systemd[1]: Started Ceph mds.k for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:53:05.458 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:05.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.559+0000 7ff3aa288640 1 -- 172.21.15.76:0/3490669243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a41012a0 msgr2=0x7ff3a4101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:05.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.559+0000 7ff3aa288640 1 --2- 172.21.15.76:0/3490669243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a41012a0 0x7ff3a4101720 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff394009900 tx=0x7ff39402bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:05.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.559+0000 7ff3aa288640 1 -- 172.21.15.76:0/3490669243 shutdown_connections 2024-03-12T05:53:05.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.559+0000 7ff3aa288640 1 --2- 172.21.15.76:0/3490669243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3a4101c60 0x7ff3a41084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:05.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.559+0000 7ff3aa288640 1 --2- 172.21.15.76:0/3490669243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a41012a0 0x7ff3a4101720 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:05.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.559+0000 7ff3aa288640 1 --2- 172.21.15.76:0/3490669243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3a41000b0 0x7ff3a41004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:05.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.559+0000 7ff3aa288640 1 -- 172.21.15.76:0/3490669243 >> 172.21.15.76:0/3490669243 conn(0x7ff3a40fb620 msgr2=0x7ff3a40fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:05.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.559+0000 7ff3aa288640 1 -- 172.21.15.76:0/3490669243 shutdown_connections 2024-03-12T05:53:05.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.559+0000 7ff3aa288640 1 -- 172.21.15.76:0/3490669243 wait complete. 2024-03-12T05:53:05.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.559+0000 7ff3aa288640 1 Processor -- start 2024-03-12T05:53:05.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3aa288640 1 -- start start 2024-03-12T05:53:05.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3aa288640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3a41000b0 0x7ff3a4197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:05.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3aa288640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3a41012a0 0x7ff3a4197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:05.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3aa288640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4101c60 0x7ff3a419e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:05.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3aa288640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a419ed40 con 0x7ff3a4101c60 2024-03-12T05:53:05.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3aa288640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a419eeb0 con 0x7ff3a41000b0 2024-03-12T05:53:05.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3aa288640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a419f070 con 0x7ff3a41012a0 2024-03-12T05:53:05.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3a8825640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4101c60 0x7ff3a419e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:05.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3a8825640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4101c60 0x7ff3a419e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40062/0 (socket says 172.21.15.76:40062) 2024-03-12T05:53:05.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3a8825640 1 -- 172.21.15.76:0/3793512185 learned_addr learned my addr 172.21.15.76:0/3793512185 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:05.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3a3fff640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3a41000b0 0x7ff3a4197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:05.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3a37fe640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3a41012a0 0x7ff3a4197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:05.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3a8825640 1 -- 172.21.15.76:0/3793512185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3a41012a0 msgr2=0x7ff3a4197790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:05.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3a8825640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3a41012a0 0x7ff3a4197790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:05.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3a8825640 1 -- 172.21.15.76:0/3793512185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3a41000b0 msgr2=0x7ff3a4197250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:05.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3a8825640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3a41000b0 0x7ff3a4197250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:05.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3a8825640 1 -- 172.21.15.76:0/3793512185 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff394009560 con 0x7ff3a4101c60 2024-03-12T05:53:05.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3a3fff640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3a41000b0 0x7ff3a4197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:53:05.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3a37fe640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3a41012a0 0x7ff3a4197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:05.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.560+0000 7ff3a8825640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4101c60 0x7ff3a419e800 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff39c00eaa0 tx=0x7ff39c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:05.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.561+0000 7ff3a17fa640 1 -- 172.21.15.76:0/3793512185 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c00cdb0 con 0x7ff3a4101c60 2024-03-12T05:53:05.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.561+0000 7ff3aa288640 1 -- 172.21.15.76:0/3793512185 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a419f350 con 0x7ff3a4101c60 2024-03-12T05:53:05.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.561+0000 7ff3a17fa640 1 -- 172.21.15.76:0/3793512185 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff39c005020 con 0x7ff3a4101c60 2024-03-12T05:53:05.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.561+0000 7ff3a17fa640 1 -- 172.21.15.76:0/3793512185 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c0116b0 con 0x7ff3a4101c60 2024-03-12T05:53:05.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.561+0000 7ff3aa288640 1 -- 172.21.15.76:0/3793512185 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a419f810 con 0x7ff3a4101c60 2024-03-12T05:53:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.562+0000 7ff3aa288640 1 -- 172.21.15.76:0/3793512185 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a410bb10 con 0x7ff3a4101c60 2024-03-12T05:53:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.563+0000 7ff3a17fa640 1 -- 172.21.15.76:0/3793512185 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff39c005520 con 0x7ff3a4101c60 2024-03-12T05:53:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.563+0000 7ff3a17fa640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff38c0a1a30 0x7ff38c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.563+0000 7ff3a3fff640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff38c0a1a30 0x7ff38c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.563+0000 7ff3a17fa640 1 -- 172.21.15.76:0/3793512185 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff39c0cd220 con 0x7ff3a4101c60 2024-03-12T05:53:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.564+0000 7ff3a3fff640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff38c0a1a30 0x7ff38c0a3ef0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff3980058c0 tx=0x7ff398005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:05.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.568+0000 7ff3a17fa640 1 -- 172.21.15.76:0/3793512185 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff39c0968d0 con 0x7ff3a4101c60 2024-03-12T05:53:05.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:05 smithi082 ceph-mon[36382]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:05.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:05 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:05.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:05 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:05.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:05 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:05.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:05 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:05.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:05 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:05.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:05 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:05.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:05 smithi088 ceph-mon[36585]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:05.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:05 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:05.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:05 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:05.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:05 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:05.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:05 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:05.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:05 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:05.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:05 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:05.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:05 smithi076 ceph-mon[32095]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:05.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:05 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:05.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:05 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:05.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:05 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:05.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:05 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:05.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:05 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:05.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:05 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:05.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.809+0000 7ff3aa288640 1 -- 172.21.15.76:0/3793512185 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3a410bca0 con 0x7ff3a4101c60 2024-03-12T05:53:05.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.810+0000 7ff3a17fa640 1 -- 172.21.15.76:0/3793512185 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2116 (secure 0 0 0) 0x7ff39c099f90 con 0x7ff3a4101c60 2024-03-12T05:53:05.813 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:05.813 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-12T05:53:05.814 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 3 2024-03-12T05:53:05.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.812+0000 7ff3aa288640 1 -- 172.21.15.76:0/3793512185 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff38c0a1a30 msgr2=0x7ff38c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:05.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.812+0000 7ff3aa288640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff38c0a1a30 0x7ff38c0a3ef0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff3980058c0 tx=0x7ff398005850 comp rx=0 tx=0).stop 2024-03-12T05:53:05.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.812+0000 7ff3aa288640 1 -- 172.21.15.76:0/3793512185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4101c60 msgr2=0x7ff3a419e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:05.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.812+0000 7ff3aa288640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4101c60 0x7ff3a419e800 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff39c00eaa0 tx=0x7ff39c00ef60 comp rx=0 tx=0).stop 2024-03-12T05:53:05.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.812+0000 7ff3aa288640 1 -- 172.21.15.76:0/3793512185 shutdown_connections 2024-03-12T05:53:05.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.812+0000 7ff3aa288640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff38c0a1a30 0x7ff38c0a3ef0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:05.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.812+0000 7ff3aa288640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4101c60 0x7ff3a419e800 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:05.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.812+0000 7ff3aa288640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3a41012a0 0x7ff3a4197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.812+0000 7ff3aa288640 1 --2- 172.21.15.76:0/3793512185 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff3a41000b0 0x7ff3a4197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.812+0000 7ff3aa288640 1 -- 172.21.15.76:0/3793512185 >> 172.21.15.76:0/3793512185 conn(0x7ff3a40fb620 msgr2=0x7ff3a40fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.812+0000 7ff3aa288640 1 -- 172.21.15.76:0/3793512185 shutdown_connections 2024-03-12T05:53:05.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:05.812+0000 7ff3aa288640 1 -- 172.21.15.76:0/3793512185 wait complete. 2024-03-12T05:53:06.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:06 smithi082 ceph-mon[36382]: Deploying daemon mds.j on smithi076 2024-03-12T05:53:06.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:06 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.82:6832/2359119459,v1:172.21.15.82:6833/2359119459] up:boot 2024-03-12T05:53:06.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:06 smithi082 ceph-mon[36382]: fsmap 2 up:standby 2024-03-12T05:53:06.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:06 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:53:06.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:06 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3793512185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:06.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:06 smithi088 ceph-mon[36585]: Deploying daemon mds.j on smithi076 2024-03-12T05:53:06.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:06 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.82:6832/2359119459,v1:172.21.15.82:6833/2359119459] up:boot 2024-03-12T05:53:06.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:06 smithi088 ceph-mon[36585]: fsmap 2 up:standby 2024-03-12T05:53:06.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:06 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:53:06.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:06 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3793512185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:06.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:06 smithi076 ceph-mon[32095]: Deploying daemon mds.j on smithi076 2024-03-12T05:53:06.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:06 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.82:6832/2359119459,v1:172.21.15.82:6833/2359119459] up:boot 2024-03-12T05:53:06.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:06 smithi076 ceph-mon[32095]: fsmap 2 up:standby 2024-03-12T05:53:06.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:06 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:53:06.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:06 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3793512185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:07.413 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 05:53:07 smithi076 systemd[1]: Starting Ceph mds.j for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T05:53:07.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:07 smithi082 ceph-mon[36382]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:07.733 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:07 smithi088 ceph-mon[36585]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:07.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:07 smithi076 ceph-mon[32095]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:07.737 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 05:53:07 smithi076 podman[72094]: 2024-03-12 05:53:07.412309811 +0000 UTC m=+0.019414214 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:53:07.737 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 05:53:07 smithi076 podman[72094]: 2024-03-12 05:53:07.542466257 +0000 UTC m=+0.149570659 container create 7656e4ec3d6a93f0bcdeb8ffdf135ca49b38f791ad4f9f0cd64ef044bd23716a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-12T05:53:07.827 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:07.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.937+0000 7fdb14335640 1 -- 172.21.15.76:0/2045791103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb0c1000b0 msgr2=0x7fdb0c1004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:07.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.937+0000 7fdb14335640 1 --2- 172.21.15.76:0/2045791103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb0c1000b0 0x7fdb0c1004d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009900 tx=0x7fdb0802bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:07.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 -- 172.21.15.76:0/2045791103 shutdown_connections 2024-03-12T05:53:07.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 --2- 172.21.15.76:0/2045791103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb0c101c60 0x7fdb0c1084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:07.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 --2- 172.21.15.76:0/2045791103 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb0c1012a0 0x7fdb0c101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:07.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 --2- 172.21.15.76:0/2045791103 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb0c1000b0 0x7fdb0c1004d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:07.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 -- 172.21.15.76:0/2045791103 >> 172.21.15.76:0/2045791103 conn(0x7fdb0c0fb620 msgr2=0x7fdb0c0fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:07.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 -- 172.21.15.76:0/2045791103 shutdown_connections 2024-03-12T05:53:07.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 -- 172.21.15.76:0/2045791103 wait complete. 2024-03-12T05:53:07.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 Processor -- start 2024-03-12T05:53:07.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 -- start start 2024-03-12T05:53:07.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb0c1000b0 0x7fdb0c196120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:07.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb0c1012a0 0x7fdb0c196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:07.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb0c101c60 0x7fdb0c19d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:07.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c19dc10 con 0x7fdb0c101c60 2024-03-12T05:53:07.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c19dd80 con 0x7fdb0c1000b0 2024-03-12T05:53:07.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb14335640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c19df40 con 0x7fdb0c1012a0 2024-03-12T05:53:07.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb128d2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb0c101c60 0x7fdb0c19d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:07.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb128d2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb0c101c60 0x7fdb0c19d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40110/0 (socket says 172.21.15.76:40110) 2024-03-12T05:53:07.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb128d2640 1 -- 172.21.15.76:0/2083046652 learned_addr learned my addr 172.21.15.76:0/2083046652 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:07.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.938+0000 7fdb120d1640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb0c1000b0 0x7fdb0c196120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:07.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.939+0000 7fdb118d0640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb0c1012a0 0x7fdb0c196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:07.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.939+0000 7fdb128d2640 1 -- 172.21.15.76:0/2083046652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb0c1012a0 msgr2=0x7fdb0c196660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:07.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.939+0000 7fdb128d2640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb0c1012a0 0x7fdb0c196660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:07.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.939+0000 7fdb128d2640 1 -- 172.21.15.76:0/2083046652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb0c1000b0 msgr2=0x7fdb0c196120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:07.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.939+0000 7fdb128d2640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb0c1000b0 0x7fdb0c196120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:07.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.939+0000 7fdb128d2640 1 -- 172.21.15.76:0/2083046652 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb08009560 con 0x7fdb0c101c60 2024-03-12T05:53:07.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.939+0000 7fdb120d1640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb0c1000b0 0x7fdb0c196120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:53:07.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.939+0000 7fdb128d2640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb0c101c60 0x7fdb0c19d6d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400eaa0 tx=0x7fdb0400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:07.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.939+0000 7fdb037fe640 1 -- 172.21.15.76:0/2083046652 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0400cdb0 con 0x7fdb0c101c60 2024-03-12T05:53:07.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.939+0000 7fdb14335640 1 -- 172.21.15.76:0/2083046652 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0c19e220 con 0x7fdb0c101c60 2024-03-12T05:53:07.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.939+0000 7fdb037fe640 1 -- 172.21.15.76:0/2083046652 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb04005020 con 0x7fdb0c101c60 2024-03-12T05:53:07.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.939+0000 7fdb037fe640 1 -- 172.21.15.76:0/2083046652 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb040116d0 con 0x7fdb0c101c60 2024-03-12T05:53:07.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.939+0000 7fdb14335640 1 -- 172.21.15.76:0/2083046652 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0c19e730 con 0x7fdb0c101c60 2024-03-12T05:53:07.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.940+0000 7fdb14335640 1 -- 172.21.15.76:0/2083046652 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0c1073d0 con 0x7fdb0c101c60 2024-03-12T05:53:07.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.941+0000 7fdb037fe640 1 -- 172.21.15.76:0/2083046652 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fdb04005490 con 0x7fdb0c101c60 2024-03-12T05:53:07.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.942+0000 7fdb037fe640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdaf80a1a30 0x7fdaf80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.942+0000 7fdb120d1640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdaf80a1a30 0x7fdaf80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.942+0000 7fdb037fe640 1 -- 172.21.15.76:0/2083046652 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdb040cd010 con 0x7fdb0c101c60 2024-03-12T05:53:07.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.942+0000 7fdb120d1640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdaf80a1a30 0x7fdaf80a3ef0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdb08005ab0 tx=0x7fdb08005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:07.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:07.947+0000 7fdb037fe640 1 -- 172.21.15.76:0/2083046652 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdb04096770 con 0x7fdb0c101c60 2024-03-12T05:53:08.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:08.167+0000 7fdb14335640 1 -- 172.21.15.76:0/2083046652 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb0c02cdd0 con 0x7fdb0c101c60 2024-03-12T05:53:08.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:08.167+0000 7fdb037fe640 1 -- 172.21.15.76:0/2083046652 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2116 (secure 0 0 0) 0x7fdb04099e30 con 0x7fdb0c101c60 2024-03-12T05:53:08.171 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:08.171 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-12T05:53:08.171 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 3 2024-03-12T05:53:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:08.169+0000 7fdb14335640 1 -- 172.21.15.76:0/2083046652 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdaf80a1a30 msgr2=0x7fdaf80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:08.169+0000 7fdb14335640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdaf80a1a30 0x7fdaf80a3ef0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdb08005ab0 tx=0x7fdb08005fd0 comp rx=0 tx=0).stop 2024-03-12T05:53:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:08.169+0000 7fdb14335640 1 -- 172.21.15.76:0/2083046652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb0c101c60 msgr2=0x7fdb0c19d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:08.169+0000 7fdb14335640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb0c101c60 0x7fdb0c19d6d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400eaa0 tx=0x7fdb0400ef60 comp rx=0 tx=0).stop 2024-03-12T05:53:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:08.170+0000 7fdb14335640 1 -- 172.21.15.76:0/2083046652 shutdown_connections 2024-03-12T05:53:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:08.170+0000 7fdb14335640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdaf80a1a30 0x7fdaf80a3ef0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:08.170+0000 7fdb14335640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdb0c101c60 0x7fdb0c19d6d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:08.170+0000 7fdb14335640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb0c1012a0 0x7fdb0c196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:08.170+0000 7fdb14335640 1 --2- 172.21.15.76:0/2083046652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdb0c1000b0 0x7fdb0c196120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:08.170+0000 7fdb14335640 1 -- 172.21.15.76:0/2083046652 >> 172.21.15.76:0/2083046652 conn(0x7fdb0c0fb620 msgr2=0x7fdb0c1045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:08.170+0000 7fdb14335640 1 -- 172.21.15.76:0/2083046652 shutdown_connections 2024-03-12T05:53:08.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:08.170+0000 7fdb14335640 1 -- 172.21.15.76:0/2083046652 wait complete. 2024-03-12T05:53:08.183 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 05:53:07 smithi076 podman[72094]: 2024-03-12 05:53:07.753342371 +0000 UTC m=+0.360446784 container init 7656e4ec3d6a93f0bcdeb8ffdf135ca49b38f791ad4f9f0cd64ef044bd23716a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-j, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.vendor=CentOS) 2024-03-12T05:53:08.183 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 05:53:07 smithi076 podman[72094]: 2024-03-12 05:53:07.756002852 +0000 UTC m=+0.363107258 container start 7656e4ec3d6a93f0bcdeb8ffdf135ca49b38f791ad4f9f0cd64ef044bd23716a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-j, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, ceph=True) 2024-03-12T05:53:08.183 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 05:53:07 smithi076 ceph-mds[72113]: -- 172.21.15.76:0/2103925447 <== mon.0 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56398991e1a0 con 0x5639898f5000 2024-03-12T05:53:08.183 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 05:53:07 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-j[72109]: starting mds.j at 2024-03-12T05:53:08.183 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 05:53:07 smithi076 bash[72094]: 7656e4ec3d6a93f0bcdeb8ffdf135ca49b38f791ad4f9f0cd64ef044bd23716a 2024-03-12T05:53:08.183 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 05:53:07 smithi076 systemd[1]: Started Ceph mds.j for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:53:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:09.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:09.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:09.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:09.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:09.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:08 smithi082 ceph-mon[36382]: Deploying daemon mds.d on smithi076 2024-03-12T05:53:09.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:08 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2083046652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:09.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:08 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.76:6834/4172787847,v1:172.21.15.76:6835/4172787847] up:boot 2024-03-12T05:53:09.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:08 smithi082 ceph-mon[36382]: fsmap 3 up:standby 2024-03-12T05:53:09.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:08 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-12T05:53:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:08 smithi088 ceph-mon[36585]: Deploying daemon mds.d on smithi076 2024-03-12T05:53:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:08 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2083046652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:08 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.76:6834/4172787847,v1:172.21.15.76:6835/4172787847] up:boot 2024-03-12T05:53:09.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:08 smithi088 ceph-mon[36585]: fsmap 3 up:standby 2024-03-12T05:53:09.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:08 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-12T05:53:09.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:09.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:09.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:09.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:09.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:09.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:09.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:08 smithi076 ceph-mon[32095]: Deploying daemon mds.d on smithi076 2024-03-12T05:53:09.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:08 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2083046652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:09.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:08 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.76:6834/4172787847,v1:172.21.15.76:6835/4172787847] up:boot 2024-03-12T05:53:09.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:08 smithi076 ceph-mon[32095]: fsmap 3 up:standby 2024-03-12T05:53:09.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:08 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-12T05:53:10.186 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:10.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:09 smithi082 ceph-mon[36382]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:10.214 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:09 smithi076 ceph-mon[32095]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:10.214 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 05:53:10 smithi076 systemd[1]: Starting Ceph mds.d for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T05:53:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:09 smithi088 ceph-mon[36585]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:10.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.293+0000 7ff53029a640 1 -- 172.21.15.76:0/2718506056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff528100120 msgr2=0x7ff528100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:10.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.293+0000 7ff53029a640 1 --2- 172.21.15.76:0/2718506056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff528100120 0x7ff528100540 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff524009900 tx=0x7ff52402bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:10.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.293+0000 7ff53029a640 1 -- 172.21.15.76:0/2718506056 shutdown_connections 2024-03-12T05:53:10.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.293+0000 7ff53029a640 1 --2- 172.21.15.76:0/2718506056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff528101cd0 0x7ff528108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:10.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.293+0000 7ff53029a640 1 --2- 172.21.15.76:0/2718506056 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff528101310 0x7ff528101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:10.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.293+0000 7ff53029a640 1 --2- 172.21.15.76:0/2718506056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff528100120 0x7ff528100540 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:10.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.293+0000 7ff53029a640 1 -- 172.21.15.76:0/2718506056 >> 172.21.15.76:0/2718506056 conn(0x7ff5280fb690 msgr2=0x7ff5280fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:10.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.293+0000 7ff53029a640 1 -- 172.21.15.76:0/2718506056 shutdown_connections 2024-03-12T05:53:10.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.293+0000 7ff53029a640 1 -- 172.21.15.76:0/2718506056 wait complete. 2024-03-12T05:53:10.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff53029a640 1 Processor -- start 2024-03-12T05:53:10.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff53029a640 1 -- start start 2024-03-12T05:53:10.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff53029a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff528100120 0x7ff5281972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:10.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff53029a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff528101310 0x7ff528197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:10.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff53029a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff528101cd0 0x7ff52819e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:10.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff53029a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52819ede0 con 0x7ff528100120 2024-03-12T05:53:10.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff53029a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52819ef50 con 0x7ff528101cd0 2024-03-12T05:53:10.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff53029a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52819f110 con 0x7ff528101310 2024-03-12T05:53:10.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff52e036640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff528100120 0x7ff5281972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:10.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff52e036640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff528100120 0x7ff5281972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40122/0 (socket says 172.21.15.76:40122) 2024-03-12T05:53:10.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff52e036640 1 -- 172.21.15.76:0/45892807 learned_addr learned my addr 172.21.15.76:0/45892807 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:10.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff52e837640 1 --2- 172.21.15.76:0/45892807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff528101cd0 0x7ff52819e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:10.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff52e036640 1 -- 172.21.15.76:0/45892807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff528101310 msgr2=0x7ff528197830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:10.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff52e036640 1 --2- 172.21.15.76:0/45892807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff528101310 0x7ff528197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:10.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff52e036640 1 -- 172.21.15.76:0/45892807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff528101cd0 msgr2=0x7ff52819e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:10.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff52e036640 1 --2- 172.21.15.76:0/45892807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff528101cd0 0x7ff52819e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:10.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff52e036640 1 -- 172.21.15.76:0/45892807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff524009560 con 0x7ff528100120 2024-03-12T05:53:10.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.294+0000 7ff52e837640 1 --2- 172.21.15.76:0/45892807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff528101cd0 0x7ff52819e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:10.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.295+0000 7ff52e036640 1 --2- 172.21.15.76:0/45892807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff528100120 0x7ff5281972f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff524000c00 tx=0x7ff524004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:10.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.295+0000 7ff51f7fe640 1 -- 172.21.15.76:0/45892807 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff524004df0 con 0x7ff528100120 2024-03-12T05:53:10.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.295+0000 7ff53029a640 1 -- 172.21.15.76:0/45892807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff52819f390 con 0x7ff528100120 2024-03-12T05:53:10.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.295+0000 7ff51f7fe640 1 -- 172.21.15.76:0/45892807 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff524005430 con 0x7ff528100120 2024-03-12T05:53:10.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.295+0000 7ff51f7fe640 1 -- 172.21.15.76:0/45892807 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52403e450 con 0x7ff528100120 2024-03-12T05:53:10.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.295+0000 7ff53029a640 1 -- 172.21.15.76:0/45892807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff52819f920 con 0x7ff528100120 2024-03-12T05:53:10.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.296+0000 7ff53029a640 1 -- 172.21.15.76:0/45892807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff52810bb00 con 0x7ff528100120 2024-03-12T05:53:10.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.297+0000 7ff51f7fe640 1 -- 172.21.15.76:0/45892807 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff524003680 con 0x7ff528100120 2024-03-12T05:53:10.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.297+0000 7ff51f7fe640 1 --2- 172.21.15.76:0/45892807 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff5140a1a30 0x7ff5140a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:10.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.297+0000 7ff52d835640 1 --2- 172.21.15.76:0/45892807 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff5140a1a30 0x7ff5140a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:10.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.297+0000 7ff51f7fe640 1 -- 172.21.15.76:0/45892807 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff5240ef540 con 0x7ff528100120 2024-03-12T05:53:10.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.298+0000 7ff52d835640 1 --2- 172.21.15.76:0/45892807 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff5140a1a30 0x7ff5140a3ef0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff518005fd0 tx=0x7ff518005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:10.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.302+0000 7ff51f7fe640 1 -- 172.21.15.76:0/45892807 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff5240b8bf0 con 0x7ff528100120 2024-03-12T05:53:10.487 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 05:53:10 smithi076 podman[72443]: 2024-03-12 05:53:10.211020473 +0000 UTC m=+0.018355695 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:53:10.487 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 05:53:10 smithi076 podman[72443]: 2024-03-12 05:53:10.397403923 +0000 UTC m=+0.204739134 container create 89bf230cda25afb00fa9fb84fdaf8816989db03ff3fd1f2facc0f9eda2bab8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-12T05:53:10.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.517+0000 7ff53029a640 1 -- 172.21.15.76:0/45892807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff52819fb40 con 0x7ff528100120 2024-03-12T05:53:10.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.517+0000 7ff51f7fe640 1 -- 172.21.15.76:0/45892807 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2907 (secure 0 0 0) 0x7ff5240bc2b0 con 0x7ff528100120 2024-03-12T05:53:10.521 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:10.521 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-12T05:53:10.521 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 4 2024-03-12T05:53:10.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.519+0000 7ff53029a640 1 -- 172.21.15.76:0/45892807 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff5140a1a30 msgr2=0x7ff5140a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:10.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.519+0000 7ff53029a640 1 --2- 172.21.15.76:0/45892807 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff5140a1a30 0x7ff5140a3ef0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff518005fd0 tx=0x7ff518005850 comp rx=0 tx=0).stop 2024-03-12T05:53:10.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.519+0000 7ff53029a640 1 -- 172.21.15.76:0/45892807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff528100120 msgr2=0x7ff5281972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:10.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.519+0000 7ff53029a640 1 --2- 172.21.15.76:0/45892807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff528100120 0x7ff5281972f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff524000c00 tx=0x7ff524004270 comp rx=0 tx=0).stop 2024-03-12T05:53:10.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.519+0000 7ff53029a640 1 -- 172.21.15.76:0/45892807 shutdown_connections 2024-03-12T05:53:10.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.519+0000 7ff53029a640 1 --2- 172.21.15.76:0/45892807 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff5140a1a30 0x7ff5140a3ef0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:10.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.519+0000 7ff53029a640 1 --2- 172.21.15.76:0/45892807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff528101cd0 0x7ff52819e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:10.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.520+0000 7ff53029a640 1 --2- 172.21.15.76:0/45892807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff528101310 0x7ff528197830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:10.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.520+0000 7ff53029a640 1 --2- 172.21.15.76:0/45892807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff528100120 0x7ff5281972f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:10.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.520+0000 7ff53029a640 1 -- 172.21.15.76:0/45892807 >> 172.21.15.76:0/45892807 conn(0x7ff5280fb690 msgr2=0x7ff5280fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:10.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.520+0000 7ff53029a640 1 -- 172.21.15.76:0/45892807 shutdown_connections 2024-03-12T05:53:10.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:10.520+0000 7ff53029a640 1 -- 172.21.15.76:0/45892807 wait complete. 2024-03-12T05:53:10.909 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 05:53:10 smithi076 podman[72443]: 2024-03-12 05:53:10.650264096 +0000 UTC m=+0.457599310 container init 89bf230cda25afb00fa9fb84fdaf8816989db03ff3fd1f2facc0f9eda2bab8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-12T05:53:10.910 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 05:53:10 smithi076 podman[72443]: 2024-03-12 05:53:10.652993787 +0000 UTC m=+0.460329004 container start 89bf230cda25afb00fa9fb84fdaf8816989db03ff3fd1f2facc0f9eda2bab8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-03-12T05:53:10.910 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 05:53:10 smithi076 ceph-mds[72502]: -- 172.21.15.76:0/756751473 <== mon.0 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d5408ae1a0 con 0x55d540885000 2024-03-12T05:53:10.910 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 05:53:10 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-d[72497]: starting mds.d at 2024-03-12T05:53:10.910 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 05:53:10 smithi076 bash[72443]: 89bf230cda25afb00fa9fb84fdaf8816989db03ff3fd1f2facc0f9eda2bab8bf 2024-03-12T05:53:10.910 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 05:53:10 smithi076 systemd[1]: Started Ceph mds.d for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:53:11.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:10 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/45892807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:11.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:10 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:11.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:10 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:11.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:10 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:11.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:10 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:11.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:10 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:11.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:10 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:11.233 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:10 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/45892807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:11.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:10 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:11.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:10 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:11.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:10 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:11.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:10 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:11.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:10 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:11.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:10 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:11.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:10 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/45892807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:11.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:10 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:11.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:10 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:11.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:10 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:11.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:10 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:11.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:10 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:11.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:10 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:12.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:11 smithi082 ceph-mon[36382]: Deploying daemon mds.e on smithi082 2024-03-12T05:53:12.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:11 smithi082 ceph-mon[36382]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:12.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:11 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.76:6836/2194036306,v1:172.21.15.76:6837/2194036306] up:boot 2024-03-12T05:53:12.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:11 smithi082 ceph-mon[36382]: fsmap 4 up:standby 2024-03-12T05:53:12.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:11 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-12T05:53:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:11 smithi088 ceph-mon[36585]: Deploying daemon mds.e on smithi082 2024-03-12T05:53:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:11 smithi088 ceph-mon[36585]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:11 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.76:6836/2194036306,v1:172.21.15.76:6837/2194036306] up:boot 2024-03-12T05:53:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:11 smithi088 ceph-mon[36585]: fsmap 4 up:standby 2024-03-12T05:53:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:11 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-12T05:53:12.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:11 smithi076 ceph-mon[32095]: Deploying daemon mds.e on smithi082 2024-03-12T05:53:12.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:11 smithi076 ceph-mon[32095]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:12.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:11 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.76:6836/2194036306,v1:172.21.15.76:6837/2194036306] up:boot 2024-03-12T05:53:12.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:11 smithi076 ceph-mon[32095]: fsmap 4 up:standby 2024-03-12T05:53:12.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:11 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-12T05:53:12.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:12.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.652+0000 7f7207b3e640 1 -- 172.21.15.76:0/539750845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7200101c60 msgr2=0x7f72001084f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:12.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.652+0000 7f7207b3e640 1 --2- 172.21.15.76:0/539750845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7200101c60 0x7f72001084f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009ca0 tx=0x7f71f802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:53:12.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.652+0000 7f7207b3e640 1 -- 172.21.15.76:0/539750845 shutdown_connections 2024-03-12T05:53:12.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.652+0000 7f7207b3e640 1 --2- 172.21.15.76:0/539750845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7200101c60 0x7f72001084f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:12.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.652+0000 7f7207b3e640 1 --2- 172.21.15.76:0/539750845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f72001012a0 0x7f7200101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:12.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.652+0000 7f7207b3e640 1 --2- 172.21.15.76:0/539750845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72001000b0 0x7f72001004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:12.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.652+0000 7f7207b3e640 1 -- 172.21.15.76:0/539750845 >> 172.21.15.76:0/539750845 conn(0x7f72000fb620 msgr2=0x7f72000fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:12.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.652+0000 7f7207b3e640 1 -- 172.21.15.76:0/539750845 shutdown_connections 2024-03-12T05:53:12.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.652+0000 7f7207b3e640 1 -- 172.21.15.76:0/539750845 wait complete. 2024-03-12T05:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f7207b3e640 1 Processor -- start 2024-03-12T05:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f7207b3e640 1 -- start start 2024-03-12T05:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f7207b3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72001000b0 0x7f72001960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f7207b3e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72001012a0 0x7f7200196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f7207b3e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7200101c60 0x7f720019d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f7207b3e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720019dbc0 con 0x7f72001012a0 2024-03-12T05:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f7207b3e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720019dd30 con 0x7f7200101c60 2024-03-12T05:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f7207b3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720019def0 con 0x7f72001000b0 2024-03-12T05:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f72050d9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72001012a0 0x7f7200196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:12.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f72050d9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72001012a0 0x7f7200196610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35398/0 (socket says 172.21.15.76:35398) 2024-03-12T05:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f72050d9640 1 -- 172.21.15.76:0/731902241 learned_addr learned my addr 172.21.15.76:0/731902241 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f72058da640 1 --2- 172.21.15.76:0/731902241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72001000b0 0x7f72001960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f72060db640 1 --2- 172.21.15.76:0/731902241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7200101c60 0x7f720019d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f72050d9640 1 -- 172.21.15.76:0/731902241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72001000b0 msgr2=0x7f72001960d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f72050d9640 1 --2- 172.21.15.76:0/731902241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72001000b0 0x7f72001960d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f72050d9640 1 -- 172.21.15.76:0/731902241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7200101c60 msgr2=0x7f720019d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f72050d9640 1 --2- 172.21.15.76:0/731902241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7200101c60 0x7f720019d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f72050d9640 1 -- 172.21.15.76:0/731902241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f8009900 con 0x7f72001012a0 2024-03-12T05:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.653+0000 7f72050d9640 1 --2- 172.21.15.76:0/731902241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72001012a0 0x7f7200196610 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00e9f0 tx=0x7f71fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.654+0000 7f71f6ffd640 1 -- 172.21.15.76:0/731902241 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc009700 con 0x7f72001012a0 2024-03-12T05:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.654+0000 7f7207b3e640 1 -- 172.21.15.76:0/731902241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f720019e1d0 con 0x7f72001012a0 2024-03-12T05:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.654+0000 7f71f6ffd640 1 -- 172.21.15.76:0/731902241 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f71fc00cd80 con 0x7f72001012a0 2024-03-12T05:53:12.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.654+0000 7f71f6ffd640 1 -- 172.21.15.76:0/731902241 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc018920 con 0x7f72001012a0 2024-03-12T05:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.654+0000 7f7207b3e640 1 -- 172.21.15.76:0/731902241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f720019e6e0 con 0x7f72001012a0 2024-03-12T05:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.655+0000 7f7207b3e640 1 -- 172.21.15.76:0/731902241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72001073d0 con 0x7f72001012a0 2024-03-12T05:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.656+0000 7f71f6ffd640 1 -- 172.21.15.76:0/731902241 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f71fc018a80 con 0x7f72001012a0 2024-03-12T05:53:12.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.656+0000 7f71f6ffd640 1 --2- 172.21.15.76:0/731902241 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f71ec0a1a30 0x7f71ec0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:12.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.656+0000 7f72058da640 1 --2- 172.21.15.76:0/731902241 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f71ec0a1a30 0x7f71ec0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:12.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.656+0000 7f71f6ffd640 1 -- 172.21.15.76:0/731902241 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f71fc014070 con 0x7f72001012a0 2024-03-12T05:53:12.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.657+0000 7f72058da640 1 --2- 172.21.15.76:0/731902241 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f71ec0a1a30 0x7f71ec0a3ef0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7200101100 tx=0x7f71f000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:12.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.664+0000 7f71f6ffd640 1 -- 172.21.15.76:0/731902241 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f71fc096ed0 con 0x7f72001012a0 2024-03-12T05:53:12.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.882+0000 7f7207b3e640 1 -- 172.21.15.76:0/731902241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f720010bbe0 con 0x7f72001012a0 2024-03-12T05:53:12.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.882+0000 7f71f6ffd640 1 -- 172.21.15.76:0/731902241 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3698 (secure 0 0 0) 0x7f71fc09a590 con 0x7f72001012a0 2024-03-12T05:53:12.885 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:12.885 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-12T05:53:12.886 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 5 2024-03-12T05:53:12.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.884+0000 7f7207b3e640 1 -- 172.21.15.76:0/731902241 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f71ec0a1a30 msgr2=0x7f71ec0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:12.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.884+0000 7f7207b3e640 1 --2- 172.21.15.76:0/731902241 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f71ec0a1a30 0x7f71ec0a3ef0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7200101100 tx=0x7f71f000b390 comp rx=0 tx=0).stop 2024-03-12T05:53:12.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.884+0000 7f7207b3e640 1 -- 172.21.15.76:0/731902241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72001012a0 msgr2=0x7f7200196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:12.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.884+0000 7f7207b3e640 1 --2- 172.21.15.76:0/731902241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72001012a0 0x7f7200196610 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00e9f0 tx=0x7f71fc00eeb0 comp rx=0 tx=0).stop 2024-03-12T05:53:12.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.884+0000 7f7207b3e640 1 -- 172.21.15.76:0/731902241 shutdown_connections 2024-03-12T05:53:12.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.884+0000 7f7207b3e640 1 --2- 172.21.15.76:0/731902241 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f71ec0a1a30 0x7f71ec0a3ef0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:12.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.884+0000 7f7207b3e640 1 --2- 172.21.15.76:0/731902241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7200101c60 0x7f720019d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:12.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.884+0000 7f7207b3e640 1 --2- 172.21.15.76:0/731902241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72001012a0 0x7f7200196610 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:12.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.884+0000 7f7207b3e640 1 --2- 172.21.15.76:0/731902241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72001000b0 0x7f72001960d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:12.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.884+0000 7f7207b3e640 1 -- 172.21.15.76:0/731902241 >> 172.21.15.76:0/731902241 conn(0x7f72000fb620 msgr2=0x7f72001045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:12.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.884+0000 7f7207b3e640 1 -- 172.21.15.76:0/731902241 shutdown_connections 2024-03-12T05:53:12.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:12.884+0000 7f7207b3e640 1 -- 172.21.15.76:0/731902241 wait complete. 2024-03-12T05:53:13.160 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:12 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/731902241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:12 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/731902241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:13.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:12 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/731902241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:13.694 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 12 05:53:13 smithi082 systemd[1]: Started Ceph mds.e for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:53:14.170 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:13 smithi088 ceph-mon[36585]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:14.170 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:14.170 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:14.170 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:14.170 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:14.171 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:14.171 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:13 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:14.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:13 smithi082 ceph-mon[36382]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:14.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:14.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:14.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:14.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:14.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:14.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:13 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:14.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:13 smithi076 ceph-mon[32095]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:14.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:14.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:14.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:14.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:14.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:14.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:13 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:14.899 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:15.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.016+0000 7fc85b5c7640 1 -- 172.21.15.76:0/1885715575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc854102d60 msgr2=0x7fc8541095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:15.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.016+0000 7fc85b5c7640 1 --2- 172.21.15.76:0/1885715575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc854102d60 0x7fc8541095f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc84c009ca0 tx=0x7fc84c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:53:15.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.016+0000 7fc85b5c7640 1 -- 172.21.15.76:0/1885715575 shutdown_connections 2024-03-12T05:53:15.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.016+0000 7fc85b5c7640 1 --2- 172.21.15.76:0/1885715575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc854102d60 0x7fc8541095f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:15.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.016+0000 7fc85b5c7640 1 --2- 172.21.15.76:0/1885715575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8541023a0 0x7fc854102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:15.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.016+0000 7fc85b5c7640 1 --2- 172.21.15.76:0/1885715575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8541011b0 0x7fc8541015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:15.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.016+0000 7fc85b5c7640 1 -- 172.21.15.76:0/1885715575 >> 172.21.15.76:0/1885715575 conn(0x7fc8540fc760 msgr2=0x7fc8540feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:15.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.016+0000 7fc85b5c7640 1 -- 172.21.15.76:0/1885715575 shutdown_connections 2024-03-12T05:53:15.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.016+0000 7fc85b5c7640 1 -- 172.21.15.76:0/1885715575 wait complete. 2024-03-12T05:53:15.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.017+0000 7fc85b5c7640 1 Processor -- start 2024-03-12T05:53:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.017+0000 7fc85b5c7640 1 -- start start 2024-03-12T05:53:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.017+0000 7fc85b5c7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8541011b0 0x7fc854197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.017+0000 7fc85b5c7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8541023a0 0x7fc8541977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.017+0000 7fc85b5c7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc854102d60 0x7fc85419e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.017+0000 7fc85b5c7640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85419ed70 con 0x7fc8541023a0 2024-03-12T05:53:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.017+0000 7fc85b5c7640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85419eee0 con 0x7fc854102d60 2024-03-12T05:53:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.017+0000 7fc85b5c7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85419f0a0 con 0x7fc8541011b0 2024-03-12T05:53:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.017+0000 7fc858b62640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8541023a0 0x7fc8541977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.017+0000 7fc858b62640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8541023a0 0x7fc8541977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35408/0 (socket says 172.21.15.76:35408) 2024-03-12T05:53:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.017+0000 7fc858b62640 1 -- 172.21.15.76:0/2947025580 learned_addr learned my addr 172.21.15.76:0/2947025580 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.017+0000 7fc859363640 1 --2- 172.21.15.76:0/2947025580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8541011b0 0x7fc854197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.017+0000 7fc859b64640 1 --2- 172.21.15.76:0/2947025580 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc854102d60 0x7fc85419e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.018+0000 7fc858b62640 1 -- 172.21.15.76:0/2947025580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8541011b0 msgr2=0x7fc854197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:15.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.018+0000 7fc858b62640 1 --2- 172.21.15.76:0/2947025580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8541011b0 0x7fc854197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:15.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.018+0000 7fc858b62640 1 -- 172.21.15.76:0/2947025580 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc854102d60 msgr2=0x7fc85419e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:15.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.018+0000 7fc858b62640 1 --2- 172.21.15.76:0/2947025580 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc854102d60 0x7fc85419e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:15.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.018+0000 7fc858b62640 1 -- 172.21.15.76:0/2947025580 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c009900 con 0x7fc8541023a0 2024-03-12T05:53:15.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.018+0000 7fc858b62640 1 --2- 172.21.15.76:0/2947025580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8541023a0 0x7fc8541977c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc84400b4a0 tx=0x7fc84400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:15.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.018+0000 7fc84a7fc640 1 -- 172.21.15.76:0/2947025580 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844010610 con 0x7fc8541023a0 2024-03-12T05:53:15.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.018+0000 7fc84a7fc640 1 -- 172.21.15.76:0/2947025580 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc844010c50 con 0x7fc8541023a0 2024-03-12T05:53:15.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.018+0000 7fc85b5c7640 1 -- 172.21.15.76:0/2947025580 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85419f380 con 0x7fc8541023a0 2024-03-12T05:53:15.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.018+0000 7fc84a7fc640 1 -- 172.21.15.76:0/2947025580 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844017e20 con 0x7fc8541023a0 2024-03-12T05:53:15.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.018+0000 7fc85b5c7640 1 -- 172.21.15.76:0/2947025580 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85419f840 con 0x7fc8541023a0 2024-03-12T05:53:15.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.019+0000 7fc85b5c7640 1 -- 172.21.15.76:0/2947025580 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8541084d0 con 0x7fc8541023a0 2024-03-12T05:53:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.020+0000 7fc84a7fc640 1 -- 172.21.15.76:0/2947025580 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc844017480 con 0x7fc8541023a0 2024-03-12T05:53:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.021+0000 7fc84a7fc640 1 --2- 172.21.15.76:0/2947025580 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc8400a1a30 0x7fc8400a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.021+0000 7fc859363640 1 --2- 172.21.15.76:0/2947025580 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc8400a1a30 0x7fc8400a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.021+0000 7fc84a7fc640 1 -- 172.21.15.76:0/2947025580 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc8440ccad0 con 0x7fc8541023a0 2024-03-12T05:53:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.021+0000 7fc859363640 1 --2- 172.21.15.76:0/2947025580 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc8400a1a30 0x7fc8400a3ef0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc850005db0 tx=0x7fc850005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:15.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.025+0000 7fc84a7fc640 1 -- 172.21.15.76:0/2947025580 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc844096180 con 0x7fc8541023a0 2024-03-12T05:53:15.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:14 smithi082 ceph-mon[36382]: Deploying daemon mds.f on smithi088 2024-03-12T05:53:15.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:14 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.82:6834/2138041661,v1:172.21.15.82:6835/2138041661] up:boot 2024-03-12T05:53:15.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:14 smithi082 ceph-mon[36382]: fsmap 5 up:standby 2024-03-12T05:53:15.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:14 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:53:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:14 smithi088 ceph-mon[36585]: Deploying daemon mds.f on smithi088 2024-03-12T05:53:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:14 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.82:6834/2138041661,v1:172.21.15.82:6835/2138041661] up:boot 2024-03-12T05:53:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:14 smithi088 ceph-mon[36585]: fsmap 5 up:standby 2024-03-12T05:53:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:14 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:53:15.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.233+0000 7fc85b5c7640 1 -- 172.21.15.76:0/2947025580 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc85410cc00 con 0x7fc8541023a0 2024-03-12T05:53:15.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.233+0000 7fc84a7fc640 1 -- 172.21.15.76:0/2947025580 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4489 (secure 0 0 0) 0x7fc8440092b0 con 0x7fc8541023a0 2024-03-12T05:53:15.236 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:15.237 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-12T05:53:15.237 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 6 2024-03-12T05:53:15.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:14 smithi076 ceph-mon[32095]: Deploying daemon mds.f on smithi088 2024-03-12T05:53:15.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:14 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.82:6834/2138041661,v1:172.21.15.82:6835/2138041661] up:boot 2024-03-12T05:53:15.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:14 smithi076 ceph-mon[32095]: fsmap 5 up:standby 2024-03-12T05:53:15.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:14 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:53:15.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.235+0000 7fc85b5c7640 1 -- 172.21.15.76:0/2947025580 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc8400a1a30 msgr2=0x7fc8400a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:15.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.235+0000 7fc85b5c7640 1 --2- 172.21.15.76:0/2947025580 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc8400a1a30 0x7fc8400a3ef0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc850005db0 tx=0x7fc850005d40 comp rx=0 tx=0).stop 2024-03-12T05:53:15.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.235+0000 7fc85b5c7640 1 -- 172.21.15.76:0/2947025580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8541023a0 msgr2=0x7fc8541977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:15.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.235+0000 7fc85b5c7640 1 --2- 172.21.15.76:0/2947025580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8541023a0 0x7fc8541977c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc84400b4a0 tx=0x7fc84400b960 comp rx=0 tx=0).stop 2024-03-12T05:53:15.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.235+0000 7fc85b5c7640 1 -- 172.21.15.76:0/2947025580 shutdown_connections 2024-03-12T05:53:15.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.235+0000 7fc85b5c7640 1 --2- 172.21.15.76:0/2947025580 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc8400a1a30 0x7fc8400a3ef0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:15.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.235+0000 7fc85b5c7640 1 --2- 172.21.15.76:0/2947025580 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc854102d60 0x7fc85419e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:15.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.235+0000 7fc85b5c7640 1 --2- 172.21.15.76:0/2947025580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8541023a0 0x7fc8541977c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:15.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.235+0000 7fc85b5c7640 1 --2- 172.21.15.76:0/2947025580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8541011b0 0x7fc854197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:15.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.235+0000 7fc85b5c7640 1 -- 172.21.15.76:0/2947025580 >> 172.21.15.76:0/2947025580 conn(0x7fc8540fc760 msgr2=0x7fc8541056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:15.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.235+0000 7fc85b5c7640 1 -- 172.21.15.76:0/2947025580 shutdown_connections 2024-03-12T05:53:15.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:15.235+0000 7fc85b5c7640 1 -- 172.21.15.76:0/2947025580 wait complete. 2024-03-12T05:53:15.654 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 05:53:15 smithi088 systemd[1]: Starting Ceph mds.f for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T05:53:15.923 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 05:53:15 smithi088 podman[67687]: 2024-03-12 05:53:15.65135016 +0000 UTC m=+0.109019594 container create e1fdcaa76ae5353ff8f2530ad38d7e5fd3fc58704620e6cb3a8256426044414c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-f, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T05:53:15.923 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 05:53:15 smithi088 podman[67687]: 2024-03-12 05:53:15.560132912 +0000 UTC m=+0.017802304 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:53:15.923 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 05:53:15 smithi088 podman[67687]: 2024-03-12 05:53:15.845397414 +0000 UTC m=+0.303066805 container init e1fdcaa76ae5353ff8f2530ad38d7e5fd3fc58704620e6cb3a8256426044414c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-12T05:53:15.923 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 05:53:15 smithi088 podman[67687]: 2024-03-12 05:53:15.848108374 +0000 UTC m=+0.305777766 container start e1fdcaa76ae5353ff8f2530ad38d7e5fd3fc58704620e6cb3a8256426044414c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T05:53:15.923 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 05:53:15 smithi088 ceph-mds[67706]: -- 172.21.15.88:0/1425331692 <== mon.1 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bced0041a0 con 0x55bcecfda800 2024-03-12T05:53:15.923 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 05:53:15 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-f[67702]: starting mds.f at 2024-03-12T05:53:15.923 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 05:53:15 smithi088 bash[67687]: e1fdcaa76ae5353ff8f2530ad38d7e5fd3fc58704620e6cb3a8256426044414c 2024-03-12T05:53:16.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:15 smithi082 ceph-mon[36382]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:16.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:15 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2947025580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:16.233 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 05:53:15 smithi088 systemd[1]: Started Ceph mds.f for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:53:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:15 smithi088 ceph-mon[36585]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:15 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2947025580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:16.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:15 smithi076 ceph-mon[32095]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:16.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:15 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2947025580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:17.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:16 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] up:boot 2024-03-12T05:53:17.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:16 smithi082 ceph-mon[36382]: fsmap 6 up:standby 2024-03-12T05:53:17.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:16 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:53:17.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:16 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:17.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:16 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:17.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:16 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:17.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:16 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:17.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:16 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:17.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:16 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:17.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:16 smithi082 ceph-mon[36382]: Deploying daemon mds.c on smithi088 2024-03-12T05:53:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:16 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] up:boot 2024-03-12T05:53:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:16 smithi088 ceph-mon[36585]: fsmap 6 up:standby 2024-03-12T05:53:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:16 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:53:17.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:16 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:17.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:16 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:17.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:16 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:17.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:16 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:17.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:16 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:17.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:16 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:17.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:16 smithi088 ceph-mon[36585]: Deploying daemon mds.c on smithi088 2024-03-12T05:53:17.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:16 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] up:boot 2024-03-12T05:53:17.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:16 smithi076 ceph-mon[32095]: fsmap 6 up:standby 2024-03-12T05:53:17.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:16 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:53:17.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:16 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:17.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:16 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:17.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:16 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:17.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:16 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:17.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:16 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:17.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:16 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:17.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:16 smithi076 ceph-mon[32095]: Deploying daemon mds.c on smithi088 2024-03-12T05:53:17.251 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:17.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.367+0000 7fc38f4d4640 1 -- 172.21.15.76:0/1961595279 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc388101310 msgr2=0x7fc388101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:17.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.367+0000 7fc38f4d4640 1 --2- 172.21.15.76:0/1961595279 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc388101310 0x7fc388101790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc378009900 tx=0x7fc37802bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:17.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 -- 172.21.15.76:0/1961595279 shutdown_connections 2024-03-12T05:53:17.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 --2- 172.21.15.76:0/1961595279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388101cd0 0x7fc388108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:17.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 --2- 172.21.15.76:0/1961595279 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc388101310 0x7fc388101790 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:17.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 --2- 172.21.15.76:0/1961595279 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc388100120 0x7fc388100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:17.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 -- 172.21.15.76:0/1961595279 >> 172.21.15.76:0/1961595279 conn(0x7fc3880fb6b0 msgr2=0x7fc3880fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:17.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 -- 172.21.15.76:0/1961595279 shutdown_connections 2024-03-12T05:53:17.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 -- 172.21.15.76:0/1961595279 wait complete. 2024-03-12T05:53:17.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 Processor -- start 2024-03-12T05:53:17.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 -- start start 2024-03-12T05:53:17.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388100120 0x7fc388196150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:17.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc388101310 0x7fc388196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:17.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc388101cd0 0x7fc38819d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:17.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38819dc40 con 0x7fc388101310 2024-03-12T05:53:17.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38819ddb0 con 0x7fc388101cd0 2024-03-12T05:53:17.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.368+0000 7fc38f4d4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38819df70 con 0x7fc388100120 2024-03-12T05:53:17.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc38ca6f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc388101310 0x7fc388196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:17.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc38ca6f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc388101310 0x7fc388196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35426/0 (socket says 172.21.15.76:35426) 2024-03-12T05:53:17.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc38ca6f640 1 -- 172.21.15.76:0/2730351871 learned_addr learned my addr 172.21.15.76:0/2730351871 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:17.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc38da71640 1 --2- 172.21.15.76:0/2730351871 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc388101cd0 0x7fc38819d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:17.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc38d270640 1 --2- 172.21.15.76:0/2730351871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388100120 0x7fc388196150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:17.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc38ca6f640 1 -- 172.21.15.76:0/2730351871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388100120 msgr2=0x7fc388196150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:17.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc38ca6f640 1 --2- 172.21.15.76:0/2730351871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388100120 0x7fc388196150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:17.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc38ca6f640 1 -- 172.21.15.76:0/2730351871 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc388101cd0 msgr2=0x7fc38819d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:17.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc38ca6f640 1 --2- 172.21.15.76:0/2730351871 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc388101cd0 0x7fc38819d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:17.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc38ca6f640 1 -- 172.21.15.76:0/2730351871 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc378009560 con 0x7fc388101310 2024-03-12T05:53:17.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc38ca6f640 1 --2- 172.21.15.76:0/2730351871 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc388101310 0x7fc388196690 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc37802c840 tx=0x7fc378004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:17.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc37e7fc640 1 -- 172.21.15.76:0/2730351871 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc378004a80 con 0x7fc388101310 2024-03-12T05:53:17.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc38f4d4640 1 -- 172.21.15.76:0/2730351871 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38819e1f0 con 0x7fc388101310 2024-03-12T05:53:17.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc37e7fc640 1 -- 172.21.15.76:0/2730351871 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3780052d0 con 0x7fc388101310 2024-03-12T05:53:17.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc37e7fc640 1 -- 172.21.15.76:0/2730351871 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc378003e20 con 0x7fc388101310 2024-03-12T05:53:17.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.369+0000 7fc38f4d4640 1 -- 172.21.15.76:0/2730351871 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38819e6d0 con 0x7fc388101310 2024-03-12T05:53:17.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.371+0000 7fc38f4d4640 1 -- 172.21.15.76:0/2730351871 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3881073e0 con 0x7fc388101310 2024-03-12T05:53:17.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.372+0000 7fc37e7fc640 1 -- 172.21.15.76:0/2730351871 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc3780035f0 con 0x7fc388101310 2024-03-12T05:53:17.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.372+0000 7fc37e7fc640 1 --2- 172.21.15.76:0/2730351871 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc3740a1a30 0x7fc3740a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:17.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.372+0000 7fc38d270640 1 --2- 172.21.15.76:0/2730351871 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc3740a1a30 0x7fc3740a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:17.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.372+0000 7fc37e7fc640 1 -- 172.21.15.76:0/2730351871 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc3780ef360 con 0x7fc388101310 2024-03-12T05:53:17.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.373+0000 7fc38d270640 1 --2- 172.21.15.76:0/2730351871 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc3740a1a30 0x7fc3740a3ef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc384005fd0 tx=0x7fc38400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:17.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.377+0000 7fc37e7fc640 1 -- 172.21.15.76:0/2730351871 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc3780b8a10 con 0x7fc388101310 2024-03-12T05:53:17.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.588+0000 7fc38f4d4640 1 -- 172.21.15.76:0/2730351871 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc38802cda0 con 0x7fc388101310 2024-03-12T05:53:17.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.589+0000 7fc37e7fc640 1 -- 172.21.15.76:0/2730351871 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5280 (secure 0 0 0) 0x7fc378005440 con 0x7fc388101310 2024-03-12T05:53:17.592 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:17.592 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-12T05:53:17.592 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 7 2024-03-12T05:53:17.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.591+0000 7fc38f4d4640 1 -- 172.21.15.76:0/2730351871 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc3740a1a30 msgr2=0x7fc3740a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:17.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.591+0000 7fc38f4d4640 1 --2- 172.21.15.76:0/2730351871 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc3740a1a30 0x7fc3740a3ef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc384005fd0 tx=0x7fc38400b410 comp rx=0 tx=0).stop 2024-03-12T05:53:17.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.591+0000 7fc38f4d4640 1 -- 172.21.15.76:0/2730351871 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc388101310 msgr2=0x7fc388196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:17.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.591+0000 7fc38f4d4640 1 --2- 172.21.15.76:0/2730351871 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc388101310 0x7fc388196690 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc37802c840 tx=0x7fc378004270 comp rx=0 tx=0).stop 2024-03-12T05:53:17.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.591+0000 7fc38f4d4640 1 -- 172.21.15.76:0/2730351871 shutdown_connections 2024-03-12T05:53:17.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.591+0000 7fc38f4d4640 1 --2- 172.21.15.76:0/2730351871 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc3740a1a30 0x7fc3740a3ef0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:17.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.591+0000 7fc38f4d4640 1 --2- 172.21.15.76:0/2730351871 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc388101cd0 0x7fc38819d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:17.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.591+0000 7fc38f4d4640 1 --2- 172.21.15.76:0/2730351871 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc388101310 0x7fc388196690 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:17.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.591+0000 7fc38f4d4640 1 --2- 172.21.15.76:0/2730351871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc388100120 0x7fc388196150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:17.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.591+0000 7fc38f4d4640 1 -- 172.21.15.76:0/2730351871 >> 172.21.15.76:0/2730351871 conn(0x7fc3880fb6b0 msgr2=0x7fc3881045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:17.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.591+0000 7fc38f4d4640 1 -- 172.21.15.76:0/2730351871 shutdown_connections 2024-03-12T05:53:17.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:17.591+0000 7fc38f4d4640 1 -- 172.21.15.76:0/2730351871 wait complete. 2024-03-12T05:53:18.198 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:17 smithi088 ceph-mon[36585]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:18.198 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:17 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2730351871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:18.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:17 smithi082 ceph-mon[36382]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:18.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:17 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2730351871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:18.459 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 05:53:18 smithi088 podman[67993]: 2024-03-12 05:53:18.434109586 +0000 UTC m=+0.344682101 container start 4550fbbd722ac36ec05d86750bdb3ebc621f37ed0c7d4f6ead8f492bd256f7fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:53:18.460 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 05:53:18 smithi088 ceph-mds[68012]: -- 172.21.15.88:0/812461044 <== mon.1 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557b2a1161a0 con 0x557b2a0ed000 2024-03-12T05:53:18.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:17 smithi076 ceph-mon[32095]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:18.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:17 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2730351871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:18.734 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 05:53:18 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-c[68008]: starting mds.c at 2024-03-12T05:53:18.734 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 05:53:18 smithi088 bash[67993]: 4550fbbd722ac36ec05d86750bdb3ebc621f37ed0c7d4f6ead8f492bd256f7fa 2024-03-12T05:53:18.734 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 05:53:18 smithi088 systemd[1]: Started Ceph mds.c for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:53:19.605 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:19.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.713+0000 7fcd6d42e640 1 -- 172.21.15.76:0/2689965009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd68106f80 msgr2=0x7fcd68109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:19.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.713+0000 7fcd6d42e640 1 --2- 172.21.15.76:0/2689965009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd68106f80 0x7fcd68109370 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcd60009ca0 tx=0x7fcd6002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:53:19.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.713+0000 7fcd6d42e640 1 -- 172.21.15.76:0/2689965009 shutdown_connections 2024-03-12T05:53:19.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.713+0000 7fcd6d42e640 1 --2- 172.21.15.76:0/2689965009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd68106f80 0x7fcd68109370 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:19.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.713+0000 7fcd6d42e640 1 --2- 172.21.15.76:0/2689965009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd68104650 0x7fcd68106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:19.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.713+0000 7fcd6d42e640 1 --2- 172.21.15.76:0/2689965009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68101d20 0x7fcd68104110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:19.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.713+0000 7fcd6d42e640 1 -- 172.21.15.76:0/2689965009 >> 172.21.15.76:0/2689965009 conn(0x7fcd680fb6b0 msgr2=0x7fcd680fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:19.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.713+0000 7fcd6d42e640 1 -- 172.21.15.76:0/2689965009 shutdown_connections 2024-03-12T05:53:19.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.714+0000 7fcd6d42e640 1 -- 172.21.15.76:0/2689965009 wait complete. 2024-03-12T05:53:19.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.714+0000 7fcd6d42e640 1 Processor -- start 2024-03-12T05:53:19.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.714+0000 7fcd6d42e640 1 -- start start 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.714+0000 7fcd6d42e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd68101d20 0x7fcd68101690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.714+0000 7fcd6d42e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd68104650 0x7fcd680ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.714+0000 7fcd6d42e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68106f80 0x7fcd68100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.714+0000 7fcd6d42e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd68100970 con 0x7fcd68104650 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.714+0000 7fcd6d42e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd68100ae0 con 0x7fcd68106f80 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.714+0000 7fcd6d42e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd681a1240 con 0x7fcd68101d20 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.714+0000 7fcd667fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd68104650 0x7fcd680ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd667fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd68104650 0x7fcd680ffcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35434/0 (socket says 172.21.15.76:35434) 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd667fc640 1 -- 172.21.15.76:0/3737342984 learned_addr learned my addr 172.21.15.76:0/3737342984 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd66ffd640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd68101d20 0x7fcd68101690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd677fe640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68106f80 0x7fcd68100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd667fc640 1 -- 172.21.15.76:0/3737342984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd68101d20 msgr2=0x7fcd68101690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd667fc640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd68101d20 0x7fcd68101690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd667fc640 1 -- 172.21.15.76:0/3737342984 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68106f80 msgr2=0x7fcd68100210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:19.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd667fc640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68106f80 0x7fcd68100210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd667fc640 1 -- 172.21.15.76:0/3737342984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd60009900 con 0x7fcd68104650 2024-03-12T05:53:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd66ffd640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd68101d20 0x7fcd68101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:53:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd667fc640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd68104650 0x7fcd680ffcd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c00c9a0 tx=0x7fcd5c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd4ffff640 1 -- 172.21.15.76:0/3737342984 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c007bb0 con 0x7fcd68104650 2024-03-12T05:53:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd4ffff640 1 -- 172.21.15.76:0/3737342984 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd5c007d10 con 0x7fcd68104650 2024-03-12T05:53:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd6d42e640 1 -- 172.21.15.76:0/3737342984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd681a14f0 con 0x7fcd68104650 2024-03-12T05:53:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd4ffff640 1 -- 172.21.15.76:0/3737342984 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c018730 con 0x7fcd68104650 2024-03-12T05:53:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.715+0000 7fcd6d42e640 1 -- 172.21.15.76:0/3737342984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd681a19b0 con 0x7fcd68104650 2024-03-12T05:53:19.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.716+0000 7fcd6d42e640 1 -- 172.21.15.76:0/3737342984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd680fd1a0 con 0x7fcd68104650 2024-03-12T05:53:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.717+0000 7fcd4ffff640 1 -- 172.21.15.76:0/3737342984 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcd5c00f450 con 0x7fcd68104650 2024-03-12T05:53:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.718+0000 7fcd4ffff640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcd500a1a30 0x7fcd500a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.718+0000 7fcd66ffd640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcd500a1a30 0x7fcd500a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.718+0000 7fcd4ffff640 1 -- 172.21.15.76:0/3737342984 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcd5c0cd0d0 con 0x7fcd68104650 2024-03-12T05:53:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.718+0000 7fcd66ffd640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcd500a1a30 0x7fcd500a3ef0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcd58009c20 tx=0x7fcd58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.722+0000 7fcd4ffff640 1 -- 172.21.15.76:0/3737342984 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcd5c096830 con 0x7fcd68104650 2024-03-12T05:53:19.897 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:19.897 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:19.898 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:19.898 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:19.898 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:19.898 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:19.898 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:19 smithi082 ceph-mon[36382]: Deploying daemon mds.h on smithi082 2024-03-12T05:53:19.898 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:19 smithi082 ceph-mon[36382]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:19.898 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:19 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.88:6836/1311843051,v1:172.21.15.88:6837/1311843051] up:boot 2024-03-12T05:53:19.898 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:19 smithi082 ceph-mon[36382]: fsmap 7 up:standby 2024-03-12T05:53:19.898 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:19 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:53:19.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.931+0000 7fcd6d42e640 1 -- 172.21.15.76:0/3737342984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd6810cbc0 con 0x7fcd68104650 2024-03-12T05:53:19.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.932+0000 7fcd4ffff640 1 -- 172.21.15.76:0/3737342984 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6071 (secure 0 0 0) 0x7fcd5c099ef0 con 0x7fcd68104650 2024-03-12T05:53:19.935 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:19.936 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-12T05:53:19.936 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 8 2024-03-12T05:53:19.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.934+0000 7fcd6d42e640 1 -- 172.21.15.76:0/3737342984 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcd500a1a30 msgr2=0x7fcd500a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:19.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.934+0000 7fcd6d42e640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcd500a1a30 0x7fcd500a3ef0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcd58009c20 tx=0x7fcd58005f90 comp rx=0 tx=0).stop 2024-03-12T05:53:19.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.934+0000 7fcd6d42e640 1 -- 172.21.15.76:0/3737342984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd68104650 msgr2=0x7fcd680ffcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:19.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.934+0000 7fcd6d42e640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd68104650 0x7fcd680ffcd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c00c9a0 tx=0x7fcd5c00ce60 comp rx=0 tx=0).stop 2024-03-12T05:53:19.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.934+0000 7fcd6d42e640 1 -- 172.21.15.76:0/3737342984 shutdown_connections 2024-03-12T05:53:19.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.934+0000 7fcd6d42e640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcd500a1a30 0x7fcd500a3ef0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:19.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.934+0000 7fcd6d42e640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd68106f80 0x7fcd68100210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:19.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.934+0000 7fcd6d42e640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd68104650 0x7fcd680ffcd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:19.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.934+0000 7fcd6d42e640 1 --2- 172.21.15.76:0/3737342984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd68101d20 0x7fcd68101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:19.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.934+0000 7fcd6d42e640 1 -- 172.21.15.76:0/3737342984 >> 172.21.15.76:0/3737342984 conn(0x7fcd680fb6b0 msgr2=0x7fcd68102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:19.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.934+0000 7fcd6d42e640 1 -- 172.21.15.76:0/3737342984 shutdown_connections 2024-03-12T05:53:19.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:19.934+0000 7fcd6d42e640 1 -- 172.21.15.76:0/3737342984 wait complete. 2024-03-12T05:53:19.947 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:19.947 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:19.947 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:19.947 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:19.948 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:19.948 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:19.948 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:19 smithi076 ceph-mon[32095]: Deploying daemon mds.h on smithi082 2024-03-12T05:53:19.948 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:19 smithi076 ceph-mon[32095]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:19.948 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:19 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.88:6836/1311843051,v1:172.21.15.88:6837/1311843051] up:boot 2024-03-12T05:53:19.948 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:19 smithi076 ceph-mon[32095]: fsmap 7 up:standby 2024-03-12T05:53:19.948 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:19 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:53:19.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:19.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:19.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:19.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:19.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:19.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:19.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:19 smithi088 ceph-mon[36585]: Deploying daemon mds.h on smithi082 2024-03-12T05:53:19.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:19 smithi088 ceph-mon[36585]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:19.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:19 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.88:6836/1311843051,v1:172.21.15.88:6837/1311843051] up:boot 2024-03-12T05:53:19.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:19 smithi088 ceph-mon[36585]: fsmap 7 up:standby 2024-03-12T05:53:19.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:19 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:53:20.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:20 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3737342984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:20.945 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 05:53:20 smithi082 systemd[1]: Starting Ceph mds.h for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T05:53:20.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:20 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3737342984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:20.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:20 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3737342984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:21.294 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 05:53:20 smithi082 podman[66214]: 2024-03-12 05:53:20.900013308 +0000 UTC m=+0.017920470 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:53:21.295 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 05:53:21 smithi082 podman[66214]: 2024-03-12 05:53:21.031403136 +0000 UTC m=+0.149310282 container create e32cf20a5ee18216e1a0819b4910d3ea956823fe750b7e40dba074e675edeca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-h, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-12T05:53:21.642 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 05:53:21 smithi082 podman[66214]: 2024-03-12 05:53:21.292290027 +0000 UTC m=+0.410197174 container init e32cf20a5ee18216e1a0819b4910d3ea956823fe750b7e40dba074e675edeca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=squid-94a5397, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-12T05:53:21.643 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 05:53:21 smithi082 podman[66214]: 2024-03-12 05:53:21.295328757 +0000 UTC m=+0.413235904 container start e32cf20a5ee18216e1a0819b4910d3ea956823fe750b7e40dba074e675edeca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-h, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:53:21.643 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 05:53:21 smithi082 ceph-mds[66233]: -- 172.21.15.82:0/2364370965 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f0770021a0 con 0x55f076fd9000 2024-03-12T05:53:21.643 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 05:53:21 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-h[66229]: starting mds.h at 2024-03-12T05:53:21.643 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 05:53:21 smithi082 bash[66214]: e32cf20a5ee18216e1a0819b4910d3ea956823fe750b7e40dba074e675edeca4 2024-03-12T05:53:21.643 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 05:53:21 smithi082 systemd[1]: Started Ceph mds.h for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:53:21.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:21 smithi082 ceph-mon[36382]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:21.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:21 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:21.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:21 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:21.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:21 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:21.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:21 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:21.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:21 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:21.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:21 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:21.949 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:21.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:21 smithi076 ceph-mon[32095]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:21.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:21.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:21.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:21.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:21.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:21.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:21 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:21.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:21 smithi088 ceph-mon[36585]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:21.983 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:21 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:21.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:21 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:21.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:21 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:21.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:21 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:21.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:21 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:21.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:21 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:22.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.059+0000 7f233d44e640 1 -- 172.21.15.76:0/1623295499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2338100120 msgr2=0x7f2338100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:22.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.059+0000 7f233d44e640 1 --2- 172.21.15.76:0/1623295499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2338100120 0x7f2338100540 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2330009ca0 tx=0x7f233002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:53:22.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.059+0000 7f233d44e640 1 -- 172.21.15.76:0/1623295499 shutdown_connections 2024-03-12T05:53:22.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.059+0000 7f233d44e640 1 --2- 172.21.15.76:0/1623295499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2338101cd0 0x7f2338108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:22.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.059+0000 7f233d44e640 1 --2- 172.21.15.76:0/1623295499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2338101310 0x7f2338101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:22.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.059+0000 7f233d44e640 1 --2- 172.21.15.76:0/1623295499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2338100120 0x7f2338100540 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:22.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.059+0000 7f233d44e640 1 -- 172.21.15.76:0/1623295499 >> 172.21.15.76:0/1623295499 conn(0x7f23380fb6b0 msgr2=0x7f23380fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:22.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.059+0000 7f233d44e640 1 -- 172.21.15.76:0/1623295499 shutdown_connections 2024-03-12T05:53:22.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.059+0000 7f233d44e640 1 -- 172.21.15.76:0/1623295499 wait complete. 2024-03-12T05:53:22.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.059+0000 7f233d44e640 1 Processor -- start 2024-03-12T05:53:22.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f233d44e640 1 -- start start 2024-03-12T05:53:22.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f233d44e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2338100120 0x7f2338196120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:22.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f233d44e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2338101310 0x7f2338196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:22.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f233d44e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2338101cd0 0x7f233819d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:22.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f233d44e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233819dc10 con 0x7f2338101310 2024-03-12T05:53:22.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f233d44e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233819dd80 con 0x7f2338100120 2024-03-12T05:53:22.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f233d44e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233819df40 con 0x7f2338101cd0 2024-03-12T05:53:22.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f23367fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2338101310 0x7f2338196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:22.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f23367fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2338101310 0x7f2338196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35490/0 (socket says 172.21.15.76:35490) 2024-03-12T05:53:22.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f23367fc640 1 -- 172.21.15.76:0/3251999229 learned_addr learned my addr 172.21.15.76:0/3251999229 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:22.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f2336ffd640 1 --2- 172.21.15.76:0/3251999229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2338100120 0x7f2338196120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:22.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f23377fe640 1 --2- 172.21.15.76:0/3251999229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2338101cd0 0x7f233819d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:22.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f23367fc640 1 -- 172.21.15.76:0/3251999229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2338101cd0 msgr2=0x7f233819d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:22.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f23367fc640 1 --2- 172.21.15.76:0/3251999229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2338101cd0 0x7f233819d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:22.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f23367fc640 1 -- 172.21.15.76:0/3251999229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2338100120 msgr2=0x7f2338196120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:22.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f23367fc640 1 --2- 172.21.15.76:0/3251999229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2338100120 0x7f2338196120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:22.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f23367fc640 1 -- 172.21.15.76:0/3251999229 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2330009900 con 0x7f2338101310 2024-03-12T05:53:22.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f23367fc640 1 --2- 172.21.15.76:0/3251999229 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2338101310 0x7f2338196660 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f232000e9f0 tx=0x7f232000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:22.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f232ffff640 1 -- 172.21.15.76:0/3251999229 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320009700 con 0x7f2338101310 2024-03-12T05:53:22.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f232ffff640 1 -- 172.21.15.76:0/3251999229 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f232000cd80 con 0x7f2338101310 2024-03-12T05:53:22.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f233d44e640 1 -- 172.21.15.76:0/3251999229 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233819e220 con 0x7f2338101310 2024-03-12T05:53:22.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.060+0000 7f232ffff640 1 -- 172.21.15.76:0/3251999229 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320018920 con 0x7f2338101310 2024-03-12T05:53:22.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.061+0000 7f233d44e640 1 -- 172.21.15.76:0/3251999229 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233819e780 con 0x7f2338101310 2024-03-12T05:53:22.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.061+0000 7f233d44e640 1 -- 172.21.15.76:0/3251999229 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2338107440 con 0x7f2338101310 2024-03-12T05:53:22.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.062+0000 7f232ffff640 1 -- 172.21.15.76:0/3251999229 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2320018a80 con 0x7f2338101310 2024-03-12T05:53:22.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.063+0000 7f232ffff640 1 --2- 172.21.15.76:0/3251999229 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f23240a1ab0 0x7f23240a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:22.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.063+0000 7f2336ffd640 1 --2- 172.21.15.76:0/3251999229 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f23240a1ab0 0x7f23240a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:22.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.063+0000 7f232ffff640 1 -- 172.21.15.76:0/3251999229 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2320014070 con 0x7f2338101310 2024-03-12T05:53:22.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.064+0000 7f2336ffd640 1 --2- 172.21.15.76:0/3251999229 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f23240a1ab0 0x7f23240a3f70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f233002c700 tx=0x7f23300326b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:22.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.068+0000 7f232ffff640 1 -- 172.21.15.76:0/3251999229 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2320052d20 con 0x7f2338101310 2024-03-12T05:53:22.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.277+0000 7f233d44e640 1 -- 172.21.15.76:0/3251999229 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f233810bac0 con 0x7f2338101310 2024-03-12T05:53:22.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.277+0000 7f232ffff640 1 -- 172.21.15.76:0/3251999229 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6862 (secure 0 0 0) 0x7f23200969b0 con 0x7f2338101310 2024-03-12T05:53:22.281 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:22.281 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-12T05:53:22.281 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 9 2024-03-12T05:53:22.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.279+0000 7f233d44e640 1 -- 172.21.15.76:0/3251999229 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f23240a1ab0 msgr2=0x7f23240a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:22.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.279+0000 7f233d44e640 1 --2- 172.21.15.76:0/3251999229 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f23240a1ab0 0x7f23240a3f70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f233002c700 tx=0x7f23300326b0 comp rx=0 tx=0).stop 2024-03-12T05:53:22.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.279+0000 7f233d44e640 1 -- 172.21.15.76:0/3251999229 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2338101310 msgr2=0x7f2338196660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:22.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.279+0000 7f233d44e640 1 --2- 172.21.15.76:0/3251999229 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2338101310 0x7f2338196660 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f232000e9f0 tx=0x7f232000eeb0 comp rx=0 tx=0).stop 2024-03-12T05:53:22.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.279+0000 7f233d44e640 1 -- 172.21.15.76:0/3251999229 shutdown_connections 2024-03-12T05:53:22.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.279+0000 7f233d44e640 1 --2- 172.21.15.76:0/3251999229 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f23240a1ab0 0x7f23240a3f70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:22.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.279+0000 7f233d44e640 1 --2- 172.21.15.76:0/3251999229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2338101cd0 0x7f233819d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:22.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.279+0000 7f233d44e640 1 --2- 172.21.15.76:0/3251999229 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2338101310 0x7f2338196660 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:22.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.279+0000 7f233d44e640 1 --2- 172.21.15.76:0/3251999229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2338100120 0x7f2338196120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:22.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.279+0000 7f233d44e640 1 -- 172.21.15.76:0/3251999229 >> 172.21.15.76:0/3251999229 conn(0x7f23380fb6b0 msgr2=0x7f2338104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:22.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.279+0000 7f233d44e640 1 -- 172.21.15.76:0/3251999229 shutdown_connections 2024-03-12T05:53:22.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:22.279+0000 7f233d44e640 1 -- 172.21.15.76:0/3251999229 wait complete. 2024-03-12T05:53:22.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:22 smithi082 ceph-mon[36382]: Deploying daemon mds.i on smithi088 2024-03-12T05:53:22.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:22 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.82:6836/2523422510,v1:172.21.15.82:6837/2523422510] up:boot 2024-03-12T05:53:22.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:22 smithi082 ceph-mon[36382]: fsmap 8 up:standby 2024-03-12T05:53:22.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:22 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:53:22.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:22 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3251999229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:22.965 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:22 smithi088 ceph-mon[36585]: Deploying daemon mds.i on smithi088 2024-03-12T05:53:22.965 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:22 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.82:6836/2523422510,v1:172.21.15.82:6837/2523422510] up:boot 2024-03-12T05:53:22.965 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:22 smithi088 ceph-mon[36585]: fsmap 8 up:standby 2024-03-12T05:53:22.966 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:22 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:53:22.966 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:22 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3251999229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:22.988 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:22 smithi076 ceph-mon[32095]: Deploying daemon mds.i on smithi088 2024-03-12T05:53:22.988 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:22 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.82:6836/2523422510,v1:172.21.15.82:6837/2523422510] up:boot 2024-03-12T05:53:22.988 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:22 smithi076 ceph-mon[32095]: fsmap 8 up:standby 2024-03-12T05:53:22.988 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:22 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:53:22.988 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:22 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3251999229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:23.652 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 05:53:23 smithi088 podman[68302]: 2024-03-12 05:53:23.606581491 +0000 UTC m=+0.120734046 container create 0f4ae941a28f35615eb4f9423c65b75a84d1ed8726b6302fdae18c4ef3354992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T05:53:23.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:23 smithi082 ceph-mon[36382]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:23.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:23 smithi088 ceph-mon[36585]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:23.984 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 05:53:23 smithi088 podman[68302]: 2024-03-12 05:53:23.800734318 +0000 UTC m=+0.314886848 container init 0f4ae941a28f35615eb4f9423c65b75a84d1ed8726b6302fdae18c4ef3354992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-i, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-12T05:53:23.984 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 05:53:23 smithi088 podman[68302]: 2024-03-12 05:53:23.803352116 +0000 UTC m=+0.317504629 container start 0f4ae941a28f35615eb4f9423c65b75a84d1ed8726b6302fdae18c4ef3354992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T05:53:23.984 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 05:53:23 smithi088 ceph-mds[68322]: -- 172.21.15.88:0/1475730392 <== mon.1 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5570e1f101a0 con 0x5570e1ee6c00 2024-03-12T05:53:23.984 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 05:53:23 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-i[68318]: starting mds.i at 2024-03-12T05:53:23.984 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 05:53:23 smithi088 bash[68302]: 0f4ae941a28f35615eb4f9423c65b75a84d1ed8726b6302fdae18c4ef3354992 2024-03-12T05:53:23.985 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 05:53:23 smithi088 systemd[1]: Started Ceph mds.i for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:53:23.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:23 smithi076 ceph-mon[32095]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:24.294 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:24.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.403+0000 7f82f933a640 1 -- 172.21.15.76:0/2744432606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f40ff840 msgr2=0x7f82f40ffc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:24.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.403+0000 7f82f933a640 1 --2- 172.21.15.76:0/2744432606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f40ff840 0x7f82f40ffc60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f82e4009ca0 tx=0x7f82e402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:53:24.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.403+0000 7f82f933a640 1 -- 172.21.15.76:0/2744432606 shutdown_connections 2024-03-12T05:53:24.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.403+0000 7f82f933a640 1 --2- 172.21.15.76:0/2744432606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82f41013f0 0x7f82f4107c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:24.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.403+0000 7f82f933a640 1 --2- 172.21.15.76:0/2744432606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f4100a30 0x7f82f4100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:24.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.403+0000 7f82f933a640 1 --2- 172.21.15.76:0/2744432606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f40ff840 0x7f82f40ffc60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:24.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.403+0000 7f82f933a640 1 -- 172.21.15.76:0/2744432606 >> 172.21.15.76:0/2744432606 conn(0x7f82f40fadb0 msgr2=0x7f82f40fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:24.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.403+0000 7f82f933a640 1 -- 172.21.15.76:0/2744432606 shutdown_connections 2024-03-12T05:53:24.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.403+0000 7f82f933a640 1 -- 172.21.15.76:0/2744432606 wait complete. 2024-03-12T05:53:24.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f933a640 1 Processor -- start 2024-03-12T05:53:24.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f933a640 1 -- start start 2024-03-12T05:53:24.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f933a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f40ff840 0x7f82f41037d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:24.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f933a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f4100a30 0x7f82f4101e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:24.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f933a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82f41013f0 0x7f82f4102350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:24.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f933a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4102ab0 con 0x7f82f4100a30 2024-03-12T05:53:24.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f933a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4102c20 con 0x7f82f41013f0 2024-03-12T05:53:24.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f933a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f41a12f0 con 0x7f82f40ff840 2024-03-12T05:53:24.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f27fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f4100a30 0x7f82f4101e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:24.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f27fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f4100a30 0x7f82f4101e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35500/0 (socket says 172.21.15.76:35500) 2024-03-12T05:53:24.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f27fc640 1 -- 172.21.15.76:0/4269141743 learned_addr learned my addr 172.21.15.76:0/4269141743 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:24.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f2ffd640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f40ff840 0x7f82f41037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:24.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f37fe640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82f41013f0 0x7f82f4102350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:24.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f27fc640 1 -- 172.21.15.76:0/4269141743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f40ff840 msgr2=0x7f82f41037d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:24.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f27fc640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f40ff840 0x7f82f41037d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:24.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f27fc640 1 -- 172.21.15.76:0/4269141743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82f41013f0 msgr2=0x7f82f4102350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:24.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f27fc640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82f41013f0 0x7f82f4102350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:24.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f27fc640 1 -- 172.21.15.76:0/4269141743 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc009490 con 0x7f82f4100a30 2024-03-12T05:53:24.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.404+0000 7f82f37fe640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82f41013f0 0x7f82f4102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:24.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.405+0000 7f82f27fc640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f4100a30 0x7f82f4101e10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00ea90 tx=0x7f82dc00ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:24.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.405+0000 7f82ebfff640 1 -- 172.21.15.76:0/4269141743 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc00ce40 con 0x7f82f4100a30 2024-03-12T05:53:24.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.405+0000 7f82ebfff640 1 -- 172.21.15.76:0/4269141743 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82dc0035f0 con 0x7f82f4100a30 2024-03-12T05:53:24.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.405+0000 7f82f933a640 1 -- 172.21.15.76:0/4269141743 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e4009900 con 0x7f82f4100a30 2024-03-12T05:53:24.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.405+0000 7f82ebfff640 1 -- 172.21.15.76:0/4269141743 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc010690 con 0x7f82f4100a30 2024-03-12T05:53:24.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.405+0000 7f82f933a640 1 -- 172.21.15.76:0/4269141743 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f41a1920 con 0x7f82f4100a30 2024-03-12T05:53:24.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.406+0000 7f82f933a640 1 -- 172.21.15.76:0/4269141743 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f410bae0 con 0x7f82f4100a30 2024-03-12T05:53:24.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.407+0000 7f82ebfff640 1 -- 172.21.15.76:0/4269141743 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f82dc003ad0 con 0x7f82f4100a30 2024-03-12T05:53:24.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.408+0000 7f82ebfff640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f82e00a1ab0 0x7f82e00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:24.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.408+0000 7f82f2ffd640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f82e00a1ab0 0x7f82e00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:24.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.408+0000 7f82ebfff640 1 -- 172.21.15.76:0/4269141743 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f82dc014070 con 0x7f82f4100a30 2024-03-12T05:53:24.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.409+0000 7f82f2ffd640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f82e00a1ab0 0x7f82e00a3f70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f82e402c700 tx=0x7f82e4009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:24.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.413+0000 7f82ebfff640 1 -- 172.21.15.76:0/4269141743 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f82dc096be0 con 0x7f82f4100a30 2024-03-12T05:53:24.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.633+0000 7f82f933a640 1 -- 172.21.15.76:0/4269141743 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82f410bc70 con 0x7f82f4100a30 2024-03-12T05:53:24.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.634+0000 7f82ebfff640 1 -- 172.21.15.76:0/4269141743 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6862 (secure 0 0 0) 0x7f82f410bc70 con 0x7f82f4100a30 2024-03-12T05:53:24.637 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:24.637 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-12T05:53:24.638 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 9 2024-03-12T05:53:24.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.636+0000 7f82f933a640 1 -- 172.21.15.76:0/4269141743 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f82e00a1ab0 msgr2=0x7f82e00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:24.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.636+0000 7f82f933a640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f82e00a1ab0 0x7f82e00a3f70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f82e402c700 tx=0x7f82e4009800 comp rx=0 tx=0).stop 2024-03-12T05:53:24.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.636+0000 7f82f933a640 1 -- 172.21.15.76:0/4269141743 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f4100a30 msgr2=0x7f82f4101e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:24.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.636+0000 7f82f933a640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f4100a30 0x7f82f4101e10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00ea90 tx=0x7f82dc00ef50 comp rx=0 tx=0).stop 2024-03-12T05:53:24.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.636+0000 7f82f933a640 1 -- 172.21.15.76:0/4269141743 shutdown_connections 2024-03-12T05:53:24.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.636+0000 7f82f933a640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f82e00a1ab0 0x7f82e00a3f70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:24.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.636+0000 7f82f933a640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f82f41013f0 0x7f82f4102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:24.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.636+0000 7f82f933a640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f82f4100a30 0x7f82f4101e10 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:24.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.636+0000 7f82f933a640 1 --2- 172.21.15.76:0/4269141743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82f40ff840 0x7f82f41037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:24.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.636+0000 7f82f933a640 1 -- 172.21.15.76:0/4269141743 >> 172.21.15.76:0/4269141743 conn(0x7f82f40fadb0 msgr2=0x7f82f40fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:24.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.636+0000 7f82f933a640 1 -- 172.21.15.76:0/4269141743 shutdown_connections 2024-03-12T05:53:24.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:24.636+0000 7f82f933a640 1 -- 172.21.15.76:0/4269141743 wait complete. 2024-03-12T05:53:25.175 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:25.175 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:25.175 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:25.175 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:25.176 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:25.176 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:25.176 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:24 smithi088 ceph-mon[36585]: Deploying daemon mds.g on smithi076 2024-03-12T05:53:25.176 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:24 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4269141743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:25.176 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:24 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] up:boot 2024-03-12T05:53:25.176 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:24 smithi088 ceph-mon[36585]: fsmap 9 up:standby 2024-03-12T05:53:25.176 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:24 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:53:25.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:25.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:25.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:25.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:25.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:25.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:25.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:24 smithi082 ceph-mon[36382]: Deploying daemon mds.g on smithi076 2024-03-12T05:53:25.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:24 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4269141743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:25.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:24 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] up:boot 2024-03-12T05:53:25.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:24 smithi082 ceph-mon[36382]: fsmap 9 up:standby 2024-03-12T05:53:25.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:24 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:53:25.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:25.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:25.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:25.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:25.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:25.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:25.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:24 smithi076 ceph-mon[32095]: Deploying daemon mds.g on smithi076 2024-03-12T05:53:25.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:24 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4269141743' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:25.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:24 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] up:boot 2024-03-12T05:53:25.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:24 smithi076 ceph-mon[32095]: fsmap 9 up:standby 2024-03-12T05:53:25.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:24 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:53:26.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:25 smithi082 ceph-mon[36382]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:25 smithi088 ceph-mon[36585]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:26.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:25 smithi076 ceph-mon[32095]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:26.238 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 05:53:26 smithi076 systemd[1]: Starting Ceph mds.g for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T05:53:26.625 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 05:53:26 smithi076 podman[73051]: 2024-03-12 05:53:26.232151942 +0000 UTC m=+0.017560559 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:53:26.625 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 05:53:26 smithi076 podman[73051]: 2024-03-12 05:53:26.353030166 +0000 UTC m=+0.138438783 container create 0c425e9597299e33f25a0116769e92ad40aaa4306843476d4fde91da52ce9aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-g, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-12T05:53:26.625 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 05:53:26 smithi076 podman[73051]: 2024-03-12 05:53:26.622746106 +0000 UTC m=+0.408154719 container init 0c425e9597299e33f25a0116769e92ad40aaa4306843476d4fde91da52ce9aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-g, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-12T05:53:26.652 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:26.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.756+0000 7fb6a26df640 1 -- 172.21.15.76:0/2956367644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb69c104560 msgr2=0x7fb69c106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:26.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.756+0000 7fb6a26df640 1 --2- 172.21.15.76:0/2956367644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb69c104560 0x7fb69c106950 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb690009900 tx=0x7fb69002bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:26.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.756+0000 7fb6a26df640 1 -- 172.21.15.76:0/2956367644 shutdown_connections 2024-03-12T05:53:26.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.756+0000 7fb6a26df640 1 --2- 172.21.15.76:0/2956367644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb69c106e90 0x7fb69c1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:26.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.756+0000 7fb6a26df640 1 --2- 172.21.15.76:0/2956367644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb69c104560 0x7fb69c106950 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:26.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.756+0000 7fb6a26df640 1 --2- 172.21.15.76:0/2956367644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb69c101c30 0x7fb69c104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:26.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.756+0000 7fb6a26df640 1 -- 172.21.15.76:0/2956367644 >> 172.21.15.76:0/2956367644 conn(0x7fb69c0fb620 msgr2=0x7fb69c0fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:26.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.756+0000 7fb6a26df640 1 -- 172.21.15.76:0/2956367644 shutdown_connections 2024-03-12T05:53:26.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.756+0000 7fb6a26df640 1 -- 172.21.15.76:0/2956367644 wait complete. 2024-03-12T05:53:26.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.756+0000 7fb6a26df640 1 Processor -- start 2024-03-12T05:53:26.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.756+0000 7fb6a26df640 1 -- start start 2024-03-12T05:53:26.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb6a26df640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb69c101c30 0x7fb69c197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:26.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb6a26df640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb69c104560 0x7fb69c197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:26.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb6a26df640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb69c106e90 0x7fb69c19e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:26.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb6a26df640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c19ed40 con 0x7fb69c106e90 2024-03-12T05:53:26.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb6a26df640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c19eeb0 con 0x7fb69c101c30 2024-03-12T05:53:26.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb6a26df640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c19f070 con 0x7fb69c104560 2024-03-12T05:53:26.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb6a0c7c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb69c106e90 0x7fb69c19e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:26.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb6a0c7c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb69c106e90 0x7fb69c19e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35542/0 (socket says 172.21.15.76:35542) 2024-03-12T05:53:26.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb6a0c7c640 1 -- 172.21.15.76:0/3517490936 learned_addr learned my addr 172.21.15.76:0/3517490936 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:26.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb69b7fe640 1 --2- 172.21.15.76:0/3517490936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb69c104560 0x7fb69c197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:26.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb69bfff640 1 --2- 172.21.15.76:0/3517490936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb69c101c30 0x7fb69c197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:26.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb6a0c7c640 1 -- 172.21.15.76:0/3517490936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb69c104560 msgr2=0x7fb69c197790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:26.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb6a0c7c640 1 --2- 172.21.15.76:0/3517490936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb69c104560 0x7fb69c197790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:26.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb6a0c7c640 1 -- 172.21.15.76:0/3517490936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb69c101c30 msgr2=0x7fb69c197250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:26.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb6a0c7c640 1 --2- 172.21.15.76:0/3517490936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb69c101c30 0x7fb69c197250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:26.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.757+0000 7fb6a0c7c640 1 -- 172.21.15.76:0/3517490936 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb690009560 con 0x7fb69c106e90 2024-03-12T05:53:26.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.758+0000 7fb6a0c7c640 1 --2- 172.21.15.76:0/3517490936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb69c106e90 0x7fb69c19e800 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb69400eaa0 tx=0x7fb69400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:26.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.758+0000 7fb6997fa640 1 -- 172.21.15.76:0/3517490936 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69400cdb0 con 0x7fb69c106e90 2024-03-12T05:53:26.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.758+0000 7fb6a26df640 1 -- 172.21.15.76:0/3517490936 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69c19f350 con 0x7fb69c106e90 2024-03-12T05:53:26.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.758+0000 7fb6997fa640 1 -- 172.21.15.76:0/3517490936 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb694005020 con 0x7fb69c106e90 2024-03-12T05:53:26.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.758+0000 7fb6997fa640 1 -- 172.21.15.76:0/3517490936 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6940116b0 con 0x7fb69c106e90 2024-03-12T05:53:26.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.758+0000 7fb6a26df640 1 -- 172.21.15.76:0/3517490936 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb69c19f7e0 con 0x7fb69c106e90 2024-03-12T05:53:26.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.759+0000 7fb6a26df640 1 -- 172.21.15.76:0/3517490936 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69c0fd110 con 0x7fb69c106e90 2024-03-12T05:53:26.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.760+0000 7fb6997fa640 1 -- 172.21.15.76:0/3517490936 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb694005520 con 0x7fb69c106e90 2024-03-12T05:53:26.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.760+0000 7fb6997fa640 1 --2- 172.21.15.76:0/3517490936 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb6840a1a30 0x7fb6840a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:26.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.760+0000 7fb69bfff640 1 --2- 172.21.15.76:0/3517490936 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb6840a1a30 0x7fb6840a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:26.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.761+0000 7fb6997fa640 1 -- 172.21.15.76:0/3517490936 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb6940cd240 con 0x7fb69c106e90 2024-03-12T05:53:26.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.761+0000 7fb69bfff640 1 --2- 172.21.15.76:0/3517490936 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb6840a1a30 0x7fb6840a3ef0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009b60 tx=0x7fb68c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:26.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:26.765+0000 7fb6997fa640 1 -- 172.21.15.76:0/3517490936 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb6940968f0 con 0x7fb69c106e90 2024-03-12T05:53:26.987 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 05:53:26 smithi076 podman[73051]: 2024-03-12 05:53:26.625670232 +0000 UTC m=+0.411078852 container start 0c425e9597299e33f25a0116769e92ad40aaa4306843476d4fde91da52ce9aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-g, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-03-12T05:53:26.987 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 05:53:26 smithi076 ceph-mds[73069]: -- 172.21.15.76:0/93991722 <== mon.0 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d8666801a0 con 0x55d866656c00 2024-03-12T05:53:26.988 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 05:53:26 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-g[73065]: starting mds.g at 2024-03-12T05:53:26.988 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 05:53:26 smithi076 bash[73051]: 0c425e9597299e33f25a0116769e92ad40aaa4306843476d4fde91da52ce9aba 2024-03-12T05:53:26.988 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 05:53:26 smithi076 systemd[1]: Started Ceph mds.g for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:53:27.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:27.014+0000 7fb6a26df640 1 -- 172.21.15.76:0/3517490936 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb69c10cbc0 con 0x7fb69c106e90 2024-03-12T05:53:27.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:27.014+0000 7fb6997fa640 1 -- 172.21.15.76:0/3517490936 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8447 (secure 0 0 0) 0x7fb694099fb0 con 0x7fb69c106e90 2024-03-12T05:53:27.017 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:27.018 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-12T05:53:27.018 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 11 2024-03-12T05:53:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:27.017+0000 7fb6a26df640 1 -- 172.21.15.76:0/3517490936 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb6840a1a30 msgr2=0x7fb6840a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:27.017+0000 7fb6a26df640 1 --2- 172.21.15.76:0/3517490936 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb6840a1a30 0x7fb6840a3ef0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009b60 tx=0x7fb68c005f90 comp rx=0 tx=0).stop 2024-03-12T05:53:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:27.017+0000 7fb6a26df640 1 -- 172.21.15.76:0/3517490936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb69c106e90 msgr2=0x7fb69c19e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:27.017+0000 7fb6a26df640 1 --2- 172.21.15.76:0/3517490936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb69c106e90 0x7fb69c19e800 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb69400eaa0 tx=0x7fb69400ef60 comp rx=0 tx=0).stop 2024-03-12T05:53:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:27.017+0000 7fb6a26df640 1 -- 172.21.15.76:0/3517490936 shutdown_connections 2024-03-12T05:53:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:27.017+0000 7fb6a26df640 1 --2- 172.21.15.76:0/3517490936 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb6840a1a30 0x7fb6840a3ef0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:27.017+0000 7fb6a26df640 1 --2- 172.21.15.76:0/3517490936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb69c106e90 0x7fb69c19e800 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:27.017+0000 7fb6a26df640 1 --2- 172.21.15.76:0/3517490936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb69c104560 0x7fb69c197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:27.017+0000 7fb6a26df640 1 --2- 172.21.15.76:0/3517490936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb69c101c30 0x7fb69c197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:27.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:27.017+0000 7fb6a26df640 1 -- 172.21.15.76:0/3517490936 >> 172.21.15.76:0/3517490936 conn(0x7fb69c0fb620 msgr2=0x7fb69c1026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:27.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:27.017+0000 7fb6a26df640 1 -- 172.21.15.76:0/3517490936 shutdown_connections 2024-03-12T05:53:27.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:27.017+0000 7fb6a26df640 1 -- 172.21.15.76:0/3517490936 wait complete. 2024-03-12T05:53:28.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:28.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:28.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:28.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:28.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:28.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:28.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:27 smithi076 ceph-mon[32095]: Deploying daemon mds.a on smithi076 2024-03-12T05:53:28.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:27 smithi076 ceph-mon[32095]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:28.141 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:27 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.76:6838/1304487620,v1:172.21.15.76:6839/1304487620] up:boot 2024-03-12T05:53:28.141 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:27 smithi076 ceph-mon[32095]: fsmap 10 up:standby 2024-03-12T05:53:28.141 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:27 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:53:28.141 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:27 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3517490936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:28.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:28.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:28.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:28.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:28.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:28.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:28.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:27 smithi082 ceph-mon[36382]: Deploying daemon mds.a on smithi076 2024-03-12T05:53:28.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:27 smithi082 ceph-mon[36382]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:28.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:27 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.76:6838/1304487620,v1:172.21.15.76:6839/1304487620] up:boot 2024-03-12T05:53:28.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:27 smithi082 ceph-mon[36382]: fsmap 10 up:standby 2024-03-12T05:53:28.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:27 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:53:28.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:27 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3517490936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:28.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:27 smithi088 ceph-mon[36585]: Deploying daemon mds.a on smithi076 2024-03-12T05:53:28.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:27 smithi088 ceph-mon[36585]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:28.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:27 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.76:6838/1304487620,v1:172.21.15.76:6839/1304487620] up:boot 2024-03-12T05:53:28.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:27 smithi088 ceph-mon[36585]: fsmap 10 up:standby 2024-03-12T05:53:28.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:27 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:53:28.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:27 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3517490936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:28.927 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 05:53:28 smithi076 systemd[1]: Starting Ceph mds.a for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T05:53:29.032 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:29.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.140+0000 7fd51d084640 1 -- 172.21.15.76:0/2923720341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd518101c60 msgr2=0x7fd5181084f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:29.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.140+0000 7fd51d084640 1 --2- 172.21.15.76:0/2923720341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd518101c60 0x7fd5181084f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd510009ca0 tx=0x7fd51002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:53:29.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.140+0000 7fd51d084640 1 -- 172.21.15.76:0/2923720341 shutdown_connections 2024-03-12T05:53:29.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.140+0000 7fd51d084640 1 --2- 172.21.15.76:0/2923720341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd518101c60 0x7fd5181084f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:29.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.140+0000 7fd51d084640 1 --2- 172.21.15.76:0/2923720341 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5181012a0 0x7fd518101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:29.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.140+0000 7fd51d084640 1 --2- 172.21.15.76:0/2923720341 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5181000b0 0x7fd5181004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:29.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.140+0000 7fd51d084640 1 -- 172.21.15.76:0/2923720341 >> 172.21.15.76:0/2923720341 conn(0x7fd5180fb620 msgr2=0x7fd5180fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:29.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.140+0000 7fd51d084640 1 -- 172.21.15.76:0/2923720341 shutdown_connections 2024-03-12T05:53:29.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.140+0000 7fd51d084640 1 -- 172.21.15.76:0/2923720341 wait complete. 2024-03-12T05:53:29.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51d084640 1 Processor -- start 2024-03-12T05:53:29.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51d084640 1 -- start start 2024-03-12T05:53:29.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51d084640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5181000b0 0x7fd518196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:29.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51d084640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5181012a0 0x7fd5181965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:29.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51d084640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd518101c60 0x7fd51819d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:29.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51d084640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51819db80 con 0x7fd5181000b0 2024-03-12T05:53:29.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51d084640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51819dcf0 con 0x7fd5181012a0 2024-03-12T05:53:29.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51d084640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51819deb0 con 0x7fd518101c60 2024-03-12T05:53:29.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd516d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5181000b0 0x7fd518196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd516d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5181000b0 0x7fd518196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:35562/0 (socket says 172.21.15.76:35562) 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd516d9d640 1 -- 172.21.15.76:0/4204384026 learned_addr learned my addr 172.21.15.76:0/4204384026 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51659c640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5181012a0 0x7fd5181965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51759e640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd518101c60 0x7fd51819d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51659c640 1 -- 172.21.15.76:0/4204384026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd518101c60 msgr2=0x7fd51819d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51659c640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd518101c60 0x7fd51819d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51659c640 1 -- 172.21.15.76:0/4204384026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5181000b0 msgr2=0x7fd518196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51659c640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5181000b0 0x7fd518196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd51659c640 1 -- 172.21.15.76:0/4204384026 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd510009900 con 0x7fd5181012a0 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.141+0000 7fd516d9d640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5181000b0 0x7fd518196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.142+0000 7fd51659c640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5181012a0 0x7fd5181965d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00e9f0 tx=0x7fd50c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.142+0000 7fd4fffff640 1 -- 172.21.15.76:0/4204384026 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c00ccc0 con 0x7fd5181012a0 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.142+0000 7fd51d084640 1 -- 172.21.15.76:0/4204384026 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51819e190 con 0x7fd5181012a0 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.142+0000 7fd4fffff640 1 -- 172.21.15.76:0/4204384026 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd50c00ce20 con 0x7fd5181012a0 2024-03-12T05:53:29.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.142+0000 7fd4fffff640 1 -- 172.21.15.76:0/4204384026 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c018920 con 0x7fd5181012a0 2024-03-12T05:53:29.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.142+0000 7fd51d084640 1 -- 172.21.15.76:0/4204384026 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd51819e6f0 con 0x7fd5181012a0 2024-03-12T05:53:29.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.143+0000 7fd51d084640 1 -- 172.21.15.76:0/4204384026 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5181073d0 con 0x7fd5181012a0 2024-03-12T05:53:29.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.144+0000 7fd4fffff640 1 -- 172.21.15.76:0/4204384026 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd50c018a80 con 0x7fd5181012a0 2024-03-12T05:53:29.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.145+0000 7fd4fffff640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd5000a1a30 0x7fd5000a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:29.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.145+0000 7fd516d9d640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd5000a1a30 0x7fd5000a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:29.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.145+0000 7fd4fffff640 1 -- 172.21.15.76:0/4204384026 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd50c014070 con 0x7fd5181012a0 2024-03-12T05:53:29.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.145+0000 7fd516d9d640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd5000a1a30 0x7fd5000a3ef0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd508005fd0 tx=0x7fd50800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:29.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.148+0000 7fd4fffff640 1 -- 172.21.15.76:0/4204384026 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd50c096940 con 0x7fd5181012a0 2024-03-12T05:53:29.224 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 05:53:29 smithi076 podman[73406]: 2024-03-12 05:53:28.992698511 +0000 UTC m=+0.019102669 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:53:29.224 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 05:53:29 smithi076 podman[73406]: 2024-03-12 05:53:29.099454427 +0000 UTC m=+0.125858572 container create c5ac474888b8c19a4992e0f80b576044dc52fee919ab59536b767e817082579f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T05:53:29.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.370+0000 7fd51d084640 1 -- 172.21.15.76:0/4204384026 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd51810bac0 con 0x7fd5181012a0 2024-03-12T05:53:29.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.371+0000 7fd4fffff640 1 -- 172.21.15.76:0/4204384026 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8447 (secure 0 0 0) 0x7fd50c09a000 con 0x7fd5181012a0 2024-03-12T05:53:29.374 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:29.374 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-12T05:53:29.375 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 11 2024-03-12T05:53:29.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.373+0000 7fd51d084640 1 -- 172.21.15.76:0/4204384026 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd5000a1a30 msgr2=0x7fd5000a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:29.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.373+0000 7fd51d084640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd5000a1a30 0x7fd5000a3ef0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd508005fd0 tx=0x7fd50800b410 comp rx=0 tx=0).stop 2024-03-12T05:53:29.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.373+0000 7fd51d084640 1 -- 172.21.15.76:0/4204384026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5181012a0 msgr2=0x7fd5181965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:29.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.373+0000 7fd51d084640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5181012a0 0x7fd5181965d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00e9f0 tx=0x7fd50c00eeb0 comp rx=0 tx=0).stop 2024-03-12T05:53:29.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.373+0000 7fd51d084640 1 -- 172.21.15.76:0/4204384026 shutdown_connections 2024-03-12T05:53:29.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.373+0000 7fd51d084640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd5000a1a30 0x7fd5000a3ef0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:29.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.373+0000 7fd51d084640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd518101c60 0x7fd51819d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:29.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.373+0000 7fd51d084640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd5181012a0 0x7fd5181965d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:29.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.373+0000 7fd51d084640 1 --2- 172.21.15.76:0/4204384026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5181000b0 0x7fd518196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:29.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.373+0000 7fd51d084640 1 -- 172.21.15.76:0/4204384026 >> 172.21.15.76:0/4204384026 conn(0x7fd5180fb620 msgr2=0x7fd5181045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:29.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.373+0000 7fd51d084640 1 -- 172.21.15.76:0/4204384026 shutdown_connections 2024-03-12T05:53:29.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:29.373+0000 7fd51d084640 1 -- 172.21.15.76:0/4204384026 wait complete. 2024-03-12T05:53:29.737 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 05:53:29 smithi076 podman[73406]: 2024-03-12 05:53:29.343674716 +0000 UTC m=+0.370078861 container init c5ac474888b8c19a4992e0f80b576044dc52fee919ab59536b767e817082579f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-12T05:53:29.737 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 05:53:29 smithi076 podman[73406]: 2024-03-12 05:53:29.346869856 +0000 UTC m=+0.373274006 container start c5ac474888b8c19a4992e0f80b576044dc52fee919ab59536b767e817082579f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-a, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-12T05:53:29.738 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 05:53:29 smithi076 ceph-mds[73472]: -- 172.21.15.76:0/82865479 <== mon.0 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557d254d81a0 con 0x557d254aec00 2024-03-12T05:53:29.738 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 05:53:29 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-a[73468]: starting mds.a at 2024-03-12T05:53:29.738 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 05:53:29 smithi076 bash[73406]: c5ac474888b8c19a4992e0f80b576044dc52fee919ab59536b767e817082579f 2024-03-12T05:53:29.738 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 05:53:29 smithi076 systemd[1]: Started Ceph mds.a for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:29 smithi082 ceph-mon[36382]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:29 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4204384026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:29 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:29 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:29 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:29 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:29 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:29 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:29 smithi088 ceph-mon[36585]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:29 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4204384026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:29 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:29 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:29 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:29 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:29 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:29 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:30.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:29 smithi076 ceph-mon[32095]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:30.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:29 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4204384026' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:30.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:29 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:30.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:29 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:30.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:29 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:30.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:29 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T05:53:30.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:29 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T05:53:30.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:29 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:31.150 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:30 smithi082 ceph-mon[36382]: Deploying daemon mds.b on smithi082 2024-03-12T05:53:31.150 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:30 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.76:6840/291567235,v1:172.21.15.76:6841/291567235] up:boot 2024-03-12T05:53:31.151 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:30 smithi082 ceph-mon[36382]: fsmap 11 up:standby 2024-03-12T05:53:31.151 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:30 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:53:31.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:30 smithi088 ceph-mon[36585]: Deploying daemon mds.b on smithi082 2024-03-12T05:53:31.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:30 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.76:6840/291567235,v1:172.21.15.76:6841/291567235] up:boot 2024-03-12T05:53:31.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:30 smithi088 ceph-mon[36585]: fsmap 11 up:standby 2024-03-12T05:53:31.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:30 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:53:31.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:30 smithi076 ceph-mon[32095]: Deploying daemon mds.b on smithi082 2024-03-12T05:53:31.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:30 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.76:6840/291567235,v1:172.21.15.76:6841/291567235] up:boot 2024-03-12T05:53:31.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:30 smithi076 ceph-mon[32095]: fsmap 11 up:standby 2024-03-12T05:53:31.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:30 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:53:31.388 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:31.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.498+0000 7f9f75c8a640 1 -- 172.21.15.76:0/4293834971 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f701045d0 msgr2=0x7f9f701069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:31.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.498+0000 7f9f75c8a640 1 --2- 172.21.15.76:0/4293834971 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f701045d0 0x7f9f701069c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9f64009900 tx=0x7f9f6402bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:31.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.498+0000 7f9f75c8a640 1 -- 172.21.15.76:0/4293834971 shutdown_connections 2024-03-12T05:53:31.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.498+0000 7f9f75c8a640 1 --2- 172.21.15.76:0/4293834971 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9f70106f00 0x7f9f70109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:31.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.498+0000 7f9f75c8a640 1 --2- 172.21.15.76:0/4293834971 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f701045d0 0x7f9f701069c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:31.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.498+0000 7f9f75c8a640 1 --2- 172.21.15.76:0/4293834971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f70101ca0 0x7f9f70104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:31.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.498+0000 7f9f75c8a640 1 -- 172.21.15.76:0/4293834971 >> 172.21.15.76:0/4293834971 conn(0x7f9f700fb6b0 msgr2=0x7f9f700fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:31.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.498+0000 7f9f75c8a640 1 -- 172.21.15.76:0/4293834971 shutdown_connections 2024-03-12T05:53:31.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.498+0000 7f9f75c8a640 1 -- 172.21.15.76:0/4293834971 wait complete. 2024-03-12T05:53:31.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f75c8a640 1 Processor -- start 2024-03-12T05:53:31.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f75c8a640 1 -- start start 2024-03-12T05:53:31.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f75c8a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9f70101ca0 0x7f9f701972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:31.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f75c8a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f701045d0 0x7f9f701977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:31.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f75c8a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70106f00 0x7f9f7019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:31.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f75c8a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7019eda0 con 0x7f9f70106f00 2024-03-12T05:53:31.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f75c8a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7019ef10 con 0x7f9f70101ca0 2024-03-12T05:53:31.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f75c8a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7019f0d0 con 0x7f9f701045d0 2024-03-12T05:53:31.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f6ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70106f00 0x7f9f7019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:31.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f6ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70106f00 0x7f9f7019e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44904/0 (socket says 172.21.15.76:44904) 2024-03-12T05:53:31.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f6ffff640 1 -- 172.21.15.76:0/2964979834 learned_addr learned my addr 172.21.15.76:0/2964979834 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:31.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f6effd640 1 --2- 172.21.15.76:0/2964979834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f701045d0 0x7f9f701977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:31.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f6f7fe640 1 --2- 172.21.15.76:0/2964979834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9f70101ca0 0x7f9f701972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:31.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f6ffff640 1 -- 172.21.15.76:0/2964979834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f701045d0 msgr2=0x7f9f701977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:31.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f6ffff640 1 --2- 172.21.15.76:0/2964979834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f701045d0 0x7f9f701977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:31.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f6ffff640 1 -- 172.21.15.76:0/2964979834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9f70101ca0 msgr2=0x7f9f701972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f6ffff640 1 --2- 172.21.15.76:0/2964979834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9f70101ca0 0x7f9f701972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.499+0000 7f9f6ffff640 1 -- 172.21.15.76:0/2964979834 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f64009560 con 0x7f9f70106f00 2024-03-12T05:53:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.500+0000 7f9f6ffff640 1 --2- 172.21.15.76:0/2964979834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70106f00 0x7f9f7019e860 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9f6800b970 tx=0x7f9f6800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.500+0000 7f9f6cff9640 1 -- 172.21.15.76:0/2964979834 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6800cb40 con 0x7f9f70106f00 2024-03-12T05:53:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.500+0000 7f9f6cff9640 1 -- 172.21.15.76:0/2964979834 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f6800cca0 con 0x7f9f70106f00 2024-03-12T05:53:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.500+0000 7f9f75c8a640 1 -- 172.21.15.76:0/2964979834 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f7019f3b0 con 0x7f9f70106f00 2024-03-12T05:53:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.500+0000 7f9f6cff9640 1 -- 172.21.15.76:0/2964979834 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f680077d0 con 0x7f9f70106f00 2024-03-12T05:53:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.500+0000 7f9f75c8a640 1 -- 172.21.15.76:0/2964979834 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f7019f8c0 con 0x7f9f70106f00 2024-03-12T05:53:31.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.501+0000 7f9f75c8a640 1 -- 172.21.15.76:0/2964979834 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f700fd1a0 con 0x7f9f70106f00 2024-03-12T05:53:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.502+0000 7f9f6cff9640 1 -- 172.21.15.76:0/2964979834 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9f68014440 con 0x7f9f70106f00 2024-03-12T05:53:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.502+0000 7f9f6cff9640 1 --2- 172.21.15.76:0/2964979834 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9f580a1a30 0x7f9f580a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.503+0000 7f9f6f7fe640 1 --2- 172.21.15.76:0/2964979834 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9f580a1a30 0x7f9f580a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.503+0000 7f9f6cff9640 1 -- 172.21.15.76:0/2964979834 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9f680cccb0 con 0x7f9f70106f00 2024-03-12T05:53:31.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.503+0000 7f9f6f7fe640 1 --2- 172.21.15.76:0/2964979834 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9f580a1a30 0x7f9f580a3ef0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9f60005fd0 tx=0x7f9f6000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:31.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.506+0000 7f9f6cff9640 1 -- 172.21.15.76:0/2964979834 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9f68096410 con 0x7f9f70106f00 2024-03-12T05:53:31.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.714+0000 7f9f75c8a640 1 -- 172.21.15.76:0/2964979834 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f7010cc20 con 0x7f9f70106f00 2024-03-12T05:53:31.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.715+0000 7f9f6cff9640 1 -- 172.21.15.76:0/2964979834 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9236 (secure 0 0 0) 0x7f9f68099ad0 con 0x7f9f70106f00 2024-03-12T05:53:31.718 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:31.718 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-12T05:53:31.718 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 12 2024-03-12T05:53:31.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.717+0000 7f9f75c8a640 1 -- 172.21.15.76:0/2964979834 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9f580a1a30 msgr2=0x7f9f580a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:31.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.717+0000 7f9f75c8a640 1 --2- 172.21.15.76:0/2964979834 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9f580a1a30 0x7f9f580a3ef0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9f60005fd0 tx=0x7f9f6000b390 comp rx=0 tx=0).stop 2024-03-12T05:53:31.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.717+0000 7f9f75c8a640 1 -- 172.21.15.76:0/2964979834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70106f00 msgr2=0x7f9f7019e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:31.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.717+0000 7f9f75c8a640 1 --2- 172.21.15.76:0/2964979834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70106f00 0x7f9f7019e860 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9f6800b970 tx=0x7f9f6800be30 comp rx=0 tx=0).stop 2024-03-12T05:53:31.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.717+0000 7f9f75c8a640 1 -- 172.21.15.76:0/2964979834 shutdown_connections 2024-03-12T05:53:31.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.717+0000 7f9f75c8a640 1 --2- 172.21.15.76:0/2964979834 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f9f580a1a30 0x7f9f580a3ef0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:31.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.717+0000 7f9f75c8a640 1 --2- 172.21.15.76:0/2964979834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f70106f00 0x7f9f7019e860 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:31.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.717+0000 7f9f75c8a640 1 --2- 172.21.15.76:0/2964979834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f701045d0 0x7f9f701977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:31.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.717+0000 7f9f75c8a640 1 --2- 172.21.15.76:0/2964979834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9f70101ca0 0x7f9f701972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:31.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.717+0000 7f9f75c8a640 1 -- 172.21.15.76:0/2964979834 >> 172.21.15.76:0/2964979834 conn(0x7f9f700fb6b0 msgr2=0x7f9f701026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:31.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.717+0000 7f9f75c8a640 1 -- 172.21.15.76:0/2964979834 shutdown_connections 2024-03-12T05:53:31.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:31.717+0000 7f9f75c8a640 1 -- 172.21.15.76:0/2964979834 wait complete. 2024-03-12T05:53:31.864 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 05:53:31 smithi082 systemd[1]: Starting Ceph mds.b for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T05:53:31.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:31 smithi076 ceph-mon[32095]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:31.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:31 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2964979834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:32.121 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:31 smithi082 ceph-mon[36382]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:32.121 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:31 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2964979834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:32.121 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 05:53:31 smithi082 podman[66533]: 2024-03-12 05:53:31.76426061 +0000 UTC m=+0.018077105 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T05:53:32.121 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 05:53:31 smithi082 podman[66533]: 2024-03-12 05:53:31.866485288 +0000 UTC m=+0.120301807 container create 1336ba7664a1854fa67575a981439d68ca11ff76f8b29db9dea9c69f580a524c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-12T05:53:32.122 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 05:53:32 smithi082 podman[66533]: 2024-03-12 05:53:32.119036236 +0000 UTC m=+0.372852717 container init 1336ba7664a1854fa67575a981439d68ca11ff76f8b29db9dea9c69f580a524c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-12T05:53:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:31 smithi088 ceph-mon[36585]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:32.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:31 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2964979834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:32.444 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 05:53:32 smithi082 podman[66533]: 2024-03-12 05:53:32.12177399 +0000 UTC m=+0.375590462 container start 1336ba7664a1854fa67575a981439d68ca11ff76f8b29db9dea9c69f580a524c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-12T05:53:32.444 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 05:53:32 smithi082 ceph-mds[66552]: -- 172.21.15.82:0/3375354390 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dcf47d01a0 con 0x55dcf47a7000 2024-03-12T05:53:32.445 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 05:53:32 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b[66548]: starting mds.b at 2024-03-12T05:53:32.445 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 05:53:32 smithi082 bash[66533]: 1336ba7664a1854fa67575a981439d68ca11ff76f8b29db9dea9c69f580a524c 2024-03-12T05:53:32.445 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 05:53:32 smithi082 systemd[1]: Started Ceph mds.b for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T05:53:33.433 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:33 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.433 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:33 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.433 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:33 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.433 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:33 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.433 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:33 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.433 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:33 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:53:33.433 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:33 smithi082 ceph-mon[36382]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:33.433 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:33 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] up:boot 2024-03-12T05:53:33.433 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:33 smithi082 ceph-mon[36382]: fsmap 12 up:standby 2024-03-12T05:53:33.433 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:33 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:53:33.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:33 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:33 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:33 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:33 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:33 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:33 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:53:33.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:33 smithi076 ceph-mon[32095]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:33.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:33 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] up:boot 2024-03-12T05:53:33.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:33 smithi076 ceph-mon[32095]: fsmap 12 up:standby 2024-03-12T05:53:33.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:33 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:53:33.732 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:33 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:33 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:33 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:33 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:33 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:33 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:53:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:33 smithi088 ceph-mon[36585]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:33 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] up:boot 2024-03-12T05:53:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:33 smithi088 ceph-mon[36585]: fsmap 12 up:standby 2024-03-12T05:53:33.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:33 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:53:33.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.853+0000 7fdbfc22a640 1 -- 172.21.15.76:0/3245684389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf4100d30 msgr2=0x7fdbf41011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.853+0000 7fdbfc22a640 1 --2- 172.21.15.76:0/3245684389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf4100d30 0x7fdbf41011b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009ca0 tx=0x7fdbec02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:53:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.853+0000 7fdbfc22a640 1 -- 172.21.15.76:0/3245684389 shutdown_connections 2024-03-12T05:53:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.853+0000 7fdbfc22a640 1 --2- 172.21.15.76:0/3245684389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf4100d30 0x7fdbf41011b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.853+0000 7fdbfc22a640 1 --2- 172.21.15.76:0/3245684389 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdbf4100390 0x7fdbf41007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.853+0000 7fdbfc22a640 1 --2- 172.21.15.76:0/3245684389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf40ffa30 0x7fdbf40ffe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.853+0000 7fdbfc22a640 1 -- 172.21.15.76:0/3245684389 >> 172.21.15.76:0/3245684389 conn(0x7fdbf40fb610 msgr2=0x7fdbf40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.853+0000 7fdbfc22a640 1 -- 172.21.15.76:0/3245684389 shutdown_connections 2024-03-12T05:53:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.853+0000 7fdbfc22a640 1 -- 172.21.15.76:0/3245684389 wait complete. 2024-03-12T05:53:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.854+0000 7fdbfc22a640 1 Processor -- start 2024-03-12T05:53:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.854+0000 7fdbfc22a640 1 -- start start 2024-03-12T05:53:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.854+0000 7fdbfc22a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf40ffa30 0x7fdbf4199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.854+0000 7fdbfc22a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdbf4100390 0x7fdbf41998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:33.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.854+0000 7fdbfc22a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf4100d30 0x7fdbf41a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.854+0000 7fdbfc22a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf41a0e60 con 0x7fdbf4100d30 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.854+0000 7fdbfc22a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf41a0fd0 con 0x7fdbf4100390 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.854+0000 7fdbfc22a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf41a1190 con 0x7fdbf40ffa30 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.854+0000 7fdbfa7c7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf4100d30 0x7fdbf41a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.854+0000 7fdbfa7c7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf4100d30 0x7fdbf41a0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44936/0 (socket says 172.21.15.76:44936) 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.854+0000 7fdbfa7c7640 1 -- 172.21.15.76:0/391005266 learned_addr learned my addr 172.21.15.76:0/391005266 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.854+0000 7fdbf9fc6640 1 --2- 172.21.15.76:0/391005266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf40ffa30 0x7fdbf4199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.854+0000 7fdbf97c5640 1 --2- 172.21.15.76:0/391005266 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdbf4100390 0x7fdbf41998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.855+0000 7fdbfa7c7640 1 -- 172.21.15.76:0/391005266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf40ffa30 msgr2=0x7fdbf4199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.855+0000 7fdbfa7c7640 1 --2- 172.21.15.76:0/391005266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf40ffa30 0x7fdbf4199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.855+0000 7fdbfa7c7640 1 -- 172.21.15.76:0/391005266 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdbf4100390 msgr2=0x7fdbf41998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.855+0000 7fdbfa7c7640 1 --2- 172.21.15.76:0/391005266 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdbf4100390 0x7fdbf41998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.855+0000 7fdbfa7c7640 1 -- 172.21.15.76:0/391005266 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbec009900 con 0x7fdbf4100d30 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.855+0000 7fdbfa7c7640 1 --2- 172.21.15.76:0/391005266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf4100d30 0x7fdbf41a0920 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdbec037040 tx=0x7fdbec009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:33.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.855+0000 7fdbeaffd640 1 -- 172.21.15.76:0/391005266 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec0329b0 con 0x7fdbf4100d30 2024-03-12T05:53:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.855+0000 7fdbeaffd640 1 -- 172.21.15.76:0/391005266 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbec003e50 con 0x7fdbf4100d30 2024-03-12T05:53:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.855+0000 7fdbfc22a640 1 -- 172.21.15.76:0/391005266 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf41a1410 con 0x7fdbf4100d30 2024-03-12T05:53:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.855+0000 7fdbeaffd640 1 -- 172.21.15.76:0/391005266 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec0048a0 con 0x7fdbf4100d30 2024-03-12T05:53:33.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.855+0000 7fdbfc22a640 1 -- 172.21.15.76:0/391005266 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf41a18c0 con 0x7fdbf4100d30 2024-03-12T05:53:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.856+0000 7fdbfc22a640 1 -- 172.21.15.76:0/391005266 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf410a890 con 0x7fdbf4100d30 2024-03-12T05:53:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.857+0000 7fdbeaffd640 1 -- 172.21.15.76:0/391005266 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fdbec00a040 con 0x7fdbf4100d30 2024-03-12T05:53:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.857+0000 7fdbeaffd640 1 --2- 172.21.15.76:0/391005266 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdbe00a1a30 0x7fdbe00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.858+0000 7fdbf9fc6640 1 --2- 172.21.15.76:0/391005266 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdbe00a1a30 0x7fdbe00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.858+0000 7fdbeaffd640 1 -- 172.21.15.76:0/391005266 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdbec0efcd0 con 0x7fdbf4100d30 2024-03-12T05:53:33.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.858+0000 7fdbf9fc6640 1 --2- 172.21.15.76:0/391005266 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdbe00a1a30 0x7fdbe00a3ef0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4009bd0 tx=0x7fdbe4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:33.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:33.862+0000 7fdbeaffd640 1 -- 172.21.15.76:0/391005266 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdbec0b9410 con 0x7fdbf4100d30 2024-03-12T05:53:34.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.133+0000 7fdbfc22a640 1 -- 172.21.15.76:0/391005266 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbf402ce40 con 0x7fdbf4100d30 2024-03-12T05:53:34.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.133+0000 7fdbeaffd640 1 -- 172.21.15.76:0/391005266 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10028 (secure 0 0 0) 0x7fdbec034e60 con 0x7fdbf4100d30 2024-03-12T05:53:34.138 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:34.138 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24518,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-12T05:53:34.138 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 13 2024-03-12T05:53:34.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.137+0000 7fdbfc22a640 1 -- 172.21.15.76:0/391005266 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdbe00a1a30 msgr2=0x7fdbe00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.137+0000 7fdbfc22a640 1 --2- 172.21.15.76:0/391005266 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdbe00a1a30 0x7fdbe00a3ef0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4009bd0 tx=0x7fdbe4005f90 comp rx=0 tx=0).stop 2024-03-12T05:53:34.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.137+0000 7fdbfc22a640 1 -- 172.21.15.76:0/391005266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf4100d30 msgr2=0x7fdbf41a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.137+0000 7fdbfc22a640 1 --2- 172.21.15.76:0/391005266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf4100d30 0x7fdbf41a0920 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdbec037040 tx=0x7fdbec009360 comp rx=0 tx=0).stop 2024-03-12T05:53:34.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.137+0000 7fdbfc22a640 1 -- 172.21.15.76:0/391005266 shutdown_connections 2024-03-12T05:53:34.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.137+0000 7fdbfc22a640 1 --2- 172.21.15.76:0/391005266 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdbe00a1a30 0x7fdbe00a3ef0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.137+0000 7fdbfc22a640 1 --2- 172.21.15.76:0/391005266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf4100d30 0x7fdbf41a0920 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.137+0000 7fdbfc22a640 1 --2- 172.21.15.76:0/391005266 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdbf4100390 0x7fdbf41998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.137+0000 7fdbfc22a640 1 --2- 172.21.15.76:0/391005266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbf40ffa30 0x7fdbf4199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.137+0000 7fdbfc22a640 1 -- 172.21.15.76:0/391005266 >> 172.21.15.76:0/391005266 conn(0x7fdbf40fb610 msgr2=0x7fdbf40fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:34.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.137+0000 7fdbfc22a640 1 -- 172.21.15.76:0/391005266 shutdown_connections 2024-03-12T05:53:34.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.137+0000 7fdbfc22a640 1 -- 172.21.15.76:0/391005266 wait complete. 2024-03-12T05:53:34.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:53:34.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.257+0000 7f54253d0640 1 -- 172.21.15.76:0/3665917759 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54201012a0 msgr2=0x7f5420101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.257+0000 7f54253d0640 1 --2- 172.21.15.76:0/3665917759 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54201012a0 0x7f5420101720 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5410009900 tx=0x7f541002bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:34.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.257+0000 7f54253d0640 1 -- 172.21.15.76:0/3665917759 shutdown_connections 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.257+0000 7f54253d0640 1 --2- 172.21.15.76:0/3665917759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5420101c60 0x7f54201084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.257+0000 7f54253d0640 1 --2- 172.21.15.76:0/3665917759 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54201012a0 0x7f5420101720 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.257+0000 7f54253d0640 1 --2- 172.21.15.76:0/3665917759 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54201000b0 0x7f54201004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.257+0000 7f54253d0640 1 -- 172.21.15.76:0/3665917759 >> 172.21.15.76:0/3665917759 conn(0x7f54200fb640 msgr2=0x7f54200fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.257+0000 7f54253d0640 1 -- 172.21.15.76:0/3665917759 shutdown_connections 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.257+0000 7f54253d0640 1 -- 172.21.15.76:0/3665917759 wait complete. 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f54253d0640 1 Processor -- start 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f54253d0640 1 -- start start 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f54253d0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54201000b0 0x7f54201961d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f54253d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54201012a0 0x7f5420196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f54253d0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5420101c60 0x7f542019d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f54253d0640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542019dcc0 con 0x7f5420101c60 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f54253d0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542019de30 con 0x7f54201000b0 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f54253d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542019dff0 con 0x7f54201012a0 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f541f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5420101c60 0x7f542019d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:34.264 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f541f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5420101c60 0x7f542019d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44956/0 (socket says 172.21.15.76:44956) 2024-03-12T05:53:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f541f7fe640 1 -- 172.21.15.76:0/2400952486 learned_addr learned my addr 172.21.15.76:0/2400952486 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f541e7fc640 1 --2- 172.21.15.76:0/2400952486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54201012a0 0x7f5420196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f541effd640 1 --2- 172.21.15.76:0/2400952486 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54201000b0 0x7f54201961d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f541f7fe640 1 -- 172.21.15.76:0/2400952486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54201012a0 msgr2=0x7f5420196710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f541f7fe640 1 --2- 172.21.15.76:0/2400952486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54201012a0 0x7f5420196710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f541f7fe640 1 -- 172.21.15.76:0/2400952486 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54201000b0 msgr2=0x7f54201961d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f541f7fe640 1 --2- 172.21.15.76:0/2400952486 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54201000b0 0x7f54201961d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.258+0000 7f541f7fe640 1 -- 172.21.15.76:0/2400952486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5410009560 con 0x7f5420101c60 2024-03-12T05:53:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.259+0000 7f541f7fe640 1 --2- 172.21.15.76:0/2400952486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5420101c60 0x7f542019d780 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f541800b970 tx=0x7f541800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.259+0000 7f5407fff640 1 -- 172.21.15.76:0/2400952486 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541800cb40 con 0x7f5420101c60 2024-03-12T05:53:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.259+0000 7f54253d0640 1 -- 172.21.15.76:0/2400952486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5420069980 con 0x7f5420101c60 2024-03-12T05:53:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.259+0000 7f5407fff640 1 -- 172.21.15.76:0/2400952486 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f541800cca0 con 0x7f5420101c60 2024-03-12T05:53:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.259+0000 7f5407fff640 1 -- 172.21.15.76:0/2400952486 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54180077d0 con 0x7f5420101c60 2024-03-12T05:53:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.259+0000 7f54253d0640 1 -- 172.21.15.76:0/2400952486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5420069e90 con 0x7f5420101c60 2024-03-12T05:53:34.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.260+0000 7f54253d0640 1 -- 172.21.15.76:0/2400952486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542010bb00 con 0x7f5420101c60 2024-03-12T05:53:34.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.261+0000 7f5407fff640 1 -- 172.21.15.76:0/2400952486 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5418014440 con 0x7f5420101c60 2024-03-12T05:53:34.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.261+0000 7f5407fff640 1 --2- 172.21.15.76:0/2400952486 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f54080a1a30 0x7f54080a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:34.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.261+0000 7f541effd640 1 --2- 172.21.15.76:0/2400952486 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f54080a1a30 0x7f54080a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:34.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.261+0000 7f5407fff640 1 -- 172.21.15.76:0/2400952486 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f54180cccb0 con 0x7f5420101c60 2024-03-12T05:53:34.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.262+0000 7f541effd640 1 --2- 172.21.15.76:0/2400952486 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f54080a1a30 0x7f54080a3ef0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5414005db0 tx=0x7f5414005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:34.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.266+0000 7f5407fff640 1 -- 172.21.15.76:0/2400952486 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5418096410 con 0x7f5420101c60 2024-03-12T05:53:34.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.478+0000 7f54253d0640 1 -- 172.21.15.76:0/2400952486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f542010bc90 con 0x7f5420101c60 2024-03-12T05:53:34.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.479+0000 7f5407fff640 1 -- 172.21.15.76:0/2400952486 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18347 (secure 0 0 0) 0x7f5418099ad0 con 0x7f5420101c60 2024-03-12T05:53:34.483 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:34.483 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":82,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:52:49.872922+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:53:34.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.481+0000 7f54253d0640 1 -- 172.21.15.76:0/2400952486 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f54080a1a30 msgr2=0x7f54080a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.481+0000 7f54253d0640 1 --2- 172.21.15.76:0/2400952486 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f54080a1a30 0x7f54080a3ef0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5414005db0 tx=0x7f5414005d40 comp rx=0 tx=0).stop 2024-03-12T05:53:34.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.481+0000 7f54253d0640 1 -- 172.21.15.76:0/2400952486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5420101c60 msgr2=0x7f542019d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.481+0000 7f54253d0640 1 --2- 172.21.15.76:0/2400952486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5420101c60 0x7f542019d780 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f541800b970 tx=0x7f541800be30 comp rx=0 tx=0).stop 2024-03-12T05:53:34.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.481+0000 7f54253d0640 1 -- 172.21.15.76:0/2400952486 shutdown_connections 2024-03-12T05:53:34.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.481+0000 7f54253d0640 1 --2- 172.21.15.76:0/2400952486 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f54080a1a30 0x7f54080a3ef0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.481+0000 7f54253d0640 1 --2- 172.21.15.76:0/2400952486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5420101c60 0x7f542019d780 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.481+0000 7f54253d0640 1 --2- 172.21.15.76:0/2400952486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54201012a0 0x7f5420196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.481+0000 7f54253d0640 1 --2- 172.21.15.76:0/2400952486 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f54201000b0 0x7f54201961d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.481+0000 7f54253d0640 1 -- 172.21.15.76:0/2400952486 >> 172.21.15.76:0/2400952486 conn(0x7f54200fb640 msgr2=0x7f54201045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:34.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.481+0000 7f54253d0640 1 -- 172.21.15.76:0/2400952486 shutdown_connections 2024-03-12T05:53:34.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.481+0000 7f54253d0640 1 -- 172.21.15.76:0/2400952486 wait complete. 2024-03-12T05:53:34.497 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:49:09.125536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:53:34.497 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:53:34.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.599+0000 7fb04fe8d640 1 -- 172.21.15.76:0/3810470384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb048101290 msgr2=0x7fb048101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.599+0000 7fb04fe8d640 1 --2- 172.21.15.76:0/3810470384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb048101290 0x7fb048101710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb038009900 tx=0x7fb03802bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:34.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.599+0000 7fb04fe8d640 1 -- 172.21.15.76:0/3810470384 shutdown_connections 2024-03-12T05:53:34.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.599+0000 7fb04fe8d640 1 --2- 172.21.15.76:0/3810470384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb048101c50 0x7fb0481084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.599+0000 7fb04fe8d640 1 --2- 172.21.15.76:0/3810470384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb048101290 0x7fb048101710 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.599+0000 7fb04fe8d640 1 --2- 172.21.15.76:0/3810470384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0481000a0 0x7fb0481004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.599+0000 7fb04fe8d640 1 -- 172.21.15.76:0/3810470384 >> 172.21.15.76:0/3810470384 conn(0x7fb0480fb610 msgr2=0x7fb0480fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:34.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.599+0000 7fb04fe8d640 1 -- 172.21.15.76:0/3810470384 shutdown_connections 2024-03-12T05:53:34.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.599+0000 7fb04fe8d640 1 -- 172.21.15.76:0/3810470384 wait complete. 2024-03-12T05:53:34.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04fe8d640 1 Processor -- start 2024-03-12T05:53:34.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04fe8d640 1 -- start start 2024-03-12T05:53:34.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04fe8d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0481000a0 0x7fb048196150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04fe8d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb048101290 0x7fb048196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04fe8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb048101c50 0x7fb04819d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04fe8d640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04819dc40 con 0x7fb0481000a0 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04fe8d640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04819ddb0 con 0x7fb048101290 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04fe8d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04819df70 con 0x7fb048101c50 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04dc29640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0481000a0 0x7fb048196150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04dc29640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0481000a0 0x7fb048196150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44972/0 (socket says 172.21.15.76:44972) 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04dc29640 1 -- 172.21.15.76:0/273303447 learned_addr learned my addr 172.21.15.76:0/273303447 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04e42a640 1 --2- 172.21.15.76:0/273303447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb048101c50 0x7fb04819d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04d428640 1 --2- 172.21.15.76:0/273303447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb048101290 0x7fb048196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04dc29640 1 -- 172.21.15.76:0/273303447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb048101c50 msgr2=0x7fb04819d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04dc29640 1 --2- 172.21.15.76:0/273303447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb048101c50 0x7fb04819d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04dc29640 1 -- 172.21.15.76:0/273303447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb048101290 msgr2=0x7fb048196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04dc29640 1 --2- 172.21.15.76:0/273303447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb048101290 0x7fb048196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.600+0000 7fb04dc29640 1 -- 172.21.15.76:0/273303447 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb038009560 con 0x7fb0481000a0 2024-03-12T05:53:34.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.601+0000 7fb04dc29640 1 --2- 172.21.15.76:0/273303447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0481000a0 0x7fb048196150 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb04400e9c0 tx=0x7fb04400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:34.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.601+0000 7fb03effd640 1 -- 172.21.15.76:0/273303447 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0440096e0 con 0x7fb0481000a0 2024-03-12T05:53:34.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.601+0000 7fb04fe8d640 1 -- 172.21.15.76:0/273303447 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04819e250 con 0x7fb0481000a0 2024-03-12T05:53:34.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.601+0000 7fb03effd640 1 -- 172.21.15.76:0/273303447 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb04400cd80 con 0x7fb0481000a0 2024-03-12T05:53:34.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.601+0000 7fb03effd640 1 -- 172.21.15.76:0/273303447 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044010430 con 0x7fb0481000a0 2024-03-12T05:53:34.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.601+0000 7fb04fe8d640 1 -- 172.21.15.76:0/273303447 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04819e6e0 con 0x7fb0481000a0 2024-03-12T05:53:34.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.602+0000 7fb04fe8d640 1 -- 172.21.15.76:0/273303447 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb048107410 con 0x7fb0481000a0 2024-03-12T05:53:34.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.603+0000 7fb03effd640 1 -- 172.21.15.76:0/273303447 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb044003680 con 0x7fb0481000a0 2024-03-12T05:53:34.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.603+0000 7fb03effd640 1 --2- 172.21.15.76:0/273303447 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb0340a1a30 0x7fb0340a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:34.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.604+0000 7fb04d428640 1 --2- 172.21.15.76:0/273303447 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb0340a1a30 0x7fb0340a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:34.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.604+0000 7fb03effd640 1 -- 172.21.15.76:0/273303447 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb044014070 con 0x7fb0481000a0 2024-03-12T05:53:34.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.604+0000 7fb04d428640 1 --2- 172.21.15.76:0/273303447 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb0340a1a30 0x7fb0340a3ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb038009230 tx=0x7fb038005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:34.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.607+0000 7fb03effd640 1 -- 172.21.15.76:0/273303447 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb044096a70 con 0x7fb0481000a0 2024-03-12T05:53:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.789+0000 7fb04fe8d640 1 -- 172.21.15.76:0/273303447 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb04810bc60 con 0x7fb0481000a0 2024-03-12T05:53:34.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.789+0000 7fb03effd640 1 -- 172.21.15.76:0/273303447 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb04409a130 con 0x7fb0481000a0 2024-03-12T05:53:34.793 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-12T05:53:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.791+0000 7fb04fe8d640 1 -- 172.21.15.76:0/273303447 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb0340a1a30 msgr2=0x7fb0340a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.791+0000 7fb04fe8d640 1 --2- 172.21.15.76:0/273303447 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb0340a1a30 0x7fb0340a3ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb038009230 tx=0x7fb038005fd0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.791+0000 7fb04fe8d640 1 -- 172.21.15.76:0/273303447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0481000a0 msgr2=0x7fb048196150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.791+0000 7fb04fe8d640 1 --2- 172.21.15.76:0/273303447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0481000a0 0x7fb048196150 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb04400e9c0 tx=0x7fb04400ee80 comp rx=0 tx=0).stop 2024-03-12T05:53:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.791+0000 7fb04fe8d640 1 -- 172.21.15.76:0/273303447 shutdown_connections 2024-03-12T05:53:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.791+0000 7fb04fe8d640 1 --2- 172.21.15.76:0/273303447 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb0340a1a30 0x7fb0340a3ef0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.791+0000 7fb04fe8d640 1 --2- 172.21.15.76:0/273303447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb048101c50 0x7fb04819d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.791+0000 7fb04fe8d640 1 --2- 172.21.15.76:0/273303447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb048101290 0x7fb048196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.791+0000 7fb04fe8d640 1 --2- 172.21.15.76:0/273303447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0481000a0 0x7fb048196150 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.792+0000 7fb04fe8d640 1 -- 172.21.15.76:0/273303447 >> 172.21.15.76:0/273303447 conn(0x7fb0480fb610 msgr2=0x7fb0481045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.792+0000 7fb04fe8d640 1 -- 172.21.15.76:0/273303447 shutdown_connections 2024-03-12T05:53:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.792+0000 7fb04fe8d640 1 -- 172.21.15.76:0/273303447 wait complete. 2024-03-12T05:53:34.807 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-12T05:53:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.918+0000 7fdd0150d640 1 -- 172.21.15.76:0/2908041452 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdcfc1045c0 msgr2=0x7fdcfc1069b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.918+0000 7fdd0150d640 1 --2- 172.21.15.76:0/2908041452 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdcfc1045c0 0x7fdcfc1069b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdce4009900 tx=0x7fdce402bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.918+0000 7fdd0150d640 1 -- 172.21.15.76:0/2908041452 shutdown_connections 2024-03-12T05:53:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.918+0000 7fdd0150d640 1 --2- 172.21.15.76:0/2908041452 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdcfc106ef0 0x7fdcfc1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.918+0000 7fdd0150d640 1 --2- 172.21.15.76:0/2908041452 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdcfc1045c0 0x7fdcfc1069b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.918+0000 7fdd0150d640 1 --2- 172.21.15.76:0/2908041452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcfc101c90 0x7fdcfc104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.918+0000 7fdd0150d640 1 -- 172.21.15.76:0/2908041452 >> 172.21.15.76:0/2908041452 conn(0x7fdcfc0fb620 msgr2=0x7fdcfc0fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.918+0000 7fdd0150d640 1 -- 172.21.15.76:0/2908041452 shutdown_connections 2024-03-12T05:53:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.918+0000 7fdd0150d640 1 -- 172.21.15.76:0/2908041452 wait complete. 2024-03-12T05:53:34.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.919+0000 7fdd0150d640 1 Processor -- start 2024-03-12T05:53:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.919+0000 7fdd0150d640 1 -- start start 2024-03-12T05:53:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.919+0000 7fdd0150d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcfc101c90 0x7fdcfc101600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.919+0000 7fdd0150d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdcfc1045c0 0x7fdcfc0ffc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:34.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.919+0000 7fdd0150d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdcfc106ef0 0x7fdcfc100180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.919+0000 7fdd0150d640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc1008e0 con 0x7fdcfc1045c0 2024-03-12T05:53:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.919+0000 7fdd0150d640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc100a50 con 0x7fdcfc106ef0 2024-03-12T05:53:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.919+0000 7fdd0150d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc1a12d0 con 0x7fdcfc101c90 2024-03-12T05:53:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.919+0000 7fdcf27fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdcfc1045c0 0x7fdcfc0ffc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.919+0000 7fdcf27fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdcfc1045c0 0x7fdcfc0ffc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:44992/0 (socket says 172.21.15.76:44992) 2024-03-12T05:53:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.919+0000 7fdcf27fc640 1 -- 172.21.15.76:0/4150533226 learned_addr learned my addr 172.21.15.76:0/4150533226 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.919+0000 7fdcfb7fe640 1 --2- 172.21.15.76:0/4150533226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdcfc106ef0 0x7fdcfc100180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.919+0000 7fdcfaffd640 1 --2- 172.21.15.76:0/4150533226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcfc101c90 0x7fdcfc101600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.920+0000 7fdcf27fc640 1 -- 172.21.15.76:0/4150533226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcfc101c90 msgr2=0x7fdcfc101600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.920+0000 7fdcf27fc640 1 --2- 172.21.15.76:0/4150533226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcfc101c90 0x7fdcfc101600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.920+0000 7fdcf27fc640 1 -- 172.21.15.76:0/4150533226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdcfc106ef0 msgr2=0x7fdcfc100180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.920+0000 7fdcf27fc640 1 --2- 172.21.15.76:0/4150533226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdcfc106ef0 0x7fdcfc100180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:34.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.920+0000 7fdcf27fc640 1 -- 172.21.15.76:0/4150533226 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce4009560 con 0x7fdcfc1045c0 2024-03-12T05:53:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.920+0000 7fdcf27fc640 1 --2- 172.21.15.76:0/4150533226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdcfc1045c0 0x7fdcfc0ffc40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdce4000c00 tx=0x7fdce4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.920+0000 7fdcf8ff9640 1 -- 172.21.15.76:0/4150533226 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce4004aa0 con 0x7fdcfc1045c0 2024-03-12T05:53:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.920+0000 7fdcf8ff9640 1 -- 172.21.15.76:0/4150533226 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdce40052d0 con 0x7fdcfc1045c0 2024-03-12T05:53:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.920+0000 7fdd0150d640 1 -- 172.21.15.76:0/4150533226 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcfc1a1520 con 0x7fdcfc1045c0 2024-03-12T05:53:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.920+0000 7fdcf8ff9640 1 -- 172.21.15.76:0/4150533226 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce4003e60 con 0x7fdcfc1045c0 2024-03-12T05:53:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.920+0000 7fdd0150d640 1 -- 172.21.15.76:0/4150533226 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcfc1a1a30 con 0x7fdcfc1045c0 2024-03-12T05:53:34.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.921+0000 7fdd0150d640 1 -- 172.21.15.76:0/4150533226 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcfc0fd110 con 0x7fdcfc1045c0 2024-03-12T05:53:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.922+0000 7fdcf8ff9640 1 -- 172.21.15.76:0/4150533226 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fdce4005440 con 0x7fdcfc1045c0 2024-03-12T05:53:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.923+0000 7fdcf8ff9640 1 --2- 172.21.15.76:0/4150533226 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdce80a1a30 0x7fdce80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.923+0000 7fdcf8ff9640 1 -- 172.21.15.76:0/4150533226 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdce40ef400 con 0x7fdcfc1045c0 2024-03-12T05:53:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.923+0000 7fdcfaffd640 1 --2- 172.21.15.76:0/4150533226 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdce80a1a30 0x7fdce80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:34.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.923+0000 7fdcfaffd640 1 --2- 172.21.15.76:0/4150533226 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdce80a1a30 0x7fdce80a3ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdcec009bd0 tx=0x7fdcec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:34.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:34.927+0000 7fdcf8ff9640 1 -- 172.21.15.76:0/4150533226 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdce40b8ab0 con 0x7fdcfc1045c0 2024-03-12T05:53:35.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.111+0000 7fdd0150d640 1 -- 172.21.15.76:0/4150533226 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fdcfc10ce10 con 0x7fdcfc1045c0 2024-03-12T05:53:35.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.111+0000 7fdcf8ff9640 1 -- 172.21.15.76:0/4150533226 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) v1 ==== 84+0+65 (secure 0 0 0) 0x7fdce40bc170 con 0x7fdcfc1045c0 2024-03-12T05:53:35.116 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:35.116 INFO:teuthology.orchestra.run.smithi076.stdout:[ 2024-03-12T05:53:35.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T05:53:35.116 INFO:teuthology.orchestra.run.smithi076.stdout: "poolnum": 1, 2024-03-12T05:53:35.116 INFO:teuthology.orchestra.run.smithi076.stdout: "poolname": ".mgr" 2024-03-12T05:53:35.116 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:53:35.116 INFO:teuthology.orchestra.run.smithi076.stdout:] 2024-03-12T05:53:35.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.113+0000 7fdd0150d640 1 -- 172.21.15.76:0/4150533226 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdce80a1a30 msgr2=0x7fdce80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:35.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.113+0000 7fdd0150d640 1 --2- 172.21.15.76:0/4150533226 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdce80a1a30 0x7fdce80a3ef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdcec009bd0 tx=0x7fdcec005f90 comp rx=0 tx=0).stop 2024-03-12T05:53:35.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.113+0000 7fdd0150d640 1 -- 172.21.15.76:0/4150533226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdcfc1045c0 msgr2=0x7fdcfc0ffc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:35.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.113+0000 7fdd0150d640 1 --2- 172.21.15.76:0/4150533226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdcfc1045c0 0x7fdcfc0ffc40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdce4000c00 tx=0x7fdce4004270 comp rx=0 tx=0).stop 2024-03-12T05:53:35.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.113+0000 7fdd0150d640 1 -- 172.21.15.76:0/4150533226 shutdown_connections 2024-03-12T05:53:35.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.113+0000 7fdd0150d640 1 --2- 172.21.15.76:0/4150533226 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdce80a1a30 0x7fdce80a3ef0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:35.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.113+0000 7fdd0150d640 1 --2- 172.21.15.76:0/4150533226 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdcfc106ef0 0x7fdcfc100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:35.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.113+0000 7fdd0150d640 1 --2- 172.21.15.76:0/4150533226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdcfc1045c0 0x7fdcfc0ffc40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:35.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.113+0000 7fdd0150d640 1 --2- 172.21.15.76:0/4150533226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcfc101c90 0x7fdcfc101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:35.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.113+0000 7fdd0150d640 1 -- 172.21.15.76:0/4150533226 >> 172.21.15.76:0/4150533226 conn(0x7fdcfc0fb620 msgr2=0x7fdcfc1025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:35.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.114+0000 7fdd0150d640 1 -- 172.21.15.76:0/4150533226 shutdown_connections 2024-03-12T05:53:35.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.114+0000 7fdd0150d640 1 -- 172.21.15.76:0/4150533226 wait complete. 2024-03-12T05:53:35.128 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-12T05:53:35.129 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-12T05:53:35.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.233+0000 7f3f04815640 1 -- 172.21.15.76:0/2745270080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f001000a0 msgr2=0x7f3f001004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:35.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.233+0000 7f3f04815640 1 --2- 172.21.15.76:0/2745270080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f001000a0 0x7f3f001004c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0009900 tx=0x7f3ef002bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:35.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.233+0000 7f3f04815640 1 -- 172.21.15.76:0/2745270080 shutdown_connections 2024-03-12T05:53:35.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.233+0000 7f3f04815640 1 --2- 172.21.15.76:0/2745270080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f00101c50 0x7f3f001084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:35.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.233+0000 7f3f04815640 1 --2- 172.21.15.76:0/2745270080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f00101290 0x7f3f00101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:35.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.233+0000 7f3f04815640 1 --2- 172.21.15.76:0/2745270080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f001000a0 0x7f3f001004c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:35.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.233+0000 7f3f04815640 1 -- 172.21.15.76:0/2745270080 >> 172.21.15.76:0/2745270080 conn(0x7f3f000fb610 msgr2=0x7f3f000fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:35.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.233+0000 7f3f04815640 1 -- 172.21.15.76:0/2745270080 shutdown_connections 2024-03-12T05:53:35.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.233+0000 7f3f04815640 1 -- 172.21.15.76:0/2745270080 wait complete. 2024-03-12T05:53:35.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.234+0000 7f3f04815640 1 Processor -- start 2024-03-12T05:53:35.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.234+0000 7f3f04815640 1 -- start start 2024-03-12T05:53:35.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.234+0000 7f3f04815640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f001000a0 0x7f3f00103fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:35.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.234+0000 7f3f04815640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f00101290 0x7f3f00102670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:35.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.234+0000 7f3f04815640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f00101c50 0x7f3f00102bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:35.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.234+0000 7f3f04815640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f00103340 con 0x7f3f001000a0 2024-03-12T05:53:35.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.234+0000 7f3f04815640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f001034b0 con 0x7f3f00101290 2024-03-12T05:53:35.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.234+0000 7f3f04815640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f001a1380 con 0x7f3f00101c50 2024-03-12T05:53:35.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.234+0000 7f3efe59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f001000a0 0x7f3f00103fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:35.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.234+0000 7f3efe59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f001000a0 0x7f3f00103fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45018/0 (socket says 172.21.15.76:45018) 2024-03-12T05:53:35.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.234+0000 7f3efe59c640 1 -- 172.21.15.76:0/3768257035 learned_addr learned my addr 172.21.15.76:0/3768257035 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:35.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.234+0000 7f3ef7fff640 1 --2- 172.21.15.76:0/3768257035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f00101290 0x7f3f00102670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:35.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.234+0000 7f3efed9d640 1 --2- 172.21.15.76:0/3768257035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f00101c50 0x7f3f00102bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:35.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.234+0000 7f3efe59c640 1 -- 172.21.15.76:0/3768257035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f00101c50 msgr2=0x7f3f00102bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:35.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.235+0000 7f3efe59c640 1 --2- 172.21.15.76:0/3768257035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f00101c50 0x7f3f00102bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:35.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.235+0000 7f3efe59c640 1 -- 172.21.15.76:0/3768257035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f00101290 msgr2=0x7f3f00102670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:35.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.235+0000 7f3efe59c640 1 --2- 172.21.15.76:0/3768257035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f00101290 0x7f3f00102670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:35.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.235+0000 7f3efe59c640 1 -- 172.21.15.76:0/3768257035 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef0009560 con 0x7f3f001000a0 2024-03-12T05:53:35.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.235+0000 7f3efe59c640 1 --2- 172.21.15.76:0/3768257035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f001000a0 0x7f3f00103fe0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0000c00 tx=0x7f3ef0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:35.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.235+0000 7f3ef77fe640 1 -- 172.21.15.76:0/3768257035 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0004aa0 con 0x7f3f001000a0 2024-03-12T05:53:35.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.235+0000 7f3f04815640 1 -- 172.21.15.76:0/3768257035 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f001a1580 con 0x7f3f001000a0 2024-03-12T05:53:35.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.235+0000 7f3ef77fe640 1 -- 172.21.15.76:0/3768257035 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ef00052d0 con 0x7f3f001000a0 2024-03-12T05:53:35.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.235+0000 7f3f04815640 1 -- 172.21.15.76:0/3768257035 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f001a1ae0 con 0x7f3f001000a0 2024-03-12T05:53:35.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.235+0000 7f3ef77fe640 1 -- 172.21.15.76:0/3768257035 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0003e60 con 0x7f3f001000a0 2024-03-12T05:53:35.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.236+0000 7f3f04815640 1 -- 172.21.15.76:0/3768257035 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0010bab0 con 0x7f3f001000a0 2024-03-12T05:53:35.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.237+0000 7f3ef77fe640 1 -- 172.21.15.76:0/3768257035 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3ef0005440 con 0x7f3f001000a0 2024-03-12T05:53:35.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.238+0000 7f3ef77fe640 1 --2- 172.21.15.76:0/3768257035 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3eec0a1a30 0x7f3eec0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:35.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.238+0000 7f3ef7fff640 1 --2- 172.21.15.76:0/3768257035 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3eec0a1a30 0x7f3eec0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:35.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.238+0000 7f3ef77fe640 1 -- 172.21.15.76:0/3768257035 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3ef00ef1c0 con 0x7f3f001000a0 2024-03-12T05:53:35.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.238+0000 7f3ef7fff640 1 --2- 172.21.15.76:0/3768257035 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3eec0a1a30 0x7f3eec0a3ef0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3f00103d50 tx=0x7f3ee8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:35.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.242+0000 7f3ef77fe640 1 -- 172.21.15.76:0/3768257035 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3ef00b8870 con 0x7f3f001000a0 2024-03-12T05:53:35.370 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:35 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.370 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:35 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.370 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:35 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:35 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:35 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/391005266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:35.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:35 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:35 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:35 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:35.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:35 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:53:35.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:35 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:35 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:53:35.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:35 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2400952486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:35.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:35 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/273303447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:53:35.372 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:35 smithi076 ceph-mon[32095]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:35.383 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:35 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.383 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:35 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.383 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:35 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.383 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:35 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.383 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:35 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/391005266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:35.383 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:35 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.383 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:35 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.383 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:35 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:35.383 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:35 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:53:35.384 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:35 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.384 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:35 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:53:35.384 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:35 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2400952486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:35.384 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:35 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/273303447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:53:35.384 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:35 smithi088 ceph-mon[36585]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:35.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:35.423+0000 7f3f04815640 1 -- 172.21.15.76:0/3768257035 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3f0010bc40 con 0x7f3f001000a0 2024-03-12T05:53:35.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:35 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:35 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:35 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:35 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:35 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/391005266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:35 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:35 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:35 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:35 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:53:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:35 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:35 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:53:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:35 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2400952486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:35 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/273303447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:53:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:35 smithi082 ceph-mon[36382]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:36.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.183+0000 7f3ef77fe640 1 -- 172.21.15.76:0/3768257035 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3ef00bbf30 con 0x7f3f001000a0 2024-03-12T05:53:36.187 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cephfs_metadata' created 2024-03-12T05:53:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.185+0000 7f3f04815640 1 -- 172.21.15.76:0/3768257035 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3eec0a1a30 msgr2=0x7f3eec0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.185+0000 7f3f04815640 1 --2- 172.21.15.76:0/3768257035 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3eec0a1a30 0x7f3eec0a3ef0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3f00103d50 tx=0x7f3ee8005f90 comp rx=0 tx=0).stop 2024-03-12T05:53:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.185+0000 7f3f04815640 1 -- 172.21.15.76:0/3768257035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f001000a0 msgr2=0x7f3f00103fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.185+0000 7f3f04815640 1 --2- 172.21.15.76:0/3768257035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f001000a0 0x7f3f00103fe0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0000c00 tx=0x7f3ef0004270 comp rx=0 tx=0).stop 2024-03-12T05:53:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.185+0000 7f3f04815640 1 -- 172.21.15.76:0/3768257035 shutdown_connections 2024-03-12T05:53:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.185+0000 7f3f04815640 1 --2- 172.21.15.76:0/3768257035 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3eec0a1a30 0x7f3eec0a3ef0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.185+0000 7f3f04815640 1 --2- 172.21.15.76:0/3768257035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f00101c50 0x7f3f00102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.185+0000 7f3f04815640 1 --2- 172.21.15.76:0/3768257035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3f00101290 0x7f3f00102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.185+0000 7f3f04815640 1 --2- 172.21.15.76:0/3768257035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f001000a0 0x7f3f00103fe0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.185+0000 7f3f04815640 1 -- 172.21.15.76:0/3768257035 >> 172.21.15.76:0/3768257035 conn(0x7f3f000fb610 msgr2=0x7f3f000fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.185+0000 7f3f04815640 1 -- 172.21.15.76:0/3768257035 shutdown_connections 2024-03-12T05:53:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.186+0000 7f3f04815640 1 -- 172.21.15.76:0/3768257035 wait complete. 2024-03-12T05:53:36.203 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-12T05:53:36.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.309+0000 7fc3d7488640 1 -- 172.21.15.76:0/4250011328 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3d0102d40 msgr2=0x7fc3d01095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:36.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.309+0000 7fc3d7488640 1 --2- 172.21.15.76:0/4250011328 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3d0102d40 0x7fc3d01095d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8009ca0 tx=0x7fc3c802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:53:36.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.309+0000 7fc3d7488640 1 -- 172.21.15.76:0/4250011328 shutdown_connections 2024-03-12T05:53:36.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.309+0000 7fc3d7488640 1 --2- 172.21.15.76:0/4250011328 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3d0102d40 0x7fc3d01095d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:36.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.309+0000 7fc3d7488640 1 --2- 172.21.15.76:0/4250011328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3d0102380 0x7fc3d0102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:36.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.309+0000 7fc3d7488640 1 --2- 172.21.15.76:0/4250011328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d0101190 0x7fc3d01015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:36.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.309+0000 7fc3d7488640 1 -- 172.21.15.76:0/4250011328 >> 172.21.15.76:0/4250011328 conn(0x7fc3d00fc6c0 msgr2=0x7fc3d00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:36.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.309+0000 7fc3d7488640 1 -- 172.21.15.76:0/4250011328 shutdown_connections 2024-03-12T05:53:36.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.309+0000 7fc3d7488640 1 -- 172.21.15.76:0/4250011328 wait complete. 2024-03-12T05:53:36.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d7488640 1 Processor -- start 2024-03-12T05:53:36.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d7488640 1 -- start start 2024-03-12T05:53:36.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d7488640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3d0101190 0x7fc3d01971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:36.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d7488640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3d0102380 0x7fc3d0197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:36.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d7488640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d0102d40 0x7fc3d019e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:36.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d7488640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d019ecc0 con 0x7fc3d0101190 2024-03-12T05:53:36.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d7488640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d019ee30 con 0x7fc3d0102380 2024-03-12T05:53:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d7488640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d019eff0 con 0x7fc3d0102d40 2024-03-12T05:53:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d5224640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3d0101190 0x7fc3d01971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d5224640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3d0101190 0x7fc3d01971d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45038/0 (socket says 172.21.15.76:45038) 2024-03-12T05:53:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d5224640 1 -- 172.21.15.76:0/1104499165 learned_addr learned my addr 172.21.15.76:0/1104499165 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d4a23640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3d0102380 0x7fc3d0197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d5a25640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d0102d40 0x7fc3d019e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d5224640 1 -- 172.21.15.76:0/1104499165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d0102d40 msgr2=0x7fc3d019e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d5224640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d0102d40 0x7fc3d019e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d5224640 1 -- 172.21.15.76:0/1104499165 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3d0102380 msgr2=0x7fc3d0197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d5224640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3d0102380 0x7fc3d0197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d5224640 1 -- 172.21.15.76:0/1104499165 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c8009900 con 0x7fc3d0101190 2024-03-12T05:53:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d5a25640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d0102d40 0x7fc3d019e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:53:36.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d5224640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3d0101190 0x7fc3d01971d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00e9c0 tx=0x7fc3cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:36.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3c67fc640 1 -- 172.21.15.76:0/1104499165 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc0096e0 con 0x7fc3d0101190 2024-03-12T05:53:36.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3c67fc640 1 -- 172.21.15.76:0/1104499165 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3cc00cd80 con 0x7fc3d0101190 2024-03-12T05:53:36.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3c67fc640 1 -- 172.21.15.76:0/1104499165 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc010430 con 0x7fc3d0101190 2024-03-12T05:53:36.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d7488640 1 -- 172.21.15.76:0/1104499165 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d019f2d0 con 0x7fc3d0101190 2024-03-12T05:53:36.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.310+0000 7fc3d7488640 1 -- 172.21.15.76:0/1104499165 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d019f790 con 0x7fc3d0101190 2024-03-12T05:53:36.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.311+0000 7fc3d7488640 1 -- 172.21.15.76:0/1104499165 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d0108450 con 0x7fc3d0101190 2024-03-12T05:53:36.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.313+0000 7fc3c67fc640 1 -- 172.21.15.76:0/1104499165 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc3cc003680 con 0x7fc3d0101190 2024-03-12T05:53:36.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.313+0000 7fc3c67fc640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc3bc0a1a30 0x7fc3bc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:36.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.313+0000 7fc3d4a23640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc3bc0a1a30 0x7fc3bc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:36.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.314+0000 7fc3c67fc640 1 -- 172.21.15.76:0/1104499165 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fc3cc014070 con 0x7fc3d0101190 2024-03-12T05:53:36.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.314+0000 7fc3d4a23640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc3bc0a1a30 0x7fc3bc0a3ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc3d01986e0 tx=0x7fc3c000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:36.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.318+0000 7fc3c67fc640 1 -- 172.21.15.76:0/1104499165 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc3cc096b20 con 0x7fc3d0101190 2024-03-12T05:53:36.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:36 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4150533226' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-12T05:53:36.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:36 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3768257035' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-12T05:53:36.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:36 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:36 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:36 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:36 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:36 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:36 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4150533226' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-12T05:53:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:36 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3768257035' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-12T05:53:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:36 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:36 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:36 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:36 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:36 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:36 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4150533226' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-12T05:53:36.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:36 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3768257035' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-12T05:53:36.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:36 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:36 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:36 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:36 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:36 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:36.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:36.505+0000 7fc3d7488640 1 -- 172.21.15.76:0/1104499165 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc3d010cba0 con 0x7fc3d0101190 2024-03-12T05:53:37.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.186+0000 7fc3c67fc640 1 -- 172.21.15.76:0/1104499165 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc3cc09a1e0 con 0x7fc3d0101190 2024-03-12T05:53:37.189 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cephfs_data' created 2024-03-12T05:53:37.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.189+0000 7fc3d7488640 1 -- 172.21.15.76:0/1104499165 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc3bc0a1a30 msgr2=0x7fc3bc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:37.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.189+0000 7fc3d7488640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc3bc0a1a30 0x7fc3bc0a3ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc3d01986e0 tx=0x7fc3c000a2d0 comp rx=0 tx=0).stop 2024-03-12T05:53:37.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.189+0000 7fc3d7488640 1 -- 172.21.15.76:0/1104499165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3d0101190 msgr2=0x7fc3d01971d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:37.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.189+0000 7fc3d7488640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3d0101190 0x7fc3d01971d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00e9c0 tx=0x7fc3cc00ee80 comp rx=0 tx=0).stop 2024-03-12T05:53:37.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.189+0000 7fc3d7488640 1 -- 172.21.15.76:0/1104499165 shutdown_connections 2024-03-12T05:53:37.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.189+0000 7fc3d7488640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc3bc0a1a30 0x7fc3bc0a3ef0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:37.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.189+0000 7fc3d7488640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3d0102d40 0x7fc3d019e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:37.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.189+0000 7fc3d7488640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3d0102380 0x7fc3d0197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:37.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.189+0000 7fc3d7488640 1 --2- 172.21.15.76:0/1104499165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3d0101190 0x7fc3d01971d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:37.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.189+0000 7fc3d7488640 1 -- 172.21.15.76:0/1104499165 >> 172.21.15.76:0/1104499165 conn(0x7fc3d00fc6c0 msgr2=0x7fc3d0105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:37.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.189+0000 7fc3d7488640 1 -- 172.21.15.76:0/1104499165 shutdown_connections 2024-03-12T05:53:37.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.189+0000 7fc3d7488640 1 -- 172.21.15.76:0/1104499165 wait complete. 2024-03-12T05:53:37.214 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-12T05:53:37.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.317+0000 7feae0e0b640 1 -- 172.21.15.76:0/3147798175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feadc0ff470 msgr2=0x7feadc0ff890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:37.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.317+0000 7feae0e0b640 1 --2- 172.21.15.76:0/3147798175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feadc0ff470 0x7feadc0ff890 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7feacc009900 tx=0x7feacc02bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:37.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.317+0000 7feae0e0b640 1 -- 172.21.15.76:0/3147798175 shutdown_connections 2024-03-12T05:53:37.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.317+0000 7feae0e0b640 1 --2- 172.21.15.76:0/3147798175 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feadc102e00 0x7feadc109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:37.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.317+0000 7feae0e0b640 1 --2- 172.21.15.76:0/3147798175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feadc102440 0x7feadc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:37.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.317+0000 7feae0e0b640 1 --2- 172.21.15.76:0/3147798175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feadc0ff470 0x7feadc0ff890 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:37.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.317+0000 7feae0e0b640 1 -- 172.21.15.76:0/3147798175 >> 172.21.15.76:0/3147798175 conn(0x7feadc0fada0 msgr2=0x7feadc0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:37.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.317+0000 7feae0e0b640 1 -- 172.21.15.76:0/3147798175 shutdown_connections 2024-03-12T05:53:37.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.317+0000 7feae0e0b640 1 -- 172.21.15.76:0/3147798175 wait complete. 2024-03-12T05:53:37.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.317+0000 7feae0e0b640 1 Processor -- start 2024-03-12T05:53:37.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feae0e0b640 1 -- start start 2024-03-12T05:53:37.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feae0e0b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feadc0ff470 0x7feadc1994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:37.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feae0e0b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feadc102440 0x7feadc199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:37.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feae0e0b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feadc102e00 0x7feadc1a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:37.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feae0e0b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feadc1a0fc0 con 0x7feadc102e00 2024-03-12T05:53:37.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feae0e0b640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feadc1a1130 con 0x7feadc0ff470 2024-03-12T05:53:37.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feae0e0b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feadc1a12f0 con 0x7feadc102440 2024-03-12T05:53:37.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feadad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feadc102e00 0x7feadc1a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:37.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feadad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feadc102e00 0x7feadc1a0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45068/0 (socket says 172.21.15.76:45068) 2024-03-12T05:53:37.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feadad9d640 1 -- 172.21.15.76:0/3509652790 learned_addr learned my addr 172.21.15.76:0/3509652790 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:37.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feada59c640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feadc0ff470 0x7feadc1994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:37.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7fead3fff640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feadc102440 0x7feadc199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:37.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feadad9d640 1 -- 172.21.15.76:0/3509652790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feadc102440 msgr2=0x7feadc199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:37.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feadad9d640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feadc102440 0x7feadc199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:37.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feadad9d640 1 -- 172.21.15.76:0/3509652790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feadc0ff470 msgr2=0x7feadc1994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:37.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feadad9d640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feadc0ff470 0x7feadc1994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:37.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feadad9d640 1 -- 172.21.15.76:0/3509652790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc009560 con 0x7feadc102e00 2024-03-12T05:53:37.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feada59c640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feadc0ff470 0x7feadc1994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:37.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7fead3fff640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feadc102440 0x7feadc199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:37.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feadad9d640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feadc102e00 0x7feadc1a0a80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fead400eaa0 tx=0x7fead400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:37.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7fead37fe640 1 -- 172.21.15.76:0/3509652790 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead400cdb0 con 0x7feadc102e00 2024-03-12T05:53:37.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7fead37fe640 1 -- 172.21.15.76:0/3509652790 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fead4005020 con 0x7feadc102e00 2024-03-12T05:53:37.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feae0e0b640 1 -- 172.21.15.76:0/3509652790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feadc1a15d0 con 0x7feadc102e00 2024-03-12T05:53:37.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7fead37fe640 1 -- 172.21.15.76:0/3509652790 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead40116b0 con 0x7feadc102e00 2024-03-12T05:53:37.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.318+0000 7feae0e0b640 1 -- 172.21.15.76:0/3509652790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feadc1a1b60 con 0x7feadc102e00 2024-03-12T05:53:37.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.319+0000 7feae0e0b640 1 -- 172.21.15.76:0/3509652790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feadc10cc60 con 0x7feadc102e00 2024-03-12T05:53:37.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.320+0000 7fead37fe640 1 -- 172.21.15.76:0/3509652790 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fead4005520 con 0x7feadc102e00 2024-03-12T05:53:37.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.321+0000 7fead37fe640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7feac80a1ab0 0x7feac80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:37.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.321+0000 7feada59c640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7feac80a1ab0 0x7feac80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:37.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.321+0000 7fead37fe640 1 -- 172.21.15.76:0/3509652790 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fead40cd8c0 con 0x7feadc102e00 2024-03-12T05:53:37.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.321+0000 7feada59c640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7feac80a1ab0 0x7feac80a3f70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feacc005ec0 tx=0x7feacc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:37.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.325+0000 7fead37fe640 1 -- 172.21.15.76:0/3509652790 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fead4096d00 con 0x7feadc102e00 2024-03-12T05:53:37.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:37 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:37.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:37 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:37.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:37 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:37.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:37 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:53:37.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:37 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:37.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:37 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3768257035' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-12T05:53:37.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:37 smithi082 ceph-mon[36382]: osdmap e83: 12 total, 12 up, 12 in 2024-03-12T05:53:37.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:37 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1104499165' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-12T05:53:37.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:37 smithi082 ceph-mon[36382]: pgmap v281: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:37 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:37 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:37 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:37 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:53:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:37 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:37 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3768257035' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-12T05:53:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:37 smithi088 ceph-mon[36585]: osdmap e83: 12 total, 12 up, 12 in 2024-03-12T05:53:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:37 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1104499165' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-12T05:53:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:37 smithi088 ceph-mon[36585]: pgmap v281: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:37.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:37 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:37.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:37 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:37.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:37 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:53:37.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:37 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:53:37.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:37 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:53:37.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:37 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3768257035' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-12T05:53:37.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:37 smithi076 ceph-mon[32095]: osdmap e83: 12 total, 12 up, 12 in 2024-03-12T05:53:37.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:37 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1104499165' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-12T05:53:37.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:37 smithi076 ceph-mon[32095]: pgmap v281: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:37.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:37.534+0000 7feae0e0b640 1 -- 172.21.15.76:0/3509652790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7feadc10cdf0 con 0x7feadc102e00 2024-03-12T05:53:38.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.360+0000 7fead37fe640 1 -- 172.21.15.76:0/3509652790 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fead401f070 con 0x7feadc102e00 2024-03-12T05:53:38.364 INFO:teuthology.orchestra.run.smithi076.stderr:new fs with metadata pool 2 and data pool 3 2024-03-12T05:53:38.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.364+0000 7feae0e0b640 1 -- 172.21.15.76:0/3509652790 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7feac80a1ab0 msgr2=0x7feac80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:38.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.364+0000 7feae0e0b640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7feac80a1ab0 0x7feac80a3f70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feacc005ec0 tx=0x7feacc005fd0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.364+0000 7feae0e0b640 1 -- 172.21.15.76:0/3509652790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feadc102e00 msgr2=0x7feadc1a0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:38.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.364+0000 7feae0e0b640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feadc102e00 0x7feadc1a0a80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fead400eaa0 tx=0x7fead400ef60 comp rx=0 tx=0).stop 2024-03-12T05:53:38.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.364+0000 7feae0e0b640 1 -- 172.21.15.76:0/3509652790 shutdown_connections 2024-03-12T05:53:38.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.364+0000 7feae0e0b640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7feac80a1ab0 0x7feac80a3f70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.364+0000 7feae0e0b640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feadc102e00 0x7feadc1a0a80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.364+0000 7feae0e0b640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feadc102440 0x7feadc199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.364+0000 7feae0e0b640 1 --2- 172.21.15.76:0/3509652790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feadc0ff470 0x7feadc1994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.364+0000 7feae0e0b640 1 -- 172.21.15.76:0/3509652790 >> 172.21.15.76:0/3509652790 conn(0x7feadc0fada0 msgr2=0x7feadc100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:38.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.364+0000 7feae0e0b640 1 -- 172.21.15.76:0/3509652790 shutdown_connections 2024-03-12T05:53:38.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.364+0000 7feae0e0b640 1 -- 172.21.15.76:0/3509652790 wait complete. 2024-03-12T05:53:38.383 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:53:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:38 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1104499165' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-12T05:53:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:38 smithi088 ceph-mon[36585]: osdmap e84: 12 total, 12 up, 12 in 2024-03-12T05:53:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:38 smithi088 ceph-mon[36585]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T05:53:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:38 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3509652790' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-12T05:53:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:38 smithi088 ceph-mon[36585]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-12T05:53:38.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:38 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1104499165' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-12T05:53:38.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:38 smithi076 ceph-mon[32095]: osdmap e84: 12 total, 12 up, 12 in 2024-03-12T05:53:38.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:38 smithi076 ceph-mon[32095]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T05:53:38.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:38 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3509652790' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-12T05:53:38.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:38 smithi076 ceph-mon[32095]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-12T05:53:38.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.487+0000 7fc829335640 1 -- 172.21.15.76:0/37358850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc824101c60 msgr2=0x7fc8241084f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:38.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.487+0000 7fc829335640 1 --2- 172.21.15.76:0/37358850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc824101c60 0x7fc8241084f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009ca0 tx=0x7fc81c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.487+0000 7fc829335640 1 -- 172.21.15.76:0/37358850 shutdown_connections 2024-03-12T05:53:38.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.487+0000 7fc829335640 1 --2- 172.21.15.76:0/37358850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc824101c60 0x7fc8241084f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.487+0000 7fc829335640 1 --2- 172.21.15.76:0/37358850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8241012a0 0x7fc824101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.487+0000 7fc829335640 1 --2- 172.21.15.76:0/37358850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8241000b0 0x7fc8241004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.487+0000 7fc829335640 1 -- 172.21.15.76:0/37358850 >> 172.21.15.76:0/37358850 conn(0x7fc8240fb620 msgr2=0x7fc8240fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:38.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.487+0000 7fc829335640 1 -- 172.21.15.76:0/37358850 shutdown_connections 2024-03-12T05:53:38.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.487+0000 7fc829335640 1 -- 172.21.15.76:0/37358850 wait complete. 2024-03-12T05:53:38.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc829335640 1 Processor -- start 2024-03-12T05:53:38.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc829335640 1 -- start start 2024-03-12T05:53:38.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc829335640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8241000b0 0x7fc824068fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:38.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc829335640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8241012a0 0x7fc82419bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:38.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc829335640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc824101c60 0x7fc82419e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:38.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc829335640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82419ec70 con 0x7fc8241000b0 2024-03-12T05:53:38.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc829335640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82419ede0 con 0x7fc8241012a0 2024-03-12T05:53:38.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc829335640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82419efa0 con 0x7fc824101c60 2024-03-12T05:53:38.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc822ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8241000b0 0x7fc824068fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:38.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc822ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8241000b0 0x7fc824068fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45094/0 (socket says 172.21.15.76:45094) 2024-03-12T05:53:38.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc822ffd640 1 -- 172.21.15.76:0/2390276672 learned_addr learned my addr 172.21.15.76:0/2390276672 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:38.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc8237fe640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc824101c60 0x7fc82419e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:38.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc8227fc640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8241012a0 0x7fc82419bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:38.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc822ffd640 1 -- 172.21.15.76:0/2390276672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc824101c60 msgr2=0x7fc82419e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:38.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc822ffd640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc824101c60 0x7fc82419e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc822ffd640 1 -- 172.21.15.76:0/2390276672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8241012a0 msgr2=0x7fc82419bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:38.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc822ffd640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8241012a0 0x7fc82419bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc822ffd640 1 -- 172.21.15.76:0/2390276672 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c009900 con 0x7fc8241000b0 2024-03-12T05:53:38.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc8237fe640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc824101c60 0x7fc82419e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:38.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc8227fc640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8241012a0 0x7fc82419bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:38.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.488+0000 7fc822ffd640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8241000b0 0x7fc824068fe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc81400c970 tx=0x7fc81400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:38.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.489+0000 7fc80bfff640 1 -- 172.21.15.76:0/2390276672 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc814007c10 con 0x7fc8241000b0 2024-03-12T05:53:38.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.489+0000 7fc80bfff640 1 -- 172.21.15.76:0/2390276672 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc814007d70 con 0x7fc8241000b0 2024-03-12T05:53:38.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.489+0000 7fc829335640 1 -- 172.21.15.76:0/2390276672 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82419f280 con 0x7fc8241000b0 2024-03-12T05:53:38.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.489+0000 7fc80bfff640 1 -- 172.21.15.76:0/2390276672 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81400f8a0 con 0x7fc8241000b0 2024-03-12T05:53:38.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.489+0000 7fc829335640 1 -- 172.21.15.76:0/2390276672 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82419f790 con 0x7fc8241000b0 2024-03-12T05:53:38.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.490+0000 7fc829335640 1 -- 172.21.15.76:0/2390276672 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc824107300 con 0x7fc8241000b0 2024-03-12T05:53:38.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.491+0000 7fc80bfff640 1 -- 172.21.15.76:0/2390276672 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc814018af0 con 0x7fc8241000b0 2024-03-12T05:53:38.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.491+0000 7fc80bfff640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc80c0a1a30 0x7fc80c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:38.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.491+0000 7fc8227fc640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc80c0a1a30 0x7fc80c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:38.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.491+0000 7fc80bfff640 1 -- 172.21.15.76:0/2390276672 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc814013070 con 0x7fc8241000b0 2024-03-12T05:53:38.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.492+0000 7fc8227fc640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc80c0a1a30 0x7fc80c0a3ef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc818006fd0 tx=0x7fc818008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:38.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.495+0000 7fc80bfff640 1 -- 172.21.15.76:0/2390276672 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc814096a70 con 0x7fc8241000b0 2024-03-12T05:53:38.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.671+0000 7fc829335640 1 -- 172.21.15.76:0/2390276672 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc82410bb00 con 0x7fc8241000b0 2024-03-12T05:53:38.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.673+0000 7fc80bfff640 1 -- 172.21.15.76:0/2390276672 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22123 (secure 0 0 0) 0x7fc81409a130 con 0x7fc8241000b0 2024-03-12T05:53:38.676 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:38.676 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":85,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:53:37.535533+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:53:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.675+0000 7fc829335640 1 -- 172.21.15.76:0/2390276672 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc80c0a1a30 msgr2=0x7fc80c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.675+0000 7fc829335640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc80c0a1a30 0x7fc80c0a3ef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc818006fd0 tx=0x7fc818008040 comp rx=0 tx=0).stop 2024-03-12T05:53:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.675+0000 7fc829335640 1 -- 172.21.15.76:0/2390276672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8241000b0 msgr2=0x7fc824068fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.675+0000 7fc829335640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8241000b0 0x7fc824068fe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc81400c970 tx=0x7fc81400ce30 comp rx=0 tx=0).stop 2024-03-12T05:53:38.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.675+0000 7fc829335640 1 -- 172.21.15.76:0/2390276672 shutdown_connections 2024-03-12T05:53:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.675+0000 7fc829335640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc80c0a1a30 0x7fc80c0a3ef0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.675+0000 7fc829335640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc824101c60 0x7fc82419e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.675+0000 7fc829335640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc8241012a0 0x7fc82419bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.675+0000 7fc829335640 1 --2- 172.21.15.76:0/2390276672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc8241000b0 0x7fc824068fe0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.675+0000 7fc829335640 1 -- 172.21.15.76:0/2390276672 >> 172.21.15.76:0/2390276672 conn(0x7fc8240fb620 msgr2=0x7fc8241044d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.675+0000 7fc829335640 1 -- 172.21.15.76:0/2390276672 shutdown_connections 2024-03-12T05:53:38.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.675+0000 7fc829335640 1 -- 172.21.15.76:0/2390276672 wait complete. 2024-03-12T05:53:38.690 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:53:38.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:38 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1104499165' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-12T05:53:38.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:38 smithi082 ceph-mon[36382]: osdmap e84: 12 total, 12 up, 12 in 2024-03-12T05:53:38.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:38 smithi082 ceph-mon[36382]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T05:53:38.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:38 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3509652790' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-12T05:53:38.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:38 smithi082 ceph-mon[36382]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-12T05:53:38.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.788+0000 7fd073a81640 1 -- 172.21.15.76:0/1520571590 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c1045d0 msgr2=0x7fd06c1069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:38.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.788+0000 7fd073a81640 1 --2- 172.21.15.76:0/1520571590 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c1045d0 0x7fd06c1069c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009900 tx=0x7fd05c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:38.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.788+0000 7fd073a81640 1 -- 172.21.15.76:0/1520571590 shutdown_connections 2024-03-12T05:53:38.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.788+0000 7fd073a81640 1 --2- 172.21.15.76:0/1520571590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd06c106f00 0x7fd06c109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.788+0000 7fd073a81640 1 --2- 172.21.15.76:0/1520571590 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c1045d0 0x7fd06c1069c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.788+0000 7fd073a81640 1 --2- 172.21.15.76:0/1520571590 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd06c101ca0 0x7fd06c104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.788+0000 7fd073a81640 1 -- 172.21.15.76:0/1520571590 >> 172.21.15.76:0/1520571590 conn(0x7fd06c0fb690 msgr2=0x7fd06c0fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:38.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.788+0000 7fd073a81640 1 -- 172.21.15.76:0/1520571590 shutdown_connections 2024-03-12T05:53:38.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.788+0000 7fd073a81640 1 -- 172.21.15.76:0/1520571590 wait complete. 2024-03-12T05:53:38.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd073a81640 1 Processor -- start 2024-03-12T05:53:38.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd073a81640 1 -- start start 2024-03-12T05:53:38.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd073a81640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd06c101ca0 0x7fd06c197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:38.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd073a81640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd06c1045d0 0x7fd06c197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:38.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd073a81640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c106f00 0x7fd06c19e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:38.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd073a81640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c19ed10 con 0x7fd06c106f00 2024-03-12T05:53:38.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd073a81640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c19ee80 con 0x7fd06c1045d0 2024-03-12T05:53:38.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd073a81640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06c19f040 con 0x7fd06c101ca0 2024-03-12T05:53:38.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd07201e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c106f00 0x7fd06c19e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:38.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd07201e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c106f00 0x7fd06c19e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45104/0 (socket says 172.21.15.76:45104) 2024-03-12T05:53:38.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd07201e640 1 -- 172.21.15.76:0/1966553865 learned_addr learned my addr 172.21.15.76:0/1966553865 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:38.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd07181d640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd06c101ca0 0x7fd06c197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:38.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd07101c640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd06c1045d0 0x7fd06c197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:38.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd07201e640 1 -- 172.21.15.76:0/1966553865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd06c101ca0 msgr2=0x7fd06c197220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:38.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd07201e640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd06c101ca0 0x7fd06c197220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd07201e640 1 -- 172.21.15.76:0/1966553865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd06c1045d0 msgr2=0x7fd06c197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:38.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd07201e640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd06c1045d0 0x7fd06c197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd07201e640 1 -- 172.21.15.76:0/1966553865 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c009560 con 0x7fd06c106f00 2024-03-12T05:53:38.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.789+0000 7fd07181d640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd06c101ca0 0x7fd06c197220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:53:38.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.790+0000 7fd07201e640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c106f00 0x7fd06c19e7d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd06400eaa0 tx=0x7fd06400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:38.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.790+0000 7fd062ffd640 1 -- 172.21.15.76:0/1966553865 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06400cdb0 con 0x7fd06c106f00 2024-03-12T05:53:38.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.790+0000 7fd062ffd640 1 -- 172.21.15.76:0/1966553865 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd064005020 con 0x7fd06c106f00 2024-03-12T05:53:38.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.790+0000 7fd073a81640 1 -- 172.21.15.76:0/1966553865 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd06c19f320 con 0x7fd06c106f00 2024-03-12T05:53:38.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.790+0000 7fd062ffd640 1 -- 172.21.15.76:0/1966553865 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0640116b0 con 0x7fd06c106f00 2024-03-12T05:53:38.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.790+0000 7fd073a81640 1 -- 172.21.15.76:0/1966553865 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd06c19f800 con 0x7fd06c106f00 2024-03-12T05:53:38.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.791+0000 7fd073a81640 1 -- 172.21.15.76:0/1966553865 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06c0fd180 con 0x7fd06c106f00 2024-03-12T05:53:38.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.792+0000 7fd062ffd640 1 -- 172.21.15.76:0/1966553865 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fd064005520 con 0x7fd06c106f00 2024-03-12T05:53:38.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.792+0000 7fd062ffd640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd0580a1a30 0x7fd0580a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:38.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.792+0000 7fd07181d640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd0580a1a30 0x7fd0580a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:38.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.792+0000 7fd062ffd640 1 -- 172.21.15.76:0/1966553865 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd0640cd950 con 0x7fd06c106f00 2024-03-12T05:53:38.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.793+0000 7fd07181d640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd0580a1a30 0x7fd0580a3ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd0680058c0 tx=0x7fd068005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:38.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.796+0000 7fd062ffd640 1 -- 172.21.15.76:0/1966553865 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd064096cb0 con 0x7fd06c106f00 2024-03-12T05:53:38.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.970+0000 7fd073a81640 1 -- 172.21.15.76:0/1966553865 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd06c10cbc0 con 0x7fd06c106f00 2024-03-12T05:53:38.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.972+0000 7fd062ffd640 1 -- 172.21.15.76:0/1966553865 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22123 (secure 0 0 0) 0x7fd06401f020 con 0x7fd06c106f00 2024-03-12T05:53:38.975 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:38.975 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":85,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:53:37.535533+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:53:38.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.974+0000 7fd073a81640 1 -- 172.21.15.76:0/1966553865 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd0580a1a30 msgr2=0x7fd0580a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:38.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.974+0000 7fd073a81640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd0580a1a30 0x7fd0580a3ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd0680058c0 tx=0x7fd068005850 comp rx=0 tx=0).stop 2024-03-12T05:53:38.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.974+0000 7fd073a81640 1 -- 172.21.15.76:0/1966553865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c106f00 msgr2=0x7fd06c19e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:38.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.974+0000 7fd073a81640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c106f00 0x7fd06c19e7d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd06400eaa0 tx=0x7fd06400ef60 comp rx=0 tx=0).stop 2024-03-12T05:53:38.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.974+0000 7fd073a81640 1 -- 172.21.15.76:0/1966553865 shutdown_connections 2024-03-12T05:53:38.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.974+0000 7fd073a81640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd0580a1a30 0x7fd0580a3ef0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.974+0000 7fd073a81640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd06c106f00 0x7fd06c19e7d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.974+0000 7fd073a81640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd06c1045d0 0x7fd06c197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.974+0000 7fd073a81640 1 --2- 172.21.15.76:0/1966553865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd06c101ca0 0x7fd06c197220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:38.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.974+0000 7fd073a81640 1 -- 172.21.15.76:0/1966553865 >> 172.21.15.76:0/1966553865 conn(0x7fd06c0fb690 msgr2=0x7fd06c102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:38.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.974+0000 7fd073a81640 1 -- 172.21.15.76:0/1966553865 shutdown_connections 2024-03-12T05:53:38.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:38.974+0000 7fd073a81640 1 -- 172.21.15.76:0/1966553865 wait complete. 2024-03-12T05:53:38.988 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-12T05:53:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 -- 172.21.15.76:0/1882632158 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4080101290 msgr2=0x7f4080101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 --2- 172.21.15.76:0/1882632158 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4080101290 0x7f4080101710 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4068009900 tx=0x7f406802bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 -- 172.21.15.76:0/1882632158 shutdown_connections 2024-03-12T05:53:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 --2- 172.21.15.76:0/1882632158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4080101c50 0x7f40801084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 --2- 172.21.15.76:0/1882632158 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4080101290 0x7f4080101710 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 --2- 172.21.15.76:0/1882632158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40801000a0 0x7f40801004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 -- 172.21.15.76:0/1882632158 >> 172.21.15.76:0/1882632158 conn(0x7f40800fb610 msgr2=0x7f40800fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:39.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 -- 172.21.15.76:0/1882632158 shutdown_connections 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 -- 172.21.15.76:0/1882632158 wait complete. 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 Processor -- start 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 -- start start 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40801000a0 0x7f4080068fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4080101290 0x7f408019bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4080101c50 0x7f408019e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408019edc0 con 0x7f40801000a0 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408019ef30 con 0x7f4080101c50 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f40871e2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408019f0f0 con 0x7f4080101290 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f4084f7e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40801000a0 0x7f4080068fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f4084f7e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40801000a0 0x7f4080068fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45124/0 (socket says 172.21.15.76:45124) 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.091+0000 7f4084f7e640 1 -- 172.21.15.76:0/2262384204 learned_addr learned my addr 172.21.15.76:0/2262384204 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.092+0000 7f407ffff640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4080101290 0x7f408019bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.092+0000 7f408577f640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4080101c50 0x7f408019e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:39.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.092+0000 7f4084f7e640 1 -- 172.21.15.76:0/2262384204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4080101290 msgr2=0x7f408019bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:39.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.092+0000 7f4084f7e640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4080101290 0x7f408019bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.092+0000 7f4084f7e640 1 -- 172.21.15.76:0/2262384204 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4080101c50 msgr2=0x7f408019e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:39.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.092+0000 7f4084f7e640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4080101c50 0x7f408019e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.092+0000 7f4084f7e640 1 -- 172.21.15.76:0/2262384204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4068009560 con 0x7f40801000a0 2024-03-12T05:53:39.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.092+0000 7f408577f640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4080101c50 0x7f408019e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:53:39.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.092+0000 7f4084f7e640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40801000a0 0x7f4080068fc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f407000e9c0 tx=0x7f407000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:39.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.092+0000 7f407dffb640 1 -- 172.21.15.76:0/2262384204 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40700096e0 con 0x7f40801000a0 2024-03-12T05:53:39.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.092+0000 7f407dffb640 1 -- 172.21.15.76:0/2262384204 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f407000cd80 con 0x7f40801000a0 2024-03-12T05:53:39.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.092+0000 7f40871e2640 1 -- 172.21.15.76:0/2262384204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408019f370 con 0x7f40801000a0 2024-03-12T05:53:39.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.092+0000 7f407dffb640 1 -- 172.21.15.76:0/2262384204 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4070010430 con 0x7f40801000a0 2024-03-12T05:53:39.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.092+0000 7f40871e2640 1 -- 172.21.15.76:0/2262384204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408019f800 con 0x7f40801000a0 2024-03-12T05:53:39.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.093+0000 7f40871e2640 1 -- 172.21.15.76:0/2262384204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4080068000 con 0x7f40801000a0 2024-03-12T05:53:39.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.094+0000 7f407dffb640 1 -- 172.21.15.76:0/2262384204 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4070003680 con 0x7f40801000a0 2024-03-12T05:53:39.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.095+0000 7f407dffb640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f406c0a1a30 0x7f406c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:39.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.095+0000 7f407ffff640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f406c0a1a30 0x7f406c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:39.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.095+0000 7f407dffb640 1 -- 172.21.15.76:0/2262384204 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4070014070 con 0x7f40801000a0 2024-03-12T05:53:39.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.095+0000 7f407ffff640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f406c0a1a30 0x7f406c0a3ef0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4068009230 tx=0x7f4068005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:39.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.099+0000 7f407dffb640 1 -- 172.21.15.76:0/2262384204 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4070096a50 con 0x7f40801000a0 2024-03-12T05:53:39.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:39 smithi076 ceph-mon[32095]: osdmap e85: 12 total, 12 up, 12 in 2024-03-12T05:53:39.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:39 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3509652790' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-12T05:53:39.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:39 smithi076 ceph-mon[32095]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-12T05:53:39.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:39 smithi076 ceph-mon[32095]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-12T05:53:39.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:39 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2390276672' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:39.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:39 smithi076 ceph-mon[32095]: pgmap v284: 97 pgs: 13 active+clean, 21 creating+peering, 63 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:39.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:39 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1966553865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:39.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.311+0000 7f40871e2640 1 -- 172.21.15.76:0/2262384204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4080107440 con 0x7f40801000a0 2024-03-12T05:53:39.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.369+0000 7f407dffb640 1 -- 172.21.15.76:0/2262384204 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f407009a110 con 0x7f40801000a0 2024-03-12T05:53:39.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.372+0000 7f40871e2640 1 -- 172.21.15.76:0/2262384204 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f406c0a1a30 msgr2=0x7f406c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:39.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.372+0000 7f40871e2640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f406c0a1a30 0x7f406c0a3ef0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4068009230 tx=0x7f4068005fd0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.372+0000 7f40871e2640 1 -- 172.21.15.76:0/2262384204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40801000a0 msgr2=0x7f4080068fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:39.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.372+0000 7f40871e2640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40801000a0 0x7f4080068fc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f407000e9c0 tx=0x7f407000ee80 comp rx=0 tx=0).stop 2024-03-12T05:53:39.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.372+0000 7f40871e2640 1 -- 172.21.15.76:0/2262384204 shutdown_connections 2024-03-12T05:53:39.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.372+0000 7f40871e2640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f406c0a1a30 0x7f406c0a3ef0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.372+0000 7f40871e2640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4080101c50 0x7f408019e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.372+0000 7f40871e2640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4080101290 0x7f408019bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.372+0000 7f40871e2640 1 --2- 172.21.15.76:0/2262384204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f40801000a0 0x7f4080068fc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.372+0000 7f40871e2640 1 -- 172.21.15.76:0/2262384204 >> 172.21.15.76:0/2262384204 conn(0x7f40800fb610 msgr2=0x7f4080104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:39.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.372+0000 7f40871e2640 1 -- 172.21.15.76:0/2262384204 shutdown_connections 2024-03-12T05:53:39.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.372+0000 7f40871e2640 1 -- 172.21.15.76:0/2262384204 wait complete. 2024-03-12T05:53:39.387 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-12T05:53:39.387 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-12T05:53:39.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:39 smithi082 ceph-mon[36382]: osdmap e85: 12 total, 12 up, 12 in 2024-03-12T05:53:39.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:39 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3509652790' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-12T05:53:39.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:39 smithi082 ceph-mon[36382]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-12T05:53:39.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:39 smithi082 ceph-mon[36382]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-12T05:53:39.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:39 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2390276672' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:39.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:39 smithi082 ceph-mon[36382]: pgmap v284: 97 pgs: 13 active+clean, 21 creating+peering, 63 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:39.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:39 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1966553865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:39.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:39 smithi088 ceph-mon[36585]: osdmap e85: 12 total, 12 up, 12 in 2024-03-12T05:53:39.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:39 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3509652790' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-12T05:53:39.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:39 smithi088 ceph-mon[36585]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-12T05:53:39.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:39 smithi088 ceph-mon[36585]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-12T05:53:39.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:39 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2390276672' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:39.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:39 smithi088 ceph-mon[36585]: pgmap v284: 97 pgs: 13 active+clean, 21 creating+peering, 63 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:53:39.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:39 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1966553865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:39.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 -- 172.21.15.76:0/2514510415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bf4101290 msgr2=0x7f3bf4101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:39.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 --2- 172.21.15.76:0/2514510415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bf4101290 0x7f3bf4101710 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3be0009900 tx=0x7f3be002bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:39.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 -- 172.21.15.76:0/2514510415 shutdown_connections 2024-03-12T05:53:39.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 --2- 172.21.15.76:0/2514510415 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4101c50 0x7f3bf41084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 --2- 172.21.15.76:0/2514510415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bf4101290 0x7f3bf4101710 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 --2- 172.21.15.76:0/2514510415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf41000a0 0x7f3bf41004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 -- 172.21.15.76:0/2514510415 >> 172.21.15.76:0/2514510415 conn(0x7f3bf40fb610 msgr2=0x7f3bf40fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 -- 172.21.15.76:0/2514510415 shutdown_connections 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 -- 172.21.15.76:0/2514510415 wait complete. 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 Processor -- start 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 -- start start 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf41000a0 0x7f3bf406a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4101290 0x7f3bf419bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bf4101c50 0x7f3bf419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf419ed90 con 0x7f3bf4101c50 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf419ef00 con 0x7f3bf4101290 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.489+0000 7f3bfb20f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf419f0c0 con 0x7f3bf41000a0 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf97ac640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bf4101c50 0x7f3bf419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:39.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf97ac640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bf4101c50 0x7f3bf419e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45130/0 (socket says 172.21.15.76:45130) 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf97ac640 1 -- 172.21.15.76:0/3251948434 learned_addr learned my addr 172.21.15.76:0/3251948434 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf8fab640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf41000a0 0x7f3bf406a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf3fff640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4101290 0x7f3bf419bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf97ac640 1 -- 172.21.15.76:0/3251948434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf41000a0 msgr2=0x7f3bf406a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf97ac640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf41000a0 0x7f3bf406a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf97ac640 1 -- 172.21.15.76:0/3251948434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4101290 msgr2=0x7f3bf419bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf97ac640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4101290 0x7f3bf419bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf97ac640 1 -- 172.21.15.76:0/3251948434 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be0009560 con 0x7f3bf4101c50 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf8fab640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf41000a0 0x7f3bf406a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf3fff640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4101290 0x7f3bf419bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf97ac640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bf4101c50 0x7f3bf419e600 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00cad0 tx=0x7f3bec00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf1ffb640 1 -- 172.21.15.76:0/3251948434 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec00ae20 con 0x7f3bf4101c50 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf1ffb640 1 -- 172.21.15.76:0/3251948434 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bec005020 con 0x7f3bf4101c50 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bfb20f640 1 -- 172.21.15.76:0/3251948434 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf419f3a0 con 0x7f3bf4101c50 2024-03-12T05:53:39.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bf1ffb640 1 -- 172.21.15.76:0/3251948434 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec00f6b0 con 0x7f3bf4101c50 2024-03-12T05:53:39.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.490+0000 7f3bfb20f640 1 -- 172.21.15.76:0/3251948434 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf419f860 con 0x7f3bf4101c50 2024-03-12T05:53:39.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.491+0000 7f3bfb20f640 1 -- 172.21.15.76:0/3251948434 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf4107410 con 0x7f3bf4101c50 2024-03-12T05:53:39.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.492+0000 7f3bf1ffb640 1 -- 172.21.15.76:0/3251948434 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3bec005520 con 0x7f3bf4101c50 2024-03-12T05:53:39.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.493+0000 7f3bf1ffb640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3be40a1a30 0x7f3be40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:39.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.493+0000 7f3bf8fab640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3be40a1a30 0x7f3be40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:39.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.493+0000 7f3bf1ffb640 1 -- 172.21.15.76:0/3251948434 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3bec0cd380 con 0x7f3bf4101c50 2024-03-12T05:53:39.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.493+0000 7f3bf8fab640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3be40a1a30 0x7f3be40a3ef0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3be80058c0 tx=0x7f3be8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:39.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.497+0000 7f3bf1ffb640 1 -- 172.21.15.76:0/3251948434 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3bec0966e0 con 0x7f3bf4101c50 2024-03-12T05:53:39.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:39.707+0000 7f3bfb20f640 1 -- 172.21.15.76:0/3251948434 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f3bf402ce40 con 0x7f3bf4101c50 2024-03-12T05:53:40.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:40 smithi076 ceph-mon[32095]: osdmap e86: 12 total, 12 up, 12 in 2024-03-12T05:53:40.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:40 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2262384204' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-12T05:53:40.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:40 smithi076 ceph-mon[32095]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T05:53:40.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:40 smithi076 ceph-mon[32095]: Cluster is now healthy 2024-03-12T05:53:40.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:40 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2262384204' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-12T05:53:40.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:40 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] up:active 2024-03-12T05:53:40.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:40 smithi076 ceph-mon[32095]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-12T05:53:40.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:40 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3251948434' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-12T05:53:40.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.370+0000 7f3bf1ffb640 1 -- 172.21.15.76:0/3251948434 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v16) v1 ==== 105+0+0 (secure 0 0 0) 0x7f3bec099da0 con 0x7f3bf4101c50 2024-03-12T05:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.373+0000 7f3bfb20f640 1 -- 172.21.15.76:0/3251948434 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3be40a1a30 msgr2=0x7f3be40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.373+0000 7f3bfb20f640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3be40a1a30 0x7f3be40a3ef0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3be80058c0 tx=0x7f3be8005850 comp rx=0 tx=0).stop 2024-03-12T05:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.373+0000 7f3bfb20f640 1 -- 172.21.15.76:0/3251948434 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bf4101c50 msgr2=0x7f3bf419e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.373+0000 7f3bfb20f640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bf4101c50 0x7f3bf419e600 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00cad0 tx=0x7f3bec00cf90 comp rx=0 tx=0).stop 2024-03-12T05:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.373+0000 7f3bfb20f640 1 -- 172.21.15.76:0/3251948434 shutdown_connections 2024-03-12T05:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.373+0000 7f3bfb20f640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3be40a1a30 0x7f3be40a3ef0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.373+0000 7f3bfb20f640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bf4101c50 0x7f3bf419e600 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.373+0000 7f3bfb20f640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3bf4101290 0x7f3bf419bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.373+0000 7f3bfb20f640 1 --2- 172.21.15.76:0/3251948434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bf41000a0 0x7f3bf406a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.373+0000 7f3bfb20f640 1 -- 172.21.15.76:0/3251948434 >> 172.21.15.76:0/3251948434 conn(0x7f3bf40fb610 msgr2=0x7f3bf41045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.373+0000 7f3bfb20f640 1 -- 172.21.15.76:0/3251948434 shutdown_connections 2024-03-12T05:53:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.373+0000 7f3bfb20f640 1 -- 172.21.15.76:0/3251948434 wait complete. 2024-03-12T05:53:40.390 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-12T05:53:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:40 smithi088 ceph-mon[36585]: osdmap e86: 12 total, 12 up, 12 in 2024-03-12T05:53:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:40 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2262384204' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-12T05:53:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:40 smithi088 ceph-mon[36585]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T05:53:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:40 smithi088 ceph-mon[36585]: Cluster is now healthy 2024-03-12T05:53:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:40 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2262384204' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-12T05:53:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:40 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] up:active 2024-03-12T05:53:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:40 smithi088 ceph-mon[36585]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-12T05:53:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:40 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3251948434' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-12T05:53:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.492+0000 7f6b9c565640 1 -- 172.21.15.76:0/2663802003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b941000a0 msgr2=0x7f6b941004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.492+0000 7f6b9c565640 1 --2- 172.21.15.76:0/2663802003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b941000a0 0x7f6b941004c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009900 tx=0x7f6b9002bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.492+0000 7f6b9c565640 1 -- 172.21.15.76:0/2663802003 shutdown_connections 2024-03-12T05:53:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.492+0000 7f6b9c565640 1 --2- 172.21.15.76:0/2663802003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b94101c50 0x7f6b941084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.492+0000 7f6b9c565640 1 --2- 172.21.15.76:0/2663802003 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b94101290 0x7f6b94101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.492+0000 7f6b9c565640 1 --2- 172.21.15.76:0/2663802003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b941000a0 0x7f6b941004c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.492+0000 7f6b9c565640 1 -- 172.21.15.76:0/2663802003 >> 172.21.15.76:0/2663802003 conn(0x7f6b940fb610 msgr2=0x7f6b940fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.492+0000 7f6b9c565640 1 -- 172.21.15.76:0/2663802003 shutdown_connections 2024-03-12T05:53:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.492+0000 7f6b9c565640 1 -- 172.21.15.76:0/2663802003 wait complete. 2024-03-12T05:53:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.493+0000 7f6b9c565640 1 Processor -- start 2024-03-12T05:53:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.493+0000 7f6b9c565640 1 -- start start 2024-03-12T05:53:40.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.493+0000 7f6b9c565640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b941000a0 0x7f6b94196190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:40.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.493+0000 7f6b9c565640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b94101290 0x7f6b941966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.493+0000 7f6b9c565640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94101c50 0x7f6b9419d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.493+0000 7f6b9c565640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9419dc80 con 0x7f6b94101c50 2024-03-12T05:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.493+0000 7f6b9c565640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9419ddf0 con 0x7f6b94101290 2024-03-12T05:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.493+0000 7f6b9c565640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9419dfb0 con 0x7f6b941000a0 2024-03-12T05:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.493+0000 7f6b9ab02640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94101c50 0x7f6b9419d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.493+0000 7f6b9ab02640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94101c50 0x7f6b9419d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45148/0 (socket says 172.21.15.76:45148) 2024-03-12T05:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.493+0000 7f6b9ab02640 1 -- 172.21.15.76:0/921123008 learned_addr learned my addr 172.21.15.76:0/921123008 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.493+0000 7f6b9a301640 1 --2- 172.21.15.76:0/921123008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b941000a0 0x7f6b94196190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.493+0000 7f6b99b00640 1 --2- 172.21.15.76:0/921123008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b94101290 0x7f6b941966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.493+0000 7f6b9ab02640 1 -- 172.21.15.76:0/921123008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b941000a0 msgr2=0x7f6b94196190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.494+0000 7f6b9ab02640 1 --2- 172.21.15.76:0/921123008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b941000a0 0x7f6b94196190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.494+0000 7f6b9ab02640 1 -- 172.21.15.76:0/921123008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b94101290 msgr2=0x7f6b941966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.494+0000 7f6b9ab02640 1 --2- 172.21.15.76:0/921123008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b94101290 0x7f6b941966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.494+0000 7f6b9ab02640 1 -- 172.21.15.76:0/921123008 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b90009560 con 0x7f6b94101c50 2024-03-12T05:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.494+0000 7f6b9ab02640 1 --2- 172.21.15.76:0/921123008 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94101c50 0x7f6b9419d740 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00cad0 tx=0x7f6b8c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.494+0000 7f6b8b7fe640 1 -- 172.21.15.76:0/921123008 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c00ae20 con 0x7f6b94101c50 2024-03-12T05:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.494+0000 7f6b8b7fe640 1 -- 172.21.15.76:0/921123008 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b8c005020 con 0x7f6b94101c50 2024-03-12T05:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.494+0000 7f6b9c565640 1 -- 172.21.15.76:0/921123008 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9419e290 con 0x7f6b94101c50 2024-03-12T05:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.494+0000 7f6b8b7fe640 1 -- 172.21.15.76:0/921123008 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c00f6b0 con 0x7f6b94101c50 2024-03-12T05:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.494+0000 7f6b9c565640 1 -- 172.21.15.76:0/921123008 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9419e7a0 con 0x7f6b94101c50 2024-03-12T05:53:40.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.495+0000 7f6b9c565640 1 -- 172.21.15.76:0/921123008 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b94107490 con 0x7f6b94101c50 2024-03-12T05:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.496+0000 7f6b8b7fe640 1 -- 172.21.15.76:0/921123008 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6b8c005520 con 0x7f6b94101c50 2024-03-12T05:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.496+0000 7f6b8b7fe640 1 --2- 172.21.15.76:0/921123008 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b800a1a30 0x7f6b800a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.496+0000 7f6b9a301640 1 --2- 172.21.15.76:0/921123008 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b800a1a30 0x7f6b800a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.497+0000 7f6b8b7fe640 1 -- 172.21.15.76:0/921123008 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6b8c0cd580 con 0x7f6b94101c50 2024-03-12T05:53:40.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.497+0000 7f6b9a301640 1 --2- 172.21.15.76:0/921123008 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b800a1a30 0x7f6b800a3ef0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009230 tx=0x7f6b90005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:40.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.500+0000 7f6b8b7fe640 1 -- 172.21.15.76:0/921123008 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6b8c096990 con 0x7f6b94101c50 2024-03-12T05:53:40.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:40 smithi082 ceph-mon[36382]: osdmap e86: 12 total, 12 up, 12 in 2024-03-12T05:53:40.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:40 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2262384204' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-12T05:53:40.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:40 smithi082 ceph-mon[36382]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T05:53:40.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:40 smithi082 ceph-mon[36382]: Cluster is now healthy 2024-03-12T05:53:40.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:40 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2262384204' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-12T05:53:40.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:40 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] up:active 2024-03-12T05:53:40.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:40 smithi082 ceph-mon[36382]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-12T05:53:40.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:40 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3251948434' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-12T05:53:40.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:40.712+0000 7f6b9c565640 1 -- 172.21.15.76:0/921123008 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f6b9410bab0 con 0x7f6b94101c50 2024-03-12T05:53:41.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.358+0000 7f6b8b7fe640 1 -- 172.21.15.76:0/921123008 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) v1 ==== 121+0+0 (secure 0 0 0) 0x7f6b8c09a050 con 0x7f6b94101c50 2024-03-12T05:53:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.360+0000 7f6b9c565640 1 -- 172.21.15.76:0/921123008 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b800a1a30 msgr2=0x7f6b800a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.360+0000 7f6b9c565640 1 --2- 172.21.15.76:0/921123008 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b800a1a30 0x7f6b800a3ef0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009230 tx=0x7f6b90005fd0 comp rx=0 tx=0).stop 2024-03-12T05:53:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.360+0000 7f6b9c565640 1 -- 172.21.15.76:0/921123008 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94101c50 msgr2=0x7f6b9419d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.360+0000 7f6b9c565640 1 --2- 172.21.15.76:0/921123008 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94101c50 0x7f6b9419d740 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00cad0 tx=0x7f6b8c00cf90 comp rx=0 tx=0).stop 2024-03-12T05:53:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.360+0000 7f6b9c565640 1 -- 172.21.15.76:0/921123008 shutdown_connections 2024-03-12T05:53:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.360+0000 7f6b9c565640 1 --2- 172.21.15.76:0/921123008 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b800a1a30 0x7f6b800a3ef0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.360+0000 7f6b9c565640 1 --2- 172.21.15.76:0/921123008 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94101c50 0x7f6b9419d740 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.360+0000 7f6b9c565640 1 --2- 172.21.15.76:0/921123008 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b94101290 0x7f6b941966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.360+0000 7f6b9c565640 1 --2- 172.21.15.76:0/921123008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b941000a0 0x7f6b94196190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.360+0000 7f6b9c565640 1 -- 172.21.15.76:0/921123008 >> 172.21.15.76:0/921123008 conn(0x7f6b940fb610 msgr2=0x7f6b94104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.360+0000 7f6b9c565640 1 -- 172.21.15.76:0/921123008 shutdown_connections 2024-03-12T05:53:41.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.360+0000 7f6b9c565640 1 -- 172.21.15.76:0/921123008 wait complete. 2024-03-12T05:53:41.380 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-12T05:53:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:41 smithi088 ceph-mon[36585]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-12T05:53:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:41 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3251948434' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-12T05:53:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:41 smithi088 ceph-mon[36585]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-12T05:53:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:41 smithi088 ceph-mon[36585]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-12T05:53:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:41 smithi088 ceph-mon[36585]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-12T05:53:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:41 smithi088 ceph-mon[36585]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-12T05:53:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:41 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/921123008' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-12T05:53:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:41 smithi088 ceph-mon[36585]: pgmap v286: 97 pgs: 66 active+clean, 21 creating+peering, 10 unknown; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2024-03-12T05:53:41.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.484+0000 7fe08e72c640 1 -- 172.21.15.76:0/1911868341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0881024e0 msgr2=0x7fe088102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:41.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.484+0000 7fe08e72c640 1 --2- 172.21.15.76:0/1911868341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0881024e0 0x7fe088102960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe07c009900 tx=0x7fe07c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:41.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.484+0000 7fe08e72c640 1 -- 172.21.15.76:0/1911868341 shutdown_connections 2024-03-12T05:53:41.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.484+0000 7fe08e72c640 1 --2- 172.21.15.76:0/1911868341 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe088102ea0 0x7fe088109730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:41.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.484+0000 7fe08e72c640 1 --2- 172.21.15.76:0/1911868341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0881024e0 0x7fe088102960 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:41.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.484+0000 7fe08e72c640 1 --2- 172.21.15.76:0/1911868341 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0880ff470 0x7fe0880ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:41.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.484+0000 7fe08e72c640 1 -- 172.21.15.76:0/1911868341 >> 172.21.15.76:0/1911868341 conn(0x7fe0880fada0 msgr2=0x7fe0880fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:41.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.484+0000 7fe08e72c640 1 -- 172.21.15.76:0/1911868341 shutdown_connections 2024-03-12T05:53:41.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.484+0000 7fe08e72c640 1 -- 172.21.15.76:0/1911868341 wait complete. 2024-03-12T05:53:41.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:41 smithi076 ceph-mon[32095]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-12T05:53:41.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:41 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3251948434' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-12T05:53:41.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:41 smithi076 ceph-mon[32095]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-12T05:53:41.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:41 smithi076 ceph-mon[32095]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-12T05:53:41.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:41 smithi076 ceph-mon[32095]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-12T05:53:41.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:41 smithi076 ceph-mon[32095]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-12T05:53:41.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:41 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/921123008' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-12T05:53:41.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:41 smithi076 ceph-mon[32095]: pgmap v286: 97 pgs: 66 active+clean, 21 creating+peering, 10 unknown; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2024-03-12T05:53:41.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.484+0000 7fe08e72c640 1 Processor -- start 2024-03-12T05:53:41.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.484+0000 7fe08e72c640 1 -- start start 2024-03-12T05:53:41.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08e72c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0880ff470 0x7fe088101e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:41.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08e72c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0881024e0 0x7fe088100490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:41.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08e72c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe088102ea0 0x7fe0881009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:41.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08e72c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe088101130 con 0x7fe088102ea0 2024-03-12T05:53:41.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08e72c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0881012a0 con 0x7fe0880ff470 2024-03-12T05:53:41.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08e72c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0881a1260 con 0x7fe0881024e0 2024-03-12T05:53:41.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08ccc9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe088102ea0 0x7fe0881009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:41.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08ccc9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe088102ea0 0x7fe0881009d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53402/0 (socket says 172.21.15.76:53402) 2024-03-12T05:53:41.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08ccc9640 1 -- 172.21.15.76:0/2710460926 learned_addr learned my addr 172.21.15.76:0/2710460926 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:41.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe087fff640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0880ff470 0x7fe088101e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:41.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe0877fe640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0881024e0 0x7fe088100490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:41.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08ccc9640 1 -- 172.21.15.76:0/2710460926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0881024e0 msgr2=0x7fe088100490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:41.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08ccc9640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0881024e0 0x7fe088100490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:41.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08ccc9640 1 -- 172.21.15.76:0/2710460926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0880ff470 msgr2=0x7fe088101e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:41.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08ccc9640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0880ff470 0x7fe088101e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:41.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08ccc9640 1 -- 172.21.15.76:0/2710460926 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c009560 con 0x7fe088102ea0 2024-03-12T05:53:41.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe0877fe640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0881024e0 0x7fe088100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:41.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe08ccc9640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe088102ea0 0x7fe0881009d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe08000ead0 tx=0x7fe08000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:41.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.485+0000 7fe0857fa640 1 -- 172.21.15.76:0/2710460926 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08000cde0 con 0x7fe088102ea0 2024-03-12T05:53:41.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.486+0000 7fe08e72c640 1 -- 172.21.15.76:0/2710460926 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0881a1510 con 0x7fe088102ea0 2024-03-12T05:53:41.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.486+0000 7fe0857fa640 1 -- 172.21.15.76:0/2710460926 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe080005020 con 0x7fe088102ea0 2024-03-12T05:53:41.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.486+0000 7fe0857fa640 1 -- 172.21.15.76:0/2710460926 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0800116b0 con 0x7fe088102ea0 2024-03-12T05:53:41.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.486+0000 7fe08e72c640 1 -- 172.21.15.76:0/2710460926 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0881a19d0 con 0x7fe088102ea0 2024-03-12T05:53:41.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.487+0000 7fe08e72c640 1 -- 172.21.15.76:0/2710460926 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe088108610 con 0x7fe088102ea0 2024-03-12T05:53:41.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.488+0000 7fe0857fa640 1 -- 172.21.15.76:0/2710460926 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fe080005520 con 0x7fe088102ea0 2024-03-12T05:53:41.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.488+0000 7fe0857fa640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe0700a1a30 0x7fe0700a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:41.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.488+0000 7fe087fff640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe0700a1a30 0x7fe0700a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:41.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.488+0000 7fe0857fa640 1 -- 172.21.15.76:0/2710460926 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe0800cd510 con 0x7fe088102ea0 2024-03-12T05:53:41.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.489+0000 7fe087fff640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe0700a1a30 0x7fe0700a3ef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe078005fd0 tx=0x7fe07800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:41.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.493+0000 7fe0857fa640 1 -- 172.21.15.76:0/2710460926 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe080096870 con 0x7fe088102ea0 2024-03-12T05:53:41.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:41 smithi082 ceph-mon[36382]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-12T05:53:41.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:41 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3251948434' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-12T05:53:41.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:41 smithi082 ceph-mon[36382]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-12T05:53:41.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:41 smithi082 ceph-mon[36382]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-12T05:53:41.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:41 smithi082 ceph-mon[36382]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-12T05:53:41.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:41 smithi082 ceph-mon[36382]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-12T05:53:41.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:41 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/921123008' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-12T05:53:41.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:41 smithi082 ceph-mon[36382]: pgmap v286: 97 pgs: 66 active+clean, 21 creating+peering, 10 unknown; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2024-03-12T05:53:41.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:41.705+0000 7fe08e72c640 1 -- 172.21.15.76:0/2710460926 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fe08802ce40 con 0x7fe088102ea0 2024-03-12T05:53:42.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.380+0000 7fe0857fa640 1 -- 172.21.15.76:0/2710460926 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe080099f30 con 0x7fe088102ea0 2024-03-12T05:53:42.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.382+0000 7fe08e72c640 1 -- 172.21.15.76:0/2710460926 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe0700a1a30 msgr2=0x7fe0700a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:42.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.382+0000 7fe08e72c640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe0700a1a30 0x7fe0700a3ef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe078005fd0 tx=0x7fe07800b390 comp rx=0 tx=0).stop 2024-03-12T05:53:42.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.382+0000 7fe08e72c640 1 -- 172.21.15.76:0/2710460926 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe088102ea0 msgr2=0x7fe0881009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:42.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.382+0000 7fe08e72c640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe088102ea0 0x7fe0881009d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe08000ead0 tx=0x7fe08000ef90 comp rx=0 tx=0).stop 2024-03-12T05:53:42.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.382+0000 7fe08e72c640 1 -- 172.21.15.76:0/2710460926 shutdown_connections 2024-03-12T05:53:42.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.382+0000 7fe08e72c640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe0700a1a30 0x7fe0700a3ef0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.382+0000 7fe08e72c640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe088102ea0 0x7fe0881009d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.382+0000 7fe08e72c640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0881024e0 0x7fe088100490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.382+0000 7fe08e72c640 1 --2- 172.21.15.76:0/2710460926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0880ff470 0x7fe088101e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.382+0000 7fe08e72c640 1 -- 172.21.15.76:0/2710460926 >> 172.21.15.76:0/2710460926 conn(0x7fe0880fada0 msgr2=0x7fe0881057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:42.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.383+0000 7fe08e72c640 1 -- 172.21.15.76:0/2710460926 shutdown_connections 2024-03-12T05:53:42.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.383+0000 7fe08e72c640 1 -- 172.21.15.76:0/2710460926 wait complete. 2024-03-12T05:53:42.398 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-12T05:53:42.398 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-12T05:53:42.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.505+0000 7f9730064640 1 -- 172.21.15.76:0/625901802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9728101290 msgr2=0x7f9728101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.505+0000 7f9730064640 1 --2- 172.21.15.76:0/625901802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9728101290 0x7f9728101710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9718009900 tx=0x7f971802bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.505+0000 7f9730064640 1 -- 172.21.15.76:0/625901802 shutdown_connections 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.505+0000 7f9730064640 1 --2- 172.21.15.76:0/625901802 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9728101c50 0x7f97281084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.505+0000 7f9730064640 1 --2- 172.21.15.76:0/625901802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9728101290 0x7f9728101710 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.505+0000 7f9730064640 1 --2- 172.21.15.76:0/625901802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97281000a0 0x7f97281004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.505+0000 7f9730064640 1 -- 172.21.15.76:0/625901802 >> 172.21.15.76:0/625901802 conn(0x7f97280fb610 msgr2=0x7f97280fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.505+0000 7f9730064640 1 -- 172.21.15.76:0/625901802 shutdown_connections 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.505+0000 7f9730064640 1 -- 172.21.15.76:0/625901802 wait complete. 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.506+0000 7f9730064640 1 Processor -- start 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.506+0000 7f9730064640 1 -- start start 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.506+0000 7f9730064640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97281000a0 0x7f97281972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.506+0000 7f9730064640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9728101290 0x7f97281977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.506+0000 7f9730064640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9728101c50 0x7f972819e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.506+0000 7f9730064640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972819ed90 con 0x7f9728101c50 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.506+0000 7f9730064640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972819ef00 con 0x7f9728101290 2024-03-12T05:53:42.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.506+0000 7f9730064640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972819f0c0 con 0x7f97281000a0 2024-03-12T05:53:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.506+0000 7f972e601640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9728101c50 0x7f972819e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.506+0000 7f972e601640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9728101c50 0x7f972819e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53422/0 (socket says 172.21.15.76:53422) 2024-03-12T05:53:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.506+0000 7f972e601640 1 -- 172.21.15.76:0/15153883 learned_addr learned my addr 172.21.15.76:0/15153883 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.506+0000 7f972de00640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97281000a0 0x7f97281972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.506+0000 7f972d5ff640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9728101290 0x7f97281977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.507+0000 7f972e601640 1 -- 172.21.15.76:0/15153883 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97281000a0 msgr2=0x7f97281972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.507+0000 7f972e601640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97281000a0 0x7f97281972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.507+0000 7f972e601640 1 -- 172.21.15.76:0/15153883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9728101290 msgr2=0x7f97281977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.507+0000 7f972e601640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9728101290 0x7f97281977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.507+0000 7f972e601640 1 -- 172.21.15.76:0/15153883 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9718009560 con 0x7f9728101c50 2024-03-12T05:53:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.507+0000 7f972d5ff640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9728101290 0x7f97281977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.507+0000 7f972e601640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9728101c50 0x7f972819e850 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f972000eaa0 tx=0x7f972000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.507+0000 7f971effd640 1 -- 172.21.15.76:0/15153883 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972000cdb0 con 0x7f9728101c50 2024-03-12T05:53:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.507+0000 7f9730064640 1 -- 172.21.15.76:0/15153883 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972819f3a0 con 0x7f9728101c50 2024-03-12T05:53:42.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.507+0000 7f971effd640 1 -- 172.21.15.76:0/15153883 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9720005020 con 0x7f9728101c50 2024-03-12T05:53:42.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.507+0000 7f971effd640 1 -- 172.21.15.76:0/15153883 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97200116b0 con 0x7f9728101c50 2024-03-12T05:53:42.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.507+0000 7f9730064640 1 -- 172.21.15.76:0/15153883 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972819f8b0 con 0x7f9728101c50 2024-03-12T05:53:42.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.508+0000 7f9730064640 1 -- 172.21.15.76:0/15153883 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f972810bb00 con 0x7f9728101c50 2024-03-12T05:53:42.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.509+0000 7f971effd640 1 -- 172.21.15.76:0/15153883 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9720005520 con 0x7f9728101c50 2024-03-12T05:53:42.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.510+0000 7f971effd640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f97140a1a30 0x7f97140a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:42.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.510+0000 7f972de00640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f97140a1a30 0x7f97140a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:42.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.510+0000 7f971effd640 1 -- 172.21.15.76:0/15153883 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f97200cd4e0 con 0x7f9728101c50 2024-03-12T05:53:42.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.510+0000 7f972de00640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f97140a1a30 0x7f97140a3ef0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9724005fd0 tx=0x7f972400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:42.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.514+0000 7f971effd640 1 -- 172.21.15.76:0/15153883 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9720096840 con 0x7f9728101c50 2024-03-12T05:53:42.690 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:42 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/921123008' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-12T05:53:42.690 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:42 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] up:active 2024-03-12T05:53:42.690 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:42 smithi076 ceph-mon[32095]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-12T05:53:42.690 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:42 smithi076 ceph-mon[32095]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-12T05:53:42.690 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:42 smithi076 ceph-mon[32095]: Cluster is now healthy 2024-03-12T05:53:42.690 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:42 smithi076 ceph-mon[32095]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-03-12T05:53:42.691 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:42 smithi076 ceph-mon[32095]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-03-12T05:53:42.691 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:42 smithi076 ceph-mon[32095]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-12T05:53:42.691 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:42 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2710460926' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-12T05:53:42.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:42 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/921123008' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-12T05:53:42.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:42 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] up:active 2024-03-12T05:53:42.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:42 smithi082 ceph-mon[36382]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-12T05:53:42.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:42 smithi082 ceph-mon[36382]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-12T05:53:42.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:42 smithi082 ceph-mon[36382]: Cluster is now healthy 2024-03-12T05:53:42.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:42 smithi082 ceph-mon[36382]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-03-12T05:53:42.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:42 smithi082 ceph-mon[36382]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-03-12T05:53:42.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:42 smithi082 ceph-mon[36382]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-12T05:53:42.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:42 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2710460926' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-12T05:53:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:42 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/921123008' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-12T05:53:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:42 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] up:active 2024-03-12T05:53:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:42 smithi088 ceph-mon[36585]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-12T05:53:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:42 smithi088 ceph-mon[36585]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-12T05:53:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:42 smithi088 ceph-mon[36585]: Cluster is now healthy 2024-03-12T05:53:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:42 smithi088 ceph-mon[36585]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-03-12T05:53:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:42 smithi088 ceph-mon[36585]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-03-12T05:53:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:42 smithi088 ceph-mon[36585]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-12T05:53:42.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:42 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2710460926' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-12T05:53:42.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.739+0000 7f9730064640 1 -- 172.21.15.76:0/15153883 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f9728068b60 con 0x7f97140a1a30 2024-03-12T05:53:42.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.762+0000 7f971effd640 1 -- 172.21.15.76:0/15153883 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9728068b60 con 0x7f97140a1a30 2024-03-12T05:53:42.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.764+0000 7f9730064640 1 -- 172.21.15.76:0/15153883 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f97140a1a30 msgr2=0x7f97140a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:42.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.764+0000 7f9730064640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f97140a1a30 0x7f97140a3ef0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9724005fd0 tx=0x7f972400b390 comp rx=0 tx=0).stop 2024-03-12T05:53:42.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.764+0000 7f9730064640 1 -- 172.21.15.76:0/15153883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9728101c50 msgr2=0x7f972819e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:42.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.764+0000 7f9730064640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9728101c50 0x7f972819e850 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f972000eaa0 tx=0x7f972000ef60 comp rx=0 tx=0).stop 2024-03-12T05:53:42.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.765+0000 7f9730064640 1 -- 172.21.15.76:0/15153883 shutdown_connections 2024-03-12T05:53:42.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.765+0000 7f9730064640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f97140a1a30 0x7f97140a3ef0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.765+0000 7f9730064640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9728101c50 0x7f972819e850 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.765+0000 7f9730064640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9728101290 0x7f97281977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.765+0000 7f9730064640 1 --2- 172.21.15.76:0/15153883 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97281000a0 0x7f97281972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.765+0000 7f9730064640 1 -- 172.21.15.76:0/15153883 >> 172.21.15.76:0/15153883 conn(0x7f97280fb610 msgr2=0x7f9728068450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:42.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.765+0000 7f9730064640 1 -- 172.21.15.76:0/15153883 shutdown_connections 2024-03-12T05:53:42.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.765+0000 7f9730064640 1 -- 172.21.15.76:0/15153883 wait complete. 2024-03-12T05:53:42.780 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-12T05:53:42.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.887+0000 7f8eb7be2640 1 -- 172.21.15.76:0/1916743677 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8eb0101290 msgr2=0x7f8eb0101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:42.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.887+0000 7f8eb7be2640 1 --2- 172.21.15.76:0/1916743677 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8eb0101290 0x7f8eb0101710 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8eac009900 tx=0x7f8eac02bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:42.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 -- 172.21.15.76:0/1916743677 shutdown_connections 2024-03-12T05:53:42.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 --2- 172.21.15.76:0/1916743677 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8eb0101c50 0x7f8eb01084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 --2- 172.21.15.76:0/1916743677 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8eb0101290 0x7f8eb0101710 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 --2- 172.21.15.76:0/1916743677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8eb01000a0 0x7f8eb01004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 -- 172.21.15.76:0/1916743677 >> 172.21.15.76:0/1916743677 conn(0x7f8eb00fb610 msgr2=0x7f8eb00fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:42.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 -- 172.21.15.76:0/1916743677 shutdown_connections 2024-03-12T05:53:42.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 -- 172.21.15.76:0/1916743677 wait complete. 2024-03-12T05:53:42.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 Processor -- start 2024-03-12T05:53:42.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 -- start start 2024-03-12T05:53:42.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8eb01000a0 0x7f8eb01973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:42.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8eb0101290 0x7f8eb0197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:42.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8eb0101c50 0x7f8eb019e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:42.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb019eee0 con 0x7f8eb0101c50 2024-03-12T05:53:42.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb019f050 con 0x7f8eb01000a0 2024-03-12T05:53:42.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb7be2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb019f210 con 0x7f8eb0101290 2024-03-12T05:53:42.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb617f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8eb0101c50 0x7f8eb019e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:42.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.888+0000 7f8eb617f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8eb0101c50 0x7f8eb019e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53458/0 (socket says 172.21.15.76:53458) 2024-03-12T05:53:42.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8eb617f640 1 -- 172.21.15.76:0/720782365 learned_addr learned my addr 172.21.15.76:0/720782365 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:42.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8eb597e640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8eb01000a0 0x7f8eb01973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:42.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8eb517d640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8eb0101290 0x7f8eb0197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:42.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8eb617f640 1 -- 172.21.15.76:0/720782365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8eb0101290 msgr2=0x7f8eb0197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:42.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8eb617f640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8eb0101290 0x7f8eb0197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8eb617f640 1 -- 172.21.15.76:0/720782365 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8eb01000a0 msgr2=0x7f8eb01973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:42.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8eb617f640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8eb01000a0 0x7f8eb01973f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:42.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8eb617f640 1 -- 172.21.15.76:0/720782365 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac009560 con 0x7f8eb0101c50 2024-03-12T05:53:42.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8eb597e640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8eb01000a0 0x7f8eb01973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:53:42.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8eb617f640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8eb0101c50 0x7f8eb019e9a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800b970 tx=0x7f8ea800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:42.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8ea6ffd640 1 -- 172.21.15.76:0/720782365 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea800cb40 con 0x7f8eb0101c50 2024-03-12T05:53:42.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8eb7be2640 1 -- 172.21.15.76:0/720782365 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb019f4f0 con 0x7f8eb0101c50 2024-03-12T05:53:42.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8ea6ffd640 1 -- 172.21.15.76:0/720782365 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ea800cca0 con 0x7f8eb0101c50 2024-03-12T05:53:42.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8ea6ffd640 1 -- 172.21.15.76:0/720782365 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea80077d0 con 0x7f8eb0101c50 2024-03-12T05:53:42.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.889+0000 7f8eb7be2640 1 -- 172.21.15.76:0/720782365 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb019fa30 con 0x7f8eb0101c50 2024-03-12T05:53:42.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.890+0000 7f8eb7be2640 1 -- 172.21.15.76:0/720782365 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb0107490 con 0x7f8eb0101c50 2024-03-12T05:53:42.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.891+0000 7f8ea6ffd640 1 -- 172.21.15.76:0/720782365 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8ea8014440 con 0x7f8eb0101c50 2024-03-12T05:53:42.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.892+0000 7f8ea6ffd640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8e9c0a1a30 0x7f8e9c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:42.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.892+0000 7f8eb597e640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8e9c0a1a30 0x7f8e9c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:42.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.892+0000 7f8ea6ffd640 1 -- 172.21.15.76:0/720782365 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8ea80ccc20 con 0x7f8eb0101c50 2024-03-12T05:53:42.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.892+0000 7f8eb597e640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8e9c0a1a30 0x7f8e9c0a3ef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8eb01010f0 tx=0x7f8ea0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:42.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:42.896+0000 7f8ea6ffd640 1 -- 172.21.15.76:0/720782365 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8ea8095f80 con 0x7f8eb0101c50 2024-03-12T05:53:42.987 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:53:42 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:53:42.743+0000 7f9ae283d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:53:42.987 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:53:42 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:53:42.743+0000 7f9ae283d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:53:42.987 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:53:42 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:53:42.744+0000 7f9ae283d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:53:42.988 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:53:42 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:53:42.744+0000 7f9ae283d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:53:42.988 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:53:42 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:53:42.744+0000 7f9ae283d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:53:43.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.104+0000 7f8eb7be2640 1 -- 172.21.15.76:0/720782365 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f8eb002ce10 con 0x7f8e9c0a1a30 2024-03-12T05:53:43.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.125+0000 7f8ea6ffd640 1 -- 172.21.15.76:0/720782365 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8eb002ce10 con 0x7f8e9c0a1a30 2024-03-12T05:53:43.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.127+0000 7f8eb7be2640 1 -- 172.21.15.76:0/720782365 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8e9c0a1a30 msgr2=0x7f8e9c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.127+0000 7f8eb7be2640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8e9c0a1a30 0x7f8e9c0a3ef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8eb01010f0 tx=0x7f8ea0008040 comp rx=0 tx=0).stop 2024-03-12T05:53:43.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.127+0000 7f8eb7be2640 1 -- 172.21.15.76:0/720782365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8eb0101c50 msgr2=0x7f8eb019e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.127+0000 7f8eb7be2640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8eb0101c50 0x7f8eb019e9a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800b970 tx=0x7f8ea800be30 comp rx=0 tx=0).stop 2024-03-12T05:53:43.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.127+0000 7f8eb7be2640 1 -- 172.21.15.76:0/720782365 shutdown_connections 2024-03-12T05:53:43.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.127+0000 7f8eb7be2640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8e9c0a1a30 0x7f8e9c0a3ef0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.127+0000 7f8eb7be2640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8eb0101c50 0x7f8eb019e9a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.127+0000 7f8eb7be2640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8eb0101290 0x7f8eb0197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.127+0000 7f8eb7be2640 1 --2- 172.21.15.76:0/720782365 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8eb01000a0 0x7f8eb01973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.127+0000 7f8eb7be2640 1 -- 172.21.15.76:0/720782365 >> 172.21.15.76:0/720782365 conn(0x7f8eb00fb610 msgr2=0x7f8eb0104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:43.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.127+0000 7f8eb7be2640 1 -- 172.21.15.76:0/720782365 shutdown_connections 2024-03-12T05:53:43.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.127+0000 7f8eb7be2640 1 -- 172.21.15.76:0/720782365 wait complete. 2024-03-12T05:53:43.141 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-12T05:53:43.194 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 05:53:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b[66548]: 2024-03-12T05:53:42.933+0000 7fb6c9863700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-12T05:53:43.194 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 05:53:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b[66548]: 2024-03-12T05:53:42.933+0000 7fb6c9863700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-12T05:53:43.195 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 05:53:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b[66548]: 2024-03-12T05:53:42.933+0000 7fb6c9863700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-12T05:53:43.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.248+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/4263748177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfcc101c20 msgr2=0x7fcfcc104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.248+0000 7fcfd1a6f640 1 --2- 172.21.15.76:0/4263748177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfcc101c20 0x7fcfcc104010 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009900 tx=0x7fcfc002bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:43.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.248+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/4263748177 shutdown_connections 2024-03-12T05:53:43.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.248+0000 7fcfd1a6f640 1 --2- 172.21.15.76:0/4263748177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcfcc106e80 0x7fcfcc1092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.248+0000 7fcfd1a6f640 1 --2- 172.21.15.76:0/4263748177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc104550 0x7fcfcc106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.248+0000 7fcfd1a6f640 1 --2- 172.21.15.76:0/4263748177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfcc101c20 0x7fcfcc104010 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.248+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/4263748177 >> 172.21.15.76:0/4263748177 conn(0x7fcfcc0fb610 msgr2=0x7fcfcc0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:43.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.248+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/4263748177 shutdown_connections 2024-03-12T05:53:43.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.248+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/4263748177 wait complete. 2024-03-12T05:53:43.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.248+0000 7fcfd1a6f640 1 Processor -- start 2024-03-12T05:53:43.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.248+0000 7fcfd1a6f640 1 -- start start 2024-03-12T05:53:43.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfd1a6f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc101c20 0x7fcfcc101590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:43.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfd1a6f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcfcc104550 0x7fcfcc0ffbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:43.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfd1a6f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfcc106e80 0x7fcfcc100110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:43.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfd1a6f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc100870 con 0x7fcfcc106e80 2024-03-12T05:53:43.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfd1a6f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc1009e0 con 0x7fcfcc104550 2024-03-12T05:53:43.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfd1a6f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc1a1310 con 0x7fcfcc101c20 2024-03-12T05:53:43.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfcbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfcc106e80 0x7fcfcc100110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfcbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfcc106e80 0x7fcfcc100110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53488/0 (socket says 172.21.15.76:53488) 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfcbfff640 1 -- 172.21.15.76:0/1613475990 learned_addr learned my addr 172.21.15.76:0/1613475990 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfcb7fe640 1 --2- 172.21.15.76:0/1613475990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc101c20 0x7fcfcc101590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfcaffd640 1 --2- 172.21.15.76:0/1613475990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcfcc104550 0x7fcfcc0ffbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfcbfff640 1 -- 172.21.15.76:0/1613475990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc101c20 msgr2=0x7fcfcc101590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfcbfff640 1 --2- 172.21.15.76:0/1613475990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc101c20 0x7fcfcc101590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfcbfff640 1 -- 172.21.15.76:0/1613475990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcfcc104550 msgr2=0x7fcfcc0ffbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfcbfff640 1 --2- 172.21.15.76:0/1613475990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcfcc104550 0x7fcfcc0ffbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfcbfff640 1 -- 172.21.15.76:0/1613475990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc0009560 con 0x7fcfcc106e80 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfcbfff640 1 --2- 172.21.15.76:0/1613475990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfcc106e80 0x7fcfcc100110 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400db20 tx=0x7fcfc400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfc8ff9640 1 -- 172.21.15.76:0/1613475990 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc400bea0 con 0x7fcfcc106e80 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/1613475990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfcc1a15c0 con 0x7fcfcc106e80 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.249+0000 7fcfc8ff9640 1 -- 172.21.15.76:0/1613475990 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfc4009940 con 0x7fcfcc106e80 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.250+0000 7fcfc8ff9640 1 -- 172.21.15.76:0/1613475990 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc40106c0 con 0x7fcfcc106e80 2024-03-12T05:53:43.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.250+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/1613475990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfcc1a1ad0 con 0x7fcfcc106e80 2024-03-12T05:53:43.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.250+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/1613475990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfcc0fd100 con 0x7fcfcc106e80 2024-03-12T05:53:43.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.252+0000 7fcfc8ff9640 1 -- 172.21.15.76:0/1613475990 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcfc4004240 con 0x7fcfcc106e80 2024-03-12T05:53:43.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.253+0000 7fcfc8ff9640 1 --2- 172.21.15.76:0/1613475990 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcfb40a1a30 0x7fcfb40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:43.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.253+0000 7fcfcb7fe640 1 --2- 172.21.15.76:0/1613475990 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcfb40a1a30 0x7fcfb40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:43.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.253+0000 7fcfc8ff9640 1 -- 172.21.15.76:0/1613475990 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcfc40cceb0 con 0x7fcfcc106e80 2024-03-12T05:53:43.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.253+0000 7fcfcb7fe640 1 --2- 172.21.15.76:0/1613475990 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcfb40a1a30 0x7fcfb40a3ef0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009230 tx=0x7fcfc0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:43.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.256+0000 7fcfc8ff9640 1 -- 172.21.15.76:0/1613475990 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcfc40fad20 con 0x7fcfcc106e80 2024-03-12T05:53:43.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.403+0000 7fcfc8ff9640 1 -- 172.21.15.76:0/1613475990 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fcfc4096690 con 0x7fcfcc106e80 2024-03-12T05:53:43.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.467+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/1613475990 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfcc0615c0 con 0x7fcfb40a1a30 2024-03-12T05:53:43.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.471+0000 7fcfc8ff9640 1 -- 172.21.15.76:0/1613475990 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fcfcc0615c0 con 0x7fcfb40a1a30 2024-03-12T05:53:43.474 INFO:teuthology.orchestra.run.smithi076.stdout:/volumes/qa/sv_0/e204b170-b728-4b7f-b81e-d35db2cc71ff 2024-03-12T05:53:43.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.473+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/1613475990 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcfb40a1a30 msgr2=0x7fcfb40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.473+0000 7fcfd1a6f640 1 --2- 172.21.15.76:0/1613475990 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcfb40a1a30 0x7fcfb40a3ef0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009230 tx=0x7fcfc0005fd0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.473+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/1613475990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfcc106e80 msgr2=0x7fcfcc100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.473+0000 7fcfd1a6f640 1 --2- 172.21.15.76:0/1613475990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfcc106e80 0x7fcfcc100110 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400db20 tx=0x7fcfc400b260 comp rx=0 tx=0).stop 2024-03-12T05:53:43.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.473+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/1613475990 shutdown_connections 2024-03-12T05:53:43.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.473+0000 7fcfd1a6f640 1 --2- 172.21.15.76:0/1613475990 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcfb40a1a30 0x7fcfb40a3ef0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.473+0000 7fcfd1a6f640 1 --2- 172.21.15.76:0/1613475990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcfcc106e80 0x7fcfcc100110 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.473+0000 7fcfd1a6f640 1 --2- 172.21.15.76:0/1613475990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcfcc104550 0x7fcfcc0ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.473+0000 7fcfd1a6f640 1 --2- 172.21.15.76:0/1613475990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc101c20 0x7fcfcc101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.473+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/1613475990 >> 172.21.15.76:0/1613475990 conn(0x7fcfcc0fb610 msgr2=0x7fcfcc102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:43.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.473+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/1613475990 shutdown_connections 2024-03-12T05:53:43.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.473+0000 7fcfd1a6f640 1 -- 172.21.15.76:0/1613475990 wait complete. 2024-03-12T05:53:43.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:43 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2710460926' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-12T05:53:43.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:43 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] up:active 2024-03-12T05:53:43.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:43 smithi076 ceph-mon[32095]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-12T05:53:43.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:43 smithi076 ceph-mon[32095]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:53:43.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:43 smithi076 ceph-mon[32095]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:43.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:43 smithi076 ceph-mon[32095]: pgmap v287: 97 pgs: 85 active+clean, 12 creating+peering; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3.8 KiB/s wr, 11 op/s 2024-03-12T05:53:43.487 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-12T05:53:43.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.591+0000 7f359360f640 1 -- 172.21.15.76:0/337986168 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f358c101c50 msgr2=0x7f358c1084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.591+0000 7f359360f640 1 --2- 172.21.15.76:0/337986168 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f358c101c50 0x7f358c1084e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3584009ca0 tx=0x7f358402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.591+0000 7f359360f640 1 -- 172.21.15.76:0/337986168 shutdown_connections 2024-03-12T05:53:43.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.591+0000 7f359360f640 1 --2- 172.21.15.76:0/337986168 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f358c101c50 0x7f358c1084e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.591+0000 7f359360f640 1 --2- 172.21.15.76:0/337986168 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f358c101290 0x7f358c101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.591+0000 7f359360f640 1 --2- 172.21.15.76:0/337986168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f358c1000a0 0x7f358c1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.591+0000 7f359360f640 1 -- 172.21.15.76:0/337986168 >> 172.21.15.76:0/337986168 conn(0x7f358c0fb610 msgr2=0x7f358c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:43.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.591+0000 7f359360f640 1 -- 172.21.15.76:0/337986168 shutdown_connections 2024-03-12T05:53:43.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.591+0000 7f359360f640 1 -- 172.21.15.76:0/337986168 wait complete. 2024-03-12T05:53:43.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f359360f640 1 Processor -- start 2024-03-12T05:53:43.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f359360f640 1 -- start start 2024-03-12T05:53:43.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f359360f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f358c1000a0 0x7f358c068fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f359360f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f358c101290 0x7f358c19bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f359360f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f358c101c50 0x7f358c19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f359360f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c19ed60 con 0x7f358c1000a0 2024-03-12T05:53:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f359360f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c19eed0 con 0x7f358c101c50 2024-03-12T05:53:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f359360f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c19f090 con 0x7f358c101290 2024-03-12T05:53:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f35913ab640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f358c1000a0 0x7f358c068fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f35913ab640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f358c1000a0 0x7f358c068fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53514/0 (socket says 172.21.15.76:53514) 2024-03-12T05:53:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f35913ab640 1 -- 172.21.15.76:0/1746014611 learned_addr learned my addr 172.21.15.76:0/1746014611 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f3590baa640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f358c101290 0x7f358c19bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f35913ab640 1 -- 172.21.15.76:0/1746014611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f358c101290 msgr2=0x7f358c19bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f35913ab640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f358c101290 0x7f358c19bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f3591bac640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f358c101c50 0x7f358c19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f35913ab640 1 -- 172.21.15.76:0/1746014611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f358c101c50 msgr2=0x7f358c19e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f35913ab640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f358c101c50 0x7f358c19e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f35913ab640 1 -- 172.21.15.76:0/1746014611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3584009900 con 0x7f358c1000a0 2024-03-12T05:53:43.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.592+0000 7f3590baa640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f358c101290 0x7f358c19bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:43.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.593+0000 7f35913ab640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f358c1000a0 0x7f358c068fc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f358800c970 tx=0x7f358800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:43.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.593+0000 7f35827fc640 1 -- 172.21.15.76:0/1746014611 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3588004220 con 0x7f358c1000a0 2024-03-12T05:53:43.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.593+0000 7f359360f640 1 -- 172.21.15.76:0/1746014611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358c19f310 con 0x7f358c1000a0 2024-03-12T05:53:43.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.593+0000 7f35827fc640 1 -- 172.21.15.76:0/1746014611 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3588007e90 con 0x7f358c1000a0 2024-03-12T05:53:43.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.593+0000 7f35827fc640 1 -- 172.21.15.76:0/1746014611 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3588004a00 con 0x7f358c1000a0 2024-03-12T05:53:43.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.593+0000 7f359360f640 1 -- 172.21.15.76:0/1746014611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f358c19f7d0 con 0x7f358c1000a0 2024-03-12T05:53:43.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.594+0000 7f359360f640 1 -- 172.21.15.76:0/1746014611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f358c068000 con 0x7f358c1000a0 2024-03-12T05:53:43.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.595+0000 7f35827fc640 1 -- 172.21.15.76:0/1746014611 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f35880079b0 con 0x7f358c1000a0 2024-03-12T05:53:43.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.595+0000 7f35827fc640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f35780a1ac0 0x7f35780a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:43.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.595+0000 7f3590baa640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f35780a1ac0 0x7f35780a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:43.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.595+0000 7f35827fc640 1 -- 172.21.15.76:0/1746014611 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f35880cd560 con 0x7f358c1000a0 2024-03-12T05:53:43.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.596+0000 7f3590baa640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f35780a1ac0 0x7f35780a3f80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f357c005d00 tx=0x7f357c009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:43.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.600+0000 7f35827fc640 1 -- 172.21.15.76:0/1746014611 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f35880967c0 con 0x7f358c1000a0 2024-03-12T05:53:43.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:43 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2710460926' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-12T05:53:43.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:43 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] up:active 2024-03-12T05:53:43.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:43 smithi082 ceph-mon[36382]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-12T05:53:43.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:43 smithi082 ceph-mon[36382]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:53:43.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:43 smithi082 ceph-mon[36382]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:43.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:43 smithi082 ceph-mon[36382]: pgmap v287: 97 pgs: 85 active+clean, 12 creating+peering; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3.8 KiB/s wr, 11 op/s 2024-03-12T05:53:43.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:43 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2710460926' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-12T05:53:43.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:43 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] up:active 2024-03-12T05:53:43.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:43 smithi088 ceph-mon[36585]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-12T05:53:43.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:43 smithi088 ceph-mon[36585]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:53:43.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:43 smithi088 ceph-mon[36585]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:43.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:43 smithi088 ceph-mon[36585]: pgmap v287: 97 pgs: 85 active+clean, 12 creating+peering; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 3.8 KiB/s wr, 11 op/s 2024-03-12T05:53:43.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.804+0000 7f359360f640 1 -- 172.21.15.76:0/1746014611 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f358c02cf40 con 0x7f35780a1ac0 2024-03-12T05:53:43.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.825+0000 7f35827fc640 1 -- 172.21.15.76:0/1746014611 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f358c02cf40 con 0x7f35780a1ac0 2024-03-12T05:53:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.828+0000 7f359360f640 1 -- 172.21.15.76:0/1746014611 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f35780a1ac0 msgr2=0x7f35780a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.828+0000 7f359360f640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f35780a1ac0 0x7f35780a3f80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f357c005d00 tx=0x7f357c009550 comp rx=0 tx=0).stop 2024-03-12T05:53:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.828+0000 7f359360f640 1 -- 172.21.15.76:0/1746014611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f358c1000a0 msgr2=0x7f358c068fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.828+0000 7f359360f640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f358c1000a0 0x7f358c068fc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f358800c970 tx=0x7f358800ce30 comp rx=0 tx=0).stop 2024-03-12T05:53:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.828+0000 7f359360f640 1 -- 172.21.15.76:0/1746014611 shutdown_connections 2024-03-12T05:53:43.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.828+0000 7f359360f640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f35780a1ac0 0x7f35780a3f80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.828+0000 7f359360f640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f358c101c50 0x7f358c19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.828+0000 7f359360f640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f358c101290 0x7f358c19bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.828+0000 7f359360f640 1 --2- 172.21.15.76:0/1746014611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f358c1000a0 0x7f358c068fc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.828+0000 7f359360f640 1 -- 172.21.15.76:0/1746014611 >> 172.21.15.76:0/1746014611 conn(0x7f358c0fb610 msgr2=0x7f358c104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.828+0000 7f359360f640 1 -- 172.21.15.76:0/1746014611 shutdown_connections 2024-03-12T05:53:43.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.828+0000 7f359360f640 1 -- 172.21.15.76:0/1746014611 wait complete. 2024-03-12T05:53:43.842 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-12T05:53:43.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 -- 172.21.15.76:0/2128910562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5078101280 msgr2=0x7f5078101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 --2- 172.21.15.76:0/2128910562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5078101280 0x7f5078101700 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5064009900 tx=0x7f506402bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:43.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 -- 172.21.15.76:0/2128910562 shutdown_connections 2024-03-12T05:53:43.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 --2- 172.21.15.76:0/2128910562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5078101c40 0x7f50781084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 --2- 172.21.15.76:0/2128910562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5078101280 0x7f5078101700 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 --2- 172.21.15.76:0/2128910562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5078100090 0x7f50781004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 -- 172.21.15.76:0/2128910562 >> 172.21.15.76:0/2128910562 conn(0x7f50780fb5e0 msgr2=0x7f50780fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:43.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 -- 172.21.15.76:0/2128910562 shutdown_connections 2024-03-12T05:53:43.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 -- 172.21.15.76:0/2128910562 wait complete. 2024-03-12T05:53:43.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 Processor -- start 2024-03-12T05:53:43.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 -- start start 2024-03-12T05:53:43.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5078100090 0x7f5078197210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:43.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5078101280 0x7f5078197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:43.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5078101c40 0x7f507819e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:43.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507819ed00 con 0x7f5078101c40 2024-03-12T05:53:43.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507819ee70 con 0x7f5078100090 2024-03-12T05:53:43.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507eb2c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507819f030 con 0x7f5078101280 2024-03-12T05:53:43.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507d0c9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5078101c40 0x7f507819e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:43.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507d0c9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5078101c40 0x7f507819e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53536/0 (socket says 172.21.15.76:53536) 2024-03-12T05:53:43.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.946+0000 7f507d0c9640 1 -- 172.21.15.76:0/3319891075 learned_addr learned my addr 172.21.15.76:0/3319891075 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:43.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.947+0000 7f507c8c8640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5078100090 0x7f5078197210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:43.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.947+0000 7f5077fff640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5078101280 0x7f5078197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:43.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.947+0000 7f507d0c9640 1 -- 172.21.15.76:0/3319891075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5078101280 msgr2=0x7f5078197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.947+0000 7f507d0c9640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5078101280 0x7f5078197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.947+0000 7f507d0c9640 1 -- 172.21.15.76:0/3319891075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5078100090 msgr2=0x7f5078197210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:43.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.947+0000 7f507d0c9640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5078100090 0x7f5078197210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:43.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.947+0000 7f507d0c9640 1 -- 172.21.15.76:0/3319891075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5064009560 con 0x7f5078101c40 2024-03-12T05:53:43.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.947+0000 7f507c8c8640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5078100090 0x7f5078197210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:43.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.947+0000 7f507d0c9640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5078101c40 0x7f507819e7c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f507000cce0 tx=0x7f507000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:43.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.947+0000 7f5075ffb640 1 -- 172.21.15.76:0/3319891075 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507000f450 con 0x7f5078101c40 2024-03-12T05:53:43.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.947+0000 7f5075ffb640 1 -- 172.21.15.76:0/3319891075 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f507000fa90 con 0x7f5078101c40 2024-03-12T05:53:43.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.947+0000 7f507eb2c640 1 -- 172.21.15.76:0/3319891075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507819f310 con 0x7f5078101c40 2024-03-12T05:53:43.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.947+0000 7f5075ffb640 1 -- 172.21.15.76:0/3319891075 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070018b90 con 0x7f5078101c40 2024-03-12T05:53:43.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.947+0000 7f507eb2c640 1 -- 172.21.15.76:0/3319891075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507819f7d0 con 0x7f5078101c40 2024-03-12T05:53:43.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.948+0000 7f507eb2c640 1 -- 172.21.15.76:0/3319891075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5078107350 con 0x7f5078101c40 2024-03-12T05:53:43.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.949+0000 7f5075ffb640 1 -- 172.21.15.76:0/3319891075 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f507000fc00 con 0x7f5078101c40 2024-03-12T05:53:43.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.950+0000 7f5075ffb640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f50680a1ac0 0x7f50680a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:43.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.950+0000 7f507c8c8640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f50680a1ac0 0x7f50680a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:43.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.950+0000 7f5075ffb640 1 -- 172.21.15.76:0/3319891075 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f50700cde00 con 0x7f5078101c40 2024-03-12T05:53:43.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.950+0000 7f507c8c8640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f50680a1ac0 0x7f50680a3f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f50781010e0 tx=0x7f506c005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:43.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:43.953+0000 7f5075ffb640 1 -- 172.21.15.76:0/3319891075 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5070097060 con 0x7f5078101c40 2024-03-12T05:53:44.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.157+0000 7f507eb2c640 1 -- 172.21.15.76:0/3319891075 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f507802ceb0 con 0x7f50680a1ac0 2024-03-12T05:53:44.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.161+0000 7f5075ffb640 1 -- 172.21.15.76:0/3319891075 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f507802ceb0 con 0x7f50680a1ac0 2024-03-12T05:53:44.164 INFO:teuthology.orchestra.run.smithi076.stdout:/volumes/qa/sv_1/43d84004-5a04-4acb-a504-caeba624e5ab 2024-03-12T05:53:44.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.163+0000 7f507eb2c640 1 -- 172.21.15.76:0/3319891075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f50680a1ac0 msgr2=0x7f50680a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.163+0000 7f507eb2c640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f50680a1ac0 0x7f50680a3f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f50781010e0 tx=0x7f506c005d30 comp rx=0 tx=0).stop 2024-03-12T05:53:44.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.163+0000 7f507eb2c640 1 -- 172.21.15.76:0/3319891075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5078101c40 msgr2=0x7f507819e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.163+0000 7f507eb2c640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5078101c40 0x7f507819e7c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f507000cce0 tx=0x7f507000a620 comp rx=0 tx=0).stop 2024-03-12T05:53:44.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.163+0000 7f507eb2c640 1 -- 172.21.15.76:0/3319891075 shutdown_connections 2024-03-12T05:53:44.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.163+0000 7f507eb2c640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f50680a1ac0 0x7f50680a3f80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.163+0000 7f507eb2c640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5078101c40 0x7f507819e7c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.163+0000 7f507eb2c640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5078101280 0x7f5078197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.163+0000 7f507eb2c640 1 --2- 172.21.15.76:0/3319891075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5078100090 0x7f5078197210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.163+0000 7f507eb2c640 1 -- 172.21.15.76:0/3319891075 >> 172.21.15.76:0/3319891075 conn(0x7f50780fb5e0 msgr2=0x7f5078104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:44.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.163+0000 7f507eb2c640 1 -- 172.21.15.76:0/3319891075 shutdown_connections 2024-03-12T05:53:44.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.163+0000 7f507eb2c640 1 -- 172.21.15.76:0/3319891075 wait complete. 2024-03-12T05:53:44.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:53:44.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.278+0000 7fc88816c640 1 -- 172.21.15.76:0/821117874 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc880102dc0 msgr2=0x7fc880109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.278+0000 7fc88816c640 1 --2- 172.21.15.76:0/821117874 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc880102dc0 0x7fc880109650 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc878009ca0 tx=0x7fc87802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.278+0000 7fc88816c640 1 -- 172.21.15.76:0/821117874 shutdown_connections 2024-03-12T05:53:44.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.278+0000 7fc88816c640 1 --2- 172.21.15.76:0/821117874 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc880102dc0 0x7fc880109650 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.278+0000 7fc88816c640 1 --2- 172.21.15.76:0/821117874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc880102400 0x7fc880102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.278+0000 7fc88816c640 1 --2- 172.21.15.76:0/821117874 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc880101210 0x7fc880101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.278+0000 7fc88816c640 1 -- 172.21.15.76:0/821117874 >> 172.21.15.76:0/821117874 conn(0x7fc8800fc7a0 msgr2=0x7fc8800febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:44.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.278+0000 7fc88816c640 1 -- 172.21.15.76:0/821117874 shutdown_connections 2024-03-12T05:53:44.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.278+0000 7fc88816c640 1 -- 172.21.15.76:0/821117874 wait complete. 2024-03-12T05:53:44.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.278+0000 7fc88816c640 1 Processor -- start 2024-03-12T05:53:44.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.278+0000 7fc88816c640 1 -- start start 2024-03-12T05:53:44.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc88816c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc880101210 0x7fc880197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:44.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc88816c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc880102400 0x7fc880197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:44.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc88816c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc880102dc0 0x7fc88019e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:44.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc88816c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88019ecf0 con 0x7fc880102400 2024-03-12T05:53:44.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc88816c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88019ee60 con 0x7fc880101210 2024-03-12T05:53:44.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc88816c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88019f020 con 0x7fc880102dc0 2024-03-12T05:53:44.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc885707640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc880102400 0x7fc880197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:44.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc885707640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc880102400 0x7fc880197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53556/0 (socket says 172.21.15.76:53556) 2024-03-12T05:53:44.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc885707640 1 -- 172.21.15.76:0/2598967956 learned_addr learned my addr 172.21.15.76:0/2598967956 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:44.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc885f08640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc880101210 0x7fc880197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:44.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc886709640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc880102dc0 0x7fc88019e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:44.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc885707640 1 -- 172.21.15.76:0/2598967956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc880102dc0 msgr2=0x7fc88019e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc885707640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc880102dc0 0x7fc88019e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc885707640 1 -- 172.21.15.76:0/2598967956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc880101210 msgr2=0x7fc880197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc885707640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc880101210 0x7fc880197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc885707640 1 -- 172.21.15.76:0/2598967956 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc878009900 con 0x7fc880102400 2024-03-12T05:53:44.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc886709640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc880102dc0 0x7fc88019e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:53:44.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc885707640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc880102400 0x7fc880197740 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc87000c970 tx=0x7fc87000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:44.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc876ffd640 1 -- 172.21.15.76:0/2598967956 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc870007bb0 con 0x7fc880102400 2024-03-12T05:53:44.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc876ffd640 1 -- 172.21.15.76:0/2598967956 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc870007d10 con 0x7fc880102400 2024-03-12T05:53:44.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc88816c640 1 -- 172.21.15.76:0/2598967956 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88019f300 con 0x7fc880102400 2024-03-12T05:53:44.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc876ffd640 1 -- 172.21.15.76:0/2598967956 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87000f450 con 0x7fc880102400 2024-03-12T05:53:44.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.279+0000 7fc88816c640 1 -- 172.21.15.76:0/2598967956 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88019f7c0 con 0x7fc880102400 2024-03-12T05:53:44.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.280+0000 7fc88816c640 1 -- 172.21.15.76:0/2598967956 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8801084d0 con 0x7fc880102400 2024-03-12T05:53:44.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.281+0000 7fc876ffd640 1 -- 172.21.15.76:0/2598967956 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fc870004f70 con 0x7fc880102400 2024-03-12T05:53:44.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.282+0000 7fc876ffd640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc86c0a1ac0 0x7fc86c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:44.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.282+0000 7fc885f08640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc86c0a1ac0 0x7fc86c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:44.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.282+0000 7fc876ffd640 1 -- 172.21.15.76:0/2598967956 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc8700cd120 con 0x7fc880102400 2024-03-12T05:53:44.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.282+0000 7fc885f08640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc86c0a1ac0 0x7fc86c0a3f80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc880102260 tx=0x7fc87c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:44.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.286+0000 7fc876ffd640 1 -- 172.21.15.76:0/2598967956 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc870096380 con 0x7fc880102400 2024-03-12T05:53:44.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:44 smithi076 ceph-mon[32095]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:44.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:44 smithi076 ceph-mon[32095]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-12T05:53:44.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:44 smithi076 ceph-mon[32095]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:44.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:44 smithi076 ceph-mon[32095]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:44.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.501+0000 7fc88816c640 1 -- 172.21.15.76:0/2598967956 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc88010cbb0 con 0x7fc880102400 2024-03-12T05:53:44.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.502+0000 7fc876ffd640 1 -- 172.21.15.76:0/2598967956 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+11328 (secure 0 0 0) 0x7fc870099a40 con 0x7fc880102400 2024-03-12T05:53:44.505 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:44.505 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:53:42.384114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:53:44.505 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 21 2024-03-12T05:53:44.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.504+0000 7fc88816c640 1 -- 172.21.15.76:0/2598967956 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc86c0a1ac0 msgr2=0x7fc86c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.504+0000 7fc88816c640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc86c0a1ac0 0x7fc86c0a3f80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc880102260 tx=0x7fc87c00a2d0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.504+0000 7fc88816c640 1 -- 172.21.15.76:0/2598967956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc880102400 msgr2=0x7fc880197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.504+0000 7fc88816c640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc880102400 0x7fc880197740 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc87000c970 tx=0x7fc87000ce30 comp rx=0 tx=0).stop 2024-03-12T05:53:44.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.504+0000 7fc88816c640 1 -- 172.21.15.76:0/2598967956 shutdown_connections 2024-03-12T05:53:44.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.504+0000 7fc88816c640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc86c0a1ac0 0x7fc86c0a3f80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.504+0000 7fc88816c640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc880102dc0 0x7fc88019e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.504+0000 7fc88816c640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc880102400 0x7fc880197740 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.504+0000 7fc88816c640 1 --2- 172.21.15.76:0/2598967956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc880101210 0x7fc880197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.504+0000 7fc88816c640 1 -- 172.21.15.76:0/2598967956 >> 172.21.15.76:0/2598967956 conn(0x7fc8800fc7a0 msgr2=0x7fc8801056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:44.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.504+0000 7fc88816c640 1 -- 172.21.15.76:0/2598967956 shutdown_connections 2024-03-12T05:53:44.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.504+0000 7fc88816c640 1 -- 172.21.15.76:0/2598967956 wait complete. 2024-03-12T05:53:44.518 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:53:44.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.616+0000 7f225882a640 1 -- 172.21.15.76:0/1767809217 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22541027a0 msgr2=0x7f2254102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.616+0000 7f225882a640 1 --2- 172.21.15.76:0/1767809217 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22541027a0 0x7f2254102bc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2244009900 tx=0x7f224402bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:44.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.616+0000 7f225882a640 1 -- 172.21.15.76:0/1767809217 shutdown_connections 2024-03-12T05:53:44.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.616+0000 7f225882a640 1 --2- 172.21.15.76:0/1767809217 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f22540ffbd0 0x7f2254109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.616+0000 7f225882a640 1 --2- 172.21.15.76:0/1767809217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22540ff210 0x7f22540ff690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.616+0000 7f225882a640 1 --2- 172.21.15.76:0/1767809217 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22541027a0 0x7f2254102bc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.616+0000 7f225882a640 1 -- 172.21.15.76:0/1767809217 >> 172.21.15.76:0/1767809217 conn(0x7f22540fadb0 msgr2=0x7f22540fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:44.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.616+0000 7f225882a640 1 -- 172.21.15.76:0/1767809217 shutdown_connections 2024-03-12T05:53:44.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.616+0000 7f225882a640 1 -- 172.21.15.76:0/1767809217 wait complete. 2024-03-12T05:53:44.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.617+0000 7f225882a640 1 Processor -- start 2024-03-12T05:53:44.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.617+0000 7f225882a640 1 -- start start 2024-03-12T05:53:44.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.617+0000 7f225882a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f22540ff210 0x7f2254101ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:44.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.617+0000 7f225882a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22540ffbd0 0x7f22541004e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:44.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.617+0000 7f225882a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22541027a0 0x7f2254100a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:44.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.617+0000 7f225882a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2254101180 con 0x7f22541027a0 2024-03-12T05:53:44.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.617+0000 7f225882a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22541012f0 con 0x7f22540ff210 2024-03-12T05:53:44.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.617+0000 7f225882a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22541a12b0 con 0x7f22540ffbd0 2024-03-12T05:53:44.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.617+0000 7f2252d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22541027a0 0x7f2254100a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:44.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.617+0000 7f2252d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22541027a0 0x7f2254100a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53576/0 (socket says 172.21.15.76:53576) 2024-03-12T05:53:44.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.617+0000 7f2252d9d640 1 -- 172.21.15.76:0/1029554712 learned_addr learned my addr 172.21.15.76:0/1029554712 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:44.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.618+0000 7f225259c640 1 --2- 172.21.15.76:0/1029554712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f22540ff210 0x7f2254101ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:44.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.618+0000 7f2252d9d640 1 -- 172.21.15.76:0/1029554712 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22540ffbd0 msgr2=0x7f22541004e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.618+0000 7f2252d9d640 1 --2- 172.21.15.76:0/1029554712 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22540ffbd0 0x7f22541004e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.618+0000 7f2252d9d640 1 -- 172.21.15.76:0/1029554712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f22540ff210 msgr2=0x7f2254101ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.618+0000 7f2252d9d640 1 --2- 172.21.15.76:0/1029554712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f22540ff210 0x7f2254101ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.618+0000 7f2252d9d640 1 -- 172.21.15.76:0/1029554712 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2244009560 con 0x7f22541027a0 2024-03-12T05:53:44.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.618+0000 7f2252d9d640 1 --2- 172.21.15.76:0/1029554712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22541027a0 0x7f2254100a20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f224c00ead0 tx=0x7f224c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:44.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.618+0000 7f22437fe640 1 -- 172.21.15.76:0/1029554712 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c00cde0 con 0x7f22541027a0 2024-03-12T05:53:44.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.618+0000 7f225882a640 1 -- 172.21.15.76:0/1029554712 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22541a1560 con 0x7f22541027a0 2024-03-12T05:53:44.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.618+0000 7f22437fe640 1 -- 172.21.15.76:0/1029554712 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f224c005020 con 0x7f22541027a0 2024-03-12T05:53:44.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.618+0000 7f22437fe640 1 -- 172.21.15.76:0/1029554712 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c0116b0 con 0x7f22541027a0 2024-03-12T05:53:44.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.618+0000 7f225882a640 1 -- 172.21.15.76:0/1029554712 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22541a1a70 con 0x7f22541027a0 2024-03-12T05:53:44.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.619+0000 7f225882a640 1 -- 172.21.15.76:0/1029554712 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2254108660 con 0x7f22541027a0 2024-03-12T05:53:44.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.620+0000 7f22437fe640 1 -- 172.21.15.76:0/1029554712 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f224c005520 con 0x7f22541027a0 2024-03-12T05:53:44.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.620+0000 7f22437fe640 1 --2- 172.21.15.76:0/1029554712 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f223c0a1ac0 0x7f223c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:44.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.621+0000 7f225259c640 1 --2- 172.21.15.76:0/1029554712 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f223c0a1ac0 0x7f223c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:44.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.621+0000 7f22437fe640 1 -- 172.21.15.76:0/1029554712 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f224c0cd6f0 con 0x7f22541027a0 2024-03-12T05:53:44.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.621+0000 7f225259c640 1 --2- 172.21.15.76:0/1029554712 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f223c0a1ac0 0x7f223c0a3f80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f224402c6a0 tx=0x7f2244005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:44.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.625+0000 7f22437fe640 1 -- 172.21.15.76:0/1029554712 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f224c096950 con 0x7f22541027a0 2024-03-12T05:53:44.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:44 smithi082 ceph-mon[36382]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:44.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:44 smithi082 ceph-mon[36382]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-12T05:53:44.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:44 smithi082 ceph-mon[36382]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:44.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:44 smithi082 ceph-mon[36382]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:44.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:44 smithi088 ceph-mon[36585]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:44.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:44 smithi088 ceph-mon[36585]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-12T05:53:44.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:44 smithi088 ceph-mon[36585]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:44.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:44 smithi088 ceph-mon[36585]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:44.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.798+0000 7f225882a640 1 -- 172.21.15.76:0/1029554712 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f225410cde0 con 0x7f22541027a0 2024-03-12T05:53:44.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.800+0000 7f22437fe640 1 -- 172.21.15.76:0/1029554712 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22110 (secure 0 0 0) 0x7f224c09a010 con 0x7f22541027a0 2024-03-12T05:53:44.803 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:44.803 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":86,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:53:39.189331+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:53:44.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.802+0000 7f225882a640 1 -- 172.21.15.76:0/1029554712 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f223c0a1ac0 msgr2=0x7f223c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.802+0000 7f225882a640 1 --2- 172.21.15.76:0/1029554712 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f223c0a1ac0 0x7f223c0a3f80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f224402c6a0 tx=0x7f2244005fd0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.802+0000 7f225882a640 1 -- 172.21.15.76:0/1029554712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22541027a0 msgr2=0x7f2254100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.802+0000 7f225882a640 1 --2- 172.21.15.76:0/1029554712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22541027a0 0x7f2254100a20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f224c00ead0 tx=0x7f224c00ef90 comp rx=0 tx=0).stop 2024-03-12T05:53:44.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.802+0000 7f225882a640 1 -- 172.21.15.76:0/1029554712 shutdown_connections 2024-03-12T05:53:44.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.802+0000 7f225882a640 1 --2- 172.21.15.76:0/1029554712 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f223c0a1ac0 0x7f223c0a3f80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.802+0000 7f225882a640 1 --2- 172.21.15.76:0/1029554712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f22541027a0 0x7f2254100a20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.802+0000 7f225882a640 1 --2- 172.21.15.76:0/1029554712 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22540ffbd0 0x7f22541004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.802+0000 7f225882a640 1 --2- 172.21.15.76:0/1029554712 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f22540ff210 0x7f2254101ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.802+0000 7f225882a640 1 -- 172.21.15.76:0/1029554712 >> 172.21.15.76:0/1029554712 conn(0x7f22540fadb0 msgr2=0x7f2254105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:44.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.802+0000 7f225882a640 1 -- 172.21.15.76:0/1029554712 shutdown_connections 2024-03-12T05:53:44.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.802+0000 7f225882a640 1 -- 172.21.15.76:0/1029554712 wait complete. 2024-03-12T05:53:44.816 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-12T05:53:44.816 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-12T05:53:44.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 -- 172.21.15.76:0/141574815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0f4102400 msgr2=0x7fc0f4102880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 --2- 172.21.15.76:0/141574815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0f4102400 0x7fc0f4102880 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009900 tx=0x7fc0e802bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:44.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 -- 172.21.15.76:0/141574815 shutdown_connections 2024-03-12T05:53:44.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 --2- 172.21.15.76:0/141574815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0f4102dc0 0x7fc0f4109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 --2- 172.21.15.76:0/141574815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0f4102400 0x7fc0f4102880 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 --2- 172.21.15.76:0/141574815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f4101210 0x7fc0f4101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 -- 172.21.15.76:0/141574815 >> 172.21.15.76:0/141574815 conn(0x7fc0f40fc7a0 msgr2=0x7fc0f40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 -- 172.21.15.76:0/141574815 shutdown_connections 2024-03-12T05:53:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 -- 172.21.15.76:0/141574815 wait complete. 2024-03-12T05:53:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 Processor -- start 2024-03-12T05:53:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 -- start start 2024-03-12T05:53:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0f4101210 0x7fc0f4197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0f4102400 0x7fc0f41977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f4102dc0 0x7fc0f419e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f419ed70 con 0x7fc0f4102400 2024-03-12T05:53:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f419eee0 con 0x7fc0f4101210 2024-03-12T05:53:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.916+0000 7fc0fac40640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f419f0a0 con 0x7fc0f4102dc0 2024-03-12T05:53:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0f3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0f4102400 0x7fc0f41977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0f3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0f4102400 0x7fc0f41977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53592/0 (socket says 172.21.15.76:53592) 2024-03-12T05:53:44.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0f3fff640 1 -- 172.21.15.76:0/1937737473 learned_addr learned my addr 172.21.15.76:0/1937737473 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0f91dd640 1 --2- 172.21.15.76:0/1937737473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f4102dc0 0x7fc0f419e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0f3fff640 1 -- 172.21.15.76:0/1937737473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f4102dc0 msgr2=0x7fc0f419e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0f3fff640 1 --2- 172.21.15.76:0/1937737473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f4102dc0 0x7fc0f419e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0f3fff640 1 -- 172.21.15.76:0/1937737473 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0f4101210 msgr2=0x7fc0f4197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0f3fff640 1 --2- 172.21.15.76:0/1937737473 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0f4101210 0x7fc0f4197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0f3fff640 1 -- 172.21.15.76:0/1937737473 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e8009560 con 0x7fc0f4102400 2024-03-12T05:53:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0f91dd640 1 --2- 172.21.15.76:0/1937737473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f4102dc0 0x7fc0f419e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0f3fff640 1 --2- 172.21.15.76:0/1937737473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0f4102400 0x7fc0f41977c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8006010 tx=0x7fc0e80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0f1ffb640 1 -- 172.21.15.76:0/1937737473 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e80044e0 con 0x7fc0f4102400 2024-03-12T05:53:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0f1ffb640 1 -- 172.21.15.76:0/1937737473 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0e80054e0 con 0x7fc0f4102400 2024-03-12T05:53:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0f1ffb640 1 -- 172.21.15.76:0/1937737473 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e803e450 con 0x7fc0f4102400 2024-03-12T05:53:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0fac40640 1 -- 172.21.15.76:0/1937737473 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f419f320 con 0x7fc0f4102400 2024-03-12T05:53:44.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.917+0000 7fc0fac40640 1 -- 172.21.15.76:0/1937737473 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f419f8b0 con 0x7fc0f4102400 2024-03-12T05:53:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.918+0000 7fc0fac40640 1 -- 172.21.15.76:0/1937737473 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f4108530 con 0x7fc0f4102400 2024-03-12T05:53:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.919+0000 7fc0f1ffb640 1 -- 172.21.15.76:0/1937737473 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fc0e80035f0 con 0x7fc0f4102400 2024-03-12T05:53:44.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.920+0000 7fc0f1ffb640 1 --2- 172.21.15.76:0/1937737473 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc0dc0a1ac0 0x7fc0dc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:44.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.920+0000 7fc0f89dc640 1 --2- 172.21.15.76:0/1937737473 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc0dc0a1ac0 0x7fc0dc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:44.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.920+0000 7fc0f1ffb640 1 -- 172.21.15.76:0/1937737473 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc0e80ef780 con 0x7fc0f4102400 2024-03-12T05:53:44.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.920+0000 7fc0f89dc640 1 --2- 172.21.15.76:0/1937737473 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc0dc0a1ac0 0x7fc0dc0a3f80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4102260 tx=0x7fc0ec009250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:44.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:44.924+0000 7fc0f1ffb640 1 -- 172.21.15.76:0/1937737473 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc0e80b8af0 con 0x7fc0f4102400 2024-03-12T05:53:45.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.093+0000 7fc0fac40640 1 -- 172.21.15.76:0/1937737473 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc0f402ceb0 con 0x7fc0dc0a1ac0 2024-03-12T05:53:45.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.101+0000 7fc0f1ffb640 1 -- 172.21.15.76:0/1937737473 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258753 (secure 0 0 0) 0x7fc0f402ceb0 con 0x7fc0dc0a1ac0 2024-03-12T05:53:45.108 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:45.109 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-03-12T05:53:45.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.104+0000 7fc0fac40640 1 -- 172.21.15.76:0/1937737473 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc0dc0a1ac0 msgr2=0x7fc0dc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:45.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.104+0000 7fc0fac40640 1 --2- 172.21.15.76:0/1937737473 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc0dc0a1ac0 0x7fc0dc0a3f80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4102260 tx=0x7fc0ec009250 comp rx=0 tx=0).stop 2024-03-12T05:53:45.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.104+0000 7fc0fac40640 1 -- 172.21.15.76:0/1937737473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0f4102400 msgr2=0x7fc0f41977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:45.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.104+0000 7fc0fac40640 1 --2- 172.21.15.76:0/1937737473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0f4102400 0x7fc0f41977c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8006010 tx=0x7fc0e80042f0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.104+0000 7fc0fac40640 1 -- 172.21.15.76:0/1937737473 shutdown_connections 2024-03-12T05:53:45.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.104+0000 7fc0fac40640 1 --2- 172.21.15.76:0/1937737473 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc0dc0a1ac0 0x7fc0dc0a3f80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.104+0000 7fc0fac40640 1 --2- 172.21.15.76:0/1937737473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0f4102dc0 0x7fc0f419e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.104+0000 7fc0fac40640 1 --2- 172.21.15.76:0/1937737473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0f4102400 0x7fc0f41977c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.104+0000 7fc0fac40640 1 --2- 172.21.15.76:0/1937737473 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0f4101210 0x7fc0f4197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.104+0000 7fc0fac40640 1 -- 172.21.15.76:0/1937737473 >> 172.21.15.76:0/1937737473 conn(0x7fc0f40fc7a0 msgr2=0x7fc0f4105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:45.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.104+0000 7fc0fac40640 1 -- 172.21.15.76:0/1937737473 shutdown_connections 2024-03-12T05:53:45.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.104+0000 7fc0fac40640 1 -- 172.21.15.76:0/1937737473 wait complete. 2024-03-12T05:53:45.118 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-03-12T05:53:44.863677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630462,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":159,"num_read_kb":157,"num_write":310,"num_write_kb":3011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":245,"ondisk_log_size":245,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":252,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":374468,"kb_used_data":12812,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124567356,"statfs":{"total":1151940427776,"available":1151556972544,"internally_reserved":0,"allocated":13119488,"data_stored":7147787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40094,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":23,"num_read_kb":38,"num_write":129,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.675848"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.319004+0000","last_change":"2024-03-12T05:53:38.207712+0000","last_active":"2024-03-12T05:53:39.319004+0000","last_peered":"2024-03-12T05:53:39.319004+0000","last_clean":"2024-03-12T05:53:39.319004+0000","last_became_active":"2024-03-12T05:53:38.207362+0000","last_became_peered":"2024-03-12T05:53:38.207362+0000","last_unstale":"2024-03-12T05:53:39.319004+0000","last_undegraded":"2024-03-12T05:53:39.319004+0000","last_fullsized":"2024-03-12T05:53:39.319004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:19.518959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.209313+0000","last_change":"2024-03-12T05:53:38.206877+0000","last_active":"2024-03-12T05:53:38.209313+0000","last_peered":"2024-03-12T05:53:38.209313+0000","last_clean":"2024-03-12T05:53:38.209313+0000","last_became_active":"2024-03-12T05:53:38.206468+0000","last_became_peered":"2024-03-12T05:53:38.206468+0000","last_unstale":"2024-03-12T05:53:38.209313+0000","last_undegraded":"2024-03-12T05:53:38.209313+0000","last_fullsized":"2024-03-12T05:53:38.209313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:18:43.309306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.215864+0000","last_change":"2024-03-12T05:53:38.214979+0000","last_active":"2024-03-12T05:53:38.215864+0000","last_peered":"2024-03-12T05:53:38.215864+0000","last_clean":"2024-03-12T05:53:38.215864+0000","last_became_active":"2024-03-12T05:53:38.214580+0000","last_became_peered":"2024-03-12T05:53:38.214580+0000","last_unstale":"2024-03-12T05:53:38.215864+0000","last_undegraded":"2024-03-12T05:53:38.215864+0000","last_fullsized":"2024-03-12T05:53:38.215864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:14:04.635961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.198731+0000","last_change":"2024-03-12T05:53:38.218645+0000","last_active":"2024-03-12T05:53:39.198731+0000","last_peered":"2024-03-12T05:53:39.198731+0000","last_clean":"2024-03-12T05:53:39.198731+0000","last_became_active":"2024-03-12T05:53:38.218254+0000","last_became_peered":"2024-03-12T05:53:38.218254+0000","last_unstale":"2024-03-12T05:53:39.198731+0000","last_undegraded":"2024-03-12T05:53:39.198731+0000","last_fullsized":"2024-03-12T05:53:39.198731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:46:09.544989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.183433+0000","last_change":"2024-03-12T05:53:38.214588+0000","last_active":"2024-03-12T05:53:40.183433+0000","last_peered":"2024-03-12T05:53:40.183433+0000","last_clean":"2024-03-12T05:53:40.183433+0000","last_became_active":"2024-03-12T05:53:38.214192+0000","last_became_peered":"2024-03-12T05:53:38.214192+0000","last_unstale":"2024-03-12T05:53:40.183433+0000","last_undegraded":"2024-03-12T05:53:40.183433+0000","last_fullsized":"2024-03-12T05:53:40.183433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:54:04.880901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.385422+0000","last_change":"2024-03-12T05:53:38.214315+0000","last_active":"2024-03-12T05:53:40.385422+0000","last_peered":"2024-03-12T05:53:40.385422+0000","last_clean":"2024-03-12T05:53:40.385422+0000","last_became_active":"2024-03-12T05:53:38.213881+0000","last_became_peered":"2024-03-12T05:53:38.213881+0000","last_unstale":"2024-03-12T05:53:40.385422+0000","last_undegraded":"2024-03-12T05:53:40.385422+0000","last_fullsized":"2024-03-12T05:53:40.385422+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:11:32.834418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465273+0000","last_change":"2024-03-12T05:53:38.214915+0000","last_active":"2024-03-12T05:53:39.465273+0000","last_peered":"2024-03-12T05:53:39.465273+0000","last_clean":"2024-03-12T05:53:39.465273+0000","last_became_active":"2024-03-12T05:53:38.214592+0000","last_became_peered":"2024-03-12T05:53:38.214592+0000","last_unstale":"2024-03-12T05:53:39.465273+0000","last_undegraded":"2024-03-12T05:53:39.465273+0000","last_fullsized":"2024-03-12T05:53:39.465273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:11.548585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197339+0000","last_change":"2024-03-12T05:53:38.219201+0000","last_active":"2024-03-12T05:53:39.197339+0000","last_peered":"2024-03-12T05:53:39.197339+0000","last_clean":"2024-03-12T05:53:39.197339+0000","last_became_active":"2024-03-12T05:53:38.218833+0000","last_became_peered":"2024-03-12T05:53:38.218833+0000","last_unstale":"2024-03-12T05:53:39.197339+0000","last_undegraded":"2024-03-12T05:53:39.197339+0000","last_fullsized":"2024-03-12T05:53:39.197339+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:13:36.177325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.385062+0000","last_change":"2024-03-12T05:53:38.205126+0000","last_active":"2024-03-12T05:53:40.385062+0000","last_peered":"2024-03-12T05:53:40.385062+0000","last_clean":"2024-03-12T05:53:40.385062+0000","last_became_active":"2024-03-12T05:53:38.204478+0000","last_became_peered":"2024-03-12T05:53:38.204478+0000","last_unstale":"2024-03-12T05:53:40.385062+0000","last_undegraded":"2024-03-12T05:53:40.385062+0000","last_fullsized":"2024-03-12T05:53:40.385062+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:25:56.475134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.387119+0000","last_change":"2024-03-12T05:53:38.215186+0000","last_active":"2024-03-12T05:53:40.387119+0000","last_peered":"2024-03-12T05:53:40.387119+0000","last_clean":"2024-03-12T05:53:40.387119+0000","last_became_active":"2024-03-12T05:53:38.214832+0000","last_became_peered":"2024-03-12T05:53:38.214832+0000","last_unstale":"2024-03-12T05:53:40.387119+0000","last_undegraded":"2024-03-12T05:53:40.387119+0000","last_fullsized":"2024-03-12T05:53:40.387119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:56:26.621990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465591+0000","last_change":"2024-03-12T05:53:38.222347+0000","last_active":"2024-03-12T05:53:39.465591+0000","last_peered":"2024-03-12T05:53:39.465591+0000","last_clean":"2024-03-12T05:53:39.465591+0000","last_became_active":"2024-03-12T05:53:38.221637+0000","last_became_peered":"2024-03-12T05:53:38.221637+0000","last_unstale":"2024-03-12T05:53:39.465591+0000","last_undegraded":"2024-03-12T05:53:39.465591+0000","last_fullsized":"2024-03-12T05:53:39.465591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:16:26.652673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.198722+0000","last_change":"2024-03-12T05:53:38.218687+0000","last_active":"2024-03-12T05:53:39.198722+0000","last_peered":"2024-03-12T05:53:39.198722+0000","last_clean":"2024-03-12T05:53:39.198722+0000","last_became_active":"2024-03-12T05:53:38.218393+0000","last_became_peered":"2024-03-12T05:53:38.218393+0000","last_unstale":"2024-03-12T05:53:39.198722+0000","last_undegraded":"2024-03-12T05:53:39.198722+0000","last_fullsized":"2024-03-12T05:53:39.198722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:41:39.918992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.912472+0000","last_change":"2024-03-12T05:53:38.207432+0000","last_active":"2024-03-12T05:53:39.912472+0000","last_peered":"2024-03-12T05:53:39.912472+0000","last_clean":"2024-03-12T05:53:39.912472+0000","last_became_active":"2024-03-12T05:53:38.207099+0000","last_became_peered":"2024-03-12T05:53:38.207099+0000","last_unstale":"2024-03-12T05:53:39.912472+0000","last_undegraded":"2024-03-12T05:53:39.912472+0000","last_fullsized":"2024-03-12T05:53:39.912472+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:20:08.504723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.209282+0000","last_change":"2024-03-12T05:53:38.207086+0000","last_active":"2024-03-12T05:53:38.209282+0000","last_peered":"2024-03-12T05:53:38.209282+0000","last_clean":"2024-03-12T05:53:38.209282+0000","last_became_active":"2024-03-12T05:53:38.206714+0000","last_became_peered":"2024-03-12T05:53:38.206714+0000","last_unstale":"2024-03-12T05:53:38.209282+0000","last_undegraded":"2024-03-12T05:53:38.209282+0000","last_fullsized":"2024-03-12T05:53:38.209282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:13:45.966851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.218308+0000","last_change":"2024-03-12T05:53:38.217179+0000","last_active":"2024-03-12T05:53:38.218308+0000","last_peered":"2024-03-12T05:53:38.218308+0000","last_clean":"2024-03-12T05:53:38.218308+0000","last_became_active":"2024-03-12T05:53:38.216820+0000","last_became_peered":"2024-03-12T05:53:38.216820+0000","last_unstale":"2024-03-12T05:53:38.218308+0000","last_undegraded":"2024-03-12T05:53:38.218308+0000","last_fullsized":"2024-03-12T05:53:38.218308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:40:00.006259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.384995+0000","last_change":"2024-03-12T05:53:38.220224+0000","last_active":"2024-03-12T05:53:40.384995+0000","last_peered":"2024-03-12T05:53:40.384995+0000","last_clean":"2024-03-12T05:53:40.384995+0000","last_became_active":"2024-03-12T05:53:38.219878+0000","last_became_peered":"2024-03-12T05:53:38.219878+0000","last_unstale":"2024-03-12T05:53:40.384995+0000","last_undegraded":"2024-03-12T05:53:40.384995+0000","last_fullsized":"2024-03-12T05:53:40.384995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:53:16.545807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.319135+0000","last_change":"2024-03-12T05:53:38.208524+0000","last_active":"2024-03-12T05:53:39.319135+0000","last_peered":"2024-03-12T05:53:39.319135+0000","last_clean":"2024-03-12T05:53:39.319135+0000","last_became_active":"2024-03-12T05:53:38.208209+0000","last_became_peered":"2024-03-12T05:53:38.208209+0000","last_unstale":"2024-03-12T05:53:39.319135+0000","last_undegraded":"2024-03-12T05:53:39.319135+0000","last_fullsized":"2024-03-12T05:53:39.319135+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:28:56.334066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.367484+0000","last_change":"2024-03-12T05:53:38.208035+0000","last_active":"2024-03-12T05:53:40.367484+0000","last_peered":"2024-03-12T05:53:40.367484+0000","last_clean":"2024-03-12T05:53:40.367484+0000","last_became_active":"2024-03-12T05:53:38.207679+0000","last_became_peered":"2024-03-12T05:53:38.207679+0000","last_unstale":"2024-03-12T05:53:40.367484+0000","last_undegraded":"2024-03-12T05:53:40.367484+0000","last_fullsized":"2024-03-12T05:53:40.367484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:40:20.620552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.334689+0000","last_change":"2024-03-12T05:53:38.207077+0000","last_active":"2024-03-12T05:53:39.334689+0000","last_peered":"2024-03-12T05:53:39.334689+0000","last_clean":"2024-03-12T05:53:39.334689+0000","last_became_active":"2024-03-12T05:53:38.206720+0000","last_became_peered":"2024-03-12T05:53:38.206720+0000","last_unstale":"2024-03-12T05:53:39.334689+0000","last_undegraded":"2024-03-12T05:53:39.334689+0000","last_fullsized":"2024-03-12T05:53:39.334689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:45:54.628394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.377150+0000","last_change":"2024-03-12T05:53:37.201637+0000","last_active":"2024-03-12T05:53:41.377150+0000","last_peered":"2024-03-12T05:53:41.377150+0000","last_clean":"2024-03-12T05:53:41.377150+0000","last_became_active":"2024-03-12T05:53:37.201289+0000","last_became_peered":"2024-03-12T05:53:37.201289+0000","last_unstale":"2024-03-12T05:53:41.377150+0000","last_undegraded":"2024-03-12T05:53:41.377150+0000","last_fullsized":"2024-03-12T05:53:41.377150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:59:43.829570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197427+0000","last_change":"2024-03-12T05:53:38.218052+0000","last_active":"2024-03-12T05:53:39.197427+0000","last_peered":"2024-03-12T05:53:39.197427+0000","last_clean":"2024-03-12T05:53:39.197427+0000","last_became_active":"2024-03-12T05:53:38.217738+0000","last_became_peered":"2024-03-12T05:53:38.217738+0000","last_unstale":"2024-03-12T05:53:39.197427+0000","last_undegraded":"2024-03-12T05:53:39.197427+0000","last_fullsized":"2024-03-12T05:53:39.197427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:59:40.599573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197645+0000","last_change":"2024-03-12T05:53:37.210321+0000","last_active":"2024-03-12T05:53:39.197645+0000","last_peered":"2024-03-12T05:53:39.197645+0000","last_clean":"2024-03-12T05:53:39.197645+0000","last_became_active":"2024-03-12T05:53:37.210018+0000","last_became_peered":"2024-03-12T05:53:37.210018+0000","last_unstale":"2024-03-12T05:53:39.197645+0000","last_undegraded":"2024-03-12T05:53:39.197645+0000","last_fullsized":"2024-03-12T05:53:39.197645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:13:45.433332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.209340+0000","last_change":"2024-03-12T05:53:38.207801+0000","last_active":"2024-03-12T05:53:38.209340+0000","last_peered":"2024-03-12T05:53:38.209340+0000","last_clean":"2024-03-12T05:53:38.209340+0000","last_became_active":"2024-03-12T05:53:38.207467+0000","last_became_peered":"2024-03-12T05:53:38.207467+0000","last_unstale":"2024-03-12T05:53:38.209340+0000","last_undegraded":"2024-03-12T05:53:38.209340+0000","last_fullsized":"2024-03-12T05:53:38.209340+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:22:30.434799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.376310+0000","last_change":"2024-03-12T05:53:37.208288+0000","last_active":"2024-03-12T05:53:41.376310+0000","last_peered":"2024-03-12T05:53:41.376310+0000","last_clean":"2024-03-12T05:53:41.376310+0000","last_became_active":"2024-03-12T05:53:37.207929+0000","last_became_peered":"2024-03-12T05:53:37.207929+0000","last_unstale":"2024-03-12T05:53:41.376310+0000","last_undegraded":"2024-03-12T05:53:41.376310+0000","last_fullsized":"2024-03-12T05:53:41.376310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:10:52.719116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.385095+0000","last_change":"2024-03-12T05:53:38.207270+0000","last_active":"2024-03-12T05:53:40.385095+0000","last_peered":"2024-03-12T05:53:40.385095+0000","last_clean":"2024-03-12T05:53:40.385095+0000","last_became_active":"2024-03-12T05:53:38.205782+0000","last_became_peered":"2024-03-12T05:53:38.205782+0000","last_unstale":"2024-03-12T05:53:40.385095+0000","last_undegraded":"2024-03-12T05:53:40.385095+0000","last_fullsized":"2024-03-12T05:53:40.385095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:02:24.875444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.183675+0000","last_change":"2024-03-12T05:53:37.203899+0000","last_active":"2024-03-12T05:53:40.183675+0000","last_peered":"2024-03-12T05:53:40.183675+0000","last_clean":"2024-03-12T05:53:40.183675+0000","last_became_active":"2024-03-12T05:53:37.203450+0000","last_became_peered":"2024-03-12T05:53:37.203450+0000","last_unstale":"2024-03-12T05:53:40.183675+0000","last_undegraded":"2024-03-12T05:53:40.183675+0000","last_fullsized":"2024-03-12T05:53:40.183675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:25.773826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197718+0000","last_change":"2024-03-12T05:53:38.211955+0000","last_active":"2024-03-12T05:53:39.197718+0000","last_peered":"2024-03-12T05:53:39.197718+0000","last_clean":"2024-03-12T05:53:39.197718+0000","last_became_active":"2024-03-12T05:53:38.211547+0000","last_became_peered":"2024-03-12T05:53:38.211547+0000","last_unstale":"2024-03-12T05:53:39.197718+0000","last_undegraded":"2024-03-12T05:53:39.197718+0000","last_fullsized":"2024-03-12T05:53:39.197718+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:53:56.984213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197670+0000","last_change":"2024-03-12T05:53:37.205401+0000","last_active":"2024-03-12T05:53:39.197670+0000","last_peered":"2024-03-12T05:53:39.197670+0000","last_clean":"2024-03-12T05:53:39.197670+0000","last_became_active":"2024-03-12T05:53:37.204993+0000","last_became_peered":"2024-03-12T05:53:37.204993+0000","last_unstale":"2024-03-12T05:53:39.197670+0000","last_undegraded":"2024-03-12T05:53:39.197670+0000","last_fullsized":"2024-03-12T05:53:39.197670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:56:06.177387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.183472+0000","last_change":"2024-03-12T05:53:38.210562+0000","last_active":"2024-03-12T05:53:40.183472+0000","last_peered":"2024-03-12T05:53:40.183472+0000","last_clean":"2024-03-12T05:53:40.183472+0000","last_became_active":"2024-03-12T05:53:38.210235+0000","last_became_peered":"2024-03-12T05:53:38.210235+0000","last_unstale":"2024-03-12T05:53:40.183472+0000","last_undegraded":"2024-03-12T05:53:40.183472+0000","last_fullsized":"2024-03-12T05:53:40.183472+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:35.445812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.374540+0000","last_change":"2024-03-12T05:53:37.204986+0000","last_active":"2024-03-12T05:53:41.374540+0000","last_peered":"2024-03-12T05:53:41.374540+0000","last_clean":"2024-03-12T05:53:41.374540+0000","last_became_active":"2024-03-12T05:53:37.204580+0000","last_became_peered":"2024-03-12T05:53:37.204580+0000","last_unstale":"2024-03-12T05:53:41.374540+0000","last_undegraded":"2024-03-12T05:53:41.374540+0000","last_fullsized":"2024-03-12T05:53:41.374540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:37:34.584395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197554+0000","last_change":"2024-03-12T05:53:38.214948+0000","last_active":"2024-03-12T05:53:39.197554+0000","last_peered":"2024-03-12T05:53:39.197554+0000","last_clean":"2024-03-12T05:53:39.197554+0000","last_became_active":"2024-03-12T05:53:38.213748+0000","last_became_peered":"2024-03-12T05:53:38.213748+0000","last_unstale":"2024-03-12T05:53:39.197554+0000","last_undegraded":"2024-03-12T05:53:39.197554+0000","last_fullsized":"2024-03-12T05:53:39.197554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:12:45.994129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.375650+0000","last_change":"2024-03-12T05:53:37.198167+0000","last_active":"2024-03-12T05:53:41.375650+0000","last_peered":"2024-03-12T05:53:41.375650+0000","last_clean":"2024-03-12T05:53:41.375650+0000","last_became_active":"2024-03-12T05:53:37.197808+0000","last_became_peered":"2024-03-12T05:53:37.197808+0000","last_unstale":"2024-03-12T05:53:41.375650+0000","last_undegraded":"2024-03-12T05:53:41.375650+0000","last_fullsized":"2024-03-12T05:53:41.375650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:40:19.498546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197306+0000","last_change":"2024-03-12T05:53:38.205464+0000","last_active":"2024-03-12T05:53:39.197306+0000","last_peered":"2024-03-12T05:53:39.197306+0000","last_clean":"2024-03-12T05:53:39.197306+0000","last_became_active":"2024-03-12T05:53:38.205129+0000","last_became_peered":"2024-03-12T05:53:38.205129+0000","last_unstale":"2024-03-12T05:53:39.197306+0000","last_undegraded":"2024-03-12T05:53:39.197306+0000","last_fullsized":"2024-03-12T05:53:39.197306+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:38:55.460862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197255+0000","last_change":"2024-03-12T05:53:37.201572+0000","last_active":"2024-03-12T05:53:39.197255+0000","last_peered":"2024-03-12T05:53:39.197255+0000","last_clean":"2024-03-12T05:53:39.197255+0000","last_became_active":"2024-03-12T05:53:37.201230+0000","last_became_peered":"2024-03-12T05:53:37.201230+0000","last_unstale":"2024-03-12T05:53:39.197255+0000","last_undegraded":"2024-03-12T05:53:39.197255+0000","last_fullsized":"2024-03-12T05:53:39.197255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:16:13.715459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.367343+0000","last_change":"2024-03-12T05:53:38.210696+0000","last_active":"2024-03-12T05:53:40.367343+0000","last_peered":"2024-03-12T05:53:40.367343+0000","last_clean":"2024-03-12T05:53:40.367343+0000","last_became_active":"2024-03-12T05:53:38.210325+0000","last_became_peered":"2024-03-12T05:53:38.210325+0000","last_unstale":"2024-03-12T05:53:40.367343+0000","last_undegraded":"2024-03-12T05:53:40.367343+0000","last_fullsized":"2024-03-12T05:53:40.367343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:25:36.397314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:42.368628+0000","last_change":"2024-03-12T05:53:37.201712+0000","last_active":"2024-03-12T05:53:42.368628+0000","last_peered":"2024-03-12T05:53:42.368628+0000","last_clean":"2024-03-12T05:53:42.368628+0000","last_became_active":"2024-03-12T05:53:37.200794+0000","last_became_peered":"2024-03-12T05:53:37.200794+0000","last_unstale":"2024-03-12T05:53:42.368628+0000","last_undegraded":"2024-03-12T05:53:42.368628+0000","last_fullsized":"2024-03-12T05:53:42.368628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:43:29.673717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.384766+0000","last_change":"2024-03-12T05:53:38.206647+0000","last_active":"2024-03-12T05:53:40.384766+0000","last_peered":"2024-03-12T05:53:40.384766+0000","last_clean":"2024-03-12T05:53:40.384766+0000","last_became_active":"2024-03-12T05:53:38.206254+0000","last_became_peered":"2024-03-12T05:53:38.206254+0000","last_unstale":"2024-03-12T05:53:40.384766+0000","last_undegraded":"2024-03-12T05:53:40.384766+0000","last_fullsized":"2024-03-12T05:53:40.384766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:47:20.455360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.387336+0000","last_change":"2024-03-12T05:53:37.208692+0000","last_active":"2024-03-12T05:53:40.387336+0000","last_peered":"2024-03-12T05:53:40.387336+0000","last_clean":"2024-03-12T05:53:40.387336+0000","last_became_active":"2024-03-12T05:53:37.208270+0000","last_became_peered":"2024-03-12T05:53:37.208270+0000","last_unstale":"2024-03-12T05:53:40.387336+0000","last_undegraded":"2024-03-12T05:53:40.387336+0000","last_fullsized":"2024-03-12T05:53:40.387336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:51:38.217999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.219740+0000","last_change":"2024-03-12T05:53:38.218202+0000","last_active":"2024-03-12T05:53:38.219740+0000","last_peered":"2024-03-12T05:53:38.219740+0000","last_clean":"2024-03-12T05:53:38.219740+0000","last_became_active":"2024-03-12T05:53:38.217798+0000","last_became_peered":"2024-03-12T05:53:38.217798+0000","last_unstale":"2024-03-12T05:53:38.219740+0000","last_undegraded":"2024-03-12T05:53:38.219740+0000","last_fullsized":"2024-03-12T05:53:38.219740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:22:16.690610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197843+0000","last_change":"2024-03-12T05:53:37.203252+0000","last_active":"2024-03-12T05:53:39.197843+0000","last_peered":"2024-03-12T05:53:39.197843+0000","last_clean":"2024-03-12T05:53:39.197843+0000","last_became_active":"2024-03-12T05:53:37.202916+0000","last_became_peered":"2024-03-12T05:53:37.202916+0000","last_unstale":"2024-03-12T05:53:39.197843+0000","last_undegraded":"2024-03-12T05:53:39.197843+0000","last_fullsized":"2024-03-12T05:53:39.197843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:09:34.629085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.385411+0000","last_change":"2024-03-12T05:53:38.220826+0000","last_active":"2024-03-12T05:53:40.385411+0000","last_peered":"2024-03-12T05:53:40.385411+0000","last_clean":"2024-03-12T05:53:40.385411+0000","last_became_active":"2024-03-12T05:53:38.220440+0000","last_became_peered":"2024-03-12T05:53:38.220440+0000","last_unstale":"2024-03-12T05:53:40.385411+0000","last_undegraded":"2024-03-12T05:53:40.385411+0000","last_fullsized":"2024-03-12T05:53:40.385411+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:53:41.311273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'19","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:43.819785+0000","last_change":"2024-03-12T05:53:37.206271+0000","last_active":"2024-03-12T05:53:43.819785+0000","last_peered":"2024-03-12T05:53:43.819785+0000","last_clean":"2024-03-12T05:53:43.819785+0000","last_became_active":"2024-03-12T05:53:37.205920+0000","last_became_peered":"2024-03-12T05:53:37.205920+0000","last_unstale":"2024-03-12T05:53:43.819785+0000","last_undegraded":"2024-03-12T05:53:43.819785+0000","last_fullsized":"2024-03-12T05:53:43.819785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:30.140159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38193,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":24,"num_write":20,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.183646+0000","last_change":"2024-03-12T05:53:38.211397+0000","last_active":"2024-03-12T05:53:40.183646+0000","last_peered":"2024-03-12T05:53:40.183646+0000","last_clean":"2024-03-12T05:53:40.183646+0000","last_became_active":"2024-03-12T05:53:38.211045+0000","last_became_peered":"2024-03-12T05:53:38.211045+0000","last_unstale":"2024-03-12T05:53:40.183646+0000","last_undegraded":"2024-03-12T05:53:40.183646+0000","last_fullsized":"2024-03-12T05:53:40.183646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:21:42.510436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.377295+0000","last_change":"2024-03-12T05:53:37.210871+0000","last_active":"2024-03-12T05:53:41.377295+0000","last_peered":"2024-03-12T05:53:41.377295+0000","last_clean":"2024-03-12T05:53:41.377295+0000","last_became_active":"2024-03-12T05:53:37.210526+0000","last_became_peered":"2024-03-12T05:53:37.210526+0000","last_unstale":"2024-03-12T05:53:41.377295+0000","last_undegraded":"2024-03-12T05:53:41.377295+0000","last_fullsized":"2024-03-12T05:53:41.377295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:40:46.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.912250+0000","last_change":"2024-03-12T05:53:38.213751+0000","last_active":"2024-03-12T05:53:39.912250+0000","last_peered":"2024-03-12T05:53:39.912250+0000","last_clean":"2024-03-12T05:53:39.912250+0000","last_became_active":"2024-03-12T05:53:38.211558+0000","last_became_peered":"2024-03-12T05:53:38.211558+0000","last_unstale":"2024-03-12T05:53:39.912250+0000","last_undegraded":"2024-03-12T05:53:39.912250+0000","last_fullsized":"2024-03-12T05:53:39.912250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:40:44.194548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197403+0000","last_change":"2024-03-12T05:53:37.207368+0000","last_active":"2024-03-12T05:53:39.197403+0000","last_peered":"2024-03-12T05:53:39.197403+0000","last_clean":"2024-03-12T05:53:39.197403+0000","last_became_active":"2024-03-12T05:53:37.207066+0000","last_became_peered":"2024-03-12T05:53:37.207066+0000","last_unstale":"2024-03-12T05:53:39.197403+0000","last_undegraded":"2024-03-12T05:53:39.197403+0000","last_fullsized":"2024-03-12T05:53:39.197403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:28:58.155360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465553+0000","last_change":"2024-03-12T05:53:38.217079+0000","last_active":"2024-03-12T05:53:39.465553+0000","last_peered":"2024-03-12T05:53:39.465553+0000","last_clean":"2024-03-12T05:53:39.465553+0000","last_became_active":"2024-03-12T05:53:38.216757+0000","last_became_peered":"2024-03-12T05:53:38.216757+0000","last_unstale":"2024-03-12T05:53:39.465553+0000","last_undegraded":"2024-03-12T05:53:39.465553+0000","last_fullsized":"2024-03-12T05:53:39.465553+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:48:13.125867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.198280+0000","last_change":"2024-03-12T05:53:37.202390+0000","last_active":"2024-03-12T05:53:39.198280+0000","last_peered":"2024-03-12T05:53:39.198280+0000","last_clean":"2024-03-12T05:53:39.198280+0000","last_became_active":"2024-03-12T05:53:37.202020+0000","last_became_peered":"2024-03-12T05:53:37.202020+0000","last_unstale":"2024-03-12T05:53:39.198280+0000","last_undegraded":"2024-03-12T05:53:39.198280+0000","last_fullsized":"2024-03-12T05:53:39.198280+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:06:35.293227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465285+0000","last_change":"2024-03-12T05:53:38.218464+0000","last_active":"2024-03-12T05:53:39.465285+0000","last_peered":"2024-03-12T05:53:39.465285+0000","last_clean":"2024-03-12T05:53:39.465285+0000","last_became_active":"2024-03-12T05:53:38.218136+0000","last_became_peered":"2024-03-12T05:53:38.218136+0000","last_unstale":"2024-03-12T05:53:39.465285+0000","last_undegraded":"2024-03-12T05:53:39.465285+0000","last_fullsized":"2024-03-12T05:53:39.465285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:31:05.409148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.376801+0000","last_change":"2024-03-12T05:53:37.205161+0000","last_active":"2024-03-12T05:53:41.376801+0000","last_peered":"2024-03-12T05:53:41.376801+0000","last_clean":"2024-03-12T05:53:41.376801+0000","last_became_active":"2024-03-12T05:53:37.204799+0000","last_became_peered":"2024-03-12T05:53:37.204799+0000","last_unstale":"2024-03-12T05:53:41.376801+0000","last_undegraded":"2024-03-12T05:53:41.376801+0000","last_fullsized":"2024-03-12T05:53:41.376801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:29:32.065635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197319+0000","last_change":"2024-03-12T05:53:38.216321+0000","last_active":"2024-03-12T05:53:39.197319+0000","last_peered":"2024-03-12T05:53:39.197319+0000","last_clean":"2024-03-12T05:53:39.197319+0000","last_became_active":"2024-03-12T05:53:38.215893+0000","last_became_peered":"2024-03-12T05:53:38.215893+0000","last_unstale":"2024-03-12T05:53:39.197319+0000","last_undegraded":"2024-03-12T05:53:39.197319+0000","last_fullsized":"2024-03-12T05:53:39.197319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:32:27.603930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.183844+0000","last_change":"2024-03-12T05:53:37.203638+0000","last_active":"2024-03-12T05:53:40.183844+0000","last_peered":"2024-03-12T05:53:40.183844+0000","last_clean":"2024-03-12T05:53:40.183844+0000","last_became_active":"2024-03-12T05:53:37.203217+0000","last_became_peered":"2024-03-12T05:53:37.203217+0000","last_unstale":"2024-03-12T05:53:40.183844+0000","last_undegraded":"2024-03-12T05:53:40.183844+0000","last_fullsized":"2024-03-12T05:53:40.183844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:10.521197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.318913+0000","last_change":"2024-03-12T05:53:38.210491+0000","last_active":"2024-03-12T05:53:39.318913+0000","last_peered":"2024-03-12T05:53:39.318913+0000","last_clean":"2024-03-12T05:53:39.318913+0000","last_became_active":"2024-03-12T05:53:38.210169+0000","last_became_peered":"2024-03-12T05:53:38.210169+0000","last_unstale":"2024-03-12T05:53:39.318913+0000","last_undegraded":"2024-03-12T05:53:39.318913+0000","last_fullsized":"2024-03-12T05:53:39.318913+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:56:05.384834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.367267+0000","last_change":"2024-03-12T05:53:37.204478+0000","last_active":"2024-03-12T05:53:40.367267+0000","last_peered":"2024-03-12T05:53:40.367267+0000","last_clean":"2024-03-12T05:53:40.367267+0000","last_became_active":"2024-03-12T05:53:37.204160+0000","last_became_peered":"2024-03-12T05:53:37.204160+0000","last_unstale":"2024-03-12T05:53:40.367267+0000","last_undegraded":"2024-03-12T05:53:40.367267+0000","last_fullsized":"2024-03-12T05:53:40.367267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:07.597731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.387503+0000","last_change":"2024-03-12T05:53:38.212913+0000","last_active":"2024-03-12T05:53:40.387503+0000","last_peered":"2024-03-12T05:53:40.387503+0000","last_clean":"2024-03-12T05:53:40.387503+0000","last_became_active":"2024-03-12T05:53:38.212558+0000","last_became_peered":"2024-03-12T05:53:38.212558+0000","last_unstale":"2024-03-12T05:53:40.387503+0000","last_undegraded":"2024-03-12T05:53:40.387503+0000","last_fullsized":"2024-03-12T05:53:40.387503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:40:59.993066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465726+0000","last_change":"2024-03-12T05:53:37.205983+0000","last_active":"2024-03-12T05:53:39.465726+0000","last_peered":"2024-03-12T05:53:39.465726+0000","last_clean":"2024-03-12T05:53:39.465726+0000","last_became_active":"2024-03-12T05:53:37.205629+0000","last_became_peered":"2024-03-12T05:53:37.205629+0000","last_unstale":"2024-03-12T05:53:39.465726+0000","last_undegraded":"2024-03-12T05:53:39.465726+0000","last_fullsized":"2024-03-12T05:53:39.465726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:37.928623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.387032+0000","last_change":"2024-03-12T05:53:38.213752+0000","last_active":"2024-03-12T05:53:40.387032+0000","last_peered":"2024-03-12T05:53:40.387032+0000","last_clean":"2024-03-12T05:53:40.387032+0000","last_became_active":"2024-03-12T05:53:38.213407+0000","last_became_peered":"2024-03-12T05:53:38.213407+0000","last_unstale":"2024-03-12T05:53:40.387032+0000","last_undegraded":"2024-03-12T05:53:40.387032+0000","last_fullsized":"2024-03-12T05:53:40.387032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:21:15.703455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197688+0000","last_change":"2024-03-12T05:53:38.218205+0000","last_active":"2024-03-12T05:53:39.197688+0000","last_peered":"2024-03-12T05:53:39.197688+0000","last_clean":"2024-03-12T05:53:39.197688+0000","last_became_active":"2024-03-12T05:53:38.217725+0000","last_became_peered":"2024-03-12T05:53:38.217725+0000","last_unstale":"2024-03-12T05:53:39.197688+0000","last_undegraded":"2024-03-12T05:53:39.197688+0000","last_fullsized":"2024-03-12T05:53:39.197688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:48:30.050343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:43.393762+0000","last_change":"2024-03-12T05:53:37.202782+0000","last_active":"2024-03-12T05:53:43.393762+0000","last_peered":"2024-03-12T05:53:43.393762+0000","last_clean":"2024-03-12T05:53:43.393762+0000","last_became_active":"2024-03-12T05:53:37.202439+0000","last_became_peered":"2024-03-12T05:53:37.202439+0000","last_unstale":"2024-03-12T05:53:43.393762+0000","last_undegraded":"2024-03-12T05:53:43.393762+0000","last_fullsized":"2024-03-12T05:53:43.393762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:07:18.939629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197714+0000","last_change":"2024-03-12T05:53:38.217993+0000","last_active":"2024-03-12T05:53:39.197714+0000","last_peered":"2024-03-12T05:53:39.197714+0000","last_clean":"2024-03-12T05:53:39.197714+0000","last_became_active":"2024-03-12T05:53:38.217613+0000","last_became_peered":"2024-03-12T05:53:38.217613+0000","last_unstale":"2024-03-12T05:53:39.197714+0000","last_undegraded":"2024-03-12T05:53:39.197714+0000","last_fullsized":"2024-03-12T05:53:39.197714+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:49:50.960004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.388344+0000","last_change":"2024-03-12T05:53:37.203662+0000","last_active":"2024-03-12T05:53:40.388344+0000","last_peered":"2024-03-12T05:53:40.388344+0000","last_clean":"2024-03-12T05:53:40.388344+0000","last_became_active":"2024-03-12T05:53:37.203309+0000","last_became_peered":"2024-03-12T05:53:37.203309+0000","last_unstale":"2024-03-12T05:53:40.388344+0000","last_undegraded":"2024-03-12T05:53:40.388344+0000","last_fullsized":"2024-03-12T05:53:40.388344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:47:24.603686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465034+0000","last_change":"2024-03-12T05:52:27.278524+0000","last_active":"2024-03-12T05:53:39.465034+0000","last_peered":"2024-03-12T05:53:39.465034+0000","last_clean":"2024-03-12T05:53:39.465034+0000","last_became_active":"2024-03-12T05:52:27.278010+0000","last_became_peered":"2024-03-12T05:52:27.278010+0000","last_unstale":"2024-03-12T05:53:39.465034+0000","last_undegraded":"2024-03-12T05:53:39.465034+0000","last_fullsized":"2024-03-12T05:53:39.465034+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:49:09.175725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:49:09.175725+0000","last_clean_scrub_stamp":"2024-03-12T05:49:09.175725+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:24:40.380724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.912134+0000","last_change":"2024-03-12T05:53:38.219451+0000","last_active":"2024-03-12T05:53:39.912134+0000","last_peered":"2024-03-12T05:53:39.912134+0000","last_clean":"2024-03-12T05:53:39.912134+0000","last_became_active":"2024-03-12T05:53:38.218840+0000","last_became_peered":"2024-03-12T05:53:38.218840+0000","last_unstale":"2024-03-12T05:53:39.912134+0000","last_undegraded":"2024-03-12T05:53:39.912134+0000","last_fullsized":"2024-03-12T05:53:39.912134+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:05.595822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.375093+0000","last_change":"2024-03-12T05:53:37.210352+0000","last_active":"2024-03-12T05:53:41.375093+0000","last_peered":"2024-03-12T05:53:41.375093+0000","last_clean":"2024-03-12T05:53:41.375093+0000","last_became_active":"2024-03-12T05:53:37.210002+0000","last_became_peered":"2024-03-12T05:53:37.210002+0000","last_unstale":"2024-03-12T05:53:41.375093+0000","last_undegraded":"2024-03-12T05:53:41.375093+0000","last_fullsized":"2024-03-12T05:53:41.375093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:49:52.368411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.385031+0000","last_change":"2024-03-12T05:53:37.201136+0000","last_active":"2024-03-12T05:53:40.385031+0000","last_peered":"2024-03-12T05:53:40.385031+0000","last_clean":"2024-03-12T05:53:40.385031+0000","last_became_active":"2024-03-12T05:53:37.200695+0000","last_became_peered":"2024-03-12T05:53:37.200695+0000","last_unstale":"2024-03-12T05:53:40.385031+0000","last_undegraded":"2024-03-12T05:53:40.385031+0000","last_fullsized":"2024-03-12T05:53:40.385031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:55:22.428183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.912404+0000","last_change":"2024-03-12T05:53:38.203951+0000","last_active":"2024-03-12T05:53:39.912404+0000","last_peered":"2024-03-12T05:53:39.912404+0000","last_clean":"2024-03-12T05:53:39.912404+0000","last_became_active":"2024-03-12T05:53:38.203617+0000","last_became_peered":"2024-03-12T05:53:38.203617+0000","last_unstale":"2024-03-12T05:53:39.912404+0000","last_undegraded":"2024-03-12T05:53:39.912404+0000","last_fullsized":"2024-03-12T05:53:39.912404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:19.138220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.377824+0000","last_change":"2024-03-12T05:53:37.203770+0000","last_active":"2024-03-12T05:53:41.377824+0000","last_peered":"2024-03-12T05:53:41.377824+0000","last_clean":"2024-03-12T05:53:41.377824+0000","last_became_active":"2024-03-12T05:53:37.203456+0000","last_became_peered":"2024-03-12T05:53:37.203456+0000","last_unstale":"2024-03-12T05:53:41.377824+0000","last_undegraded":"2024-03-12T05:53:41.377824+0000","last_fullsized":"2024-03-12T05:53:41.377824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:32.291536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197684+0000","last_change":"2024-03-12T05:53:38.214872+0000","last_active":"2024-03-12T05:53:39.197684+0000","last_peered":"2024-03-12T05:53:39.197684+0000","last_clean":"2024-03-12T05:53:39.197684+0000","last_became_active":"2024-03-12T05:53:38.214139+0000","last_became_peered":"2024-03-12T05:53:38.214139+0000","last_unstale":"2024-03-12T05:53:39.197684+0000","last_undegraded":"2024-03-12T05:53:39.197684+0000","last_fullsized":"2024-03-12T05:53:39.197684+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:20:21.808039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.385592+0000","last_change":"2024-03-12T05:53:37.203852+0000","last_active":"2024-03-12T05:53:40.385592+0000","last_peered":"2024-03-12T05:53:40.385592+0000","last_clean":"2024-03-12T05:53:40.385592+0000","last_became_active":"2024-03-12T05:53:37.203416+0000","last_became_peered":"2024-03-12T05:53:37.203416+0000","last_unstale":"2024-03-12T05:53:40.385592+0000","last_undegraded":"2024-03-12T05:53:40.385592+0000","last_fullsized":"2024-03-12T05:53:40.385592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:09:08.095957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.213958+0000","last_change":"2024-03-12T05:53:38.211722+0000","last_active":"2024-03-12T05:53:38.213958+0000","last_peered":"2024-03-12T05:53:38.213958+0000","last_clean":"2024-03-12T05:53:38.213958+0000","last_became_active":"2024-03-12T05:53:38.211362+0000","last_became_peered":"2024-03-12T05:53:38.211362+0000","last_unstale":"2024-03-12T05:53:38.213958+0000","last_undegraded":"2024-03-12T05:53:38.213958+0000","last_fullsized":"2024-03-12T05:53:38.213958+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:21.465441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.387757+0000","last_change":"2024-03-12T05:53:37.207444+0000","last_active":"2024-03-12T05:53:40.387757+0000","last_peered":"2024-03-12T05:53:40.387757+0000","last_clean":"2024-03-12T05:53:40.387757+0000","last_became_active":"2024-03-12T05:53:37.207119+0000","last_became_peered":"2024-03-12T05:53:37.207119+0000","last_unstale":"2024-03-12T05:53:40.387757+0000","last_undegraded":"2024-03-12T05:53:40.387757+0000","last_fullsized":"2024-03-12T05:53:40.387757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:20:36.489684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.183387+0000","last_change":"2024-03-12T05:53:38.207757+0000","last_active":"2024-03-12T05:53:40.183387+0000","last_peered":"2024-03-12T05:53:40.183387+0000","last_clean":"2024-03-12T05:53:40.183387+0000","last_became_active":"2024-03-12T05:53:38.207424+0000","last_became_peered":"2024-03-12T05:53:38.207424+0000","last_unstale":"2024-03-12T05:53:40.183387+0000","last_undegraded":"2024-03-12T05:53:40.183387+0000","last_fullsized":"2024-03-12T05:53:40.183387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:33:16.017907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.388215+0000","last_change":"2024-03-12T05:53:37.208512+0000","last_active":"2024-03-12T05:53:40.388215+0000","last_peered":"2024-03-12T05:53:40.388215+0000","last_clean":"2024-03-12T05:53:40.388215+0000","last_became_active":"2024-03-12T05:53:37.207628+0000","last_became_peered":"2024-03-12T05:53:37.207628+0000","last_unstale":"2024-03-12T05:53:40.388215+0000","last_undegraded":"2024-03-12T05:53:40.388215+0000","last_fullsized":"2024-03-12T05:53:40.388215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:25:28.825779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.318575+0000","last_change":"2024-03-12T05:53:38.205654+0000","last_active":"2024-03-12T05:53:39.318575+0000","last_peered":"2024-03-12T05:53:39.318575+0000","last_clean":"2024-03-12T05:53:39.318575+0000","last_became_active":"2024-03-12T05:53:38.205325+0000","last_became_peered":"2024-03-12T05:53:38.205325+0000","last_unstale":"2024-03-12T05:53:39.318575+0000","last_undegraded":"2024-03-12T05:53:39.318575+0000","last_fullsized":"2024-03-12T05:53:39.318575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:33:35.668625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.334333+0000","last_change":"2024-03-12T05:53:37.211430+0000","last_active":"2024-03-12T05:53:39.334333+0000","last_peered":"2024-03-12T05:53:39.334333+0000","last_clean":"2024-03-12T05:53:39.334333+0000","last_became_active":"2024-03-12T05:53:37.210501+0000","last_became_peered":"2024-03-12T05:53:37.210501+0000","last_unstale":"2024-03-12T05:53:39.334333+0000","last_undegraded":"2024-03-12T05:53:39.334333+0000","last_fullsized":"2024-03-12T05:53:39.334333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:48:33.555134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197495+0000","last_change":"2024-03-12T05:53:38.212837+0000","last_active":"2024-03-12T05:53:39.197495+0000","last_peered":"2024-03-12T05:53:39.197495+0000","last_clean":"2024-03-12T05:53:39.197495+0000","last_became_active":"2024-03-12T05:53:38.212513+0000","last_became_peered":"2024-03-12T05:53:38.212513+0000","last_unstale":"2024-03-12T05:53:39.197495+0000","last_undegraded":"2024-03-12T05:53:39.197495+0000","last_fullsized":"2024-03-12T05:53:39.197495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:47:33.879954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.387273+0000","last_change":"2024-03-12T05:53:37.207465+0000","last_active":"2024-03-12T05:53:40.387273+0000","last_peered":"2024-03-12T05:53:40.387273+0000","last_clean":"2024-03-12T05:53:40.387273+0000","last_became_active":"2024-03-12T05:53:37.207107+0000","last_became_peered":"2024-03-12T05:53:37.207107+0000","last_unstale":"2024-03-12T05:53:40.387273+0000","last_undegraded":"2024-03-12T05:53:40.387273+0000","last_fullsized":"2024-03-12T05:53:40.387273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:34.018856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465402+0000","last_change":"2024-03-12T05:53:38.221470+0000","last_active":"2024-03-12T05:53:39.465402+0000","last_peered":"2024-03-12T05:53:39.465402+0000","last_clean":"2024-03-12T05:53:39.465402+0000","last_became_active":"2024-03-12T05:53:38.221061+0000","last_became_peered":"2024-03-12T05:53:38.221061+0000","last_unstale":"2024-03-12T05:53:39.465402+0000","last_undegraded":"2024-03-12T05:53:39.465402+0000","last_fullsized":"2024-03-12T05:53:39.465402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:01:54.148877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.376844+0000","last_change":"2024-03-12T05:53:37.203822+0000","last_active":"2024-03-12T05:53:41.376844+0000","last_peered":"2024-03-12T05:53:41.376844+0000","last_clean":"2024-03-12T05:53:41.376844+0000","last_became_active":"2024-03-12T05:53:37.203431+0000","last_became_peered":"2024-03-12T05:53:37.203431+0000","last_unstale":"2024-03-12T05:53:41.376844+0000","last_undegraded":"2024-03-12T05:53:41.376844+0000","last_fullsized":"2024-03-12T05:53:41.376844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:25:04.447261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465777+0000","last_change":"2024-03-12T05:53:38.222045+0000","last_active":"2024-03-12T05:53:39.465777+0000","last_peered":"2024-03-12T05:53:39.465777+0000","last_clean":"2024-03-12T05:53:39.465777+0000","last_became_active":"2024-03-12T05:53:38.220904+0000","last_became_peered":"2024-03-12T05:53:38.220904+0000","last_unstale":"2024-03-12T05:53:39.465777+0000","last_undegraded":"2024-03-12T05:53:39.465777+0000","last_fullsized":"2024-03-12T05:53:39.465777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:30:43.964599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197795+0000","last_change":"2024-03-12T05:53:37.206753+0000","last_active":"2024-03-12T05:53:39.197795+0000","last_peered":"2024-03-12T05:53:39.197795+0000","last_clean":"2024-03-12T05:53:39.197795+0000","last_became_active":"2024-03-12T05:53:37.206078+0000","last_became_peered":"2024-03-12T05:53:37.206078+0000","last_unstale":"2024-03-12T05:53:39.197795+0000","last_undegraded":"2024-03-12T05:53:39.197795+0000","last_fullsized":"2024-03-12T05:53:39.197795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:42:18.859528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465670+0000","last_change":"2024-03-12T05:53:38.215671+0000","last_active":"2024-03-12T05:53:39.465670+0000","last_peered":"2024-03-12T05:53:39.465670+0000","last_clean":"2024-03-12T05:53:39.465670+0000","last_became_active":"2024-03-12T05:53:38.215309+0000","last_became_peered":"2024-03-12T05:53:38.215309+0000","last_unstale":"2024-03-12T05:53:39.465670+0000","last_undegraded":"2024-03-12T05:53:39.465670+0000","last_fullsized":"2024-03-12T05:53:39.465670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:50:12.176935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.912403+0000","last_change":"2024-03-12T05:53:37.205627+0000","last_active":"2024-03-12T05:53:39.912403+0000","last_peered":"2024-03-12T05:53:39.912403+0000","last_clean":"2024-03-12T05:53:39.912403+0000","last_became_active":"2024-03-12T05:53:37.205289+0000","last_became_peered":"2024-03-12T05:53:37.205289+0000","last_unstale":"2024-03-12T05:53:39.912403+0000","last_undegraded":"2024-03-12T05:53:39.912403+0000","last_fullsized":"2024-03-12T05:53:39.912403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:56.682746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465593+0000","last_change":"2024-03-12T05:53:38.214376+0000","last_active":"2024-03-12T05:53:39.465593+0000","last_peered":"2024-03-12T05:53:39.465593+0000","last_clean":"2024-03-12T05:53:39.465593+0000","last_became_active":"2024-03-12T05:53:38.214011+0000","last_became_peered":"2024-03-12T05:53:38.214011+0000","last_unstale":"2024-03-12T05:53:39.465593+0000","last_undegraded":"2024-03-12T05:53:39.465593+0000","last_fullsized":"2024-03-12T05:53:39.465593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:32.605216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.318744+0000","last_change":"2024-03-12T05:53:38.205337+0000","last_active":"2024-03-12T05:53:39.318744+0000","last_peered":"2024-03-12T05:53:39.318744+0000","last_clean":"2024-03-12T05:53:39.318744+0000","last_became_active":"2024-03-12T05:53:38.205005+0000","last_became_peered":"2024-03-12T05:53:38.205005+0000","last_unstale":"2024-03-12T05:53:39.318744+0000","last_undegraded":"2024-03-12T05:53:39.318744+0000","last_fullsized":"2024-03-12T05:53:39.318744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:44:16.296370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.334470+0000","last_change":"2024-03-12T05:53:38.214694+0000","last_active":"2024-03-12T05:53:39.334470+0000","last_peered":"2024-03-12T05:53:39.334470+0000","last_clean":"2024-03-12T05:53:39.334470+0000","last_became_active":"2024-03-12T05:53:38.214285+0000","last_became_peered":"2024-03-12T05:53:38.214285+0000","last_unstale":"2024-03-12T05:53:39.334470+0000","last_undegraded":"2024-03-12T05:53:39.334470+0000","last_fullsized":"2024-03-12T05:53:39.334470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:28:32.253332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.334386+0000","last_change":"2024-03-12T05:53:38.213069+0000","last_active":"2024-03-12T05:53:39.334386+0000","last_peered":"2024-03-12T05:53:39.334386+0000","last_clean":"2024-03-12T05:53:39.334386+0000","last_became_active":"2024-03-12T05:53:38.212694+0000","last_became_peered":"2024-03-12T05:53:38.212694+0000","last_unstale":"2024-03-12T05:53:39.334386+0000","last_undegraded":"2024-03-12T05:53:39.334386+0000","last_fullsized":"2024-03-12T05:53:39.334386+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:26:29.350292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.214060+0000","last_change":"2024-03-12T05:53:38.212084+0000","last_active":"2024-03-12T05:53:38.214060+0000","last_peered":"2024-03-12T05:53:38.214060+0000","last_clean":"2024-03-12T05:53:38.214060+0000","last_became_active":"2024-03-12T05:53:38.211750+0000","last_became_peered":"2024-03-12T05:53:38.211750+0000","last_unstale":"2024-03-12T05:53:38.214060+0000","last_undegraded":"2024-03-12T05:53:38.214060+0000","last_fullsized":"2024-03-12T05:53:38.214060+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:39:39.892739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.912470+0000","last_change":"2024-03-12T05:53:38.212071+0000","last_active":"2024-03-12T05:53:39.912470+0000","last_peered":"2024-03-12T05:53:39.912470+0000","last_clean":"2024-03-12T05:53:39.912470+0000","last_became_active":"2024-03-12T05:53:38.211722+0000","last_became_peered":"2024-03-12T05:53:38.211722+0000","last_unstale":"2024-03-12T05:53:39.912470+0000","last_undegraded":"2024-03-12T05:53:39.912470+0000","last_fullsized":"2024-03-12T05:53:39.912470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:19:46.581837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.318876+0000","last_change":"2024-03-12T05:53:38.206819+0000","last_active":"2024-03-12T05:53:39.318876+0000","last_peered":"2024-03-12T05:53:39.318876+0000","last_clean":"2024-03-12T05:53:39.318876+0000","last_became_active":"2024-03-12T05:53:38.206496+0000","last_became_peered":"2024-03-12T05:53:38.206496+0000","last_unstale":"2024-03-12T05:53:39.318876+0000","last_undegraded":"2024-03-12T05:53:39.318876+0000","last_fullsized":"2024-03-12T05:53:39.318876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:31:56.920976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.334811+0000","last_change":"2024-03-12T05:53:38.208018+0000","last_active":"2024-03-12T05:53:39.334811+0000","last_peered":"2024-03-12T05:53:39.334811+0000","last_clean":"2024-03-12T05:53:39.334811+0000","last_became_active":"2024-03-12T05:53:38.207682+0000","last_became_peered":"2024-03-12T05:53:38.207682+0000","last_unstale":"2024-03-12T05:53:39.334811+0000","last_undegraded":"2024-03-12T05:53:39.334811+0000","last_fullsized":"2024-03-12T05:53:39.334811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:56:10.903917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197761+0000","last_change":"2024-03-12T05:53:38.217540+0000","last_active":"2024-03-12T05:53:39.197761+0000","last_peered":"2024-03-12T05:53:39.197761+0000","last_clean":"2024-03-12T05:53:39.197761+0000","last_became_active":"2024-03-12T05:53:38.217144+0000","last_became_peered":"2024-03-12T05:53:38.217144+0000","last_unstale":"2024-03-12T05:53:39.197761+0000","last_undegraded":"2024-03-12T05:53:39.197761+0000","last_fullsized":"2024-03-12T05:53:39.197761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:46:18.905909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197969+0000","last_change":"2024-03-12T05:53:38.215006+0000","last_active":"2024-03-12T05:53:39.197969+0000","last_peered":"2024-03-12T05:53:39.197969+0000","last_clean":"2024-03-12T05:53:39.197969+0000","last_became_active":"2024-03-12T05:53:38.213756+0000","last_became_peered":"2024-03-12T05:53:38.213756+0000","last_unstale":"2024-03-12T05:53:39.197969+0000","last_undegraded":"2024-03-12T05:53:39.197969+0000","last_fullsized":"2024-03-12T05:53:39.197969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:14:57.642381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197539+0000","last_change":"2024-03-12T05:53:38.216096+0000","last_active":"2024-03-12T05:53:39.197539+0000","last_peered":"2024-03-12T05:53:39.197539+0000","last_clean":"2024-03-12T05:53:39.197539+0000","last_became_active":"2024-03-12T05:53:38.215153+0000","last_became_peered":"2024-03-12T05:53:38.215153+0000","last_unstale":"2024-03-12T05:53:39.197539+0000","last_undegraded":"2024-03-12T05:53:39.197539+0000","last_fullsized":"2024-03-12T05:53:39.197539+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:41:03.893641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.384862+0000","last_change":"2024-03-12T05:53:38.219165+0000","last_active":"2024-03-12T05:53:40.384862+0000","last_peered":"2024-03-12T05:53:40.384862+0000","last_clean":"2024-03-12T05:53:40.384862+0000","last_became_active":"2024-03-12T05:53:38.218579+0000","last_became_peered":"2024-03-12T05:53:38.218579+0000","last_unstale":"2024-03-12T05:53:40.384862+0000","last_undegraded":"2024-03-12T05:53:40.384862+0000","last_fullsized":"2024-03-12T05:53:40.384862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:06:37.531293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.912329+0000","last_change":"2024-03-12T05:53:38.212685+0000","last_active":"2024-03-12T05:53:39.912329+0000","last_peered":"2024-03-12T05:53:39.912329+0000","last_clean":"2024-03-12T05:53:39.912329+0000","last_became_active":"2024-03-12T05:53:38.211106+0000","last_became_peered":"2024-03-12T05:53:38.211106+0000","last_unstale":"2024-03-12T05:53:39.912329+0000","last_undegraded":"2024-03-12T05:53:39.912329+0000","last_fullsized":"2024-03-12T05:53:39.912329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:26:00.339270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.318641+0000","last_change":"2024-03-12T05:53:38.218402+0000","last_active":"2024-03-12T05:53:39.318641+0000","last_peered":"2024-03-12T05:53:39.318641+0000","last_clean":"2024-03-12T05:53:39.318641+0000","last_became_active":"2024-03-12T05:53:38.217699+0000","last_became_peered":"2024-03-12T05:53:38.217699+0000","last_unstale":"2024-03-12T05:53:39.318641+0000","last_undegraded":"2024-03-12T05:53:39.318641+0000","last_fullsized":"2024-03-12T05:53:39.318641+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:23:03.221710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":39977,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":23,"num_read_kb":38,"num_write":128,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":92906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":929792,"data_stored":444507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612627,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32724,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712428,"statfs":{"total":95995035648,"available":95961526272,"internally_reserved":0,"allocated":1585152,"data_stored":1056883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776152,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":946176,"data_stored":444743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037789,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":444663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332129,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32676,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712476,"statfs":{"total":95995035648,"available":95961575424,"internally_reserved":0,"allocated":1536000,"data_stored":1036085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":444132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":464820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":917504,"data_stored":435445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":453878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":444209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36244,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708908,"statfs":{"total":95995035648,"available":95957921792,"internally_reserved":0,"allocated":929792,"data_stored":443947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28520,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716632,"statfs":{"total":95995035648,"available":95965831168,"internally_reserved":0,"allocated":1531904,"data_stored":1034475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":38193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":9953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":39771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T05:53:45.123 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-12T05:53:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.259+0000 7f4b279c7640 1 -- 172.21.15.76:0/842460592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b20100120 msgr2=0x7f4b20100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.259+0000 7f4b279c7640 1 --2- 172.21.15.76:0/842460592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b20100120 0x7f4b20100540 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c009900 tx=0x7f4b1c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:53:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.259+0000 7f4b279c7640 1 -- 172.21.15.76:0/842460592 shutdown_connections 2024-03-12T05:53:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.259+0000 7f4b279c7640 1 --2- 172.21.15.76:0/842460592 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b20101cd0 0x7f4b20108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.259+0000 7f4b279c7640 1 --2- 172.21.15.76:0/842460592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b20101310 0x7f4b20101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.259+0000 7f4b279c7640 1 --2- 172.21.15.76:0/842460592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b20100120 0x7f4b20100540 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.259+0000 7f4b279c7640 1 -- 172.21.15.76:0/842460592 >> 172.21.15.76:0/842460592 conn(0x7f4b200fb6b0 msgr2=0x7f4b200fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.259+0000 7f4b279c7640 1 -- 172.21.15.76:0/842460592 shutdown_connections 2024-03-12T05:53:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.259+0000 7f4b279c7640 1 -- 172.21.15.76:0/842460592 wait complete. 2024-03-12T05:53:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b279c7640 1 Processor -- start 2024-03-12T05:53:45.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b279c7640 1 -- start start 2024-03-12T05:53:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b279c7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b20100120 0x7f4b2006a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b279c7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b20101310 0x7f4b2019bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b279c7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b20101cd0 0x7f4b2019e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b279c7640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2019ed80 con 0x7f4b20101310 2024-03-12T05:53:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b279c7640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2019eef0 con 0x7f4b20100120 2024-03-12T05:53:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b279c7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2019f0b0 con 0x7f4b20101cd0 2024-03-12T05:53:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b24f62640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b20101310 0x7f4b2019bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b24f62640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b20101310 0x7f4b2019bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53606/0 (socket says 172.21.15.76:53606) 2024-03-12T05:53:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b24f62640 1 -- 172.21.15.76:0/498021310 learned_addr learned my addr 172.21.15.76:0/498021310 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b24f62640 1 -- 172.21.15.76:0/498021310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b20101cd0 msgr2=0x7f4b2019e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b24f62640 1 --2- 172.21.15.76:0/498021310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b20101cd0 0x7f4b2019e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b24f62640 1 -- 172.21.15.76:0/498021310 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b20100120 msgr2=0x7f4b2006a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:45.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b24f62640 1 --2- 172.21.15.76:0/498021310 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b20100120 0x7f4b2006a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.260+0000 7f4b24f62640 1 -- 172.21.15.76:0/498021310 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1c009560 con 0x7f4b20101310 2024-03-12T05:53:45.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.261+0000 7f4b24f62640 1 --2- 172.21.15.76:0/498021310 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b20101310 0x7f4b2019bfb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4b1000d950 tx=0x7f4b1000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:45.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.261+0000 7f4b167fc640 1 -- 172.21.15.76:0/498021310 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b10009c30 con 0x7f4b20101310 2024-03-12T05:53:45.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.261+0000 7f4b167fc640 1 -- 172.21.15.76:0/498021310 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b10009d90 con 0x7f4b20101310 2024-03-12T05:53:45.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.261+0000 7f4b279c7640 1 -- 172.21.15.76:0/498021310 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b2019f390 con 0x7f4b20101310 2024-03-12T05:53:45.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.261+0000 7f4b167fc640 1 -- 172.21.15.76:0/498021310 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1000f7e0 con 0x7f4b20101310 2024-03-12T05:53:45.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.261+0000 7f4b279c7640 1 -- 172.21.15.76:0/498021310 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b2019f8a0 con 0x7f4b20101310 2024-03-12T05:53:45.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.262+0000 7f4b279c7640 1 -- 172.21.15.76:0/498021310 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b20107510 con 0x7f4b20101310 2024-03-12T05:53:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.263+0000 7f4b167fc640 1 -- 172.21.15.76:0/498021310 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f4b10010ad0 con 0x7f4b20101310 2024-03-12T05:53:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.263+0000 7f4b167fc640 1 --2- 172.21.15.76:0/498021310 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f4b0c0a1ac0 0x7f4b0c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.263+0000 7f4b25763640 1 --2- 172.21.15.76:0/498021310 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f4b0c0a1ac0 0x7f4b0c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.263+0000 7f4b167fc640 1 -- 172.21.15.76:0/498021310 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4b100cd120 con 0x7f4b20101310 2024-03-12T05:53:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.264+0000 7f4b25763640 1 --2- 172.21.15.76:0/498021310 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f4b0c0a1ac0 0x7f4b0c0a3f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c02c6a0 tx=0x7f4b1c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:45.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.267+0000 7f4b167fc640 1 -- 172.21.15.76:0/498021310 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4b10096380 con 0x7f4b20101310 2024-03-12T05:53:45.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.440+0000 7f4b279c7640 1 -- 172.21.15.76:0/498021310 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4b2002cf10 con 0x7f4b0c0a1ac0 2024-03-12T05:53:45.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.447+0000 7f4b167fc640 1 -- 172.21.15.76:0/498021310 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258753 (secure 0 0 0) 0x7f4b2002cf10 con 0x7f4b0c0a1ac0 2024-03-12T05:53:45.452 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:45.453 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-03-12T05:53:45.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.450+0000 7f4b279c7640 1 -- 172.21.15.76:0/498021310 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f4b0c0a1ac0 msgr2=0x7f4b0c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:45.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.450+0000 7f4b279c7640 1 --2- 172.21.15.76:0/498021310 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f4b0c0a1ac0 0x7f4b0c0a3f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c02c6a0 tx=0x7f4b1c000bc0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.450+0000 7f4b279c7640 1 -- 172.21.15.76:0/498021310 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b20101310 msgr2=0x7f4b2019bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:45.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.450+0000 7f4b279c7640 1 --2- 172.21.15.76:0/498021310 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b20101310 0x7f4b2019bfb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4b1000d950 tx=0x7f4b1000de10 comp rx=0 tx=0).stop 2024-03-12T05:53:45.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.450+0000 7f4b279c7640 1 -- 172.21.15.76:0/498021310 shutdown_connections 2024-03-12T05:53:45.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.450+0000 7f4b279c7640 1 --2- 172.21.15.76:0/498021310 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f4b0c0a1ac0 0x7f4b0c0a3f80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.450+0000 7f4b279c7640 1 --2- 172.21.15.76:0/498021310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b20101cd0 0x7f4b2019e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.450+0000 7f4b279c7640 1 --2- 172.21.15.76:0/498021310 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b20101310 0x7f4b2019bfb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.450+0000 7f4b279c7640 1 --2- 172.21.15.76:0/498021310 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4b20100120 0x7f4b2006a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:45.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.450+0000 7f4b279c7640 1 -- 172.21.15.76:0/498021310 >> 172.21.15.76:0/498021310 conn(0x7f4b200fb6b0 msgr2=0x7f4b201046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:45.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.450+0000 7f4b279c7640 1 -- 172.21.15.76:0/498021310 shutdown_connections 2024-03-12T05:53:45.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:45.450+0000 7f4b279c7640 1 -- 172.21.15.76:0/498021310 wait complete. 2024-03-12T05:53:45.464 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-03-12T05:53:44.863677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630462,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":159,"num_read_kb":157,"num_write":310,"num_write_kb":3011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":245,"ondisk_log_size":245,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":252,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":374468,"kb_used_data":12812,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124567356,"statfs":{"total":1151940427776,"available":1151556972544,"internally_reserved":0,"allocated":13119488,"data_stored":7147787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40094,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":23,"num_read_kb":38,"num_write":129,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.675848"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.319004+0000","last_change":"2024-03-12T05:53:38.207712+0000","last_active":"2024-03-12T05:53:39.319004+0000","last_peered":"2024-03-12T05:53:39.319004+0000","last_clean":"2024-03-12T05:53:39.319004+0000","last_became_active":"2024-03-12T05:53:38.207362+0000","last_became_peered":"2024-03-12T05:53:38.207362+0000","last_unstale":"2024-03-12T05:53:39.319004+0000","last_undegraded":"2024-03-12T05:53:39.319004+0000","last_fullsized":"2024-03-12T05:53:39.319004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:19.518959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.209313+0000","last_change":"2024-03-12T05:53:38.206877+0000","last_active":"2024-03-12T05:53:38.209313+0000","last_peered":"2024-03-12T05:53:38.209313+0000","last_clean":"2024-03-12T05:53:38.209313+0000","last_became_active":"2024-03-12T05:53:38.206468+0000","last_became_peered":"2024-03-12T05:53:38.206468+0000","last_unstale":"2024-03-12T05:53:38.209313+0000","last_undegraded":"2024-03-12T05:53:38.209313+0000","last_fullsized":"2024-03-12T05:53:38.209313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:18:43.309306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.215864+0000","last_change":"2024-03-12T05:53:38.214979+0000","last_active":"2024-03-12T05:53:38.215864+0000","last_peered":"2024-03-12T05:53:38.215864+0000","last_clean":"2024-03-12T05:53:38.215864+0000","last_became_active":"2024-03-12T05:53:38.214580+0000","last_became_peered":"2024-03-12T05:53:38.214580+0000","last_unstale":"2024-03-12T05:53:38.215864+0000","last_undegraded":"2024-03-12T05:53:38.215864+0000","last_fullsized":"2024-03-12T05:53:38.215864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:14:04.635961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.198731+0000","last_change":"2024-03-12T05:53:38.218645+0000","last_active":"2024-03-12T05:53:39.198731+0000","last_peered":"2024-03-12T05:53:39.198731+0000","last_clean":"2024-03-12T05:53:39.198731+0000","last_became_active":"2024-03-12T05:53:38.218254+0000","last_became_peered":"2024-03-12T05:53:38.218254+0000","last_unstale":"2024-03-12T05:53:39.198731+0000","last_undegraded":"2024-03-12T05:53:39.198731+0000","last_fullsized":"2024-03-12T05:53:39.198731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:46:09.544989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.183433+0000","last_change":"2024-03-12T05:53:38.214588+0000","last_active":"2024-03-12T05:53:40.183433+0000","last_peered":"2024-03-12T05:53:40.183433+0000","last_clean":"2024-03-12T05:53:40.183433+0000","last_became_active":"2024-03-12T05:53:38.214192+0000","last_became_peered":"2024-03-12T05:53:38.214192+0000","last_unstale":"2024-03-12T05:53:40.183433+0000","last_undegraded":"2024-03-12T05:53:40.183433+0000","last_fullsized":"2024-03-12T05:53:40.183433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:54:04.880901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.385422+0000","last_change":"2024-03-12T05:53:38.214315+0000","last_active":"2024-03-12T05:53:40.385422+0000","last_peered":"2024-03-12T05:53:40.385422+0000","last_clean":"2024-03-12T05:53:40.385422+0000","last_became_active":"2024-03-12T05:53:38.213881+0000","last_became_peered":"2024-03-12T05:53:38.213881+0000","last_unstale":"2024-03-12T05:53:40.385422+0000","last_undegraded":"2024-03-12T05:53:40.385422+0000","last_fullsized":"2024-03-12T05:53:40.385422+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:11:32.834418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465273+0000","last_change":"2024-03-12T05:53:38.214915+0000","last_active":"2024-03-12T05:53:39.465273+0000","last_peered":"2024-03-12T05:53:39.465273+0000","last_clean":"2024-03-12T05:53:39.465273+0000","last_became_active":"2024-03-12T05:53:38.214592+0000","last_became_peered":"2024-03-12T05:53:38.214592+0000","last_unstale":"2024-03-12T05:53:39.465273+0000","last_undegraded":"2024-03-12T05:53:39.465273+0000","last_fullsized":"2024-03-12T05:53:39.465273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:11.548585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197339+0000","last_change":"2024-03-12T05:53:38.219201+0000","last_active":"2024-03-12T05:53:39.197339+0000","last_peered":"2024-03-12T05:53:39.197339+0000","last_clean":"2024-03-12T05:53:39.197339+0000","last_became_active":"2024-03-12T05:53:38.218833+0000","last_became_peered":"2024-03-12T05:53:38.218833+0000","last_unstale":"2024-03-12T05:53:39.197339+0000","last_undegraded":"2024-03-12T05:53:39.197339+0000","last_fullsized":"2024-03-12T05:53:39.197339+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:13:36.177325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.385062+0000","last_change":"2024-03-12T05:53:38.205126+0000","last_active":"2024-03-12T05:53:40.385062+0000","last_peered":"2024-03-12T05:53:40.385062+0000","last_clean":"2024-03-12T05:53:40.385062+0000","last_became_active":"2024-03-12T05:53:38.204478+0000","last_became_peered":"2024-03-12T05:53:38.204478+0000","last_unstale":"2024-03-12T05:53:40.385062+0000","last_undegraded":"2024-03-12T05:53:40.385062+0000","last_fullsized":"2024-03-12T05:53:40.385062+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:25:56.475134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.387119+0000","last_change":"2024-03-12T05:53:38.215186+0000","last_active":"2024-03-12T05:53:40.387119+0000","last_peered":"2024-03-12T05:53:40.387119+0000","last_clean":"2024-03-12T05:53:40.387119+0000","last_became_active":"2024-03-12T05:53:38.214832+0000","last_became_peered":"2024-03-12T05:53:38.214832+0000","last_unstale":"2024-03-12T05:53:40.387119+0000","last_undegraded":"2024-03-12T05:53:40.387119+0000","last_fullsized":"2024-03-12T05:53:40.387119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:56:26.621990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465591+0000","last_change":"2024-03-12T05:53:38.222347+0000","last_active":"2024-03-12T05:53:39.465591+0000","last_peered":"2024-03-12T05:53:39.465591+0000","last_clean":"2024-03-12T05:53:39.465591+0000","last_became_active":"2024-03-12T05:53:38.221637+0000","last_became_peered":"2024-03-12T05:53:38.221637+0000","last_unstale":"2024-03-12T05:53:39.465591+0000","last_undegraded":"2024-03-12T05:53:39.465591+0000","last_fullsized":"2024-03-12T05:53:39.465591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:16:26.652673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.198722+0000","last_change":"2024-03-12T05:53:38.218687+0000","last_active":"2024-03-12T05:53:39.198722+0000","last_peered":"2024-03-12T05:53:39.198722+0000","last_clean":"2024-03-12T05:53:39.198722+0000","last_became_active":"2024-03-12T05:53:38.218393+0000","last_became_peered":"2024-03-12T05:53:38.218393+0000","last_unstale":"2024-03-12T05:53:39.198722+0000","last_undegraded":"2024-03-12T05:53:39.198722+0000","last_fullsized":"2024-03-12T05:53:39.198722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:41:39.918992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.912472+0000","last_change":"2024-03-12T05:53:38.207432+0000","last_active":"2024-03-12T05:53:39.912472+0000","last_peered":"2024-03-12T05:53:39.912472+0000","last_clean":"2024-03-12T05:53:39.912472+0000","last_became_active":"2024-03-12T05:53:38.207099+0000","last_became_peered":"2024-03-12T05:53:38.207099+0000","last_unstale":"2024-03-12T05:53:39.912472+0000","last_undegraded":"2024-03-12T05:53:39.912472+0000","last_fullsized":"2024-03-12T05:53:39.912472+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:20:08.504723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.209282+0000","last_change":"2024-03-12T05:53:38.207086+0000","last_active":"2024-03-12T05:53:38.209282+0000","last_peered":"2024-03-12T05:53:38.209282+0000","last_clean":"2024-03-12T05:53:38.209282+0000","last_became_active":"2024-03-12T05:53:38.206714+0000","last_became_peered":"2024-03-12T05:53:38.206714+0000","last_unstale":"2024-03-12T05:53:38.209282+0000","last_undegraded":"2024-03-12T05:53:38.209282+0000","last_fullsized":"2024-03-12T05:53:38.209282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:13:45.966851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.218308+0000","last_change":"2024-03-12T05:53:38.217179+0000","last_active":"2024-03-12T05:53:38.218308+0000","last_peered":"2024-03-12T05:53:38.218308+0000","last_clean":"2024-03-12T05:53:38.218308+0000","last_became_active":"2024-03-12T05:53:38.216820+0000","last_became_peered":"2024-03-12T05:53:38.216820+0000","last_unstale":"2024-03-12T05:53:38.218308+0000","last_undegraded":"2024-03-12T05:53:38.218308+0000","last_fullsized":"2024-03-12T05:53:38.218308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:40:00.006259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.384995+0000","last_change":"2024-03-12T05:53:38.220224+0000","last_active":"2024-03-12T05:53:40.384995+0000","last_peered":"2024-03-12T05:53:40.384995+0000","last_clean":"2024-03-12T05:53:40.384995+0000","last_became_active":"2024-03-12T05:53:38.219878+0000","last_became_peered":"2024-03-12T05:53:38.219878+0000","last_unstale":"2024-03-12T05:53:40.384995+0000","last_undegraded":"2024-03-12T05:53:40.384995+0000","last_fullsized":"2024-03-12T05:53:40.384995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:53:16.545807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.319135+0000","last_change":"2024-03-12T05:53:38.208524+0000","last_active":"2024-03-12T05:53:39.319135+0000","last_peered":"2024-03-12T05:53:39.319135+0000","last_clean":"2024-03-12T05:53:39.319135+0000","last_became_active":"2024-03-12T05:53:38.208209+0000","last_became_peered":"2024-03-12T05:53:38.208209+0000","last_unstale":"2024-03-12T05:53:39.319135+0000","last_undegraded":"2024-03-12T05:53:39.319135+0000","last_fullsized":"2024-03-12T05:53:39.319135+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:28:56.334066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.367484+0000","last_change":"2024-03-12T05:53:38.208035+0000","last_active":"2024-03-12T05:53:40.367484+0000","last_peered":"2024-03-12T05:53:40.367484+0000","last_clean":"2024-03-12T05:53:40.367484+0000","last_became_active":"2024-03-12T05:53:38.207679+0000","last_became_peered":"2024-03-12T05:53:38.207679+0000","last_unstale":"2024-03-12T05:53:40.367484+0000","last_undegraded":"2024-03-12T05:53:40.367484+0000","last_fullsized":"2024-03-12T05:53:40.367484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:40:20.620552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.334689+0000","last_change":"2024-03-12T05:53:38.207077+0000","last_active":"2024-03-12T05:53:39.334689+0000","last_peered":"2024-03-12T05:53:39.334689+0000","last_clean":"2024-03-12T05:53:39.334689+0000","last_became_active":"2024-03-12T05:53:38.206720+0000","last_became_peered":"2024-03-12T05:53:38.206720+0000","last_unstale":"2024-03-12T05:53:39.334689+0000","last_undegraded":"2024-03-12T05:53:39.334689+0000","last_fullsized":"2024-03-12T05:53:39.334689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:45:54.628394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.377150+0000","last_change":"2024-03-12T05:53:37.201637+0000","last_active":"2024-03-12T05:53:41.377150+0000","last_peered":"2024-03-12T05:53:41.377150+0000","last_clean":"2024-03-12T05:53:41.377150+0000","last_became_active":"2024-03-12T05:53:37.201289+0000","last_became_peered":"2024-03-12T05:53:37.201289+0000","last_unstale":"2024-03-12T05:53:41.377150+0000","last_undegraded":"2024-03-12T05:53:41.377150+0000","last_fullsized":"2024-03-12T05:53:41.377150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:59:43.829570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197427+0000","last_change":"2024-03-12T05:53:38.218052+0000","last_active":"2024-03-12T05:53:39.197427+0000","last_peered":"2024-03-12T05:53:39.197427+0000","last_clean":"2024-03-12T05:53:39.197427+0000","last_became_active":"2024-03-12T05:53:38.217738+0000","last_became_peered":"2024-03-12T05:53:38.217738+0000","last_unstale":"2024-03-12T05:53:39.197427+0000","last_undegraded":"2024-03-12T05:53:39.197427+0000","last_fullsized":"2024-03-12T05:53:39.197427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:59:40.599573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197645+0000","last_change":"2024-03-12T05:53:37.210321+0000","last_active":"2024-03-12T05:53:39.197645+0000","last_peered":"2024-03-12T05:53:39.197645+0000","last_clean":"2024-03-12T05:53:39.197645+0000","last_became_active":"2024-03-12T05:53:37.210018+0000","last_became_peered":"2024-03-12T05:53:37.210018+0000","last_unstale":"2024-03-12T05:53:39.197645+0000","last_undegraded":"2024-03-12T05:53:39.197645+0000","last_fullsized":"2024-03-12T05:53:39.197645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:13:45.433332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.209340+0000","last_change":"2024-03-12T05:53:38.207801+0000","last_active":"2024-03-12T05:53:38.209340+0000","last_peered":"2024-03-12T05:53:38.209340+0000","last_clean":"2024-03-12T05:53:38.209340+0000","last_became_active":"2024-03-12T05:53:38.207467+0000","last_became_peered":"2024-03-12T05:53:38.207467+0000","last_unstale":"2024-03-12T05:53:38.209340+0000","last_undegraded":"2024-03-12T05:53:38.209340+0000","last_fullsized":"2024-03-12T05:53:38.209340+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:22:30.434799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.376310+0000","last_change":"2024-03-12T05:53:37.208288+0000","last_active":"2024-03-12T05:53:41.376310+0000","last_peered":"2024-03-12T05:53:41.376310+0000","last_clean":"2024-03-12T05:53:41.376310+0000","last_became_active":"2024-03-12T05:53:37.207929+0000","last_became_peered":"2024-03-12T05:53:37.207929+0000","last_unstale":"2024-03-12T05:53:41.376310+0000","last_undegraded":"2024-03-12T05:53:41.376310+0000","last_fullsized":"2024-03-12T05:53:41.376310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:10:52.719116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.385095+0000","last_change":"2024-03-12T05:53:38.207270+0000","last_active":"2024-03-12T05:53:40.385095+0000","last_peered":"2024-03-12T05:53:40.385095+0000","last_clean":"2024-03-12T05:53:40.385095+0000","last_became_active":"2024-03-12T05:53:38.205782+0000","last_became_peered":"2024-03-12T05:53:38.205782+0000","last_unstale":"2024-03-12T05:53:40.385095+0000","last_undegraded":"2024-03-12T05:53:40.385095+0000","last_fullsized":"2024-03-12T05:53:40.385095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:02:24.875444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.183675+0000","last_change":"2024-03-12T05:53:37.203899+0000","last_active":"2024-03-12T05:53:40.183675+0000","last_peered":"2024-03-12T05:53:40.183675+0000","last_clean":"2024-03-12T05:53:40.183675+0000","last_became_active":"2024-03-12T05:53:37.203450+0000","last_became_peered":"2024-03-12T05:53:37.203450+0000","last_unstale":"2024-03-12T05:53:40.183675+0000","last_undegraded":"2024-03-12T05:53:40.183675+0000","last_fullsized":"2024-03-12T05:53:40.183675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:25.773826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197718+0000","last_change":"2024-03-12T05:53:38.211955+0000","last_active":"2024-03-12T05:53:39.197718+0000","last_peered":"2024-03-12T05:53:39.197718+0000","last_clean":"2024-03-12T05:53:39.197718+0000","last_became_active":"2024-03-12T05:53:38.211547+0000","last_became_peered":"2024-03-12T05:53:38.211547+0000","last_unstale":"2024-03-12T05:53:39.197718+0000","last_undegraded":"2024-03-12T05:53:39.197718+0000","last_fullsized":"2024-03-12T05:53:39.197718+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:53:56.984213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197670+0000","last_change":"2024-03-12T05:53:37.205401+0000","last_active":"2024-03-12T05:53:39.197670+0000","last_peered":"2024-03-12T05:53:39.197670+0000","last_clean":"2024-03-12T05:53:39.197670+0000","last_became_active":"2024-03-12T05:53:37.204993+0000","last_became_peered":"2024-03-12T05:53:37.204993+0000","last_unstale":"2024-03-12T05:53:39.197670+0000","last_undegraded":"2024-03-12T05:53:39.197670+0000","last_fullsized":"2024-03-12T05:53:39.197670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:56:06.177387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.183472+0000","last_change":"2024-03-12T05:53:38.210562+0000","last_active":"2024-03-12T05:53:40.183472+0000","last_peered":"2024-03-12T05:53:40.183472+0000","last_clean":"2024-03-12T05:53:40.183472+0000","last_became_active":"2024-03-12T05:53:38.210235+0000","last_became_peered":"2024-03-12T05:53:38.210235+0000","last_unstale":"2024-03-12T05:53:40.183472+0000","last_undegraded":"2024-03-12T05:53:40.183472+0000","last_fullsized":"2024-03-12T05:53:40.183472+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:35.445812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.374540+0000","last_change":"2024-03-12T05:53:37.204986+0000","last_active":"2024-03-12T05:53:41.374540+0000","last_peered":"2024-03-12T05:53:41.374540+0000","last_clean":"2024-03-12T05:53:41.374540+0000","last_became_active":"2024-03-12T05:53:37.204580+0000","last_became_peered":"2024-03-12T05:53:37.204580+0000","last_unstale":"2024-03-12T05:53:41.374540+0000","last_undegraded":"2024-03-12T05:53:41.374540+0000","last_fullsized":"2024-03-12T05:53:41.374540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:37:34.584395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197554+0000","last_change":"2024-03-12T05:53:38.214948+0000","last_active":"2024-03-12T05:53:39.197554+0000","last_peered":"2024-03-12T05:53:39.197554+0000","last_clean":"2024-03-12T05:53:39.197554+0000","last_became_active":"2024-03-12T05:53:38.213748+0000","last_became_peered":"2024-03-12T05:53:38.213748+0000","last_unstale":"2024-03-12T05:53:39.197554+0000","last_undegraded":"2024-03-12T05:53:39.197554+0000","last_fullsized":"2024-03-12T05:53:39.197554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:12:45.994129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.375650+0000","last_change":"2024-03-12T05:53:37.198167+0000","last_active":"2024-03-12T05:53:41.375650+0000","last_peered":"2024-03-12T05:53:41.375650+0000","last_clean":"2024-03-12T05:53:41.375650+0000","last_became_active":"2024-03-12T05:53:37.197808+0000","last_became_peered":"2024-03-12T05:53:37.197808+0000","last_unstale":"2024-03-12T05:53:41.375650+0000","last_undegraded":"2024-03-12T05:53:41.375650+0000","last_fullsized":"2024-03-12T05:53:41.375650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:40:19.498546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197306+0000","last_change":"2024-03-12T05:53:38.205464+0000","last_active":"2024-03-12T05:53:39.197306+0000","last_peered":"2024-03-12T05:53:39.197306+0000","last_clean":"2024-03-12T05:53:39.197306+0000","last_became_active":"2024-03-12T05:53:38.205129+0000","last_became_peered":"2024-03-12T05:53:38.205129+0000","last_unstale":"2024-03-12T05:53:39.197306+0000","last_undegraded":"2024-03-12T05:53:39.197306+0000","last_fullsized":"2024-03-12T05:53:39.197306+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:38:55.460862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197255+0000","last_change":"2024-03-12T05:53:37.201572+0000","last_active":"2024-03-12T05:53:39.197255+0000","last_peered":"2024-03-12T05:53:39.197255+0000","last_clean":"2024-03-12T05:53:39.197255+0000","last_became_active":"2024-03-12T05:53:37.201230+0000","last_became_peered":"2024-03-12T05:53:37.201230+0000","last_unstale":"2024-03-12T05:53:39.197255+0000","last_undegraded":"2024-03-12T05:53:39.197255+0000","last_fullsized":"2024-03-12T05:53:39.197255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:16:13.715459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.367343+0000","last_change":"2024-03-12T05:53:38.210696+0000","last_active":"2024-03-12T05:53:40.367343+0000","last_peered":"2024-03-12T05:53:40.367343+0000","last_clean":"2024-03-12T05:53:40.367343+0000","last_became_active":"2024-03-12T05:53:38.210325+0000","last_became_peered":"2024-03-12T05:53:38.210325+0000","last_unstale":"2024-03-12T05:53:40.367343+0000","last_undegraded":"2024-03-12T05:53:40.367343+0000","last_fullsized":"2024-03-12T05:53:40.367343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:25:36.397314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:42.368628+0000","last_change":"2024-03-12T05:53:37.201712+0000","last_active":"2024-03-12T05:53:42.368628+0000","last_peered":"2024-03-12T05:53:42.368628+0000","last_clean":"2024-03-12T05:53:42.368628+0000","last_became_active":"2024-03-12T05:53:37.200794+0000","last_became_peered":"2024-03-12T05:53:37.200794+0000","last_unstale":"2024-03-12T05:53:42.368628+0000","last_undegraded":"2024-03-12T05:53:42.368628+0000","last_fullsized":"2024-03-12T05:53:42.368628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:43:29.673717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.384766+0000","last_change":"2024-03-12T05:53:38.206647+0000","last_active":"2024-03-12T05:53:40.384766+0000","last_peered":"2024-03-12T05:53:40.384766+0000","last_clean":"2024-03-12T05:53:40.384766+0000","last_became_active":"2024-03-12T05:53:38.206254+0000","last_became_peered":"2024-03-12T05:53:38.206254+0000","last_unstale":"2024-03-12T05:53:40.384766+0000","last_undegraded":"2024-03-12T05:53:40.384766+0000","last_fullsized":"2024-03-12T05:53:40.384766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:47:20.455360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.387336+0000","last_change":"2024-03-12T05:53:37.208692+0000","last_active":"2024-03-12T05:53:40.387336+0000","last_peered":"2024-03-12T05:53:40.387336+0000","last_clean":"2024-03-12T05:53:40.387336+0000","last_became_active":"2024-03-12T05:53:37.208270+0000","last_became_peered":"2024-03-12T05:53:37.208270+0000","last_unstale":"2024-03-12T05:53:40.387336+0000","last_undegraded":"2024-03-12T05:53:40.387336+0000","last_fullsized":"2024-03-12T05:53:40.387336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:51:38.217999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.219740+0000","last_change":"2024-03-12T05:53:38.218202+0000","last_active":"2024-03-12T05:53:38.219740+0000","last_peered":"2024-03-12T05:53:38.219740+0000","last_clean":"2024-03-12T05:53:38.219740+0000","last_became_active":"2024-03-12T05:53:38.217798+0000","last_became_peered":"2024-03-12T05:53:38.217798+0000","last_unstale":"2024-03-12T05:53:38.219740+0000","last_undegraded":"2024-03-12T05:53:38.219740+0000","last_fullsized":"2024-03-12T05:53:38.219740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:22:16.690610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197843+0000","last_change":"2024-03-12T05:53:37.203252+0000","last_active":"2024-03-12T05:53:39.197843+0000","last_peered":"2024-03-12T05:53:39.197843+0000","last_clean":"2024-03-12T05:53:39.197843+0000","last_became_active":"2024-03-12T05:53:37.202916+0000","last_became_peered":"2024-03-12T05:53:37.202916+0000","last_unstale":"2024-03-12T05:53:39.197843+0000","last_undegraded":"2024-03-12T05:53:39.197843+0000","last_fullsized":"2024-03-12T05:53:39.197843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:09:34.629085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.385411+0000","last_change":"2024-03-12T05:53:38.220826+0000","last_active":"2024-03-12T05:53:40.385411+0000","last_peered":"2024-03-12T05:53:40.385411+0000","last_clean":"2024-03-12T05:53:40.385411+0000","last_became_active":"2024-03-12T05:53:38.220440+0000","last_became_peered":"2024-03-12T05:53:38.220440+0000","last_unstale":"2024-03-12T05:53:40.385411+0000","last_undegraded":"2024-03-12T05:53:40.385411+0000","last_fullsized":"2024-03-12T05:53:40.385411+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:53:41.311273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'19","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:43.819785+0000","last_change":"2024-03-12T05:53:37.206271+0000","last_active":"2024-03-12T05:53:43.819785+0000","last_peered":"2024-03-12T05:53:43.819785+0000","last_clean":"2024-03-12T05:53:43.819785+0000","last_became_active":"2024-03-12T05:53:37.205920+0000","last_became_peered":"2024-03-12T05:53:37.205920+0000","last_unstale":"2024-03-12T05:53:43.819785+0000","last_undegraded":"2024-03-12T05:53:43.819785+0000","last_fullsized":"2024-03-12T05:53:43.819785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:30.140159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38193,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":24,"num_write":20,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.183646+0000","last_change":"2024-03-12T05:53:38.211397+0000","last_active":"2024-03-12T05:53:40.183646+0000","last_peered":"2024-03-12T05:53:40.183646+0000","last_clean":"2024-03-12T05:53:40.183646+0000","last_became_active":"2024-03-12T05:53:38.211045+0000","last_became_peered":"2024-03-12T05:53:38.211045+0000","last_unstale":"2024-03-12T05:53:40.183646+0000","last_undegraded":"2024-03-12T05:53:40.183646+0000","last_fullsized":"2024-03-12T05:53:40.183646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:21:42.510436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.377295+0000","last_change":"2024-03-12T05:53:37.210871+0000","last_active":"2024-03-12T05:53:41.377295+0000","last_peered":"2024-03-12T05:53:41.377295+0000","last_clean":"2024-03-12T05:53:41.377295+0000","last_became_active":"2024-03-12T05:53:37.210526+0000","last_became_peered":"2024-03-12T05:53:37.210526+0000","last_unstale":"2024-03-12T05:53:41.377295+0000","last_undegraded":"2024-03-12T05:53:41.377295+0000","last_fullsized":"2024-03-12T05:53:41.377295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:40:46.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.912250+0000","last_change":"2024-03-12T05:53:38.213751+0000","last_active":"2024-03-12T05:53:39.912250+0000","last_peered":"2024-03-12T05:53:39.912250+0000","last_clean":"2024-03-12T05:53:39.912250+0000","last_became_active":"2024-03-12T05:53:38.211558+0000","last_became_peered":"2024-03-12T05:53:38.211558+0000","last_unstale":"2024-03-12T05:53:39.912250+0000","last_undegraded":"2024-03-12T05:53:39.912250+0000","last_fullsized":"2024-03-12T05:53:39.912250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:40:44.194548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197403+0000","last_change":"2024-03-12T05:53:37.207368+0000","last_active":"2024-03-12T05:53:39.197403+0000","last_peered":"2024-03-12T05:53:39.197403+0000","last_clean":"2024-03-12T05:53:39.197403+0000","last_became_active":"2024-03-12T05:53:37.207066+0000","last_became_peered":"2024-03-12T05:53:37.207066+0000","last_unstale":"2024-03-12T05:53:39.197403+0000","last_undegraded":"2024-03-12T05:53:39.197403+0000","last_fullsized":"2024-03-12T05:53:39.197403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:28:58.155360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465553+0000","last_change":"2024-03-12T05:53:38.217079+0000","last_active":"2024-03-12T05:53:39.465553+0000","last_peered":"2024-03-12T05:53:39.465553+0000","last_clean":"2024-03-12T05:53:39.465553+0000","last_became_active":"2024-03-12T05:53:38.216757+0000","last_became_peered":"2024-03-12T05:53:38.216757+0000","last_unstale":"2024-03-12T05:53:39.465553+0000","last_undegraded":"2024-03-12T05:53:39.465553+0000","last_fullsized":"2024-03-12T05:53:39.465553+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:48:13.125867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.198280+0000","last_change":"2024-03-12T05:53:37.202390+0000","last_active":"2024-03-12T05:53:39.198280+0000","last_peered":"2024-03-12T05:53:39.198280+0000","last_clean":"2024-03-12T05:53:39.198280+0000","last_became_active":"2024-03-12T05:53:37.202020+0000","last_became_peered":"2024-03-12T05:53:37.202020+0000","last_unstale":"2024-03-12T05:53:39.198280+0000","last_undegraded":"2024-03-12T05:53:39.198280+0000","last_fullsized":"2024-03-12T05:53:39.198280+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:06:35.293227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465285+0000","last_change":"2024-03-12T05:53:38.218464+0000","last_active":"2024-03-12T05:53:39.465285+0000","last_peered":"2024-03-12T05:53:39.465285+0000","last_clean":"2024-03-12T05:53:39.465285+0000","last_became_active":"2024-03-12T05:53:38.218136+0000","last_became_peered":"2024-03-12T05:53:38.218136+0000","last_unstale":"2024-03-12T05:53:39.465285+0000","last_undegraded":"2024-03-12T05:53:39.465285+0000","last_fullsized":"2024-03-12T05:53:39.465285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:31:05.409148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.376801+0000","last_change":"2024-03-12T05:53:37.205161+0000","last_active":"2024-03-12T05:53:41.376801+0000","last_peered":"2024-03-12T05:53:41.376801+0000","last_clean":"2024-03-12T05:53:41.376801+0000","last_became_active":"2024-03-12T05:53:37.204799+0000","last_became_peered":"2024-03-12T05:53:37.204799+0000","last_unstale":"2024-03-12T05:53:41.376801+0000","last_undegraded":"2024-03-12T05:53:41.376801+0000","last_fullsized":"2024-03-12T05:53:41.376801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:29:32.065635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197319+0000","last_change":"2024-03-12T05:53:38.216321+0000","last_active":"2024-03-12T05:53:39.197319+0000","last_peered":"2024-03-12T05:53:39.197319+0000","last_clean":"2024-03-12T05:53:39.197319+0000","last_became_active":"2024-03-12T05:53:38.215893+0000","last_became_peered":"2024-03-12T05:53:38.215893+0000","last_unstale":"2024-03-12T05:53:39.197319+0000","last_undegraded":"2024-03-12T05:53:39.197319+0000","last_fullsized":"2024-03-12T05:53:39.197319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:32:27.603930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.183844+0000","last_change":"2024-03-12T05:53:37.203638+0000","last_active":"2024-03-12T05:53:40.183844+0000","last_peered":"2024-03-12T05:53:40.183844+0000","last_clean":"2024-03-12T05:53:40.183844+0000","last_became_active":"2024-03-12T05:53:37.203217+0000","last_became_peered":"2024-03-12T05:53:37.203217+0000","last_unstale":"2024-03-12T05:53:40.183844+0000","last_undegraded":"2024-03-12T05:53:40.183844+0000","last_fullsized":"2024-03-12T05:53:40.183844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:10.521197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.318913+0000","last_change":"2024-03-12T05:53:38.210491+0000","last_active":"2024-03-12T05:53:39.318913+0000","last_peered":"2024-03-12T05:53:39.318913+0000","last_clean":"2024-03-12T05:53:39.318913+0000","last_became_active":"2024-03-12T05:53:38.210169+0000","last_became_peered":"2024-03-12T05:53:38.210169+0000","last_unstale":"2024-03-12T05:53:39.318913+0000","last_undegraded":"2024-03-12T05:53:39.318913+0000","last_fullsized":"2024-03-12T05:53:39.318913+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:56:05.384834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.367267+0000","last_change":"2024-03-12T05:53:37.204478+0000","last_active":"2024-03-12T05:53:40.367267+0000","last_peered":"2024-03-12T05:53:40.367267+0000","last_clean":"2024-03-12T05:53:40.367267+0000","last_became_active":"2024-03-12T05:53:37.204160+0000","last_became_peered":"2024-03-12T05:53:37.204160+0000","last_unstale":"2024-03-12T05:53:40.367267+0000","last_undegraded":"2024-03-12T05:53:40.367267+0000","last_fullsized":"2024-03-12T05:53:40.367267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:07.597731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.387503+0000","last_change":"2024-03-12T05:53:38.212913+0000","last_active":"2024-03-12T05:53:40.387503+0000","last_peered":"2024-03-12T05:53:40.387503+0000","last_clean":"2024-03-12T05:53:40.387503+0000","last_became_active":"2024-03-12T05:53:38.212558+0000","last_became_peered":"2024-03-12T05:53:38.212558+0000","last_unstale":"2024-03-12T05:53:40.387503+0000","last_undegraded":"2024-03-12T05:53:40.387503+0000","last_fullsized":"2024-03-12T05:53:40.387503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:40:59.993066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465726+0000","last_change":"2024-03-12T05:53:37.205983+0000","last_active":"2024-03-12T05:53:39.465726+0000","last_peered":"2024-03-12T05:53:39.465726+0000","last_clean":"2024-03-12T05:53:39.465726+0000","last_became_active":"2024-03-12T05:53:37.205629+0000","last_became_peered":"2024-03-12T05:53:37.205629+0000","last_unstale":"2024-03-12T05:53:39.465726+0000","last_undegraded":"2024-03-12T05:53:39.465726+0000","last_fullsized":"2024-03-12T05:53:39.465726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:37.928623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.387032+0000","last_change":"2024-03-12T05:53:38.213752+0000","last_active":"2024-03-12T05:53:40.387032+0000","last_peered":"2024-03-12T05:53:40.387032+0000","last_clean":"2024-03-12T05:53:40.387032+0000","last_became_active":"2024-03-12T05:53:38.213407+0000","last_became_peered":"2024-03-12T05:53:38.213407+0000","last_unstale":"2024-03-12T05:53:40.387032+0000","last_undegraded":"2024-03-12T05:53:40.387032+0000","last_fullsized":"2024-03-12T05:53:40.387032+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:21:15.703455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197688+0000","last_change":"2024-03-12T05:53:38.218205+0000","last_active":"2024-03-12T05:53:39.197688+0000","last_peered":"2024-03-12T05:53:39.197688+0000","last_clean":"2024-03-12T05:53:39.197688+0000","last_became_active":"2024-03-12T05:53:38.217725+0000","last_became_peered":"2024-03-12T05:53:38.217725+0000","last_unstale":"2024-03-12T05:53:39.197688+0000","last_undegraded":"2024-03-12T05:53:39.197688+0000","last_fullsized":"2024-03-12T05:53:39.197688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:48:30.050343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:43.393762+0000","last_change":"2024-03-12T05:53:37.202782+0000","last_active":"2024-03-12T05:53:43.393762+0000","last_peered":"2024-03-12T05:53:43.393762+0000","last_clean":"2024-03-12T05:53:43.393762+0000","last_became_active":"2024-03-12T05:53:37.202439+0000","last_became_peered":"2024-03-12T05:53:37.202439+0000","last_unstale":"2024-03-12T05:53:43.393762+0000","last_undegraded":"2024-03-12T05:53:43.393762+0000","last_fullsized":"2024-03-12T05:53:43.393762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:07:18.939629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197714+0000","last_change":"2024-03-12T05:53:38.217993+0000","last_active":"2024-03-12T05:53:39.197714+0000","last_peered":"2024-03-12T05:53:39.197714+0000","last_clean":"2024-03-12T05:53:39.197714+0000","last_became_active":"2024-03-12T05:53:38.217613+0000","last_became_peered":"2024-03-12T05:53:38.217613+0000","last_unstale":"2024-03-12T05:53:39.197714+0000","last_undegraded":"2024-03-12T05:53:39.197714+0000","last_fullsized":"2024-03-12T05:53:39.197714+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:49:50.960004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.388344+0000","last_change":"2024-03-12T05:53:37.203662+0000","last_active":"2024-03-12T05:53:40.388344+0000","last_peered":"2024-03-12T05:53:40.388344+0000","last_clean":"2024-03-12T05:53:40.388344+0000","last_became_active":"2024-03-12T05:53:37.203309+0000","last_became_peered":"2024-03-12T05:53:37.203309+0000","last_unstale":"2024-03-12T05:53:40.388344+0000","last_undegraded":"2024-03-12T05:53:40.388344+0000","last_fullsized":"2024-03-12T05:53:40.388344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:47:24.603686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465034+0000","last_change":"2024-03-12T05:52:27.278524+0000","last_active":"2024-03-12T05:53:39.465034+0000","last_peered":"2024-03-12T05:53:39.465034+0000","last_clean":"2024-03-12T05:53:39.465034+0000","last_became_active":"2024-03-12T05:52:27.278010+0000","last_became_peered":"2024-03-12T05:52:27.278010+0000","last_unstale":"2024-03-12T05:53:39.465034+0000","last_undegraded":"2024-03-12T05:53:39.465034+0000","last_fullsized":"2024-03-12T05:53:39.465034+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:49:09.175725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:49:09.175725+0000","last_clean_scrub_stamp":"2024-03-12T05:49:09.175725+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:24:40.380724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.912134+0000","last_change":"2024-03-12T05:53:38.219451+0000","last_active":"2024-03-12T05:53:39.912134+0000","last_peered":"2024-03-12T05:53:39.912134+0000","last_clean":"2024-03-12T05:53:39.912134+0000","last_became_active":"2024-03-12T05:53:38.218840+0000","last_became_peered":"2024-03-12T05:53:38.218840+0000","last_unstale":"2024-03-12T05:53:39.912134+0000","last_undegraded":"2024-03-12T05:53:39.912134+0000","last_fullsized":"2024-03-12T05:53:39.912134+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:05.595822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.375093+0000","last_change":"2024-03-12T05:53:37.210352+0000","last_active":"2024-03-12T05:53:41.375093+0000","last_peered":"2024-03-12T05:53:41.375093+0000","last_clean":"2024-03-12T05:53:41.375093+0000","last_became_active":"2024-03-12T05:53:37.210002+0000","last_became_peered":"2024-03-12T05:53:37.210002+0000","last_unstale":"2024-03-12T05:53:41.375093+0000","last_undegraded":"2024-03-12T05:53:41.375093+0000","last_fullsized":"2024-03-12T05:53:41.375093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:49:52.368411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.385031+0000","last_change":"2024-03-12T05:53:37.201136+0000","last_active":"2024-03-12T05:53:40.385031+0000","last_peered":"2024-03-12T05:53:40.385031+0000","last_clean":"2024-03-12T05:53:40.385031+0000","last_became_active":"2024-03-12T05:53:37.200695+0000","last_became_peered":"2024-03-12T05:53:37.200695+0000","last_unstale":"2024-03-12T05:53:40.385031+0000","last_undegraded":"2024-03-12T05:53:40.385031+0000","last_fullsized":"2024-03-12T05:53:40.385031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:55:22.428183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.912404+0000","last_change":"2024-03-12T05:53:38.203951+0000","last_active":"2024-03-12T05:53:39.912404+0000","last_peered":"2024-03-12T05:53:39.912404+0000","last_clean":"2024-03-12T05:53:39.912404+0000","last_became_active":"2024-03-12T05:53:38.203617+0000","last_became_peered":"2024-03-12T05:53:38.203617+0000","last_unstale":"2024-03-12T05:53:39.912404+0000","last_undegraded":"2024-03-12T05:53:39.912404+0000","last_fullsized":"2024-03-12T05:53:39.912404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:19.138220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.377824+0000","last_change":"2024-03-12T05:53:37.203770+0000","last_active":"2024-03-12T05:53:41.377824+0000","last_peered":"2024-03-12T05:53:41.377824+0000","last_clean":"2024-03-12T05:53:41.377824+0000","last_became_active":"2024-03-12T05:53:37.203456+0000","last_became_peered":"2024-03-12T05:53:37.203456+0000","last_unstale":"2024-03-12T05:53:41.377824+0000","last_undegraded":"2024-03-12T05:53:41.377824+0000","last_fullsized":"2024-03-12T05:53:41.377824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:32.291536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197684+0000","last_change":"2024-03-12T05:53:38.214872+0000","last_active":"2024-03-12T05:53:39.197684+0000","last_peered":"2024-03-12T05:53:39.197684+0000","last_clean":"2024-03-12T05:53:39.197684+0000","last_became_active":"2024-03-12T05:53:38.214139+0000","last_became_peered":"2024-03-12T05:53:38.214139+0000","last_unstale":"2024-03-12T05:53:39.197684+0000","last_undegraded":"2024-03-12T05:53:39.197684+0000","last_fullsized":"2024-03-12T05:53:39.197684+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:20:21.808039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.385592+0000","last_change":"2024-03-12T05:53:37.203852+0000","last_active":"2024-03-12T05:53:40.385592+0000","last_peered":"2024-03-12T05:53:40.385592+0000","last_clean":"2024-03-12T05:53:40.385592+0000","last_became_active":"2024-03-12T05:53:37.203416+0000","last_became_peered":"2024-03-12T05:53:37.203416+0000","last_unstale":"2024-03-12T05:53:40.385592+0000","last_undegraded":"2024-03-12T05:53:40.385592+0000","last_fullsized":"2024-03-12T05:53:40.385592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:09:08.095957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.213958+0000","last_change":"2024-03-12T05:53:38.211722+0000","last_active":"2024-03-12T05:53:38.213958+0000","last_peered":"2024-03-12T05:53:38.213958+0000","last_clean":"2024-03-12T05:53:38.213958+0000","last_became_active":"2024-03-12T05:53:38.211362+0000","last_became_peered":"2024-03-12T05:53:38.211362+0000","last_unstale":"2024-03-12T05:53:38.213958+0000","last_undegraded":"2024-03-12T05:53:38.213958+0000","last_fullsized":"2024-03-12T05:53:38.213958+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:21.465441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.387757+0000","last_change":"2024-03-12T05:53:37.207444+0000","last_active":"2024-03-12T05:53:40.387757+0000","last_peered":"2024-03-12T05:53:40.387757+0000","last_clean":"2024-03-12T05:53:40.387757+0000","last_became_active":"2024-03-12T05:53:37.207119+0000","last_became_peered":"2024-03-12T05:53:37.207119+0000","last_unstale":"2024-03-12T05:53:40.387757+0000","last_undegraded":"2024-03-12T05:53:40.387757+0000","last_fullsized":"2024-03-12T05:53:40.387757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:20:36.489684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.183387+0000","last_change":"2024-03-12T05:53:38.207757+0000","last_active":"2024-03-12T05:53:40.183387+0000","last_peered":"2024-03-12T05:53:40.183387+0000","last_clean":"2024-03-12T05:53:40.183387+0000","last_became_active":"2024-03-12T05:53:38.207424+0000","last_became_peered":"2024-03-12T05:53:38.207424+0000","last_unstale":"2024-03-12T05:53:40.183387+0000","last_undegraded":"2024-03-12T05:53:40.183387+0000","last_fullsized":"2024-03-12T05:53:40.183387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:33:16.017907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.388215+0000","last_change":"2024-03-12T05:53:37.208512+0000","last_active":"2024-03-12T05:53:40.388215+0000","last_peered":"2024-03-12T05:53:40.388215+0000","last_clean":"2024-03-12T05:53:40.388215+0000","last_became_active":"2024-03-12T05:53:37.207628+0000","last_became_peered":"2024-03-12T05:53:37.207628+0000","last_unstale":"2024-03-12T05:53:40.388215+0000","last_undegraded":"2024-03-12T05:53:40.388215+0000","last_fullsized":"2024-03-12T05:53:40.388215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:25:28.825779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.318575+0000","last_change":"2024-03-12T05:53:38.205654+0000","last_active":"2024-03-12T05:53:39.318575+0000","last_peered":"2024-03-12T05:53:39.318575+0000","last_clean":"2024-03-12T05:53:39.318575+0000","last_became_active":"2024-03-12T05:53:38.205325+0000","last_became_peered":"2024-03-12T05:53:38.205325+0000","last_unstale":"2024-03-12T05:53:39.318575+0000","last_undegraded":"2024-03-12T05:53:39.318575+0000","last_fullsized":"2024-03-12T05:53:39.318575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:33:35.668625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.334333+0000","last_change":"2024-03-12T05:53:37.211430+0000","last_active":"2024-03-12T05:53:39.334333+0000","last_peered":"2024-03-12T05:53:39.334333+0000","last_clean":"2024-03-12T05:53:39.334333+0000","last_became_active":"2024-03-12T05:53:37.210501+0000","last_became_peered":"2024-03-12T05:53:37.210501+0000","last_unstale":"2024-03-12T05:53:39.334333+0000","last_undegraded":"2024-03-12T05:53:39.334333+0000","last_fullsized":"2024-03-12T05:53:39.334333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:48:33.555134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197495+0000","last_change":"2024-03-12T05:53:38.212837+0000","last_active":"2024-03-12T05:53:39.197495+0000","last_peered":"2024-03-12T05:53:39.197495+0000","last_clean":"2024-03-12T05:53:39.197495+0000","last_became_active":"2024-03-12T05:53:38.212513+0000","last_became_peered":"2024-03-12T05:53:38.212513+0000","last_unstale":"2024-03-12T05:53:39.197495+0000","last_undegraded":"2024-03-12T05:53:39.197495+0000","last_fullsized":"2024-03-12T05:53:39.197495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:47:33.879954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.387273+0000","last_change":"2024-03-12T05:53:37.207465+0000","last_active":"2024-03-12T05:53:40.387273+0000","last_peered":"2024-03-12T05:53:40.387273+0000","last_clean":"2024-03-12T05:53:40.387273+0000","last_became_active":"2024-03-12T05:53:37.207107+0000","last_became_peered":"2024-03-12T05:53:37.207107+0000","last_unstale":"2024-03-12T05:53:40.387273+0000","last_undegraded":"2024-03-12T05:53:40.387273+0000","last_fullsized":"2024-03-12T05:53:40.387273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:34.018856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465402+0000","last_change":"2024-03-12T05:53:38.221470+0000","last_active":"2024-03-12T05:53:39.465402+0000","last_peered":"2024-03-12T05:53:39.465402+0000","last_clean":"2024-03-12T05:53:39.465402+0000","last_became_active":"2024-03-12T05:53:38.221061+0000","last_became_peered":"2024-03-12T05:53:38.221061+0000","last_unstale":"2024-03-12T05:53:39.465402+0000","last_undegraded":"2024-03-12T05:53:39.465402+0000","last_fullsized":"2024-03-12T05:53:39.465402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:01:54.148877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:41.376844+0000","last_change":"2024-03-12T05:53:37.203822+0000","last_active":"2024-03-12T05:53:41.376844+0000","last_peered":"2024-03-12T05:53:41.376844+0000","last_clean":"2024-03-12T05:53:41.376844+0000","last_became_active":"2024-03-12T05:53:37.203431+0000","last_became_peered":"2024-03-12T05:53:37.203431+0000","last_unstale":"2024-03-12T05:53:41.376844+0000","last_undegraded":"2024-03-12T05:53:41.376844+0000","last_fullsized":"2024-03-12T05:53:41.376844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:25:04.447261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465777+0000","last_change":"2024-03-12T05:53:38.222045+0000","last_active":"2024-03-12T05:53:39.465777+0000","last_peered":"2024-03-12T05:53:39.465777+0000","last_clean":"2024-03-12T05:53:39.465777+0000","last_became_active":"2024-03-12T05:53:38.220904+0000","last_became_peered":"2024-03-12T05:53:38.220904+0000","last_unstale":"2024-03-12T05:53:39.465777+0000","last_undegraded":"2024-03-12T05:53:39.465777+0000","last_fullsized":"2024-03-12T05:53:39.465777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:30:43.964599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197795+0000","last_change":"2024-03-12T05:53:37.206753+0000","last_active":"2024-03-12T05:53:39.197795+0000","last_peered":"2024-03-12T05:53:39.197795+0000","last_clean":"2024-03-12T05:53:39.197795+0000","last_became_active":"2024-03-12T05:53:37.206078+0000","last_became_peered":"2024-03-12T05:53:37.206078+0000","last_unstale":"2024-03-12T05:53:39.197795+0000","last_undegraded":"2024-03-12T05:53:39.197795+0000","last_fullsized":"2024-03-12T05:53:39.197795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:42:18.859528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465670+0000","last_change":"2024-03-12T05:53:38.215671+0000","last_active":"2024-03-12T05:53:39.465670+0000","last_peered":"2024-03-12T05:53:39.465670+0000","last_clean":"2024-03-12T05:53:39.465670+0000","last_became_active":"2024-03-12T05:53:38.215309+0000","last_became_peered":"2024-03-12T05:53:38.215309+0000","last_unstale":"2024-03-12T05:53:39.465670+0000","last_undegraded":"2024-03-12T05:53:39.465670+0000","last_fullsized":"2024-03-12T05:53:39.465670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:50:12.176935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.912403+0000","last_change":"2024-03-12T05:53:37.205627+0000","last_active":"2024-03-12T05:53:39.912403+0000","last_peered":"2024-03-12T05:53:39.912403+0000","last_clean":"2024-03-12T05:53:39.912403+0000","last_became_active":"2024-03-12T05:53:37.205289+0000","last_became_peered":"2024-03-12T05:53:37.205289+0000","last_unstale":"2024-03-12T05:53:39.912403+0000","last_undegraded":"2024-03-12T05:53:39.912403+0000","last_fullsized":"2024-03-12T05:53:39.912403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:56.682746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.465593+0000","last_change":"2024-03-12T05:53:38.214376+0000","last_active":"2024-03-12T05:53:39.465593+0000","last_peered":"2024-03-12T05:53:39.465593+0000","last_clean":"2024-03-12T05:53:39.465593+0000","last_became_active":"2024-03-12T05:53:38.214011+0000","last_became_peered":"2024-03-12T05:53:38.214011+0000","last_unstale":"2024-03-12T05:53:39.465593+0000","last_undegraded":"2024-03-12T05:53:39.465593+0000","last_fullsized":"2024-03-12T05:53:39.465593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:32.605216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.318744+0000","last_change":"2024-03-12T05:53:38.205337+0000","last_active":"2024-03-12T05:53:39.318744+0000","last_peered":"2024-03-12T05:53:39.318744+0000","last_clean":"2024-03-12T05:53:39.318744+0000","last_became_active":"2024-03-12T05:53:38.205005+0000","last_became_peered":"2024-03-12T05:53:38.205005+0000","last_unstale":"2024-03-12T05:53:39.318744+0000","last_undegraded":"2024-03-12T05:53:39.318744+0000","last_fullsized":"2024-03-12T05:53:39.318744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:44:16.296370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.334470+0000","last_change":"2024-03-12T05:53:38.214694+0000","last_active":"2024-03-12T05:53:39.334470+0000","last_peered":"2024-03-12T05:53:39.334470+0000","last_clean":"2024-03-12T05:53:39.334470+0000","last_became_active":"2024-03-12T05:53:38.214285+0000","last_became_peered":"2024-03-12T05:53:38.214285+0000","last_unstale":"2024-03-12T05:53:39.334470+0000","last_undegraded":"2024-03-12T05:53:39.334470+0000","last_fullsized":"2024-03-12T05:53:39.334470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:28:32.253332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.334386+0000","last_change":"2024-03-12T05:53:38.213069+0000","last_active":"2024-03-12T05:53:39.334386+0000","last_peered":"2024-03-12T05:53:39.334386+0000","last_clean":"2024-03-12T05:53:39.334386+0000","last_became_active":"2024-03-12T05:53:38.212694+0000","last_became_peered":"2024-03-12T05:53:38.212694+0000","last_unstale":"2024-03-12T05:53:39.334386+0000","last_undegraded":"2024-03-12T05:53:39.334386+0000","last_fullsized":"2024-03-12T05:53:39.334386+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:26:29.350292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-12T05:53:38.214060+0000","last_change":"2024-03-12T05:53:38.212084+0000","last_active":"2024-03-12T05:53:38.214060+0000","last_peered":"2024-03-12T05:53:38.214060+0000","last_clean":"2024-03-12T05:53:38.214060+0000","last_became_active":"2024-03-12T05:53:38.211750+0000","last_became_peered":"2024-03-12T05:53:38.211750+0000","last_unstale":"2024-03-12T05:53:38.214060+0000","last_undegraded":"2024-03-12T05:53:38.214060+0000","last_fullsized":"2024-03-12T05:53:38.214060+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:39:39.892739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.912470+0000","last_change":"2024-03-12T05:53:38.212071+0000","last_active":"2024-03-12T05:53:39.912470+0000","last_peered":"2024-03-12T05:53:39.912470+0000","last_clean":"2024-03-12T05:53:39.912470+0000","last_became_active":"2024-03-12T05:53:38.211722+0000","last_became_peered":"2024-03-12T05:53:38.211722+0000","last_unstale":"2024-03-12T05:53:39.912470+0000","last_undegraded":"2024-03-12T05:53:39.912470+0000","last_fullsized":"2024-03-12T05:53:39.912470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:19:46.581837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.318876+0000","last_change":"2024-03-12T05:53:38.206819+0000","last_active":"2024-03-12T05:53:39.318876+0000","last_peered":"2024-03-12T05:53:39.318876+0000","last_clean":"2024-03-12T05:53:39.318876+0000","last_became_active":"2024-03-12T05:53:38.206496+0000","last_became_peered":"2024-03-12T05:53:38.206496+0000","last_unstale":"2024-03-12T05:53:39.318876+0000","last_undegraded":"2024-03-12T05:53:39.318876+0000","last_fullsized":"2024-03-12T05:53:39.318876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:31:56.920976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.334811+0000","last_change":"2024-03-12T05:53:38.208018+0000","last_active":"2024-03-12T05:53:39.334811+0000","last_peered":"2024-03-12T05:53:39.334811+0000","last_clean":"2024-03-12T05:53:39.334811+0000","last_became_active":"2024-03-12T05:53:38.207682+0000","last_became_peered":"2024-03-12T05:53:38.207682+0000","last_unstale":"2024-03-12T05:53:39.334811+0000","last_undegraded":"2024-03-12T05:53:39.334811+0000","last_fullsized":"2024-03-12T05:53:39.334811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:56:10.903917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197761+0000","last_change":"2024-03-12T05:53:38.217540+0000","last_active":"2024-03-12T05:53:39.197761+0000","last_peered":"2024-03-12T05:53:39.197761+0000","last_clean":"2024-03-12T05:53:39.197761+0000","last_became_active":"2024-03-12T05:53:38.217144+0000","last_became_peered":"2024-03-12T05:53:38.217144+0000","last_unstale":"2024-03-12T05:53:39.197761+0000","last_undegraded":"2024-03-12T05:53:39.197761+0000","last_fullsized":"2024-03-12T05:53:39.197761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:46:18.905909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197969+0000","last_change":"2024-03-12T05:53:38.215006+0000","last_active":"2024-03-12T05:53:39.197969+0000","last_peered":"2024-03-12T05:53:39.197969+0000","last_clean":"2024-03-12T05:53:39.197969+0000","last_became_active":"2024-03-12T05:53:38.213756+0000","last_became_peered":"2024-03-12T05:53:38.213756+0000","last_unstale":"2024-03-12T05:53:39.197969+0000","last_undegraded":"2024-03-12T05:53:39.197969+0000","last_fullsized":"2024-03-12T05:53:39.197969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:14:57.642381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.197539+0000","last_change":"2024-03-12T05:53:38.216096+0000","last_active":"2024-03-12T05:53:39.197539+0000","last_peered":"2024-03-12T05:53:39.197539+0000","last_clean":"2024-03-12T05:53:39.197539+0000","last_became_active":"2024-03-12T05:53:38.215153+0000","last_became_peered":"2024-03-12T05:53:38.215153+0000","last_unstale":"2024-03-12T05:53:39.197539+0000","last_undegraded":"2024-03-12T05:53:39.197539+0000","last_fullsized":"2024-03-12T05:53:39.197539+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:41:03.893641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:40.384862+0000","last_change":"2024-03-12T05:53:38.219165+0000","last_active":"2024-03-12T05:53:40.384862+0000","last_peered":"2024-03-12T05:53:40.384862+0000","last_clean":"2024-03-12T05:53:40.384862+0000","last_became_active":"2024-03-12T05:53:38.218579+0000","last_became_peered":"2024-03-12T05:53:38.218579+0000","last_unstale":"2024-03-12T05:53:40.384862+0000","last_undegraded":"2024-03-12T05:53:40.384862+0000","last_fullsized":"2024-03-12T05:53:40.384862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:06:37.531293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.912329+0000","last_change":"2024-03-12T05:53:38.212685+0000","last_active":"2024-03-12T05:53:39.912329+0000","last_peered":"2024-03-12T05:53:39.912329+0000","last_clean":"2024-03-12T05:53:39.912329+0000","last_became_active":"2024-03-12T05:53:38.211106+0000","last_became_peered":"2024-03-12T05:53:38.211106+0000","last_unstale":"2024-03-12T05:53:39.912329+0000","last_undegraded":"2024-03-12T05:53:39.912329+0000","last_fullsized":"2024-03-12T05:53:39.912329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:26:00.339270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-12T05:53:39.318641+0000","last_change":"2024-03-12T05:53:38.218402+0000","last_active":"2024-03-12T05:53:39.318641+0000","last_peered":"2024-03-12T05:53:39.318641+0000","last_clean":"2024-03-12T05:53:39.318641+0000","last_became_active":"2024-03-12T05:53:38.217699+0000","last_became_peered":"2024-03-12T05:53:38.217699+0000","last_unstale":"2024-03-12T05:53:39.318641+0000","last_undegraded":"2024-03-12T05:53:39.318641+0000","last_fullsized":"2024-03-12T05:53:39.318641+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:23:03.221710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":39977,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":23,"num_read_kb":38,"num_write":128,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":92906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":929792,"data_stored":444507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612627,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32724,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712428,"statfs":{"total":95995035648,"available":95961526272,"internally_reserved":0,"allocated":1585152,"data_stored":1056883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776152,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":946176,"data_stored":444743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037789,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":444663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332129,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32676,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712476,"statfs":{"total":95995035648,"available":95961575424,"internally_reserved":0,"allocated":1536000,"data_stored":1036085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":444132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":954368,"data_stored":464820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":917504,"data_stored":435445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":453878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":444209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36244,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708908,"statfs":{"total":95995035648,"available":95957921792,"internally_reserved":0,"allocated":929792,"data_stored":443947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757291,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28520,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716632,"statfs":{"total":95995035648,"available":95965831168,"internally_reserved":0,"allocated":1531904,"data_stored":1034475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":38193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":9953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":39771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T05:53:45.468 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-12T05:53:45.468 INFO:tasks.cephadm:Setting up client nodes... 2024-03-12T05:53:45.468 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T05:53:45.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:45 smithi082 ceph-mon[36382]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:45.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:45 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2598967956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:45.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:45 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1029554712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:45.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:45 smithi082 ceph-mon[36382]: pgmap v288: 97 pgs: 97 active+clean; 616 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9.5 KiB/s wr, 19 op/s 2024-03-12T05:53:45.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:45 smithi088 ceph-mon[36585]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:45.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:45 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2598967956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:45.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:45 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1029554712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:45.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:45 smithi088 ceph-mon[36585]: pgmap v288: 97 pgs: 97 active+clean; 616 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9.5 KiB/s wr, 19 op/s 2024-03-12T05:53:45.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:45 smithi076 ceph-mon[32095]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:53:45.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:45 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2598967956' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:53:45.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:45 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1029554712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:45.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:45 smithi076 ceph-mon[32095]: pgmap v288: 97 pgs: 97 active+clean; 616 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9.5 KiB/s wr, 19 op/s 2024-03-12T05:53:45.743 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:46.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:46 smithi076 ceph-mon[32095]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:53:46.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:46 smithi076 ceph-mon[32095]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:53:46.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:46 smithi082 ceph-mon[36382]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:53:46.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:46 smithi082 ceph-mon[36382]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:53:46.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:46 smithi088 ceph-mon[36585]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:53:46.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:46 smithi088 ceph-mon[36585]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:53:47.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.367+0000 7fa757403700 1 -- 172.21.15.76:0/1202167450 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa750102790 msgr2=0x7fa750109040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:47.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.367+0000 7fa757403700 1 --2- 172.21.15.76:0/1202167450 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa750102790 0x7fa750109040 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa74800b0d0 tx=0x7fa74800b3e0 comp rx=0 tx=0).stop 2024-03-12T05:53:47.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.368+0000 7fa757403700 1 -- 172.21.15.76:0/1202167450 shutdown_connections 2024-03-12T05:53:47.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.368+0000 7fa757403700 1 --2- 172.21.15.76:0/1202167450 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa750102790 0x7fa750109040 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:47.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.368+0000 7fa757403700 1 --2- 172.21.15.76:0/1202167450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa750100bf0 0x7fa750101010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:47.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.368+0000 7fa757403700 1 --2- 172.21.15.76:0/1202167450 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa750101df0 0x7fa750102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:47.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.368+0000 7fa757403700 1 -- 172.21.15.76:0/1202167450 >> 172.21.15.76:0/1202167450 conn(0x7fa7500fc190 msgr2=0x7fa7500fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:47.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.368+0000 7fa757403700 1 -- 172.21.15.76:0/1202167450 shutdown_connections 2024-03-12T05:53:47.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.368+0000 7fa757403700 1 -- 172.21.15.76:0/1202167450 wait complete. 2024-03-12T05:53:47.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.368+0000 7fa757403700 1 Processor -- start 2024-03-12T05:53:47.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.368+0000 7fa757403700 1 -- start start 2024-03-12T05:53:47.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa757403700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa750100bf0 0x7fa750196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:47.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa757403700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa750101df0 0x7fa7501970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:47.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa757403700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa750102790 0x7fa75019e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:47.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa757403700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75019e730 con 0x7fa750102790 2024-03-12T05:53:47.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa757403700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75019e870 con 0x7fa750100bf0 2024-03-12T05:53:47.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa757403700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75019ea00 con 0x7fa750101df0 2024-03-12T05:53:47.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa7559a0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa750102790 0x7fa75019e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:47.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa75499e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa750101df0 0x7fa7501970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:47.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa75499e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa750101df0 0x7fa7501970e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.76:58370/0 (socket says 172.21.15.76:58370) 2024-03-12T05:53:47.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa75499e700 1 -- 172.21.15.76:0/1790095721 learned_addr learned my addr 172.21.15.76:0/1790095721 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:47.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa7559a0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa750102790 0x7fa75019e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53630/0 (socket says 172.21.15.76:53630) 2024-03-12T05:53:47.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa75519f700 1 --2- 172.21.15.76:0/1790095721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa750100bf0 0x7fa750196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:47.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa7559a0700 1 -- 172.21.15.76:0/1790095721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa750101df0 msgr2=0x7fa7501970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:47.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa7559a0700 1 --2- 172.21.15.76:0/1790095721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa750101df0 0x7fa7501970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:47.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa7559a0700 1 -- 172.21.15.76:0/1790095721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa750100bf0 msgr2=0x7fa750196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:47.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa7559a0700 1 --2- 172.21.15.76:0/1790095721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa750100bf0 0x7fa750196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:47.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa7559a0700 1 -- 172.21.15.76:0/1790095721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa748009d00 con 0x7fa750102790 2024-03-12T05:53:47.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa75499e700 1 --2- 172.21.15.76:0/1790095721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa750101df0 0x7fa7501970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:53:47.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa7559a0700 1 --2- 172.21.15.76:0/1790095721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa750102790 0x7fa75019e160 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa74800b770 tx=0x7fa74800bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:47.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa7467fc700 1 -- 172.21.15.76:0/1790095721 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74800e040 con 0x7fa750102790 2024-03-12T05:53:47.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa757403700 1 -- 172.21.15.76:0/1790095721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75019ec80 con 0x7fa750102790 2024-03-12T05:53:47.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa7467fc700 1 -- 172.21.15.76:0/1790095721 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa748003e10 con 0x7fa750102790 2024-03-12T05:53:47.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.369+0000 7fa7467fc700 1 -- 172.21.15.76:0/1790095721 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74801da60 con 0x7fa750102790 2024-03-12T05:53:47.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.370+0000 7fa757403700 1 -- 172.21.15.76:0/1790095721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75019f090 con 0x7fa750102790 2024-03-12T05:53:47.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.371+0000 7fa757403700 1 -- 172.21.15.76:0/1790095721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa750107ee0 con 0x7fa750102790 2024-03-12T05:53:47.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.371+0000 7fa7467fc700 1 -- 172.21.15.76:0/1790095721 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fa748005670 con 0x7fa750102790 2024-03-12T05:53:47.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.372+0000 7fa7467fc700 1 --2- 172.21.15.76:0/1790095721 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa73c0a1c00 0x7fa73c0a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:47.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.372+0000 7fa75519f700 1 --2- 172.21.15.76:0/1790095721 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa73c0a1c00 0x7fa73c0a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:47.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.372+0000 7fa7467fc700 1 -- 172.21.15.76:0/1790095721 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa7480ce6c0 con 0x7fa750102790 2024-03-12T05:53:47.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.372+0000 7fa75519f700 1 --2- 172.21.15.76:0/1790095721 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa73c0a1c00 0x7fa73c0a40c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa750101c50 tx=0x7fa74c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:47.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.376+0000 7fa7467fc700 1 -- 172.21.15.76:0/1790095721 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa748097a40 con 0x7fa750102790 2024-03-12T05:53:47.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:47 smithi076 ceph-mon[32095]: pgmap v289: 97 pgs: 97 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9.1 KiB/s wr, 20 op/s 2024-03-12T05:53:47.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.650+0000 7fa757403700 1 -- 172.21.15.76:0/1790095721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fa75010c670 con 0x7fa750102790 2024-03-12T05:53:47.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.652+0000 7fa7467fc700 1 -- 172.21.15.76:0/1790095721 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7fa74809b100 con 0x7fa750102790 2024-03-12T05:53:47.655 INFO:teuthology.orchestra.run.smithi076.stdout:[client.0] 2024-03-12T05:53:47.655 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQDr7e9l24fVJhAARyrvHJLWXKgEowq5UV6Mbg== 2024-03-12T05:53:47.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.655+0000 7fa757403700 1 -- 172.21.15.76:0/1790095721 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa73c0a1c00 msgr2=0x7fa73c0a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:47.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.655+0000 7fa757403700 1 --2- 172.21.15.76:0/1790095721 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa73c0a1c00 0x7fa73c0a40c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa750101c50 tx=0x7fa74c009260 comp rx=0 tx=0).stop 2024-03-12T05:53:47.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.655+0000 7fa757403700 1 -- 172.21.15.76:0/1790095721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa750102790 msgr2=0x7fa75019e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:47.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.655+0000 7fa757403700 1 --2- 172.21.15.76:0/1790095721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa750102790 0x7fa75019e160 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa74800b770 tx=0x7fa74800bcd0 comp rx=0 tx=0).stop 2024-03-12T05:53:47.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.655+0000 7fa757403700 1 -- 172.21.15.76:0/1790095721 shutdown_connections 2024-03-12T05:53:47.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.655+0000 7fa757403700 1 --2- 172.21.15.76:0/1790095721 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa73c0a1c00 0x7fa73c0a40c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:47.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.655+0000 7fa757403700 1 --2- 172.21.15.76:0/1790095721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa750102790 0x7fa75019e160 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:47.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.655+0000 7fa757403700 1 --2- 172.21.15.76:0/1790095721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa750100bf0 0x7fa750196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:47.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.655+0000 7fa757403700 1 --2- 172.21.15.76:0/1790095721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa750101df0 0x7fa7501970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:47.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.655+0000 7fa757403700 1 -- 172.21.15.76:0/1790095721 >> 172.21.15.76:0/1790095721 conn(0x7fa7500fc190 msgr2=0x7fa750105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:47.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.655+0000 7fa757403700 1 -- 172.21.15.76:0/1790095721 shutdown_connections 2024-03-12T05:53:47.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:47.655+0000 7fa757403700 1 -- 172.21.15.76:0/1790095721 wait complete. 2024-03-12T05:53:47.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:47 smithi082 ceph-mon[36382]: pgmap v289: 97 pgs: 97 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9.1 KiB/s wr, 20 op/s 2024-03-12T05:53:47.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:47 smithi088 ceph-mon[36585]: pgmap v289: 97 pgs: 97 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9.1 KiB/s wr, 20 op/s 2024-03-12T05:53:48.383 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:53:48.383 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-12T05:53:48.383 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-12T05:53:48.426 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-12T05:53:48.427 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-12T05:53:48.427 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-12T05:53:48.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:48 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1790095721' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T05:53:48.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:48 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1790095721' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T05:53:48.706 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:48.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:48 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1790095721' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T05:53:48.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:48 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1790095721' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T05:53:48.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:48 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1790095721' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T05:53:48.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:48 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1790095721' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T05:53:49.694 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:49 smithi082 ceph-mon[36382]: pgmap v290: 97 pgs: 97 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-03-12T05:53:49.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:49 smithi076 ceph-mon[32095]: pgmap v290: 97 pgs: 97 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-03-12T05:53:49.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:49 smithi088 ceph-mon[36585]: pgmap v290: 97 pgs: 97 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-03-12T05:53:50.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.196+0000 7f1a71370700 1 -- 172.21.15.76:0/1280543246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1a6c06af20 msgr2=0x7f1a6c1068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:50.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.196+0000 7f1a71370700 1 --2- 172.21.15.76:0/1280543246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1a6c06af20 0x7f1a6c1068b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1a60009990 tx=0x7f1a60009ca0 comp rx=0 tx=0).stop 2024-03-12T05:53:50.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.196+0000 7f1a71370700 1 -- 172.21.15.76:0/1280543246 shutdown_connections 2024-03-12T05:53:50.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.196+0000 7f1a71370700 1 --2- 172.21.15.76:0/1280543246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a6c106fe0 0x7f1a6c10eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:50.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.196+0000 7f1a71370700 1 --2- 172.21.15.76:0/1280543246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1a6c06a570 0x7f1a6c06a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:50.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.196+0000 7f1a71370700 1 --2- 172.21.15.76:0/1280543246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1a6c06af20 0x7f1a6c1068b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:50.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.196+0000 7f1a71370700 1 -- 172.21.15.76:0/1280543246 >> 172.21.15.76:0/1280543246 conn(0x7f1a6c077ae0 msgr2=0x7f1a6c077ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:50.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.196+0000 7f1a71370700 1 -- 172.21.15.76:0/1280543246 shutdown_connections 2024-03-12T05:53:50.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.196+0000 7f1a71370700 1 -- 172.21.15.76:0/1280543246 wait complete. 2024-03-12T05:53:50.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.197+0000 7f1a71370700 1 Processor -- start 2024-03-12T05:53:50.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.197+0000 7f1a71370700 1 -- start start 2024-03-12T05:53:50.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.197+0000 7f1a71370700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1a6c06a570 0x7f1a6c100c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:50.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.197+0000 7f1a71370700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1a6c06af20 0x7f1a6c101170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:50.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.197+0000 7f1a71370700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a6c106fe0 0x7f1a6c1016b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:50.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.197+0000 7f1a71370700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1a65f0 con 0x7f1a6c06af20 2024-03-12T05:53:50.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.197+0000 7f1a71370700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1a6760 con 0x7f1a6c06a570 2024-03-12T05:53:50.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.197+0000 7f1a71370700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c1a6920 con 0x7f1a6c106fe0 2024-03-12T05:53:50.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.197+0000 7f1a6a7fc700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1a6c06af20 0x7f1a6c101170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:50.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.197+0000 7f1a6a7fc700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1a6c06af20 0x7f1a6c101170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53650/0 (socket says 172.21.15.76:53650) 2024-03-12T05:53:50.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.197+0000 7f1a6a7fc700 1 -- 172.21.15.76:0/1432039060 learned_addr learned my addr 172.21.15.76:0/1432039060 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:50.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.198+0000 7f1a6affd700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1a6c06a570 0x7f1a6c100c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:50.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.198+0000 7f1a6b7fe700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a6c106fe0 0x7f1a6c1016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:50.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.198+0000 7f1a6a7fc700 1 -- 172.21.15.76:0/1432039060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a6c106fe0 msgr2=0x7f1a6c1016b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:50.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.198+0000 7f1a6a7fc700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a6c106fe0 0x7f1a6c1016b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:50.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.198+0000 7f1a6a7fc700 1 -- 172.21.15.76:0/1432039060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1a6c06a570 msgr2=0x7f1a6c100c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:50.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.198+0000 7f1a6a7fc700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1a6c06a570 0x7f1a6c100c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:50.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.198+0000 7f1a6a7fc700 1 -- 172.21.15.76:0/1432039060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a600095f0 con 0x7f1a6c06af20 2024-03-12T05:53:50.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.198+0000 7f1a6b7fe700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a6c106fe0 0x7f1a6c1016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:53:50.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.198+0000 7f1a6a7fc700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1a6c06af20 0x7f1a6c101170 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1a60006010 tx=0x7f1a6000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:50.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.198+0000 7f1a53fff700 1 -- 172.21.15.76:0/1432039060 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6001d070 con 0x7f1a6c06af20 2024-03-12T05:53:50.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.198+0000 7f1a53fff700 1 -- 172.21.15.76:0/1432039060 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a60003680 con 0x7f1a6c06af20 2024-03-12T05:53:50.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.198+0000 7f1a71370700 1 -- 172.21.15.76:0/1432039060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6c1a6ba0 con 0x7f1a6c06af20 2024-03-12T05:53:50.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.198+0000 7f1a53fff700 1 -- 172.21.15.76:0/1432039060 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a600173f0 con 0x7f1a6c06af20 2024-03-12T05:53:50.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.198+0000 7f1a71370700 1 -- 172.21.15.76:0/1432039060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6c1a6fb0 con 0x7f1a6c06af20 2024-03-12T05:53:50.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.199+0000 7f1a71370700 1 -- 172.21.15.76:0/1432039060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6c10a190 con 0x7f1a6c06af20 2024-03-12T05:53:50.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.201+0000 7f1a53fff700 1 -- 172.21.15.76:0/1432039060 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f1a60022470 con 0x7f1a6c06af20 2024-03-12T05:53:50.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.201+0000 7f1a53fff700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1a540a1b50 0x7f1a540a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:50.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.201+0000 7f1a6affd700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1a540a1b50 0x7f1a540a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:50.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.201+0000 7f1a53fff700 1 -- 172.21.15.76:0/1432039060 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1a600ce950 con 0x7f1a6c06af20 2024-03-12T05:53:50.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.202+0000 7f1a6affd700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1a540a1b50 0x7f1a540a4010 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c005fd0 tx=0x7f1a5c005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:50.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.205+0000 7f1a53fff700 1 -- 172.21.15.76:0/1432039060 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1a60097c40 con 0x7f1a6c06af20 2024-03-12T05:53:50.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.455+0000 7f1a71370700 1 -- 172.21.15.76:0/1432039060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f1a6c103390 con 0x7f1a6c06af20 2024-03-12T05:53:50.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.460+0000 7f1a53fff700 1 -- 172.21.15.76:0/1432039060 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260662 (secure 0 0 0) 0x7f1a60086020 con 0x7f1a6c06af20 2024-03-12T05:53:50.466 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:50.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.462+0000 7f1a71370700 1 -- 172.21.15.76:0/1432039060 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1a540a1b50 msgr2=0x7f1a540a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:50.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.462+0000 7f1a71370700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1a540a1b50 0x7f1a540a4010 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c005fd0 tx=0x7f1a5c005f60 comp rx=0 tx=0).stop 2024-03-12T05:53:50.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.462+0000 7f1a71370700 1 -- 172.21.15.76:0/1432039060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1a6c06af20 msgr2=0x7f1a6c101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:50.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.462+0000 7f1a71370700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1a6c06af20 0x7f1a6c101170 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1a60006010 tx=0x7f1a6000f740 comp rx=0 tx=0).stop 2024-03-12T05:53:50.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.462+0000 7f1a71370700 1 -- 172.21.15.76:0/1432039060 shutdown_connections 2024-03-12T05:53:50.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.462+0000 7f1a71370700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f1a540a1b50 0x7f1a540a4010 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:50.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.462+0000 7f1a71370700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a6c106fe0 0x7f1a6c1016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:50.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.462+0000 7f1a71370700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1a6c06a570 0x7f1a6c100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:50.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.462+0000 7f1a71370700 1 --2- 172.21.15.76:0/1432039060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1a6c06af20 0x7f1a6c101170 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:50.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.462+0000 7f1a71370700 1 -- 172.21.15.76:0/1432039060 >> 172.21.15.76:0/1432039060 conn(0x7f1a6c077ae0 msgr2=0x7f1a6c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:50.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.463+0000 7f1a71370700 1 -- 172.21.15.76:0/1432039060 shutdown_connections 2024-03-12T05:53:50.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:50.463+0000 7f1a71370700 1 -- 172.21.15.76:0/1432039060 wait complete. 2024-03-12T05:53:50.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:50 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1432039060' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T05:53:50.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:50 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1432039060' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T05:53:50.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:50 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1432039060' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T05:53:51.273 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":1718825251},{"type":"v1","addr":"172.21.15.76:6801","nonce":1718825251}]},"active_addr":"172.21.15.76:6801/1718825251","active_change":"2024-03-12T05:46:40.781860+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.76:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":1939795424}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":3998885619}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":2299856451}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":189327369}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":4131633709}]}]} 2024-03-12T05:53:51.280 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-12T05:53:51.280 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-12T05:53:51.280 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T05:53:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:51.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:51 smithi076 ceph-mon[32095]: pgmap v291: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 7.5 KiB/s wr, 18 op/s 2024-03-12T05:53:51.944 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:51 smithi082 ceph-mon[36382]: pgmap v291: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 7.5 KiB/s wr, 18 op/s 2024-03-12T05:53:51.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:51 smithi088 ceph-mon[36585]: pgmap v291: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 7.5 KiB/s wr, 18 op/s 2024-03-12T05:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.207+0000 7f59733b3700 1 -- 172.21.15.76:0/2567257094 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f596c106ff0 msgr2=0x7f596c10ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.207+0000 7f59733b3700 1 --2- 172.21.15.76:0/2567257094 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f596c106ff0 0x7f596c10ebc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f596400b0a0 tx=0x7f596400b3b0 comp rx=0 tx=0).stop 2024-03-12T05:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.207+0000 7f59733b3700 1 -- 172.21.15.76:0/2567257094 shutdown_connections 2024-03-12T05:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.207+0000 7f59733b3700 1 --2- 172.21.15.76:0/2567257094 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f596c106ff0 0x7f596c10ebc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.207+0000 7f59733b3700 1 --2- 172.21.15.76:0/2567257094 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f596c101f20 0x7f596c102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.207+0000 7f59733b3700 1 --2- 172.21.15.76:0/2567257094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f596c1028d0 0x7f596c106910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.207+0000 7f59733b3700 1 -- 172.21.15.76:0/2567257094 >> 172.21.15.76:0/2567257094 conn(0x7f596c078dd0 msgr2=0x7f596c0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.207+0000 7f59733b3700 1 -- 172.21.15.76:0/2567257094 shutdown_connections 2024-03-12T05:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.207+0000 7f59733b3700 1 -- 172.21.15.76:0/2567257094 wait complete. 2024-03-12T05:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.208+0000 7f59733b3700 1 Processor -- start 2024-03-12T05:53:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.208+0000 7f59733b3700 1 -- start start 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.208+0000 7f59733b3700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f596c101f20 0x7f596c198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.208+0000 7f59733b3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f596c1028d0 0x7f596c198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.208+0000 7f59733b3700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f596c106ff0 0x7f596c19cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.208+0000 7f59733b3700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c19d230 con 0x7f596c106ff0 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.208+0000 7f59733b3700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c19d3a0 con 0x7f596c101f20 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.208+0000 7f59733b3700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c19d560 con 0x7f596c1028d0 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.208+0000 7f5971950700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f596c106ff0 0x7f596c19cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.208+0000 7f597114f700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f596c101f20 0x7f596c198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.208+0000 7f597094e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f596c1028d0 0x7f596c198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.208+0000 7f5971950700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f596c106ff0 0x7f596c19cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49100/0 (socket says 172.21.15.76:49100) 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.208+0000 7f5971950700 1 -- 172.21.15.76:0/3488613237 learned_addr learned my addr 172.21.15.76:0/3488613237 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.208+0000 7f597114f700 1 --2- 172.21.15.76:0/3488613237 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f596c101f20 0x7f596c198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:50202/0 (socket says 172.21.15.76:50202) 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.209+0000 7f5971950700 1 -- 172.21.15.76:0/3488613237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f596c1028d0 msgr2=0x7f596c198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.209+0000 7f5971950700 1 --2- 172.21.15.76:0/3488613237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f596c1028d0 0x7f596c198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.209+0000 7f5971950700 1 -- 172.21.15.76:0/3488613237 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f596c101f20 msgr2=0x7f596c198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:53.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.209+0000 7f5971950700 1 --2- 172.21.15.76:0/3488613237 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f596c101f20 0x7f596c198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:53.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.209+0000 7f5971950700 1 -- 172.21.15.76:0/3488613237 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5964009d00 con 0x7f596c106ff0 2024-03-12T05:53:53.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.209+0000 7f5971950700 1 --2- 172.21.15.76:0/3488613237 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f596c106ff0 0x7f596c19cc20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f596400b070 tx=0x7f5964004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:53.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.209+0000 7f59627fc700 1 -- 172.21.15.76:0/3488613237 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596400e040 con 0x7f596c106ff0 2024-03-12T05:53:53.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.209+0000 7f59627fc700 1 -- 172.21.15.76:0/3488613237 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5964007850 con 0x7f596c106ff0 2024-03-12T05:53:53.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.209+0000 7f59733b3700 1 -- 172.21.15.76:0/3488613237 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c19d7e0 con 0x7f596c106ff0 2024-03-12T05:53:53.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.209+0000 7f59627fc700 1 -- 172.21.15.76:0/3488613237 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596401b850 con 0x7f596c106ff0 2024-03-12T05:53:53.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.209+0000 7f59733b3700 1 -- 172.21.15.76:0/3488613237 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c1a50a0 con 0x7f596c106ff0 2024-03-12T05:53:53.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.210+0000 7f59733b3700 1 -- 172.21.15.76:0/3488613237 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c10a310 con 0x7f596c106ff0 2024-03-12T05:53:53.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.211+0000 7f59627fc700 1 -- 172.21.15.76:0/3488613237 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f596401da80 con 0x7f596c106ff0 2024-03-12T05:53:53.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.212+0000 7f59627fc700 1 --2- 172.21.15.76:0/3488613237 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f59580a1b50 0x7f59580a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:53.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.212+0000 7f597114f700 1 --2- 172.21.15.76:0/3488613237 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f59580a1b50 0x7f59580a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:53.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.212+0000 7f59627fc700 1 -- 172.21.15.76:0/3488613237 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f59640ceb90 con 0x7f596c106ff0 2024-03-12T05:53:53.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.212+0000 7f597114f700 1 --2- 172.21.15.76:0/3488613237 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f59580a1b50 0x7f59580a4010 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5968007be0 tx=0x7f59680073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:53.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.216+0000 7f59627fc700 1 -- 172.21.15.76:0/3488613237 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5964097e80 con 0x7f596c106ff0 2024-03-12T05:53:53.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.414+0000 7f59733b3700 1 -- 172.21.15.76:0/3488613237 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f596c1996b0 con 0x7f596c106ff0 2024-03-12T05:53:53.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.415+0000 7f59627fc700 1 -- 172.21.15.76:0/3488613237 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22110 (secure 0 0 0) 0x7f596409b540 con 0x7f596c106ff0 2024-03-12T05:53:53.419 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:53.419 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":86,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:53:39.189331+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:53:53.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.418+0000 7f59733b3700 1 -- 172.21.15.76:0/3488613237 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f59580a1b50 msgr2=0x7f59580a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:53.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.418+0000 7f59733b3700 1 --2- 172.21.15.76:0/3488613237 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f59580a1b50 0x7f59580a4010 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5968007be0 tx=0x7f59680073d0 comp rx=0 tx=0).stop 2024-03-12T05:53:53.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.418+0000 7f59733b3700 1 -- 172.21.15.76:0/3488613237 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f596c106ff0 msgr2=0x7f596c19cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:53.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.418+0000 7f59733b3700 1 --2- 172.21.15.76:0/3488613237 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f596c106ff0 0x7f596c19cc20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f596400b070 tx=0x7f5964004e40 comp rx=0 tx=0).stop 2024-03-12T05:53:53.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.418+0000 7f59733b3700 1 -- 172.21.15.76:0/3488613237 shutdown_connections 2024-03-12T05:53:53.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.418+0000 7f59733b3700 1 --2- 172.21.15.76:0/3488613237 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f59580a1b50 0x7f59580a4010 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:53.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.418+0000 7f59733b3700 1 --2- 172.21.15.76:0/3488613237 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f596c106ff0 0x7f596c19cc20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:53.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.418+0000 7f59733b3700 1 --2- 172.21.15.76:0/3488613237 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f596c101f20 0x7f596c198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:53.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.418+0000 7f59733b3700 1 --2- 172.21.15.76:0/3488613237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f596c1028d0 0x7f596c198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:53.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.418+0000 7f59733b3700 1 -- 172.21.15.76:0/3488613237 >> 172.21.15.76:0/3488613237 conn(0x7f596c078dd0 msgr2=0x7f596c0ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:53.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.418+0000 7f59733b3700 1 -- 172.21.15.76:0/3488613237 shutdown_connections 2024-03-12T05:53:53.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:53.418+0000 7f59733b3700 1 -- 172.21.15.76:0/3488613237 wait complete. 2024-03-12T05:53:53.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:53 smithi076 ceph-mon[32095]: pgmap v292: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.8 KiB/s wr, 16 op/s 2024-03-12T05:53:53.988 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:53 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3488613237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:54.070 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-12T05:53:54.071 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T05:53:54.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:53 smithi082 ceph-mon[36382]: pgmap v292: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.8 KiB/s wr, 16 op/s 2024-03-12T05:53:54.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:53 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3488613237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:54.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:53 smithi088 ceph-mon[36585]: pgmap v292: 97 pgs: 97 active+clean; 623 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.8 KiB/s wr, 16 op/s 2024-03-12T05:53:54.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:53 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3488613237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:54.314 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:55.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.909+0000 7f2adec19700 1 -- 172.21.15.76:0/4205526313 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad8105f20 msgr2=0x7f2ad810daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:55.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.909+0000 7f2adec19700 1 --2- 172.21.15.76:0/4205526313 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad8105f20 0x7f2ad810daf0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8009990 tx=0x7f2ac8009ca0 comp rx=0 tx=0).stop 2024-03-12T05:53:55.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.909+0000 7f2adec19700 1 -- 172.21.15.76:0/4205526313 shutdown_connections 2024-03-12T05:53:55.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.909+0000 7f2adec19700 1 --2- 172.21.15.76:0/4205526313 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad8105f20 0x7f2ad810daf0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:55.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.909+0000 7f2adec19700 1 --2- 172.21.15.76:0/4205526313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ad8100e40 0x7f2ad8101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:55.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.909+0000 7f2adec19700 1 --2- 172.21.15.76:0/4205526313 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ad81017f0 0x7f2ad8105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:55.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.909+0000 7f2adec19700 1 -- 172.21.15.76:0/4205526313 >> 172.21.15.76:0/4205526313 conn(0x7f2ad8077ca0 msgr2=0x7f2ad80780b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.909+0000 7f2adec19700 1 -- 172.21.15.76:0/4205526313 shutdown_connections 2024-03-12T05:53:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.909+0000 7f2adec19700 1 -- 172.21.15.76:0/4205526313 wait complete. 2024-03-12T05:53:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.909+0000 7f2adec19700 1 Processor -- start 2024-03-12T05:53:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2adec19700 1 -- start start 2024-03-12T05:53:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2adec19700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ad8100e40 0x7f2ad81971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2adec19700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad81017f0 0x7f2ad8197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2adec19700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ad8105f20 0x7f2ad819bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2adec19700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad819c0c0 con 0x7f2ad81017f0 2024-03-12T05:53:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2adec19700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad819c230 con 0x7f2ad8100e40 2024-03-12T05:53:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2adec19700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad819c3f0 con 0x7f2ad8105f20 2024-03-12T05:53:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2ad7fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad81017f0 0x7f2ad8197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2ad7fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad81017f0 0x7f2ad8197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49116/0 (socket says 172.21.15.76:49116) 2024-03-12T05:53:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2ad7fff700 1 -- 172.21.15.76:0/1431923399 learned_addr learned my addr 172.21.15.76:0/1431923399 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:53:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2add1b6700 1 --2- 172.21.15.76:0/1431923399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ad8105f20 0x7f2ad819bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:55.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2adc9b5700 1 --2- 172.21.15.76:0/1431923399 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ad8100e40 0x7f2ad81971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:55.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2ad7fff700 1 -- 172.21.15.76:0/1431923399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ad8105f20 msgr2=0x7f2ad819bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:55.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2ad7fff700 1 --2- 172.21.15.76:0/1431923399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ad8105f20 0x7f2ad819bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:55.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2ad7fff700 1 -- 172.21.15.76:0/1431923399 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ad8100e40 msgr2=0x7f2ad81971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:55.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2ad7fff700 1 --2- 172.21.15.76:0/1431923399 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ad8100e40 0x7f2ad81971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:55.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2ad7fff700 1 -- 172.21.15.76:0/1431923399 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac80095f0 con 0x7f2ad81017f0 2024-03-12T05:53:55.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.910+0000 7f2ad7fff700 1 --2- 172.21.15.76:0/1431923399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad81017f0 0x7f2ad8197720 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000b500 tx=0x7f2ac000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:55.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.911+0000 7f2ad5ffb700 1 -- 172.21.15.76:0/1431923399 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0011830 con 0x7f2ad81017f0 2024-03-12T05:53:55.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.911+0000 7f2adec19700 1 -- 172.21.15.76:0/1431923399 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad819c6d0 con 0x7f2ad81017f0 2024-03-12T05:53:55.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.911+0000 7f2ad5ffb700 1 -- 172.21.15.76:0/1431923399 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ac0011e70 con 0x7f2ad81017f0 2024-03-12T05:53:55.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.911+0000 7f2ad5ffb700 1 -- 172.21.15.76:0/1431923399 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac000f2e0 con 0x7f2ad81017f0 2024-03-12T05:53:55.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.911+0000 7f2adec19700 1 -- 172.21.15.76:0/1431923399 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad81a3f30 con 0x7f2ad81017f0 2024-03-12T05:53:55.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.912+0000 7f2adec19700 1 -- 172.21.15.76:0/1431923399 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad8109240 con 0x7f2ad81017f0 2024-03-12T05:53:55.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.913+0000 7f2ad5ffb700 1 -- 172.21.15.76:0/1431923399 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f2ac0010610 con 0x7f2ad81017f0 2024-03-12T05:53:55.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.914+0000 7f2ad5ffb700 1 --2- 172.21.15.76:0/1431923399 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2ac40a1b50 0x7f2ac40a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:53:55.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.914+0000 7f2adc9b5700 1 --2- 172.21.15.76:0/1431923399 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2ac40a1b50 0x7f2ac40a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:53:55.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.914+0000 7f2ad5ffb700 1 -- 172.21.15.76:0/1431923399 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2ac00cccf0 con 0x7f2ad81017f0 2024-03-12T05:53:55.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.914+0000 7f2adc9b5700 1 --2- 172.21.15.76:0/1431923399 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2ac40a1b50 0x7f2ac40a4010 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0009f20 tx=0x7f2ad00094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:53:55.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:55.918+0000 7f2ad5ffb700 1 -- 172.21.15.76:0/1431923399 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2ac0096060 con 0x7f2ad81017f0 2024-03-12T05:53:55.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:55 smithi076 ceph-mon[32095]: pgmap v293: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2024-03-12T05:53:56.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:56.111+0000 7f2adec19700 1 -- 172.21.15.76:0/1431923399 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ad8068930 con 0x7f2ad81017f0 2024-03-12T05:53:56.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:56.113+0000 7f2ad5ffb700 1 -- 172.21.15.76:0/1431923399 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22110 (secure 0 0 0) 0x7f2ac0099720 con 0x7f2ad81017f0 2024-03-12T05:53:56.116 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:53:56.116 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":86,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:53:39.189331+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:53:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:56.115+0000 7f2adec19700 1 -- 172.21.15.76:0/1431923399 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2ac40a1b50 msgr2=0x7f2ac40a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:56.115+0000 7f2adec19700 1 --2- 172.21.15.76:0/1431923399 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2ac40a1b50 0x7f2ac40a4010 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0009f20 tx=0x7f2ad00094f0 comp rx=0 tx=0).stop 2024-03-12T05:53:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:56.115+0000 7f2adec19700 1 -- 172.21.15.76:0/1431923399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad81017f0 msgr2=0x7f2ad8197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:53:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:56.115+0000 7f2adec19700 1 --2- 172.21.15.76:0/1431923399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad81017f0 0x7f2ad8197720 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000b500 tx=0x7f2ac000b810 comp rx=0 tx=0).stop 2024-03-12T05:53:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:56.115+0000 7f2adec19700 1 -- 172.21.15.76:0/1431923399 shutdown_connections 2024-03-12T05:53:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:56.115+0000 7f2adec19700 1 --2- 172.21.15.76:0/1431923399 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2ac40a1b50 0x7f2ac40a4010 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:56.115+0000 7f2adec19700 1 --2- 172.21.15.76:0/1431923399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ad8105f20 0x7f2ad819bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:56.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:56.115+0000 7f2adec19700 1 --2- 172.21.15.76:0/1431923399 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2ad8100e40 0x7f2ad81971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:56.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:56.115+0000 7f2adec19700 1 --2- 172.21.15.76:0/1431923399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad81017f0 0x7f2ad8197720 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:53:56.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:56.115+0000 7f2adec19700 1 -- 172.21.15.76:0/1431923399 >> 172.21.15.76:0/1431923399 conn(0x7f2ad8077ca0 msgr2=0x7f2ad80feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:53:56.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:56.115+0000 7f2adec19700 1 -- 172.21.15.76:0/1431923399 shutdown_connections 2024-03-12T05:53:56.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:53:56.115+0000 7f2adec19700 1 -- 172.21.15.76:0/1431923399 wait complete. 2024-03-12T05:53:56.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:55 smithi082 ceph-mon[36382]: pgmap v293: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2024-03-12T05:53:56.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:55 smithi088 ceph-mon[36585]: pgmap v293: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2024-03-12T05:53:56.755 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-12T05:53:56.755 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-12T05:53:56.756 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-12T05:53:56.756 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-12T05:53:56.756 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-12T05:53:56.757 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-12T05:53:56.757 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-12T05:53:56.758 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-12T05:53:56.758 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-12T05:53:56.759 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-12T05:53:56.759 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-12T05:53:56.759 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-12T05:53:57.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:56 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1431923399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:57.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:56 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1431923399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:57.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:56 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1431923399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:53:57.509 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:57.509 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:57.510 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:57.510 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:57.518 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:57.518 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:57.518 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:57.522 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:57.525 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:57.528 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:57.532 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:57.584 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:53:58.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:57 smithi082 ceph-mon[36382]: pgmap v294: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-12T05:53:58.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:57 smithi088 ceph-mon[36585]: pgmap v294: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-12T05:53:58.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:57 smithi076 ceph-mon[32095]: pgmap v294: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-12T05:54:00.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:53:59 smithi082 ceph-mon[36382]: pgmap v295: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-12T05:54:00.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:53:59 smithi088 ceph-mon[36585]: pgmap v295: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-12T05:54:00.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:53:59 smithi076 ceph-mon[32095]: pgmap v295: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-12T05:54:02.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:01 smithi082 ceph-mon[36382]: pgmap v296: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-12T05:54:02.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:01 smithi088 ceph-mon[36585]: pgmap v296: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-12T05:54:02.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:01 smithi076 ceph-mon[32095]: pgmap v296: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-12T05:54:04.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:03 smithi082 ceph-mon[36382]: pgmap v297: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-12T05:54:04.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:03 smithi088 ceph-mon[36585]: pgmap v297: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-12T05:54:04.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:03 smithi076 ceph-mon[32095]: pgmap v297: 97 pgs: 97 active+clean; 635 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-12T05:54:05.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.130+0000 7feddad39700 1 -- 172.21.15.76:0/1165046833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedd4106ff0 msgr2=0x7fedd410eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:05.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.130+0000 7feddad39700 1 --2- 172.21.15.76:0/1165046833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedd4106ff0 0x7fedd410eb70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fedcc009990 tx=0x7fedcc009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:05.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.130+0000 7feddad39700 1 -- 172.21.15.76:0/1165046833 shutdown_connections 2024-03-12T05:54:05.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.130+0000 7feddad39700 1 --2- 172.21.15.76:0/1165046833 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedd4106ff0 0x7fedd410eb70 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:05.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.130+0000 7feddad39700 1 --2- 172.21.15.76:0/1165046833 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fedd406a410 0x7fedd406a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:05.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.130+0000 7feddad39700 1 --2- 172.21.15.76:0/1165046833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedd406adc0 0x7fedd41068c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:05.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.130+0000 7feddad39700 1 -- 172.21.15.76:0/1165046833 >> 172.21.15.76:0/1165046833 conn(0x7fedd4076bc0 msgr2=0x7fedd4076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:05.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.130+0000 7feddad39700 1 -- 172.21.15.76:0/1165046833 shutdown_connections 2024-03-12T05:54:05.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.130+0000 7feddad39700 1 -- 172.21.15.76:0/1165046833 wait complete. 2024-03-12T05:54:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7feddad39700 1 Processor -- start 2024-03-12T05:54:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7feddad39700 1 -- start start 2024-03-12T05:54:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7feddad39700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedd406a410 0x7fedd419a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7feddad39700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedd406adc0 0x7fedd419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7feddad39700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fedd4106ff0 0x7fedd419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7feddad39700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd419f250 con 0x7fedd406a410 2024-03-12T05:54:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7feddad39700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd419f3c0 con 0x7fedd4106ff0 2024-03-12T05:54:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7feddad39700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd419f580 con 0x7fedd406adc0 2024-03-12T05:54:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7fedd8ad5700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedd406a410 0x7fedd419a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7fedd8ad5700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedd406a410 0x7fedd419a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54502/0 (socket says 172.21.15.76:54502) 2024-03-12T05:54:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7fedd8ad5700 1 -- 172.21.15.76:0/1886138608 learned_addr learned my addr 172.21.15.76:0/1886138608 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7fedd3fff700 1 --2- 172.21.15.76:0/1886138608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedd406adc0 0x7fedd419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7fedd92d6700 1 --2- 172.21.15.76:0/1886138608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fedd4106ff0 0x7fedd419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:05.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7fedd8ad5700 1 -- 172.21.15.76:0/1886138608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedd406adc0 msgr2=0x7fedd419a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7fedd8ad5700 1 --2- 172.21.15.76:0/1886138608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedd406adc0 0x7fedd419a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7fedd8ad5700 1 -- 172.21.15.76:0/1886138608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fedd4106ff0 msgr2=0x7fedd419ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7fedd8ad5700 1 --2- 172.21.15.76:0/1886138608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fedd4106ff0 0x7fedd419ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7fedd8ad5700 1 -- 172.21.15.76:0/1886138608 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedcc0095f0 con 0x7fedd406a410 2024-03-12T05:54:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.131+0000 7fedd8ad5700 1 --2- 172.21.15.76:0/1886138608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedd406a410 0x7fedd419a370 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fedc4007e20 tx=0x7fedc400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.132+0000 7fedd1ffb700 1 -- 172.21.15.76:0/1886138608 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc400ac90 con 0x7fedd406a410 2024-03-12T05:54:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.132+0000 7fedd1ffb700 1 -- 172.21.15.76:0/1886138608 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedc400adf0 con 0x7fedd406a410 2024-03-12T05:54:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.132+0000 7feddad39700 1 -- 172.21.15.76:0/1886138608 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd419f860 con 0x7fedd406a410 2024-03-12T05:54:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.132+0000 7fedd1ffb700 1 -- 172.21.15.76:0/1886138608 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc40117a0 con 0x7fedd406a410 2024-03-12T05:54:05.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.132+0000 7feddad39700 1 -- 172.21.15.76:0/1886138608 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd41a70c0 con 0x7fedd406a410 2024-03-12T05:54:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.133+0000 7feddad39700 1 -- 172.21.15.76:0/1886138608 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fedd4112070 con 0x7fedd406a410 2024-03-12T05:54:05.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.134+0000 7fedd1ffb700 1 -- 172.21.15.76:0/1886138608 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fedc4012420 con 0x7fedd406a410 2024-03-12T05:54:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.134+0000 7fedd1ffb700 1 --2- 172.21.15.76:0/1886138608 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fedc00a1b50 0x7fedc00a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.134+0000 7fedd1ffb700 1 -- 172.21.15.76:0/1886138608 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fedc40cd6f0 con 0x7fedd406a410 2024-03-12T05:54:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.134+0000 7fedd3fff700 1 --2- 172.21.15.76:0/1886138608 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fedc00a1b50 0x7fedc00a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.134+0000 7fedd1ffb700 1 --2- 172.21.15.76:0/1886138608 >> [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838] conn(0x7fedc00ab330 0x7fedc00ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.134+0000 7fedd1ffb700 1 -- 172.21.15.76:0/1886138608 --> [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fedc00ade00 con 0x7fedc00ab330 2024-03-12T05:54:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.134+0000 7fedd1ffb700 1 -- 172.21.15.76:0/1886138608 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedc40cdb90 con 0x7fedd406a410 2024-03-12T05:54:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.135+0000 7fedd3fff700 1 --2- 172.21.15.76:0/1886138608 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fedc00a1b50 0x7fedc00a4010 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fedd410e9f0 tx=0x7fedbc009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.135+0000 7fedd92d6700 1 --2- 172.21.15.76:0/1886138608 >> [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838] conn(0x7fedc00ab330 0x7fedc00ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.135+0000 7fedd92d6700 1 --2- 172.21.15.76:0/1886138608 >> [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838] conn(0x7fedc00ab330 0x7fedc00ad750 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:05.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.136+0000 7fedd1ffb700 1 -- 172.21.15.76:0/1886138608 <== osd.2 v2:172.21.15.88:6800/1437946838 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fedc00ade00 con 0x7fedc00ab330 2024-03-12T05:54:05.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.152+0000 7feddad39700 1 -- 172.21.15.76:0/1886138608 --> [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fedd410d830 con 0x7fedc00ab330 2024-03-12T05:54:05.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.153+0000 7fedd1ffb700 1 -- 172.21.15.76:0/1886138608 <== osd.2 v2:172.21.15.88:6800/1437946838 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fedd410d830 con 0x7fedc00ab330 2024-03-12T05:54:05.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.153+0000 7feddad39700 1 -- 172.21.15.76:0/1886138608 >> [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838] conn(0x7fedc00ab330 msgr2=0x7fedc00ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:05.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.153+0000 7feddad39700 1 --2- 172.21.15.76:0/1886138608 >> [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838] conn(0x7fedc00ab330 0x7fedc00ad750 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:05.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.153+0000 7feddad39700 1 -- 172.21.15.76:0/1886138608 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fedc00a1b50 msgr2=0x7fedc00a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:05.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.153+0000 7feddad39700 1 --2- 172.21.15.76:0/1886138608 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fedc00a1b50 0x7fedc00a4010 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fedd410e9f0 tx=0x7fedbc009190 comp rx=0 tx=0).stop 2024-03-12T05:54:05.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.153+0000 7feddad39700 1 -- 172.21.15.76:0/1886138608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedd406a410 msgr2=0x7fedd419a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:05.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.153+0000 7feddad39700 1 --2- 172.21.15.76:0/1886138608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedd406a410 0x7fedd419a370 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fedc4007e20 tx=0x7fedc400a340 comp rx=0 tx=0).stop 2024-03-12T05:54:05.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.154+0000 7fedd92d6700 1 -- 172.21.15.76:0/1886138608 reap_dead start 2024-03-12T05:54:05.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.154+0000 7feddad39700 1 -- 172.21.15.76:0/1886138608 shutdown_connections 2024-03-12T05:54:05.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.154+0000 7feddad39700 1 -- 172.21.15.76:0/1886138608 >> 172.21.15.76:0/1886138608 conn(0x7fedd4076bc0 msgr2=0x7fedd410ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:05.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.154+0000 7feddad39700 1 -- 172.21.15.76:0/1886138608 shutdown_connections 2024-03-12T05:54:05.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:05.154+0000 7feddad39700 1 -- 172.21.15.76:0/1886138608 wait complete. 2024-03-12T05:54:06.067 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:05 smithi076 ceph-mon[32095]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-12T05:54:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:05 smithi088 ceph-mon[36585]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-12T05:54:06.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:05 smithi082 ceph-mon[36382]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-12T05:54:06.463 INFO:teuthology.orchestra.run.smithi076.stdout:77309411390 2024-03-12T05:54:06.463 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-12T05:54:06.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.603+0000 7ff9ee094700 1 -- 172.21.15.76:0/3041387656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9e8100b70 msgr2=0x7ff9e8100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:06.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.603+0000 7ff9ee094700 1 --2- 172.21.15.76:0/3041387656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9e8100b70 0x7ff9e8100ff0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009990 tx=0x7ff9d8009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:06.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.603+0000 7ff9ee094700 1 -- 172.21.15.76:0/3041387656 shutdown_connections 2024-03-12T05:54:06.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.603+0000 7ff9ee094700 1 --2- 172.21.15.76:0/3041387656 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9e8101530 0x7ff9e810d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:06.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.603+0000 7ff9ee094700 1 --2- 172.21.15.76:0/3041387656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9e8106b80 0x7ff9e8106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:06.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.603+0000 7ff9ee094700 1 --2- 172.21.15.76:0/3041387656 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9e8100b70 0x7ff9e8100ff0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:06.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.603+0000 7ff9ee094700 1 -- 172.21.15.76:0/3041387656 >> 172.21.15.76:0/3041387656 conn(0x7ff9e8077aa0 msgr2=0x7ff9e8077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:06.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.603+0000 7ff9ee094700 1 -- 172.21.15.76:0/3041387656 shutdown_connections 2024-03-12T05:54:06.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.603+0000 7ff9ee094700 1 -- 172.21.15.76:0/3041387656 wait complete. 2024-03-12T05:54:06.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.604+0000 7ff9ee094700 1 Processor -- start 2024-03-12T05:54:06.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.604+0000 7ff9ee094700 1 -- start start 2024-03-12T05:54:06.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.604+0000 7ff9ee094700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9e8100b70 0x7ff9e81027b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.604+0000 7ff9ee094700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9e8101530 0x7ff9e8102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.604+0000 7ff9ee094700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9e8106b80 0x7ff9e8103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.604+0000 7ff9ee094700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e81a64c0 con 0x7ff9e8100b70 2024-03-12T05:54:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.604+0000 7ff9ee094700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e81a6630 con 0x7ff9e8101530 2024-03-12T05:54:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.604+0000 7ff9ee094700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e81a67f0 con 0x7ff9e8106b80 2024-03-12T05:54:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.604+0000 7ff9e77fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9e8100b70 0x7ff9e81027b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.604+0000 7ff9e77fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9e8100b70 0x7ff9e81027b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54524/0 (socket says 172.21.15.76:54524) 2024-03-12T05:54:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.604+0000 7ff9e77fe700 1 -- 172.21.15.76:0/3558857563 learned_addr learned my addr 172.21.15.76:0/3558857563 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9e6ffd700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9e8101530 0x7ff9e8102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9e7fff700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9e8106b80 0x7ff9e8103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9e77fe700 1 -- 172.21.15.76:0/3558857563 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9e8106b80 msgr2=0x7ff9e8103230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9e77fe700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9e8106b80 0x7ff9e8103230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9e77fe700 1 -- 172.21.15.76:0/3558857563 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9e8101530 msgr2=0x7ff9e8102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9e77fe700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9e8101530 0x7ff9e8102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:06.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9e77fe700 1 -- 172.21.15.76:0/3558857563 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d80095f0 con 0x7ff9e8100b70 2024-03-12T05:54:06.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9e7fff700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9e8106b80 0x7ff9e8103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:54:06.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9e6ffd700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9e8101530 0x7ff9e8102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:54:06.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9e77fe700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9e8100b70 0x7ff9e81027b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00c920 tx=0x7ff9dc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:06.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9e4ff9700 1 -- 172.21.15.76:0/3558857563 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc007b00 con 0x7ff9e8100b70 2024-03-12T05:54:06.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9ee094700 1 -- 172.21.15.76:0/3558857563 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e81a6ad0 con 0x7ff9e8100b70 2024-03-12T05:54:06.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9e4ff9700 1 -- 172.21.15.76:0/3558857563 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9dc004d10 con 0x7ff9e8100b70 2024-03-12T05:54:06.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9e4ff9700 1 -- 172.21.15.76:0/3558857563 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc0056c0 con 0x7ff9e8100b70 2024-03-12T05:54:06.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.605+0000 7ff9ee094700 1 -- 172.21.15.76:0/3558857563 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e81a7020 con 0x7ff9e8100b70 2024-03-12T05:54:06.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.606+0000 7ff9ee094700 1 -- 172.21.15.76:0/3558857563 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff9e8110e90 con 0x7ff9e8100b70 2024-03-12T05:54:06.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.607+0000 7ff9e4ff9700 1 -- 172.21.15.76:0/3558857563 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7ff9dc007c60 con 0x7ff9e8100b70 2024-03-12T05:54:06.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.608+0000 7ff9e4ff9700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff9d00a1bd0 0x7ff9d00a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:06.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.608+0000 7ff9e6ffd700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff9d00a1bd0 0x7ff9d00a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:06.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.608+0000 7ff9e4ff9700 1 -- 172.21.15.76:0/3558857563 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff9dc0cd350 con 0x7ff9e8100b70 2024-03-12T05:54:06.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.608+0000 7ff9e4ff9700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649] conn(0x7ff9d00ab3b0 0x7ff9d00ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:06.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.608+0000 7ff9e4ff9700 1 -- 172.21.15.76:0/3558857563 --> [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9d00ade80 con 0x7ff9d00ab3b0 2024-03-12T05:54:06.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.608+0000 7ff9e4ff9700 1 -- 172.21.15.76:0/3558857563 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9dc0cd7f0 con 0x7ff9e8100b70 2024-03-12T05:54:06.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.608+0000 7ff9e6ffd700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff9d00a1bd0 0x7ff9d00a4090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8006010 tx=0x7ff9d801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:06.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.608+0000 7ff9e7fff700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649] conn(0x7ff9d00ab3b0 0x7ff9d00ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:06.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.609+0000 7ff9e7fff700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649] conn(0x7ff9d00ab3b0 0x7ff9d00ad7d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:06.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.610+0000 7ff9e4ff9700 1 -- 172.21.15.76:0/3558857563 <== osd.11 v2:172.21.15.88:6824/3057841649 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff9d00ade80 con 0x7ff9d00ab3b0 2024-03-12T05:54:06.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.626+0000 7ff9ee094700 1 -- 172.21.15.76:0/3558857563 --> [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff9e8104b20 con 0x7ff9d00ab3b0 2024-03-12T05:54:06.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.626+0000 7ff9e4ff9700 1 -- 172.21.15.76:0/3558857563 <== osd.11 v2:172.21.15.88:6824/3057841649 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff9e8104b20 con 0x7ff9d00ab3b0 2024-03-12T05:54:06.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.626+0000 7ff9ee094700 1 -- 172.21.15.76:0/3558857563 >> [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649] conn(0x7ff9d00ab3b0 msgr2=0x7ff9d00ad7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:06.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.626+0000 7ff9ee094700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649] conn(0x7ff9d00ab3b0 0x7ff9d00ad7d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:06.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.627+0000 7ff9ee094700 1 -- 172.21.15.76:0/3558857563 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff9d00a1bd0 msgr2=0x7ff9d00a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:06.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.627+0000 7ff9ee094700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff9d00a1bd0 0x7ff9d00a4090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8006010 tx=0x7ff9d801a040 comp rx=0 tx=0).stop 2024-03-12T05:54:06.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.627+0000 7ff9ee094700 1 -- 172.21.15.76:0/3558857563 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9e8100b70 msgr2=0x7ff9e81027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:06.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.627+0000 7ff9ee094700 1 --2- 172.21.15.76:0/3558857563 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9e8100b70 0x7ff9e81027b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00c920 tx=0x7ff9dc00cce0 comp rx=0 tx=0).stop 2024-03-12T05:54:06.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.627+0000 7ff9e7fff700 1 -- 172.21.15.76:0/3558857563 reap_dead start 2024-03-12T05:54:06.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.627+0000 7ff9ee094700 1 -- 172.21.15.76:0/3558857563 shutdown_connections 2024-03-12T05:54:06.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.627+0000 7ff9ee094700 1 -- 172.21.15.76:0/3558857563 >> 172.21.15.76:0/3558857563 conn(0x7ff9e8077aa0 msgr2=0x7ff9e80ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:06.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.627+0000 7ff9ee094700 1 -- 172.21.15.76:0/3558857563 shutdown_connections 2024-03-12T05:54:06.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:06.627+0000 7ff9ee094700 1 -- 172.21.15.76:0/3558857563 wait complete. 2024-03-12T05:54:07.941 INFO:teuthology.orchestra.run.smithi076.stdout:347892350995 2024-03-12T05:54:07.941 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-12T05:54:08.172 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:07 smithi076 ceph-mon[32095]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-12T05:54:08.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:07 smithi088 ceph-mon[36585]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-12T05:54:08.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:07 smithi082 ceph-mon[36382]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-12T05:54:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:09 smithi088 ceph-mon[36585]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-12T05:54:10.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:09 smithi076 ceph-mon[32095]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-12T05:54:10.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:09 smithi082 ceph-mon[36382]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-12T05:54:12.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.022+0000 7fdd28b03700 1 -- 172.21.15.76:0/1949318481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd24106d60 msgr2=0x7fdd24107140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:12.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.022+0000 7fdd28b03700 1 --2- 172.21.15.76:0/1949318481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd24106d60 0x7fdd24107140 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009990 tx=0x7fdd18009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.022+0000 7fdd28b03700 1 -- 172.21.15.76:0/1949318481 shutdown_connections 2024-03-12T05:54:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.022+0000 7fdd28b03700 1 --2- 172.21.15.76:0/1949318481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdd24101810 0x7fdd2410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.022+0000 7fdd28b03700 1 --2- 172.21.15.76:0/1949318481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd24106d60 0x7fdd24107140 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.022+0000 7fdd28b03700 1 --2- 172.21.15.76:0/1949318481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd24100e50 0x7fdd241012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.022+0000 7fdd28b03700 1 -- 172.21.15.76:0/1949318481 >> 172.21.15.76:0/1949318481 conn(0x7fdd240779d0 msgr2=0x7fdd24077de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.022+0000 7fdd28b03700 1 -- 172.21.15.76:0/1949318481 shutdown_connections 2024-03-12T05:54:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.022+0000 7fdd28b03700 1 -- 172.21.15.76:0/1949318481 wait complete. 2024-03-12T05:54:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.023+0000 7fdd28b03700 1 Processor -- start 2024-03-12T05:54:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.023+0000 7fdd28b03700 1 -- start start 2024-03-12T05:54:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.023+0000 7fdd28b03700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd24100e50 0x7fdd24197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.023+0000 7fdd28b03700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd24101810 0x7fdd24197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:12.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.023+0000 7fdd28b03700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdd24106d60 0x7fdd2419bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.023+0000 7fdd28b03700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2419c0c0 con 0x7fdd24101810 2024-03-12T05:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.023+0000 7fdd28b03700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2419c230 con 0x7fdd24106d60 2024-03-12T05:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.023+0000 7fdd28b03700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2419c3f0 con 0x7fdd24100e50 2024-03-12T05:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd21d9b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd24101810 0x7fdd24197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd21d9b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd24101810 0x7fdd24197770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43526/0 (socket says 172.21.15.76:43526) 2024-03-12T05:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd21d9b700 1 -- 172.21.15.76:0/1208298699 learned_addr learned my addr 172.21.15.76:0/1208298699 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd2259c700 1 --2- 172.21.15.76:0/1208298699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd24100e50 0x7fdd24197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd21d9b700 1 -- 172.21.15.76:0/1208298699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd24100e50 msgr2=0x7fdd24197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd21d9b700 1 --2- 172.21.15.76:0/1208298699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd24100e50 0x7fdd24197230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd22d9d700 1 --2- 172.21.15.76:0/1208298699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdd24106d60 0x7fdd2419bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd21d9b700 1 -- 172.21.15.76:0/1208298699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdd24106d60 msgr2=0x7fdd2419bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd21d9b700 1 --2- 172.21.15.76:0/1208298699 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdd24106d60 0x7fdd2419bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd21d9b700 1 -- 172.21.15.76:0/1208298699 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd180095f0 con 0x7fdd24101810 2024-03-12T05:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd21d9b700 1 --2- 172.21.15.76:0/1208298699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd24101810 0x7fdd24197770 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400d950 tx=0x7fdd1400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd137fe700 1 -- 172.21.15.76:0/1208298699 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd14009940 con 0x7fdd24101810 2024-03-12T05:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd28b03700 1 -- 172.21.15.76:0/1208298699 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2419c6d0 con 0x7fdd24101810 2024-03-12T05:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd137fe700 1 -- 172.21.15.76:0/1208298699 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd14010460 con 0x7fdd24101810 2024-03-12T05:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd137fe700 1 -- 172.21.15.76:0/1208298699 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1400f620 con 0x7fdd24101810 2024-03-12T05:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.024+0000 7fdd28b03700 1 -- 172.21.15.76:0/1208298699 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd241a3f30 con 0x7fdd24101810 2024-03-12T05:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.025+0000 7fdd28b03700 1 -- 172.21.15.76:0/1208298699 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdd24110f50 con 0x7fdd24101810 2024-03-12T05:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.026+0000 7fdd137fe700 1 -- 172.21.15.76:0/1208298699 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fdd140105d0 con 0x7fdd24101810 2024-03-12T05:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.027+0000 7fdd137fe700 1 --2- 172.21.15.76:0/1208298699 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdd0c0a1b50 0x7fdd0c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.027+0000 7fdd2259c700 1 --2- 172.21.15.76:0/1208298699 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdd0c0a1b50 0x7fdd0c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.027+0000 7fdd137fe700 1 -- 172.21.15.76:0/1208298699 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fdd140cd7c0 con 0x7fdd24101810 2024-03-12T05:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.027+0000 7fdd137fe700 1 --2- 172.21.15.76:0/1208298699 >> [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805] conn(0x7fdd0c0ab330 0x7fdd0c0ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.027+0000 7fdd137fe700 1 -- 172.21.15.76:0/1208298699 --> [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd0c0ade00 con 0x7fdd0c0ab330 2024-03-12T05:54:12.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.027+0000 7fdd137fe700 1 -- 172.21.15.76:0/1208298699 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd140cdc60 con 0x7fdd24101810 2024-03-12T05:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.027+0000 7fdd2259c700 1 --2- 172.21.15.76:0/1208298699 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdd0c0a1b50 0x7fdd0c0a4010 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800b5c0 tx=0x7fdd18005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.027+0000 7fdd22d9d700 1 --2- 172.21.15.76:0/1208298699 >> [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805] conn(0x7fdd0c0ab330 0x7fdd0c0ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.028+0000 7fdd22d9d700 1 --2- 172.21.15.76:0/1208298699 >> [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805] conn(0x7fdd0c0ab330 0x7fdd0c0ad750 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.028+0000 7fdd137fe700 1 -- 172.21.15.76:0/1208298699 <== osd.0 v2:172.21.15.76:6802/1797183805 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdd0c0ade00 con 0x7fdd0c0ab330 2024-03-12T05:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.044+0000 7fdd28b03700 1 -- 172.21.15.76:0/1208298699 --> [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdd24062300 con 0x7fdd0c0ab330 2024-03-12T05:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.044+0000 7fdd137fe700 1 -- 172.21.15.76:0/1208298699 <== osd.0 v2:172.21.15.76:6802/1797183805 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdd24062300 con 0x7fdd0c0ab330 2024-03-12T05:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.045+0000 7fdd28b03700 1 -- 172.21.15.76:0/1208298699 >> [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805] conn(0x7fdd0c0ab330 msgr2=0x7fdd0c0ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.045+0000 7fdd28b03700 1 --2- 172.21.15.76:0/1208298699 >> [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805] conn(0x7fdd0c0ab330 0x7fdd0c0ad750 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.045+0000 7fdd28b03700 1 -- 172.21.15.76:0/1208298699 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdd0c0a1b50 msgr2=0x7fdd0c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.045+0000 7fdd28b03700 1 --2- 172.21.15.76:0/1208298699 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdd0c0a1b50 0x7fdd0c0a4010 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800b5c0 tx=0x7fdd18005960 comp rx=0 tx=0).stop 2024-03-12T05:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.045+0000 7fdd28b03700 1 -- 172.21.15.76:0/1208298699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd24101810 msgr2=0x7fdd24197770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.045+0000 7fdd28b03700 1 --2- 172.21.15.76:0/1208298699 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd24101810 0x7fdd24197770 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400d950 tx=0x7fdd1400dc60 comp rx=0 tx=0).stop 2024-03-12T05:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.045+0000 7fdd22d9d700 1 -- 172.21.15.76:0/1208298699 reap_dead start 2024-03-12T05:54:12.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.045+0000 7fdd28b03700 1 -- 172.21.15.76:0/1208298699 shutdown_connections 2024-03-12T05:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.045+0000 7fdd28b03700 1 -- 172.21.15.76:0/1208298699 >> 172.21.15.76:0/1208298699 conn(0x7fdd240779d0 msgr2=0x7fdd240ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.045+0000 7fdd28b03700 1 -- 172.21.15.76:0/1208298699 shutdown_connections 2024-03-12T05:54:12.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.045+0000 7fdd28b03700 1 -- 172.21.15.76:0/1208298699 wait complete. 2024-03-12T05:54:12.061 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:11 smithi076 ceph-mon[32095]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-12T05:54:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:11 smithi088 ceph-mon[36585]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-12T05:54:12.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:11 smithi082 ceph-mon[36382]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-12T05:54:12.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.889+0000 7f25f9d4f700 1 -- 172.21.15.76:0/2682701757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25f406a2c0 msgr2=0x7f25f410eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:12.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.889+0000 7f25f9d4f700 1 --2- 172.21.15.76:0/2682701757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25f406a2c0 0x7f25f410eb00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f25ec00b0d0 tx=0x7f25ec00b3e0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.889+0000 7f25f9d4f700 1 -- 172.21.15.76:0/2682701757 shutdown_connections 2024-03-12T05:54:12.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.889+0000 7f25f9d4f700 1 --2- 172.21.15.76:0/2682701757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25f406a2c0 0x7f25f410eb00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.889+0000 7f25f9d4f700 1 --2- 172.21.15.76:0/2682701757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25f4107cc0 0x7f25f41080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.889+0000 7f25f9d4f700 1 --2- 172.21.15.76:0/2682701757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25f4069920 0x7f25f4069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.889+0000 7f25f9d4f700 1 -- 172.21.15.76:0/2682701757 >> 172.21.15.76:0/2682701757 conn(0x7f25f4077aa0 msgr2=0x7f25f4077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:12.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.889+0000 7f25f9d4f700 1 -- 172.21.15.76:0/2682701757 shutdown_connections 2024-03-12T05:54:12.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.889+0000 7f25f9d4f700 1 -- 172.21.15.76:0/2682701757 wait complete. 2024-03-12T05:54:12.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f9d4f700 1 Processor -- start 2024-03-12T05:54:12.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f9d4f700 1 -- start start 2024-03-12T05:54:12.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f9d4f700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25f4069920 0x7f25f4100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:12.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f9d4f700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25f406a2c0 0x7f25f4101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:12.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f9d4f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25f4107cc0 0x7f25f41016c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:12.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f9d4f700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f41a6640 con 0x7f25f406a2c0 2024-03-12T05:54:12.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f9d4f700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f41a67b0 con 0x7f25f4069920 2024-03-12T05:54:12.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f9d4f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f41a6970 con 0x7f25f4107cc0 2024-03-12T05:54:12.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f2ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25f406a2c0 0x7f25f4101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:12.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f2ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25f406a2c0 0x7f25f4101180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43536/0 (socket says 172.21.15.76:43536) 2024-03-12T05:54:12.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f2ffd700 1 -- 172.21.15.76:0/3068207488 learned_addr learned my addr 172.21.15.76:0/3068207488 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:12.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f37fe700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25f4069920 0x7f25f4100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:12.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f3fff700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25f4107cc0 0x7f25f41016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:12.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f2ffd700 1 -- 172.21.15.76:0/3068207488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25f4107cc0 msgr2=0x7f25f41016c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:12.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f2ffd700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25f4107cc0 0x7f25f41016c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f2ffd700 1 -- 172.21.15.76:0/3068207488 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25f4069920 msgr2=0x7f25f4100c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:12.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f2ffd700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25f4069920 0x7f25f4100c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f2ffd700 1 -- 172.21.15.76:0/3068207488 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec009d00 con 0x7f25f406a2c0 2024-03-12T05:54:12.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f3fff700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25f4107cc0 0x7f25f41016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:54:12.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.890+0000 7f25f37fe700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f25f4069920 0x7f25f4100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:54:12.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.891+0000 7f25f2ffd700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25f406a2c0 0x7f25f4101180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f25e800b530 tx=0x7f25e800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:12.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.891+0000 7f25f0ff9700 1 -- 172.21.15.76:0/3068207488 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8011830 con 0x7f25f406a2c0 2024-03-12T05:54:12.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.891+0000 7f25f9d4f700 1 -- 172.21.15.76:0/3068207488 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f41a6c50 con 0x7f25f406a2c0 2024-03-12T05:54:12.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.891+0000 7f25f0ff9700 1 -- 172.21.15.76:0/3068207488 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25e8011e70 con 0x7f25f406a2c0 2024-03-12T05:54:12.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.891+0000 7f25f0ff9700 1 -- 172.21.15.76:0/3068207488 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e800f2e0 con 0x7f25f406a2c0 2024-03-12T05:54:12.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.891+0000 7f25f9d4f700 1 -- 172.21.15.76:0/3068207488 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f41a71a0 con 0x7f25f406a2c0 2024-03-12T05:54:12.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.892+0000 7f25f9d4f700 1 -- 172.21.15.76:0/3068207488 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f25f4111fd0 con 0x7f25f406a2c0 2024-03-12T05:54:12.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.893+0000 7f25f0ff9700 1 -- 172.21.15.76:0/3068207488 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f25e80105f0 con 0x7f25f406a2c0 2024-03-12T05:54:12.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.893+0000 7f25f0ff9700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f25dc0a1b50 0x7f25dc0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:12.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.894+0000 7f25f37fe700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f25dc0a1b50 0x7f25dc0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:12.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.894+0000 7f25f0ff9700 1 -- 172.21.15.76:0/3068207488 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f25e80cd350 con 0x7f25f406a2c0 2024-03-12T05:54:12.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.894+0000 7f25f0ff9700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522] conn(0x7f25dc0ab330 0x7f25dc0ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:12.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.894+0000 7f25f0ff9700 1 -- 172.21.15.76:0/3068207488 --> [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25dc0ade00 con 0x7f25dc0ab330 2024-03-12T05:54:12.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.894+0000 7f25f0ff9700 1 -- 172.21.15.76:0/3068207488 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25e80cd7f0 con 0x7f25f406a2c0 2024-03-12T05:54:12.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.894+0000 7f25f37fe700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f25dc0a1b50 0x7f25dc0a4010 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f25e4005fd0 tx=0x7f25e4005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:12.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.894+0000 7f25f3fff700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522] conn(0x7f25dc0ab330 0x7f25dc0ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:12.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.895+0000 7f25f3fff700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522] conn(0x7f25dc0ab330 0x7f25dc0ad750 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:12.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.895+0000 7f25f0ff9700 1 -- 172.21.15.76:0/3068207488 <== osd.7 v2:172.21.15.82:6816/88004522 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f25dc0ade00 con 0x7f25dc0ab330 2024-03-12T05:54:12.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.912+0000 7f25f9d4f700 1 -- 172.21.15.76:0/3068207488 --> [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f25f4062330 con 0x7f25dc0ab330 2024-03-12T05:54:12.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.912+0000 7f25f0ff9700 1 -- 172.21.15.76:0/3068207488 <== osd.7 v2:172.21.15.82:6816/88004522 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f25f4062330 con 0x7f25dc0ab330 2024-03-12T05:54:12.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.912+0000 7f25f9d4f700 1 -- 172.21.15.76:0/3068207488 >> [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522] conn(0x7f25dc0ab330 msgr2=0x7f25dc0ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:12.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.913+0000 7f25f9d4f700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522] conn(0x7f25dc0ab330 0x7f25dc0ad750 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.913+0000 7f25f9d4f700 1 -- 172.21.15.76:0/3068207488 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f25dc0a1b50 msgr2=0x7f25dc0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:12.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.913+0000 7f25f9d4f700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f25dc0a1b50 0x7f25dc0a4010 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f25e4005fd0 tx=0x7f25e4005e30 comp rx=0 tx=0).stop 2024-03-12T05:54:12.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.913+0000 7f25f9d4f700 1 -- 172.21.15.76:0/3068207488 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25f406a2c0 msgr2=0x7f25f4101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:12.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.913+0000 7f25f9d4f700 1 --2- 172.21.15.76:0/3068207488 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f25f406a2c0 0x7f25f4101180 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f25e800b530 tx=0x7f25e800b8f0 comp rx=0 tx=0).stop 2024-03-12T05:54:12.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.913+0000 7f25f3fff700 1 -- 172.21.15.76:0/3068207488 reap_dead start 2024-03-12T05:54:12.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.913+0000 7f25f9d4f700 1 -- 172.21.15.76:0/3068207488 shutdown_connections 2024-03-12T05:54:12.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.913+0000 7f25f9d4f700 1 -- 172.21.15.76:0/3068207488 >> 172.21.15.76:0/3068207488 conn(0x7f25f4077aa0 msgr2=0x7f25f40ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:12.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.913+0000 7f25f9d4f700 1 -- 172.21.15.76:0/3068207488 shutdown_connections 2024-03-12T05:54:12.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:12.913+0000 7f25f9d4f700 1 -- 172.21.15.76:0/3068207488 wait complete. 2024-03-12T05:54:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.139+0000 7fb6f4192700 1 -- 172.21.15.76:0/1416593585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6ec100260 msgr2=0x7fb6ec1006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.139+0000 7fb6f4192700 1 --2- 172.21.15.76:0/1416593585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6ec100260 0x7fb6ec1006e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009990 tx=0x7fb6dc009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 -- 172.21.15.76:0/1416593585 shutdown_connections 2024-03-12T05:54:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 --2- 172.21.15.76:0/1416593585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6ec100c20 0x7fb6ec10eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 --2- 172.21.15.76:0/1416593585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6ec106ce0 0x7fb6ec1070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 --2- 172.21.15.76:0/1416593585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6ec100260 0x7fb6ec1006e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 -- 172.21.15.76:0/1416593585 >> 172.21.15.76:0/1416593585 conn(0x7fb6ec077170 msgr2=0x7fb6ec0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 -- 172.21.15.76:0/1416593585 shutdown_connections 2024-03-12T05:54:13.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 -- 172.21.15.76:0/1416593585 wait complete. 2024-03-12T05:54:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 Processor -- start 2024-03-12T05:54:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 -- start start 2024-03-12T05:54:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6ec100260 0x7fb6ec101440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6ec100c20 0x7fb6ec101980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6ec106ce0 0x7fb6ec101ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec1a67f0 con 0x7fb6ec100c20 2024-03-12T05:54:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec1a6960 con 0x7fb6ec106ce0 2024-03-12T05:54:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.140+0000 7fb6f4192700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ec1a6b20 con 0x7fb6ec100260 2024-03-12T05:54:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6f172d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6ec100c20 0x7fb6ec101980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6f172d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6ec100c20 0x7fb6ec101980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43548/0 (socket says 172.21.15.76:43548) 2024-03-12T05:54:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6f172d700 1 -- 172.21.15.76:0/4200072811 learned_addr learned my addr 172.21.15.76:0/4200072811 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6f1f2e700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6ec100260 0x7fb6ec101440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6f172d700 1 -- 172.21.15.76:0/4200072811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6ec100260 msgr2=0x7fb6ec101440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:13.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6f172d700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6ec100260 0x7fb6ec101440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6f272f700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6ec106ce0 0x7fb6ec101ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6f172d700 1 -- 172.21.15.76:0/4200072811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6ec106ce0 msgr2=0x7fb6ec101ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6f172d700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6ec106ce0 0x7fb6ec101ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6f172d700 1 -- 172.21.15.76:0/4200072811 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc0095f0 con 0x7fb6ec100c20 2024-03-12T05:54:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6f1f2e700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6ec100260 0x7fb6ec101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:54:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6f172d700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6ec100c20 0x7fb6ec101980 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc000c00 tx=0x7fb6dc00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6e2ffd700 1 -- 172.21.15.76:0/4200072811 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc01d070 con 0x7fb6ec100c20 2024-03-12T05:54:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6e2ffd700 1 -- 172.21.15.76:0/4200072811 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6dc00fc30 con 0x7fb6ec100c20 2024-03-12T05:54:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6e2ffd700 1 -- 172.21.15.76:0/4200072811 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc0176e0 con 0x7fb6ec100c20 2024-03-12T05:54:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6f4192700 1 -- 172.21.15.76:0/4200072811 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ec1a6da0 con 0x7fb6ec100c20 2024-03-12T05:54:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.141+0000 7fb6f4192700 1 -- 172.21.15.76:0/4200072811 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ec1a7160 con 0x7fb6ec100c20 2024-03-12T05:54:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.142+0000 7fb6f4192700 1 -- 172.21.15.76:0/4200072811 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb6ec1121e0 con 0x7fb6ec100c20 2024-03-12T05:54:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.143+0000 7fb6e2ffd700 1 -- 172.21.15.76:0/4200072811 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fb6dc0229e0 con 0x7fb6ec100c20 2024-03-12T05:54:13.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.144+0000 7fb6e2ffd700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb6d80a1bd0 0x7fb6d80a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.144+0000 7fb6f1f2e700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb6d80a1bd0 0x7fb6d80a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.144+0000 7fb6e2ffd700 1 -- 172.21.15.76:0/4200072811 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb6dc0cee50 con 0x7fb6ec100c20 2024-03-12T05:54:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.144+0000 7fb6e2ffd700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247] conn(0x7fb6d80ab3b0 0x7fb6d80ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.144+0000 7fb6e2ffd700 1 -- 172.21.15.76:0/4200072811 --> [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6d80ade80 con 0x7fb6d80ab3b0 2024-03-12T05:54:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.144+0000 7fb6e2ffd700 1 -- 172.21.15.76:0/4200072811 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6dc0cf2f0 con 0x7fb6ec100c20 2024-03-12T05:54:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.144+0000 7fb6f1f2e700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb6d80a1bd0 0x7fb6d80a4090 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8007900 tx=0x7fb6e8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.144+0000 7fb6f272f700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247] conn(0x7fb6d80ab3b0 0x7fb6d80ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.145+0000 7fb6f272f700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247] conn(0x7fb6d80ab3b0 0x7fb6d80ad7d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:13.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.146+0000 7fb6e2ffd700 1 -- 172.21.15.76:0/4200072811 <== osd.8 v2:172.21.15.88:6816/529149247 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb6d80ade80 con 0x7fb6d80ab3b0 2024-03-12T05:54:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.162+0000 7fb6f4192700 1 -- 172.21.15.76:0/4200072811 --> [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb6ec062300 con 0x7fb6d80ab3b0 2024-03-12T05:54:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.163+0000 7fb6e2ffd700 1 -- 172.21.15.76:0/4200072811 <== osd.8 v2:172.21.15.88:6816/529149247 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb6ec062300 con 0x7fb6d80ab3b0 2024-03-12T05:54:13.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.163+0000 7fb6f4192700 1 -- 172.21.15.76:0/4200072811 >> [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247] conn(0x7fb6d80ab3b0 msgr2=0x7fb6d80ad7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:13.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.163+0000 7fb6f4192700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247] conn(0x7fb6d80ab3b0 0x7fb6d80ad7d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:13.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.163+0000 7fb6f4192700 1 -- 172.21.15.76:0/4200072811 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb6d80a1bd0 msgr2=0x7fb6d80a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:13.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.163+0000 7fb6f4192700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb6d80a1bd0 0x7fb6d80a4090 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8007900 tx=0x7fb6e8008040 comp rx=0 tx=0).stop 2024-03-12T05:54:13.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.163+0000 7fb6f4192700 1 -- 172.21.15.76:0/4200072811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6ec100c20 msgr2=0x7fb6ec101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:13.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.163+0000 7fb6f4192700 1 --2- 172.21.15.76:0/4200072811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6ec100c20 0x7fb6ec101980 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc000c00 tx=0x7fb6dc00f740 comp rx=0 tx=0).stop 2024-03-12T05:54:13.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.163+0000 7fb6f272f700 1 -- 172.21.15.76:0/4200072811 reap_dead start 2024-03-12T05:54:13.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.163+0000 7fb6f4192700 1 -- 172.21.15.76:0/4200072811 shutdown_connections 2024-03-12T05:54:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.163+0000 7fb6f4192700 1 -- 172.21.15.76:0/4200072811 >> 172.21.15.76:0/4200072811 conn(0x7fb6ec077170 msgr2=0x7fb6ec104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.163+0000 7fb6f4192700 1 -- 172.21.15.76:0/4200072811 shutdown_connections 2024-03-12T05:54:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.164+0000 7fb6f4192700 1 -- 172.21.15.76:0/4200072811 wait complete. 2024-03-12T05:54:13.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.352+0000 7fa6616f4700 1 -- 172.21.15.76:0/3932663711 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa65c1011c0 msgr2=0x7fa65c106900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:13.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.352+0000 7fa6616f4700 1 --2- 172.21.15.76:0/3932663711 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa65c1011c0 0x7fa65c106900 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa644009990 tx=0x7fa644009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:13.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.352+0000 7fa6616f4700 1 -- 172.21.15.76:0/3932663711 shutdown_connections 2024-03-12T05:54:13.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.352+0000 7fa6616f4700 1 --2- 172.21.15.76:0/3932663711 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa65c107030 0x7fa65c10ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:13.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.352+0000 7fa6616f4700 1 --2- 172.21.15.76:0/3932663711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa65c100810 0x7fa65c100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:13.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.352+0000 7fa6616f4700 1 --2- 172.21.15.76:0/3932663711 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa65c1011c0 0x7fa65c106900 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:13.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.352+0000 7fa6616f4700 1 -- 172.21.15.76:0/3932663711 >> 172.21.15.76:0/3932663711 conn(0x7fa65c075eb0 msgr2=0x7fa65c0762c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:13.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.352+0000 7fa6616f4700 1 -- 172.21.15.76:0/3932663711 shutdown_connections 2024-03-12T05:54:13.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.352+0000 7fa6616f4700 1 -- 172.21.15.76:0/3932663711 wait complete. 2024-03-12T05:54:13.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.352+0000 7fa6616f4700 1 Processor -- start 2024-03-12T05:54:13.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.352+0000 7fa6616f4700 1 -- start start 2024-03-12T05:54:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa6616f4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa65c100810 0x7fa65c198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa6616f4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa65c1011c0 0x7fa65c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa6616f4700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa65c107030 0x7fa65c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa6616f4700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c19d030 con 0x7fa65c107030 2024-03-12T05:54:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa6616f4700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c19d1a0 con 0x7fa65c1011c0 2024-03-12T05:54:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa6616f4700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c19d360 con 0x7fa65c100810 2024-03-12T05:54:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa65b7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa65c107030 0x7fa65c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa65b7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa65c107030 0x7fa65c19ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43570/0 (socket says 172.21.15.76:43570) 2024-03-12T05:54:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa65b7fe700 1 -- 172.21.15.76:0/837577597 learned_addr learned my addr 172.21.15.76:0/837577597 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa65affd700 1 --2- 172.21.15.76:0/837577597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa65c100810 0x7fa65c198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa65a7fc700 1 --2- 172.21.15.76:0/837577597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa65c1011c0 0x7fa65c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa65b7fe700 1 -- 172.21.15.76:0/837577597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa65c100810 msgr2=0x7fa65c198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa65b7fe700 1 --2- 172.21.15.76:0/837577597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa65c100810 0x7fa65c198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa65b7fe700 1 -- 172.21.15.76:0/837577597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa65c1011c0 msgr2=0x7fa65c198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa65b7fe700 1 --2- 172.21.15.76:0/837577597 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa65c1011c0 0x7fa65c198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa65b7fe700 1 -- 172.21.15.76:0/837577597 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6440095f0 con 0x7fa65c107030 2024-03-12T05:54:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa65b7fe700 1 --2- 172.21.15.76:0/837577597 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa65c107030 0x7fa65c19ca20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa65400c900 tx=0x7fa65400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa653fff700 1 -- 172.21.15.76:0/837577597 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6540078d0 con 0x7fa65c107030 2024-03-12T05:54:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa653fff700 1 -- 172.21.15.76:0/837577597 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa65400f450 con 0x7fa65c107030 2024-03-12T05:54:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa653fff700 1 -- 172.21.15.76:0/837577597 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa654018660 con 0x7fa65c107030 2024-03-12T05:54:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa6616f4700 1 -- 172.21.15.76:0/837577597 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65c19d640 con 0x7fa65c107030 2024-03-12T05:54:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.353+0000 7fa6616f4700 1 -- 172.21.15.76:0/837577597 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65c1a4ea0 con 0x7fa65c107030 2024-03-12T05:54:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.354+0000 7fa6616f4700 1 -- 172.21.15.76:0/837577597 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa65c111f90 con 0x7fa65c107030 2024-03-12T05:54:13.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.355+0000 7fa653fff700 1 -- 172.21.15.76:0/837577597 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fa65400f5c0 con 0x7fa65c107030 2024-03-12T05:54:13.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.356+0000 7fa653fff700 1 --2- 172.21.15.76:0/837577597 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa6480a1b50 0x7fa6480a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.356+0000 7fa65affd700 1 --2- 172.21.15.76:0/837577597 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa6480a1b50 0x7fa6480a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.356+0000 7fa653fff700 1 -- 172.21.15.76:0/837577597 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa6540cd5e0 con 0x7fa65c107030 2024-03-12T05:54:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.356+0000 7fa653fff700 1 --2- 172.21.15.76:0/837577597 >> [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264] conn(0x7fa6480ab330 0x7fa6480ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.356+0000 7fa653fff700 1 -- 172.21.15.76:0/837577597 --> [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6480ade00 con 0x7fa6480ab330 2024-03-12T05:54:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.356+0000 7fa653fff700 1 -- 172.21.15.76:0/837577597 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6540cda80 con 0x7fa65c107030 2024-03-12T05:54:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.356+0000 7fa65affd700 1 --2- 172.21.15.76:0/837577597 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa6480a1b50 0x7fa6480a4010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa64c0060b0 tx=0x7fa64c00c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.357+0000 7fa65a7fc700 1 --2- 172.21.15.76:0/837577597 >> [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264] conn(0x7fa6480ab330 0x7fa6480ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.357+0000 7fa65a7fc700 1 --2- 172.21.15.76:0/837577597 >> [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264] conn(0x7fa6480ab330 0x7fa6480ad750 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.358+0000 7fa653fff700 1 -- 172.21.15.76:0/837577597 <== osd.5 v2:172.21.15.88:6808/2894543264 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa6480ade00 con 0x7fa6480ab330 2024-03-12T05:54:13.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.374+0000 7fa6616f4700 1 -- 172.21.15.76:0/837577597 --> [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa65c0623b0 con 0x7fa6480ab330 2024-03-12T05:54:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.375+0000 7fa653fff700 1 -- 172.21.15.76:0/837577597 <== osd.5 v2:172.21.15.88:6808/2894543264 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa65c0623b0 con 0x7fa6480ab330 2024-03-12T05:54:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.375+0000 7fa6616f4700 1 -- 172.21.15.76:0/837577597 >> [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264] conn(0x7fa6480ab330 msgr2=0x7fa6480ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.375+0000 7fa6616f4700 1 --2- 172.21.15.76:0/837577597 >> [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264] conn(0x7fa6480ab330 0x7fa6480ad750 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.375+0000 7fa6616f4700 1 -- 172.21.15.76:0/837577597 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa6480a1b50 msgr2=0x7fa6480a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.375+0000 7fa6616f4700 1 --2- 172.21.15.76:0/837577597 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa6480a1b50 0x7fa6480a4010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa64c0060b0 tx=0x7fa64c00c040 comp rx=0 tx=0).stop 2024-03-12T05:54:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.375+0000 7fa6616f4700 1 -- 172.21.15.76:0/837577597 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa65c107030 msgr2=0x7fa65c19ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.375+0000 7fa6616f4700 1 --2- 172.21.15.76:0/837577597 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa65c107030 0x7fa65c19ca20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa65400c900 tx=0x7fa65400cc10 comp rx=0 tx=0).stop 2024-03-12T05:54:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.375+0000 7fa65b7fe700 1 -- 172.21.15.76:0/837577597 reap_dead start 2024-03-12T05:54:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.375+0000 7fa6616f4700 1 -- 172.21.15.76:0/837577597 shutdown_connections 2024-03-12T05:54:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.375+0000 7fa6616f4700 1 -- 172.21.15.76:0/837577597 >> 172.21.15.76:0/837577597 conn(0x7fa65c075eb0 msgr2=0x7fa65c0fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.375+0000 7fa6616f4700 1 -- 172.21.15.76:0/837577597 shutdown_connections 2024-03-12T05:54:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:13.375+0000 7fa6616f4700 1 -- 172.21.15.76:0/837577597 wait complete. 2024-03-12T05:54:14.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:13 smithi088 ceph-mon[36585]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-12T05:54:14.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:13 smithi076 ceph-mon[32095]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-12T05:54:14.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.605+0000 7f325f76c700 1 -- 172.21.15.76:0/492480370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3258105ab0 msgr2=0x7f3258105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:14.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.605+0000 7f325f76c700 1 --2- 172.21.15.76:0/492480370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3258105ab0 0x7f3258105e90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3254009990 tx=0x7f3254009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:14.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.605+0000 7f325f76c700 1 -- 172.21.15.76:0/492480370 shutdown_connections 2024-03-12T05:54:14.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.605+0000 7f325f76c700 1 --2- 172.21.15.76:0/492480370 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3258101510 0x7f325810d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:14.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.605+0000 7f325f76c700 1 --2- 172.21.15.76:0/492480370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3258105ab0 0x7f3258105e90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:14.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.605+0000 7f325f76c700 1 --2- 172.21.15.76:0/492480370 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3258100b50 0x7f3258100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:14.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.605+0000 7f325f76c700 1 -- 172.21.15.76:0/492480370 >> 172.21.15.76:0/492480370 conn(0x7f3258077b80 msgr2=0x7f3258077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:14.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.605+0000 7f325f76c700 1 -- 172.21.15.76:0/492480370 shutdown_connections 2024-03-12T05:54:14.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.605+0000 7f325f76c700 1 -- 172.21.15.76:0/492480370 wait complete. 2024-03-12T05:54:14.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325f76c700 1 Processor -- start 2024-03-12T05:54:14.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325f76c700 1 -- start start 2024-03-12T05:54:14.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325f76c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3258100b50 0x7f3258198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:14.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325f76c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3258101510 0x7f3258198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:14.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325f76c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3258105ab0 0x7f325819cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:14.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325f76c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325819d1d0 con 0x7f3258105ab0 2024-03-12T05:54:14.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325f76c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325819d340 con 0x7f3258101510 2024-03-12T05:54:14.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325f76c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325819d500 con 0x7f3258100b50 2024-03-12T05:54:14.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325dd09700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3258105ab0 0x7f325819cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:14.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325dd09700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3258105ab0 0x7f325819cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:54542/0 (socket says 172.21.15.76:54542) 2024-03-12T05:54:14.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325dd09700 1 -- 172.21.15.76:0/1917646754 learned_addr learned my addr 172.21.15.76:0/1917646754 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:14.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325d508700 1 --2- 172.21.15.76:0/1917646754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3258100b50 0x7f3258198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:14.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325cd07700 1 --2- 172.21.15.76:0/1917646754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3258101510 0x7f3258198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:14.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325dd09700 1 -- 172.21.15.76:0/1917646754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3258100b50 msgr2=0x7f3258198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:14.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.606+0000 7f325dd09700 1 --2- 172.21.15.76:0/1917646754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3258100b50 0x7f3258198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:14.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.607+0000 7f325dd09700 1 -- 172.21.15.76:0/1917646754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3258101510 msgr2=0x7f3258198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:14.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.607+0000 7f325dd09700 1 --2- 172.21.15.76:0/1917646754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3258101510 0x7f3258198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:14.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.607+0000 7f325dd09700 1 -- 172.21.15.76:0/1917646754 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32540095f0 con 0x7f3258105ab0 2024-03-12T05:54:14.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.607+0000 7f325dd09700 1 --2- 172.21.15.76:0/1917646754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3258105ab0 0x7f325819cbc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f325000b970 tx=0x7f325000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:14.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.607+0000 7f324e7fc700 1 -- 172.21.15.76:0/1917646754 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325000c9b0 con 0x7f3258105ab0 2024-03-12T05:54:14.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.607+0000 7f324e7fc700 1 -- 172.21.15.76:0/1917646754 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3250014440 con 0x7f3258105ab0 2024-03-12T05:54:14.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.607+0000 7f325f76c700 1 -- 172.21.15.76:0/1917646754 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325819d7e0 con 0x7f3258105ab0 2024-03-12T05:54:14.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.607+0000 7f324e7fc700 1 -- 172.21.15.76:0/1917646754 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32500075d0 con 0x7f3258105ab0 2024-03-12T05:54:14.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.607+0000 7f325f76c700 1 -- 172.21.15.76:0/1917646754 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32581a5040 con 0x7f3258105ab0 2024-03-12T05:54:14.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.607+0000 7f325f76c700 1 -- 172.21.15.76:0/1917646754 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3258110f10 con 0x7f3258105ab0 2024-03-12T05:54:14.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.609+0000 7f324e7fc700 1 -- 172.21.15.76:0/1917646754 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f3250014920 con 0x7f3258105ab0 2024-03-12T05:54:14.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.609+0000 7f324e7fc700 1 --2- 172.21.15.76:0/1917646754 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f32440a1bd0 0x7f32440a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:14.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.609+0000 7f325d508700 1 --2- 172.21.15.76:0/1917646754 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f32440a1bd0 0x7f32440a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:14.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.609+0000 7f324e7fc700 1 -- 172.21.15.76:0/1917646754 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3250006e40 con 0x7f3258105ab0 2024-03-12T05:54:14.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.610+0000 7f324e7fc700 1 --2- 172.21.15.76:0/1917646754 >> [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327] conn(0x7f32440ab410 0x7f32440ad830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:14.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.610+0000 7f324e7fc700 1 -- 172.21.15.76:0/1917646754 --> [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32440adee0 con 0x7f32440ab410 2024-03-12T05:54:14.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.610+0000 7f324e7fc700 1 -- 172.21.15.76:0/1917646754 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32500cd450 con 0x7f3258105ab0 2024-03-12T05:54:14.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.610+0000 7f325d508700 1 --2- 172.21.15.76:0/1917646754 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f32440a1bd0 0x7f32440a4090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3254006010 tx=0x7f325400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:14.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.610+0000 7f325cd07700 1 --2- 172.21.15.76:0/1917646754 >> [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327] conn(0x7f32440ab410 0x7f32440ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:14.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.610+0000 7f325cd07700 1 --2- 172.21.15.76:0/1917646754 >> [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327] conn(0x7f32440ab410 0x7f32440ad830 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:14.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.611+0000 7f324e7fc700 1 -- 172.21.15.76:0/1917646754 <== osd.10 v2:172.21.15.82:6824/2430204327 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f32440adee0 con 0x7f32440ab410 2024-03-12T05:54:14.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.627+0000 7f325f76c700 1 -- 172.21.15.76:0/1917646754 --> [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f32580fec90 con 0x7f32440ab410 2024-03-12T05:54:14.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.628+0000 7f324e7fc700 1 -- 172.21.15.76:0/1917646754 <== osd.10 v2:172.21.15.82:6824/2430204327 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f32580fec90 con 0x7f32440ab410 2024-03-12T05:54:14.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.628+0000 7f325f76c700 1 -- 172.21.15.76:0/1917646754 >> [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327] conn(0x7f32440ab410 msgr2=0x7f32440ad830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:14.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.628+0000 7f325f76c700 1 --2- 172.21.15.76:0/1917646754 >> [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327] conn(0x7f32440ab410 0x7f32440ad830 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:14.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.628+0000 7f325f76c700 1 -- 172.21.15.76:0/1917646754 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f32440a1bd0 msgr2=0x7f32440a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:14.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.628+0000 7f325f76c700 1 --2- 172.21.15.76:0/1917646754 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f32440a1bd0 0x7f32440a4090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3254006010 tx=0x7f325400b540 comp rx=0 tx=0).stop 2024-03-12T05:54:14.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.628+0000 7f325f76c700 1 -- 172.21.15.76:0/1917646754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3258105ab0 msgr2=0x7f325819cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:14.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.628+0000 7f325f76c700 1 --2- 172.21.15.76:0/1917646754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3258105ab0 0x7f325819cbc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f325000b970 tx=0x7f325000bd30 comp rx=0 tx=0).stop 2024-03-12T05:54:14.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.628+0000 7f325dd09700 1 -- 172.21.15.76:0/1917646754 reap_dead start 2024-03-12T05:54:14.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.628+0000 7f325f76c700 1 -- 172.21.15.76:0/1917646754 shutdown_connections 2024-03-12T05:54:14.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.628+0000 7f325f76c700 1 -- 172.21.15.76:0/1917646754 >> 172.21.15.76:0/1917646754 conn(0x7f3258077b80 msgr2=0x7f3258104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:14.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.629+0000 7f325f76c700 1 -- 172.21.15.76:0/1917646754 shutdown_connections 2024-03-12T05:54:14.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:10.629+0000 7f325f76c700 1 -- 172.21.15.76:0/1917646754 wait complete. 2024-03-12T05:54:14.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:13 smithi082 ceph-mon[36382]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-12T05:54:14.987 INFO:teuthology.orchestra.run.smithi076.stdout:313532612633 2024-03-12T05:54:14.987 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-12T05:54:15.468 INFO:teuthology.orchestra.run.smithi076.stdout:34359738442 2024-03-12T05:54:15.469 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-12T05:54:16.214 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:15 smithi076 ceph-mon[32095]: pgmap v303: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-12T05:54:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:15 smithi088 ceph-mon[36585]: pgmap v303: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-12T05:54:16.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:15 smithi082 ceph-mon[36382]: pgmap v303: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-12T05:54:16.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.725+0000 7f6178b06700 1 -- 172.21.15.76:0/4254624394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f617406a410 msgr2=0x7f617406a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:16.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.725+0000 7f6178b06700 1 --2- 172.21.15.76:0/4254624394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f617406a410 0x7f617406a7f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6164009990 tx=0x7f6164009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:16.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.725+0000 7f6178b06700 1 -- 172.21.15.76:0/4254624394 shutdown_connections 2024-03-12T05:54:16.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.725+0000 7f6178b06700 1 --2- 172.21.15.76:0/4254624394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6174106ff0 0x7f617410eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:16.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.725+0000 7f6178b06700 1 --2- 172.21.15.76:0/4254624394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f617406a410 0x7f617406a7f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:16.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.725+0000 7f6178b06700 1 --2- 172.21.15.76:0/4254624394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f617406adc0 0x7f61741068c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:16.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.725+0000 7f6178b06700 1 -- 172.21.15.76:0/4254624394 >> 172.21.15.76:0/4254624394 conn(0x7f6174076bc0 msgr2=0x7f6174076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:16.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.725+0000 7f6178b06700 1 -- 172.21.15.76:0/4254624394 shutdown_connections 2024-03-12T05:54:16.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.725+0000 7f6178b06700 1 -- 172.21.15.76:0/4254624394 wait complete. 2024-03-12T05:54:16.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.725+0000 7f6178b06700 1 Processor -- start 2024-03-12T05:54:16.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.725+0000 7f6178b06700 1 -- start start 2024-03-12T05:54:16.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6178b06700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f617406a410 0x7f6174198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:16.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6178b06700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f617406adc0 0x7f61741986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:16.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6178b06700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6174106ff0 0x7f617419ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:16.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6178b06700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617419d070 con 0x7f6174106ff0 2024-03-12T05:54:16.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6178b06700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617419d1e0 con 0x7f617406adc0 2024-03-12T05:54:16.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6178b06700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617419d3a0 con 0x7f617406a410 2024-03-12T05:54:16.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6172d9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6174106ff0 0x7f617419ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:16.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f617259c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f617406a410 0x7f6174198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:16.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6172d9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6174106ff0 0x7f617419ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43596/0 (socket says 172.21.15.76:43596) 2024-03-12T05:54:16.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6172d9d700 1 -- 172.21.15.76:0/3727085549 learned_addr learned my addr 172.21.15.76:0/3727085549 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:16.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f617259c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f617406a410 0x7f6174198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.76:44362/0 (socket says 172.21.15.76:44362) 2024-03-12T05:54:16.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6171d9b700 1 --2- 172.21.15.76:0/3727085549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f617406adc0 0x7f61741986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:16.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6172d9d700 1 -- 172.21.15.76:0/3727085549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f617406a410 msgr2=0x7f6174198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:16.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6172d9d700 1 --2- 172.21.15.76:0/3727085549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f617406a410 0x7f6174198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:16.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6172d9d700 1 -- 172.21.15.76:0/3727085549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f617406adc0 msgr2=0x7f61741986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:16.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6172d9d700 1 --2- 172.21.15.76:0/3727085549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f617406adc0 0x7f61741986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:16.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6172d9d700 1 -- 172.21.15.76:0/3727085549 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61640095f0 con 0x7f6174106ff0 2024-03-12T05:54:16.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.726+0000 7f6172d9d700 1 --2- 172.21.15.76:0/3727085549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6174106ff0 0x7f617419ca60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f616c00ada0 tx=0x7f616c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:16.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.727+0000 7f61637fe700 1 -- 172.21.15.76:0/3727085549 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c00f070 con 0x7f6174106ff0 2024-03-12T05:54:16.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.727+0000 7f61637fe700 1 -- 172.21.15.76:0/3727085549 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f616c007b30 con 0x7f6174106ff0 2024-03-12T05:54:16.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.727+0000 7f6178b06700 1 -- 172.21.15.76:0/3727085549 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617419d680 con 0x7f6174106ff0 2024-03-12T05:54:16.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.727+0000 7f61637fe700 1 -- 172.21.15.76:0/3727085549 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c0053c0 con 0x7f6174106ff0 2024-03-12T05:54:16.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.727+0000 7f6178b06700 1 -- 172.21.15.76:0/3727085549 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61741a4ee0 con 0x7f6174106ff0 2024-03-12T05:54:16.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.728+0000 7f6178b06700 1 -- 172.21.15.76:0/3727085549 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6174112010 con 0x7f6174106ff0 2024-03-12T05:54:16.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.729+0000 7f61637fe700 1 -- 172.21.15.76:0/3727085549 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f616c00c980 con 0x7f6174106ff0 2024-03-12T05:54:16.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.729+0000 7f61637fe700 1 --2- 172.21.15.76:0/3727085549 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f615c0a1b50 0x7f615c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:16.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.729+0000 7f617259c700 1 --2- 172.21.15.76:0/3727085549 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f615c0a1b50 0x7f615c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:16.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.729+0000 7f61637fe700 1 -- 172.21.15.76:0/3727085549 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f616c0cc9c0 con 0x7f6174106ff0 2024-03-12T05:54:16.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.729+0000 7f61637fe700 1 --2- 172.21.15.76:0/3727085549 >> [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488] conn(0x7f615c0ab330 0x7f615c0ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:16.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.729+0000 7f61637fe700 1 -- 172.21.15.76:0/3727085549 --> [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f615c0ade00 con 0x7f615c0ab330 2024-03-12T05:54:16.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.729+0000 7f61637fe700 1 -- 172.21.15.76:0/3727085549 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f616c0d12c0 con 0x7f6174106ff0 2024-03-12T05:54:16.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.730+0000 7f617259c700 1 --2- 172.21.15.76:0/3727085549 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f615c0a1b50 0x7f615c0a4010 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f616400b5c0 tx=0x7f6164005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:16.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.730+0000 7f6171d9b700 1 --2- 172.21.15.76:0/3727085549 >> [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488] conn(0x7f615c0ab330 0x7f615c0ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:16.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.730+0000 7f6171d9b700 1 --2- 172.21.15.76:0/3727085549 >> [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488] conn(0x7f615c0ab330 0x7f615c0ad750 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:16.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.731+0000 7f61637fe700 1 -- 172.21.15.76:0/3727085549 <== osd.1 v2:172.21.15.82:6800/1086375488 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f615c0ade00 con 0x7f615c0ab330 2024-03-12T05:54:16.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.747+0000 7f6178b06700 1 -- 172.21.15.76:0/3727085549 --> [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6174062380 con 0x7f615c0ab330 2024-03-12T05:54:16.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.748+0000 7f61637fe700 1 -- 172.21.15.76:0/3727085549 <== osd.1 v2:172.21.15.82:6800/1086375488 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6174062380 con 0x7f615c0ab330 2024-03-12T05:54:16.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.748+0000 7f6178b06700 1 -- 172.21.15.76:0/3727085549 >> [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488] conn(0x7f615c0ab330 msgr2=0x7f615c0ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:16.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.748+0000 7f6178b06700 1 --2- 172.21.15.76:0/3727085549 >> [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488] conn(0x7f615c0ab330 0x7f615c0ad750 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:16.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.748+0000 7f6178b06700 1 -- 172.21.15.76:0/3727085549 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f615c0a1b50 msgr2=0x7f615c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:16.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.748+0000 7f6178b06700 1 --2- 172.21.15.76:0/3727085549 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f615c0a1b50 0x7f615c0a4010 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f616400b5c0 tx=0x7f6164005960 comp rx=0 tx=0).stop 2024-03-12T05:54:16.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.748+0000 7f6178b06700 1 -- 172.21.15.76:0/3727085549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6174106ff0 msgr2=0x7f617419ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:16.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.748+0000 7f6178b06700 1 --2- 172.21.15.76:0/3727085549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6174106ff0 0x7f617419ca60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f616c00ada0 tx=0x7f616c00c330 comp rx=0 tx=0).stop 2024-03-12T05:54:16.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.748+0000 7f6172d9d700 1 -- 172.21.15.76:0/3727085549 reap_dead start 2024-03-12T05:54:16.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.748+0000 7f6178b06700 1 -- 172.21.15.76:0/3727085549 shutdown_connections 2024-03-12T05:54:16.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.748+0000 7f6178b06700 1 -- 172.21.15.76:0/3727085549 >> 172.21.15.76:0/3727085549 conn(0x7f6174076bc0 msgr2=0x7f61740fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:16.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.748+0000 7f6178b06700 1 -- 172.21.15.76:0/3727085549 shutdown_connections 2024-03-12T05:54:16.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:16.748+0000 7f6178b06700 1 -- 172.21.15.76:0/3727085549 wait complete. 2024-03-12T05:54:17.175 INFO:teuthology.orchestra.run.smithi076.stdout:219043332135 2024-03-12T05:54:17.176 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-12T05:54:18.181 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:17 smithi076 ceph-mon[32095]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:54:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:17 smithi088 ceph-mon[36585]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:54:18.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.283+0000 7f8c31f96700 1 -- 172.21.15.76:0/3650666074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c2c101810 msgr2=0x7f8c2c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:18.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.283+0000 7f8c31f96700 1 --2- 172.21.15.76:0/3650666074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c2c101810 0x7f8c2c10da40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400b0a0 tx=0x7f8c2400b3b0 comp rx=0 tx=0).stop 2024-03-12T05:54:18.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.283+0000 7f8c31f96700 1 -- 172.21.15.76:0/3650666074 shutdown_connections 2024-03-12T05:54:18.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.283+0000 7f8c31f96700 1 --2- 172.21.15.76:0/3650666074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c2c101810 0x7f8c2c10da40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:18.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.283+0000 7f8c31f96700 1 --2- 172.21.15.76:0/3650666074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c2c106d60 0x7f8c2c107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:18.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.283+0000 7f8c31f96700 1 --2- 172.21.15.76:0/3650666074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c2c100e50 0x7f8c2c1012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:18.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.283+0000 7f8c31f96700 1 -- 172.21.15.76:0/3650666074 >> 172.21.15.76:0/3650666074 conn(0x7f8c2c0779d0 msgr2=0x7f8c2c077de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:18.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.283+0000 7f8c31f96700 1 -- 172.21.15.76:0/3650666074 shutdown_connections 2024-03-12T05:54:18.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.283+0000 7f8c31f96700 1 -- 172.21.15.76:0/3650666074 wait complete. 2024-03-12T05:54:18.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.284+0000 7f8c31f96700 1 Processor -- start 2024-03-12T05:54:18.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.284+0000 7f8c31f96700 1 -- start start 2024-03-12T05:54:18.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c31f96700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c2c100e50 0x7f8c2c197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:18.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c31f96700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c2c101810 0x7f8c2c197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:18.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c31f96700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c2c106d60 0x7f8c2c19ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:18.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c31f96700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2c19c090 con 0x7f8c2c106d60 2024-03-12T05:54:18.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c31f96700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2c19c200 con 0x7f8c2c100e50 2024-03-12T05:54:18.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c31f96700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2c19c3c0 con 0x7f8c2c101810 2024-03-12T05:54:18.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c2bfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c2c106d60 0x7f8c2c19ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:18.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c2b7fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c2c100e50 0x7f8c2c197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:18.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c2bfff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c2c106d60 0x7f8c2c19ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43606/0 (socket says 172.21.15.76:43606) 2024-03-12T05:54:18.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c2bfff700 1 -- 172.21.15.76:0/104852459 learned_addr learned my addr 172.21.15.76:0/104852459 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:18.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c2affd700 1 --2- 172.21.15.76:0/104852459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c2c101810 0x7f8c2c197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:18.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c2bfff700 1 -- 172.21.15.76:0/104852459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c2c101810 msgr2=0x7f8c2c197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:18.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c2bfff700 1 --2- 172.21.15.76:0/104852459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c2c101810 0x7f8c2c197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:18.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c2bfff700 1 -- 172.21.15.76:0/104852459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c2c100e50 msgr2=0x7f8c2c197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:18.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c2bfff700 1 --2- 172.21.15.76:0/104852459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c2c100e50 0x7f8c2c197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:18.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c2bfff700 1 -- 172.21.15.76:0/104852459 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c24009d00 con 0x7f8c2c106d60 2024-03-12T05:54:18.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c2bfff700 1 --2- 172.21.15.76:0/104852459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c2c106d60 0x7f8c2c19ba80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400b070 tx=0x7f8c24004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:18.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c28ff9700 1 -- 172.21.15.76:0/104852459 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2400e040 con 0x7f8c2c106d60 2024-03-12T05:54:18.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c31f96700 1 -- 172.21.15.76:0/104852459 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c2c19c640 con 0x7f8c2c106d60 2024-03-12T05:54:18.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.285+0000 7f8c28ff9700 1 -- 172.21.15.76:0/104852459 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c24007850 con 0x7f8c2c106d60 2024-03-12T05:54:18.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.286+0000 7f8c28ff9700 1 -- 172.21.15.76:0/104852459 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2401b850 con 0x7f8c2c106d60 2024-03-12T05:54:18.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.286+0000 7f8c31f96700 1 -- 172.21.15.76:0/104852459 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c2c1a3f00 con 0x7f8c2c106d60 2024-03-12T05:54:18.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.286+0000 7f8c31f96700 1 -- 172.21.15.76:0/104852459 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8c2c110ee0 con 0x7f8c2c106d60 2024-03-12T05:54:18.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.288+0000 7f8c28ff9700 1 -- 172.21.15.76:0/104852459 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f8c2401da80 con 0x7f8c2c106d60 2024-03-12T05:54:18.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.288+0000 7f8c28ff9700 1 --2- 172.21.15.76:0/104852459 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8c140a1b50 0x7f8c140a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:18.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.288+0000 7f8c28ff9700 1 -- 172.21.15.76:0/104852459 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8c240cf640 con 0x7f8c2c106d60 2024-03-12T05:54:18.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.288+0000 7f8c2b7fe700 1 --2- 172.21.15.76:0/104852459 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8c140a1b50 0x7f8c140a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:18.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.288+0000 7f8c28ff9700 1 --2- 172.21.15.76:0/104852459 >> [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618] conn(0x7f8c140ab330 0x7f8c140ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:18.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.288+0000 7f8c28ff9700 1 -- 172.21.15.76:0/104852459 --> [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c140ade00 con 0x7f8c140ab330 2024-03-12T05:54:18.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.289+0000 7f8c28ff9700 1 -- 172.21.15.76:0/104852459 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c240cfae0 con 0x7f8c2c106d60 2024-03-12T05:54:18.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.289+0000 7f8c2affd700 1 --2- 172.21.15.76:0/104852459 >> [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618] conn(0x7f8c140ab330 0x7f8c140ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:18.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.289+0000 7f8c2b7fe700 1 --2- 172.21.15.76:0/104852459 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8c140a1b50 0x7f8c140a4010 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c005950 tx=0x7f8c1c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:18.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.289+0000 7f8c2affd700 1 --2- 172.21.15.76:0/104852459 >> [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618] conn(0x7f8c140ab330 0x7f8c140ad750 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:18.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.290+0000 7f8c28ff9700 1 -- 172.21.15.76:0/104852459 <== osd.3 v2:172.21.15.76:6810/159124618 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8c140ade00 con 0x7f8c140ab330 2024-03-12T05:54:18.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.306+0000 7f8c31f96700 1 -- 172.21.15.76:0/104852459 --> [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8c2c062380 con 0x7f8c140ab330 2024-03-12T05:54:18.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.306+0000 7f8c28ff9700 1 -- 172.21.15.76:0/104852459 <== osd.3 v2:172.21.15.76:6810/159124618 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8c2c062380 con 0x7f8c140ab330 2024-03-12T05:54:18.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.307+0000 7f8c31f96700 1 -- 172.21.15.76:0/104852459 >> [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618] conn(0x7f8c140ab330 msgr2=0x7f8c140ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:18.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.307+0000 7f8c31f96700 1 --2- 172.21.15.76:0/104852459 >> [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618] conn(0x7f8c140ab330 0x7f8c140ad750 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:18.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.307+0000 7f8c31f96700 1 -- 172.21.15.76:0/104852459 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8c140a1b50 msgr2=0x7f8c140a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:18.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.307+0000 7f8c31f96700 1 --2- 172.21.15.76:0/104852459 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8c140a1b50 0x7f8c140a4010 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c005950 tx=0x7f8c1c00b410 comp rx=0 tx=0).stop 2024-03-12T05:54:18.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.307+0000 7f8c31f96700 1 -- 172.21.15.76:0/104852459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c2c106d60 msgr2=0x7f8c2c19ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:18.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.307+0000 7f8c31f96700 1 --2- 172.21.15.76:0/104852459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c2c106d60 0x7f8c2c19ba80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400b070 tx=0x7f8c24004e40 comp rx=0 tx=0).stop 2024-03-12T05:54:18.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.307+0000 7f8c2bfff700 1 -- 172.21.15.76:0/104852459 reap_dead start 2024-03-12T05:54:18.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.307+0000 7f8c31f96700 1 -- 172.21.15.76:0/104852459 shutdown_connections 2024-03-12T05:54:18.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.307+0000 7f8c31f96700 1 -- 172.21.15.76:0/104852459 >> 172.21.15.76:0/104852459 conn(0x7f8c2c0779d0 msgr2=0x7f8c2c0ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:18.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.307+0000 7f8c31f96700 1 -- 172.21.15.76:0/104852459 shutdown_connections 2024-03-12T05:54:18.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:18.307+0000 7f8c31f96700 1 -- 172.21.15.76:0/104852459 wait complete. 2024-03-12T05:54:18.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:17 smithi082 ceph-mon[36382]: pgmap v304: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:54:19.000 INFO:teuthology.orchestra.run.smithi076.stdout:55834574918 2024-03-12T05:54:19.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-12T05:54:19.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.207+0000 7f215179e700 1 -- 172.21.15.76:0/1057676406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f214c100a50 msgr2=0x7f214c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.207+0000 7f215179e700 1 --2- 172.21.15.76:0/1057676406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f214c100a50 0x7f214c100ed0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2140009990 tx=0x7f2140009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.208+0000 7f215179e700 1 -- 172.21.15.76:0/1057676406 shutdown_connections 2024-03-12T05:54:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.208+0000 7f215179e700 1 --2- 172.21.15.76:0/1057676406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214c101410 0x7f214c10d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.208+0000 7f215179e700 1 --2- 172.21.15.76:0/1057676406 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f214c106a50 0x7f214c106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.208+0000 7f215179e700 1 --2- 172.21.15.76:0/1057676406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f214c100a50 0x7f214c100ed0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.208+0000 7f215179e700 1 -- 172.21.15.76:0/1057676406 >> 172.21.15.76:0/1057676406 conn(0x7f214c0fc540 msgr2=0x7f214c0fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.208+0000 7f215179e700 1 -- 172.21.15.76:0/1057676406 shutdown_connections 2024-03-12T05:54:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.208+0000 7f215179e700 1 -- 172.21.15.76:0/1057676406 wait complete. 2024-03-12T05:54:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.208+0000 7f215179e700 1 Processor -- start 2024-03-12T05:54:19.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.208+0000 7f215179e700 1 -- start start 2024-03-12T05:54:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f215179e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f214c100a50 0x7f214c0fc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f215179e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f214c101410 0x7f214c0fcf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f215179e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214c106a50 0x7f214c0fd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f215179e700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c1a64a0 con 0x7f214c101410 2024-03-12T05:54:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f215179e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c1a6610 con 0x7f214c100a50 2024-03-12T05:54:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f215179e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c1a67d0 con 0x7f214c106a50 2024-03-12T05:54:19.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f214b7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f214c101410 0x7f214c0fcf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f214b7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f214c101410 0x7f214c0fcf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:43634/0 (socket says 172.21.15.76:43634) 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f214b7fe700 1 -- 172.21.15.76:0/3722162273 learned_addr learned my addr 172.21.15.76:0/3722162273 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f214bfff700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f214c100a50 0x7f214c0fc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f2150f9d700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214c106a50 0x7f214c0fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f214b7fe700 1 -- 172.21.15.76:0/3722162273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214c106a50 msgr2=0x7f214c0fd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f214b7fe700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214c106a50 0x7f214c0fd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f214b7fe700 1 -- 172.21.15.76:0/3722162273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f214c100a50 msgr2=0x7f214c0fc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f214b7fe700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f214c100a50 0x7f214c0fc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f214b7fe700 1 -- 172.21.15.76:0/3722162273 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21400095f0 con 0x7f214c101410 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f214bfff700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f214c100a50 0x7f214c0fc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f2150f9d700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f214c106a50 0x7f214c0fd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f214b7fe700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f214c101410 0x7f214c0fcf30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f21400098e0 tx=0x7f214000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.209+0000 7f21497fa700 1 -- 172.21.15.76:0/3722162273 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214001d070 con 0x7f214c101410 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.210+0000 7f21497fa700 1 -- 172.21.15.76:0/3722162273 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f214000fc70 con 0x7f214c101410 2024-03-12T05:54:19.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.210+0000 7f215179e700 1 -- 172.21.15.76:0/3722162273 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214c1a6a50 con 0x7f214c101410 2024-03-12T05:54:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.210+0000 7f21497fa700 1 -- 172.21.15.76:0/3722162273 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21400176e0 con 0x7f214c101410 2024-03-12T05:54:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.210+0000 7f215179e700 1 -- 172.21.15.76:0/3722162273 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214c1a6e60 con 0x7f214c101410 2024-03-12T05:54:19.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.210+0000 7f215179e700 1 -- 172.21.15.76:0/3722162273 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f214c110d50 con 0x7f214c101410 2024-03-12T05:54:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.211+0000 7f21497fa700 1 -- 172.21.15.76:0/3722162273 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f21400229e0 con 0x7f214c101410 2024-03-12T05:54:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.212+0000 7f21497fa700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21340a1b50 0x7f21340a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.212+0000 7f214bfff700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21340a1b50 0x7f21340a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.212+0000 7f21497fa700 1 -- 172.21.15.76:0/3722162273 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f21400cebc0 con 0x7f214c101410 2024-03-12T05:54:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.212+0000 7f21497fa700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191] conn(0x7f21340ab330 0x7f21340ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.212+0000 7f21497fa700 1 -- 172.21.15.76:0/3722162273 --> [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21340ade00 con 0x7f21340ab330 2024-03-12T05:54:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.212+0000 7f21497fa700 1 -- 172.21.15.76:0/3722162273 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21400cf060 con 0x7f214c101410 2024-03-12T05:54:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.212+0000 7f2150f9d700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191] conn(0x7f21340ab330 0x7f21340ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:19.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.213+0000 7f214bfff700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21340a1b50 0x7f21340a4010 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f213c005fd0 tx=0x7f213c005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.213+0000 7f2150f9d700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191] conn(0x7f21340ab330 0x7f21340ad750 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.213+0000 7f21497fa700 1 -- 172.21.15.76:0/3722162273 <== osd.9 v2:172.21.15.76:6826/3035286191 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21340ade00 con 0x7f21340ab330 2024-03-12T05:54:19.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.231+0000 7f215179e700 1 -- 172.21.15.76:0/3722162273 --> [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f214c062380 con 0x7f21340ab330 2024-03-12T05:54:19.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.232+0000 7f21497fa700 1 -- 172.21.15.76:0/3722162273 <== osd.9 v2:172.21.15.76:6826/3035286191 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f214c062380 con 0x7f21340ab330 2024-03-12T05:54:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.232+0000 7f215179e700 1 -- 172.21.15.76:0/3722162273 >> [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191] conn(0x7f21340ab330 msgr2=0x7f21340ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.232+0000 7f215179e700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191] conn(0x7f21340ab330 0x7f21340ad750 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.232+0000 7f215179e700 1 -- 172.21.15.76:0/3722162273 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21340a1b50 msgr2=0x7f21340a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.232+0000 7f215179e700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21340a1b50 0x7f21340a4010 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f213c005fd0 tx=0x7f213c005d40 comp rx=0 tx=0).stop 2024-03-12T05:54:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.232+0000 7f215179e700 1 -- 172.21.15.76:0/3722162273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f214c101410 msgr2=0x7f214c0fcf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.232+0000 7f215179e700 1 --2- 172.21.15.76:0/3722162273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f214c101410 0x7f214c0fcf30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f21400098e0 tx=0x7f214000f740 comp rx=0 tx=0).stop 2024-03-12T05:54:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.232+0000 7f2150f9d700 1 -- 172.21.15.76:0/3722162273 reap_dead start 2024-03-12T05:54:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.232+0000 7f215179e700 1 -- 172.21.15.76:0/3722162273 shutdown_connections 2024-03-12T05:54:19.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.232+0000 7f215179e700 1 -- 172.21.15.76:0/3722162273 >> 172.21.15.76:0/3722162273 conn(0x7f214c0fc540 msgr2=0x7f214c069c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:19.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.232+0000 7f215179e700 1 -- 172.21.15.76:0/3722162273 shutdown_connections 2024-03-12T05:54:19.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:19.232+0000 7f215179e700 1 -- 172.21.15.76:0/3722162273 wait complete. 2024-03-12T05:54:19.750 INFO:teuthology.orchestra.run.smithi076.stdout:163208757298 2024-03-12T05:54:19.751 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-12T05:54:19.987 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:19 smithi076 ceph-mon[32095]: pgmap v305: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:54:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:19 smithi088 ceph-mon[36585]: pgmap v305: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:54:20.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:19 smithi082 ceph-mon[36382]: pgmap v305: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:54:20.544 INFO:teuthology.orchestra.run.smithi076.stdout:257698037796 2024-03-12T05:54:20.545 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-12T05:54:21.351 INFO:teuthology.orchestra.run.smithi076.stdout:292057776160 2024-03-12T05:54:21.352 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-12T05:54:21.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.562+0000 7f2898476700 1 -- 172.21.15.76:0/1089760949 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2890107dd0 msgr2=0x7f28901081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:21.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.562+0000 7f2898476700 1 --2- 172.21.15.76:0/1089760949 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2890107dd0 0x7f28901081b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f288c009990 tx=0x7f288c009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:21.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.562+0000 7f2898476700 1 -- 172.21.15.76:0/1089760949 shutdown_connections 2024-03-12T05:54:21.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.562+0000 7f2898476700 1 --2- 172.21.15.76:0/1089760949 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f28901026b0 0x7f289010eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:21.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.562+0000 7f2898476700 1 --2- 172.21.15.76:0/1089760949 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2890107dd0 0x7f28901081b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:21.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.562+0000 7f2898476700 1 --2- 172.21.15.76:0/1089760949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2890101cf0 0x7f2890102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:21.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.562+0000 7f2898476700 1 -- 172.21.15.76:0/1089760949 >> 172.21.15.76:0/1089760949 conn(0x7f2890078d30 msgr2=0x7f2890079140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:21.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.563+0000 7f2898476700 1 -- 172.21.15.76:0/1089760949 shutdown_connections 2024-03-12T05:54:21.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.563+0000 7f2898476700 1 -- 172.21.15.76:0/1089760949 wait complete. 2024-03-12T05:54:21.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.563+0000 7f2898476700 1 Processor -- start 2024-03-12T05:54:21.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.563+0000 7f2898476700 1 -- start start 2024-03-12T05:54:21.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2898476700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2890101cf0 0x7f28901981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:21.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2898476700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28901026b0 0x7f28901986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:21.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2898476700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2890107dd0 0x7f289019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:21.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2898476700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289019d090 con 0x7f28901026b0 2024-03-12T05:54:21.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2898476700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289019d200 con 0x7f2890101cf0 2024-03-12T05:54:21.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2898476700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f289019d3c0 con 0x7f2890107dd0 2024-03-12T05:54:21.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2895a11700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28901026b0 0x7f28901986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:21.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2895a11700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28901026b0 0x7f28901986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60590/0 (socket says 172.21.15.76:60590) 2024-03-12T05:54:21.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2895a11700 1 -- 172.21.15.76:0/1093118948 learned_addr learned my addr 172.21.15.76:0/1093118948 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:21.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2896a13700 1 --2- 172.21.15.76:0/1093118948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2890107dd0 0x7f289019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:21.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2896212700 1 --2- 172.21.15.76:0/1093118948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2890101cf0 0x7f28901981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:21.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2896a13700 1 -- 172.21.15.76:0/1093118948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2890101cf0 msgr2=0x7f28901981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:21.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2896a13700 1 --2- 172.21.15.76:0/1093118948 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2890101cf0 0x7f28901981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:21.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2896a13700 1 -- 172.21.15.76:0/1093118948 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28901026b0 msgr2=0x7f28901986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:21.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2896a13700 1 --2- 172.21.15.76:0/1093118948 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28901026b0 0x7f28901986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:21.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2896a13700 1 -- 172.21.15.76:0/1093118948 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288c0095f0 con 0x7f2890107dd0 2024-03-12T05:54:21.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.564+0000 7f2896a13700 1 --2- 172.21.15.76:0/1093118948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2890107dd0 0x7f289019ca80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2888007f10 tx=0x7f288800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:21.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.565+0000 7f28877fe700 1 -- 172.21.15.76:0/1093118948 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2888004db0 con 0x7f2890107dd0 2024-03-12T05:54:21.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.565+0000 7f2898476700 1 -- 172.21.15.76:0/1093118948 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289019d6a0 con 0x7f2890107dd0 2024-03-12T05:54:21.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.565+0000 7f2898476700 1 -- 172.21.15.76:0/1093118948 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28901a4f00 con 0x7f2890107dd0 2024-03-12T05:54:21.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.565+0000 7f28877fe700 1 -- 172.21.15.76:0/1093118948 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f288800f040 con 0x7f2890107dd0 2024-03-12T05:54:21.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.565+0000 7f28877fe700 1 -- 172.21.15.76:0/1093118948 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2888005a90 con 0x7f2890107dd0 2024-03-12T05:54:21.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.566+0000 7f2898476700 1 -- 172.21.15.76:0/1093118948 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2874001030 con 0x7f2890107dd0 2024-03-12T05:54:21.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.567+0000 7f28877fe700 1 -- 172.21.15.76:0/1093118948 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f288800db20 con 0x7f2890107dd0 2024-03-12T05:54:21.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.568+0000 7f28877fe700 1 --2- 172.21.15.76:0/1093118948 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f287c0a1c10 0x7f287c0a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:21.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.568+0000 7f28877fe700 1 -- 172.21.15.76:0/1093118948 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f28880cd590 con 0x7f2890107dd0 2024-03-12T05:54:21.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.568+0000 7f2896212700 1 --2- 172.21.15.76:0/1093118948 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f287c0a1c10 0x7f287c0a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:21.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.568+0000 7f28877fe700 1 --2- 172.21.15.76:0/1093118948 >> [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628] conn(0x7f287c0ab280 0x7f287c0ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:21.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.568+0000 7f28877fe700 1 -- 172.21.15.76:0/1093118948 --> [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f287c0add50 con 0x7f287c0ab280 2024-03-12T05:54:21.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.568+0000 7f28877fe700 1 -- 172.21.15.76:0/1093118948 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28880cda30 con 0x7f2890107dd0 2024-03-12T05:54:21.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.569+0000 7f2895a11700 1 --2- 172.21.15.76:0/1093118948 >> [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628] conn(0x7f287c0ab280 0x7f287c0ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:21.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.569+0000 7f2896212700 1 --2- 172.21.15.76:0/1093118948 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f287c0a1c10 0x7f287c0a40d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f288c00b2e0 tx=0x7f288c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:21.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.569+0000 7f2895a11700 1 --2- 172.21.15.76:0/1093118948 >> [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628] conn(0x7f287c0ab280 0x7f287c0ad6a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:21.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.570+0000 7f28877fe700 1 -- 172.21.15.76:0/1093118948 <== osd.4 v2:172.21.15.82:6808/3036883628 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f287c0add50 con 0x7f287c0ab280 2024-03-12T05:54:21.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.586+0000 7f2898476700 1 -- 172.21.15.76:0/1093118948 --> [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2874002dc0 con 0x7f287c0ab280 2024-03-12T05:54:21.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.587+0000 7f28877fe700 1 -- 172.21.15.76:0/1093118948 <== osd.4 v2:172.21.15.82:6808/3036883628 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2874002dc0 con 0x7f287c0ab280 2024-03-12T05:54:21.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.587+0000 7f2898476700 1 -- 172.21.15.76:0/1093118948 >> [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628] conn(0x7f287c0ab280 msgr2=0x7f287c0ad6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:21.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.587+0000 7f2898476700 1 --2- 172.21.15.76:0/1093118948 >> [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628] conn(0x7f287c0ab280 0x7f287c0ad6a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:21.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.587+0000 7f2898476700 1 -- 172.21.15.76:0/1093118948 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f287c0a1c10 msgr2=0x7f287c0a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:21.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.587+0000 7f2898476700 1 --2- 172.21.15.76:0/1093118948 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f287c0a1c10 0x7f287c0a40d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f288c00b2e0 tx=0x7f288c005fb0 comp rx=0 tx=0).stop 2024-03-12T05:54:21.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.587+0000 7f2898476700 1 -- 172.21.15.76:0/1093118948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2890107dd0 msgr2=0x7f289019ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:21.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.587+0000 7f2898476700 1 --2- 172.21.15.76:0/1093118948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2890107dd0 0x7f289019ca80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2888007f10 tx=0x7f288800d3b0 comp rx=0 tx=0).stop 2024-03-12T05:54:21.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.587+0000 7f2896a13700 1 -- 172.21.15.76:0/1093118948 reap_dead start 2024-03-12T05:54:21.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.587+0000 7f2898476700 1 -- 172.21.15.76:0/1093118948 shutdown_connections 2024-03-12T05:54:21.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.587+0000 7f2898476700 1 -- 172.21.15.76:0/1093118948 >> 172.21.15.76:0/1093118948 conn(0x7f2890078d30 msgr2=0x7f28901007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:21.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.587+0000 7f2898476700 1 -- 172.21.15.76:0/1093118948 shutdown_connections 2024-03-12T05:54:21.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:21.587+0000 7f2898476700 1 -- 172.21.15.76:0/1093118948 wait complete. 2024-03-12T05:54:22.111 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:21 smithi076 ceph-mon[32095]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-12T05:54:22.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:21 smithi088 ceph-mon[36585]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-12T05:54:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.342+0000 7fab71f6d700 1 -- 172.21.15.76:0/3499858625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab6c101510 msgr2=0x7fab6c10d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.342+0000 7fab71f6d700 1 --2- 172.21.15.76:0/3499858625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab6c101510 0x7fab6c10d9b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fab6400b0a0 tx=0x7fab6400b3b0 comp rx=0 tx=0).stop 2024-03-12T05:54:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.342+0000 7fab71f6d700 1 -- 172.21.15.76:0/3499858625 shutdown_connections 2024-03-12T05:54:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.342+0000 7fab71f6d700 1 --2- 172.21.15.76:0/3499858625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab6c101510 0x7fab6c10d9b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.342+0000 7fab71f6d700 1 --2- 172.21.15.76:0/3499858625 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab6c105ab0 0x7fab6c105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.342+0000 7fab71f6d700 1 --2- 172.21.15.76:0/3499858625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab6c100b50 0x7fab6c100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.342+0000 7fab71f6d700 1 -- 172.21.15.76:0/3499858625 >> 172.21.15.76:0/3499858625 conn(0x7fab6c077b80 msgr2=0x7fab6c077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.342+0000 7fab71f6d700 1 -- 172.21.15.76:0/3499858625 shutdown_connections 2024-03-12T05:54:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.342+0000 7fab71f6d700 1 -- 172.21.15.76:0/3499858625 wait complete. 2024-03-12T05:54:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.343+0000 7fab71f6d700 1 Processor -- start 2024-03-12T05:54:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.343+0000 7fab71f6d700 1 -- start start 2024-03-12T05:54:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.343+0000 7fab71f6d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab6c100b50 0x7fab6c198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.343+0000 7fab71f6d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab6c101510 0x7fab6c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.343+0000 7fab71f6d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab6c105ab0 0x7fab6c19ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.343+0000 7fab71f6d700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c19d0a0 con 0x7fab6c101510 2024-03-12T05:54:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.343+0000 7fab71f6d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c19d210 con 0x7fab6c100b50 2024-03-12T05:54:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.343+0000 7fab71f6d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c19d3d0 con 0x7fab6c105ab0 2024-03-12T05:54:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.343+0000 7fab6affd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab6c101510 0x7fab6c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.343+0000 7fab6affd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab6c101510 0x7fab6c198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60610/0 (socket says 172.21.15.76:60610) 2024-03-12T05:54:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.343+0000 7fab6affd700 1 -- 172.21.15.76:0/3106461885 learned_addr learned my addr 172.21.15.76:0/3106461885 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.343+0000 7fab6b7fe700 1 --2- 172.21.15.76:0/3106461885 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab6c100b50 0x7fab6c198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.344+0000 7fab6affd700 1 -- 172.21.15.76:0/3106461885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab6c105ab0 msgr2=0x7fab6c19ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.344+0000 7fab6bfff700 1 --2- 172.21.15.76:0/3106461885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab6c105ab0 0x7fab6c19ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.344+0000 7fab6affd700 1 --2- 172.21.15.76:0/3106461885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab6c105ab0 0x7fab6c19ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.344+0000 7fab6affd700 1 -- 172.21.15.76:0/3106461885 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab6c100b50 msgr2=0x7fab6c198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.344+0000 7fab6affd700 1 --2- 172.21.15.76:0/3106461885 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab6c100b50 0x7fab6c198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:22.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.344+0000 7fab6affd700 1 -- 172.21.15.76:0/3106461885 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab64009d00 con 0x7fab6c101510 2024-03-12T05:54:22.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.344+0000 7fab6affd700 1 --2- 172.21.15.76:0/3106461885 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab6c101510 0x7fab6c198750 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fab6000c9b0 tx=0x7fab6000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:22.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.344+0000 7fab68ff9700 1 -- 172.21.15.76:0/3106461885 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60007b00 con 0x7fab6c101510 2024-03-12T05:54:22.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.344+0000 7fab68ff9700 1 -- 172.21.15.76:0/3106461885 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab60007c60 con 0x7fab6c101510 2024-03-12T05:54:22.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.344+0000 7fab68ff9700 1 -- 172.21.15.76:0/3106461885 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60018660 con 0x7fab6c101510 2024-03-12T05:54:22.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.344+0000 7fab71f6d700 1 -- 172.21.15.76:0/3106461885 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6c19d6b0 con 0x7fab6c101510 2024-03-12T05:54:22.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.344+0000 7fab71f6d700 1 -- 172.21.15.76:0/3106461885 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6c1a4f10 con 0x7fab6c101510 2024-03-12T05:54:22.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.345+0000 7fab71f6d700 1 -- 172.21.15.76:0/3106461885 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fab6c110ee0 con 0x7fab6c101510 2024-03-12T05:54:22.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.346+0000 7fab68ff9700 1 -- 172.21.15.76:0/3106461885 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fab6000f450 con 0x7fab6c101510 2024-03-12T05:54:22.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.346+0000 7fab68ff9700 1 --2- 172.21.15.76:0/3106461885 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fab540a1b50 0x7fab540a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:22.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.347+0000 7fab6b7fe700 1 --2- 172.21.15.76:0/3106461885 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fab540a1b50 0x7fab540a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:22.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.347+0000 7fab68ff9700 1 -- 172.21.15.76:0/3106461885 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fab600cd840 con 0x7fab6c101510 2024-03-12T05:54:22.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.347+0000 7fab68ff9700 1 --2- 172.21.15.76:0/3106461885 >> [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935] conn(0x7fab540ab330 0x7fab540ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:22.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.347+0000 7fab68ff9700 1 -- 172.21.15.76:0/3106461885 --> [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab540ade00 con 0x7fab540ab330 2024-03-12T05:54:22.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.347+0000 7fab68ff9700 1 -- 172.21.15.76:0/3106461885 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab600cdce0 con 0x7fab6c101510 2024-03-12T05:54:22.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.347+0000 7fab6bfff700 1 --2- 172.21.15.76:0/3106461885 >> [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935] conn(0x7fab540ab330 0x7fab540ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:22.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.347+0000 7fab6b7fe700 1 --2- 172.21.15.76:0/3106461885 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fab540a1b50 0x7fab540a4010 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fab5c005950 tx=0x7fab5c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:22.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.347+0000 7fab6bfff700 1 --2- 172.21.15.76:0/3106461885 >> [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935] conn(0x7fab540ab330 0x7fab540ad750 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:22.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.348+0000 7fab68ff9700 1 -- 172.21.15.76:0/3106461885 <== osd.6 v2:172.21.15.76:6818/4101985935 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fab540ade00 con 0x7fab540ab330 2024-03-12T05:54:22.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.364+0000 7fab71f6d700 1 -- 172.21.15.76:0/3106461885 --> [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fab6c06a660 con 0x7fab540ab330 2024-03-12T05:54:22.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.365+0000 7fab68ff9700 1 -- 172.21.15.76:0/3106461885 <== osd.6 v2:172.21.15.76:6818/4101985935 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fab6c06a660 con 0x7fab540ab330 2024-03-12T05:54:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.365+0000 7fab71f6d700 1 -- 172.21.15.76:0/3106461885 >> [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935] conn(0x7fab540ab330 msgr2=0x7fab540ad750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.365+0000 7fab71f6d700 1 --2- 172.21.15.76:0/3106461885 >> [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935] conn(0x7fab540ab330 0x7fab540ad750 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.365+0000 7fab71f6d700 1 -- 172.21.15.76:0/3106461885 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fab540a1b50 msgr2=0x7fab540a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.365+0000 7fab71f6d700 1 --2- 172.21.15.76:0/3106461885 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fab540a1b50 0x7fab540a4010 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fab5c005950 tx=0x7fab5c00b410 comp rx=0 tx=0).stop 2024-03-12T05:54:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.365+0000 7fab71f6d700 1 -- 172.21.15.76:0/3106461885 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab6c101510 msgr2=0x7fab6c198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.365+0000 7fab71f6d700 1 --2- 172.21.15.76:0/3106461885 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab6c101510 0x7fab6c198750 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fab6000c9b0 tx=0x7fab6000ccc0 comp rx=0 tx=0).stop 2024-03-12T05:54:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.365+0000 7fab6bfff700 1 -- 172.21.15.76:0/3106461885 reap_dead start 2024-03-12T05:54:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.365+0000 7fab71f6d700 1 -- 172.21.15.76:0/3106461885 shutdown_connections 2024-03-12T05:54:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.365+0000 7fab71f6d700 1 -- 172.21.15.76:0/3106461885 >> 172.21.15.76:0/3106461885 conn(0x7fab6c077b80 msgr2=0x7fab6c1048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.365+0000 7fab71f6d700 1 -- 172.21.15.76:0/3106461885 shutdown_connections 2024-03-12T05:54:22.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:22.365+0000 7fab71f6d700 1 -- 172.21.15.76:0/3106461885 wait complete. 2024-03-12T05:54:22.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:21 smithi082 ceph-mon[36382]: pgmap v306: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-12T05:54:23.178 INFO:teuthology.orchestra.run.smithi076.stdout:107374182461 2024-03-12T05:54:23.178 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-12T05:54:23.770 INFO:teuthology.orchestra.run.smithi076.stdout:197568495663 2024-03-12T05:54:23.771 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-12T05:54:24.113 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:23 smithi076 ceph-mon[32095]: pgmap v307: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:54:24.124 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:24.128 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:24.130 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:24.182 INFO:teuthology.orchestra.run.smithi076.stdout:141733920825 2024-03-12T05:54:24.183 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-12T05:54:24.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:23 smithi088 ceph-mon[36585]: pgmap v307: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:54:24.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:23 smithi082 ceph-mon[36382]: pgmap v307: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:54:24.464 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:24.464 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:24.476 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:25.806 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:25.806 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:26.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:25 smithi088 ceph-mon[36585]: pgmap v308: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-12T05:54:26.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:25 smithi076 ceph-mon[32095]: pgmap v308: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-12T05:54:26.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:25 smithi082 ceph-mon[36382]: pgmap v308: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-12T05:54:28.150 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:27 smithi076 ceph-mon[32095]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:54:28.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:27 smithi088 ceph-mon[36585]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:54:28.247 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:27 smithi082 ceph-mon[36382]: pgmap v309: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:54:30.105 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:29 smithi076 ceph-mon[32095]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:54:30.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:29 smithi088 ceph-mon[36585]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:54:30.439 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:29 smithi082 ceph-mon[36382]: pgmap v310: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:54:31.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.707+0000 7fedef5b3700 1 -- 172.21.15.76:0/4059707426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede8101cc0 msgr2=0x7fede8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:31.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.707+0000 7fedef5b3700 1 --2- 172.21.15.76:0/4059707426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede8101cc0 0x7fede8102140 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fedd80099c0 tx=0x7fedd8009cd0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.707+0000 7fedef5b3700 1 -- 172.21.15.76:0/4059707426 shutdown_connections 2024-03-12T05:54:31.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.707+0000 7fedef5b3700 1 --2- 172.21.15.76:0/4059707426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fede8102680 0x7fede810eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.707+0000 7fedef5b3700 1 --2- 172.21.15.76:0/4059707426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fede8107ca0 0x7fede8108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.707+0000 7fedef5b3700 1 --2- 172.21.15.76:0/4059707426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede8101cc0 0x7fede8102140 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.707+0000 7fedef5b3700 1 -- 172.21.15.76:0/4059707426 >> 172.21.15.76:0/4059707426 conn(0x7fede8078da0 msgr2=0x7fede80791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:31.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.707+0000 7fedef5b3700 1 -- 172.21.15.76:0/4059707426 shutdown_connections 2024-03-12T05:54:31.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.707+0000 7fedef5b3700 1 -- 172.21.15.76:0/4059707426 wait complete. 2024-03-12T05:54:31.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.707+0000 7fedef5b3700 1 Processor -- start 2024-03-12T05:54:31.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedef5b3700 1 -- start start 2024-03-12T05:54:31.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedef5b3700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fede8101cc0 0x7fede8198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:31.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedef5b3700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede8102680 0x7fede8198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:31.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedef5b3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fede8107ca0 0x7fede819ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:31.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedef5b3700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede819d030 con 0x7fede8102680 2024-03-12T05:54:31.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedef5b3700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede819d1a0 con 0x7fede8101cc0 2024-03-12T05:54:31.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedef5b3700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede819d360 con 0x7fede8107ca0 2024-03-12T05:54:31.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedecb4e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede8102680 0x7fede8198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:31.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedecb4e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede8102680 0x7fede8198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34450/0 (socket says 172.21.15.76:34450) 2024-03-12T05:54:31.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedecb4e700 1 -- 172.21.15.76:0/2805084343 learned_addr learned my addr 172.21.15.76:0/2805084343 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:31.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7feded34f700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fede8101cc0 0x7fede8198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fededb50700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fede8107ca0 0x7fede819ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedecb4e700 1 -- 172.21.15.76:0/2805084343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fede8107ca0 msgr2=0x7fede819ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedecb4e700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fede8107ca0 0x7fede819ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedecb4e700 1 -- 172.21.15.76:0/2805084343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fede8101cc0 msgr2=0x7fede8198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedecb4e700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fede8101cc0 0x7fede8198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedecb4e700 1 -- 172.21.15.76:0/2805084343 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd80095f0 con 0x7fede8102680 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7feded34f700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fede8101cc0 0x7fede8198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fededb50700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fede8107ca0 0x7fede819ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedecb4e700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede8102680 0x7fede8198690 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fedd800b580 tx=0x7fedd8003710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedde7fc700 1 -- 172.21.15.76:0/2805084343 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd801d070 con 0x7fede8102680 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedef5b3700 1 -- 172.21.15.76:0/2805084343 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede819d5e0 con 0x7fede8102680 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.708+0000 7fedde7fc700 1 -- 172.21.15.76:0/2805084343 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedd80040d0 con 0x7fede8102680 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.709+0000 7fedde7fc700 1 -- 172.21.15.76:0/2805084343 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd800f460 con 0x7fede8102680 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.709+0000 7fedef5b3700 1 -- 172.21.15.76:0/2805084343 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede81a4ea0 con 0x7fede8102680 2024-03-12T05:54:31.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.709+0000 7fedef5b3700 1 -- 172.21.15.76:0/2805084343 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede810a180 con 0x7fede8102680 2024-03-12T05:54:31.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.710+0000 7fedde7fc700 1 -- 172.21.15.76:0/2805084343 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fedd8022720 con 0x7fede8102680 2024-03-12T05:54:31.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.711+0000 7fedde7fc700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fedd40a1b50 0x7fedd40a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:31.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.711+0000 7feded34f700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fedd40a1b50 0x7fedd40a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:31.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.711+0000 7fedde7fc700 1 -- 172.21.15.76:0/2805084343 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fedd80ce710 con 0x7fede8102680 2024-03-12T05:54:31.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.711+0000 7feded34f700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fedd40a1b50 0x7fedd40a4010 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fede4005950 tx=0x7fede4005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:31.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.716+0000 7fedde7fc700 1 -- 172.21.15.76:0/2805084343 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fedd805dd20 con 0x7fede8102680 2024-03-12T05:54:31.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.921+0000 7fedef5b3700 1 -- 172.21.15.76:0/2805084343 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fede81994b0 con 0x7fede8102680 2024-03-12T05:54:31.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.921+0000 7fedde7fc700 1 -- 172.21.15.76:0/2805084343 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fedd8097a00 con 0x7fede8102680 2024-03-12T05:54:31.925 INFO:teuthology.orchestra.run.smithi076.stdout:77309411395 2024-03-12T05:54:31.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.922+0000 7f5a8c43c700 1 -- 172.21.15.76:0/4141179630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a8406a850 msgr2=0x7f5a8406acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:31.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.922+0000 7f5a8c43c700 1 --2- 172.21.15.76:0/4141179630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a8406a850 0x7f5a8406acb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5a74009990 tx=0x7f5a74009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.922+0000 7f5a8c43c700 1 -- 172.21.15.76:0/4141179630 shutdown_connections 2024-03-12T05:54:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.922+0000 7f5a8c43c700 1 --2- 172.21.15.76:0/4141179630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a8410e550 0x7f5a84110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.922+0000 7f5a8c43c700 1 --2- 172.21.15.76:0/4141179630 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a84102bd0 0x7f5a8406a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.922+0000 7f5a8c43c700 1 --2- 172.21.15.76:0/4141179630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a8406a850 0x7f5a8406acb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.922+0000 7f5a8c43c700 1 -- 172.21.15.76:0/4141179630 >> 172.21.15.76:0/4141179630 conn(0x7f5a84077a30 msgr2=0x7f5a84077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.922+0000 7f5a8c43c700 1 -- 172.21.15.76:0/4141179630 shutdown_connections 2024-03-12T05:54:31.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.922+0000 7f5a8c43c700 1 -- 172.21.15.76:0/4141179630 wait complete. 2024-03-12T05:54:31.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.923+0000 7f5a8c43c700 1 Processor -- start 2024-03-12T05:54:31.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.923+0000 7f5a8c43c700 1 -- start start 2024-03-12T05:54:31.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.923+0000 7f5a8c43c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a8406a850 0x7f5a8419a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:31.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.923+0000 7f5a8c43c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a84102bd0 0x7f5a8419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:31.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.923+0000 7f5a8c43c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a8410e550 0x7f5a8419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:31.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.923+0000 7f5a8c43c700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8419f2c0 con 0x7f5a8410e550 2024-03-12T05:54:31.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.923+0000 7f5a8c43c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8419f430 con 0x7f5a8406a850 2024-03-12T05:54:31.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.923+0000 7f5a8c43c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8419f5f0 con 0x7f5a84102bd0 2024-03-12T05:54:31.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.923+0000 7f5a8a9d9700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a8410e550 0x7f5a8419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:31.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.923+0000 7f5a8a9d9700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a8410e550 0x7f5a8419ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34474/0 (socket says 172.21.15.76:34474) 2024-03-12T05:54:31.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.923+0000 7f5a8a9d9700 1 -- 172.21.15.76:0/701888983 learned_addr learned my addr 172.21.15.76:0/701888983 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:31.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.923+0000 7f5a899d7700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a84102bd0 0x7f5a8419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:31.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.923+0000 7f5a8a1d8700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a8406a850 0x7f5a8419a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:31.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7fedef5b3700 1 -- 172.21.15.76:0/2805084343 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fedd40a1b50 msgr2=0x7fedd40a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7fedef5b3700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fedd40a1b50 0x7fedd40a4010 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fede4005950 tx=0x7fede4005f50 comp rx=0 tx=0).stop 2024-03-12T05:54:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7fedef5b3700 1 -- 172.21.15.76:0/2805084343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede8102680 msgr2=0x7fede8198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7fedef5b3700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede8102680 0x7fede8198690 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fedd800b580 tx=0x7fedd8003710 comp rx=0 tx=0).stop 2024-03-12T05:54:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7fedef5b3700 1 -- 172.21.15.76:0/2805084343 shutdown_connections 2024-03-12T05:54:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7f5a899d7700 1 -- 172.21.15.76:0/701888983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a8406a850 msgr2=0x7f5a8419a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7f5a899d7700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a8406a850 0x7f5a8419a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7f5a899d7700 1 -- 172.21.15.76:0/701888983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a8410e550 msgr2=0x7f5a8419ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7fedef5b3700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fedd40a1b50 0x7fedd40a4010 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7fedef5b3700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fede8107ca0 0x7fede819ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7fedef5b3700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fede8101cc0 0x7fede8198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7fedef5b3700 1 --2- 172.21.15.76:0/2805084343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede8102680 0x7fede8198690 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7fedef5b3700 1 -- 172.21.15.76:0/2805084343 >> 172.21.15.76:0/2805084343 conn(0x7fede8078da0 msgr2=0x7fede8100910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7f5a899d7700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a8410e550 0x7f5a8419ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7f5a899d7700 1 -- 172.21.15.76:0/701888983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a740095f0 con 0x7f5a84102bd0 2024-03-12T05:54:31.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7f5a8a9d9700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a8410e550 0x7f5a8419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:54:31.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7fedef5b3700 1 -- 172.21.15.76:0/2805084343 shutdown_connections 2024-03-12T05:54:31.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7fedef5b3700 1 -- 172.21.15.76:0/2805084343 wait complete. 2024-03-12T05:54:31.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7f5a899d7700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a84102bd0 0x7f5a8419a920 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5a74006010 tx=0x7f5a74005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:31.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7f5a7b7fe700 1 -- 172.21.15.76:0/701888983 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7401d070 con 0x7f5a84102bd0 2024-03-12T05:54:31.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7f5a8c43c700 1 -- 172.21.15.76:0/701888983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8419f870 con 0x7f5a84102bd0 2024-03-12T05:54:31.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7f5a7b7fe700 1 -- 172.21.15.76:0/701888983 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a7400b610 con 0x7f5a84102bd0 2024-03-12T05:54:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7f5a8c43c700 1 -- 172.21.15.76:0/701888983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a841a7130 con 0x7f5a84102bd0 2024-03-12T05:54:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.924+0000 7f5a7b7fe700 1 -- 172.21.15.76:0/701888983 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a74022e90 con 0x7f5a84102bd0 2024-03-12T05:54:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.925+0000 7f5a8c43c700 1 -- 172.21.15.76:0/701888983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a841011d0 con 0x7f5a84102bd0 2024-03-12T05:54:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.927+0000 7f5a7b7fe700 1 -- 172.21.15.76:0/701888983 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f5a7400fad0 con 0x7f5a84102bd0 2024-03-12T05:54:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.928+0000 7f5a7b7fe700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5a700a1b50 0x7f5a700a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.928+0000 7f5a7b7fe700 1 -- 172.21.15.76:0/701888983 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5a740ce660 con 0x7f5a84102bd0 2024-03-12T05:54:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.928+0000 7f5a8a1d8700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5a700a1b50 0x7f5a700a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:31.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.929+0000 7f5a8a1d8700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5a700a1b50 0x7f5a700a4010 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009c60 tx=0x7f5a80009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:31.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:31.931+0000 7f5a7b7fe700 1 -- 172.21.15.76:0/701888983 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5a74097950 con 0x7f5a84102bd0 2024-03-12T05:54:32.102 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:31 smithi076 ceph-mon[32095]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:54:32.102 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:31 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2805084343' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T05:54:32.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:32.137+0000 7f5a8c43c700 1 -- 172.21.15.76:0/701888983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f5a8419b7a0 con 0x7f5a84102bd0 2024-03-12T05:54:32.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:32.138+0000 7f5a7b7fe700 1 -- 172.21.15.76:0/701888983 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5a7409b010 con 0x7f5a84102bd0 2024-03-12T05:54:32.141 INFO:teuthology.orchestra.run.smithi076.stdout:107374182463 2024-03-12T05:54:32.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:32.140+0000 7f5a8c43c700 1 -- 172.21.15.76:0/701888983 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5a700a1b50 msgr2=0x7f5a700a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:32.140+0000 7f5a8c43c700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5a700a1b50 0x7f5a700a4010 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5a80009c60 tx=0x7f5a80009260 comp rx=0 tx=0).stop 2024-03-12T05:54:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:32.140+0000 7f5a8c43c700 1 -- 172.21.15.76:0/701888983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a84102bd0 msgr2=0x7f5a8419a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:32.140+0000 7f5a8c43c700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a84102bd0 0x7f5a8419a920 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5a74006010 tx=0x7f5a74005740 comp rx=0 tx=0).stop 2024-03-12T05:54:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:32.140+0000 7f5a8c43c700 1 -- 172.21.15.76:0/701888983 shutdown_connections 2024-03-12T05:54:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:32.140+0000 7f5a8c43c700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5a700a1b50 0x7f5a700a4010 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:32.140+0000 7f5a8c43c700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a8410e550 0x7f5a8419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:32.140+0000 7f5a8c43c700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a8406a850 0x7f5a8419a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:32.140+0000 7f5a8c43c700 1 --2- 172.21.15.76:0/701888983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a84102bd0 0x7f5a8419a920 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:32.140+0000 7f5a8c43c700 1 -- 172.21.15.76:0/701888983 >> 172.21.15.76:0/701888983 conn(0x7f5a84077a30 msgr2=0x7f5a840ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:32.140+0000 7f5a8c43c700 1 -- 172.21.15.76:0/701888983 shutdown_connections 2024-03-12T05:54:32.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:32.140+0000 7f5a8c43c700 1 -- 172.21.15.76:0/701888983 wait complete. 2024-03-12T05:54:32.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:31 smithi082 ceph-mon[36382]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:54:32.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:31 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2805084343' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T05:54:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:31 smithi088 ceph-mon[36585]: pgmap v311: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:54:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:31 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2805084343' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T05:54:32.944 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411390 got 77309411395 for osd.2 2024-03-12T05:54:32.944 DEBUG:teuthology.parallel:result is None 2024-03-12T05:54:33.071 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:32 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/701888983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T05:54:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:33.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:32 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/701888983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T05:54:33.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:32 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/701888983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T05:54:33.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.633+0000 7f3b74b73700 1 -- 172.21.15.76:0/3137452331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b70106ce0 msgr2=0x7f3b701070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:33.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.633+0000 7f3b74b73700 1 --2- 172.21.15.76:0/3137452331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b70106ce0 0x7f3b701070c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009990 tx=0x7f3b60009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:33.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.633+0000 7f3b74b73700 1 -- 172.21.15.76:0/3137452331 shutdown_connections 2024-03-12T05:54:33.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.633+0000 7f3b74b73700 1 --2- 172.21.15.76:0/3137452331 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b70100c20 0x7f3b7010eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:33.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.633+0000 7f3b74b73700 1 --2- 172.21.15.76:0/3137452331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b70106ce0 0x7f3b701070c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:33.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.633+0000 7f3b74b73700 1 --2- 172.21.15.76:0/3137452331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b70100260 0x7f3b701006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:33.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.633+0000 7f3b74b73700 1 -- 172.21.15.76:0/3137452331 >> 172.21.15.76:0/3137452331 conn(0x7f3b70077170 msgr2=0x7f3b700fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:33.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.633+0000 7f3b74b73700 1 -- 172.21.15.76:0/3137452331 shutdown_connections 2024-03-12T05:54:33.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.633+0000 7f3b74b73700 1 -- 172.21.15.76:0/3137452331 wait complete. 2024-03-12T05:54:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b74b73700 1 Processor -- start 2024-03-12T05:54:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b74b73700 1 -- start start 2024-03-12T05:54:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b74b73700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b70100260 0x7f3b70198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b74b73700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b70100c20 0x7f3b701987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b74b73700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b70106ce0 0x7f3b7019cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b74b73700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7019d120 con 0x7f3b70100260 2024-03-12T05:54:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b74b73700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7019d290 con 0x7f3b70106ce0 2024-03-12T05:54:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b74b73700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7019d450 con 0x7f3b70100c20 2024-03-12T05:54:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b6e59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b70100260 0x7f3b70198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b6e59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b70100260 0x7f3b70198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34514/0 (socket says 172.21.15.76:34514) 2024-03-12T05:54:33.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b6e59c700 1 -- 172.21.15.76:0/793600807 learned_addr learned my addr 172.21.15.76:0/793600807 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b6dd9b700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b70100c20 0x7f3b701987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b6ed9d700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b70106ce0 0x7f3b7019cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b6e59c700 1 -- 172.21.15.76:0/793600807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b70100c20 msgr2=0x7f3b701987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b6e59c700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b70100c20 0x7f3b701987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b6e59c700 1 -- 172.21.15.76:0/793600807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b70106ce0 msgr2=0x7f3b7019cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b6e59c700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b70106ce0 0x7f3b7019cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b6e59c700 1 -- 172.21.15.76:0/793600807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b600095f0 con 0x7f3b70100260 2024-03-12T05:54:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b6ed9d700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b70106ce0 0x7f3b7019cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:54:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.634+0000 7f3b6e59c700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b70100260 0x7f3b70198290 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3b6000b2e0 tx=0x7f3b60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.635+0000 7f3b5f7fe700 1 -- 172.21.15.76:0/793600807 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6001d070 con 0x7f3b70100260 2024-03-12T05:54:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.635+0000 7f3b5f7fe700 1 -- 172.21.15.76:0/793600807 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b6000b610 con 0x7f3b70100260 2024-03-12T05:54:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.635+0000 7f3b5f7fe700 1 -- 172.21.15.76:0/793600807 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b600173f0 con 0x7f3b70100260 2024-03-12T05:54:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.635+0000 7f3b74b73700 1 -- 172.21.15.76:0/793600807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b7019d6d0 con 0x7f3b70100260 2024-03-12T05:54:33.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.635+0000 7f3b74b73700 1 -- 172.21.15.76:0/793600807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b701a4dc0 con 0x7f3b70100260 2024-03-12T05:54:33.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.636+0000 7f3b74b73700 1 -- 172.21.15.76:0/793600807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b7010a2b0 con 0x7f3b70100260 2024-03-12T05:54:33.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.637+0000 7f3b5f7fe700 1 -- 172.21.15.76:0/793600807 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f3b60021410 con 0x7f3b70100260 2024-03-12T05:54:33.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.637+0000 7f3b5f7fe700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3b580a1bd0 0x7f3b580a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:33.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.637+0000 7f3b6dd9b700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3b580a1bd0 0x7f3b580a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:33.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.637+0000 7f3b5f7fe700 1 -- 172.21.15.76:0/793600807 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3b600ce910 con 0x7f3b70100260 2024-03-12T05:54:33.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.638+0000 7f3b6dd9b700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3b580a1bd0 0x7f3b580a4090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3b700ff7d0 tx=0x7f3b64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:33.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.641+0000 7f3b5f7fe700 1 -- 172.21.15.76:0/793600807 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3b60097c00 con 0x7f3b70100260 2024-03-12T05:54:33.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.844+0000 7f3b74b73700 1 -- 172.21.15.76:0/793600807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3b701995a0 con 0x7f3b70100260 2024-03-12T05:54:33.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.845+0000 7f3b5f7fe700 1 -- 172.21.15.76:0/793600807 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3b6009b2c0 con 0x7f3b70100260 2024-03-12T05:54:33.849 INFO:teuthology.orchestra.run.smithi076.stdout:163208757302 2024-03-12T05:54:33.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.847+0000 7f3b74b73700 1 -- 172.21.15.76:0/793600807 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3b580a1bd0 msgr2=0x7f3b580a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:33.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.847+0000 7f3b74b73700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3b580a1bd0 0x7f3b580a4090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3b700ff7d0 tx=0x7f3b64009260 comp rx=0 tx=0).stop 2024-03-12T05:54:33.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.847+0000 7f3b74b73700 1 -- 172.21.15.76:0/793600807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b70100260 msgr2=0x7f3b70198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:33.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.847+0000 7f3b74b73700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b70100260 0x7f3b70198290 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3b6000b2e0 tx=0x7f3b60004730 comp rx=0 tx=0).stop 2024-03-12T05:54:33.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.847+0000 7f3b74b73700 1 -- 172.21.15.76:0/793600807 shutdown_connections 2024-03-12T05:54:33.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.847+0000 7f3b74b73700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3b580a1bd0 0x7f3b580a4090 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:33.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.847+0000 7f3b74b73700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3b70106ce0 0x7f3b7019cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:33.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.847+0000 7f3b74b73700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3b70100260 0x7f3b70198290 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:33.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.847+0000 7f3b74b73700 1 --2- 172.21.15.76:0/793600807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b70100c20 0x7f3b701987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:33.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.847+0000 7f3b74b73700 1 -- 172.21.15.76:0/793600807 >> 172.21.15.76:0/793600807 conn(0x7f3b70077170 msgr2=0x7f3b70104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:33.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.847+0000 7f3b74b73700 1 -- 172.21.15.76:0/793600807 shutdown_connections 2024-03-12T05:54:33.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.847+0000 7f3b74b73700 1 -- 172.21.15.76:0/793600807 wait complete. 2024-03-12T05:54:34.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:33 smithi076 ceph-mon[32095]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:54:34.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:33 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2643645109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T05:54:34.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:33 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/793600807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T05:54:34.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.297+0000 7f6aa328a700 1 -- 172.21.15.76:0/281283862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c1017b0 msgr2=0x7f6a9c1057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:34.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.297+0000 7f6aa328a700 1 --2- 172.21.15.76:0/281283862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c1017b0 0x7f6a9c1057b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009990 tx=0x7f6a98009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:34.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.297+0000 7f6aa328a700 1 -- 172.21.15.76:0/281283862 shutdown_connections 2024-03-12T05:54:34.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.297+0000 7f6aa328a700 1 --2- 172.21.15.76:0/281283862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a9c105e90 0x7f6a9c10da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:34.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.297+0000 7f6aa328a700 1 --2- 172.21.15.76:0/281283862 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a9c100e00 0x7f6a9c1011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:34.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.297+0000 7f6aa328a700 1 --2- 172.21.15.76:0/281283862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c1017b0 0x7f6a9c1057b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:34.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.297+0000 7f6aa328a700 1 -- 172.21.15.76:0/281283862 >> 172.21.15.76:0/281283862 conn(0x7f6a9c076b90 msgr2=0x7f6a9c076fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:34.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.297+0000 7f6aa328a700 1 -- 172.21.15.76:0/281283862 shutdown_connections 2024-03-12T05:54:34.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.297+0000 7f6aa328a700 1 -- 172.21.15.76:0/281283862 wait complete. 2024-03-12T05:54:34.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.298+0000 7f6aa328a700 1 Processor -- start 2024-03-12T05:54:34.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.298+0000 7f6aa328a700 1 -- start start 2024-03-12T05:54:34.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.298+0000 7f6aa328a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c100e00 0x7f6a9c198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:34.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.298+0000 7f6aa328a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a9c1017b0 0x7f6a9c1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:34.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.298+0000 7f6aa328a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a9c105e90 0x7f6a9c19cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.298+0000 7f6aa328a700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c19d150 con 0x7f6a9c100e00 2024-03-12T05:54:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.298+0000 7f6aa328a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c19d2c0 con 0x7f6a9c1017b0 2024-03-12T05:54:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.298+0000 7f6aa328a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c19d480 con 0x7f6a9c105e90 2024-03-12T05:54:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.298+0000 7f6aa1026700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c100e00 0x7f6a9c198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.298+0000 7f6aa1026700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c100e00 0x7f6a9c198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34542/0 (socket says 172.21.15.76:34542) 2024-03-12T05:54:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.298+0000 7f6aa1026700 1 -- 172.21.15.76:0/3572473527 learned_addr learned my addr 172.21.15.76:0/3572473527 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.299+0000 7f6aa0825700 1 --2- 172.21.15.76:0/3572473527 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a9c1017b0 0x7f6a9c1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.299+0000 7f6aa1026700 1 -- 172.21.15.76:0/3572473527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a9c105e90 msgr2=0x7f6a9c19cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.299+0000 7f6aa1026700 1 --2- 172.21.15.76:0/3572473527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a9c105e90 0x7f6a9c19cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.299+0000 7f6aa1026700 1 -- 172.21.15.76:0/3572473527 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a9c1017b0 msgr2=0x7f6a9c1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.299+0000 7f6aa1026700 1 --2- 172.21.15.76:0/3572473527 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a9c1017b0 0x7f6a9c1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.299+0000 7f6aa1026700 1 -- 172.21.15.76:0/3572473527 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a980095f0 con 0x7f6a9c100e00 2024-03-12T05:54:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.299+0000 7f6aa1026700 1 --2- 172.21.15.76:0/3572473527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c100e00 0x7f6a9c198270 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00c920 tx=0x7f6a8c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.299+0000 7f6a927fc700 1 -- 172.21.15.76:0/3572473527 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c007b00 con 0x7f6a9c100e00 2024-03-12T05:54:34.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.299+0000 7f6a927fc700 1 -- 172.21.15.76:0/3572473527 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a8c00ce70 con 0x7f6a9c100e00 2024-03-12T05:54:34.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.299+0000 7f6aa328a700 1 -- 172.21.15.76:0/3572473527 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9c19d760 con 0x7f6a9c100e00 2024-03-12T05:54:34.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.299+0000 7f6a927fc700 1 -- 172.21.15.76:0/3572473527 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c005610 con 0x7f6a9c100e00 2024-03-12T05:54:34.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.299+0000 7f6aa328a700 1 -- 172.21.15.76:0/3572473527 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a9c1a4fc0 con 0x7f6a9c100e00 2024-03-12T05:54:34.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.300+0000 7f6aa328a700 1 -- 172.21.15.76:0/3572473527 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a9c109140 con 0x7f6a9c100e00 2024-03-12T05:54:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.301+0000 7f6a927fc700 1 -- 172.21.15.76:0/3572473527 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f6a8c004750 con 0x7f6a9c100e00 2024-03-12T05:54:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.302+0000 7f6a927fc700 1 --2- 172.21.15.76:0/3572473527 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6a880a1bd0 0x7f6a880a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.302+0000 7f6aa0825700 1 --2- 172.21.15.76:0/3572473527 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6a880a1bd0 0x7f6a880a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.302+0000 7f6a927fc700 1 -- 172.21.15.76:0/3572473527 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6a8c0cd400 con 0x7f6a9c100e00 2024-03-12T05:54:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.302+0000 7f6aa0825700 1 --2- 172.21.15.76:0/3572473527 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6a880a1bd0 0x7f6a880a4090 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6a98006010 tx=0x7f6a98005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:34.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.305+0000 7f6a927fc700 1 -- 172.21.15.76:0/3572473527 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6a8c0966e0 con 0x7f6a9c100e00 2024-03-12T05:54:34.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:33 smithi082 ceph-mon[36382]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:54:34.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:33 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2643645109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T05:54:34.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:33 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/793600807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T05:54:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:33 smithi088 ceph-mon[36585]: pgmap v312: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:54:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:33 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2643645109' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T05:54:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:33 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/793600807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T05:54:34.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.511+0000 7f6aa328a700 1 -- 172.21.15.76:0/3572473527 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f6a9c111d40 con 0x7f6a9c100e00 2024-03-12T05:54:34.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.512+0000 7f6a927fc700 1 -- 172.21.15.76:0/3572473527 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f6a8c01d020 con 0x7f6a9c100e00 2024-03-12T05:54:34.515 INFO:teuthology.orchestra.run.smithi076.stdout:313532612637 2024-03-12T05:54:34.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.514+0000 7f6aa328a700 1 -- 172.21.15.76:0/3572473527 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6a880a1bd0 msgr2=0x7f6a880a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:34.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.514+0000 7f6aa328a700 1 --2- 172.21.15.76:0/3572473527 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6a880a1bd0 0x7f6a880a4090 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6a98006010 tx=0x7f6a98005960 comp rx=0 tx=0).stop 2024-03-12T05:54:34.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.514+0000 7f6aa328a700 1 -- 172.21.15.76:0/3572473527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c100e00 msgr2=0x7f6a9c198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:34.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.514+0000 7f6aa328a700 1 --2- 172.21.15.76:0/3572473527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c100e00 0x7f6a9c198270 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00c920 tx=0x7f6a8c00cc30 comp rx=0 tx=0).stop 2024-03-12T05:54:34.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.514+0000 7f6aa328a700 1 -- 172.21.15.76:0/3572473527 shutdown_connections 2024-03-12T05:54:34.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.514+0000 7f6aa328a700 1 --2- 172.21.15.76:0/3572473527 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6a880a1bd0 0x7f6a880a4090 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:34.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.514+0000 7f6aa328a700 1 --2- 172.21.15.76:0/3572473527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a9c105e90 0x7f6a9c19cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:34.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.514+0000 7f6aa328a700 1 --2- 172.21.15.76:0/3572473527 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a9c100e00 0x7f6a9c198270 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:34.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.514+0000 7f6aa328a700 1 --2- 172.21.15.76:0/3572473527 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6a9c1017b0 0x7f6a9c1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:34.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.514+0000 7f6aa328a700 1 -- 172.21.15.76:0/3572473527 >> 172.21.15.76:0/3572473527 conn(0x7f6a9c076b90 msgr2=0x7f6a9c100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:34.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.514+0000 7f6aa328a700 1 -- 172.21.15.76:0/3572473527 shutdown_connections 2024-03-12T05:54:34.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:34.514+0000 7f6aa328a700 1 -- 172.21.15.76:0/3572473527 wait complete. 2024-03-12T05:54:35.186 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:34 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3572473527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-12T05:54:35.211 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612633 got 313532612637 for osd.10 2024-03-12T05:54:35.211 DEBUG:teuthology.parallel:result is None 2024-03-12T05:54:35.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:34 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3572473527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-12T05:54:35.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:34 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3572473527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-12T05:54:35.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.630+0000 7f2cd78ec700 1 -- 172.21.15.76:0/1297640143 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2cd0106bd0 msgr2=0x7f2cd0106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:35.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.630+0000 7f2cd78ec700 1 --2- 172.21.15.76:0/1297640143 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2cd0106bd0 0x7f2cd0106fb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800b0a0 tx=0x7f2cc800b3b0 comp rx=0 tx=0).stop 2024-03-12T05:54:35.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.630+0000 7f2cd78ec700 1 -- 172.21.15.76:0/1297640143 shutdown_connections 2024-03-12T05:54:35.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.630+0000 7f2cd78ec700 1 --2- 172.21.15.76:0/1297640143 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cd0101560 0x7f2cd010d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:35.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.630+0000 7f2cd78ec700 1 --2- 172.21.15.76:0/1297640143 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2cd0106bd0 0x7f2cd0106fb0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:35.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.630+0000 7f2cd78ec700 1 --2- 172.21.15.76:0/1297640143 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2cd0100ba0 0x7f2cd0101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:35.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.630+0000 7f2cd78ec700 1 -- 172.21.15.76:0/1297640143 >> 172.21.15.76:0/1297640143 conn(0x7f2cd0076bc0 msgr2=0x7f2cd0076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:35.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.630+0000 7f2cd78ec700 1 -- 172.21.15.76:0/1297640143 shutdown_connections 2024-03-12T05:54:35.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.630+0000 7f2cd78ec700 1 -- 172.21.15.76:0/1297640143 wait complete. 2024-03-12T05:54:35.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.631+0000 7f2cd78ec700 1 Processor -- start 2024-03-12T05:54:35.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.631+0000 7f2cd78ec700 1 -- start start 2024-03-12T05:54:35.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.631+0000 7f2cd78ec700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2cd0100ba0 0x7f2cd0197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:35.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.631+0000 7f2cd78ec700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cd0101560 0x7f2cd0197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:35.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.631+0000 7f2cd78ec700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2cd0106bd0 0x7f2cd019bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:35.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.631+0000 7f2cd78ec700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd019c200 con 0x7f2cd0106bd0 2024-03-12T05:54:35.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.631+0000 7f2cd78ec700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd019c370 con 0x7f2cd0100ba0 2024-03-12T05:54:35.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.631+0000 7f2cd78ec700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd019c530 con 0x7f2cd0101560 2024-03-12T05:54:35.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.631+0000 7f2cd5e89700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2cd0106bd0 0x7f2cd019bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:35.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.631+0000 7f2cd5e89700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2cd0106bd0 0x7f2cd019bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34570/0 (socket says 172.21.15.76:34570) 2024-03-12T05:54:35.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.631+0000 7f2cd5e89700 1 -- 172.21.15.76:0/3031971080 learned_addr learned my addr 172.21.15.76:0/3031971080 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:35.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.632+0000 7f2cd5688700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2cd0100ba0 0x7f2cd0197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:35.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.632+0000 7f2cd4e87700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cd0101560 0x7f2cd0197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:35.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.632+0000 7f2cd5e89700 1 -- 172.21.15.76:0/3031971080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cd0101560 msgr2=0x7f2cd0197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:35.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.632+0000 7f2cd5e89700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cd0101560 0x7f2cd0197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:35.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.632+0000 7f2cd5e89700 1 -- 172.21.15.76:0/3031971080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2cd0100ba0 msgr2=0x7f2cd0197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:35.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.632+0000 7f2cd5e89700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2cd0100ba0 0x7f2cd0197320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:35.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.632+0000 7f2cd5e89700 1 -- 172.21.15.76:0/3031971080 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc8009d00 con 0x7f2cd0106bd0 2024-03-12T05:54:35.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.632+0000 7f2cd4e87700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cd0101560 0x7f2cd0197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:54:35.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.632+0000 7f2cd5e89700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2cd0106bd0 0x7f2cd019bbf0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2cc000b550 tx=0x7f2cc000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:35.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.632+0000 7f2cce7fc700 1 -- 172.21.15.76:0/3031971080 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0011830 con 0x7f2cd0106bd0 2024-03-12T05:54:35.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.632+0000 7f2cd78ec700 1 -- 172.21.15.76:0/3031971080 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd019c810 con 0x7f2cd0106bd0 2024-03-12T05:54:35.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.632+0000 7f2cd78ec700 1 -- 172.21.15.76:0/3031971080 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd01a4070 con 0x7f2cd0106bd0 2024-03-12T05:54:35.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.632+0000 7f2cce7fc700 1 -- 172.21.15.76:0/3031971080 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2cc0011e70 con 0x7f2cd0106bd0 2024-03-12T05:54:35.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.632+0000 7f2cce7fc700 1 -- 172.21.15.76:0/3031971080 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc000f2e0 con 0x7f2cd0106bd0 2024-03-12T05:54:35.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.633+0000 7f2cd78ec700 1 -- 172.21.15.76:0/3031971080 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cbc005330 con 0x7f2cd0106bd0 2024-03-12T05:54:35.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.634+0000 7f2cce7fc700 1 -- 172.21.15.76:0/3031971080 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f2cc0011990 con 0x7f2cd0106bd0 2024-03-12T05:54:35.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.635+0000 7f2cce7fc700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2cb80a1bd0 0x7f2cb80a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:35.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.635+0000 7f2cd5688700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2cb80a1bd0 0x7f2cb80a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:35.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.635+0000 7f2cce7fc700 1 -- 172.21.15.76:0/3031971080 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2cc00ccbf0 con 0x7f2cd0106bd0 2024-03-12T05:54:35.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.635+0000 7f2cd5688700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2cb80a1bd0 0x7f2cb80a4090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800b830 tx=0x7f2cc8009be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:35.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.639+0000 7f2cce7fc700 1 -- 172.21.15.76:0/3031971080 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2cc0095f60 con 0x7f2cd0106bd0 2024-03-12T05:54:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.837+0000 7f2cd78ec700 1 -- 172.21.15.76:0/3031971080 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f2cbc0058b0 con 0x7f2cd0106bd0 2024-03-12T05:54:35.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.837+0000 7f2cce7fc700 1 -- 172.21.15.76:0/3031971080 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f2cc0099620 con 0x7f2cd0106bd0 2024-03-12T05:54:35.841 INFO:teuthology.orchestra.run.smithi076.stdout:347892351001 2024-03-12T05:54:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.839+0000 7f2cd78ec700 1 -- 172.21.15.76:0/3031971080 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2cb80a1bd0 msgr2=0x7f2cb80a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.839+0000 7f2cd78ec700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2cb80a1bd0 0x7f2cb80a4090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800b830 tx=0x7f2cc8009be0 comp rx=0 tx=0).stop 2024-03-12T05:54:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.839+0000 7f2cd78ec700 1 -- 172.21.15.76:0/3031971080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2cd0106bd0 msgr2=0x7f2cd019bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.839+0000 7f2cd78ec700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2cd0106bd0 0x7f2cd019bbf0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2cc000b550 tx=0x7f2cc000b910 comp rx=0 tx=0).stop 2024-03-12T05:54:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.839+0000 7f2cd78ec700 1 -- 172.21.15.76:0/3031971080 shutdown_connections 2024-03-12T05:54:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.839+0000 7f2cd78ec700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2cb80a1bd0 0x7f2cb80a4090 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.839+0000 7f2cd78ec700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2cd0106bd0 0x7f2cd019bbf0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.839+0000 7f2cd78ec700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2cd0100ba0 0x7f2cd0197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.839+0000 7f2cd78ec700 1 --2- 172.21.15.76:0/3031971080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cd0101560 0x7f2cd0197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.839+0000 7f2cd78ec700 1 -- 172.21.15.76:0/3031971080 >> 172.21.15.76:0/3031971080 conn(0x7f2cd0076bc0 msgr2=0x7f2cd00ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.839+0000 7f2cd78ec700 1 -- 172.21.15.76:0/3031971080 shutdown_connections 2024-03-12T05:54:35.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:35.839+0000 7f2cd78ec700 1 -- 172.21.15.76:0/3031971080 wait complete. 2024-03-12T05:54:36.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:35 smithi076 ceph-mon[32095]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-12T05:54:36.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:35 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3031971080' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-12T05:54:36.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:35 smithi082 ceph-mon[36382]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-12T05:54:36.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:35 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3031971080' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-12T05:54:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:35 smithi088 ceph-mon[36585]: pgmap v313: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-03-12T05:54:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:35 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3031971080' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-12T05:54:36.712 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182461 got 107374182463 for osd.3 2024-03-12T05:54:36.712 DEBUG:teuthology.parallel:result is None 2024-03-12T05:54:36.844 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757298 got 163208757302 for osd.5 2024-03-12T05:54:36.845 DEBUG:teuthology.parallel:result is None 2024-03-12T05:54:37.256 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:36 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:54:37.270 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:36 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:54:37.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:36 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:54:37.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.530+0000 7f7f70af0700 1 -- 172.21.15.76:0/3982696266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f6c100260 msgr2=0x7f7f6c1006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:37.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.530+0000 7f7f70af0700 1 --2- 172.21.15.76:0/3982696266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f6c100260 0x7f7f6c1006e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009990 tx=0x7f7f54009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:37.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.531+0000 7f7f70af0700 1 -- 172.21.15.76:0/3982696266 shutdown_connections 2024-03-12T05:54:37.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.531+0000 7f7f70af0700 1 --2- 172.21.15.76:0/3982696266 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f6c100c20 0x7f7f6c10eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:37.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.531+0000 7f7f70af0700 1 --2- 172.21.15.76:0/3982696266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f6c106ce0 0x7f7f6c1070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:37.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.531+0000 7f7f70af0700 1 --2- 172.21.15.76:0/3982696266 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f6c100260 0x7f7f6c1006e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:37.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.531+0000 7f7f70af0700 1 -- 172.21.15.76:0/3982696266 >> 172.21.15.76:0/3982696266 conn(0x7f7f6c077170 msgr2=0x7f7f6c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:37.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.531+0000 7f7f70af0700 1 -- 172.21.15.76:0/3982696266 shutdown_connections 2024-03-12T05:54:37.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.531+0000 7f7f70af0700 1 -- 172.21.15.76:0/3982696266 wait complete. 2024-03-12T05:54:37.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.531+0000 7f7f70af0700 1 Processor -- start 2024-03-12T05:54:37.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.531+0000 7f7f70af0700 1 -- start start 2024-03-12T05:54:37.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f70af0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f6c100260 0x7f7f6c1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:37.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f70af0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f6c100c20 0x7f7f6c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:37.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f70af0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f6c106ce0 0x7f7f6c19cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:37.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f70af0700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c19d0c0 con 0x7f7f6c100c20 2024-03-12T05:54:37.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f70af0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c19d230 con 0x7f7f6c106ce0 2024-03-12T05:54:37.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f70af0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c19d3f0 con 0x7f7f6c100260 2024-03-12T05:54:37.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f69d9b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f6c100c20 0x7f7f6c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:37.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f69d9b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f6c100c20 0x7f7f6c198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34496/0 (socket says 172.21.15.76:34496) 2024-03-12T05:54:37.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f69d9b700 1 -- 172.21.15.76:0/2643645109 learned_addr learned my addr 172.21.15.76:0/2643645109 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:37.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f6a59c700 1 --2- 172.21.15.76:0/2643645109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f6c100260 0x7f7f6c1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:37.469 INFO:teuthology.orchestra.run.smithi076.stdout:34359738446 2024-03-12T05:54:37.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f6ad9d700 1 --2- 172.21.15.76:0/2643645109 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f6c106ce0 0x7f7f6c19cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:37.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f69d9b700 1 -- 172.21.15.76:0/2643645109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f6c100260 msgr2=0x7f7f6c1981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:37.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f69d9b700 1 --2- 172.21.15.76:0/2643645109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f6c100260 0x7f7f6c1981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:37.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f69d9b700 1 -- 172.21.15.76:0/2643645109 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f6c106ce0 msgr2=0x7f7f6c19cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:37.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f69d9b700 1 --2- 172.21.15.76:0/2643645109 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f6c106ce0 0x7f7f6c19cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:37.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f69d9b700 1 -- 172.21.15.76:0/2643645109 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f540095f0 con 0x7f7f6c100c20 2024-03-12T05:54:37.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f69d9b700 1 --2- 172.21.15.76:0/2643645109 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f6c100c20 0x7f7f6c198720 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7f540098e0 tx=0x7f7f5400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:37.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.532+0000 7f7f637fe700 1 -- 172.21.15.76:0/2643645109 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5401d070 con 0x7f7f6c100c20 2024-03-12T05:54:37.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.533+0000 7f7f70af0700 1 -- 172.21.15.76:0/2643645109 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f6c19d670 con 0x7f7f6c100c20 2024-03-12T05:54:37.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.533+0000 7f7f637fe700 1 -- 172.21.15.76:0/2643645109 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f5400fc70 con 0x7f7f6c100c20 2024-03-12T05:54:37.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.533+0000 7f7f637fe700 1 -- 172.21.15.76:0/2643645109 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f540176e0 con 0x7f7f6c100c20 2024-03-12T05:54:37.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.533+0000 7f7f70af0700 1 -- 172.21.15.76:0/2643645109 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6c1a4f30 con 0x7f7f6c100c20 2024-03-12T05:54:37.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.534+0000 7f7f70af0700 1 -- 172.21.15.76:0/2643645109 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6c103580 con 0x7f7f6c100c20 2024-03-12T05:54:37.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.535+0000 7f7f637fe700 1 -- 172.21.15.76:0/2643645109 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f7f540229e0 con 0x7f7f6c100c20 2024-03-12T05:54:37.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.536+0000 7f7f637fe700 1 --2- 172.21.15.76:0/2643645109 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f7f580a1b50 0x7f7f580a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:37.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.536+0000 7f7f6a59c700 1 --2- 172.21.15.76:0/2643645109 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f7f580a1b50 0x7f7f580a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:37.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.536+0000 7f7f637fe700 1 -- 172.21.15.76:0/2643645109 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7f540cea40 con 0x7f7f6c100c20 2024-03-12T05:54:37.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.536+0000 7f7f6a59c700 1 --2- 172.21.15.76:0/2643645109 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f7f580a1b50 0x7f7f580a4010 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400e270 tx=0x7f7f64009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:37.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.539+0000 7f7f637fe700 1 -- 172.21.15.76:0/2643645109 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7f54097c30 con 0x7f7f6c100c20 2024-03-12T05:54:37.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.751+0000 7f7f70af0700 1 -- 172.21.15.76:0/2643645109 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7f6c19d8b0 con 0x7f7f6c100c20 2024-03-12T05:54:37.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.751+0000 7f7f637fe700 1 -- 172.21.15.76:0/2643645109 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7f5409b2f0 con 0x7f7f6c100c20 2024-03-12T05:54:37.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.753+0000 7f7f70af0700 1 -- 172.21.15.76:0/2643645109 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f7f580a1b50 msgr2=0x7f7f580a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:37.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.753+0000 7f7f70af0700 1 --2- 172.21.15.76:0/2643645109 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f7f580a1b50 0x7f7f580a4010 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400e270 tx=0x7f7f64009040 comp rx=0 tx=0).stop 2024-03-12T05:54:37.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.753+0000 7f7f70af0700 1 -- 172.21.15.76:0/2643645109 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f6c100c20 msgr2=0x7f7f6c198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:37.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.753+0000 7f7f70af0700 1 --2- 172.21.15.76:0/2643645109 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f6c100c20 0x7f7f6c198720 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7f540098e0 tx=0x7f7f5400f740 comp rx=0 tx=0).stop 2024-03-12T05:54:37.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.753+0000 7f7f70af0700 1 -- 172.21.15.76:0/2643645109 shutdown_connections 2024-03-12T05:54:37.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.753+0000 7f7f70af0700 1 --2- 172.21.15.76:0/2643645109 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f7f580a1b50 0x7f7f580a4010 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:37.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.753+0000 7f7f70af0700 1 --2- 172.21.15.76:0/2643645109 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7f6c106ce0 0x7f7f6c19cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:37.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.753+0000 7f7f70af0700 1 --2- 172.21.15.76:0/2643645109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f6c100260 0x7f7f6c1981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:37.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.753+0000 7f7f70af0700 1 --2- 172.21.15.76:0/2643645109 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7f6c100c20 0x7f7f6c198720 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:37.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.753+0000 7f7f70af0700 1 -- 172.21.15.76:0/2643645109 >> 172.21.15.76:0/2643645109 conn(0x7f7f6c077170 msgr2=0x7f7f6c104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:37.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.753+0000 7f7f70af0700 1 -- 172.21.15.76:0/2643645109 shutdown_connections 2024-03-12T05:54:37.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:33.753+0000 7f7f70af0700 1 -- 172.21.15.76:0/2643645109 wait complete. 2024-03-12T05:54:38.103 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:38.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:38 smithi082 ceph-mon[36382]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:54:38.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:38 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:38.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:38 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:38.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:38 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:38.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:38 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:38.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:37 smithi076 ceph-mon[32095]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:54:38.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:37 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:38.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:37 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:38.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:38 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:38.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:38 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:38 smithi088 ceph-mon[36585]: pgmap v314: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:54:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:38 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:38 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:38 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:38 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:38.507 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738442 got 34359738446 for osd.0 2024-03-12T05:54:38.507 DEBUG:teuthology.parallel:result is None 2024-03-12T05:54:39.363 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:39.803 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350995 got 347892351001 for osd.11 2024-03-12T05:54:39.804 DEBUG:teuthology.parallel:result is None 2024-03-12T05:54:40.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:40 smithi082 ceph-mon[36382]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:54:40.295 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:40 smithi076 ceph-mon[32095]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:54:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:40 smithi088 ceph-mon[36585]: pgmap v315: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:54:41.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-12T05:54:41.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-12T05:54:41.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-03-12T05:54:41.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-03-12T05:54:41.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-03-12T05:54:41.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-12T05:54:41.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-03-12T05:54:41.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:41 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-03-12T05:54:41.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-12T05:54:41.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-12T05:54:41.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-03-12T05:54:41.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-03-12T05:54:41.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-03-12T05:54:41.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-12T05:54:41.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-03-12T05:54:41.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:41 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-03-12T05:54:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-12T05:54:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-12T05:54:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-03-12T05:54:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-03-12T05:54:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-03-12T05:54:41.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-12T05:54:41.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-03-12T05:54:41.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:41 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-03-12T05:54:42.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:42 smithi076 ceph-mon[32095]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:54:42.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:42 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-12T05:54:42.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:42 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-12T05:54:42.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:42 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-03-12T05:54:42.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:42 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-03-12T05:54:42.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:42 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-03-12T05:54:42.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:42 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-12T05:54:42.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:42 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-03-12T05:54:42.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:42 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-03-12T05:54:42.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:42 smithi076 ceph-mon[32095]: osdmap e87: 12 total, 12 up, 12 in 2024-03-12T05:54:42.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:42 smithi082 ceph-mon[36382]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:54:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:42 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-12T05:54:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:42 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-12T05:54:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:42 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-03-12T05:54:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:42 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-03-12T05:54:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:42 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-03-12T05:54:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:42 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-12T05:54:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:42 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-03-12T05:54:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:42 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-03-12T05:54:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:42 smithi082 ceph-mon[36382]: osdmap e87: 12 total, 12 up, 12 in 2024-03-12T05:54:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:42 smithi088 ceph-mon[36585]: pgmap v316: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:54:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:42 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-12T05:54:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:42 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-12T05:54:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:42 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-03-12T05:54:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:42 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-03-12T05:54:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:42 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-03-12T05:54:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:42 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-12T05:54:42.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:42 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-03-12T05:54:42.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:42 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-03-12T05:54:42.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:42 smithi088 ceph-mon[36585]: osdmap e87: 12 total, 12 up, 12 in 2024-03-12T05:54:43.116 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:43 smithi076 ceph-mon[32095]: osdmap e88: 12 total, 12 up, 12 in 2024-03-12T05:54:43.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:43 smithi082 ceph-mon[36382]: osdmap e88: 12 total, 12 up, 12 in 2024-03-12T05:54:43.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:43 smithi088 ceph-mon[36585]: osdmap e88: 12 total, 12 up, 12 in 2024-03-12T05:54:44.098 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:44 smithi076 ceph-mon[32095]: pgmap v319: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-03-12T05:54:44.098 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:44 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4091578019' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-12T05:54:44.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.575+0000 7f941d682700 1 -- 172.21.15.76:0/2557368887 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f941806adc0 msgr2=0x7f9418106870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:44.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.575+0000 7f941d682700 1 --2- 172.21.15.76:0/2557368887 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f941806adc0 0x7f9418106870 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9408009990 tx=0x7f9408009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:44.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.575+0000 7f941d682700 1 -- 172.21.15.76:0/2557368887 shutdown_connections 2024-03-12T05:54:44.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.575+0000 7f941d682700 1 --2- 172.21.15.76:0/2557368887 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9418106f50 0x7f941810eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:44.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.575+0000 7f941d682700 1 --2- 172.21.15.76:0/2557368887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f941806a410 0x7f941806a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:44.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.575+0000 7f941d682700 1 --2- 172.21.15.76:0/2557368887 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f941806adc0 0x7f9418106870 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:44.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.575+0000 7f941d682700 1 -- 172.21.15.76:0/2557368887 >> 172.21.15.76:0/2557368887 conn(0x7f9418076bc0 msgr2=0x7f9418076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:44.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.575+0000 7f941d682700 1 -- 172.21.15.76:0/2557368887 shutdown_connections 2024-03-12T05:54:44.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.575+0000 7f941d682700 1 -- 172.21.15.76:0/2557368887 wait complete. 2024-03-12T05:54:44.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.575+0000 7f941d682700 1 Processor -- start 2024-03-12T05:54:44.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f941d682700 1 -- start start 2024-03-12T05:54:44.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f941d682700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f941806a410 0x7f9418100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:44.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f941d682700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f941806adc0 0x7f94181011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:44.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f941d682700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9418106f50 0x7f94181016f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:44.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f941d682700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94181a65e0 con 0x7f941806a410 2024-03-12T05:54:44.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f941d682700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94181a6750 con 0x7f941806adc0 2024-03-12T05:54:44.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f941d682700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94181a6910 con 0x7f9418106f50 2024-03-12T05:54:44.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f9416ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f941806a410 0x7f9418100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:44.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f9416ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f941806a410 0x7f9418100c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47456/0 (socket says 172.21.15.76:47456) 2024-03-12T05:54:44.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f9416ffd700 1 -- 172.21.15.76:0/4091578019 learned_addr learned my addr 172.21.15.76:0/4091578019 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:44.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f94167fc700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f941806adc0 0x7f94181011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:44.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f94177fe700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9418106f50 0x7f94181016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:44.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f9416ffd700 1 -- 172.21.15.76:0/4091578019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9418106f50 msgr2=0x7f94181016f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:44.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f9416ffd700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9418106f50 0x7f94181016f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:44.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f9416ffd700 1 -- 172.21.15.76:0/4091578019 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f941806adc0 msgr2=0x7f94181011b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f9416ffd700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f941806adc0 0x7f94181011b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f9416ffd700 1 -- 172.21.15.76:0/4091578019 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94080095f0 con 0x7f941806a410 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f94167fc700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f941806adc0 0x7f94181011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.576+0000 7f9416ffd700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f941806a410 0x7f9418100c70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f940000d950 tx=0x7f940000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.577+0000 7f940ffff700 1 -- 172.21.15.76:0/4091578019 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9400009b40 con 0x7f941806a410 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.577+0000 7f940ffff700 1 -- 172.21.15.76:0/4091578019 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9400009ca0 con 0x7f941806a410 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.577+0000 7f941d682700 1 -- 172.21.15.76:0/4091578019 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94181a6bf0 con 0x7f941806a410 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.577+0000 7f940ffff700 1 -- 172.21.15.76:0/4091578019 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940000f6b0 con 0x7f941806a410 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.577+0000 7f941d682700 1 -- 172.21.15.76:0/4091578019 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94181a70f0 con 0x7f941806a410 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.578+0000 7f941d682700 1 -- 172.21.15.76:0/4091578019 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f941810a270 con 0x7f941806a410 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.579+0000 7f940ffff700 1 -- 172.21.15.76:0/4091578019 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f9400010460 con 0x7f941806a410 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.579+0000 7f940ffff700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f94040a1b50 0x7f94040a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.579+0000 7f94167fc700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f94040a1b50 0x7f94040a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.579+0000 7f940ffff700 1 -- 172.21.15.76:0/4091578019 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f94000cd950 con 0x7f941806a410 2024-03-12T05:54:44.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.579+0000 7f94167fc700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f94040a1b50 0x7f94040a4010 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9418108560 tx=0x7f940800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:44.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.584+0000 7f940ffff700 1 -- 172.21.15.76:0/4091578019 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f94000969b0 con 0x7f941806a410 2024-03-12T05:54:44.295 INFO:teuthology.orchestra.run.smithi076.stdout:257698037802 2024-03-12T05:54:44.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.788+0000 7f941d682700 1 -- 172.21.15.76:0/4091578019 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f9418111d00 con 0x7f941806a410 2024-03-12T05:54:44.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.788+0000 7f940ffff700 1 -- 172.21.15.76:0/4091578019 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f940009a070 con 0x7f941806a410 2024-03-12T05:54:44.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.791+0000 7f941d682700 1 -- 172.21.15.76:0/4091578019 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f94040a1b50 msgr2=0x7f94040a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:44.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.791+0000 7f941d682700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f94040a1b50 0x7f94040a4010 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9418108560 tx=0x7f940800b540 comp rx=0 tx=0).stop 2024-03-12T05:54:44.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.791+0000 7f941d682700 1 -- 172.21.15.76:0/4091578019 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f941806a410 msgr2=0x7f9418100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:44.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.791+0000 7f941d682700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f941806a410 0x7f9418100c70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f940000d950 tx=0x7f940000dd10 comp rx=0 tx=0).stop 2024-03-12T05:54:44.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.791+0000 7f941d682700 1 -- 172.21.15.76:0/4091578019 shutdown_connections 2024-03-12T05:54:44.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.791+0000 7f941d682700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f94040a1b50 0x7f94040a4010 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:44.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.791+0000 7f941d682700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9418106f50 0x7f94181016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:44.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.791+0000 7f941d682700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f941806a410 0x7f9418100c70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:44.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.791+0000 7f941d682700 1 --2- 172.21.15.76:0/4091578019 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f941806adc0 0x7f94181011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:44.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.791+0000 7f941d682700 1 -- 172.21.15.76:0/4091578019 >> 172.21.15.76:0/4091578019 conn(0x7f9418076bc0 msgr2=0x7f94180fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:44.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.791+0000 7f941d682700 1 -- 172.21.15.76:0/4091578019 shutdown_connections 2024-03-12T05:54:44.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:43.791+0000 7f941d682700 1 -- 172.21.15.76:0/4091578019 wait complete. 2024-03-12T05:54:44.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:44 smithi082 ceph-mon[36382]: pgmap v319: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-03-12T05:54:44.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:44 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4091578019' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-12T05:54:44.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:44 smithi088 ceph-mon[36585]: pgmap v319: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-03-12T05:54:44.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:44 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4091578019' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-12T05:54:44.888 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037796 got 257698037802 for osd.8 2024-03-12T05:54:44.888 DEBUG:teuthology.parallel:result is None 2024-03-12T05:54:46.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.078+0000 7f6ba1117700 1 -- 172.21.15.76:0/4257191082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9c102860 msgr2=0x7f6b9c1068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.078+0000 7f6ba1117700 1 --2- 172.21.15.76:0/4257191082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9c102860 0x7f6b9c1068b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009990 tx=0x7f6b90009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.078+0000 7f6ba1117700 1 -- 172.21.15.76:0/4257191082 shutdown_connections 2024-03-12T05:54:46.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.078+0000 7f6ba1117700 1 --2- 172.21.15.76:0/4257191082 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b9c106f90 0x7f6b9c10eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.078+0000 7f6ba1117700 1 --2- 172.21.15.76:0/4257191082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b9c101eb0 0x7f6b9c102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.078+0000 7f6ba1117700 1 --2- 172.21.15.76:0/4257191082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9c102860 0x7f6b9c1068b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.078+0000 7f6ba1117700 1 -- 172.21.15.76:0/4257191082 >> 172.21.15.76:0/4257191082 conn(0x7f6b9c0fd720 msgr2=0x7f6b9c0ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:46.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.079+0000 7f6ba1117700 1 -- 172.21.15.76:0/4257191082 shutdown_connections 2024-03-12T05:54:46.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.079+0000 7f6ba1117700 1 -- 172.21.15.76:0/4257191082 wait complete. 2024-03-12T05:54:46.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.079+0000 7f6ba1117700 1 Processor -- start 2024-03-12T05:54:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.079+0000 7f6ba1117700 1 -- start start 2024-03-12T05:54:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6ba1117700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b9c101eb0 0x7f6b9c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6ba1117700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b9c102860 0x7f6b9c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6ba1117700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9c106f90 0x7f6b9c19c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6ba1117700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c19cfd0 con 0x7f6b9c106f90 2024-03-12T05:54:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6ba1117700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c19d140 con 0x7f6b9c101eb0 2024-03-12T05:54:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6ba1117700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9c19d300 con 0x7f6b9c102860 2024-03-12T05:54:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6b9b59e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9c106f90 0x7f6b9c19c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6b9a59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b9c102860 0x7f6b9c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6b9b59e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9c106f90 0x7f6b9c19c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47484/0 (socket says 172.21.15.76:47484) 2024-03-12T05:54:46.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6b9b59e700 1 -- 172.21.15.76:0/248226661 learned_addr learned my addr 172.21.15.76:0/248226661 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6b9ad9d700 1 --2- 172.21.15.76:0/248226661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b9c101eb0 0x7f6b9c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6b9b59e700 1 -- 172.21.15.76:0/248226661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b9c102860 msgr2=0x7f6b9c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6b9b59e700 1 --2- 172.21.15.76:0/248226661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b9c102860 0x7f6b9c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6b9b59e700 1 -- 172.21.15.76:0/248226661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b9c101eb0 msgr2=0x7f6b9c1980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6b9b59e700 1 --2- 172.21.15.76:0/248226661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b9c101eb0 0x7f6b9c1980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6b9b59e700 1 -- 172.21.15.76:0/248226661 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b900095f0 con 0x7f6b9c106f90 2024-03-12T05:54:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6b9b59e700 1 --2- 172.21.15.76:0/248226661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9c106f90 0x7f6b9c19c9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6b94007d70 tx=0x7f6b9400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.080+0000 7f6b83fff700 1 -- 172.21.15.76:0/248226661 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400ad40 con 0x7f6b9c106f90 2024-03-12T05:54:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.081+0000 7f6ba1117700 1 -- 172.21.15.76:0/248226661 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9c19d5e0 con 0x7f6b9c106f90 2024-03-12T05:54:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.081+0000 7f6b83fff700 1 -- 172.21.15.76:0/248226661 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b94012900 con 0x7f6b9c106f90 2024-03-12T05:54:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.081+0000 7f6b83fff700 1 -- 172.21.15.76:0/248226661 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94011780 con 0x7f6b9c106f90 2024-03-12T05:54:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.081+0000 7f6ba1117700 1 -- 172.21.15.76:0/248226661 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9c1a4e40 con 0x7f6b9c106f90 2024-03-12T05:54:46.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.082+0000 7f6ba1117700 1 -- 172.21.15.76:0/248226661 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b9c10a2b0 con 0x7f6b9c106f90 2024-03-12T05:54:46.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.083+0000 7f6b83fff700 1 -- 172.21.15.76:0/248226661 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f6b94012a70 con 0x7f6b9c106f90 2024-03-12T05:54:46.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.083+0000 7f6b83fff700 1 --2- 172.21.15.76:0/248226661 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b840a1bd0 0x7f6b840a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:46.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.083+0000 7f6b9ad9d700 1 --2- 172.21.15.76:0/248226661 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b840a1bd0 0x7f6b840a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:46.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.083+0000 7f6b83fff700 1 -- 172.21.15.76:0/248226661 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6b940cd6f0 con 0x7f6b9c106f90 2024-03-12T05:54:46.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.084+0000 7f6b9ad9d700 1 --2- 172.21.15.76:0/248226661 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b840a1bd0 0x7f6b840a4090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c005950 tx=0x7f6b8c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:46.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.088+0000 7f6b83fff700 1 -- 172.21.15.76:0/248226661 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6b94096860 con 0x7f6b9c106f90 2024-03-12T05:54:46.188 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:46 smithi076 ceph-mon[32095]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 17 B/s, 0 objects/s recovering 2024-03-12T05:54:46.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.286+0000 7f5335209700 1 -- 172.21.15.76:0/336308848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5330069920 msgr2=0x7f5330069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.286+0000 7f5335209700 1 --2- 172.21.15.76:0/336308848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5330069920 0x7f5330069d80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f53240099c0 tx=0x7f5324009cd0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.286+0000 7f5335209700 1 -- 172.21.15.76:0/336308848 shutdown_connections 2024-03-12T05:54:46.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.286+0000 7f5335209700 1 --2- 172.21.15.76:0/336308848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f533006a2c0 0x7f533010eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.286+0000 7f5335209700 1 --2- 172.21.15.76:0/336308848 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5330107cc0 0x7f53301080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.286+0000 7f5335209700 1 --2- 172.21.15.76:0/336308848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5330069920 0x7f5330069d80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.286+0000 7f5335209700 1 -- 172.21.15.76:0/336308848 >> 172.21.15.76:0/336308848 conn(0x7f5330077aa0 msgr2=0x7f5330077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:46.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.286+0000 7f5335209700 1 -- 172.21.15.76:0/336308848 shutdown_connections 2024-03-12T05:54:46.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.286+0000 7f5335209700 1 -- 172.21.15.76:0/336308848 wait complete. 2024-03-12T05:54:46.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f5335209700 1 Processor -- start 2024-03-12T05:54:46.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f5335209700 1 -- start start 2024-03-12T05:54:46.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f5335209700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5330069920 0x7f5330100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:46.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f5335209700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f533006a2c0 0x7f5330101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:46.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f5335209700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5330107cc0 0x7f53301016c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:46.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f5335209700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53301a65a0 con 0x7f533006a2c0 2024-03-12T05:54:46.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f5335209700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53301a6710 con 0x7f5330107cc0 2024-03-12T05:54:46.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f5335209700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53301a68d0 con 0x7f5330069920 2024-03-12T05:54:46.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f532e59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f533006a2c0 0x7f5330101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:46.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f532e59c700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f533006a2c0 0x7f5330101180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47504/0 (socket says 172.21.15.76:47504) 2024-03-12T05:54:46.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f532e59c700 1 -- 172.21.15.76:0/1443224644 learned_addr learned my addr 172.21.15.76:0/1443224644 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:46.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f532f59e700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5330107cc0 0x7f53301016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:46.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f532ed9d700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5330069920 0x7f5330100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:46.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f532e59c700 1 -- 172.21.15.76:0/1443224644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5330069920 msgr2=0x7f5330100c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f532e59c700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5330069920 0x7f5330100c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f532e59c700 1 -- 172.21.15.76:0/1443224644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5330107cc0 msgr2=0x7f53301016c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.287+0000 7f532e59c700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5330107cc0 0x7f53301016c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.288+0000 7f532e59c700 1 -- 172.21.15.76:0/1443224644 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53240095f0 con 0x7f533006a2c0 2024-03-12T05:54:46.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.288+0000 7f532f59e700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5330107cc0 0x7f53301016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:54:46.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.288+0000 7f532ed9d700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5330069920 0x7f5330100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:54:46.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.288+0000 7f532e59c700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f533006a2c0 0x7f5330101180 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5324009990 tx=0x7f5324005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:46.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.288+0000 7f5317fff700 1 -- 172.21.15.76:0/1443224644 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532401d070 con 0x7f533006a2c0 2024-03-12T05:54:46.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.288+0000 7f5335209700 1 -- 172.21.15.76:0/1443224644 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53301a6b50 con 0x7f533006a2c0 2024-03-12T05:54:46.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.288+0000 7f5317fff700 1 -- 172.21.15.76:0/1443224644 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5324004cc0 con 0x7f533006a2c0 2024-03-12T05:54:46.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.288+0000 7f5317fff700 1 -- 172.21.15.76:0/1443224644 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324003d50 con 0x7f533006a2c0 2024-03-12T05:54:46.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.288+0000 7f5335209700 1 -- 172.21.15.76:0/1443224644 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53301a6f60 con 0x7f533006a2c0 2024-03-12T05:54:46.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.289+0000 7f6ba1117700 1 -- 172.21.15.76:0/248226661 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6b9c199480 con 0x7f6b9c106f90 2024-03-12T05:54:46.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.289+0000 7f6b83fff700 1 -- 172.21.15.76:0/248226661 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6b94099f20 con 0x7f6b9c106f90 2024-03-12T05:54:46.295 INFO:teuthology.orchestra.run.smithi076.stdout:55834574924 2024-03-12T05:54:46.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.289+0000 7f5335209700 1 -- 172.21.15.76:0/1443224644 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533010a1e0 con 0x7f533006a2c0 2024-03-12T05:54:46.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.290+0000 7f5317fff700 1 -- 172.21.15.76:0/1443224644 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f53240050c0 con 0x7f533006a2c0 2024-03-12T05:54:46.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.291+0000 7f5317fff700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f53180a1b50 0x7f53180a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:46.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.291+0000 7f532ed9d700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f53180a1b50 0x7f53180a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:46.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.291+0000 7f5317fff700 1 -- 172.21.15.76:0/1443224644 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f53240ce440 con 0x7f533006a2c0 2024-03-12T05:54:46.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.291+0000 7f6ba1117700 1 -- 172.21.15.76:0/248226661 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b840a1bd0 msgr2=0x7f6b840a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.291+0000 7f6ba1117700 1 --2- 172.21.15.76:0/248226661 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b840a1bd0 0x7f6b840a4090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c005950 tx=0x7f6b8c0058e0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.291+0000 7f6ba1117700 1 -- 172.21.15.76:0/248226661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9c106f90 msgr2=0x7f6b9c19c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.291+0000 7f6ba1117700 1 --2- 172.21.15.76:0/248226661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9c106f90 0x7f6b9c19c9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6b94007d70 tx=0x7f6b9400a340 comp rx=0 tx=0).stop 2024-03-12T05:54:46.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.291+0000 7f6ba1117700 1 -- 172.21.15.76:0/248226661 shutdown_connections 2024-03-12T05:54:46.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.291+0000 7f6ba1117700 1 --2- 172.21.15.76:0/248226661 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b840a1bd0 0x7f6b840a4090 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.291+0000 7f6ba1117700 1 --2- 172.21.15.76:0/248226661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9c106f90 0x7f6b9c19c9c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.291+0000 7f6ba1117700 1 --2- 172.21.15.76:0/248226661 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b9c101eb0 0x7f6b9c1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.291+0000 7f6ba1117700 1 --2- 172.21.15.76:0/248226661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b9c102860 0x7f6b9c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.291+0000 7f6ba1117700 1 -- 172.21.15.76:0/248226661 >> 172.21.15.76:0/248226661 conn(0x7f6b9c0fd720 msgr2=0x7f6b9c0fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:46.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.292+0000 7f532ed9d700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f53180a1b50 0x7f53180a4010 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5320005e10 tx=0x7f5320005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:46.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.292+0000 7f6ba1117700 1 -- 172.21.15.76:0/248226661 shutdown_connections 2024-03-12T05:54:46.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.292+0000 7f6ba1117700 1 -- 172.21.15.76:0/248226661 wait complete. 2024-03-12T05:54:46.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.296+0000 7f5317fff700 1 -- 172.21.15.76:0/1443224644 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5324097590 con 0x7f533006a2c0 2024-03-12T05:54:46.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:46 smithi082 ceph-mon[36382]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 17 B/s, 0 objects/s recovering 2024-03-12T05:54:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:46 smithi088 ceph-mon[36585]: pgmap v320: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 17 B/s, 0 objects/s recovering 2024-03-12T05:54:46.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.494+0000 7f5335209700 1 -- 172.21.15.76:0/1443224644 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f5330111cf0 con 0x7f533006a2c0 2024-03-12T05:54:46.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.495+0000 7f5317fff700 1 -- 172.21.15.76:0/1443224644 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5324095e60 con 0x7f533006a2c0 2024-03-12T05:54:46.498 INFO:teuthology.orchestra.run.smithi076.stdout:197568495668 2024-03-12T05:54:46.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.497+0000 7f5335209700 1 -- 172.21.15.76:0/1443224644 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f53180a1b50 msgr2=0x7f53180a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.497+0000 7f5335209700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f53180a1b50 0x7f53180a4010 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5320005e10 tx=0x7f5320005d60 comp rx=0 tx=0).stop 2024-03-12T05:54:46.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.497+0000 7f5335209700 1 -- 172.21.15.76:0/1443224644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f533006a2c0 msgr2=0x7f5330101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.497+0000 7f5335209700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f533006a2c0 0x7f5330101180 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5324009990 tx=0x7f5324005e80 comp rx=0 tx=0).stop 2024-03-12T05:54:46.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.497+0000 7f5335209700 1 -- 172.21.15.76:0/1443224644 shutdown_connections 2024-03-12T05:54:46.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.497+0000 7f5335209700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f53180a1b50 0x7f53180a4010 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.497+0000 7f5335209700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5330107cc0 0x7f53301016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.497+0000 7f5335209700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5330069920 0x7f5330100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.497+0000 7f5335209700 1 --2- 172.21.15.76:0/1443224644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f533006a2c0 0x7f5330101180 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.497+0000 7f5335209700 1 -- 172.21.15.76:0/1443224644 >> 172.21.15.76:0/1443224644 conn(0x7f5330077aa0 msgr2=0x7f53300ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:46.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.497+0000 7f5335209700 1 -- 172.21.15.76:0/1443224644 shutdown_connections 2024-03-12T05:54:46.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.497+0000 7f5335209700 1 -- 172.21.15.76:0/1443224644 wait complete. 2024-03-12T05:54:46.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.745+0000 7f30aaeb8700 1 -- 172.21.15.76:0/144933807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30a4101cc0 msgr2=0x7f30a4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.745+0000 7f30aaeb8700 1 --2- 172.21.15.76:0/144933807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30a4101cc0 0x7f30a4102140 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f308c0099c0 tx=0x7f308c009cd0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.745+0000 7f30aaeb8700 1 -- 172.21.15.76:0/144933807 shutdown_connections 2024-03-12T05:54:46.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.745+0000 7f30aaeb8700 1 --2- 172.21.15.76:0/144933807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f30a4102680 0x7f30a410eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.745+0000 7f30aaeb8700 1 --2- 172.21.15.76:0/144933807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30a4107ca0 0x7f30a4108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.745+0000 7f30aaeb8700 1 --2- 172.21.15.76:0/144933807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30a4101cc0 0x7f30a4102140 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.745+0000 7f30aaeb8700 1 -- 172.21.15.76:0/144933807 >> 172.21.15.76:0/144933807 conn(0x7f30a4078da0 msgr2=0x7f30a40791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:46.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.745+0000 7f30aaeb8700 1 -- 172.21.15.76:0/144933807 shutdown_connections 2024-03-12T05:54:46.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.745+0000 7f30aaeb8700 1 -- 172.21.15.76:0/144933807 wait complete. 2024-03-12T05:54:46.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.746+0000 7f30aaeb8700 1 Processor -- start 2024-03-12T05:54:46.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.746+0000 7f30aaeb8700 1 -- start start 2024-03-12T05:54:46.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.746+0000 7f30aaeb8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f30a4101cc0 0x7f30a4198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:46.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.746+0000 7f30aaeb8700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30a4102680 0x7f30a4198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:46.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.746+0000 7f30aaeb8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30a4107ca0 0x7f30a419c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:46.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.746+0000 7f30aaeb8700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a419d000 con 0x7f30a4102680 2024-03-12T05:54:46.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.746+0000 7f30aaeb8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a419d170 con 0x7f30a4101cc0 2024-03-12T05:54:46.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.746+0000 7f30aaeb8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a419d330 con 0x7f30a4107ca0 2024-03-12T05:54:46.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.746+0000 7f30a3fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30a4102680 0x7f30a4198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:46.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.746+0000 7f30a3fff700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30a4102680 0x7f30a4198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47514/0 (socket says 172.21.15.76:47514) 2024-03-12T05:54:46.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.746+0000 7f30a3fff700 1 -- 172.21.15.76:0/3071441236 learned_addr learned my addr 172.21.15.76:0/3071441236 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:46.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.747+0000 7f30a8c54700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f30a4101cc0 0x7f30a4198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:46.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.747+0000 7f30a9455700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30a4107ca0 0x7f30a419c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:46.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.747+0000 7f30a3fff700 1 -- 172.21.15.76:0/3071441236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30a4107ca0 msgr2=0x7f30a419c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.747+0000 7f30a3fff700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30a4107ca0 0x7f30a419c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.747+0000 7f30a3fff700 1 -- 172.21.15.76:0/3071441236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f30a4101cc0 msgr2=0x7f30a4198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.747+0000 7f30a3fff700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f30a4101cc0 0x7f30a4198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.747+0000 7f30a3fff700 1 -- 172.21.15.76:0/3071441236 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c0095f0 con 0x7f30a4102680 2024-03-12T05:54:46.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.747+0000 7f30a8c54700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f30a4101cc0 0x7f30a4198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:54:46.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.747+0000 7f30a3fff700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30a4102680 0x7f30a4198660 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f308c00b580 tx=0x7f308c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:46.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.747+0000 7f30a1ffb700 1 -- 172.21.15.76:0/3071441236 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c01d070 con 0x7f30a4102680 2024-03-12T05:54:46.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.747+0000 7f30a1ffb700 1 -- 172.21.15.76:0/3071441236 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f308c003fd0 con 0x7f30a4102680 2024-03-12T05:54:46.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.747+0000 7f30a1ffb700 1 -- 172.21.15.76:0/3071441236 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c022e50 con 0x7f30a4102680 2024-03-12T05:54:46.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.747+0000 7f30aaeb8700 1 -- 172.21.15.76:0/3071441236 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a419d5b0 con 0x7f30a4102680 2024-03-12T05:54:46.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.747+0000 7f30aaeb8700 1 -- 172.21.15.76:0/3071441236 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a41a4e70 con 0x7f30a4102680 2024-03-12T05:54:46.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.748+0000 7f30aaeb8700 1 -- 172.21.15.76:0/3071441236 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a410a180 con 0x7f30a4102680 2024-03-12T05:54:46.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.749+0000 7f30a1ffb700 1 -- 172.21.15.76:0/3071441236 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f308c022590 con 0x7f30a4102680 2024-03-12T05:54:46.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.750+0000 7f30a1ffb700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f30900a1bd0 0x7f30900a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:46.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.750+0000 7f30a8c54700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f30900a1bd0 0x7f30900a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:46.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.750+0000 7f30a1ffb700 1 -- 172.21.15.76:0/3071441236 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f308c0ce7f0 con 0x7f30a4102680 2024-03-12T05:54:46.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.750+0000 7f30a8c54700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f30900a1bd0 0x7f30900a4090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f309c009dd0 tx=0x7f309c009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:46.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.753+0000 7f30a1ffb700 1 -- 172.21.15.76:0/3071441236 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f308c097940 con 0x7f30a4102680 2024-03-12T05:54:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.958+0000 7f30aaeb8700 1 -- 172.21.15.76:0/3071441236 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f30a4199480 con 0x7f30a4102680 2024-03-12T05:54:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.958+0000 7f30a1ffb700 1 -- 172.21.15.76:0/3071441236 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f308c09b000 con 0x7f30a4102680 2024-03-12T05:54:46.962 INFO:teuthology.orchestra.run.smithi076.stdout:292057776165 2024-03-12T05:54:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.960+0000 7f30aaeb8700 1 -- 172.21.15.76:0/3071441236 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f30900a1bd0 msgr2=0x7f30900a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.960+0000 7f30aaeb8700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f30900a1bd0 0x7f30900a4090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f309c009dd0 tx=0x7f309c009040 comp rx=0 tx=0).stop 2024-03-12T05:54:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.960+0000 7f30aaeb8700 1 -- 172.21.15.76:0/3071441236 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30a4102680 msgr2=0x7f30a4198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.960+0000 7f30aaeb8700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30a4102680 0x7f30a4198660 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f308c00b580 tx=0x7f308c005e80 comp rx=0 tx=0).stop 2024-03-12T05:54:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.960+0000 7f30aaeb8700 1 -- 172.21.15.76:0/3071441236 shutdown_connections 2024-03-12T05:54:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.960+0000 7f30aaeb8700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f30900a1bd0 0x7f30900a4090 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.960+0000 7f30aaeb8700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30a4107ca0 0x7f30a419c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.960+0000 7f30aaeb8700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f30a4101cc0 0x7f30a4198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.960+0000 7f30aaeb8700 1 --2- 172.21.15.76:0/3071441236 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f30a4102680 0x7f30a4198660 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.960+0000 7f30aaeb8700 1 -- 172.21.15.76:0/3071441236 >> 172.21.15.76:0/3071441236 conn(0x7f30a4078da0 msgr2=0x7f30a4100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.960+0000 7f30aaeb8700 1 -- 172.21.15.76:0/3071441236 shutdown_connections 2024-03-12T05:54:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:46.960+0000 7f30aaeb8700 1 -- 172.21.15.76:0/3071441236 wait complete. 2024-03-12T05:54:47.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:47 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/248226661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T05:54:47.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:47 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1443224644' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T05:54:47.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:47 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3071441236' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-12T05:54:47.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:47 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/248226661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T05:54:47.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:47 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1443224644' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T05:54:47.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:47 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3071441236' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-12T05:54:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:47 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/248226661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T05:54:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:47 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1443224644' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T05:54:47.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:47 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3071441236' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-12T05:54:47.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.135+0000 7feacb62a700 1 -- 172.21.15.76:0/3956836824 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feac4106b80 msgr2=0x7feac4106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:47.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.135+0000 7feacb62a700 1 --2- 172.21.15.76:0/3956836824 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feac4106b80 0x7feac4106f60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feac0009990 tx=0x7feac0009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:47.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.135+0000 7feacb62a700 1 -- 172.21.15.76:0/3956836824 shutdown_connections 2024-03-12T05:54:47.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.135+0000 7feacb62a700 1 --2- 172.21.15.76:0/3956836824 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac4101530 0x7feac410d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.135+0000 7feacb62a700 1 --2- 172.21.15.76:0/3956836824 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feac4106b80 0x7feac4106f60 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.135+0000 7feacb62a700 1 --2- 172.21.15.76:0/3956836824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feac4100b70 0x7feac4100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.135+0000 7feacb62a700 1 -- 172.21.15.76:0/3956836824 >> 172.21.15.76:0/3956836824 conn(0x7feac4077aa0 msgr2=0x7feac4077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.135+0000 7feacb62a700 1 -- 172.21.15.76:0/3956836824 shutdown_connections 2024-03-12T05:54:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.135+0000 7feacb62a700 1 -- 172.21.15.76:0/3956836824 wait complete. 2024-03-12T05:54:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feacb62a700 1 Processor -- start 2024-03-12T05:54:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feacb62a700 1 -- start start 2024-03-12T05:54:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feacb62a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feac4100b70 0x7feac4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feacb62a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac4101530 0x7feac4198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feacb62a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feac4106b80 0x7feac419c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feacb62a700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac419ced0 con 0x7feac4106b80 2024-03-12T05:54:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feacb62a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac419d040 con 0x7feac4101530 2024-03-12T05:54:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feacb62a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac419d200 con 0x7feac4100b70 2024-03-12T05:54:47.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feac9bc7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feac4106b80 0x7feac419c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feac9bc7700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feac4106b80 0x7feac419c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47532/0 (socket says 172.21.15.76:47532) 2024-03-12T05:54:47.489 INFO:teuthology.orchestra.run.smithi076.stdout:141733920830 2024-03-12T05:54:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feac9bc7700 1 -- 172.21.15.76:0/487155390 learned_addr learned my addr 172.21.15.76:0/487155390 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feac93c6700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feac4100b70 0x7feac4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:47.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feac8bc5700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac4101530 0x7feac4198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feac9bc7700 1 -- 172.21.15.76:0/487155390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feac4100b70 msgr2=0x7feac4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feac9bc7700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feac4100b70 0x7feac4197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feac9bc7700 1 -- 172.21.15.76:0/487155390 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac4101530 msgr2=0x7feac4198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feac9bc7700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac4101530 0x7feac4198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feac9bc7700 1 -- 172.21.15.76:0/487155390 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac00095f0 con 0x7feac4106b80 2024-03-12T05:54:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.136+0000 7feac93c6700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feac4100b70 0x7feac4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:54:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.137+0000 7feac9bc7700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feac4106b80 0x7feac419c8c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7feabc00ada0 tx=0x7feabc00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.137+0000 7feaba7fc700 1 -- 172.21.15.76:0/487155390 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc00f070 con 0x7feac4106b80 2024-03-12T05:54:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.137+0000 7feacb62a700 1 -- 172.21.15.76:0/487155390 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac419d4e0 con 0x7feac4106b80 2024-03-12T05:54:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.137+0000 7feaba7fc700 1 -- 172.21.15.76:0/487155390 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feabc007b30 con 0x7feac4106b80 2024-03-12T05:54:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.137+0000 7feaba7fc700 1 -- 172.21.15.76:0/487155390 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc0053c0 con 0x7feac4106b80 2024-03-12T05:54:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.137+0000 7feacb62a700 1 -- 172.21.15.76:0/487155390 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac41a4d40 con 0x7feac4106b80 2024-03-12T05:54:47.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.138+0000 7feacb62a700 1 -- 172.21.15.76:0/487155390 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac4068930 con 0x7feac4106b80 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.139+0000 7feaba7fc700 1 -- 172.21.15.76:0/487155390 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7feabc00c980 con 0x7feac4106b80 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.139+0000 7feaba7fc700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7feab00a1b50 0x7feab00a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.140+0000 7feac93c6700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7feab00a1b50 0x7feab00a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.140+0000 7feaba7fc700 1 -- 172.21.15.76:0/487155390 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7feabc0ccb90 con 0x7feac4106b80 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.140+0000 7feac93c6700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7feab00a1b50 0x7feab00a4010 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feac0006010 tx=0x7feac000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.144+0000 7feaba7fc700 1 -- 172.21.15.76:0/487155390 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7feabc095ce0 con 0x7feac4106b80 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.343+0000 7feacb62a700 1 -- 172.21.15.76:0/487155390 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7feac4110b90 con 0x7feac4106b80 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.343+0000 7feaba7fc700 1 -- 172.21.15.76:0/487155390 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7feabc0993a0 con 0x7feac4106b80 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.345+0000 7feacb62a700 1 -- 172.21.15.76:0/487155390 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7feab00a1b50 msgr2=0x7feab00a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.345+0000 7feacb62a700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7feab00a1b50 0x7feab00a4010 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feac0006010 tx=0x7feac000b540 comp rx=0 tx=0).stop 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.345+0000 7feacb62a700 1 -- 172.21.15.76:0/487155390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feac4106b80 msgr2=0x7feac419c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.345+0000 7feacb62a700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feac4106b80 0x7feac419c8c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7feabc00ada0 tx=0x7feabc00c330 comp rx=0 tx=0).stop 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.345+0000 7feacb62a700 1 -- 172.21.15.76:0/487155390 shutdown_connections 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.345+0000 7feacb62a700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7feab00a1b50 0x7feab00a4010 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:47.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.345+0000 7feacb62a700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feac4106b80 0x7feac419c8c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:47.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.345+0000 7feacb62a700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feac4100b70 0x7feac4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:47.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.345+0000 7feacb62a700 1 --2- 172.21.15.76:0/487155390 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feac4101530 0x7feac4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:47.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.345+0000 7feacb62a700 1 -- 172.21.15.76:0/487155390 >> 172.21.15.76:0/487155390 conn(0x7feac4077aa0 msgr2=0x7feac40ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:47.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.346+0000 7feacb62a700 1 -- 172.21.15.76:0/487155390 shutdown_connections 2024-03-12T05:54:47.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:47.346+0000 7feacb62a700 1 -- 172.21.15.76:0/487155390 wait complete. 2024-03-12T05:54:48.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:48 smithi076 ceph-mon[32095]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 17 B/s, 0 objects/s recovering 2024-03-12T05:54:48.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:48 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/487155390' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T05:54:48.437 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920825 got 141733920830 for osd.4 2024-03-12T05:54:48.438 DEBUG:teuthology.parallel:result is None 2024-03-12T05:54:48.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:48 smithi082 ceph-mon[36382]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 17 B/s, 0 objects/s recovering 2024-03-12T05:54:48.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:48 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/487155390' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T05:54:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:48 smithi088 ceph-mon[36585]: pgmap v321: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 17 B/s, 0 objects/s recovering 2024-03-12T05:54:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:48 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/487155390' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T05:54:48.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.953+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/558588741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28100bd0 msgr2=0x7f3d28101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:48.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.953+0000 7f3d2d4d9700 1 --2- 172.21.15.76:0/558588741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28100bd0 0x7f3d28101050 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009990 tx=0x7f3d1c009ca0 comp rx=0 tx=0).stop 2024-03-12T05:54:48.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.953+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/558588741 shutdown_connections 2024-03-12T05:54:48.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.953+0000 7f3d2d4d9700 1 --2- 172.21.15.76:0/558588741 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d28101590 0x7f3d2810d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:48.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.953+0000 7f3d2d4d9700 1 --2- 172.21.15.76:0/558588741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d28106c00 0x7f3d28106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:48.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.953+0000 7f3d2d4d9700 1 --2- 172.21.15.76:0/558588741 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28100bd0 0x7f3d28101050 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:48.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.953+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/558588741 >> 172.21.15.76:0/558588741 conn(0x7f3d28076bc0 msgr2=0x7f3d28076fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:48.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.953+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/558588741 shutdown_connections 2024-03-12T05:54:48.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.953+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/558588741 wait complete. 2024-03-12T05:54:48.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.954+0000 7f3d2d4d9700 1 Processor -- start 2024-03-12T05:54:48.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.954+0000 7f3d2d4d9700 1 -- start start 2024-03-12T05:54:48.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.954+0000 7f3d2d4d9700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28100bd0 0x7f3d2819a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:48.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.954+0000 7f3d2d4d9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d28101590 0x7f3d2819a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:48.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.954+0000 7f3d2d4d9700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d28106c00 0x7f3d2819ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:48.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.954+0000 7f3d2d4d9700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2819f340 con 0x7f3d28100bd0 2024-03-12T05:54:48.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.954+0000 7f3d2d4d9700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2819f4b0 con 0x7f3d28106c00 2024-03-12T05:54:48.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.954+0000 7f3d2d4d9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2819f670 con 0x7f3d28101590 2024-03-12T05:54:48.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.954+0000 7f3d26ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28100bd0 0x7f3d2819a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:48.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.954+0000 7f3d26ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28100bd0 0x7f3d2819a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47554/0 (socket says 172.21.15.76:47554) 2024-03-12T05:54:48.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.954+0000 7f3d26ffd700 1 -- 172.21.15.76:0/1851598298 learned_addr learned my addr 172.21.15.76:0/1851598298 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:48.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.954+0000 7f3d277fe700 1 --2- 172.21.15.76:0/1851598298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d28106c00 0x7f3d2819ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:48.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.954+0000 7f3d267fc700 1 --2- 172.21.15.76:0/1851598298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d28101590 0x7f3d2819a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:48.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.955+0000 7f3d26ffd700 1 -- 172.21.15.76:0/1851598298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d28101590 msgr2=0x7f3d2819a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:48.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.955+0000 7f3d26ffd700 1 --2- 172.21.15.76:0/1851598298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d28101590 0x7f3d2819a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:48.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.955+0000 7f3d26ffd700 1 -- 172.21.15.76:0/1851598298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d28106c00 msgr2=0x7f3d2819ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:48.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.955+0000 7f3d26ffd700 1 --2- 172.21.15.76:0/1851598298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d28106c00 0x7f3d2819ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:48.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.955+0000 7f3d26ffd700 1 -- 172.21.15.76:0/1851598298 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c0095f0 con 0x7f3d28100bd0 2024-03-12T05:54:48.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.955+0000 7f3d26ffd700 1 --2- 172.21.15.76:0/1851598298 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28100bd0 0x7f3d2819a460 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800b530 tx=0x7f3d1800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:48.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.955+0000 7f3d0ffff700 1 -- 172.21.15.76:0/1851598298 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d18011830 con 0x7f3d28100bd0 2024-03-12T05:54:48.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.955+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/1851598298 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2819f950 con 0x7f3d28100bd0 2024-03-12T05:54:48.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.955+0000 7f3d0ffff700 1 -- 172.21.15.76:0/1851598298 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d18011e70 con 0x7f3d28100bd0 2024-03-12T05:54:48.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.955+0000 7f3d0ffff700 1 -- 172.21.15.76:0/1851598298 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1800f2e0 con 0x7f3d28100bd0 2024-03-12T05:54:48.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.955+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/1851598298 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d281a71b0 con 0x7f3d28100bd0 2024-03-12T05:54:48.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.956+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/1851598298 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d28103e90 con 0x7f3d28100bd0 2024-03-12T05:54:48.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.957+0000 7f3d0ffff700 1 -- 172.21.15.76:0/1851598298 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f3d18010610 con 0x7f3d28100bd0 2024-03-12T05:54:48.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.957+0000 7f3d0ffff700 1 --2- 172.21.15.76:0/1851598298 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3d100a1b50 0x7f3d100a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:48.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.958+0000 7f3d267fc700 1 --2- 172.21.15.76:0/1851598298 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3d100a1b50 0x7f3d100a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:48.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.958+0000 7f3d0ffff700 1 -- 172.21.15.76:0/1851598298 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3d180cce00 con 0x7f3d28100bd0 2024-03-12T05:54:48.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.958+0000 7f3d267fc700 1 --2- 172.21.15.76:0/1851598298 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3d100a1b50 0x7f3d100a4010 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00b5c0 tx=0x7f3d1c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:48.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:48.962+0000 7f3d0ffff700 1 -- 172.21.15.76:0/1851598298 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3d18095fd0 con 0x7f3d28100bd0 2024-03-12T05:54:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:49.161+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/1851598298 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f3d2819b7c0 con 0x7f3d28100bd0 2024-03-12T05:54:49.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:49.162+0000 7f3d0ffff700 1 -- 172.21.15.76:0/1851598298 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3d18099690 con 0x7f3d28100bd0 2024-03-12T05:54:49.165 INFO:teuthology.orchestra.run.smithi076.stdout:219043332143 2024-03-12T05:54:49.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:49.164+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/1851598298 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3d100a1b50 msgr2=0x7f3d100a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:49.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:49.164+0000 7f3d2d4d9700 1 --2- 172.21.15.76:0/1851598298 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3d100a1b50 0x7f3d100a4010 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00b5c0 tx=0x7f3d1c005960 comp rx=0 tx=0).stop 2024-03-12T05:54:49.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:49.164+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/1851598298 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28100bd0 msgr2=0x7f3d2819a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:49.164+0000 7f3d2d4d9700 1 --2- 172.21.15.76:0/1851598298 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28100bd0 0x7f3d2819a460 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800b530 tx=0x7f3d1800b840 comp rx=0 tx=0).stop 2024-03-12T05:54:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:49.164+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/1851598298 shutdown_connections 2024-03-12T05:54:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:49.164+0000 7f3d2d4d9700 1 --2- 172.21.15.76:0/1851598298 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3d100a1b50 0x7f3d100a4010 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:49.164+0000 7f3d2d4d9700 1 --2- 172.21.15.76:0/1851598298 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d28106c00 0x7f3d2819ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:49.164+0000 7f3d2d4d9700 1 --2- 172.21.15.76:0/1851598298 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28100bd0 0x7f3d2819a460 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:49.164+0000 7f3d2d4d9700 1 --2- 172.21.15.76:0/1851598298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d28101590 0x7f3d2819a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:49.164+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/1851598298 >> 172.21.15.76:0/1851598298 conn(0x7f3d28076bc0 msgr2=0x7f3d280ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:49.164+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/1851598298 shutdown_connections 2024-03-12T05:54:49.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:49.164+0000 7f3d2d4d9700 1 -- 172.21.15.76:0/1851598298 wait complete. 2024-03-12T05:54:49.735 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495663 got 197568495668 for osd.6 2024-03-12T05:54:49.736 DEBUG:teuthology.parallel:result is None 2024-03-12T05:54:50.237 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574918 got 55834574924 for osd.1 2024-03-12T05:54:50.237 DEBUG:teuthology.parallel:result is None 2024-03-12T05:54:50.338 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:50 smithi076 ceph-mon[32095]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 17 B/s, 0 objects/s recovering 2024-03-12T05:54:50.338 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:50 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1851598298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T05:54:50.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:50 smithi082 ceph-mon[36382]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 17 B/s, 0 objects/s recovering 2024-03-12T05:54:50.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:50 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1851598298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T05:54:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:50 smithi088 ceph-mon[36585]: pgmap v322: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 17 B/s, 0 objects/s recovering 2024-03-12T05:54:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:50 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1851598298' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T05:54:50.645 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332135 got 219043332143 for osd.7 2024-03-12T05:54:50.645 DEBUG:teuthology.parallel:result is None 2024-03-12T05:54:51.018 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776160 got 292057776165 for osd.9 2024-03-12T05:54:51.018 DEBUG:teuthology.parallel:result is None 2024-03-12T05:54:51.018 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-12T05:54:51.018 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T05:54:51.291 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:52.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:52 smithi076 ceph-mon[32095]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 14 B/s, 0 objects/s recovering 2024-03-12T05:54:52.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:52 smithi082 ceph-mon[36382]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 14 B/s, 0 objects/s recovering 2024-03-12T05:54:52.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:52 smithi088 ceph-mon[36585]: pgmap v323: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 14 B/s, 0 objects/s recovering 2024-03-12T05:54:53.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:52.999+0000 7f48b0366700 1 -- 172.21.15.76:0/1467948926 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8107ca0 msgr2=0x7f48a8108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:53.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:52.999+0000 7f48b0366700 1 --2- 172.21.15.76:0/1467948926 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8107ca0 0x7f48a8108080 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f48a40099c0 tx=0x7f48a4009cd0 comp rx=0 tx=0).stop 2024-03-12T05:54:53.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:52.999+0000 7f48b0366700 1 -- 172.21.15.76:0/1467948926 shutdown_connections 2024-03-12T05:54:53.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:52.999+0000 7f48b0366700 1 --2- 172.21.15.76:0/1467948926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48a8102680 0x7f48a810eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:53.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:52.999+0000 7f48b0366700 1 --2- 172.21.15.76:0/1467948926 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8107ca0 0x7f48a8108080 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:53.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:52.999+0000 7f48b0366700 1 --2- 172.21.15.76:0/1467948926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48a8101cc0 0x7f48a8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:53.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:52.999+0000 7f48b0366700 1 -- 172.21.15.76:0/1467948926 >> 172.21.15.76:0/1467948926 conn(0x7f48a8078da0 msgr2=0x7f48a80791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:53.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:52.999+0000 7f48b0366700 1 -- 172.21.15.76:0/1467948926 shutdown_connections 2024-03-12T05:54:53.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:52.999+0000 7f48b0366700 1 -- 172.21.15.76:0/1467948926 wait complete. 2024-03-12T05:54:53.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:52.999+0000 7f48b0366700 1 Processor -- start 2024-03-12T05:54:53.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:52.999+0000 7f48b0366700 1 -- start start 2024-03-12T05:54:53.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48b0366700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8101cc0 0x7f48a8198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:53.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48b0366700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48a8102680 0x7f48a81986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:53.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48b0366700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48a8107ca0 0x7f48a819c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:53.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48b0366700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a819cfb0 con 0x7f48a8101cc0 2024-03-12T05:54:53.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48b0366700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a819d120 con 0x7f48a8107ca0 2024-03-12T05:54:53.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48b0366700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a819d2e0 con 0x7f48a8102680 2024-03-12T05:54:53.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48ae102700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8101cc0 0x7f48a8198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:53.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48ae102700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8101cc0 0x7f48a8198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58638/0 (socket says 172.21.15.76:58638) 2024-03-12T05:54:53.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48ae102700 1 -- 172.21.15.76:0/3802286029 learned_addr learned my addr 172.21.15.76:0/3802286029 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:53.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48ae903700 1 --2- 172.21.15.76:0/3802286029 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48a8107ca0 0x7f48a819c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:53.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48ad901700 1 --2- 172.21.15.76:0/3802286029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48a8102680 0x7f48a81986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:53.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48ae102700 1 -- 172.21.15.76:0/3802286029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48a8102680 msgr2=0x7f48a81986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:53.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48ae102700 1 --2- 172.21.15.76:0/3802286029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48a8102680 0x7f48a81986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:53.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48ae102700 1 -- 172.21.15.76:0/3802286029 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48a8107ca0 msgr2=0x7f48a819c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:53.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48ae102700 1 --2- 172.21.15.76:0/3802286029 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48a8107ca0 0x7f48a819c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:53.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48ae102700 1 -- 172.21.15.76:0/3802286029 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a40095f0 con 0x7f48a8101cc0 2024-03-12T05:54:53.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f48ae102700 1 --2- 172.21.15.76:0/3802286029 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8101cc0 0x7f48a8198170 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f48a400b580 tx=0x7f48a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:53.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.000+0000 7f489f7fe700 1 -- 172.21.15.76:0/3802286029 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a401d070 con 0x7f48a8101cc0 2024-03-12T05:54:53.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.001+0000 7f48b0366700 1 -- 172.21.15.76:0/3802286029 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a819d560 con 0x7f48a8101cc0 2024-03-12T05:54:53.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.001+0000 7f489f7fe700 1 -- 172.21.15.76:0/3802286029 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48a4003f50 con 0x7f48a8101cc0 2024-03-12T05:54:53.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.001+0000 7f489f7fe700 1 -- 172.21.15.76:0/3802286029 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4022d10 con 0x7f48a8101cc0 2024-03-12T05:54:53.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.001+0000 7f48b0366700 1 -- 172.21.15.76:0/3802286029 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a81a4e20 con 0x7f48a8101cc0 2024-03-12T05:54:53.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.001+0000 7f48b0366700 1 -- 172.21.15.76:0/3802286029 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a810a180 con 0x7f48a8101cc0 2024-03-12T05:54:53.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.003+0000 7f489f7fe700 1 -- 172.21.15.76:0/3802286029 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f48a4022470 con 0x7f48a8101cc0 2024-03-12T05:54:53.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.003+0000 7f489f7fe700 1 --2- 172.21.15.76:0/3802286029 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f48940a1b50 0x7f48940a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:53.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.003+0000 7f48ad901700 1 --2- 172.21.15.76:0/3802286029 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f48940a1b50 0x7f48940a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:53.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.003+0000 7f489f7fe700 1 -- 172.21.15.76:0/3802286029 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f48a40ce6d0 con 0x7f48a8101cc0 2024-03-12T05:54:53.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.004+0000 7f48ad901700 1 --2- 172.21.15.76:0/3802286029 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f48940a1b50 0x7f48940a4010 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f48a8079410 tx=0x7f489800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:53.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.007+0000 7f489f7fe700 1 -- 172.21.15.76:0/3802286029 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f48a4097820 con 0x7f48a8101cc0 2024-03-12T05:54:53.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.200+0000 7f48b0366700 1 -- 172.21.15.76:0/3802286029 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f48a8199430 con 0x7f48940a1b50 2024-03-12T05:54:53.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.207+0000 7f489f7fe700 1 -- 172.21.15.76:0/3802286029 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258721 (secure 0 0 0) 0x7f48a8199430 con 0x7f48940a1b50 2024-03-12T05:54:53.218 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:54:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.211+0000 7f48b0366700 1 -- 172.21.15.76:0/3802286029 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f48940a1b50 msgr2=0x7f48940a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.211+0000 7f48b0366700 1 --2- 172.21.15.76:0/3802286029 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f48940a1b50 0x7f48940a4010 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f48a8079410 tx=0x7f489800b410 comp rx=0 tx=0).stop 2024-03-12T05:54:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.211+0000 7f48b0366700 1 -- 172.21.15.76:0/3802286029 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8101cc0 msgr2=0x7f48a8198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.211+0000 7f48b0366700 1 --2- 172.21.15.76:0/3802286029 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8101cc0 0x7f48a8198170 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f48a400b580 tx=0x7f48a4004730 comp rx=0 tx=0).stop 2024-03-12T05:54:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.212+0000 7f48b0366700 1 -- 172.21.15.76:0/3802286029 shutdown_connections 2024-03-12T05:54:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.212+0000 7f48b0366700 1 --2- 172.21.15.76:0/3802286029 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f48940a1b50 0x7f48940a4010 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.212+0000 7f48b0366700 1 --2- 172.21.15.76:0/3802286029 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48a8107ca0 0x7f48a819c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:53.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.212+0000 7f48b0366700 1 --2- 172.21.15.76:0/3802286029 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f48a8101cc0 0x7f48a8198170 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.212+0000 7f48b0366700 1 --2- 172.21.15.76:0/3802286029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48a8102680 0x7f48a81986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.212+0000 7f48b0366700 1 -- 172.21.15.76:0/3802286029 >> 172.21.15.76:0/3802286029 conn(0x7f48a8078da0 msgr2=0x7f48a8100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.212+0000 7f48b0366700 1 -- 172.21.15.76:0/3802286029 shutdown_connections 2024-03-12T05:54:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:53.212+0000 7f48b0366700 1 -- 172.21.15.76:0/3802286029 wait complete. 2024-03-12T05:54:53.221 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-03-12T05:54:53.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:53 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:53.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:53 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:53.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:53 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:54:53.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:53 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:54:53.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:53 smithi082 ceph-mon[36382]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:53 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:53 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:53 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:54:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:53 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:54:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:53 smithi088 ceph-mon[36585]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:53.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:53 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:53.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:53 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:53.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:53 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:54:53.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:53 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:54:53.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:53 smithi076 ceph-mon[32095]: from='mgr.14152 172.21.15.76:0/1536779940' entity='mgr.x' 2024-03-12T05:54:53.893 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-03-12T05:54:52.881670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655649,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":572,"num_read_kb":404,"num_write":339,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180875,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375032,"kb_used_data":13308,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566792,"statfs":{"total":1151940427776,"available":1151556395008,"internally_reserved":0,"allocated":13627392,"data_stored":7499271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":59,"num_read_kb":29,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.869598"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.012006+0000","last_change":"2024-03-12T05:53:38.207712+0000","last_active":"2024-03-12T05:54:42.012006+0000","last_peered":"2024-03-12T05:54:42.012006+0000","last_clean":"2024-03-12T05:54:42.012006+0000","last_became_active":"2024-03-12T05:53:38.207362+0000","last_became_peered":"2024-03-12T05:53:38.207362+0000","last_unstale":"2024-03-12T05:54:42.012006+0000","last_undegraded":"2024-03-12T05:54:42.012006+0000","last_fullsized":"2024-03-12T05:54:42.012006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:19.518959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014177+0000","last_change":"2024-03-12T05:53:38.206877+0000","last_active":"2024-03-12T05:54:42.014177+0000","last_peered":"2024-03-12T05:54:42.014177+0000","last_clean":"2024-03-12T05:54:42.014177+0000","last_became_active":"2024-03-12T05:53:38.206468+0000","last_became_peered":"2024-03-12T05:53:38.206468+0000","last_unstale":"2024-03-12T05:54:42.014177+0000","last_undegraded":"2024-03-12T05:54:42.014177+0000","last_fullsized":"2024-03-12T05:54:42.014177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:02:45.138265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014360+0000","last_change":"2024-03-12T05:53:38.214979+0000","last_active":"2024-03-12T05:54:42.014360+0000","last_peered":"2024-03-12T05:54:42.014360+0000","last_clean":"2024-03-12T05:54:42.014360+0000","last_became_active":"2024-03-12T05:53:38.214580+0000","last_became_peered":"2024-03-12T05:53:38.214580+0000","last_unstale":"2024-03-12T05:54:42.014360+0000","last_undegraded":"2024-03-12T05:54:42.014360+0000","last_fullsized":"2024-03-12T05:54:42.014360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:39:58.989757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.478043+0000","last_change":"2024-03-12T05:53:38.218645+0000","last_active":"2024-03-12T05:54:42.478043+0000","last_peered":"2024-03-12T05:54:42.478043+0000","last_clean":"2024-03-12T05:54:42.478043+0000","last_became_active":"2024-03-12T05:53:38.218254+0000","last_became_peered":"2024-03-12T05:53:38.218254+0000","last_unstale":"2024-03-12T05:54:42.478043+0000","last_undegraded":"2024-03-12T05:54:42.478043+0000","last_fullsized":"2024-03-12T05:54:42.478043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:46:09.544989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014191+0000","last_change":"2024-03-12T05:53:38.214588+0000","last_active":"2024-03-12T05:54:42.014191+0000","last_peered":"2024-03-12T05:54:42.014191+0000","last_clean":"2024-03-12T05:54:42.014191+0000","last_became_active":"2024-03-12T05:53:38.214192+0000","last_became_peered":"2024-03-12T05:53:38.214192+0000","last_unstale":"2024-03-12T05:54:42.014191+0000","last_undegraded":"2024-03-12T05:54:42.014191+0000","last_fullsized":"2024-03-12T05:54:42.014191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:54:04.880901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014106+0000","last_change":"2024-03-12T05:53:38.214315+0000","last_active":"2024-03-12T05:54:42.014106+0000","last_peered":"2024-03-12T05:54:42.014106+0000","last_clean":"2024-03-12T05:54:42.014106+0000","last_became_active":"2024-03-12T05:53:38.213881+0000","last_became_peered":"2024-03-12T05:53:38.213881+0000","last_unstale":"2024-03-12T05:54:42.014106+0000","last_undegraded":"2024-03-12T05:54:42.014106+0000","last_fullsized":"2024-03-12T05:54:42.014106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:11:32.834418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686368+0000","last_change":"2024-03-12T05:53:38.214915+0000","last_active":"2024-03-12T05:54:42.686368+0000","last_peered":"2024-03-12T05:54:42.686368+0000","last_clean":"2024-03-12T05:54:42.686368+0000","last_became_active":"2024-03-12T05:53:38.214592+0000","last_became_peered":"2024-03-12T05:53:38.214592+0000","last_unstale":"2024-03-12T05:54:42.686368+0000","last_undegraded":"2024-03-12T05:54:42.686368+0000","last_fullsized":"2024-03-12T05:54:42.686368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:11.548585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016632+0000","last_change":"2024-03-12T05:53:38.219201+0000","last_active":"2024-03-12T05:54:42.016632+0000","last_peered":"2024-03-12T05:54:42.016632+0000","last_clean":"2024-03-12T05:54:42.016632+0000","last_became_active":"2024-03-12T05:53:38.218833+0000","last_became_peered":"2024-03-12T05:53:38.218833+0000","last_unstale":"2024-03-12T05:54:42.016632+0000","last_undegraded":"2024-03-12T05:54:42.016632+0000","last_fullsized":"2024-03-12T05:54:42.016632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:13:36.177325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016442+0000","last_change":"2024-03-12T05:53:38.205126+0000","last_active":"2024-03-12T05:54:42.016442+0000","last_peered":"2024-03-12T05:54:42.016442+0000","last_clean":"2024-03-12T05:54:42.016442+0000","last_became_active":"2024-03-12T05:53:38.204478+0000","last_became_peered":"2024-03-12T05:53:38.204478+0000","last_unstale":"2024-03-12T05:54:42.016442+0000","last_undegraded":"2024-03-12T05:54:42.016442+0000","last_fullsized":"2024-03-12T05:54:42.016442+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:25:56.475134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014402+0000","last_change":"2024-03-12T05:53:38.215186+0000","last_active":"2024-03-12T05:54:42.014402+0000","last_peered":"2024-03-12T05:54:42.014402+0000","last_clean":"2024-03-12T05:54:42.014402+0000","last_became_active":"2024-03-12T05:53:38.214832+0000","last_became_peered":"2024-03-12T05:53:38.214832+0000","last_unstale":"2024-03-12T05:54:42.014402+0000","last_undegraded":"2024-03-12T05:54:42.014402+0000","last_fullsized":"2024-03-12T05:54:42.014402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:56:26.621990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686736+0000","last_change":"2024-03-12T05:53:38.222347+0000","last_active":"2024-03-12T05:54:42.686736+0000","last_peered":"2024-03-12T05:54:42.686736+0000","last_clean":"2024-03-12T05:54:42.686736+0000","last_became_active":"2024-03-12T05:53:38.221637+0000","last_became_peered":"2024-03-12T05:53:38.221637+0000","last_unstale":"2024-03-12T05:54:42.686736+0000","last_undegraded":"2024-03-12T05:54:42.686736+0000","last_fullsized":"2024-03-12T05:54:42.686736+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:16:26.652673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.478305+0000","last_change":"2024-03-12T05:53:38.218687+0000","last_active":"2024-03-12T05:54:42.478305+0000","last_peered":"2024-03-12T05:54:42.478305+0000","last_clean":"2024-03-12T05:54:42.478305+0000","last_became_active":"2024-03-12T05:53:38.218393+0000","last_became_peered":"2024-03-12T05:53:38.218393+0000","last_unstale":"2024-03-12T05:54:42.478305+0000","last_undegraded":"2024-03-12T05:54:42.478305+0000","last_fullsized":"2024-03-12T05:54:42.478305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:41:39.918992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017284+0000","last_change":"2024-03-12T05:53:38.207432+0000","last_active":"2024-03-12T05:54:42.017284+0000","last_peered":"2024-03-12T05:54:42.017284+0000","last_clean":"2024-03-12T05:54:42.017284+0000","last_became_active":"2024-03-12T05:53:38.207099+0000","last_became_peered":"2024-03-12T05:53:38.207099+0000","last_unstale":"2024-03-12T05:54:42.017284+0000","last_undegraded":"2024-03-12T05:54:42.017284+0000","last_fullsized":"2024-03-12T05:54:42.017284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:20:08.504723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014250+0000","last_change":"2024-03-12T05:53:38.207086+0000","last_active":"2024-03-12T05:54:42.014250+0000","last_peered":"2024-03-12T05:54:42.014250+0000","last_clean":"2024-03-12T05:54:42.014250+0000","last_became_active":"2024-03-12T05:53:38.206714+0000","last_became_peered":"2024-03-12T05:53:38.206714+0000","last_unstale":"2024-03-12T05:54:42.014250+0000","last_undegraded":"2024-03-12T05:54:42.014250+0000","last_fullsized":"2024-03-12T05:54:42.014250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:07:13.012272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014011+0000","last_change":"2024-03-12T05:53:38.217179+0000","last_active":"2024-03-12T05:54:42.014011+0000","last_peered":"2024-03-12T05:54:42.014011+0000","last_clean":"2024-03-12T05:54:42.014011+0000","last_became_active":"2024-03-12T05:53:38.216820+0000","last_became_peered":"2024-03-12T05:53:38.216820+0000","last_unstale":"2024-03-12T05:54:42.014011+0000","last_undegraded":"2024-03-12T05:54:42.014011+0000","last_fullsized":"2024-03-12T05:54:42.014011+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:01:08.842611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.015965+0000","last_change":"2024-03-12T05:53:38.220224+0000","last_active":"2024-03-12T05:54:42.015965+0000","last_peered":"2024-03-12T05:54:42.015965+0000","last_clean":"2024-03-12T05:54:42.015965+0000","last_became_active":"2024-03-12T05:53:38.219878+0000","last_became_peered":"2024-03-12T05:53:38.219878+0000","last_unstale":"2024-03-12T05:54:42.015965+0000","last_undegraded":"2024-03-12T05:54:42.015965+0000","last_fullsized":"2024-03-12T05:54:42.015965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:53:16.545807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.012133+0000","last_change":"2024-03-12T05:53:38.208524+0000","last_active":"2024-03-12T05:54:42.012133+0000","last_peered":"2024-03-12T05:54:42.012133+0000","last_clean":"2024-03-12T05:54:42.012133+0000","last_became_active":"2024-03-12T05:53:38.208209+0000","last_became_peered":"2024-03-12T05:53:38.208209+0000","last_unstale":"2024-03-12T05:54:42.012133+0000","last_undegraded":"2024-03-12T05:54:42.012133+0000","last_fullsized":"2024-03-12T05:54:42.012133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:28:56.334066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013915+0000","last_change":"2024-03-12T05:53:38.208035+0000","last_active":"2024-03-12T05:54:42.013915+0000","last_peered":"2024-03-12T05:54:42.013915+0000","last_clean":"2024-03-12T05:54:42.013915+0000","last_became_active":"2024-03-12T05:53:38.207679+0000","last_became_peered":"2024-03-12T05:53:38.207679+0000","last_unstale":"2024-03-12T05:54:42.013915+0000","last_undegraded":"2024-03-12T05:54:42.013915+0000","last_fullsized":"2024-03-12T05:54:42.013915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:40:20.620552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019897+0000","last_change":"2024-03-12T05:53:38.207077+0000","last_active":"2024-03-12T05:54:42.019897+0000","last_peered":"2024-03-12T05:54:42.019897+0000","last_clean":"2024-03-12T05:54:42.019897+0000","last_became_active":"2024-03-12T05:53:38.206720+0000","last_became_peered":"2024-03-12T05:53:38.206720+0000","last_unstale":"2024-03-12T05:54:42.019897+0000","last_undegraded":"2024-03-12T05:54:42.019897+0000","last_fullsized":"2024-03-12T05:54:42.019897+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:45:54.628394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013942+0000","last_change":"2024-03-12T05:53:37.201637+0000","last_active":"2024-03-12T05:54:42.013942+0000","last_peered":"2024-03-12T05:54:42.013942+0000","last_clean":"2024-03-12T05:54:42.013942+0000","last_became_active":"2024-03-12T05:53:37.201289+0000","last_became_peered":"2024-03-12T05:53:37.201289+0000","last_unstale":"2024-03-12T05:54:42.013942+0000","last_undegraded":"2024-03-12T05:54:42.013942+0000","last_fullsized":"2024-03-12T05:54:42.013942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:59:43.829570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018092+0000","last_change":"2024-03-12T05:53:38.218052+0000","last_active":"2024-03-12T05:54:42.018092+0000","last_peered":"2024-03-12T05:54:42.018092+0000","last_clean":"2024-03-12T05:54:42.018092+0000","last_became_active":"2024-03-12T05:53:38.217738+0000","last_became_peered":"2024-03-12T05:53:38.217738+0000","last_unstale":"2024-03-12T05:54:42.018092+0000","last_undegraded":"2024-03-12T05:54:42.018092+0000","last_fullsized":"2024-03-12T05:54:42.018092+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:59:40.599573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018614+0000","last_change":"2024-03-12T05:53:37.210321+0000","last_active":"2024-03-12T05:54:42.018614+0000","last_peered":"2024-03-12T05:54:42.018614+0000","last_clean":"2024-03-12T05:54:42.018614+0000","last_became_active":"2024-03-12T05:53:37.210018+0000","last_became_peered":"2024-03-12T05:53:37.210018+0000","last_unstale":"2024-03-12T05:54:42.018614+0000","last_undegraded":"2024-03-12T05:54:42.018614+0000","last_fullsized":"2024-03-12T05:54:42.018614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:13:45.433332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013944+0000","last_change":"2024-03-12T05:53:38.207801+0000","last_active":"2024-03-12T05:54:42.013944+0000","last_peered":"2024-03-12T05:54:42.013944+0000","last_clean":"2024-03-12T05:54:42.013944+0000","last_became_active":"2024-03-12T05:53:38.207467+0000","last_became_peered":"2024-03-12T05:53:38.207467+0000","last_unstale":"2024-03-12T05:54:42.013944+0000","last_undegraded":"2024-03-12T05:54:42.013944+0000","last_fullsized":"2024-03-12T05:54:42.013944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:56:14.469662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014569+0000","last_change":"2024-03-12T05:53:37.208288+0000","last_active":"2024-03-12T05:54:42.014569+0000","last_peered":"2024-03-12T05:54:42.014569+0000","last_clean":"2024-03-12T05:54:42.014569+0000","last_became_active":"2024-03-12T05:53:37.207929+0000","last_became_peered":"2024-03-12T05:53:37.207929+0000","last_unstale":"2024-03-12T05:54:42.014569+0000","last_undegraded":"2024-03-12T05:54:42.014569+0000","last_fullsized":"2024-03-12T05:54:42.014569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:10:52.719116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016593+0000","last_change":"2024-03-12T05:53:38.207270+0000","last_active":"2024-03-12T05:54:42.016593+0000","last_peered":"2024-03-12T05:54:42.016593+0000","last_clean":"2024-03-12T05:54:42.016593+0000","last_became_active":"2024-03-12T05:53:38.205782+0000","last_became_peered":"2024-03-12T05:53:38.205782+0000","last_unstale":"2024-03-12T05:54:42.016593+0000","last_undegraded":"2024-03-12T05:54:42.016593+0000","last_fullsized":"2024-03-12T05:54:42.016593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:02:24.875444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014278+0000","last_change":"2024-03-12T05:53:37.203899+0000","last_active":"2024-03-12T05:54:42.014278+0000","last_peered":"2024-03-12T05:54:42.014278+0000","last_clean":"2024-03-12T05:54:42.014278+0000","last_became_active":"2024-03-12T05:53:37.203450+0000","last_became_peered":"2024-03-12T05:53:37.203450+0000","last_unstale":"2024-03-12T05:54:42.014278+0000","last_undegraded":"2024-03-12T05:54:42.014278+0000","last_fullsized":"2024-03-12T05:54:42.014278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:25.773826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.480629+0000","last_change":"2024-03-12T05:54:42.480247+0000","last_active":"2024-03-12T05:54:42.480629+0000","last_peered":"2024-03-12T05:54:42.480629+0000","last_clean":"2024-03-12T05:54:42.480629+0000","last_became_active":"2024-03-12T05:54:42.479952+0000","last_became_peered":"2024-03-12T05:54:42.479952+0000","last_unstale":"2024-03-12T05:54:42.480629+0000","last_undegraded":"2024-03-12T05:54:42.480629+0000","last_fullsized":"2024-03-12T05:54:42.480629+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:45:20.439878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":97,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:48.461130+0000","last_change":"2024-03-12T05:53:37.205401+0000","last_active":"2024-03-12T05:54:48.461130+0000","last_peered":"2024-03-12T05:54:48.461130+0000","last_clean":"2024-03-12T05:54:48.461130+0000","last_became_active":"2024-03-12T05:53:37.204993+0000","last_became_peered":"2024-03-12T05:53:37.204993+0000","last_unstale":"2024-03-12T05:54:48.461130+0000","last_undegraded":"2024-03-12T05:54:48.461130+0000","last_fullsized":"2024-03-12T05:54:48.461130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:56:06.177387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014148+0000","last_change":"2024-03-12T05:53:38.210562+0000","last_active":"2024-03-12T05:54:42.014148+0000","last_peered":"2024-03-12T05:54:42.014148+0000","last_clean":"2024-03-12T05:54:42.014148+0000","last_became_active":"2024-03-12T05:53:38.210235+0000","last_became_peered":"2024-03-12T05:53:38.210235+0000","last_unstale":"2024-03-12T05:54:42.014148+0000","last_undegraded":"2024-03-12T05:54:42.014148+0000","last_fullsized":"2024-03-12T05:54:42.014148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:35.445812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014118+0000","last_change":"2024-03-12T05:53:37.204986+0000","last_active":"2024-03-12T05:54:42.014118+0000","last_peered":"2024-03-12T05:54:42.014118+0000","last_clean":"2024-03-12T05:54:42.014118+0000","last_became_active":"2024-03-12T05:53:37.204580+0000","last_became_peered":"2024-03-12T05:53:37.204580+0000","last_unstale":"2024-03-12T05:54:42.014118+0000","last_undegraded":"2024-03-12T05:54:42.014118+0000","last_fullsized":"2024-03-12T05:54:42.014118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:37:34.584395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017753+0000","last_change":"2024-03-12T05:53:38.214948+0000","last_active":"2024-03-12T05:54:42.017753+0000","last_peered":"2024-03-12T05:54:42.017753+0000","last_clean":"2024-03-12T05:54:42.017753+0000","last_became_active":"2024-03-12T05:53:38.213748+0000","last_became_peered":"2024-03-12T05:53:38.213748+0000","last_unstale":"2024-03-12T05:54:42.017753+0000","last_undegraded":"2024-03-12T05:54:42.017753+0000","last_fullsized":"2024-03-12T05:54:42.017753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:12:45.994129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017009+0000","last_change":"2024-03-12T05:53:37.198167+0000","last_active":"2024-03-12T05:54:42.017009+0000","last_peered":"2024-03-12T05:54:42.017009+0000","last_clean":"2024-03-12T05:54:42.017009+0000","last_became_active":"2024-03-12T05:53:37.197808+0000","last_became_peered":"2024-03-12T05:53:37.197808+0000","last_unstale":"2024-03-12T05:54:42.017009+0000","last_undegraded":"2024-03-12T05:54:42.017009+0000","last_fullsized":"2024-03-12T05:54:42.017009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:40:19.498546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016514+0000","last_change":"2024-03-12T05:53:38.205464+0000","last_active":"2024-03-12T05:54:42.016514+0000","last_peered":"2024-03-12T05:54:42.016514+0000","last_clean":"2024-03-12T05:54:42.016514+0000","last_became_active":"2024-03-12T05:53:38.205129+0000","last_became_peered":"2024-03-12T05:53:38.205129+0000","last_unstale":"2024-03-12T05:54:42.016514+0000","last_undegraded":"2024-03-12T05:54:42.016514+0000","last_fullsized":"2024-03-12T05:54:42.016514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:38:55.460862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017494+0000","last_change":"2024-03-12T05:53:37.201572+0000","last_active":"2024-03-12T05:54:42.017494+0000","last_peered":"2024-03-12T05:54:42.017494+0000","last_clean":"2024-03-12T05:54:42.017494+0000","last_became_active":"2024-03-12T05:53:37.201230+0000","last_became_peered":"2024-03-12T05:53:37.201230+0000","last_unstale":"2024-03-12T05:54:42.017494+0000","last_undegraded":"2024-03-12T05:54:42.017494+0000","last_fullsized":"2024-03-12T05:54:42.017494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:16:13.715459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013535+0000","last_change":"2024-03-12T05:53:38.210696+0000","last_active":"2024-03-12T05:54:42.013535+0000","last_peered":"2024-03-12T05:54:42.013535+0000","last_clean":"2024-03-12T05:54:42.013535+0000","last_became_active":"2024-03-12T05:53:38.210325+0000","last_became_peered":"2024-03-12T05:53:38.210325+0000","last_unstale":"2024-03-12T05:54:42.013535+0000","last_undegraded":"2024-03-12T05:54:42.013535+0000","last_fullsized":"2024-03-12T05:54:42.013535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:25:36.397314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:47.458874+0000","last_change":"2024-03-12T05:53:37.201712+0000","last_active":"2024-03-12T05:54:47.458874+0000","last_peered":"2024-03-12T05:54:47.458874+0000","last_clean":"2024-03-12T05:54:47.458874+0000","last_became_active":"2024-03-12T05:53:37.200794+0000","last_became_peered":"2024-03-12T05:53:37.200794+0000","last_unstale":"2024-03-12T05:54:47.458874+0000","last_undegraded":"2024-03-12T05:54:47.458874+0000","last_fullsized":"2024-03-12T05:54:47.458874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:43:29.673717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016427+0000","last_change":"2024-03-12T05:53:38.206647+0000","last_active":"2024-03-12T05:54:42.016427+0000","last_peered":"2024-03-12T05:54:42.016427+0000","last_clean":"2024-03-12T05:54:42.016427+0000","last_became_active":"2024-03-12T05:53:38.206254+0000","last_became_peered":"2024-03-12T05:53:38.206254+0000","last_unstale":"2024-03-12T05:54:42.016427+0000","last_undegraded":"2024-03-12T05:54:42.016427+0000","last_fullsized":"2024-03-12T05:54:42.016427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:47:20.455360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687585+0000","last_change":"2024-03-12T05:53:37.208692+0000","last_active":"2024-03-12T05:54:42.687585+0000","last_peered":"2024-03-12T05:54:42.687585+0000","last_clean":"2024-03-12T05:54:42.687585+0000","last_became_active":"2024-03-12T05:53:37.208270+0000","last_became_peered":"2024-03-12T05:53:37.208270+0000","last_unstale":"2024-03-12T05:54:42.687585+0000","last_undegraded":"2024-03-12T05:54:42.687585+0000","last_fullsized":"2024-03-12T05:54:42.687585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:51:38.217999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013830+0000","last_change":"2024-03-12T05:53:38.218202+0000","last_active":"2024-03-12T05:54:42.013830+0000","last_peered":"2024-03-12T05:54:42.013830+0000","last_clean":"2024-03-12T05:54:42.013830+0000","last_became_active":"2024-03-12T05:53:38.217798+0000","last_became_peered":"2024-03-12T05:53:38.217798+0000","last_unstale":"2024-03-12T05:54:42.013830+0000","last_undegraded":"2024-03-12T05:54:42.013830+0000","last_fullsized":"2024-03-12T05:54:42.013830+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:09:29.422125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019722+0000","last_change":"2024-03-12T05:53:37.203252+0000","last_active":"2024-03-12T05:54:42.019722+0000","last_peered":"2024-03-12T05:54:42.019722+0000","last_clean":"2024-03-12T05:54:42.019722+0000","last_became_active":"2024-03-12T05:53:37.202916+0000","last_became_peered":"2024-03-12T05:53:37.202916+0000","last_unstale":"2024-03-12T05:54:42.019722+0000","last_undegraded":"2024-03-12T05:54:42.019722+0000","last_fullsized":"2024-03-12T05:54:42.019722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:09:34.629085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.021528+0000","last_change":"2024-03-12T05:54:42.021528+0000","last_active":"2024-03-12T05:54:42.021528+0000","last_peered":"2024-03-12T05:54:42.021528+0000","last_clean":"2024-03-12T05:54:42.021528+0000","last_became_active":"2024-03-12T05:54:42.021015+0000","last_became_peered":"2024-03-12T05:54:42.021015+0000","last_unstale":"2024-03-12T05:54:42.021528+0000","last_undegraded":"2024-03-12T05:54:42.021528+0000","last_fullsized":"2024-03-12T05:54:42.021528+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:16.998017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'28","reported_seq":191,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:48.460268+0000","last_change":"2024-03-12T05:53:37.206271+0000","last_active":"2024-03-12T05:54:48.460268+0000","last_peered":"2024-03-12T05:54:48.460268+0000","last_clean":"2024-03-12T05:54:48.460268+0000","last_became_active":"2024-03-12T05:53:37.205920+0000","last_became_peered":"2024-03-12T05:53:37.205920+0000","last_unstale":"2024-03-12T05:54:48.460268+0000","last_undegraded":"2024-03-12T05:54:48.460268+0000","last_fullsized":"2024-03-12T05:54:48.460268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:30.140159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":144,"num_read_kb":131,"num_write":29,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014385+0000","last_change":"2024-03-12T05:53:38.211397+0000","last_active":"2024-03-12T05:54:42.014385+0000","last_peered":"2024-03-12T05:54:42.014385+0000","last_clean":"2024-03-12T05:54:42.014385+0000","last_became_active":"2024-03-12T05:53:38.211045+0000","last_became_peered":"2024-03-12T05:53:38.211045+0000","last_unstale":"2024-03-12T05:54:42.014385+0000","last_undegraded":"2024-03-12T05:54:42.014385+0000","last_fullsized":"2024-03-12T05:54:42.014385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:21:42.510436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019264+0000","last_change":"2024-03-12T05:53:37.210871+0000","last_active":"2024-03-12T05:54:42.019264+0000","last_peered":"2024-03-12T05:54:42.019264+0000","last_clean":"2024-03-12T05:54:42.019264+0000","last_became_active":"2024-03-12T05:53:37.210526+0000","last_became_peered":"2024-03-12T05:53:37.210526+0000","last_unstale":"2024-03-12T05:54:42.019264+0000","last_undegraded":"2024-03-12T05:54:42.019264+0000","last_fullsized":"2024-03-12T05:54:42.019264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:40:46.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017098+0000","last_change":"2024-03-12T05:53:38.213751+0000","last_active":"2024-03-12T05:54:42.017098+0000","last_peered":"2024-03-12T05:54:42.017098+0000","last_clean":"2024-03-12T05:54:42.017098+0000","last_became_active":"2024-03-12T05:53:38.211558+0000","last_became_peered":"2024-03-12T05:53:38.211558+0000","last_unstale":"2024-03-12T05:54:42.017098+0000","last_undegraded":"2024-03-12T05:54:42.017098+0000","last_fullsized":"2024-03-12T05:54:42.017098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:40:44.194548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018330+0000","last_change":"2024-03-12T05:53:37.207368+0000","last_active":"2024-03-12T05:54:42.018330+0000","last_peered":"2024-03-12T05:54:42.018330+0000","last_clean":"2024-03-12T05:54:42.018330+0000","last_became_active":"2024-03-12T05:53:37.207066+0000","last_became_peered":"2024-03-12T05:53:37.207066+0000","last_unstale":"2024-03-12T05:54:42.018330+0000","last_undegraded":"2024-03-12T05:54:42.018330+0000","last_fullsized":"2024-03-12T05:54:42.018330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:28:58.155360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686427+0000","last_change":"2024-03-12T05:53:38.217079+0000","last_active":"2024-03-12T05:54:42.686427+0000","last_peered":"2024-03-12T05:54:42.686427+0000","last_clean":"2024-03-12T05:54:42.686427+0000","last_became_active":"2024-03-12T05:53:38.216757+0000","last_became_peered":"2024-03-12T05:53:38.216757+0000","last_unstale":"2024-03-12T05:54:42.686427+0000","last_undegraded":"2024-03-12T05:54:42.686427+0000","last_fullsized":"2024-03-12T05:54:42.686427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:48:13.125867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":93,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:49.488902+0000","last_change":"2024-03-12T05:53:37.202390+0000","last_active":"2024-03-12T05:54:49.488902+0000","last_peered":"2024-03-12T05:54:49.488902+0000","last_clean":"2024-03-12T05:54:49.488902+0000","last_became_active":"2024-03-12T05:53:37.202020+0000","last_became_peered":"2024-03-12T05:53:37.202020+0000","last_unstale":"2024-03-12T05:54:49.488902+0000","last_undegraded":"2024-03-12T05:54:49.488902+0000","last_fullsized":"2024-03-12T05:54:49.488902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:06:35.293227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686330+0000","last_change":"2024-03-12T05:53:38.218464+0000","last_active":"2024-03-12T05:54:42.686330+0000","last_peered":"2024-03-12T05:54:42.686330+0000","last_clean":"2024-03-12T05:54:42.686330+0000","last_became_active":"2024-03-12T05:53:38.218136+0000","last_became_peered":"2024-03-12T05:53:38.218136+0000","last_unstale":"2024-03-12T05:54:42.686330+0000","last_undegraded":"2024-03-12T05:54:42.686330+0000","last_fullsized":"2024-03-12T05:54:42.686330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:31:05.409148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013820+0000","last_change":"2024-03-12T05:53:37.205161+0000","last_active":"2024-03-12T05:54:42.013820+0000","last_peered":"2024-03-12T05:54:42.013820+0000","last_clean":"2024-03-12T05:54:42.013820+0000","last_became_active":"2024-03-12T05:53:37.204799+0000","last_became_peered":"2024-03-12T05:53:37.204799+0000","last_unstale":"2024-03-12T05:54:42.013820+0000","last_undegraded":"2024-03-12T05:54:42.013820+0000","last_fullsized":"2024-03-12T05:54:42.013820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:29:32.065635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017614+0000","last_change":"2024-03-12T05:53:38.216321+0000","last_active":"2024-03-12T05:54:42.017614+0000","last_peered":"2024-03-12T05:54:42.017614+0000","last_clean":"2024-03-12T05:54:42.017614+0000","last_became_active":"2024-03-12T05:53:38.215893+0000","last_became_peered":"2024-03-12T05:53:38.215893+0000","last_unstale":"2024-03-12T05:54:42.017614+0000","last_undegraded":"2024-03-12T05:54:42.017614+0000","last_fullsized":"2024-03-12T05:54:42.017614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:32:27.603930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014660+0000","last_change":"2024-03-12T05:53:37.203638+0000","last_active":"2024-03-12T05:54:42.014660+0000","last_peered":"2024-03-12T05:54:42.014660+0000","last_clean":"2024-03-12T05:54:42.014660+0000","last_became_active":"2024-03-12T05:53:37.203217+0000","last_became_peered":"2024-03-12T05:53:37.203217+0000","last_unstale":"2024-03-12T05:54:42.014660+0000","last_undegraded":"2024-03-12T05:54:42.014660+0000","last_fullsized":"2024-03-12T05:54:42.014660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:10.521197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.011922+0000","last_change":"2024-03-12T05:53:38.210491+0000","last_active":"2024-03-12T05:54:42.011922+0000","last_peered":"2024-03-12T05:54:42.011922+0000","last_clean":"2024-03-12T05:54:42.011922+0000","last_became_active":"2024-03-12T05:53:38.210169+0000","last_became_peered":"2024-03-12T05:53:38.210169+0000","last_unstale":"2024-03-12T05:54:42.011922+0000","last_undegraded":"2024-03-12T05:54:42.011922+0000","last_fullsized":"2024-03-12T05:54:42.011922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:56:05.384834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013770+0000","last_change":"2024-03-12T05:53:37.204478+0000","last_active":"2024-03-12T05:54:42.013770+0000","last_peered":"2024-03-12T05:54:42.013770+0000","last_clean":"2024-03-12T05:54:42.013770+0000","last_became_active":"2024-03-12T05:53:37.204160+0000","last_became_peered":"2024-03-12T05:53:37.204160+0000","last_unstale":"2024-03-12T05:54:42.013770+0000","last_undegraded":"2024-03-12T05:54:42.013770+0000","last_fullsized":"2024-03-12T05:54:42.013770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:07.597731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.026040+0000","last_change":"2024-03-12T05:54:42.026040+0000","last_active":"2024-03-12T05:54:42.026040+0000","last_peered":"2024-03-12T05:54:42.026040+0000","last_clean":"2024-03-12T05:54:42.026040+0000","last_became_active":"2024-03-12T05:54:42.018230+0000","last_became_peered":"2024-03-12T05:54:42.018230+0000","last_unstale":"2024-03-12T05:54:42.026040+0000","last_undegraded":"2024-03-12T05:54:42.026040+0000","last_fullsized":"2024-03-12T05:54:42.026040+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:32:28.703339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687385+0000","last_change":"2024-03-12T05:53:37.205983+0000","last_active":"2024-03-12T05:54:42.687385+0000","last_peered":"2024-03-12T05:54:42.687385+0000","last_clean":"2024-03-12T05:54:42.687385+0000","last_became_active":"2024-03-12T05:53:37.205629+0000","last_became_peered":"2024-03-12T05:53:37.205629+0000","last_unstale":"2024-03-12T05:54:42.687385+0000","last_undegraded":"2024-03-12T05:54:42.687385+0000","last_fullsized":"2024-03-12T05:54:42.687385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:37.928623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014350+0000","last_change":"2024-03-12T05:53:38.213752+0000","last_active":"2024-03-12T05:54:42.014350+0000","last_peered":"2024-03-12T05:54:42.014350+0000","last_clean":"2024-03-12T05:54:42.014350+0000","last_became_active":"2024-03-12T05:53:38.213407+0000","last_became_peered":"2024-03-12T05:53:38.213407+0000","last_unstale":"2024-03-12T05:54:42.014350+0000","last_undegraded":"2024-03-12T05:54:42.014350+0000","last_fullsized":"2024-03-12T05:54:42.014350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:21:15.703455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.022196+0000","last_change":"2024-03-12T05:54:42.022196+0000","last_active":"2024-03-12T05:54:42.022196+0000","last_peered":"2024-03-12T05:54:42.022196+0000","last_clean":"2024-03-12T05:54:42.022196+0000","last_became_active":"2024-03-12T05:54:42.021774+0000","last_became_peered":"2024-03-12T05:54:42.021774+0000","last_unstale":"2024-03-12T05:54:42.022196+0000","last_undegraded":"2024-03-12T05:54:42.022196+0000","last_fullsized":"2024-03-12T05:54:42.022196+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:34:39.290359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:48.486891+0000","last_change":"2024-03-12T05:53:37.202782+0000","last_active":"2024-03-12T05:54:48.486891+0000","last_peered":"2024-03-12T05:54:48.486891+0000","last_clean":"2024-03-12T05:54:48.486891+0000","last_became_active":"2024-03-12T05:53:37.202439+0000","last_became_peered":"2024-03-12T05:53:37.202439+0000","last_unstale":"2024-03-12T05:54:48.486891+0000","last_undegraded":"2024-03-12T05:54:48.486891+0000","last_fullsized":"2024-03-12T05:54:48.486891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:07:18.939629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017722+0000","last_change":"2024-03-12T05:53:38.217993+0000","last_active":"2024-03-12T05:54:42.017722+0000","last_peered":"2024-03-12T05:54:42.017722+0000","last_clean":"2024-03-12T05:54:42.017722+0000","last_became_active":"2024-03-12T05:53:38.217613+0000","last_became_peered":"2024-03-12T05:53:38.217613+0000","last_unstale":"2024-03-12T05:54:42.017722+0000","last_undegraded":"2024-03-12T05:54:42.017722+0000","last_fullsized":"2024-03-12T05:54:42.017722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:49:50.960004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.478908+0000","last_change":"2024-03-12T05:53:37.203662+0000","last_active":"2024-03-12T05:54:42.478908+0000","last_peered":"2024-03-12T05:54:42.478908+0000","last_clean":"2024-03-12T05:54:42.478908+0000","last_became_active":"2024-03-12T05:53:37.203309+0000","last_became_peered":"2024-03-12T05:53:37.203309+0000","last_unstale":"2024-03-12T05:54:42.478908+0000","last_undegraded":"2024-03-12T05:54:42.478908+0000","last_fullsized":"2024-03-12T05:54:42.478908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:47:24.603686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":351,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687252+0000","last_change":"2024-03-12T05:52:27.278524+0000","last_active":"2024-03-12T05:54:42.687252+0000","last_peered":"2024-03-12T05:54:42.687252+0000","last_clean":"2024-03-12T05:54:42.687252+0000","last_became_active":"2024-03-12T05:52:27.278010+0000","last_became_peered":"2024-03-12T05:52:27.278010+0000","last_unstale":"2024-03-12T05:54:42.687252+0000","last_undegraded":"2024-03-12T05:54:42.687252+0000","last_fullsized":"2024-03-12T05:54:42.687252+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:49:09.175725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:49:09.175725+0000","last_clean_scrub_stamp":"2024-03-12T05:49:09.175725+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:24:40.380724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016346+0000","last_change":"2024-03-12T05:53:38.219451+0000","last_active":"2024-03-12T05:54:42.016346+0000","last_peered":"2024-03-12T05:54:42.016346+0000","last_clean":"2024-03-12T05:54:42.016346+0000","last_became_active":"2024-03-12T05:53:38.218840+0000","last_became_peered":"2024-03-12T05:53:38.218840+0000","last_unstale":"2024-03-12T05:54:42.016346+0000","last_undegraded":"2024-03-12T05:54:42.016346+0000","last_fullsized":"2024-03-12T05:54:42.016346+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:05.595822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687398+0000","last_change":"2024-03-12T05:53:37.210352+0000","last_active":"2024-03-12T05:54:42.687398+0000","last_peered":"2024-03-12T05:54:42.687398+0000","last_clean":"2024-03-12T05:54:42.687398+0000","last_became_active":"2024-03-12T05:53:37.210002+0000","last_became_peered":"2024-03-12T05:53:37.210002+0000","last_unstale":"2024-03-12T05:54:42.687398+0000","last_undegraded":"2024-03-12T05:54:42.687398+0000","last_fullsized":"2024-03-12T05:54:42.687398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:49:52.368411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.022961+0000","last_change":"2024-03-12T05:54:42.022489+0000","last_active":"2024-03-12T05:54:42.022961+0000","last_peered":"2024-03-12T05:54:42.022961+0000","last_clean":"2024-03-12T05:54:42.022961+0000","last_became_active":"2024-03-12T05:54:42.016400+0000","last_became_peered":"2024-03-12T05:54:42.016400+0000","last_unstale":"2024-03-12T05:54:42.022961+0000","last_undegraded":"2024-03-12T05:54:42.022961+0000","last_fullsized":"2024-03-12T05:54:42.022961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:32:05.713420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017332+0000","last_change":"2024-03-12T05:53:38.203951+0000","last_active":"2024-03-12T05:54:42.017332+0000","last_peered":"2024-03-12T05:54:42.017332+0000","last_clean":"2024-03-12T05:54:42.017332+0000","last_became_active":"2024-03-12T05:53:38.203617+0000","last_became_peered":"2024-03-12T05:53:38.203617+0000","last_unstale":"2024-03-12T05:54:42.017332+0000","last_undegraded":"2024-03-12T05:54:42.017332+0000","last_fullsized":"2024-03-12T05:54:42.017332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:19.138220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013822+0000","last_change":"2024-03-12T05:53:37.203770+0000","last_active":"2024-03-12T05:54:42.013822+0000","last_peered":"2024-03-12T05:54:42.013822+0000","last_clean":"2024-03-12T05:54:42.013822+0000","last_became_active":"2024-03-12T05:53:37.203456+0000","last_became_peered":"2024-03-12T05:53:37.203456+0000","last_unstale":"2024-03-12T05:54:42.013822+0000","last_undegraded":"2024-03-12T05:54:42.013822+0000","last_fullsized":"2024-03-12T05:54:42.013822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:32.291536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018655+0000","last_change":"2024-03-12T05:53:38.214872+0000","last_active":"2024-03-12T05:54:42.018655+0000","last_peered":"2024-03-12T05:54:42.018655+0000","last_clean":"2024-03-12T05:54:42.018655+0000","last_became_active":"2024-03-12T05:53:38.214139+0000","last_became_peered":"2024-03-12T05:53:38.214139+0000","last_unstale":"2024-03-12T05:54:42.018655+0000","last_undegraded":"2024-03-12T05:54:42.018655+0000","last_fullsized":"2024-03-12T05:54:42.018655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:20:21.808039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.479035+0000","last_change":"2024-03-12T05:53:37.203852+0000","last_active":"2024-03-12T05:54:42.479035+0000","last_peered":"2024-03-12T05:54:42.479035+0000","last_clean":"2024-03-12T05:54:42.479035+0000","last_became_active":"2024-03-12T05:53:37.203416+0000","last_became_peered":"2024-03-12T05:53:37.203416+0000","last_unstale":"2024-03-12T05:54:42.479035+0000","last_undegraded":"2024-03-12T05:54:42.479035+0000","last_fullsized":"2024-03-12T05:54:42.479035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:09:08.095957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.688395+0000","last_change":"2024-03-12T05:54:42.688395+0000","last_active":"2024-03-12T05:54:42.688395+0000","last_peered":"2024-03-12T05:54:42.688395+0000","last_clean":"2024-03-12T05:54:42.688395+0000","last_became_active":"2024-03-12T05:54:42.688094+0000","last_became_peered":"2024-03-12T05:54:42.688094+0000","last_unstale":"2024-03-12T05:54:42.688395+0000","last_undegraded":"2024-03-12T05:54:42.688395+0000","last_fullsized":"2024-03-12T05:54:42.688395+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:29:42.921726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.025218+0000","last_change":"2024-03-12T05:54:42.025218+0000","last_active":"2024-03-12T05:54:42.025218+0000","last_peered":"2024-03-12T05:54:42.025218+0000","last_clean":"2024-03-12T05:54:42.025218+0000","last_became_active":"2024-03-12T05:54:42.021022+0000","last_became_peered":"2024-03-12T05:54:42.021022+0000","last_unstale":"2024-03-12T05:54:42.025218+0000","last_undegraded":"2024-03-12T05:54:42.025218+0000","last_fullsized":"2024-03-12T05:54:42.025218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:20:57.232145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013944+0000","last_change":"2024-03-12T05:53:38.207757+0000","last_active":"2024-03-12T05:54:42.013944+0000","last_peered":"2024-03-12T05:54:42.013944+0000","last_clean":"2024-03-12T05:54:42.013944+0000","last_became_active":"2024-03-12T05:53:38.207424+0000","last_became_peered":"2024-03-12T05:53:38.207424+0000","last_unstale":"2024-03-12T05:54:42.013944+0000","last_undegraded":"2024-03-12T05:54:42.013944+0000","last_fullsized":"2024-03-12T05:54:42.013944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:33:16.017907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016731+0000","last_change":"2024-03-12T05:53:37.208512+0000","last_active":"2024-03-12T05:54:42.016731+0000","last_peered":"2024-03-12T05:54:42.016731+0000","last_clean":"2024-03-12T05:54:42.016731+0000","last_became_active":"2024-03-12T05:53:37.207628+0000","last_became_peered":"2024-03-12T05:53:37.207628+0000","last_unstale":"2024-03-12T05:54:42.016731+0000","last_undegraded":"2024-03-12T05:54:42.016731+0000","last_fullsized":"2024-03-12T05:54:42.016731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:25:28.825779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.011698+0000","last_change":"2024-03-12T05:53:38.205654+0000","last_active":"2024-03-12T05:54:42.011698+0000","last_peered":"2024-03-12T05:54:42.011698+0000","last_clean":"2024-03-12T05:54:42.011698+0000","last_became_active":"2024-03-12T05:53:38.205325+0000","last_became_peered":"2024-03-12T05:53:38.205325+0000","last_unstale":"2024-03-12T05:54:42.011698+0000","last_undegraded":"2024-03-12T05:54:42.011698+0000","last_fullsized":"2024-03-12T05:54:42.011698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:33:35.668625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019632+0000","last_change":"2024-03-12T05:53:37.211430+0000","last_active":"2024-03-12T05:54:42.019632+0000","last_peered":"2024-03-12T05:54:42.019632+0000","last_clean":"2024-03-12T05:54:42.019632+0000","last_became_active":"2024-03-12T05:53:37.210501+0000","last_became_peered":"2024-03-12T05:53:37.210501+0000","last_unstale":"2024-03-12T05:54:42.019632+0000","last_undegraded":"2024-03-12T05:54:42.019632+0000","last_fullsized":"2024-03-12T05:54:42.019632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:48:33.555134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017930+0000","last_change":"2024-03-12T05:53:38.212837+0000","last_active":"2024-03-12T05:54:42.017930+0000","last_peered":"2024-03-12T05:54:42.017930+0000","last_clean":"2024-03-12T05:54:42.017930+0000","last_became_active":"2024-03-12T05:53:38.212513+0000","last_became_peered":"2024-03-12T05:53:38.212513+0000","last_unstale":"2024-03-12T05:54:42.017930+0000","last_undegraded":"2024-03-12T05:54:42.017930+0000","last_fullsized":"2024-03-12T05:54:42.017930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:47:33.879954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014092+0000","last_change":"2024-03-12T05:53:37.207465+0000","last_active":"2024-03-12T05:54:42.014092+0000","last_peered":"2024-03-12T05:54:42.014092+0000","last_clean":"2024-03-12T05:54:42.014092+0000","last_became_active":"2024-03-12T05:53:37.207107+0000","last_became_peered":"2024-03-12T05:53:37.207107+0000","last_unstale":"2024-03-12T05:54:42.014092+0000","last_undegraded":"2024-03-12T05:54:42.014092+0000","last_fullsized":"2024-03-12T05:54:42.014092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:34.018856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686518+0000","last_change":"2024-03-12T05:53:38.221470+0000","last_active":"2024-03-12T05:54:42.686518+0000","last_peered":"2024-03-12T05:54:42.686518+0000","last_clean":"2024-03-12T05:54:42.686518+0000","last_became_active":"2024-03-12T05:53:38.221061+0000","last_became_peered":"2024-03-12T05:53:38.221061+0000","last_unstale":"2024-03-12T05:54:42.686518+0000","last_undegraded":"2024-03-12T05:54:42.686518+0000","last_fullsized":"2024-03-12T05:54:42.686518+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:01:54.148877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019769+0000","last_change":"2024-03-12T05:53:37.203822+0000","last_active":"2024-03-12T05:54:42.019769+0000","last_peered":"2024-03-12T05:54:42.019769+0000","last_clean":"2024-03-12T05:54:42.019769+0000","last_became_active":"2024-03-12T05:53:37.203431+0000","last_became_peered":"2024-03-12T05:53:37.203431+0000","last_unstale":"2024-03-12T05:54:42.019769+0000","last_undegraded":"2024-03-12T05:54:42.019769+0000","last_fullsized":"2024-03-12T05:54:42.019769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:25:04.447261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686535+0000","last_change":"2024-03-12T05:53:38.222045+0000","last_active":"2024-03-12T05:54:42.686535+0000","last_peered":"2024-03-12T05:54:42.686535+0000","last_clean":"2024-03-12T05:54:42.686535+0000","last_became_active":"2024-03-12T05:53:38.220904+0000","last_became_peered":"2024-03-12T05:53:38.220904+0000","last_unstale":"2024-03-12T05:54:42.686535+0000","last_undegraded":"2024-03-12T05:54:42.686535+0000","last_fullsized":"2024-03-12T05:54:42.686535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:30:43.964599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019437+0000","last_change":"2024-03-12T05:53:37.206753+0000","last_active":"2024-03-12T05:54:42.019437+0000","last_peered":"2024-03-12T05:54:42.019437+0000","last_clean":"2024-03-12T05:54:42.019437+0000","last_became_active":"2024-03-12T05:53:37.206078+0000","last_became_peered":"2024-03-12T05:53:37.206078+0000","last_unstale":"2024-03-12T05:54:42.019437+0000","last_undegraded":"2024-03-12T05:54:42.019437+0000","last_fullsized":"2024-03-12T05:54:42.019437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:42:18.859528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686435+0000","last_change":"2024-03-12T05:53:38.215671+0000","last_active":"2024-03-12T05:54:42.686435+0000","last_peered":"2024-03-12T05:54:42.686435+0000","last_clean":"2024-03-12T05:54:42.686435+0000","last_became_active":"2024-03-12T05:53:38.215309+0000","last_became_peered":"2024-03-12T05:53:38.215309+0000","last_unstale":"2024-03-12T05:54:42.686435+0000","last_undegraded":"2024-03-12T05:54:42.686435+0000","last_fullsized":"2024-03-12T05:54:42.686435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:50:12.176935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017343+0000","last_change":"2024-03-12T05:53:37.205627+0000","last_active":"2024-03-12T05:54:42.017343+0000","last_peered":"2024-03-12T05:54:42.017343+0000","last_clean":"2024-03-12T05:54:42.017343+0000","last_became_active":"2024-03-12T05:53:37.205289+0000","last_became_peered":"2024-03-12T05:53:37.205289+0000","last_unstale":"2024-03-12T05:54:42.017343+0000","last_undegraded":"2024-03-12T05:54:42.017343+0000","last_fullsized":"2024-03-12T05:54:42.017343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:56.682746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686583+0000","last_change":"2024-03-12T05:53:38.214376+0000","last_active":"2024-03-12T05:54:42.686583+0000","last_peered":"2024-03-12T05:54:42.686583+0000","last_clean":"2024-03-12T05:54:42.686583+0000","last_became_active":"2024-03-12T05:53:38.214011+0000","last_became_peered":"2024-03-12T05:53:38.214011+0000","last_unstale":"2024-03-12T05:54:42.686583+0000","last_undegraded":"2024-03-12T05:54:42.686583+0000","last_fullsized":"2024-03-12T05:54:42.686583+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:32.605216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.011774+0000","last_change":"2024-03-12T05:53:38.205337+0000","last_active":"2024-03-12T05:54:42.011774+0000","last_peered":"2024-03-12T05:54:42.011774+0000","last_clean":"2024-03-12T05:54:42.011774+0000","last_became_active":"2024-03-12T05:53:38.205005+0000","last_became_peered":"2024-03-12T05:53:38.205005+0000","last_unstale":"2024-03-12T05:54:42.011774+0000","last_undegraded":"2024-03-12T05:54:42.011774+0000","last_fullsized":"2024-03-12T05:54:42.011774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:44:16.296370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018964+0000","last_change":"2024-03-12T05:53:38.214694+0000","last_active":"2024-03-12T05:54:42.018964+0000","last_peered":"2024-03-12T05:54:42.018964+0000","last_clean":"2024-03-12T05:54:42.018964+0000","last_became_active":"2024-03-12T05:53:38.214285+0000","last_became_peered":"2024-03-12T05:53:38.214285+0000","last_unstale":"2024-03-12T05:54:42.018964+0000","last_undegraded":"2024-03-12T05:54:42.018964+0000","last_fullsized":"2024-03-12T05:54:42.018964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:28:32.253332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019495+0000","last_change":"2024-03-12T05:53:38.213069+0000","last_active":"2024-03-12T05:54:42.019495+0000","last_peered":"2024-03-12T05:54:42.019495+0000","last_clean":"2024-03-12T05:54:42.019495+0000","last_became_active":"2024-03-12T05:53:38.212694+0000","last_became_peered":"2024-03-12T05:53:38.212694+0000","last_unstale":"2024-03-12T05:54:42.019495+0000","last_undegraded":"2024-03-12T05:54:42.019495+0000","last_fullsized":"2024-03-12T05:54:42.019495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:26:29.350292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014551+0000","last_change":"2024-03-12T05:53:38.212084+0000","last_active":"2024-03-12T05:54:42.014551+0000","last_peered":"2024-03-12T05:54:42.014551+0000","last_clean":"2024-03-12T05:54:42.014551+0000","last_became_active":"2024-03-12T05:53:38.211750+0000","last_became_peered":"2024-03-12T05:53:38.211750+0000","last_unstale":"2024-03-12T05:54:42.014551+0000","last_undegraded":"2024-03-12T05:54:42.014551+0000","last_fullsized":"2024-03-12T05:54:42.014551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:05:02.099121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017345+0000","last_change":"2024-03-12T05:53:38.212071+0000","last_active":"2024-03-12T05:54:42.017345+0000","last_peered":"2024-03-12T05:54:42.017345+0000","last_clean":"2024-03-12T05:54:42.017345+0000","last_became_active":"2024-03-12T05:53:38.211722+0000","last_became_peered":"2024-03-12T05:53:38.211722+0000","last_unstale":"2024-03-12T05:54:42.017345+0000","last_undegraded":"2024-03-12T05:54:42.017345+0000","last_fullsized":"2024-03-12T05:54:42.017345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:19:46.581837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.015717+0000","last_change":"2024-03-12T05:54:42.015717+0000","last_active":"2024-03-12T05:54:42.015717+0000","last_peered":"2024-03-12T05:54:42.015717+0000","last_clean":"2024-03-12T05:54:42.015717+0000","last_became_active":"2024-03-12T05:54:42.015266+0000","last_became_peered":"2024-03-12T05:54:42.015266+0000","last_unstale":"2024-03-12T05:54:42.015717+0000","last_undegraded":"2024-03-12T05:54:42.015717+0000","last_fullsized":"2024-03-12T05:54:42.015717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:33:48.250458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.020263+0000","last_change":"2024-03-12T05:53:38.208018+0000","last_active":"2024-03-12T05:54:42.020263+0000","last_peered":"2024-03-12T05:54:42.020263+0000","last_clean":"2024-03-12T05:54:42.020263+0000","last_became_active":"2024-03-12T05:53:38.207682+0000","last_became_peered":"2024-03-12T05:53:38.207682+0000","last_unstale":"2024-03-12T05:54:42.020263+0000","last_undegraded":"2024-03-12T05:54:42.020263+0000","last_fullsized":"2024-03-12T05:54:42.020263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:56:10.903917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018645+0000","last_change":"2024-03-12T05:53:38.217540+0000","last_active":"2024-03-12T05:54:42.018645+0000","last_peered":"2024-03-12T05:54:42.018645+0000","last_clean":"2024-03-12T05:54:42.018645+0000","last_became_active":"2024-03-12T05:53:38.217144+0000","last_became_peered":"2024-03-12T05:53:38.217144+0000","last_unstale":"2024-03-12T05:54:42.018645+0000","last_undegraded":"2024-03-12T05:54:42.018645+0000","last_fullsized":"2024-03-12T05:54:42.018645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:46:18.905909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017540+0000","last_change":"2024-03-12T05:53:38.215006+0000","last_active":"2024-03-12T05:54:42.017540+0000","last_peered":"2024-03-12T05:54:42.017540+0000","last_clean":"2024-03-12T05:54:42.017540+0000","last_became_active":"2024-03-12T05:53:38.213756+0000","last_became_peered":"2024-03-12T05:53:38.213756+0000","last_unstale":"2024-03-12T05:54:42.017540+0000","last_undegraded":"2024-03-12T05:54:42.017540+0000","last_fullsized":"2024-03-12T05:54:42.017540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:14:57.642381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016570+0000","last_change":"2024-03-12T05:53:38.216096+0000","last_active":"2024-03-12T05:54:42.016570+0000","last_peered":"2024-03-12T05:54:42.016570+0000","last_clean":"2024-03-12T05:54:42.016570+0000","last_became_active":"2024-03-12T05:53:38.215153+0000","last_became_peered":"2024-03-12T05:53:38.215153+0000","last_unstale":"2024-03-12T05:54:42.016570+0000","last_undegraded":"2024-03-12T05:54:42.016570+0000","last_fullsized":"2024-03-12T05:54:42.016570+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:41:03.893641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016566+0000","last_change":"2024-03-12T05:53:38.219165+0000","last_active":"2024-03-12T05:54:42.016566+0000","last_peered":"2024-03-12T05:54:42.016566+0000","last_clean":"2024-03-12T05:54:42.016566+0000","last_became_active":"2024-03-12T05:53:38.218579+0000","last_became_peered":"2024-03-12T05:53:38.218579+0000","last_unstale":"2024-03-12T05:54:42.016566+0000","last_undegraded":"2024-03-12T05:54:42.016566+0000","last_fullsized":"2024-03-12T05:54:42.016566+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:06:37.531293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017229+0000","last_change":"2024-03-12T05:53:38.212685+0000","last_active":"2024-03-12T05:54:42.017229+0000","last_peered":"2024-03-12T05:54:42.017229+0000","last_clean":"2024-03-12T05:54:42.017229+0000","last_became_active":"2024-03-12T05:53:38.211106+0000","last_became_peered":"2024-03-12T05:53:38.211106+0000","last_unstale":"2024-03-12T05:54:42.017229+0000","last_undegraded":"2024-03-12T05:54:42.017229+0000","last_fullsized":"2024-03-12T05:54:42.017229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:26:00.339270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.011757+0000","last_change":"2024-03-12T05:53:38.218402+0000","last_active":"2024-03-12T05:54:42.011757+0000","last_peered":"2024-03-12T05:54:42.011757+0000","last_clean":"2024-03-12T05:54:42.011757+0000","last_became_active":"2024-03-12T05:53:38.217699+0000","last_became_peered":"2024-03-12T05:53:38.217699+0000","last_unstale":"2024-03-12T05:54:42.011757+0000","last_undegraded":"2024-03-12T05:54:42.011757+0000","last_fullsized":"2024-03-12T05:54:42.011757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:23:03.221710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65047,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":436,"num_read_kb":285,"num_write":156,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":195141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351004,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":461631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612641,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32796,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712356,"statfs":{"total":95995035648,"available":95961452544,"internally_reserved":0,"allocated":1650688,"data_stored":1114136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776166,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":461962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037804,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":461743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332143,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32712,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712440,"statfs":{"total":95995035648,"available":95961538560,"internally_reserved":0,"allocated":1564672,"data_stored":1053695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495669,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":461234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1019904,"data_stored":522073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":462505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":523814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182468,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":461428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920831,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":962560,"data_stored":461671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757306,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28564,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716588,"statfs":{"total":95995035648,"available":95965786112,"internally_reserved":0,"allocated":1576960,"data_stored":1053379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T05:54:53.898 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T05:54:54.143 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:54.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:54 smithi076 ceph-mon[32095]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 12 B/s, 0 objects/s recovering 2024-03-12T05:54:54.386 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:54 smithi088 ceph-mon[36585]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 12 B/s, 0 objects/s recovering 2024-03-12T05:54:54.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:54 smithi082 ceph-mon[36382]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 12 B/s, 0 objects/s recovering 2024-03-12T05:54:55.204 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:55 smithi076 ceph-mon[32095]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:54:55.402 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:55 smithi088 ceph-mon[36585]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:54:55.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:55 smithi082 ceph-mon[36382]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:54:55.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.789+0000 7fec40fd1700 1 -- 172.21.15.76:0/2762789671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec3c106f90 msgr2=0x7fec3c10eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:55.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.789+0000 7fec40fd1700 1 --2- 172.21.15.76:0/2762789671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec3c106f90 0x7fec3c10eb60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fec3400b0a0 tx=0x7fec3400b3b0 comp rx=0 tx=0).stop 2024-03-12T05:54:55.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.789+0000 7fec40fd1700 1 -- 172.21.15.76:0/2762789671 shutdown_connections 2024-03-12T05:54:55.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.789+0000 7fec40fd1700 1 --2- 172.21.15.76:0/2762789671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec3c106f90 0x7fec3c10eb60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:55.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.789+0000 7fec40fd1700 1 --2- 172.21.15.76:0/2762789671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec3c101eb0 0x7fec3c102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:55.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.789+0000 7fec40fd1700 1 --2- 172.21.15.76:0/2762789671 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fec3c102860 0x7fec3c1068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:55.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.789+0000 7fec40fd1700 1 -- 172.21.15.76:0/2762789671 >> 172.21.15.76:0/2762789671 conn(0x7fec3c0fd720 msgr2=0x7fec3c0ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:55.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.789+0000 7fec40fd1700 1 -- 172.21.15.76:0/2762789671 shutdown_connections 2024-03-12T05:54:55.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.789+0000 7fec40fd1700 1 -- 172.21.15.76:0/2762789671 wait complete. 2024-03-12T05:54:55.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.790+0000 7fec40fd1700 1 Processor -- start 2024-03-12T05:54:55.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.790+0000 7fec40fd1700 1 -- start start 2024-03-12T05:54:55.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.790+0000 7fec40fd1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec3c101eb0 0x7fec3c073f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:55.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec40fd1700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec3c102860 0x7fec3c0744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:55.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec40fd1700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fec3c106f90 0x7fec3c0749f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:55.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec40fd1700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c1a6650 con 0x7fec3c102860 2024-03-12T05:54:55.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec40fd1700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c1a67c0 con 0x7fec3c106f90 2024-03-12T05:54:55.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec40fd1700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c1a6980 con 0x7fec3c101eb0 2024-03-12T05:54:55.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec39d9b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec3c102860 0x7fec3c0744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:55.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec39d9b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec3c102860 0x7fec3c0744b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58662/0 (socket says 172.21.15.76:58662) 2024-03-12T05:54:55.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec39d9b700 1 -- 172.21.15.76:0/2628149747 learned_addr learned my addr 172.21.15.76:0/2628149747 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:55.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec3a59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec3c101eb0 0x7fec3c073f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:55.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec3ad9d700 1 --2- 172.21.15.76:0/2628149747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fec3c106f90 0x7fec3c0749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:55.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec39d9b700 1 -- 172.21.15.76:0/2628149747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec3c101eb0 msgr2=0x7fec3c073f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:55.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec39d9b700 1 --2- 172.21.15.76:0/2628149747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec3c101eb0 0x7fec3c073f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:55.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec39d9b700 1 -- 172.21.15.76:0/2628149747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fec3c106f90 msgr2=0x7fec3c0749f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:55.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec39d9b700 1 --2- 172.21.15.76:0/2628149747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fec3c106f90 0x7fec3c0749f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:55.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec39d9b700 1 -- 172.21.15.76:0/2628149747 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec34009d00 con 0x7fec3c102860 2024-03-12T05:54:55.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec3ad9d700 1 --2- 172.21.15.76:0/2628149747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fec3c106f90 0x7fec3c0749f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:54:55.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec39d9b700 1 --2- 172.21.15.76:0/2628149747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec3c102860 0x7fec3c0744b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fec3000c9e0 tx=0x7fec3000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:55.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec2b7fe700 1 -- 172.21.15.76:0/2628149747 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30007aa0 con 0x7fec3c102860 2024-03-12T05:54:55.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec2b7fe700 1 -- 172.21.15.76:0/2628149747 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec30007c00 con 0x7fec3c102860 2024-03-12T05:54:55.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec40fd1700 1 -- 172.21.15.76:0/2628149747 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3c1a6c60 con 0x7fec3c102860 2024-03-12T05:54:55.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec2b7fe700 1 -- 172.21.15.76:0/2628149747 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30018660 con 0x7fec3c102860 2024-03-12T05:54:55.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.791+0000 7fec40fd1700 1 -- 172.21.15.76:0/2628149747 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3c1a71b0 con 0x7fec3c102860 2024-03-12T05:54:55.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.792+0000 7fec40fd1700 1 -- 172.21.15.76:0/2628149747 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3c068930 con 0x7fec3c102860 2024-03-12T05:54:55.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.793+0000 7fec2b7fe700 1 -- 172.21.15.76:0/2628149747 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fec3000f450 con 0x7fec3c102860 2024-03-12T05:54:55.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.794+0000 7fec2b7fe700 1 --2- 172.21.15.76:0/2628149747 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fec240a1bd0 0x7fec240a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:55.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.794+0000 7fec3a59c700 1 --2- 172.21.15.76:0/2628149747 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fec240a1bd0 0x7fec240a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:55.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.794+0000 7fec2b7fe700 1 -- 172.21.15.76:0/2628149747 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fec300cd420 con 0x7fec3c102860 2024-03-12T05:54:55.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.794+0000 7fec3a59c700 1 --2- 172.21.15.76:0/2628149747 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fec240a1bd0 0x7fec240a4090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fec2c005950 tx=0x7fec2c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.798+0000 7fec2b7fe700 1 -- 172.21.15.76:0/2628149747 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fec300964e0 con 0x7fec3c102860 2024-03-12T05:54:55.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.988+0000 7fec40fd1700 1 -- 172.21.15.76:0/2628149747 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fec3c111e20 con 0x7fec240a1bd0 2024-03-12T05:54:55.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.996+0000 7fec2b7fe700 1 -- 172.21.15.76:0/2628149747 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258721 (secure 0 0 0) 0x7fec3c111e20 con 0x7fec240a1bd0 2024-03-12T05:54:56.002 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:54:56.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.999+0000 7fec40fd1700 1 -- 172.21.15.76:0/2628149747 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fec240a1bd0 msgr2=0x7fec240a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:56.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.999+0000 7fec40fd1700 1 --2- 172.21.15.76:0/2628149747 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fec240a1bd0 0x7fec240a4090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fec2c005950 tx=0x7fec2c00b410 comp rx=0 tx=0).stop 2024-03-12T05:54:56.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.999+0000 7fec40fd1700 1 -- 172.21.15.76:0/2628149747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec3c102860 msgr2=0x7fec3c0744b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:56.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.999+0000 7fec40fd1700 1 --2- 172.21.15.76:0/2628149747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec3c102860 0x7fec3c0744b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fec3000c9e0 tx=0x7fec3000ccf0 comp rx=0 tx=0).stop 2024-03-12T05:54:56.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.999+0000 7fec40fd1700 1 -- 172.21.15.76:0/2628149747 shutdown_connections 2024-03-12T05:54:56.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.999+0000 7fec40fd1700 1 --2- 172.21.15.76:0/2628149747 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fec240a1bd0 0x7fec240a4090 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:56.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.999+0000 7fec40fd1700 1 --2- 172.21.15.76:0/2628149747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fec3c106f90 0x7fec3c0749f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:56.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.999+0000 7fec40fd1700 1 --2- 172.21.15.76:0/2628149747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec3c101eb0 0x7fec3c073f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:56.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.999+0000 7fec40fd1700 1 --2- 172.21.15.76:0/2628149747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec3c102860 0x7fec3c0744b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:56.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.999+0000 7fec40fd1700 1 -- 172.21.15.76:0/2628149747 >> 172.21.15.76:0/2628149747 conn(0x7fec3c0fd720 msgr2=0x7fec3c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:56.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.999+0000 7fec40fd1700 1 -- 172.21.15.76:0/2628149747 shutdown_connections 2024-03-12T05:54:56.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:55.999+0000 7fec40fd1700 1 -- 172.21.15.76:0/2628149747 wait complete. 2024-03-12T05:54:56.005 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-03-12T05:54:56.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:56 smithi082 ceph-mon[36382]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 11 B/s, 0 objects/s recovering 2024-03-12T05:54:56.479 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:56 smithi076 ceph-mon[32095]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 11 B/s, 0 objects/s recovering 2024-03-12T05:54:56.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:56 smithi088 ceph-mon[36585]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 11 B/s, 0 objects/s recovering 2024-03-12T05:54:56.802 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-03-12T05:54:54.882078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655649,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":592,"num_read_kb":419,"num_write":339,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180875,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375032,"kb_used_data":13308,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566792,"statfs":{"total":1151940427776,"available":1151556395008,"internally_reserved":0,"allocated":13627392,"data_stored":7499271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":79,"num_read_kb":44,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002849"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.012006+0000","last_change":"2024-03-12T05:53:38.207712+0000","last_active":"2024-03-12T05:54:42.012006+0000","last_peered":"2024-03-12T05:54:42.012006+0000","last_clean":"2024-03-12T05:54:42.012006+0000","last_became_active":"2024-03-12T05:53:38.207362+0000","last_became_peered":"2024-03-12T05:53:38.207362+0000","last_unstale":"2024-03-12T05:54:42.012006+0000","last_undegraded":"2024-03-12T05:54:42.012006+0000","last_fullsized":"2024-03-12T05:54:42.012006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:19.518959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014177+0000","last_change":"2024-03-12T05:53:38.206877+0000","last_active":"2024-03-12T05:54:42.014177+0000","last_peered":"2024-03-12T05:54:42.014177+0000","last_clean":"2024-03-12T05:54:42.014177+0000","last_became_active":"2024-03-12T05:53:38.206468+0000","last_became_peered":"2024-03-12T05:53:38.206468+0000","last_unstale":"2024-03-12T05:54:42.014177+0000","last_undegraded":"2024-03-12T05:54:42.014177+0000","last_fullsized":"2024-03-12T05:54:42.014177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:02:45.138265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014360+0000","last_change":"2024-03-12T05:53:38.214979+0000","last_active":"2024-03-12T05:54:42.014360+0000","last_peered":"2024-03-12T05:54:42.014360+0000","last_clean":"2024-03-12T05:54:42.014360+0000","last_became_active":"2024-03-12T05:53:38.214580+0000","last_became_peered":"2024-03-12T05:53:38.214580+0000","last_unstale":"2024-03-12T05:54:42.014360+0000","last_undegraded":"2024-03-12T05:54:42.014360+0000","last_fullsized":"2024-03-12T05:54:42.014360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:39:58.989757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.478043+0000","last_change":"2024-03-12T05:53:38.218645+0000","last_active":"2024-03-12T05:54:42.478043+0000","last_peered":"2024-03-12T05:54:42.478043+0000","last_clean":"2024-03-12T05:54:42.478043+0000","last_became_active":"2024-03-12T05:53:38.218254+0000","last_became_peered":"2024-03-12T05:53:38.218254+0000","last_unstale":"2024-03-12T05:54:42.478043+0000","last_undegraded":"2024-03-12T05:54:42.478043+0000","last_fullsized":"2024-03-12T05:54:42.478043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:46:09.544989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014191+0000","last_change":"2024-03-12T05:53:38.214588+0000","last_active":"2024-03-12T05:54:42.014191+0000","last_peered":"2024-03-12T05:54:42.014191+0000","last_clean":"2024-03-12T05:54:42.014191+0000","last_became_active":"2024-03-12T05:53:38.214192+0000","last_became_peered":"2024-03-12T05:53:38.214192+0000","last_unstale":"2024-03-12T05:54:42.014191+0000","last_undegraded":"2024-03-12T05:54:42.014191+0000","last_fullsized":"2024-03-12T05:54:42.014191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:54:04.880901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014106+0000","last_change":"2024-03-12T05:53:38.214315+0000","last_active":"2024-03-12T05:54:42.014106+0000","last_peered":"2024-03-12T05:54:42.014106+0000","last_clean":"2024-03-12T05:54:42.014106+0000","last_became_active":"2024-03-12T05:53:38.213881+0000","last_became_peered":"2024-03-12T05:53:38.213881+0000","last_unstale":"2024-03-12T05:54:42.014106+0000","last_undegraded":"2024-03-12T05:54:42.014106+0000","last_fullsized":"2024-03-12T05:54:42.014106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:11:32.834418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686368+0000","last_change":"2024-03-12T05:53:38.214915+0000","last_active":"2024-03-12T05:54:42.686368+0000","last_peered":"2024-03-12T05:54:42.686368+0000","last_clean":"2024-03-12T05:54:42.686368+0000","last_became_active":"2024-03-12T05:53:38.214592+0000","last_became_peered":"2024-03-12T05:53:38.214592+0000","last_unstale":"2024-03-12T05:54:42.686368+0000","last_undegraded":"2024-03-12T05:54:42.686368+0000","last_fullsized":"2024-03-12T05:54:42.686368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:11.548585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016632+0000","last_change":"2024-03-12T05:53:38.219201+0000","last_active":"2024-03-12T05:54:42.016632+0000","last_peered":"2024-03-12T05:54:42.016632+0000","last_clean":"2024-03-12T05:54:42.016632+0000","last_became_active":"2024-03-12T05:53:38.218833+0000","last_became_peered":"2024-03-12T05:53:38.218833+0000","last_unstale":"2024-03-12T05:54:42.016632+0000","last_undegraded":"2024-03-12T05:54:42.016632+0000","last_fullsized":"2024-03-12T05:54:42.016632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:13:36.177325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016442+0000","last_change":"2024-03-12T05:53:38.205126+0000","last_active":"2024-03-12T05:54:42.016442+0000","last_peered":"2024-03-12T05:54:42.016442+0000","last_clean":"2024-03-12T05:54:42.016442+0000","last_became_active":"2024-03-12T05:53:38.204478+0000","last_became_peered":"2024-03-12T05:53:38.204478+0000","last_unstale":"2024-03-12T05:54:42.016442+0000","last_undegraded":"2024-03-12T05:54:42.016442+0000","last_fullsized":"2024-03-12T05:54:42.016442+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:25:56.475134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014402+0000","last_change":"2024-03-12T05:53:38.215186+0000","last_active":"2024-03-12T05:54:42.014402+0000","last_peered":"2024-03-12T05:54:42.014402+0000","last_clean":"2024-03-12T05:54:42.014402+0000","last_became_active":"2024-03-12T05:53:38.214832+0000","last_became_peered":"2024-03-12T05:53:38.214832+0000","last_unstale":"2024-03-12T05:54:42.014402+0000","last_undegraded":"2024-03-12T05:54:42.014402+0000","last_fullsized":"2024-03-12T05:54:42.014402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:56:26.621990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686736+0000","last_change":"2024-03-12T05:53:38.222347+0000","last_active":"2024-03-12T05:54:42.686736+0000","last_peered":"2024-03-12T05:54:42.686736+0000","last_clean":"2024-03-12T05:54:42.686736+0000","last_became_active":"2024-03-12T05:53:38.221637+0000","last_became_peered":"2024-03-12T05:53:38.221637+0000","last_unstale":"2024-03-12T05:54:42.686736+0000","last_undegraded":"2024-03-12T05:54:42.686736+0000","last_fullsized":"2024-03-12T05:54:42.686736+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:16:26.652673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.478305+0000","last_change":"2024-03-12T05:53:38.218687+0000","last_active":"2024-03-12T05:54:42.478305+0000","last_peered":"2024-03-12T05:54:42.478305+0000","last_clean":"2024-03-12T05:54:42.478305+0000","last_became_active":"2024-03-12T05:53:38.218393+0000","last_became_peered":"2024-03-12T05:53:38.218393+0000","last_unstale":"2024-03-12T05:54:42.478305+0000","last_undegraded":"2024-03-12T05:54:42.478305+0000","last_fullsized":"2024-03-12T05:54:42.478305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:41:39.918992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017284+0000","last_change":"2024-03-12T05:53:38.207432+0000","last_active":"2024-03-12T05:54:42.017284+0000","last_peered":"2024-03-12T05:54:42.017284+0000","last_clean":"2024-03-12T05:54:42.017284+0000","last_became_active":"2024-03-12T05:53:38.207099+0000","last_became_peered":"2024-03-12T05:53:38.207099+0000","last_unstale":"2024-03-12T05:54:42.017284+0000","last_undegraded":"2024-03-12T05:54:42.017284+0000","last_fullsized":"2024-03-12T05:54:42.017284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:20:08.504723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014250+0000","last_change":"2024-03-12T05:53:38.207086+0000","last_active":"2024-03-12T05:54:42.014250+0000","last_peered":"2024-03-12T05:54:42.014250+0000","last_clean":"2024-03-12T05:54:42.014250+0000","last_became_active":"2024-03-12T05:53:38.206714+0000","last_became_peered":"2024-03-12T05:53:38.206714+0000","last_unstale":"2024-03-12T05:54:42.014250+0000","last_undegraded":"2024-03-12T05:54:42.014250+0000","last_fullsized":"2024-03-12T05:54:42.014250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:07:13.012272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014011+0000","last_change":"2024-03-12T05:53:38.217179+0000","last_active":"2024-03-12T05:54:42.014011+0000","last_peered":"2024-03-12T05:54:42.014011+0000","last_clean":"2024-03-12T05:54:42.014011+0000","last_became_active":"2024-03-12T05:53:38.216820+0000","last_became_peered":"2024-03-12T05:53:38.216820+0000","last_unstale":"2024-03-12T05:54:42.014011+0000","last_undegraded":"2024-03-12T05:54:42.014011+0000","last_fullsized":"2024-03-12T05:54:42.014011+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:01:08.842611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.015965+0000","last_change":"2024-03-12T05:53:38.220224+0000","last_active":"2024-03-12T05:54:42.015965+0000","last_peered":"2024-03-12T05:54:42.015965+0000","last_clean":"2024-03-12T05:54:42.015965+0000","last_became_active":"2024-03-12T05:53:38.219878+0000","last_became_peered":"2024-03-12T05:53:38.219878+0000","last_unstale":"2024-03-12T05:54:42.015965+0000","last_undegraded":"2024-03-12T05:54:42.015965+0000","last_fullsized":"2024-03-12T05:54:42.015965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:53:16.545807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.012133+0000","last_change":"2024-03-12T05:53:38.208524+0000","last_active":"2024-03-12T05:54:42.012133+0000","last_peered":"2024-03-12T05:54:42.012133+0000","last_clean":"2024-03-12T05:54:42.012133+0000","last_became_active":"2024-03-12T05:53:38.208209+0000","last_became_peered":"2024-03-12T05:53:38.208209+0000","last_unstale":"2024-03-12T05:54:42.012133+0000","last_undegraded":"2024-03-12T05:54:42.012133+0000","last_fullsized":"2024-03-12T05:54:42.012133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:28:56.334066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013915+0000","last_change":"2024-03-12T05:53:38.208035+0000","last_active":"2024-03-12T05:54:42.013915+0000","last_peered":"2024-03-12T05:54:42.013915+0000","last_clean":"2024-03-12T05:54:42.013915+0000","last_became_active":"2024-03-12T05:53:38.207679+0000","last_became_peered":"2024-03-12T05:53:38.207679+0000","last_unstale":"2024-03-12T05:54:42.013915+0000","last_undegraded":"2024-03-12T05:54:42.013915+0000","last_fullsized":"2024-03-12T05:54:42.013915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:40:20.620552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019897+0000","last_change":"2024-03-12T05:53:38.207077+0000","last_active":"2024-03-12T05:54:42.019897+0000","last_peered":"2024-03-12T05:54:42.019897+0000","last_clean":"2024-03-12T05:54:42.019897+0000","last_became_active":"2024-03-12T05:53:38.206720+0000","last_became_peered":"2024-03-12T05:53:38.206720+0000","last_unstale":"2024-03-12T05:54:42.019897+0000","last_undegraded":"2024-03-12T05:54:42.019897+0000","last_fullsized":"2024-03-12T05:54:42.019897+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:45:54.628394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013942+0000","last_change":"2024-03-12T05:53:37.201637+0000","last_active":"2024-03-12T05:54:42.013942+0000","last_peered":"2024-03-12T05:54:42.013942+0000","last_clean":"2024-03-12T05:54:42.013942+0000","last_became_active":"2024-03-12T05:53:37.201289+0000","last_became_peered":"2024-03-12T05:53:37.201289+0000","last_unstale":"2024-03-12T05:54:42.013942+0000","last_undegraded":"2024-03-12T05:54:42.013942+0000","last_fullsized":"2024-03-12T05:54:42.013942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:59:43.829570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018092+0000","last_change":"2024-03-12T05:53:38.218052+0000","last_active":"2024-03-12T05:54:42.018092+0000","last_peered":"2024-03-12T05:54:42.018092+0000","last_clean":"2024-03-12T05:54:42.018092+0000","last_became_active":"2024-03-12T05:53:38.217738+0000","last_became_peered":"2024-03-12T05:53:38.217738+0000","last_unstale":"2024-03-12T05:54:42.018092+0000","last_undegraded":"2024-03-12T05:54:42.018092+0000","last_fullsized":"2024-03-12T05:54:42.018092+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:59:40.599573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018614+0000","last_change":"2024-03-12T05:53:37.210321+0000","last_active":"2024-03-12T05:54:42.018614+0000","last_peered":"2024-03-12T05:54:42.018614+0000","last_clean":"2024-03-12T05:54:42.018614+0000","last_became_active":"2024-03-12T05:53:37.210018+0000","last_became_peered":"2024-03-12T05:53:37.210018+0000","last_unstale":"2024-03-12T05:54:42.018614+0000","last_undegraded":"2024-03-12T05:54:42.018614+0000","last_fullsized":"2024-03-12T05:54:42.018614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:13:45.433332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013944+0000","last_change":"2024-03-12T05:53:38.207801+0000","last_active":"2024-03-12T05:54:42.013944+0000","last_peered":"2024-03-12T05:54:42.013944+0000","last_clean":"2024-03-12T05:54:42.013944+0000","last_became_active":"2024-03-12T05:53:38.207467+0000","last_became_peered":"2024-03-12T05:53:38.207467+0000","last_unstale":"2024-03-12T05:54:42.013944+0000","last_undegraded":"2024-03-12T05:54:42.013944+0000","last_fullsized":"2024-03-12T05:54:42.013944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:56:14.469662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014569+0000","last_change":"2024-03-12T05:53:37.208288+0000","last_active":"2024-03-12T05:54:42.014569+0000","last_peered":"2024-03-12T05:54:42.014569+0000","last_clean":"2024-03-12T05:54:42.014569+0000","last_became_active":"2024-03-12T05:53:37.207929+0000","last_became_peered":"2024-03-12T05:53:37.207929+0000","last_unstale":"2024-03-12T05:54:42.014569+0000","last_undegraded":"2024-03-12T05:54:42.014569+0000","last_fullsized":"2024-03-12T05:54:42.014569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:10:52.719116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016593+0000","last_change":"2024-03-12T05:53:38.207270+0000","last_active":"2024-03-12T05:54:42.016593+0000","last_peered":"2024-03-12T05:54:42.016593+0000","last_clean":"2024-03-12T05:54:42.016593+0000","last_became_active":"2024-03-12T05:53:38.205782+0000","last_became_peered":"2024-03-12T05:53:38.205782+0000","last_unstale":"2024-03-12T05:54:42.016593+0000","last_undegraded":"2024-03-12T05:54:42.016593+0000","last_fullsized":"2024-03-12T05:54:42.016593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:02:24.875444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014278+0000","last_change":"2024-03-12T05:53:37.203899+0000","last_active":"2024-03-12T05:54:42.014278+0000","last_peered":"2024-03-12T05:54:42.014278+0000","last_clean":"2024-03-12T05:54:42.014278+0000","last_became_active":"2024-03-12T05:53:37.203450+0000","last_became_peered":"2024-03-12T05:53:37.203450+0000","last_unstale":"2024-03-12T05:54:42.014278+0000","last_undegraded":"2024-03-12T05:54:42.014278+0000","last_fullsized":"2024-03-12T05:54:42.014278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:25.773826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.480629+0000","last_change":"2024-03-12T05:54:42.480247+0000","last_active":"2024-03-12T05:54:42.480629+0000","last_peered":"2024-03-12T05:54:42.480629+0000","last_clean":"2024-03-12T05:54:42.480629+0000","last_became_active":"2024-03-12T05:54:42.479952+0000","last_became_peered":"2024-03-12T05:54:42.479952+0000","last_unstale":"2024-03-12T05:54:42.480629+0000","last_undegraded":"2024-03-12T05:54:42.480629+0000","last_fullsized":"2024-03-12T05:54:42.480629+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:45:20.439878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":97,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:48.461130+0000","last_change":"2024-03-12T05:53:37.205401+0000","last_active":"2024-03-12T05:54:48.461130+0000","last_peered":"2024-03-12T05:54:48.461130+0000","last_clean":"2024-03-12T05:54:48.461130+0000","last_became_active":"2024-03-12T05:53:37.204993+0000","last_became_peered":"2024-03-12T05:53:37.204993+0000","last_unstale":"2024-03-12T05:54:48.461130+0000","last_undegraded":"2024-03-12T05:54:48.461130+0000","last_fullsized":"2024-03-12T05:54:48.461130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:56:06.177387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014148+0000","last_change":"2024-03-12T05:53:38.210562+0000","last_active":"2024-03-12T05:54:42.014148+0000","last_peered":"2024-03-12T05:54:42.014148+0000","last_clean":"2024-03-12T05:54:42.014148+0000","last_became_active":"2024-03-12T05:53:38.210235+0000","last_became_peered":"2024-03-12T05:53:38.210235+0000","last_unstale":"2024-03-12T05:54:42.014148+0000","last_undegraded":"2024-03-12T05:54:42.014148+0000","last_fullsized":"2024-03-12T05:54:42.014148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:35.445812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014118+0000","last_change":"2024-03-12T05:53:37.204986+0000","last_active":"2024-03-12T05:54:42.014118+0000","last_peered":"2024-03-12T05:54:42.014118+0000","last_clean":"2024-03-12T05:54:42.014118+0000","last_became_active":"2024-03-12T05:53:37.204580+0000","last_became_peered":"2024-03-12T05:53:37.204580+0000","last_unstale":"2024-03-12T05:54:42.014118+0000","last_undegraded":"2024-03-12T05:54:42.014118+0000","last_fullsized":"2024-03-12T05:54:42.014118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:37:34.584395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017753+0000","last_change":"2024-03-12T05:53:38.214948+0000","last_active":"2024-03-12T05:54:42.017753+0000","last_peered":"2024-03-12T05:54:42.017753+0000","last_clean":"2024-03-12T05:54:42.017753+0000","last_became_active":"2024-03-12T05:53:38.213748+0000","last_became_peered":"2024-03-12T05:53:38.213748+0000","last_unstale":"2024-03-12T05:54:42.017753+0000","last_undegraded":"2024-03-12T05:54:42.017753+0000","last_fullsized":"2024-03-12T05:54:42.017753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:12:45.994129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017009+0000","last_change":"2024-03-12T05:53:37.198167+0000","last_active":"2024-03-12T05:54:42.017009+0000","last_peered":"2024-03-12T05:54:42.017009+0000","last_clean":"2024-03-12T05:54:42.017009+0000","last_became_active":"2024-03-12T05:53:37.197808+0000","last_became_peered":"2024-03-12T05:53:37.197808+0000","last_unstale":"2024-03-12T05:54:42.017009+0000","last_undegraded":"2024-03-12T05:54:42.017009+0000","last_fullsized":"2024-03-12T05:54:42.017009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:40:19.498546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016514+0000","last_change":"2024-03-12T05:53:38.205464+0000","last_active":"2024-03-12T05:54:42.016514+0000","last_peered":"2024-03-12T05:54:42.016514+0000","last_clean":"2024-03-12T05:54:42.016514+0000","last_became_active":"2024-03-12T05:53:38.205129+0000","last_became_peered":"2024-03-12T05:53:38.205129+0000","last_unstale":"2024-03-12T05:54:42.016514+0000","last_undegraded":"2024-03-12T05:54:42.016514+0000","last_fullsized":"2024-03-12T05:54:42.016514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:38:55.460862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017494+0000","last_change":"2024-03-12T05:53:37.201572+0000","last_active":"2024-03-12T05:54:42.017494+0000","last_peered":"2024-03-12T05:54:42.017494+0000","last_clean":"2024-03-12T05:54:42.017494+0000","last_became_active":"2024-03-12T05:53:37.201230+0000","last_became_peered":"2024-03-12T05:53:37.201230+0000","last_unstale":"2024-03-12T05:54:42.017494+0000","last_undegraded":"2024-03-12T05:54:42.017494+0000","last_fullsized":"2024-03-12T05:54:42.017494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:16:13.715459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013535+0000","last_change":"2024-03-12T05:53:38.210696+0000","last_active":"2024-03-12T05:54:42.013535+0000","last_peered":"2024-03-12T05:54:42.013535+0000","last_clean":"2024-03-12T05:54:42.013535+0000","last_became_active":"2024-03-12T05:53:38.210325+0000","last_became_peered":"2024-03-12T05:53:38.210325+0000","last_unstale":"2024-03-12T05:54:42.013535+0000","last_undegraded":"2024-03-12T05:54:42.013535+0000","last_fullsized":"2024-03-12T05:54:42.013535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:25:36.397314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":95,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:52.466273+0000","last_change":"2024-03-12T05:53:37.201712+0000","last_active":"2024-03-12T05:54:52.466273+0000","last_peered":"2024-03-12T05:54:52.466273+0000","last_clean":"2024-03-12T05:54:52.466273+0000","last_became_active":"2024-03-12T05:53:37.200794+0000","last_became_peered":"2024-03-12T05:53:37.200794+0000","last_unstale":"2024-03-12T05:54:52.466273+0000","last_undegraded":"2024-03-12T05:54:52.466273+0000","last_fullsized":"2024-03-12T05:54:52.466273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:43:29.673717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016427+0000","last_change":"2024-03-12T05:53:38.206647+0000","last_active":"2024-03-12T05:54:42.016427+0000","last_peered":"2024-03-12T05:54:42.016427+0000","last_clean":"2024-03-12T05:54:42.016427+0000","last_became_active":"2024-03-12T05:53:38.206254+0000","last_became_peered":"2024-03-12T05:53:38.206254+0000","last_unstale":"2024-03-12T05:54:42.016427+0000","last_undegraded":"2024-03-12T05:54:42.016427+0000","last_fullsized":"2024-03-12T05:54:42.016427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:47:20.455360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687585+0000","last_change":"2024-03-12T05:53:37.208692+0000","last_active":"2024-03-12T05:54:42.687585+0000","last_peered":"2024-03-12T05:54:42.687585+0000","last_clean":"2024-03-12T05:54:42.687585+0000","last_became_active":"2024-03-12T05:53:37.208270+0000","last_became_peered":"2024-03-12T05:53:37.208270+0000","last_unstale":"2024-03-12T05:54:42.687585+0000","last_undegraded":"2024-03-12T05:54:42.687585+0000","last_fullsized":"2024-03-12T05:54:42.687585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:51:38.217999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013830+0000","last_change":"2024-03-12T05:53:38.218202+0000","last_active":"2024-03-12T05:54:42.013830+0000","last_peered":"2024-03-12T05:54:42.013830+0000","last_clean":"2024-03-12T05:54:42.013830+0000","last_became_active":"2024-03-12T05:53:38.217798+0000","last_became_peered":"2024-03-12T05:53:38.217798+0000","last_unstale":"2024-03-12T05:54:42.013830+0000","last_undegraded":"2024-03-12T05:54:42.013830+0000","last_fullsized":"2024-03-12T05:54:42.013830+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:09:29.422125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019722+0000","last_change":"2024-03-12T05:53:37.203252+0000","last_active":"2024-03-12T05:54:42.019722+0000","last_peered":"2024-03-12T05:54:42.019722+0000","last_clean":"2024-03-12T05:54:42.019722+0000","last_became_active":"2024-03-12T05:53:37.202916+0000","last_became_peered":"2024-03-12T05:53:37.202916+0000","last_unstale":"2024-03-12T05:54:42.019722+0000","last_undegraded":"2024-03-12T05:54:42.019722+0000","last_fullsized":"2024-03-12T05:54:42.019722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:09:34.629085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.021528+0000","last_change":"2024-03-12T05:54:42.021528+0000","last_active":"2024-03-12T05:54:42.021528+0000","last_peered":"2024-03-12T05:54:42.021528+0000","last_clean":"2024-03-12T05:54:42.021528+0000","last_became_active":"2024-03-12T05:54:42.021015+0000","last_became_peered":"2024-03-12T05:54:42.021015+0000","last_unstale":"2024-03-12T05:54:42.021528+0000","last_undegraded":"2024-03-12T05:54:42.021528+0000","last_fullsized":"2024-03-12T05:54:42.021528+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:16.998017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'28","reported_seq":201,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:53.467639+0000","last_change":"2024-03-12T05:53:37.206271+0000","last_active":"2024-03-12T05:54:53.467639+0000","last_peered":"2024-03-12T05:54:53.467639+0000","last_clean":"2024-03-12T05:54:53.467639+0000","last_became_active":"2024-03-12T05:53:37.205920+0000","last_became_peered":"2024-03-12T05:53:37.205920+0000","last_unstale":"2024-03-12T05:54:53.467639+0000","last_undegraded":"2024-03-12T05:54:53.467639+0000","last_fullsized":"2024-03-12T05:54:53.467639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:30.140159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":154,"num_read_kb":136,"num_write":29,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014385+0000","last_change":"2024-03-12T05:53:38.211397+0000","last_active":"2024-03-12T05:54:42.014385+0000","last_peered":"2024-03-12T05:54:42.014385+0000","last_clean":"2024-03-12T05:54:42.014385+0000","last_became_active":"2024-03-12T05:53:38.211045+0000","last_became_peered":"2024-03-12T05:53:38.211045+0000","last_unstale":"2024-03-12T05:54:42.014385+0000","last_undegraded":"2024-03-12T05:54:42.014385+0000","last_fullsized":"2024-03-12T05:54:42.014385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:21:42.510436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019264+0000","last_change":"2024-03-12T05:53:37.210871+0000","last_active":"2024-03-12T05:54:42.019264+0000","last_peered":"2024-03-12T05:54:42.019264+0000","last_clean":"2024-03-12T05:54:42.019264+0000","last_became_active":"2024-03-12T05:53:37.210526+0000","last_became_peered":"2024-03-12T05:53:37.210526+0000","last_unstale":"2024-03-12T05:54:42.019264+0000","last_undegraded":"2024-03-12T05:54:42.019264+0000","last_fullsized":"2024-03-12T05:54:42.019264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:40:46.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017098+0000","last_change":"2024-03-12T05:53:38.213751+0000","last_active":"2024-03-12T05:54:42.017098+0000","last_peered":"2024-03-12T05:54:42.017098+0000","last_clean":"2024-03-12T05:54:42.017098+0000","last_became_active":"2024-03-12T05:53:38.211558+0000","last_became_peered":"2024-03-12T05:53:38.211558+0000","last_unstale":"2024-03-12T05:54:42.017098+0000","last_undegraded":"2024-03-12T05:54:42.017098+0000","last_fullsized":"2024-03-12T05:54:42.017098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:40:44.194548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018330+0000","last_change":"2024-03-12T05:53:37.207368+0000","last_active":"2024-03-12T05:54:42.018330+0000","last_peered":"2024-03-12T05:54:42.018330+0000","last_clean":"2024-03-12T05:54:42.018330+0000","last_became_active":"2024-03-12T05:53:37.207066+0000","last_became_peered":"2024-03-12T05:53:37.207066+0000","last_unstale":"2024-03-12T05:54:42.018330+0000","last_undegraded":"2024-03-12T05:54:42.018330+0000","last_fullsized":"2024-03-12T05:54:42.018330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:28:58.155360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686427+0000","last_change":"2024-03-12T05:53:38.217079+0000","last_active":"2024-03-12T05:54:42.686427+0000","last_peered":"2024-03-12T05:54:42.686427+0000","last_clean":"2024-03-12T05:54:42.686427+0000","last_became_active":"2024-03-12T05:53:38.216757+0000","last_became_peered":"2024-03-12T05:53:38.216757+0000","last_unstale":"2024-03-12T05:54:42.686427+0000","last_undegraded":"2024-03-12T05:54:42.686427+0000","last_fullsized":"2024-03-12T05:54:42.686427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:48:13.125867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":93,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:49.488902+0000","last_change":"2024-03-12T05:53:37.202390+0000","last_active":"2024-03-12T05:54:49.488902+0000","last_peered":"2024-03-12T05:54:49.488902+0000","last_clean":"2024-03-12T05:54:49.488902+0000","last_became_active":"2024-03-12T05:53:37.202020+0000","last_became_peered":"2024-03-12T05:53:37.202020+0000","last_unstale":"2024-03-12T05:54:49.488902+0000","last_undegraded":"2024-03-12T05:54:49.488902+0000","last_fullsized":"2024-03-12T05:54:49.488902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:06:35.293227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686330+0000","last_change":"2024-03-12T05:53:38.218464+0000","last_active":"2024-03-12T05:54:42.686330+0000","last_peered":"2024-03-12T05:54:42.686330+0000","last_clean":"2024-03-12T05:54:42.686330+0000","last_became_active":"2024-03-12T05:53:38.218136+0000","last_became_peered":"2024-03-12T05:53:38.218136+0000","last_unstale":"2024-03-12T05:54:42.686330+0000","last_undegraded":"2024-03-12T05:54:42.686330+0000","last_fullsized":"2024-03-12T05:54:42.686330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:31:05.409148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013820+0000","last_change":"2024-03-12T05:53:37.205161+0000","last_active":"2024-03-12T05:54:42.013820+0000","last_peered":"2024-03-12T05:54:42.013820+0000","last_clean":"2024-03-12T05:54:42.013820+0000","last_became_active":"2024-03-12T05:53:37.204799+0000","last_became_peered":"2024-03-12T05:53:37.204799+0000","last_unstale":"2024-03-12T05:54:42.013820+0000","last_undegraded":"2024-03-12T05:54:42.013820+0000","last_fullsized":"2024-03-12T05:54:42.013820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:29:32.065635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017614+0000","last_change":"2024-03-12T05:53:38.216321+0000","last_active":"2024-03-12T05:54:42.017614+0000","last_peered":"2024-03-12T05:54:42.017614+0000","last_clean":"2024-03-12T05:54:42.017614+0000","last_became_active":"2024-03-12T05:53:38.215893+0000","last_became_peered":"2024-03-12T05:53:38.215893+0000","last_unstale":"2024-03-12T05:54:42.017614+0000","last_undegraded":"2024-03-12T05:54:42.017614+0000","last_fullsized":"2024-03-12T05:54:42.017614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:32:27.603930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014660+0000","last_change":"2024-03-12T05:53:37.203638+0000","last_active":"2024-03-12T05:54:42.014660+0000","last_peered":"2024-03-12T05:54:42.014660+0000","last_clean":"2024-03-12T05:54:42.014660+0000","last_became_active":"2024-03-12T05:53:37.203217+0000","last_became_peered":"2024-03-12T05:53:37.203217+0000","last_unstale":"2024-03-12T05:54:42.014660+0000","last_undegraded":"2024-03-12T05:54:42.014660+0000","last_fullsized":"2024-03-12T05:54:42.014660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:10.521197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.011922+0000","last_change":"2024-03-12T05:53:38.210491+0000","last_active":"2024-03-12T05:54:42.011922+0000","last_peered":"2024-03-12T05:54:42.011922+0000","last_clean":"2024-03-12T05:54:42.011922+0000","last_became_active":"2024-03-12T05:53:38.210169+0000","last_became_peered":"2024-03-12T05:53:38.210169+0000","last_unstale":"2024-03-12T05:54:42.011922+0000","last_undegraded":"2024-03-12T05:54:42.011922+0000","last_fullsized":"2024-03-12T05:54:42.011922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:56:05.384834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013770+0000","last_change":"2024-03-12T05:53:37.204478+0000","last_active":"2024-03-12T05:54:42.013770+0000","last_peered":"2024-03-12T05:54:42.013770+0000","last_clean":"2024-03-12T05:54:42.013770+0000","last_became_active":"2024-03-12T05:53:37.204160+0000","last_became_peered":"2024-03-12T05:53:37.204160+0000","last_unstale":"2024-03-12T05:54:42.013770+0000","last_undegraded":"2024-03-12T05:54:42.013770+0000","last_fullsized":"2024-03-12T05:54:42.013770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:07.597731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.026040+0000","last_change":"2024-03-12T05:54:42.026040+0000","last_active":"2024-03-12T05:54:42.026040+0000","last_peered":"2024-03-12T05:54:42.026040+0000","last_clean":"2024-03-12T05:54:42.026040+0000","last_became_active":"2024-03-12T05:54:42.018230+0000","last_became_peered":"2024-03-12T05:54:42.018230+0000","last_unstale":"2024-03-12T05:54:42.026040+0000","last_undegraded":"2024-03-12T05:54:42.026040+0000","last_fullsized":"2024-03-12T05:54:42.026040+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:32:28.703339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687385+0000","last_change":"2024-03-12T05:53:37.205983+0000","last_active":"2024-03-12T05:54:42.687385+0000","last_peered":"2024-03-12T05:54:42.687385+0000","last_clean":"2024-03-12T05:54:42.687385+0000","last_became_active":"2024-03-12T05:53:37.205629+0000","last_became_peered":"2024-03-12T05:53:37.205629+0000","last_unstale":"2024-03-12T05:54:42.687385+0000","last_undegraded":"2024-03-12T05:54:42.687385+0000","last_fullsized":"2024-03-12T05:54:42.687385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:37.928623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014350+0000","last_change":"2024-03-12T05:53:38.213752+0000","last_active":"2024-03-12T05:54:42.014350+0000","last_peered":"2024-03-12T05:54:42.014350+0000","last_clean":"2024-03-12T05:54:42.014350+0000","last_became_active":"2024-03-12T05:53:38.213407+0000","last_became_peered":"2024-03-12T05:53:38.213407+0000","last_unstale":"2024-03-12T05:54:42.014350+0000","last_undegraded":"2024-03-12T05:54:42.014350+0000","last_fullsized":"2024-03-12T05:54:42.014350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:21:15.703455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.022196+0000","last_change":"2024-03-12T05:54:42.022196+0000","last_active":"2024-03-12T05:54:42.022196+0000","last_peered":"2024-03-12T05:54:42.022196+0000","last_clean":"2024-03-12T05:54:42.022196+0000","last_became_active":"2024-03-12T05:54:42.021774+0000","last_became_peered":"2024-03-12T05:54:42.021774+0000","last_unstale":"2024-03-12T05:54:42.022196+0000","last_undegraded":"2024-03-12T05:54:42.022196+0000","last_fullsized":"2024-03-12T05:54:42.022196+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:34:39.290359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:53.494229+0000","last_change":"2024-03-12T05:53:37.202782+0000","last_active":"2024-03-12T05:54:53.494229+0000","last_peered":"2024-03-12T05:54:53.494229+0000","last_clean":"2024-03-12T05:54:53.494229+0000","last_became_active":"2024-03-12T05:53:37.202439+0000","last_became_peered":"2024-03-12T05:53:37.202439+0000","last_unstale":"2024-03-12T05:54:53.494229+0000","last_undegraded":"2024-03-12T05:54:53.494229+0000","last_fullsized":"2024-03-12T05:54:53.494229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:07:18.939629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017722+0000","last_change":"2024-03-12T05:53:38.217993+0000","last_active":"2024-03-12T05:54:42.017722+0000","last_peered":"2024-03-12T05:54:42.017722+0000","last_clean":"2024-03-12T05:54:42.017722+0000","last_became_active":"2024-03-12T05:53:38.217613+0000","last_became_peered":"2024-03-12T05:53:38.217613+0000","last_unstale":"2024-03-12T05:54:42.017722+0000","last_undegraded":"2024-03-12T05:54:42.017722+0000","last_fullsized":"2024-03-12T05:54:42.017722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:49:50.960004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.478908+0000","last_change":"2024-03-12T05:53:37.203662+0000","last_active":"2024-03-12T05:54:42.478908+0000","last_peered":"2024-03-12T05:54:42.478908+0000","last_clean":"2024-03-12T05:54:42.478908+0000","last_became_active":"2024-03-12T05:53:37.203309+0000","last_became_peered":"2024-03-12T05:53:37.203309+0000","last_unstale":"2024-03-12T05:54:42.478908+0000","last_undegraded":"2024-03-12T05:54:42.478908+0000","last_fullsized":"2024-03-12T05:54:42.478908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:47:24.603686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":351,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687252+0000","last_change":"2024-03-12T05:52:27.278524+0000","last_active":"2024-03-12T05:54:42.687252+0000","last_peered":"2024-03-12T05:54:42.687252+0000","last_clean":"2024-03-12T05:54:42.687252+0000","last_became_active":"2024-03-12T05:52:27.278010+0000","last_became_peered":"2024-03-12T05:52:27.278010+0000","last_unstale":"2024-03-12T05:54:42.687252+0000","last_undegraded":"2024-03-12T05:54:42.687252+0000","last_fullsized":"2024-03-12T05:54:42.687252+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:49:09.175725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:49:09.175725+0000","last_clean_scrub_stamp":"2024-03-12T05:49:09.175725+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:24:40.380724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016346+0000","last_change":"2024-03-12T05:53:38.219451+0000","last_active":"2024-03-12T05:54:42.016346+0000","last_peered":"2024-03-12T05:54:42.016346+0000","last_clean":"2024-03-12T05:54:42.016346+0000","last_became_active":"2024-03-12T05:53:38.218840+0000","last_became_peered":"2024-03-12T05:53:38.218840+0000","last_unstale":"2024-03-12T05:54:42.016346+0000","last_undegraded":"2024-03-12T05:54:42.016346+0000","last_fullsized":"2024-03-12T05:54:42.016346+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:05.595822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687398+0000","last_change":"2024-03-12T05:53:37.210352+0000","last_active":"2024-03-12T05:54:42.687398+0000","last_peered":"2024-03-12T05:54:42.687398+0000","last_clean":"2024-03-12T05:54:42.687398+0000","last_became_active":"2024-03-12T05:53:37.210002+0000","last_became_peered":"2024-03-12T05:53:37.210002+0000","last_unstale":"2024-03-12T05:54:42.687398+0000","last_undegraded":"2024-03-12T05:54:42.687398+0000","last_fullsized":"2024-03-12T05:54:42.687398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:49:52.368411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.022961+0000","last_change":"2024-03-12T05:54:42.022489+0000","last_active":"2024-03-12T05:54:42.022961+0000","last_peered":"2024-03-12T05:54:42.022961+0000","last_clean":"2024-03-12T05:54:42.022961+0000","last_became_active":"2024-03-12T05:54:42.016400+0000","last_became_peered":"2024-03-12T05:54:42.016400+0000","last_unstale":"2024-03-12T05:54:42.022961+0000","last_undegraded":"2024-03-12T05:54:42.022961+0000","last_fullsized":"2024-03-12T05:54:42.022961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:32:05.713420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017332+0000","last_change":"2024-03-12T05:53:38.203951+0000","last_active":"2024-03-12T05:54:42.017332+0000","last_peered":"2024-03-12T05:54:42.017332+0000","last_clean":"2024-03-12T05:54:42.017332+0000","last_became_active":"2024-03-12T05:53:38.203617+0000","last_became_peered":"2024-03-12T05:53:38.203617+0000","last_unstale":"2024-03-12T05:54:42.017332+0000","last_undegraded":"2024-03-12T05:54:42.017332+0000","last_fullsized":"2024-03-12T05:54:42.017332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:19.138220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013822+0000","last_change":"2024-03-12T05:53:37.203770+0000","last_active":"2024-03-12T05:54:42.013822+0000","last_peered":"2024-03-12T05:54:42.013822+0000","last_clean":"2024-03-12T05:54:42.013822+0000","last_became_active":"2024-03-12T05:53:37.203456+0000","last_became_peered":"2024-03-12T05:53:37.203456+0000","last_unstale":"2024-03-12T05:54:42.013822+0000","last_undegraded":"2024-03-12T05:54:42.013822+0000","last_fullsized":"2024-03-12T05:54:42.013822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:32.291536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018655+0000","last_change":"2024-03-12T05:53:38.214872+0000","last_active":"2024-03-12T05:54:42.018655+0000","last_peered":"2024-03-12T05:54:42.018655+0000","last_clean":"2024-03-12T05:54:42.018655+0000","last_became_active":"2024-03-12T05:53:38.214139+0000","last_became_peered":"2024-03-12T05:53:38.214139+0000","last_unstale":"2024-03-12T05:54:42.018655+0000","last_undegraded":"2024-03-12T05:54:42.018655+0000","last_fullsized":"2024-03-12T05:54:42.018655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:20:21.808039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.479035+0000","last_change":"2024-03-12T05:53:37.203852+0000","last_active":"2024-03-12T05:54:42.479035+0000","last_peered":"2024-03-12T05:54:42.479035+0000","last_clean":"2024-03-12T05:54:42.479035+0000","last_became_active":"2024-03-12T05:53:37.203416+0000","last_became_peered":"2024-03-12T05:53:37.203416+0000","last_unstale":"2024-03-12T05:54:42.479035+0000","last_undegraded":"2024-03-12T05:54:42.479035+0000","last_fullsized":"2024-03-12T05:54:42.479035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:09:08.095957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.688395+0000","last_change":"2024-03-12T05:54:42.688395+0000","last_active":"2024-03-12T05:54:42.688395+0000","last_peered":"2024-03-12T05:54:42.688395+0000","last_clean":"2024-03-12T05:54:42.688395+0000","last_became_active":"2024-03-12T05:54:42.688094+0000","last_became_peered":"2024-03-12T05:54:42.688094+0000","last_unstale":"2024-03-12T05:54:42.688395+0000","last_undegraded":"2024-03-12T05:54:42.688395+0000","last_fullsized":"2024-03-12T05:54:42.688395+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:29:42.921726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.025218+0000","last_change":"2024-03-12T05:54:42.025218+0000","last_active":"2024-03-12T05:54:42.025218+0000","last_peered":"2024-03-12T05:54:42.025218+0000","last_clean":"2024-03-12T05:54:42.025218+0000","last_became_active":"2024-03-12T05:54:42.021022+0000","last_became_peered":"2024-03-12T05:54:42.021022+0000","last_unstale":"2024-03-12T05:54:42.025218+0000","last_undegraded":"2024-03-12T05:54:42.025218+0000","last_fullsized":"2024-03-12T05:54:42.025218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:20:57.232145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013944+0000","last_change":"2024-03-12T05:53:38.207757+0000","last_active":"2024-03-12T05:54:42.013944+0000","last_peered":"2024-03-12T05:54:42.013944+0000","last_clean":"2024-03-12T05:54:42.013944+0000","last_became_active":"2024-03-12T05:53:38.207424+0000","last_became_peered":"2024-03-12T05:53:38.207424+0000","last_unstale":"2024-03-12T05:54:42.013944+0000","last_undegraded":"2024-03-12T05:54:42.013944+0000","last_fullsized":"2024-03-12T05:54:42.013944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:33:16.017907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016731+0000","last_change":"2024-03-12T05:53:37.208512+0000","last_active":"2024-03-12T05:54:42.016731+0000","last_peered":"2024-03-12T05:54:42.016731+0000","last_clean":"2024-03-12T05:54:42.016731+0000","last_became_active":"2024-03-12T05:53:37.207628+0000","last_became_peered":"2024-03-12T05:53:37.207628+0000","last_unstale":"2024-03-12T05:54:42.016731+0000","last_undegraded":"2024-03-12T05:54:42.016731+0000","last_fullsized":"2024-03-12T05:54:42.016731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:25:28.825779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.011698+0000","last_change":"2024-03-12T05:53:38.205654+0000","last_active":"2024-03-12T05:54:42.011698+0000","last_peered":"2024-03-12T05:54:42.011698+0000","last_clean":"2024-03-12T05:54:42.011698+0000","last_became_active":"2024-03-12T05:53:38.205325+0000","last_became_peered":"2024-03-12T05:53:38.205325+0000","last_unstale":"2024-03-12T05:54:42.011698+0000","last_undegraded":"2024-03-12T05:54:42.011698+0000","last_fullsized":"2024-03-12T05:54:42.011698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:33:35.668625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019632+0000","last_change":"2024-03-12T05:53:37.211430+0000","last_active":"2024-03-12T05:54:42.019632+0000","last_peered":"2024-03-12T05:54:42.019632+0000","last_clean":"2024-03-12T05:54:42.019632+0000","last_became_active":"2024-03-12T05:53:37.210501+0000","last_became_peered":"2024-03-12T05:53:37.210501+0000","last_unstale":"2024-03-12T05:54:42.019632+0000","last_undegraded":"2024-03-12T05:54:42.019632+0000","last_fullsized":"2024-03-12T05:54:42.019632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:48:33.555134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017930+0000","last_change":"2024-03-12T05:53:38.212837+0000","last_active":"2024-03-12T05:54:42.017930+0000","last_peered":"2024-03-12T05:54:42.017930+0000","last_clean":"2024-03-12T05:54:42.017930+0000","last_became_active":"2024-03-12T05:53:38.212513+0000","last_became_peered":"2024-03-12T05:53:38.212513+0000","last_unstale":"2024-03-12T05:54:42.017930+0000","last_undegraded":"2024-03-12T05:54:42.017930+0000","last_fullsized":"2024-03-12T05:54:42.017930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:47:33.879954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014092+0000","last_change":"2024-03-12T05:53:37.207465+0000","last_active":"2024-03-12T05:54:42.014092+0000","last_peered":"2024-03-12T05:54:42.014092+0000","last_clean":"2024-03-12T05:54:42.014092+0000","last_became_active":"2024-03-12T05:53:37.207107+0000","last_became_peered":"2024-03-12T05:53:37.207107+0000","last_unstale":"2024-03-12T05:54:42.014092+0000","last_undegraded":"2024-03-12T05:54:42.014092+0000","last_fullsized":"2024-03-12T05:54:42.014092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:34.018856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686518+0000","last_change":"2024-03-12T05:53:38.221470+0000","last_active":"2024-03-12T05:54:42.686518+0000","last_peered":"2024-03-12T05:54:42.686518+0000","last_clean":"2024-03-12T05:54:42.686518+0000","last_became_active":"2024-03-12T05:53:38.221061+0000","last_became_peered":"2024-03-12T05:53:38.221061+0000","last_unstale":"2024-03-12T05:54:42.686518+0000","last_undegraded":"2024-03-12T05:54:42.686518+0000","last_fullsized":"2024-03-12T05:54:42.686518+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:01:54.148877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019769+0000","last_change":"2024-03-12T05:53:37.203822+0000","last_active":"2024-03-12T05:54:42.019769+0000","last_peered":"2024-03-12T05:54:42.019769+0000","last_clean":"2024-03-12T05:54:42.019769+0000","last_became_active":"2024-03-12T05:53:37.203431+0000","last_became_peered":"2024-03-12T05:53:37.203431+0000","last_unstale":"2024-03-12T05:54:42.019769+0000","last_undegraded":"2024-03-12T05:54:42.019769+0000","last_fullsized":"2024-03-12T05:54:42.019769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:25:04.447261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686535+0000","last_change":"2024-03-12T05:53:38.222045+0000","last_active":"2024-03-12T05:54:42.686535+0000","last_peered":"2024-03-12T05:54:42.686535+0000","last_clean":"2024-03-12T05:54:42.686535+0000","last_became_active":"2024-03-12T05:53:38.220904+0000","last_became_peered":"2024-03-12T05:53:38.220904+0000","last_unstale":"2024-03-12T05:54:42.686535+0000","last_undegraded":"2024-03-12T05:54:42.686535+0000","last_fullsized":"2024-03-12T05:54:42.686535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:30:43.964599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019437+0000","last_change":"2024-03-12T05:53:37.206753+0000","last_active":"2024-03-12T05:54:42.019437+0000","last_peered":"2024-03-12T05:54:42.019437+0000","last_clean":"2024-03-12T05:54:42.019437+0000","last_became_active":"2024-03-12T05:53:37.206078+0000","last_became_peered":"2024-03-12T05:53:37.206078+0000","last_unstale":"2024-03-12T05:54:42.019437+0000","last_undegraded":"2024-03-12T05:54:42.019437+0000","last_fullsized":"2024-03-12T05:54:42.019437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:42:18.859528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686435+0000","last_change":"2024-03-12T05:53:38.215671+0000","last_active":"2024-03-12T05:54:42.686435+0000","last_peered":"2024-03-12T05:54:42.686435+0000","last_clean":"2024-03-12T05:54:42.686435+0000","last_became_active":"2024-03-12T05:53:38.215309+0000","last_became_peered":"2024-03-12T05:53:38.215309+0000","last_unstale":"2024-03-12T05:54:42.686435+0000","last_undegraded":"2024-03-12T05:54:42.686435+0000","last_fullsized":"2024-03-12T05:54:42.686435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:50:12.176935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017343+0000","last_change":"2024-03-12T05:53:37.205627+0000","last_active":"2024-03-12T05:54:42.017343+0000","last_peered":"2024-03-12T05:54:42.017343+0000","last_clean":"2024-03-12T05:54:42.017343+0000","last_became_active":"2024-03-12T05:53:37.205289+0000","last_became_peered":"2024-03-12T05:53:37.205289+0000","last_unstale":"2024-03-12T05:54:42.017343+0000","last_undegraded":"2024-03-12T05:54:42.017343+0000","last_fullsized":"2024-03-12T05:54:42.017343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:56.682746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.686583+0000","last_change":"2024-03-12T05:53:38.214376+0000","last_active":"2024-03-12T05:54:42.686583+0000","last_peered":"2024-03-12T05:54:42.686583+0000","last_clean":"2024-03-12T05:54:42.686583+0000","last_became_active":"2024-03-12T05:53:38.214011+0000","last_became_peered":"2024-03-12T05:53:38.214011+0000","last_unstale":"2024-03-12T05:54:42.686583+0000","last_undegraded":"2024-03-12T05:54:42.686583+0000","last_fullsized":"2024-03-12T05:54:42.686583+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:32.605216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.011774+0000","last_change":"2024-03-12T05:53:38.205337+0000","last_active":"2024-03-12T05:54:42.011774+0000","last_peered":"2024-03-12T05:54:42.011774+0000","last_clean":"2024-03-12T05:54:42.011774+0000","last_became_active":"2024-03-12T05:53:38.205005+0000","last_became_peered":"2024-03-12T05:53:38.205005+0000","last_unstale":"2024-03-12T05:54:42.011774+0000","last_undegraded":"2024-03-12T05:54:42.011774+0000","last_fullsized":"2024-03-12T05:54:42.011774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:44:16.296370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018964+0000","last_change":"2024-03-12T05:53:38.214694+0000","last_active":"2024-03-12T05:54:42.018964+0000","last_peered":"2024-03-12T05:54:42.018964+0000","last_clean":"2024-03-12T05:54:42.018964+0000","last_became_active":"2024-03-12T05:53:38.214285+0000","last_became_peered":"2024-03-12T05:53:38.214285+0000","last_unstale":"2024-03-12T05:54:42.018964+0000","last_undegraded":"2024-03-12T05:54:42.018964+0000","last_fullsized":"2024-03-12T05:54:42.018964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:28:32.253332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019495+0000","last_change":"2024-03-12T05:53:38.213069+0000","last_active":"2024-03-12T05:54:42.019495+0000","last_peered":"2024-03-12T05:54:42.019495+0000","last_clean":"2024-03-12T05:54:42.019495+0000","last_became_active":"2024-03-12T05:53:38.212694+0000","last_became_peered":"2024-03-12T05:53:38.212694+0000","last_unstale":"2024-03-12T05:54:42.019495+0000","last_undegraded":"2024-03-12T05:54:42.019495+0000","last_fullsized":"2024-03-12T05:54:42.019495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:26:29.350292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014551+0000","last_change":"2024-03-12T05:53:38.212084+0000","last_active":"2024-03-12T05:54:42.014551+0000","last_peered":"2024-03-12T05:54:42.014551+0000","last_clean":"2024-03-12T05:54:42.014551+0000","last_became_active":"2024-03-12T05:53:38.211750+0000","last_became_peered":"2024-03-12T05:53:38.211750+0000","last_unstale":"2024-03-12T05:54:42.014551+0000","last_undegraded":"2024-03-12T05:54:42.014551+0000","last_fullsized":"2024-03-12T05:54:42.014551+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:05:02.099121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017345+0000","last_change":"2024-03-12T05:53:38.212071+0000","last_active":"2024-03-12T05:54:42.017345+0000","last_peered":"2024-03-12T05:54:42.017345+0000","last_clean":"2024-03-12T05:54:42.017345+0000","last_became_active":"2024-03-12T05:53:38.211722+0000","last_became_peered":"2024-03-12T05:53:38.211722+0000","last_unstale":"2024-03-12T05:54:42.017345+0000","last_undegraded":"2024-03-12T05:54:42.017345+0000","last_fullsized":"2024-03-12T05:54:42.017345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:19:46.581837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.015717+0000","last_change":"2024-03-12T05:54:42.015717+0000","last_active":"2024-03-12T05:54:42.015717+0000","last_peered":"2024-03-12T05:54:42.015717+0000","last_clean":"2024-03-12T05:54:42.015717+0000","last_became_active":"2024-03-12T05:54:42.015266+0000","last_became_peered":"2024-03-12T05:54:42.015266+0000","last_unstale":"2024-03-12T05:54:42.015717+0000","last_undegraded":"2024-03-12T05:54:42.015717+0000","last_fullsized":"2024-03-12T05:54:42.015717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:33:48.250458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.020263+0000","last_change":"2024-03-12T05:53:38.208018+0000","last_active":"2024-03-12T05:54:42.020263+0000","last_peered":"2024-03-12T05:54:42.020263+0000","last_clean":"2024-03-12T05:54:42.020263+0000","last_became_active":"2024-03-12T05:53:38.207682+0000","last_became_peered":"2024-03-12T05:53:38.207682+0000","last_unstale":"2024-03-12T05:54:42.020263+0000","last_undegraded":"2024-03-12T05:54:42.020263+0000","last_fullsized":"2024-03-12T05:54:42.020263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:56:10.903917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018645+0000","last_change":"2024-03-12T05:53:38.217540+0000","last_active":"2024-03-12T05:54:42.018645+0000","last_peered":"2024-03-12T05:54:42.018645+0000","last_clean":"2024-03-12T05:54:42.018645+0000","last_became_active":"2024-03-12T05:53:38.217144+0000","last_became_peered":"2024-03-12T05:53:38.217144+0000","last_unstale":"2024-03-12T05:54:42.018645+0000","last_undegraded":"2024-03-12T05:54:42.018645+0000","last_fullsized":"2024-03-12T05:54:42.018645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:46:18.905909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017540+0000","last_change":"2024-03-12T05:53:38.215006+0000","last_active":"2024-03-12T05:54:42.017540+0000","last_peered":"2024-03-12T05:54:42.017540+0000","last_clean":"2024-03-12T05:54:42.017540+0000","last_became_active":"2024-03-12T05:53:38.213756+0000","last_became_peered":"2024-03-12T05:53:38.213756+0000","last_unstale":"2024-03-12T05:54:42.017540+0000","last_undegraded":"2024-03-12T05:54:42.017540+0000","last_fullsized":"2024-03-12T05:54:42.017540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:14:57.642381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016570+0000","last_change":"2024-03-12T05:53:38.216096+0000","last_active":"2024-03-12T05:54:42.016570+0000","last_peered":"2024-03-12T05:54:42.016570+0000","last_clean":"2024-03-12T05:54:42.016570+0000","last_became_active":"2024-03-12T05:53:38.215153+0000","last_became_peered":"2024-03-12T05:53:38.215153+0000","last_unstale":"2024-03-12T05:54:42.016570+0000","last_undegraded":"2024-03-12T05:54:42.016570+0000","last_fullsized":"2024-03-12T05:54:42.016570+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:41:03.893641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016566+0000","last_change":"2024-03-12T05:53:38.219165+0000","last_active":"2024-03-12T05:54:42.016566+0000","last_peered":"2024-03-12T05:54:42.016566+0000","last_clean":"2024-03-12T05:54:42.016566+0000","last_became_active":"2024-03-12T05:53:38.218579+0000","last_became_peered":"2024-03-12T05:53:38.218579+0000","last_unstale":"2024-03-12T05:54:42.016566+0000","last_undegraded":"2024-03-12T05:54:42.016566+0000","last_fullsized":"2024-03-12T05:54:42.016566+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:06:37.531293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017229+0000","last_change":"2024-03-12T05:53:38.212685+0000","last_active":"2024-03-12T05:54:42.017229+0000","last_peered":"2024-03-12T05:54:42.017229+0000","last_clean":"2024-03-12T05:54:42.017229+0000","last_became_active":"2024-03-12T05:53:38.211106+0000","last_became_peered":"2024-03-12T05:53:38.211106+0000","last_unstale":"2024-03-12T05:54:42.017229+0000","last_undegraded":"2024-03-12T05:54:42.017229+0000","last_fullsized":"2024-03-12T05:54:42.017229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:26:00.339270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.011757+0000","last_change":"2024-03-12T05:53:38.218402+0000","last_active":"2024-03-12T05:54:42.011757+0000","last_peered":"2024-03-12T05:54:42.011757+0000","last_clean":"2024-03-12T05:54:42.011757+0000","last_became_active":"2024-03-12T05:53:38.217699+0000","last_became_peered":"2024-03-12T05:53:38.217699+0000","last_unstale":"2024-03-12T05:54:42.011757+0000","last_undegraded":"2024-03-12T05:54:42.011757+0000","last_fullsized":"2024-03-12T05:54:42.011757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:23:03.221710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65047,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":456,"num_read_kb":300,"num_write":156,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":195141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351005,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":461631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612642,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32796,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712356,"statfs":{"total":95995035648,"available":95961452544,"internally_reserved":0,"allocated":1650688,"data_stored":1114136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776167,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":461962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037804,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":461743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332144,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32712,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712440,"statfs":{"total":95995035648,"available":95961538560,"internally_reserved":0,"allocated":1564672,"data_stored":1053695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495670,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":461234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1019904,"data_stored":522073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":462505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":523814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182468,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":461428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920831,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":962560,"data_stored":461671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757306,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28564,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716588,"statfs":{"total":95995035648,"available":95965786112,"internally_reserved":0,"allocated":1576960,"data_stored":1053379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T05:54:56.806 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-12T05:54:56.806 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-12T05:54:56.806 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-12T05:54:56.806 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-12T05:54:57.052 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:54:57.373 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:57 smithi076 ceph-mon[32095]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:54:57.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:57 smithi082 ceph-mon[36382]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:54:57.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:57 smithi088 ceph-mon[36585]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:54:58.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:58 smithi076 ceph-mon[32095]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:54:58.346 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:58 smithi082 ceph-mon[36382]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:54:58.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:58 smithi088 ceph-mon[36585]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:54:58.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.577+0000 7fbe68136700 1 -- 172.21.15.76:0/1045033823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe60101cc0 msgr2=0x7fbe60102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:58.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.577+0000 7fbe68136700 1 --2- 172.21.15.76:0/1045033823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe60101cc0 0x7fbe60102140 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0099c0 tx=0x7fbe5c009cd0 comp rx=0 tx=0).stop 2024-03-12T05:54:58.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.577+0000 7fbe68136700 1 -- 172.21.15.76:0/1045033823 shutdown_connections 2024-03-12T05:54:58.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.577+0000 7fbe68136700 1 --2- 172.21.15.76:0/1045033823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe60102680 0x7fbe6010eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:58.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.577+0000 7fbe68136700 1 --2- 172.21.15.76:0/1045033823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe60107ca0 0x7fbe60108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:58.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.577+0000 7fbe68136700 1 --2- 172.21.15.76:0/1045033823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe60101cc0 0x7fbe60102140 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:58.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.577+0000 7fbe68136700 1 -- 172.21.15.76:0/1045033823 >> 172.21.15.76:0/1045033823 conn(0x7fbe60078da0 msgr2=0x7fbe600791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:58.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.578+0000 7fbe68136700 1 -- 172.21.15.76:0/1045033823 shutdown_connections 2024-03-12T05:54:58.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.578+0000 7fbe68136700 1 -- 172.21.15.76:0/1045033823 wait complete. 2024-03-12T05:54:58.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.578+0000 7fbe68136700 1 Processor -- start 2024-03-12T05:54:58.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.578+0000 7fbe68136700 1 -- start start 2024-03-12T05:54:58.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.578+0000 7fbe68136700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe60101cc0 0x7fbe60198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:58.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.578+0000 7fbe68136700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe60102680 0x7fbe601985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:58.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.578+0000 7fbe68136700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe60107ca0 0x7fbe6019c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:58.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.578+0000 7fbe68136700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6019cf60 con 0x7fbe60107ca0 2024-03-12T05:54:58.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.578+0000 7fbe68136700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6019d0d0 con 0x7fbe60101cc0 2024-03-12T05:54:58.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.578+0000 7fbe68136700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6019d290 con 0x7fbe60102680 2024-03-12T05:54:58.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.578+0000 7fbe666d3700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe60107ca0 0x7fbe6019c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:58.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe666d3700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe60107ca0 0x7fbe6019c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58666/0 (socket says 172.21.15.76:58666) 2024-03-12T05:54:58.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe666d3700 1 -- 172.21.15.76:0/3719816263 learned_addr learned my addr 172.21.15.76:0/3719816263 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:58.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe656d1700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe60102680 0x7fbe601985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:58.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe65ed2700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe60101cc0 0x7fbe60198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:58.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe666d3700 1 -- 172.21.15.76:0/3719816263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe60102680 msgr2=0x7fbe601985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:58.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe666d3700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe60102680 0x7fbe601985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:58.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe666d3700 1 -- 172.21.15.76:0/3719816263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe60101cc0 msgr2=0x7fbe60198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:58.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe666d3700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe60101cc0 0x7fbe60198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:58.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe666d3700 1 -- 172.21.15.76:0/3719816263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c0095f0 con 0x7fbe60107ca0 2024-03-12T05:54:58.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe65ed2700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe60101cc0 0x7fbe60198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:54:58.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe666d3700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe60107ca0 0x7fbe6019c950 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbe58007d70 tx=0x7fbe5800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:58.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe56ffd700 1 -- 172.21.15.76:0/3719816263 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800ad40 con 0x7fbe60107ca0 2024-03-12T05:54:58.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe68136700 1 -- 172.21.15.76:0/3719816263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe6019d570 con 0x7fbe60107ca0 2024-03-12T05:54:58.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe56ffd700 1 -- 172.21.15.76:0/3719816263 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe58012900 con 0x7fbe60107ca0 2024-03-12T05:54:58.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe68136700 1 -- 172.21.15.76:0/3719816263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe601a4dd0 con 0x7fbe60107ca0 2024-03-12T05:54:58.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.579+0000 7fbe56ffd700 1 -- 172.21.15.76:0/3719816263 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58011780 con 0x7fbe60107ca0 2024-03-12T05:54:58.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.580+0000 7fbe68136700 1 -- 172.21.15.76:0/3719816263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6010a180 con 0x7fbe60107ca0 2024-03-12T05:54:58.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.581+0000 7fbe56ffd700 1 -- 172.21.15.76:0/3719816263 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fbe58012a70 con 0x7fbe60107ca0 2024-03-12T05:54:58.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.582+0000 7fbe56ffd700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbe4c0a1b50 0x7fbe4c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:58.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.582+0000 7fbe65ed2700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbe4c0a1b50 0x7fbe4c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:58.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.582+0000 7fbe56ffd700 1 -- 172.21.15.76:0/3719816263 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbe580cdb30 con 0x7fbe60107ca0 2024-03-12T05:54:58.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.582+0000 7fbe65ed2700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbe4c0a1b50 0x7fbe4c0a4010 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbe50005d90 tx=0x7fbe50005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:58.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.585+0000 7fbe56ffd700 1 -- 172.21.15.76:0/3719816263 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbe58096d00 con 0x7fbe60107ca0 2024-03-12T05:54:58.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.825+0000 7fbe68136700 1 -- 172.21.15.76:0/3719816263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbe60068930 con 0x7fbe60107ca0 2024-03-12T05:54:58.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.825+0000 7fbe56ffd700 1 -- 172.21.15.76:0/3719816263 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fbe5809a3c0 con 0x7fbe60107ca0 2024-03-12T05:54:58.829 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:54:58.829 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-12T05:54:58.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.827+0000 7fbe68136700 1 -- 172.21.15.76:0/3719816263 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbe4c0a1b50 msgr2=0x7fbe4c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:58.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.827+0000 7fbe68136700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbe4c0a1b50 0x7fbe4c0a4010 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbe50005d90 tx=0x7fbe50005d00 comp rx=0 tx=0).stop 2024-03-12T05:54:58.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.827+0000 7fbe68136700 1 -- 172.21.15.76:0/3719816263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe60107ca0 msgr2=0x7fbe6019c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:58.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.827+0000 7fbe68136700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe60107ca0 0x7fbe6019c950 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbe58007d70 tx=0x7fbe5800a340 comp rx=0 tx=0).stop 2024-03-12T05:54:58.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.827+0000 7fbe68136700 1 -- 172.21.15.76:0/3719816263 shutdown_connections 2024-03-12T05:54:58.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.827+0000 7fbe68136700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbe4c0a1b50 0x7fbe4c0a4010 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:58.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.827+0000 7fbe68136700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe60107ca0 0x7fbe6019c950 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:58.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.827+0000 7fbe68136700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe60101cc0 0x7fbe60198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:58.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.827+0000 7fbe68136700 1 --2- 172.21.15.76:0/3719816263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe60102680 0x7fbe601985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:58.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.827+0000 7fbe68136700 1 -- 172.21.15.76:0/3719816263 >> 172.21.15.76:0/3719816263 conn(0x7fbe60078da0 msgr2=0x7fbe601007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:58.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.827+0000 7fbe68136700 1 -- 172.21.15.76:0/3719816263 shutdown_connections 2024-03-12T05:54:58.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:58.827+0000 7fbe68136700 1 -- 172.21.15.76:0/3719816263 wait complete. 2024-03-12T05:54:59.184 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:59 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3719816263' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T05:54:59.184 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:54:59 smithi076 ceph-mon[32095]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:54:59.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:59 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3719816263' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T05:54:59.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:54:59 smithi082 ceph-mon[36382]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:54:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:59 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3719816263' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T05:54:59.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:54:59 smithi088 ceph-mon[36585]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:54:59.505 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-12T05:54:59.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:54:59.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.609+0000 7fe047064640 1 -- 172.21.15.76:0/2609084895 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0401000b0 msgr2=0x7fe0401004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:59.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.609+0000 7fe047064640 1 --2- 172.21.15.76:0/2609084895 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0401000b0 0x7fe0401004d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009900 tx=0x7fe03c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:54:59.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 -- 172.21.15.76:0/2609084895 shutdown_connections 2024-03-12T05:54:59.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 --2- 172.21.15.76:0/2609084895 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe040101c60 0x7fe0401084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 --2- 172.21.15.76:0/2609084895 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe0401012a0 0x7fe040101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 --2- 172.21.15.76:0/2609084895 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0401000b0 0x7fe0401004d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 -- 172.21.15.76:0/2609084895 >> 172.21.15.76:0/2609084895 conn(0x7fe0400fb620 msgr2=0x7fe0400fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:59.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 -- 172.21.15.76:0/2609084895 shutdown_connections 2024-03-12T05:54:59.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 -- 172.21.15.76:0/2609084895 wait complete. 2024-03-12T05:54:59.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 Processor -- start 2024-03-12T05:54:59.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 -- start start 2024-03-12T05:54:59.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0401000b0 0x7fe0401972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:59.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0401012a0 0x7fe040197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:59.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe040101c60 0x7fe04019e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:59.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04019edc0 con 0x7fe0401012a0 2024-03-12T05:54:59.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04019ef30 con 0x7fe040101c60 2024-03-12T05:54:59.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe047064640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04019f0f0 con 0x7fe0401000b0 2024-03-12T05:54:59.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe037fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0401012a0 0x7fe040197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:59.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe037fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0401012a0 0x7fe040197810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58672/0 (socket says 172.21.15.76:58672) 2024-03-12T05:54:59.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe037fff640 1 -- 172.21.15.76:0/353510636 learned_addr learned my addr 172.21.15.76:0/353510636 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:59.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.610+0000 7fe045601640 1 --2- 172.21.15.76:0/353510636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe040101c60 0x7fe04019e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:59.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.611+0000 7fe037fff640 1 -- 172.21.15.76:0/353510636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0401000b0 msgr2=0x7fe0401972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:59.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.611+0000 7fe037fff640 1 --2- 172.21.15.76:0/353510636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0401000b0 0x7fe0401972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.611+0000 7fe037fff640 1 -- 172.21.15.76:0/353510636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe040101c60 msgr2=0x7fe04019e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:59.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.611+0000 7fe037fff640 1 --2- 172.21.15.76:0/353510636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe040101c60 0x7fe04019e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.611+0000 7fe037fff640 1 -- 172.21.15.76:0/353510636 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c009560 con 0x7fe0401012a0 2024-03-12T05:54:59.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.611+0000 7fe045601640 1 --2- 172.21.15.76:0/353510636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe040101c60 0x7fe04019e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:54:59.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.611+0000 7fe037fff640 1 --2- 172.21.15.76:0/353510636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0401012a0 0x7fe040197810 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe03000d950 tx=0x7fe03000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:59.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.611+0000 7fe035ffb640 1 -- 172.21.15.76:0/353510636 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe030009c30 con 0x7fe0401012a0 2024-03-12T05:54:59.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.611+0000 7fe035ffb640 1 -- 172.21.15.76:0/353510636 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe030009d90 con 0x7fe0401012a0 2024-03-12T05:54:59.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.611+0000 7fe047064640 1 -- 172.21.15.76:0/353510636 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe04019f3d0 con 0x7fe0401012a0 2024-03-12T05:54:59.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.611+0000 7fe035ffb640 1 -- 172.21.15.76:0/353510636 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03000f7e0 con 0x7fe0401012a0 2024-03-12T05:54:59.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.611+0000 7fe047064640 1 -- 172.21.15.76:0/353510636 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe04019f960 con 0x7fe0401012a0 2024-03-12T05:54:59.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.612+0000 7fe047064640 1 -- 172.21.15.76:0/353510636 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe04010bb00 con 0x7fe0401012a0 2024-03-12T05:54:59.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.613+0000 7fe035ffb640 1 -- 172.21.15.76:0/353510636 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fe030010ad0 con 0x7fe0401012a0 2024-03-12T05:54:59.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.614+0000 7fe035ffb640 1 --2- 172.21.15.76:0/353510636 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe0280a1ac0 0x7fe0280a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:59.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.614+0000 7fe044e00640 1 --2- 172.21.15.76:0/353510636 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe0280a1ac0 0x7fe0280a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:59.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.614+0000 7fe035ffb640 1 -- 172.21.15.76:0/353510636 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe0300cd2c0 con 0x7fe0401012a0 2024-03-12T05:54:59.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.614+0000 7fe044e00640 1 --2- 172.21.15.76:0/353510636 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe0280a1ac0 0x7fe0280a3f80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe03c02c6a0 tx=0x7fe03c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:59.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.619+0000 7fe035ffb640 1 -- 172.21.15.76:0/353510636 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe030096380 con 0x7fe0401012a0 2024-03-12T05:54:59.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.794+0000 7fe047064640 1 -- 172.21.15.76:0/353510636 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe04010bc90 con 0x7fe0401012a0 2024-03-12T05:54:59.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.796+0000 7fe035ffb640 1 -- 172.21.15.76:0/353510636 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22541 (secure 0 0 0) 0x7fe030099a40 con 0x7fe0401012a0 2024-03-12T05:54:59.800 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:54:59.800 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:54:42.007701+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:54:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.798+0000 7fe047064640 1 -- 172.21.15.76:0/353510636 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe0280a1ac0 msgr2=0x7fe0280a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.798+0000 7fe047064640 1 --2- 172.21.15.76:0/353510636 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe0280a1ac0 0x7fe0280a3f80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe03c02c6a0 tx=0x7fe03c000bc0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.798+0000 7fe047064640 1 -- 172.21.15.76:0/353510636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0401012a0 msgr2=0x7fe040197810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.798+0000 7fe047064640 1 --2- 172.21.15.76:0/353510636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0401012a0 0x7fe040197810 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe03000d950 tx=0x7fe03000de10 comp rx=0 tx=0).stop 2024-03-12T05:54:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.798+0000 7fe047064640 1 -- 172.21.15.76:0/353510636 shutdown_connections 2024-03-12T05:54:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.798+0000 7fe047064640 1 --2- 172.21.15.76:0/353510636 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fe0280a1ac0 0x7fe0280a3f80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.798+0000 7fe047064640 1 --2- 172.21.15.76:0/353510636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe040101c60 0x7fe04019e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.798+0000 7fe047064640 1 --2- 172.21.15.76:0/353510636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0401012a0 0x7fe040197810 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.798+0000 7fe047064640 1 --2- 172.21.15.76:0/353510636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe0401000b0 0x7fe0401972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.798+0000 7fe047064640 1 -- 172.21.15.76:0/353510636 >> 172.21.15.76:0/353510636 conn(0x7fe0400fb620 msgr2=0x7fe0400fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.798+0000 7fe047064640 1 -- 172.21.15.76:0/353510636 shutdown_connections 2024-03-12T05:54:59.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.798+0000 7fe047064640 1 -- 172.21.15.76:0/353510636 wait complete. 2024-03-12T05:54:59.814 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:49:09.125536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:53:35.663129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:53:36.740826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:54:59.815 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:54:59.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.917+0000 7f662a793640 1 -- 172.21.15.76:0/4046205771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6624100090 msgr2=0x7f66241004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:59.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.917+0000 7f662a793640 1 --2- 172.21.15.76:0/4046205771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6624100090 0x7f66241004b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6614009900 tx=0x7f661402bc30 comp rx=0 tx=0).stop 2024-03-12T05:54:59.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.917+0000 7f662a793640 1 -- 172.21.15.76:0/4046205771 shutdown_connections 2024-03-12T05:54:59.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.917+0000 7f662a793640 1 --2- 172.21.15.76:0/4046205771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6624101c40 0x7f66241084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.917+0000 7f662a793640 1 --2- 172.21.15.76:0/4046205771 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6624101280 0x7f6624101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.917+0000 7f662a793640 1 --2- 172.21.15.76:0/4046205771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6624100090 0x7f66241004b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.917+0000 7f662a793640 1 -- 172.21.15.76:0/4046205771 >> 172.21.15.76:0/4046205771 conn(0x7f66240fb5e0 msgr2=0x7f66240fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:54:59.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.917+0000 7f662a793640 1 -- 172.21.15.76:0/4046205771 shutdown_connections 2024-03-12T05:54:59.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.917+0000 7f662a793640 1 -- 172.21.15.76:0/4046205771 wait complete. 2024-03-12T05:54:59.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.917+0000 7f662a793640 1 Processor -- start 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.917+0000 7f662a793640 1 -- start start 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f662a793640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6624100090 0x7f662406a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f662a793640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6624101280 0x7f662419bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f662a793640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6624101c40 0x7f662419e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f662a793640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662419edb0 con 0x7f6624100090 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f662a793640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662419ef20 con 0x7f6624101c40 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f662a793640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662419f0e0 con 0x7f6624101280 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f6623fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6624100090 0x7f662406a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f6623fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6624100090 0x7f662406a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58688/0 (socket says 172.21.15.76:58688) 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f6623fff640 1 -- 172.21.15.76:0/2617815326 learned_addr learned my addr 172.21.15.76:0/2617815326 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f6628d30640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6624101c40 0x7f662419e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f66237fe640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6624101280 0x7f662419bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f6623fff640 1 -- 172.21.15.76:0/2617815326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6624101280 msgr2=0x7f662419bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f6623fff640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6624101280 0x7f662419bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f6623fff640 1 -- 172.21.15.76:0/2617815326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6624101c40 msgr2=0x7f662419e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:54:59.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f6623fff640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6624101c40 0x7f662419e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:54:59.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f6623fff640 1 -- 172.21.15.76:0/2617815326 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6614009560 con 0x7f6624100090 2024-03-12T05:54:59.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f6628d30640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6624101c40 0x7f662419e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:54:59.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f6623fff640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6624100090 0x7f662406a3d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6614000c00 tx=0x7f6614004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:59.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f66217fa640 1 -- 172.21.15.76:0/2617815326 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614004df0 con 0x7f6624100090 2024-03-12T05:54:59.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f66217fa640 1 -- 172.21.15.76:0/2617815326 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6614005430 con 0x7f6624100090 2024-03-12T05:54:59.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f662a793640 1 -- 172.21.15.76:0/2617815326 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662419f300 con 0x7f6624100090 2024-03-12T05:54:59.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f66217fa640 1 -- 172.21.15.76:0/2617815326 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661403e450 con 0x7f6624100090 2024-03-12T05:54:59.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.918+0000 7f662a793640 1 -- 172.21.15.76:0/2617815326 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f662419f860 con 0x7f6624100090 2024-03-12T05:54:59.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.919+0000 7f662a793640 1 -- 172.21.15.76:0/2617815326 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6624107430 con 0x7f6624100090 2024-03-12T05:54:59.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.920+0000 7f66217fa640 1 -- 172.21.15.76:0/2617815326 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f6614003680 con 0x7f6624100090 2024-03-12T05:54:59.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.921+0000 7f66217fa640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f660c0a1b40 0x7f660c0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:54:59.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.921+0000 7f66217fa640 1 -- 172.21.15.76:0/2617815326 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f66140eff90 con 0x7f6624100090 2024-03-12T05:54:59.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.921+0000 7f66237fe640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f660c0a1b40 0x7f660c0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:54:59.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.921+0000 7f66237fe640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f660c0a1b40 0x7f660c0a4000 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6618005fd0 tx=0x7f6618005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:54:59.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:54:59.924+0000 7f66217fa640 1 -- 172.21.15.76:0/2617815326 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f66140b9160 con 0x7f6624100090 2024-03-12T05:55:00.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.098+0000 7f662a793640 1 -- 172.21.15.76:0/2617815326 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f662402cda0 con 0x7f6624100090 2024-03-12T05:55:00.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.098+0000 7f66217fa640 1 -- 172.21.15.76:0/2617815326 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f66140bc820 con 0x7f6624100090 2024-03-12T05:55:00.102 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-12T05:55:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.100+0000 7f662a793640 1 -- 172.21.15.76:0/2617815326 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f660c0a1b40 msgr2=0x7f660c0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.100+0000 7f662a793640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f660c0a1b40 0x7f660c0a4000 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6618005fd0 tx=0x7f6618005c70 comp rx=0 tx=0).stop 2024-03-12T05:55:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.100+0000 7f662a793640 1 -- 172.21.15.76:0/2617815326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6624100090 msgr2=0x7f662406a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.100+0000 7f662a793640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6624100090 0x7f662406a3d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6614000c00 tx=0x7f6614004270 comp rx=0 tx=0).stop 2024-03-12T05:55:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.100+0000 7f662a793640 1 -- 172.21.15.76:0/2617815326 shutdown_connections 2024-03-12T05:55:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.100+0000 7f662a793640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f660c0a1b40 0x7f660c0a4000 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.100+0000 7f662a793640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6624101c40 0x7f662419e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.100+0000 7f662a793640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6624101280 0x7f662419bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.100+0000 7f662a793640 1 --2- 172.21.15.76:0/2617815326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6624100090 0x7f662406a3d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.100+0000 7f662a793640 1 -- 172.21.15.76:0/2617815326 >> 172.21.15.76:0/2617815326 conn(0x7f66240fb5e0 msgr2=0x7f6624104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.100+0000 7f662a793640 1 -- 172.21.15.76:0/2617815326 shutdown_connections 2024-03-12T05:55:00.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.100+0000 7f662a793640 1 -- 172.21.15.76:0/2617815326 wait complete. 2024-03-12T05:55:00.115 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:55:00.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.216+0000 7fbc5f464640 1 -- 172.21.15.76:0/80330782 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc58101c50 msgr2=0x7fbc581084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.216+0000 7fbc5f464640 1 --2- 172.21.15.76:0/80330782 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc58101c50 0x7fbc581084e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbc48009900 tx=0x7fbc4802bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:00.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.216+0000 7fbc5f464640 1 -- 172.21.15.76:0/80330782 shutdown_connections 2024-03-12T05:55:00.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.216+0000 7fbc5f464640 1 --2- 172.21.15.76:0/80330782 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc58101c50 0x7fbc581084e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.216+0000 7fbc5f464640 1 --2- 172.21.15.76:0/80330782 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc58101290 0x7fbc58101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.216+0000 7fbc5f464640 1 --2- 172.21.15.76:0/80330782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc581000a0 0x7fbc581004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.216+0000 7fbc5f464640 1 -- 172.21.15.76:0/80330782 >> 172.21.15.76:0/80330782 conn(0x7fbc580fb610 msgr2=0x7fbc580fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:00.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.216+0000 7fbc5f464640 1 -- 172.21.15.76:0/80330782 shutdown_connections 2024-03-12T05:55:00.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.216+0000 7fbc5f464640 1 -- 172.21.15.76:0/80330782 wait complete. 2024-03-12T05:55:00.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.216+0000 7fbc5f464640 1 Processor -- start 2024-03-12T05:55:00.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.216+0000 7fbc5f464640 1 -- start start 2024-03-12T05:55:00.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5f464640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc581000a0 0x7fbc581960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:00.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5f464640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc58101290 0x7fbc581965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:00.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5f464640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc58101c50 0x7fbc5819d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:00.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5f464640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5819dba0 con 0x7fbc58101c50 2024-03-12T05:55:00.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5f464640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5819dd10 con 0x7fbc58101290 2024-03-12T05:55:00.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5f464640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5819ded0 con 0x7fbc581000a0 2024-03-12T05:55:00.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5da01640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc58101c50 0x7fbc5819d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:00.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5da01640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc58101c50 0x7fbc5819d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58704/0 (socket says 172.21.15.76:58704) 2024-03-12T05:55:00.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5da01640 1 -- 172.21.15.76:0/3419695459 learned_addr learned my addr 172.21.15.76:0/3419695459 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:00.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5d200640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc581000a0 0x7fbc581960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:00.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5c9ff640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc58101290 0x7fbc581965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:00.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5da01640 1 -- 172.21.15.76:0/3419695459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc581000a0 msgr2=0x7fbc581960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5da01640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc581000a0 0x7fbc581960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5da01640 1 -- 172.21.15.76:0/3419695459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc58101290 msgr2=0x7fbc581965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5da01640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc58101290 0x7fbc581965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5da01640 1 -- 172.21.15.76:0/3419695459 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc48009560 con 0x7fbc58101c50 2024-03-12T05:55:00.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5c9ff640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc58101290 0x7fbc581965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:00.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5da01640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc58101c50 0x7fbc5819d660 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbc48009230 tx=0x7fbc48004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:00.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc567fc640 1 -- 172.21.15.76:0/3419695459 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc48004df0 con 0x7fbc58101c50 2024-03-12T05:55:00.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc567fc640 1 -- 172.21.15.76:0/3419695459 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc48005430 con 0x7fbc58101c50 2024-03-12T05:55:00.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc567fc640 1 -- 172.21.15.76:0/3419695459 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc48003de0 con 0x7fbc58101c50 2024-03-12T05:55:00.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.217+0000 7fbc5f464640 1 -- 172.21.15.76:0/3419695459 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5819e150 con 0x7fbc58101c50 2024-03-12T05:55:00.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.218+0000 7fbc5f464640 1 -- 172.21.15.76:0/3419695459 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5819e660 con 0x7fbc58101c50 2024-03-12T05:55:00.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.219+0000 7fbc5f464640 1 -- 172.21.15.76:0/3419695459 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc58107360 con 0x7fbc58101c50 2024-03-12T05:55:00.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.220+0000 7fbc567fc640 1 -- 172.21.15.76:0/3419695459 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fbc480055a0 con 0x7fbc58101c50 2024-03-12T05:55:00.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.220+0000 7fbc567fc640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbc400a1ac0 0x7fbc400a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:00.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.220+0000 7fbc5d200640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbc400a1ac0 0x7fbc400a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:00.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.220+0000 7fbc567fc640 1 -- 172.21.15.76:0/3419695459 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbc480ef860 con 0x7fbc58101c50 2024-03-12T05:55:00.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.221+0000 7fbc5d200640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbc400a1ac0 0x7fbc400a3f80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbc581010f0 tx=0x7fbc50009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:00.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.225+0000 7fbc567fc640 1 -- 172.21.15.76:0/3419695459 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbc480b8a30 con 0x7fbc58101c50 2024-03-12T05:55:00.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.396+0000 7fbc5f464640 1 -- 172.21.15.76:0/3419695459 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbc5802cda0 con 0x7fbc58101c50 2024-03-12T05:55:00.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.396+0000 7fbc567fc640 1 -- 172.21.15.76:0/3419695459 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbc480bc0f0 con 0x7fbc58101c50 2024-03-12T05:55:00.400 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-12T05:55:00.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.399+0000 7fbc5f464640 1 -- 172.21.15.76:0/3419695459 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbc400a1ac0 msgr2=0x7fbc400a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.399+0000 7fbc5f464640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbc400a1ac0 0x7fbc400a3f80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbc581010f0 tx=0x7fbc50009680 comp rx=0 tx=0).stop 2024-03-12T05:55:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.399+0000 7fbc5f464640 1 -- 172.21.15.76:0/3419695459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc58101c50 msgr2=0x7fbc5819d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.399+0000 7fbc5f464640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc58101c50 0x7fbc5819d660 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbc48009230 tx=0x7fbc48004270 comp rx=0 tx=0).stop 2024-03-12T05:55:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.399+0000 7fbc5f464640 1 -- 172.21.15.76:0/3419695459 shutdown_connections 2024-03-12T05:55:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.399+0000 7fbc5f464640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbc400a1ac0 0x7fbc400a3f80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.399+0000 7fbc5f464640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc58101c50 0x7fbc5819d660 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.399+0000 7fbc5f464640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc58101290 0x7fbc581965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.399+0000 7fbc5f464640 1 --2- 172.21.15.76:0/3419695459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc581000a0 0x7fbc581960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.399+0000 7fbc5f464640 1 -- 172.21.15.76:0/3419695459 >> 172.21.15.76:0/3419695459 conn(0x7fbc580fb610 msgr2=0x7fbc58104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.399+0000 7fbc5f464640 1 -- 172.21.15.76:0/3419695459 shutdown_connections 2024-03-12T05:55:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.399+0000 7fbc5f464640 1 -- 172.21.15.76:0/3419695459 wait complete. 2024-03-12T05:55:00.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:00 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/353510636' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:00.414 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:55:00.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:00 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/353510636' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:00 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/353510636' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:00.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.515+0000 7f3d2e93a640 1 -- 172.21.15.76:0/2055400241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28104540 msgr2=0x7f3d28106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.515+0000 7f3d2e93a640 1 --2- 172.21.15.76:0/2055400241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28104540 0x7f3d28106930 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009900 tx=0x7f3d1c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:00.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.515+0000 7f3d2e93a640 1 -- 172.21.15.76:0/2055400241 shutdown_connections 2024-03-12T05:55:00.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.515+0000 7f3d2e93a640 1 --2- 172.21.15.76:0/2055400241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d28106e70 0x7f3d281092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.515+0000 7f3d2e93a640 1 --2- 172.21.15.76:0/2055400241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28104540 0x7f3d28106930 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.515+0000 7f3d2e93a640 1 --2- 172.21.15.76:0/2055400241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d28101c10 0x7f3d28104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.515+0000 7f3d2e93a640 1 -- 172.21.15.76:0/2055400241 >> 172.21.15.76:0/2055400241 conn(0x7f3d280fb5e0 msgr2=0x7f3d280fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:00.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.515+0000 7f3d2e93a640 1 -- 172.21.15.76:0/2055400241 shutdown_connections 2024-03-12T05:55:00.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.515+0000 7f3d2e93a640 1 -- 172.21.15.76:0/2055400241 wait complete. 2024-03-12T05:55:00.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2e93a640 1 Processor -- start 2024-03-12T05:55:00.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2e93a640 1 -- start start 2024-03-12T05:55:00.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2e93a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d28101c10 0x7f3d28101580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:00.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2e93a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d28104540 0x7f3d280ffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:00.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2e93a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28106e70 0x7f3d28100100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:00.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2e93a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d28100860 con 0x7f3d28106e70 2024-03-12T05:55:00.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2e93a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d281009d0 con 0x7f3d28104540 2024-03-12T05:55:00.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2e93a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d281a1290 con 0x7f3d28101c10 2024-03-12T05:55:00.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2ced7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28106e70 0x7f3d28100100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:00.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2ced7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28106e70 0x7f3d28100100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58734/0 (socket says 172.21.15.76:58734) 2024-03-12T05:55:00.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2ced7640 1 -- 172.21.15.76:0/3864049036 learned_addr learned my addr 172.21.15.76:0/3864049036 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:00.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d27fff640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d28101c10 0x7f3d28101580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:00.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d277fe640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d28104540 0x7f3d280ffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:00.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2ced7640 1 -- 172.21.15.76:0/3864049036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d28101c10 msgr2=0x7f3d28101580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2ced7640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d28101c10 0x7f3d28101580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2ced7640 1 -- 172.21.15.76:0/3864049036 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d28104540 msgr2=0x7f3d280ffbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2ced7640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d28104540 0x7f3d280ffbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2ced7640 1 -- 172.21.15.76:0/3864049036 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c009560 con 0x7f3d28106e70 2024-03-12T05:55:00.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d277fe640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d28104540 0x7f3d280ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:00.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.516+0000 7f3d2ced7640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28106e70 0x7f3d28100100 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000cd10 tx=0x7f3d2000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:00.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.517+0000 7f3d257fa640 1 -- 172.21.15.76:0/3864049036 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2000f450 con 0x7f3d28106e70 2024-03-12T05:55:00.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.517+0000 7f3d257fa640 1 -- 172.21.15.76:0/3864049036 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d2000fa90 con 0x7f3d28106e70 2024-03-12T05:55:00.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.517+0000 7f3d257fa640 1 -- 172.21.15.76:0/3864049036 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20018b70 con 0x7f3d28106e70 2024-03-12T05:55:00.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.517+0000 7f3d2e93a640 1 -- 172.21.15.76:0/3864049036 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d281a1540 con 0x7f3d28106e70 2024-03-12T05:55:00.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.517+0000 7f3d2e93a640 1 -- 172.21.15.76:0/3864049036 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d281a1a50 con 0x7f3d28106e70 2024-03-12T05:55:00.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.518+0000 7f3d2e93a640 1 -- 172.21.15.76:0/3864049036 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d280fd0d0 con 0x7f3d28106e70 2024-03-12T05:55:00.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.519+0000 7f3d257fa640 1 -- 172.21.15.76:0/3864049036 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f3d2000fc00 con 0x7f3d28106e70 2024-03-12T05:55:00.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.519+0000 7f3d257fa640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3d100a1b40 0x7f3d100a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:00.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.519+0000 7f3d27fff640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3d100a1b40 0x7f3d100a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:00.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.519+0000 7f3d257fa640 1 -- 172.21.15.76:0/3864049036 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3d200cd930 con 0x7f3d28106e70 2024-03-12T05:55:00.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.519+0000 7f3d27fff640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3d100a1b40 0x7f3d100a4000 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800bbe0 tx=0x7f3d1800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:00.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.523+0000 7f3d257fa640 1 -- 172.21.15.76:0/3864049036 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3d200969f0 con 0x7f3d28106e70 2024-03-12T05:55:00.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.700+0000 7f3d2e93a640 1 -- 172.21.15.76:0/3864049036 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3d2802ce40 con 0x7f3d28106e70 2024-03-12T05:55:00.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.701+0000 7f3d257fa640 1 -- 172.21.15.76:0/3864049036 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3d2009a0b0 con 0x7f3d28106e70 2024-03-12T05:55:00.704 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-12T05:55:00.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.702+0000 7f3d2e93a640 1 -- 172.21.15.76:0/3864049036 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3d100a1b40 msgr2=0x7f3d100a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.702+0000 7f3d2e93a640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3d100a1b40 0x7f3d100a4000 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800bbe0 tx=0x7f3d1800b390 comp rx=0 tx=0).stop 2024-03-12T05:55:00.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.702+0000 7f3d2e93a640 1 -- 172.21.15.76:0/3864049036 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28106e70 msgr2=0x7f3d28100100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.702+0000 7f3d2e93a640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28106e70 0x7f3d28100100 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000cd10 tx=0x7f3d2000a620 comp rx=0 tx=0).stop 2024-03-12T05:55:00.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.703+0000 7f3d2e93a640 1 -- 172.21.15.76:0/3864049036 shutdown_connections 2024-03-12T05:55:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.703+0000 7f3d2e93a640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3d100a1b40 0x7f3d100a4000 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.703+0000 7f3d2e93a640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d28106e70 0x7f3d28100100 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.703+0000 7f3d2e93a640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d28104540 0x7f3d280ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.703+0000 7f3d2e93a640 1 --2- 172.21.15.76:0/3864049036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d28101c10 0x7f3d28101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.703+0000 7f3d2e93a640 1 -- 172.21.15.76:0/3864049036 >> 172.21.15.76:0/3864049036 conn(0x7f3d280fb5e0 msgr2=0x7f3d28102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.703+0000 7f3d2e93a640 1 -- 172.21.15.76:0/3864049036 shutdown_connections 2024-03-12T05:55:00.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.703+0000 7f3d2e93a640 1 -- 172.21.15.76:0/3864049036 wait complete. 2024-03-12T05:55:00.717 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:55:00.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.815+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/1479453500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbb4101c30 msgr2=0x7fbbb4104020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.815+0000 7fbbb9a0f640 1 --2- 172.21.15.76:0/1479453500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbb4101c30 0x7fbbb4104020 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbba4009900 tx=0x7fbba402bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:00.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.815+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/1479453500 shutdown_connections 2024-03-12T05:55:00.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.815+0000 7fbbb9a0f640 1 --2- 172.21.15.76:0/1479453500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb4106e90 0x7fbbb41092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.815+0000 7fbbb9a0f640 1 --2- 172.21.15.76:0/1479453500 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbb4104560 0x7fbbb4106950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.815+0000 7fbbb9a0f640 1 --2- 172.21.15.76:0/1479453500 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbb4101c30 0x7fbbb4104020 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.815+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/1479453500 >> 172.21.15.76:0/1479453500 conn(0x7fbbb40fb620 msgr2=0x7fbbb40fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:00.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.815+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/1479453500 shutdown_connections 2024-03-12T05:55:00.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.815+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/1479453500 wait complete. 2024-03-12T05:55:00.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.816+0000 7fbbb9a0f640 1 Processor -- start 2024-03-12T05:55:00.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.816+0000 7fbbb9a0f640 1 -- start start 2024-03-12T05:55:00.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.816+0000 7fbbb9a0f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbb4101c30 0x7fbbb4197210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:00.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.816+0000 7fbbb9a0f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb4104560 0x7fbbb4197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:00.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.816+0000 7fbbb9a0f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbb4106e90 0x7fbbb419e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:00.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.816+0000 7fbbb9a0f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb419ed00 con 0x7fbbb4106e90 2024-03-12T05:55:00.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.816+0000 7fbbb9a0f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb419ee70 con 0x7fbbb4101c30 2024-03-12T05:55:00.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.816+0000 7fbbb9a0f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb419f030 con 0x7fbbb4104560 2024-03-12T05:55:00.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.816+0000 7fbbb37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbb4106e90 0x7fbbb419e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:00.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.816+0000 7fbbb37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbb4106e90 0x7fbbb419e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58750/0 (socket says 172.21.15.76:58750) 2024-03-12T05:55:00.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.816+0000 7fbbb37fe640 1 -- 172.21.15.76:0/61974120 learned_addr learned my addr 172.21.15.76:0/61974120 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:00.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.817+0000 7fbbb37fe640 1 -- 172.21.15.76:0/61974120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb4104560 msgr2=0x7fbbb4197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.817+0000 7fbbb37fe640 1 --2- 172.21.15.76:0/61974120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb4104560 0x7fbbb4197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.817+0000 7fbbb37fe640 1 -- 172.21.15.76:0/61974120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbb4101c30 msgr2=0x7fbbb4197210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:00.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.817+0000 7fbbb37fe640 1 --2- 172.21.15.76:0/61974120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbb4101c30 0x7fbbb4197210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:00.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.817+0000 7fbbb37fe640 1 -- 172.21.15.76:0/61974120 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba4009560 con 0x7fbbb4106e90 2024-03-12T05:55:00.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.817+0000 7fbbb37fe640 1 --2- 172.21.15.76:0/61974120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbb4106e90 0x7fbbb419e7c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00cce0 tx=0x7fbbac00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:00.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.817+0000 7fbbb8a0d640 1 -- 172.21.15.76:0/61974120 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac00f450 con 0x7fbbb4106e90 2024-03-12T05:55:00.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.817+0000 7fbbb8a0d640 1 -- 172.21.15.76:0/61974120 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbac00fa90 con 0x7fbbb4106e90 2024-03-12T05:55:00.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.817+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/61974120 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb419f310 con 0x7fbbb4106e90 2024-03-12T05:55:00.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.817+0000 7fbbb8a0d640 1 -- 172.21.15.76:0/61974120 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac018b90 con 0x7fbbb4106e90 2024-03-12T05:55:00.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.817+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/61974120 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb419f820 con 0x7fbbb4106e90 2024-03-12T05:55:00.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.818+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/61974120 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb40fd110 con 0x7fbbb4106e90 2024-03-12T05:55:00.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.819+0000 7fbbb8a0d640 1 -- 172.21.15.76:0/61974120 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fbbac00fc00 con 0x7fbbb4106e90 2024-03-12T05:55:00.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.819+0000 7fbbb8a0d640 1 --2- 172.21.15.76:0/61974120 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbb9c0a1b40 0x7fbb9c0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:00.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.820+0000 7fbbb2ffd640 1 --2- 172.21.15.76:0/61974120 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbb9c0a1b40 0x7fbb9c0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:00.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.820+0000 7fbbb8a0d640 1 -- 172.21.15.76:0/61974120 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbbac0cdd40 con 0x7fbbb4106e90 2024-03-12T05:55:00.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.820+0000 7fbbb2ffd640 1 --2- 172.21.15.76:0/61974120 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbb9c0a1b40 0x7fbb9c0a4000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbba402c6a0 tx=0x7fbba4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:00.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:00.824+0000 7fbbb8a0d640 1 -- 172.21.15.76:0/61974120 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbbac096e00 con 0x7fbbb4106e90 2024-03-12T05:55:01.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.034+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/61974120 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbb410cbc0 con 0x7fbbb4106e90 2024-03-12T05:55:01.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.035+0000 7fbbb8a0d640 1 -- 172.21.15.76:0/61974120 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+11328 (secure 0 0 0) 0x7fbbac09a4c0 con 0x7fbbb4106e90 2024-03-12T05:55:01.038 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:01.038 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:53:42.384114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:55:01.039 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 21 2024-03-12T05:55:01.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.037+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/61974120 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbb9c0a1b40 msgr2=0x7fbb9c0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.037+0000 7fbbb9a0f640 1 --2- 172.21.15.76:0/61974120 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbb9c0a1b40 0x7fbb9c0a4000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbba402c6a0 tx=0x7fbba4000bc0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.037+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/61974120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbb4106e90 msgr2=0x7fbbb419e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.037+0000 7fbbb9a0f640 1 --2- 172.21.15.76:0/61974120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbb4106e90 0x7fbbb419e7c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00cce0 tx=0x7fbbac00a620 comp rx=0 tx=0).stop 2024-03-12T05:55:01.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.037+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/61974120 shutdown_connections 2024-03-12T05:55:01.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.037+0000 7fbbb9a0f640 1 --2- 172.21.15.76:0/61974120 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbb9c0a1b40 0x7fbb9c0a4000 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.037+0000 7fbbb9a0f640 1 --2- 172.21.15.76:0/61974120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbb4106e90 0x7fbbb419e7c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.037+0000 7fbbb9a0f640 1 --2- 172.21.15.76:0/61974120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb4104560 0x7fbbb4197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.037+0000 7fbbb9a0f640 1 --2- 172.21.15.76:0/61974120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbb4101c30 0x7fbbb4197210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.037+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/61974120 >> 172.21.15.76:0/61974120 conn(0x7fbbb40fb620 msgr2=0x7fbbb4102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:01.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.037+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/61974120 shutdown_connections 2024-03-12T05:55:01.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.037+0000 7fbbb9a0f640 1 -- 172.21.15.76:0/61974120 wait complete. 2024-03-12T05:55:01.052 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:55:01.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.150+0000 7fa5e5288640 1 -- 172.21.15.76:0/1881429724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa5e0102dc0 msgr2=0x7fa5e0109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.150+0000 7fa5e5288640 1 --2- 172.21.15.76:0/1881429724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa5e0102dc0 0x7fa5e0109650 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8009ca0 tx=0x7fa5d802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.150+0000 7fa5e5288640 1 -- 172.21.15.76:0/1881429724 shutdown_connections 2024-03-12T05:55:01.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.150+0000 7fa5e5288640 1 --2- 172.21.15.76:0/1881429724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa5e0102dc0 0x7fa5e0109650 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.150+0000 7fa5e5288640 1 --2- 172.21.15.76:0/1881429724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5e0102400 0x7fa5e0102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.151+0000 7fa5e5288640 1 --2- 172.21.15.76:0/1881429724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5e0101210 0x7fa5e0101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.151+0000 7fa5e5288640 1 -- 172.21.15.76:0/1881429724 >> 172.21.15.76:0/1881429724 conn(0x7fa5e00fc7a0 msgr2=0x7fa5e00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:01.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.151+0000 7fa5e5288640 1 -- 172.21.15.76:0/1881429724 shutdown_connections 2024-03-12T05:55:01.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.151+0000 7fa5e5288640 1 -- 172.21.15.76:0/1881429724 wait complete. 2024-03-12T05:55:01.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.151+0000 7fa5e5288640 1 Processor -- start 2024-03-12T05:55:01.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.151+0000 7fa5e5288640 1 -- start start 2024-03-12T05:55:01.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.151+0000 7fa5e5288640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa5e0101210 0x7fa5e0197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:01.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.151+0000 7fa5e5288640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5e0102400 0x7fa5e0197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:01.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.151+0000 7fa5e5288640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5e0102dc0 0x7fa5e019e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:01.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.151+0000 7fa5e5288640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e019ecf0 con 0x7fa5e0101210 2024-03-12T05:55:01.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.151+0000 7fa5e5288640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e019ee60 con 0x7fa5e0102dc0 2024-03-12T05:55:01.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.151+0000 7fa5e5288640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e019f020 con 0x7fa5e0102400 2024-03-12T05:55:01.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5deffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa5e0101210 0x7fa5e0197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:01.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5deffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa5e0101210 0x7fa5e0197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58770/0 (socket says 172.21.15.76:58770) 2024-03-12T05:55:01.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5deffd640 1 -- 172.21.15.76:0/2952973704 learned_addr learned my addr 172.21.15.76:0/2952973704 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:01.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5de7fc640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5e0102400 0x7fa5e0197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:01.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5deffd640 1 -- 172.21.15.76:0/2952973704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5e0102400 msgr2=0x7fa5e0197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5deffd640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5e0102400 0x7fa5e0197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5deffd640 1 -- 172.21.15.76:0/2952973704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5e0102dc0 msgr2=0x7fa5e019e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5df7fe640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5e0102dc0 0x7fa5e019e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:01.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5deffd640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5e0102dc0 0x7fa5e019e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5deffd640 1 -- 172.21.15.76:0/2952973704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d8009900 con 0x7fa5e0101210 2024-03-12T05:55:01.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5de7fc640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5e0102400 0x7fa5e0197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:01.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5deffd640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa5e0101210 0x7fa5e0197200 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa5d000d770 tx=0x7fa5d000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:01.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5d7fff640 1 -- 172.21.15.76:0/2952973704 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0009c30 con 0x7fa5e0101210 2024-03-12T05:55:01.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5e5288640 1 -- 172.21.15.76:0/2952973704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e019f300 con 0x7fa5e0101210 2024-03-12T05:55:01.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5d7fff640 1 -- 172.21.15.76:0/2952973704 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5d000de80 con 0x7fa5e0101210 2024-03-12T05:55:01.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5d7fff640 1 -- 172.21.15.76:0/2952973704 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0010460 con 0x7fa5e0101210 2024-03-12T05:55:01.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.152+0000 7fa5e5288640 1 -- 172.21.15.76:0/2952973704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e019f860 con 0x7fa5e0101210 2024-03-12T05:55:01.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.153+0000 7fa5e5288640 1 -- 172.21.15.76:0/2952973704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e01084d0 con 0x7fa5e0101210 2024-03-12T05:55:01.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.154+0000 7fa5d7fff640 1 -- 172.21.15.76:0/2952973704 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fa5d0003f60 con 0x7fa5e0101210 2024-03-12T05:55:01.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.155+0000 7fa5d7fff640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa5cc0a1ac0 0x7fa5cc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:01.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.155+0000 7fa5de7fc640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa5cc0a1ac0 0x7fa5cc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:01.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.155+0000 7fa5d7fff640 1 -- 172.21.15.76:0/2952973704 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa5d00cd340 con 0x7fa5e0101210 2024-03-12T05:55:01.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.155+0000 7fa5de7fc640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa5cc0a1ac0 0x7fa5cc0a3f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80058c0 tx=0x7fa5c8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:01.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.159+0000 7fa5d7fff640 1 -- 172.21.15.76:0/2952973704 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa5d0096400 con 0x7fa5e0101210 2024-03-12T05:55:01.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.334+0000 7fa5e5288640 1 -- 172.21.15.76:0/2952973704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5e010cbb0 con 0x7fa5e0101210 2024-03-12T05:55:01.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.336+0000 7fa5d7fff640 1 -- 172.21.15.76:0/2952973704 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22541 (secure 0 0 0) 0x7fa5d0099ac0 con 0x7fa5e0101210 2024-03-12T05:55:01.340 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:01.340 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:54:42.007701+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:55:01.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.338+0000 7fa5e5288640 1 -- 172.21.15.76:0/2952973704 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa5cc0a1ac0 msgr2=0x7fa5cc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.338+0000 7fa5e5288640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa5cc0a1ac0 0x7fa5cc0a3f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80058c0 tx=0x7fa5c8005850 comp rx=0 tx=0).stop 2024-03-12T05:55:01.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.338+0000 7fa5e5288640 1 -- 172.21.15.76:0/2952973704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa5e0101210 msgr2=0x7fa5e0197200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.338+0000 7fa5e5288640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa5e0101210 0x7fa5e0197200 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa5d000d770 tx=0x7fa5d000dc30 comp rx=0 tx=0).stop 2024-03-12T05:55:01.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.338+0000 7fa5e5288640 1 -- 172.21.15.76:0/2952973704 shutdown_connections 2024-03-12T05:55:01.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.338+0000 7fa5e5288640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa5cc0a1ac0 0x7fa5cc0a3f80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.338+0000 7fa5e5288640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa5e0102dc0 0x7fa5e019e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.338+0000 7fa5e5288640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5e0102400 0x7fa5e0197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.338+0000 7fa5e5288640 1 --2- 172.21.15.76:0/2952973704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa5e0101210 0x7fa5e0197200 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.338+0000 7fa5e5288640 1 -- 172.21.15.76:0/2952973704 >> 172.21.15.76:0/2952973704 conn(0x7fa5e00fc7a0 msgr2=0x7fa5e01056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:01.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.338+0000 7fa5e5288640 1 -- 172.21.15.76:0/2952973704 shutdown_connections 2024-03-12T05:55:01.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.338+0000 7fa5e5288640 1 -- 172.21.15.76:0/2952973704 wait complete. 2024-03-12T05:55:01.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:01 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2617815326' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:01.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:01 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3419695459' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:01.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:01 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3864049036' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:01.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:01 smithi076 ceph-mon[32095]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:55:01.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:01 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/61974120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:01.353 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:49:09.125536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:53:35.663129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:53:36.740826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:55:01.354 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:55:01.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:01 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2617815326' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:01.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:01 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3419695459' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:01.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:01 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3864049036' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:01.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:01 smithi082 ceph-mon[36382]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:55:01.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:01 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/61974120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:01.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.455+0000 7ff1a333a640 1 -- 172.21.15.76:0/2041496517 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff19c101290 msgr2=0x7ff19c101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.455+0000 7ff1a333a640 1 --2- 172.21.15.76:0/2041496517 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff19c101290 0x7ff19c101710 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff18c009900 tx=0x7ff18c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:01.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.455+0000 7ff1a333a640 1 -- 172.21.15.76:0/2041496517 shutdown_connections 2024-03-12T05:55:01.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.455+0000 7ff1a333a640 1 --2- 172.21.15.76:0/2041496517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff19c101c50 0x7ff19c1084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.455+0000 7ff1a333a640 1 --2- 172.21.15.76:0/2041496517 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff19c101290 0x7ff19c101710 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.455+0000 7ff1a333a640 1 --2- 172.21.15.76:0/2041496517 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff19c1000a0 0x7ff19c1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.455+0000 7ff1a333a640 1 -- 172.21.15.76:0/2041496517 >> 172.21.15.76:0/2041496517 conn(0x7ff19c0fb610 msgr2=0x7ff19c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:01.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.455+0000 7ff1a333a640 1 -- 172.21.15.76:0/2041496517 shutdown_connections 2024-03-12T05:55:01.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.455+0000 7ff1a333a640 1 -- 172.21.15.76:0/2041496517 wait complete. 2024-03-12T05:55:01.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a333a640 1 Processor -- start 2024-03-12T05:55:01.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a333a640 1 -- start start 2024-03-12T05:55:01.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a333a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff19c1000a0 0x7ff19c068fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:01.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a333a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff19c101290 0x7ff19c19c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:01.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a333a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff19c101c50 0x7ff19c19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:01.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a333a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c19ee20 con 0x7ff19c1000a0 2024-03-12T05:55:01.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a333a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c19ef90 con 0x7ff19c101290 2024-03-12T05:55:01.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a333a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c19f150 con 0x7ff19c101c50 2024-03-12T05:55:01.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a10d6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff19c1000a0 0x7ff19c068fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:01.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a10d6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff19c1000a0 0x7ff19c068fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60246/0 (socket says 172.21.15.76:60246) 2024-03-12T05:55:01.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a10d6640 1 -- 172.21.15.76:0/2525794224 learned_addr learned my addr 172.21.15.76:0/2525794224 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:01.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a18d7640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff19c101c50 0x7ff19c19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:01.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a08d5640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff19c101290 0x7ff19c19c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:01.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a10d6640 1 -- 172.21.15.76:0/2525794224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff19c101c50 msgr2=0x7ff19c19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a10d6640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff19c101c50 0x7ff19c19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a10d6640 1 -- 172.21.15.76:0/2525794224 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff19c101290 msgr2=0x7ff19c19c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a10d6640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff19c101290 0x7ff19c19c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a10d6640 1 -- 172.21.15.76:0/2525794224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c009560 con 0x7ff19c1000a0 2024-03-12T05:55:01.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.456+0000 7ff1a08d5640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff19c101290 0x7ff19c19c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:01.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.457+0000 7ff1a10d6640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff19c1000a0 0x7ff19c068fc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff19400b450 tx=0x7ff19400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:01.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.457+0000 7ff1927fc640 1 -- 172.21.15.76:0/2525794224 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff194011890 con 0x7ff19c1000a0 2024-03-12T05:55:01.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.457+0000 7ff1927fc640 1 -- 172.21.15.76:0/2525794224 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1940103e0 con 0x7ff19c1000a0 2024-03-12T05:55:01.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.457+0000 7ff1a333a640 1 -- 172.21.15.76:0/2525794224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19c19f3d0 con 0x7ff19c1000a0 2024-03-12T05:55:01.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.457+0000 7ff1927fc640 1 -- 172.21.15.76:0/2525794224 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19400f580 con 0x7ff19c1000a0 2024-03-12T05:55:01.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.457+0000 7ff1a333a640 1 -- 172.21.15.76:0/2525794224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19c19f9b0 con 0x7ff19c1000a0 2024-03-12T05:55:01.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.458+0000 7ff1a333a640 1 -- 172.21.15.76:0/2525794224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19c107410 con 0x7ff19c1000a0 2024-03-12T05:55:01.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.459+0000 7ff1927fc640 1 -- 172.21.15.76:0/2525794224 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7ff1940108e0 con 0x7ff19c1000a0 2024-03-12T05:55:01.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.459+0000 7ff1927fc640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff1880a1b40 0x7ff1880a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:01.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.460+0000 7ff1a08d5640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff1880a1b40 0x7ff1880a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:01.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.460+0000 7ff1927fc640 1 -- 172.21.15.76:0/2525794224 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff1940cd240 con 0x7ff19c1000a0 2024-03-12T05:55:01.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.460+0000 7ff1a08d5640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff1880a1b40 0x7ff1880a4000 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff18c036040 tx=0x7ff18c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:01.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.464+0000 7ff1927fc640 1 -- 172.21.15.76:0/2525794224 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff194096410 con 0x7ff19c1000a0 2024-03-12T05:55:01.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:01 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2617815326' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:01.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:01 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3419695459' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:01.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:01 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3864049036' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:01.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:01 smithi088 ceph-mon[36585]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:55:01.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:01 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/61974120' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:01.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.636+0000 7ff1a333a640 1 -- 172.21.15.76:0/2525794224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff19c10bcc0 con 0x7ff19c1000a0 2024-03-12T05:55:01.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.637+0000 7ff1927fc640 1 -- 172.21.15.76:0/2525794224 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff194099ad0 con 0x7ff19c1000a0 2024-03-12T05:55:01.640 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-12T05:55:01.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.638+0000 7ff1a333a640 1 -- 172.21.15.76:0/2525794224 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff1880a1b40 msgr2=0x7ff1880a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.638+0000 7ff1a333a640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff1880a1b40 0x7ff1880a4000 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff18c036040 tx=0x7ff18c000bc0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.638+0000 7ff1a333a640 1 -- 172.21.15.76:0/2525794224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff19c1000a0 msgr2=0x7ff19c068fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.638+0000 7ff1a333a640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff19c1000a0 0x7ff19c068fc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff19400b450 tx=0x7ff19400b910 comp rx=0 tx=0).stop 2024-03-12T05:55:01.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.639+0000 7ff1a333a640 1 -- 172.21.15.76:0/2525794224 shutdown_connections 2024-03-12T05:55:01.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.639+0000 7ff1a333a640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff1880a1b40 0x7ff1880a4000 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.639+0000 7ff1a333a640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff19c101c50 0x7ff19c19e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.639+0000 7ff1a333a640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff19c101290 0x7ff19c19c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.639+0000 7ff1a333a640 1 --2- 172.21.15.76:0/2525794224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff19c1000a0 0x7ff19c068fc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.639+0000 7ff1a333a640 1 -- 172.21.15.76:0/2525794224 >> 172.21.15.76:0/2525794224 conn(0x7ff19c0fb610 msgr2=0x7ff19c1045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:01.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.639+0000 7ff1a333a640 1 -- 172.21.15.76:0/2525794224 shutdown_connections 2024-03-12T05:55:01.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.639+0000 7ff1a333a640 1 -- 172.21.15.76:0/2525794224 wait complete. 2024-03-12T05:55:01.654 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:55:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.755+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/2454898188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b94102380 msgr2=0x7f2b94102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.755+0000 7f2b9a0f4640 1 --2- 172.21.15.76:0/2454898188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b94102380 0x7f2b94102800 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009900 tx=0x7f2b8802bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.755+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/2454898188 shutdown_connections 2024-03-12T05:55:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.755+0000 7f2b9a0f4640 1 --2- 172.21.15.76:0/2454898188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b94102d40 0x7f2b941095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.755+0000 7f2b9a0f4640 1 --2- 172.21.15.76:0/2454898188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b94102380 0x7f2b94102800 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.755+0000 7f2b9a0f4640 1 --2- 172.21.15.76:0/2454898188 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b94101190 0x7f2b941015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.755+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/2454898188 >> 172.21.15.76:0/2454898188 conn(0x7f2b940fc700 msgr2=0x7f2b940feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.755+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/2454898188 shutdown_connections 2024-03-12T05:55:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.755+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/2454898188 wait complete. 2024-03-12T05:55:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b9a0f4640 1 Processor -- start 2024-03-12T05:55:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b9a0f4640 1 -- start start 2024-03-12T05:55:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b9a0f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b94101190 0x7f2b941973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b9a0f4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b94102380 0x7f2b94197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b9a0f4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b94102d40 0x7f2b9419e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:01.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b9a0f4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9419eee0 con 0x7f2b94102380 2024-03-12T05:55:01.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b9a0f4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9419f050 con 0x7f2b94102d40 2024-03-12T05:55:01.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b9a0f4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9419f210 con 0x7f2b94101190 2024-03-12T05:55:01.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b92ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b94102380 0x7f2b94197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:01.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b92ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b94102380 0x7f2b94197930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60254/0 (socket says 172.21.15.76:60254) 2024-03-12T05:55:01.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b92ffd640 1 -- 172.21.15.76:0/3696396489 learned_addr learned my addr 172.21.15.76:0/3696396489 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:01.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b937fe640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b94101190 0x7f2b941973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:01.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b93fff640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b94102d40 0x7f2b9419e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:01.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b92ffd640 1 -- 172.21.15.76:0/3696396489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b94101190 msgr2=0x7f2b941973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b92ffd640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b94101190 0x7f2b941973f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b92ffd640 1 -- 172.21.15.76:0/3696396489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b94102d40 msgr2=0x7f2b9419e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b92ffd640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b94102d40 0x7f2b9419e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b92ffd640 1 -- 172.21.15.76:0/3696396489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b88009560 con 0x7f2b94102380 2024-03-12T05:55:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.756+0000 7f2b93fff640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b94102d40 0x7f2b9419e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.757+0000 7f2b92ffd640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b94102380 0x7f2b94197930 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2b88005ab0 tx=0x7f2b880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.757+0000 7f2b90ff9640 1 -- 172.21.15.76:0/3696396489 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b880044e0 con 0x7f2b94102380 2024-03-12T05:55:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.757+0000 7f2b90ff9640 1 -- 172.21.15.76:0/3696396489 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b880054e0 con 0x7f2b94102380 2024-03-12T05:55:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.757+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/3696396489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9419f490 con 0x7f2b94102380 2024-03-12T05:55:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.757+0000 7f2b90ff9640 1 -- 172.21.15.76:0/3696396489 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b8803e450 con 0x7f2b94102380 2024-03-12T05:55:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.757+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/3696396489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9419fa70 con 0x7f2b94102380 2024-03-12T05:55:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.758+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/3696396489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b941085d0 con 0x7f2b94102380 2024-03-12T05:55:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.759+0000 7f2b90ff9640 1 -- 172.21.15.76:0/3696396489 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f2b880035f0 con 0x7f2b94102380 2024-03-12T05:55:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.759+0000 7f2b90ff9640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2b7c0a1b40 0x7f2b7c0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.759+0000 7f2b937fe640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2b7c0a1b40 0x7f2b7c0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.759+0000 7f2b90ff9640 1 -- 172.21.15.76:0/3696396489 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2b880ef9e0 con 0x7f2b94102380 2024-03-12T05:55:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.760+0000 7f2b937fe640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2b7c0a1b40 0x7f2b7c0a4000 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2b941021e0 tx=0x7f2b84008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:01.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.764+0000 7f2b90ff9640 1 -- 172.21.15.76:0/3696396489 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2b880b8bb0 con 0x7f2b94102380 2024-03-12T05:55:01.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.936+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/3696396489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2b9410ced0 con 0x7f2b94102380 2024-03-12T05:55:01.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.936+0000 7f2b90ff9640 1 -- 172.21.15.76:0/3696396489 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2b880bc270 con 0x7f2b94102380 2024-03-12T05:55:01.940 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-12T05:55:01.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.938+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/3696396489 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2b7c0a1b40 msgr2=0x7f2b7c0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.938+0000 7f2b9a0f4640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2b7c0a1b40 0x7f2b7c0a4000 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2b941021e0 tx=0x7f2b84008040 comp rx=0 tx=0).stop 2024-03-12T05:55:01.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.938+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/3696396489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b94102380 msgr2=0x7f2b94197930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:01.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.938+0000 7f2b9a0f4640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b94102380 0x7f2b94197930 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2b88005ab0 tx=0x7f2b880042f0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.938+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/3696396489 shutdown_connections 2024-03-12T05:55:01.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.938+0000 7f2b9a0f4640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2b7c0a1b40 0x7f2b7c0a4000 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.938+0000 7f2b9a0f4640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b94102d40 0x7f2b9419e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.938+0000 7f2b9a0f4640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b94102380 0x7f2b94197930 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.938+0000 7f2b9a0f4640 1 --2- 172.21.15.76:0/3696396489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b94101190 0x7f2b941973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:01.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.938+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/3696396489 >> 172.21.15.76:0/3696396489 conn(0x7f2b940fc700 msgr2=0x7f2b941057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:01.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.938+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/3696396489 shutdown_connections 2024-03-12T05:55:01.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:01.938+0000 7f2b9a0f4640 1 -- 172.21.15.76:0/3696396489 wait complete. 2024-03-12T05:55:01.953 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:55:02.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.054+0000 7f44011e2640 1 -- 172.21.15.76:0/3375668043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43fc101290 msgr2=0x7f43fc101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.054+0000 7f44011e2640 1 --2- 172.21.15.76:0/3375668043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43fc101290 0x7f43fc101710 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f43ec009900 tx=0x7f43ec02bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:02.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.054+0000 7f44011e2640 1 -- 172.21.15.76:0/3375668043 shutdown_connections 2024-03-12T05:55:02.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.054+0000 7f44011e2640 1 --2- 172.21.15.76:0/3375668043 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43fc101c50 0x7f43fc1084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.054+0000 7f44011e2640 1 --2- 172.21.15.76:0/3375668043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43fc101290 0x7f43fc101710 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.054+0000 7f44011e2640 1 --2- 172.21.15.76:0/3375668043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43fc1000a0 0x7f43fc1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.054+0000 7f44011e2640 1 -- 172.21.15.76:0/3375668043 >> 172.21.15.76:0/3375668043 conn(0x7f43fc0fb610 msgr2=0x7f43fc0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:02.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.054+0000 7f44011e2640 1 -- 172.21.15.76:0/3375668043 shutdown_connections 2024-03-12T05:55:02.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.054+0000 7f44011e2640 1 -- 172.21.15.76:0/3375668043 wait complete. 2024-03-12T05:55:02.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.055+0000 7f44011e2640 1 Processor -- start 2024-03-12T05:55:02.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.055+0000 7f44011e2640 1 -- start start 2024-03-12T05:55:02.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.055+0000 7f44011e2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43fc1000a0 0x7f43fc1961d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.055+0000 7f44011e2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43fc101290 0x7f43fc196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.055+0000 7f44011e2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43fc101c50 0x7f43fc19d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.055+0000 7f44011e2640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43fc19dcc0 con 0x7f43fc101290 2024-03-12T05:55:02.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.055+0000 7f44011e2640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43fc19de30 con 0x7f43fc101c50 2024-03-12T05:55:02.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.055+0000 7f44011e2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43fc19dff0 con 0x7f43fc1000a0 2024-03-12T05:55:02.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.055+0000 7f43fa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43fc101290 0x7f43fc196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:02.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.055+0000 7f43fa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43fc101290 0x7f43fc196710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60272/0 (socket says 172.21.15.76:60272) 2024-03-12T05:55:02.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.055+0000 7f43fa59c640 1 -- 172.21.15.76:0/3599220751 learned_addr learned my addr 172.21.15.76:0/3599220751 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:02.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f43fb59e640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43fc101c50 0x7f43fc19d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:02.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f43fa59c640 1 -- 172.21.15.76:0/3599220751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43fc1000a0 msgr2=0x7f43fc1961d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f43fad9d640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43fc1000a0 0x7f43fc1961d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:02.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f43fa59c640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43fc1000a0 0x7f43fc1961d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f43fa59c640 1 -- 172.21.15.76:0/3599220751 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43fc101c50 msgr2=0x7f43fc19d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f43fa59c640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43fc101c50 0x7f43fc19d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f43fa59c640 1 -- 172.21.15.76:0/3599220751 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ec009560 con 0x7f43fc101290 2024-03-12T05:55:02.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f43fb59e640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43fc101c50 0x7f43fc19d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:02.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f43fad9d640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43fc1000a0 0x7f43fc1961d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:02.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f43fa59c640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43fc101290 0x7f43fc196710 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f43ec006010 tx=0x7f43ec0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:02.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f43e3fff640 1 -- 172.21.15.76:0/3599220751 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ec0044e0 con 0x7f43fc101290 2024-03-12T05:55:02.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f43e3fff640 1 -- 172.21.15.76:0/3599220751 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43ec0054e0 con 0x7f43fc101290 2024-03-12T05:55:02.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f44011e2640 1 -- 172.21.15.76:0/3599220751 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43fc19e270 con 0x7f43fc101290 2024-03-12T05:55:02.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f43e3fff640 1 -- 172.21.15.76:0/3599220751 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ec03e450 con 0x7f43fc101290 2024-03-12T05:55:02.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.056+0000 7f44011e2640 1 -- 172.21.15.76:0/3599220751 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43fc19e800 con 0x7f43fc101290 2024-03-12T05:55:02.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.057+0000 7f44011e2640 1 -- 172.21.15.76:0/3599220751 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43fc107410 con 0x7f43fc101290 2024-03-12T05:55:02.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.058+0000 7f43e3fff640 1 -- 172.21.15.76:0/3599220751 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f43ec0035f0 con 0x7f43fc101290 2024-03-12T05:55:02.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.059+0000 7f43e3fff640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f43e40a1ac0 0x7f43e40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.059+0000 7f43fad9d640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f43e40a1ac0 0x7f43e40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:02.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.059+0000 7f43e3fff640 1 -- 172.21.15.76:0/3599220751 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f43ec0ef8d0 con 0x7f43fc101290 2024-03-12T05:55:02.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.059+0000 7f43fad9d640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f43e40a1ac0 0x7f43e40a3f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f43f0006fd0 tx=0x7f43f0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:02.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.063+0000 7f43e3fff640 1 -- 172.21.15.76:0/3599220751 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f43ec0b8aa0 con 0x7f43fc101290 2024-03-12T05:55:02.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:02 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2952973704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:02.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:02 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2525794224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:02.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:02 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3696396489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:02.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.236+0000 7f44011e2640 1 -- 172.21.15.76:0/3599220751 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f43fc10bd60 con 0x7f43fc101290 2024-03-12T05:55:02.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.236+0000 7f43e3fff640 1 -- 172.21.15.76:0/3599220751 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f43ec0bc160 con 0x7f43fc101290 2024-03-12T05:55:02.240 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-12T05:55:02.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.238+0000 7f44011e2640 1 -- 172.21.15.76:0/3599220751 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f43e40a1ac0 msgr2=0x7f43e40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.238+0000 7f44011e2640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f43e40a1ac0 0x7f43e40a3f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f43f0006fd0 tx=0x7f43f0008040 comp rx=0 tx=0).stop 2024-03-12T05:55:02.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.238+0000 7f44011e2640 1 -- 172.21.15.76:0/3599220751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43fc101290 msgr2=0x7f43fc196710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.238+0000 7f44011e2640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43fc101290 0x7f43fc196710 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f43ec006010 tx=0x7f43ec0042f0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.238+0000 7f44011e2640 1 -- 172.21.15.76:0/3599220751 shutdown_connections 2024-03-12T05:55:02.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.238+0000 7f44011e2640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f43e40a1ac0 0x7f43e40a3f80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.238+0000 7f44011e2640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43fc101c50 0x7f43fc19d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.238+0000 7f44011e2640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43fc101290 0x7f43fc196710 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.238+0000 7f44011e2640 1 --2- 172.21.15.76:0/3599220751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43fc1000a0 0x7f43fc1961d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.238+0000 7f44011e2640 1 -- 172.21.15.76:0/3599220751 >> 172.21.15.76:0/3599220751 conn(0x7f43fc0fb610 msgr2=0x7f43fc1045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:02.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.238+0000 7f44011e2640 1 -- 172.21.15.76:0/3599220751 shutdown_connections 2024-03-12T05:55:02.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.238+0000 7f44011e2640 1 -- 172.21.15.76:0/3599220751 wait complete. 2024-03-12T05:55:02.253 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:55:02.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.350+0000 7fafa7c81640 1 -- 172.21.15.76:0/387932535 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa01000b0 msgr2=0x7fafa01004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.350+0000 7fafa7c81640 1 --2- 172.21.15.76:0/387932535 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa01000b0 0x7fafa01004d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7faf90009900 tx=0x7faf9002bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:02.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 -- 172.21.15.76:0/387932535 shutdown_connections 2024-03-12T05:55:02.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 --2- 172.21.15.76:0/387932535 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fafa0101c60 0x7fafa01084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 --2- 172.21.15.76:0/387932535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa01012a0 0x7fafa0101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 --2- 172.21.15.76:0/387932535 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa01000b0 0x7fafa01004d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 -- 172.21.15.76:0/387932535 >> 172.21.15.76:0/387932535 conn(0x7fafa00fb620 msgr2=0x7fafa00fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 -- 172.21.15.76:0/387932535 shutdown_connections 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 -- 172.21.15.76:0/387932535 wait complete. 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 Processor -- start 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 -- start start 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa01000b0 0x7fafa0196180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa01012a0 0x7fafa01966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fafa0101c60 0x7fafa019d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa019dc20 con 0x7fafa01012a0 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa019dd90 con 0x7fafa0101c60 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa7c81640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa019df50 con 0x7fafa01000b0 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa521c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa01012a0 0x7fafa01966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa521c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa01012a0 0x7fafa01966c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60302/0 (socket says 172.21.15.76:60302) 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.351+0000 7fafa521c640 1 -- 172.21.15.76:0/3312617622 learned_addr learned my addr 172.21.15.76:0/3312617622 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:02.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.352+0000 7fafa5a1d640 1 --2- 172.21.15.76:0/3312617622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa01000b0 0x7fafa0196180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:02.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.352+0000 7fafa521c640 1 -- 172.21.15.76:0/3312617622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa01000b0 msgr2=0x7fafa0196180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.352+0000 7fafa521c640 1 --2- 172.21.15.76:0/3312617622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa01000b0 0x7fafa0196180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.352+0000 7fafa521c640 1 -- 172.21.15.76:0/3312617622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fafa0101c60 msgr2=0x7fafa019d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.352+0000 7fafa521c640 1 --2- 172.21.15.76:0/3312617622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fafa0101c60 0x7fafa019d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.352+0000 7fafa521c640 1 -- 172.21.15.76:0/3312617622 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf90009560 con 0x7fafa01012a0 2024-03-12T05:55:02.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.352+0000 7fafa521c640 1 --2- 172.21.15.76:0/3312617622 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa01012a0 0x7fafa01966c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00d950 tx=0x7faf9c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:02.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.352+0000 7faf96ffd640 1 -- 172.21.15.76:0/3312617622 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9c009c30 con 0x7fafa01012a0 2024-03-12T05:55:02.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.352+0000 7fafa7c81640 1 -- 172.21.15.76:0/3312617622 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa019e230 con 0x7fafa01012a0 2024-03-12T05:55:02.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.352+0000 7faf96ffd640 1 -- 172.21.15.76:0/3312617622 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf9c009d90 con 0x7fafa01012a0 2024-03-12T05:55:02.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.352+0000 7faf96ffd640 1 -- 172.21.15.76:0/3312617622 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9c00f7e0 con 0x7fafa01012a0 2024-03-12T05:55:02.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.352+0000 7fafa7c81640 1 -- 172.21.15.76:0/3312617622 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa019e740 con 0x7fafa01012a0 2024-03-12T05:55:02.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.353+0000 7fafa7c81640 1 -- 172.21.15.76:0/3312617622 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa01073d0 con 0x7fafa01012a0 2024-03-12T05:55:02.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.354+0000 7faf96ffd640 1 -- 172.21.15.76:0/3312617622 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7faf9c010ad0 con 0x7fafa01012a0 2024-03-12T05:55:02.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.355+0000 7faf96ffd640 1 --2- 172.21.15.76:0/3312617622 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faf8c0a1ac0 0x7faf8c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.355+0000 7faf96ffd640 1 -- 172.21.15.76:0/3312617622 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7faf9c0cd1b0 con 0x7fafa01012a0 2024-03-12T05:55:02.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.355+0000 7fafa5a1d640 1 --2- 172.21.15.76:0/3312617622 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faf8c0a1ac0 0x7faf8c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:02.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.355+0000 7fafa5a1d640 1 --2- 172.21.15.76:0/3312617622 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faf8c0a1ac0 0x7faf8c0a3f80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7faf90005ab0 tx=0x7faf90005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:02.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.358+0000 7faf96ffd640 1 -- 172.21.15.76:0/3312617622 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faf9c096270 con 0x7fafa01012a0 2024-03-12T05:55:02.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:02 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2952973704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:02.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:02 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2525794224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:02.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:02 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3696396489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:02.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:02 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2952973704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:02.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:02 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2525794224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:02.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:02 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3696396489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:02.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.567+0000 7fafa7c81640 1 -- 172.21.15.76:0/3312617622 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafa010bba0 con 0x7fafa01012a0 2024-03-12T05:55:02.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.568+0000 7faf96ffd640 1 -- 172.21.15.76:0/3312617622 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+11328 (secure 0 0 0) 0x7faf9c017090 con 0x7fafa01012a0 2024-03-12T05:55:02.572 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:02.572 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:53:42.384114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:55:02.572 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 21 2024-03-12T05:55:02.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.570+0000 7fafa7c81640 1 -- 172.21.15.76:0/3312617622 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faf8c0a1ac0 msgr2=0x7faf8c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.570+0000 7fafa7c81640 1 --2- 172.21.15.76:0/3312617622 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faf8c0a1ac0 0x7faf8c0a3f80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7faf90005ab0 tx=0x7faf90005fd0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.570+0000 7fafa7c81640 1 -- 172.21.15.76:0/3312617622 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa01012a0 msgr2=0x7fafa01966c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.570+0000 7fafa7c81640 1 --2- 172.21.15.76:0/3312617622 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa01012a0 0x7fafa01966c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00d950 tx=0x7faf9c00de10 comp rx=0 tx=0).stop 2024-03-12T05:55:02.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.570+0000 7fafa7c81640 1 -- 172.21.15.76:0/3312617622 shutdown_connections 2024-03-12T05:55:02.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.570+0000 7fafa7c81640 1 --2- 172.21.15.76:0/3312617622 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faf8c0a1ac0 0x7faf8c0a3f80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.570+0000 7fafa7c81640 1 --2- 172.21.15.76:0/3312617622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fafa0101c60 0x7fafa019d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.570+0000 7fafa7c81640 1 --2- 172.21.15.76:0/3312617622 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa01012a0 0x7fafa01966c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.570+0000 7fafa7c81640 1 --2- 172.21.15.76:0/3312617622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa01000b0 0x7fafa0196180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.570+0000 7fafa7c81640 1 -- 172.21.15.76:0/3312617622 >> 172.21.15.76:0/3312617622 conn(0x7fafa00fb620 msgr2=0x7fafa01045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:02.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.570+0000 7fafa7c81640 1 -- 172.21.15.76:0/3312617622 shutdown_connections 2024-03-12T05:55:02.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.570+0000 7fafa7c81640 1 -- 172.21.15.76:0/3312617622 wait complete. 2024-03-12T05:55:02.585 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:55:02.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.683+0000 7f649664e640 1 -- 172.21.15.76:0/835320336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6490101210 msgr2=0x7f6490101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.683+0000 7f649664e640 1 --2- 172.21.15.76:0/835320336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6490101210 0x7f6490101630 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6484009900 tx=0x7f648402bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:02.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.683+0000 7f649664e640 1 -- 172.21.15.76:0/835320336 shutdown_connections 2024-03-12T05:55:02.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.683+0000 7f649664e640 1 --2- 172.21.15.76:0/835320336 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6490102dc0 0x7f6490109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.683+0000 7f649664e640 1 --2- 172.21.15.76:0/835320336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6490102400 0x7f6490102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.683+0000 7f649664e640 1 --2- 172.21.15.76:0/835320336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6490101210 0x7f6490101630 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.683+0000 7f649664e640 1 -- 172.21.15.76:0/835320336 >> 172.21.15.76:0/835320336 conn(0x7f64900fc7a0 msgr2=0x7f64900febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:02.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.683+0000 7f649664e640 1 -- 172.21.15.76:0/835320336 shutdown_connections 2024-03-12T05:55:02.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.683+0000 7f649664e640 1 -- 172.21.15.76:0/835320336 wait complete. 2024-03-12T05:55:02.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.684+0000 7f649664e640 1 Processor -- start 2024-03-12T05:55:02.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.684+0000 7f649664e640 1 -- start start 2024-03-12T05:55:02.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.684+0000 7f649664e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6490101210 0x7f64901971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.684+0000 7f649664e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6490102400 0x7f64901976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.684+0000 7f649664e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6490102dc0 0x7f649019e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.684+0000 7f649664e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649019eca0 con 0x7f6490101210 2024-03-12T05:55:02.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.684+0000 7f649664e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649019ee10 con 0x7f6490102dc0 2024-03-12T05:55:02.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.684+0000 7f649664e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649019efd0 con 0x7f6490102400 2024-03-12T05:55:02.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.684+0000 7f648ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6490101210 0x7f64901971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:02.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.684+0000 7f648ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6490101210 0x7f64901971b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60316/0 (socket says 172.21.15.76:60316) 2024-03-12T05:55:02.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.684+0000 7f648ffff640 1 -- 172.21.15.76:0/2283398312 learned_addr learned my addr 172.21.15.76:0/2283398312 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:02.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.684+0000 7f648f7fe640 1 --2- 172.21.15.76:0/2283398312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6490102400 0x7f64901976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:02.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.684+0000 7f648ffff640 1 -- 172.21.15.76:0/2283398312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6490102400 msgr2=0x7f64901976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.685+0000 7f648ffff640 1 --2- 172.21.15.76:0/2283398312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6490102400 0x7f64901976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.685+0000 7f648ffff640 1 -- 172.21.15.76:0/2283398312 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6490102dc0 msgr2=0x7f649019e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.685+0000 7f648ffff640 1 --2- 172.21.15.76:0/2283398312 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6490102dc0 0x7f649019e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.685+0000 7f648ffff640 1 -- 172.21.15.76:0/2283398312 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6484009560 con 0x7f6490101210 2024-03-12T05:55:02.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.685+0000 7f648f7fe640 1 --2- 172.21.15.76:0/2283398312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6490102400 0x7f64901976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:02.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.685+0000 7f648ffff640 1 --2- 172.21.15.76:0/2283398312 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6490101210 0x7f64901971b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6484000c00 tx=0x7f6484004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:02.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.685+0000 7f648d7fa640 1 -- 172.21.15.76:0/2283398312 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484004df0 con 0x7f6490101210 2024-03-12T05:55:02.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.685+0000 7f649664e640 1 -- 172.21.15.76:0/2283398312 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649019f250 con 0x7f6490101210 2024-03-12T05:55:02.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.685+0000 7f648d7fa640 1 -- 172.21.15.76:0/2283398312 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6484005430 con 0x7f6490101210 2024-03-12T05:55:02.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.685+0000 7f648d7fa640 1 -- 172.21.15.76:0/2283398312 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648403e450 con 0x7f6490101210 2024-03-12T05:55:02.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.685+0000 7f649664e640 1 -- 172.21.15.76:0/2283398312 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649019f760 con 0x7f6490101210 2024-03-12T05:55:02.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.686+0000 7f649664e640 1 -- 172.21.15.76:0/2283398312 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6490108580 con 0x7f6490101210 2024-03-12T05:55:02.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.687+0000 7f648d7fa640 1 -- 172.21.15.76:0/2283398312 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f6484003680 con 0x7f6490101210 2024-03-12T05:55:02.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.687+0000 7f648d7fa640 1 --2- 172.21.15.76:0/2283398312 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f64780a1ac0 0x7f64780a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.688+0000 7f648f7fe640 1 --2- 172.21.15.76:0/2283398312 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f64780a1ac0 0x7f64780a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:02.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.688+0000 7f648d7fa640 1 -- 172.21.15.76:0/2283398312 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f64840efa50 con 0x7f6490101210 2024-03-12T05:55:02.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.688+0000 7f648f7fe640 1 --2- 172.21.15.76:0/2283398312 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f64780a1ac0 0x7f64780a3f80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f64901986c0 tx=0x7f6480005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:02.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.692+0000 7f648d7fa640 1 -- 172.21.15.76:0/2283398312 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f64840b8c20 con 0x7f6490101210 2024-03-12T05:55:02.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.866+0000 7f649664e640 1 -- 172.21.15.76:0/2283398312 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f649010cb70 con 0x7f6490101210 2024-03-12T05:55:02.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.867+0000 7f648d7fa640 1 -- 172.21.15.76:0/2283398312 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22541 (secure 0 0 0) 0x7f64840bc2e0 con 0x7f6490101210 2024-03-12T05:55:02.871 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:02.871 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:54:42.007701+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:55:02.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.869+0000 7f649664e640 1 -- 172.21.15.76:0/2283398312 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f64780a1ac0 msgr2=0x7f64780a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.869+0000 7f649664e640 1 --2- 172.21.15.76:0/2283398312 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f64780a1ac0 0x7f64780a3f80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f64901986c0 tx=0x7f6480005d90 comp rx=0 tx=0).stop 2024-03-12T05:55:02.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.869+0000 7f649664e640 1 -- 172.21.15.76:0/2283398312 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6490101210 msgr2=0x7f64901971b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.869+0000 7f649664e640 1 --2- 172.21.15.76:0/2283398312 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6490101210 0x7f64901971b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6484000c00 tx=0x7f6484004270 comp rx=0 tx=0).stop 2024-03-12T05:55:02.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.870+0000 7f649664e640 1 -- 172.21.15.76:0/2283398312 shutdown_connections 2024-03-12T05:55:02.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.870+0000 7f649664e640 1 --2- 172.21.15.76:0/2283398312 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f64780a1ac0 0x7f64780a3f80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.870+0000 7f649664e640 1 --2- 172.21.15.76:0/2283398312 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6490102dc0 0x7f649019e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.870+0000 7f649664e640 1 --2- 172.21.15.76:0/2283398312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6490102400 0x7f64901976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.870+0000 7f649664e640 1 --2- 172.21.15.76:0/2283398312 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6490101210 0x7f64901971b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.870+0000 7f649664e640 1 -- 172.21.15.76:0/2283398312 >> 172.21.15.76:0/2283398312 conn(0x7f64900fc7a0 msgr2=0x7f6490105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:02.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.870+0000 7f649664e640 1 -- 172.21.15.76:0/2283398312 shutdown_connections 2024-03-12T05:55:02.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.870+0000 7f649664e640 1 -- 172.21.15.76:0/2283398312 wait complete. 2024-03-12T05:55:02.885 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:55:02.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.982+0000 7f92229a0640 1 -- 172.21.15.76:0/1734291629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f921c102c30 msgr2=0x7f921c1094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.982+0000 7f92229a0640 1 --2- 172.21.15.76:0/1734291629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f921c102c30 0x7f921c1094c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9214009ca0 tx=0x7f921402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.982+0000 7f92229a0640 1 -- 172.21.15.76:0/1734291629 shutdown_connections 2024-03-12T05:55:02.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.982+0000 7f92229a0640 1 --2- 172.21.15.76:0/1734291629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f921c102c30 0x7f921c1094c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.982+0000 7f92229a0640 1 --2- 172.21.15.76:0/1734291629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f921c102270 0x7f921c1026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.982+0000 7f92229a0640 1 --2- 172.21.15.76:0/1734291629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f921c101080 0x7f921c1014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.982+0000 7f92229a0640 1 -- 172.21.15.76:0/1734291629 >> 172.21.15.76:0/1734291629 conn(0x7f921c0fc610 msgr2=0x7f921c0fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:02.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.982+0000 7f92229a0640 1 -- 172.21.15.76:0/1734291629 shutdown_connections 2024-03-12T05:55:02.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.982+0000 7f92229a0640 1 -- 172.21.15.76:0/1734291629 wait complete. 2024-03-12T05:55:02.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f92229a0640 1 Processor -- start 2024-03-12T05:55:02.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f92229a0640 1 -- start start 2024-03-12T05:55:02.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f92229a0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f921c101080 0x7f921c1970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f92229a0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f921c102270 0x7f921c197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f92229a0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f921c102c30 0x7f921c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f92229a0640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c19ebd0 con 0x7f921c102270 2024-03-12T05:55:02.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f92229a0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c19ed40 con 0x7f921c102c30 2024-03-12T05:55:02.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f92229a0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c19ef00 con 0x7f921c101080 2024-03-12T05:55:02.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f922119d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f921c102270 0x7f921c197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:02.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f922119d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f921c102270 0x7f921c197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60328/0 (socket says 172.21.15.76:60328) 2024-03-12T05:55:02.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f922119d640 1 -- 172.21.15.76:0/3434864429 learned_addr learned my addr 172.21.15.76:0/3434864429 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:02.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f922199e640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f921c101080 0x7f921c1970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:02.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f922219f640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f921c102c30 0x7f921c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:02.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f922119d640 1 -- 172.21.15.76:0/3434864429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f921c101080 msgr2=0x7f921c1970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f922119d640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f921c101080 0x7f921c1970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f922119d640 1 -- 172.21.15.76:0/3434864429 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f921c102c30 msgr2=0x7f921c19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:02.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.983+0000 7f922119d640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f921c102c30 0x7f921c19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:02.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.984+0000 7f922119d640 1 -- 172.21.15.76:0/3434864429 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9214009900 con 0x7f921c102270 2024-03-12T05:55:02.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.984+0000 7f922219f640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f921c102c30 0x7f921c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:02.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.984+0000 7f922119d640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f921c102270 0x7f921c197620 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f920c00c970 tx=0x7f920c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:02.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.984+0000 7f9212ffd640 1 -- 172.21.15.76:0/3434864429 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c007bb0 con 0x7f921c102270 2024-03-12T05:55:02.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.984+0000 7f9212ffd640 1 -- 172.21.15.76:0/3434864429 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f920c007d10 con 0x7f921c102270 2024-03-12T05:55:02.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.984+0000 7f9212ffd640 1 -- 172.21.15.76:0/3434864429 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c00f450 con 0x7f921c102270 2024-03-12T05:55:02.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.984+0000 7f92229a0640 1 -- 172.21.15.76:0/3434864429 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921c19f1e0 con 0x7f921c102270 2024-03-12T05:55:02.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.984+0000 7f92229a0640 1 -- 172.21.15.76:0/3434864429 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921c19f6a0 con 0x7f921c102270 2024-03-12T05:55:02.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.985+0000 7f92229a0640 1 -- 172.21.15.76:0/3434864429 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f921c108340 con 0x7f921c102270 2024-03-12T05:55:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.986+0000 7f9212ffd640 1 -- 172.21.15.76:0/3434864429 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f920c004f70 con 0x7f921c102270 2024-03-12T05:55:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.986+0000 7f9212ffd640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f92080a1ab0 0x7f92080a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.986+0000 7f922199e640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f92080a1ab0 0x7f92080a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.986+0000 7f9212ffd640 1 -- 172.21.15.76:0/3434864429 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f920c0cd2c0 con 0x7f921c102270 2024-03-12T05:55:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.987+0000 7f922199e640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f92080a1ab0 0x7f92080a3f70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f921c1020d0 tx=0x7f921800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:02.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:02.991+0000 7f9212ffd640 1 -- 172.21.15.76:0/3434864429 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f920c096380 con 0x7f921c102270 2024-03-12T05:55:03.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.201+0000 7f92229a0640 1 -- 172.21.15.76:0/3434864429 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f921c10ca90 con 0x7f921c102270 2024-03-12T05:55:03.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.202+0000 7f9212ffd640 1 -- 172.21.15.76:0/3434864429 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+11328 (secure 0 0 0) 0x7f920c099a40 con 0x7f921c102270 2024-03-12T05:55:03.206 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:03.206 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:53:42.384114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:55:03.206 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 21 2024-03-12T05:55:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.204+0000 7f92229a0640 1 -- 172.21.15.76:0/3434864429 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f92080a1ab0 msgr2=0x7f92080a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.204+0000 7f92229a0640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f92080a1ab0 0x7f92080a3f70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f921c1020d0 tx=0x7f921800a2d0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.204+0000 7f92229a0640 1 -- 172.21.15.76:0/3434864429 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f921c102270 msgr2=0x7f921c197620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.204+0000 7f92229a0640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f921c102270 0x7f921c197620 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f920c00c970 tx=0x7f920c00ce30 comp rx=0 tx=0).stop 2024-03-12T05:55:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.204+0000 7f92229a0640 1 -- 172.21.15.76:0/3434864429 shutdown_connections 2024-03-12T05:55:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.204+0000 7f92229a0640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f92080a1ab0 0x7f92080a3f70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.204+0000 7f92229a0640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f921c102c30 0x7f921c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.204+0000 7f92229a0640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f921c102270 0x7f921c197620 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.204+0000 7f92229a0640 1 --2- 172.21.15.76:0/3434864429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f921c101080 0x7f921c1970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.204+0000 7f92229a0640 1 -- 172.21.15.76:0/3434864429 >> 172.21.15.76:0/3434864429 conn(0x7f921c0fc610 msgr2=0x7f921c105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.204+0000 7f92229a0640 1 -- 172.21.15.76:0/3434864429 shutdown_connections 2024-03-12T05:55:03.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.204+0000 7f92229a0640 1 -- 172.21.15.76:0/3434864429 wait complete. 2024-03-12T05:55:03.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:03 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3599220751' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:03.218 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:03 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3312617622' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:03.218 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:03 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2283398312' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:03.218 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:03 smithi076 ceph-mon[32095]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:55:03.219 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-12T05:53:37.535927+0000', 'modified': '2024-03-12T05:53:42.384114+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24518, 'mds_1': 24481, 'mds_2': 24451}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24449': {'gid': 24449, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.82:6835/2138041661', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6834', 'nonce': 2138041661}, {'type': 'v1', 'addr': '172.21.15.82:6835', 'nonce': 2138041661}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.88:6835/3940245802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 3940245802}, {'type': 'v1', 'addr': '172.21.15.88:6835', 'nonce': 3940245802}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.88:6837/1311843051', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6836', 'nonce': 1311843051}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 1311843051}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24479': {'gid': 24479, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.82:6837/2523422510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6836', 'nonce': 2523422510}, {'type': 'v1', 'addr': '172.21.15.82:6837', 'nonce': 2523422510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24481': {'gid': 24481, 'name': 'i', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.88:6839/1164228359', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6838', 'nonce': 1164228359}, {'type': 'v1', 'addr': '172.21.15.88:6839', 'nonce': 1164228359}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24518': {'gid': 24518, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.82:6839/2242595718', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6838', 'nonce': 2242595718}, {'type': 'v1', 'addr': '172.21.15.82:6839', 'nonce': 2242595718}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-12T05:55:03.220 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-12T05:55:03.220 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-12T05:55:03.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 -- 172.21.15.76:0/3572392673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f195c102790 msgr2=0x7f195c102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 --2- 172.21.15.76:0/3572392673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f195c102790 0x7f195c102bb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1950009900 tx=0x7f195002bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:03.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 -- 172.21.15.76:0/3572392673 shutdown_connections 2024-03-12T05:55:03.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 --2- 172.21.15.76:0/3572392673 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195c0ffbc0 0x7f195c109770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 --2- 172.21.15.76:0/3572392673 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f195c0ff200 0x7f195c0ff680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 --2- 172.21.15.76:0/3572392673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f195c102790 0x7f195c102bb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 -- 172.21.15.76:0/3572392673 >> 172.21.15.76:0/3572392673 conn(0x7f195c0fada0 msgr2=0x7f195c0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 -- 172.21.15.76:0/3572392673 shutdown_connections 2024-03-12T05:55:03.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 -- 172.21.15.76:0/3572392673 wait complete. 2024-03-12T05:55:03.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 Processor -- start 2024-03-12T05:55:03.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 -- start start 2024-03-12T05:55:03.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f195c0ff200 0x7f195c197210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f195c0ffbc0 0x7f195c197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195c102790 0x7f195c19e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c19ed00 con 0x7f195c0ffbc0 2024-03-12T05:55:03.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c19ee70 con 0x7f195c0ff200 2024-03-12T05:55:03.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.319+0000 7f196384e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195c19f030 con 0x7f195c102790 2024-03-12T05:55:03.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f1960de9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f195c0ffbc0 0x7f195c197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f1960de9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f195c0ffbc0 0x7f195c197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60348/0 (socket says 172.21.15.76:60348) 2024-03-12T05:55:03.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f1960de9640 1 -- 172.21.15.76:0/2326899267 learned_addr learned my addr 172.21.15.76:0/2326899267 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f19615ea640 1 --2- 172.21.15.76:0/2326899267 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f195c0ff200 0x7f195c197210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f1961deb640 1 --2- 172.21.15.76:0/2326899267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195c102790 0x7f195c19e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f1960de9640 1 -- 172.21.15.76:0/2326899267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195c102790 msgr2=0x7f195c19e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f1960de9640 1 --2- 172.21.15.76:0/2326899267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195c102790 0x7f195c19e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f1960de9640 1 -- 172.21.15.76:0/2326899267 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f195c0ff200 msgr2=0x7f195c197210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f1960de9640 1 --2- 172.21.15.76:0/2326899267 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f195c0ff200 0x7f195c197210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f1960de9640 1 -- 172.21.15.76:0/2326899267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1950009560 con 0x7f195c0ffbc0 2024-03-12T05:55:03.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f1960de9640 1 --2- 172.21.15.76:0/2326899267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f195c0ffbc0 0x7f195c197750 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f195800c890 tx=0x7f195800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f194e7fc640 1 -- 172.21.15.76:0/2326899267 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958004220 con 0x7f195c0ffbc0 2024-03-12T05:55:03.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f194e7fc640 1 -- 172.21.15.76:0/2326899267 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1958007e90 con 0x7f195c0ffbc0 2024-03-12T05:55:03.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f196384e640 1 -- 172.21.15.76:0/2326899267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195c19f310 con 0x7f195c0ffbc0 2024-03-12T05:55:03.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f194e7fc640 1 -- 172.21.15.76:0/2326899267 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958004a00 con 0x7f195c0ffbc0 2024-03-12T05:55:03.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.320+0000 7f196384e640 1 -- 172.21.15.76:0/2326899267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195c19f820 con 0x7f195c0ffbc0 2024-03-12T05:55:03.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.323+0000 7f1966423740 1 Processor -- start 2024-03-12T05:55:03.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.323+0000 7f194e7fc640 1 -- 172.21.15.76:0/2326899267 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f19580079b0 con 0x7f195c0ffbc0 2024-03-12T05:55:03.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.323+0000 7f1966423740 1 -- start start 2024-03-12T05:55:03.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.323+0000 7f1966423740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 0x55fb593b73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.323+0000 7f1966423740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593bc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.323+0000 7f1966423740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb593bef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.323+0000 7f1966423740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb59372cd0 con 0x55fb593b7970 2024-03-12T05:55:03.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.323+0000 7f1966423740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb59325f20 con 0x55fb593bcae0 2024-03-12T05:55:03.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.323+0000 7f1966423740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb59323450 con 0x55fb593b6f80 2024-03-12T05:55:03.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.323+0000 7f19615ea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593bc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f19615ea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593bc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60356/0 (socket says 172.21.15.76:60356) 2024-03-12T05:55:03.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f19615ea640 1 -- 172.21.15.76:0/2399269291 learned_addr learned my addr 172.21.15.76:0/2399269291 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f1961deb640 1 --2- 172.21.15.76:0/2399269291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 0x55fb593b73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f1960de9640 1 --2- 172.21.15.76:0/2399269291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb593bef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f19615ea640 1 -- 172.21.15.76:0/2399269291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 msgr2=0x55fb593b73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f19615ea640 1 --2- 172.21.15.76:0/2399269291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 0x55fb593b73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f19615ea640 1 -- 172.21.15.76:0/2399269291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 msgr2=0x55fb593bef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f19615ea640 1 --2- 172.21.15.76:0/2399269291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb593bef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f194e7fc640 1 --2- 172.21.15.76:0/2326899267 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f19480a1ac0 0x7f19480a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f19615ea640 1 -- 172.21.15.76:0/2399269291 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb592d3f20 con 0x55fb593b7970 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f1961deb640 1 --2- 172.21.15.76:0/2326899267 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f19480a1ac0 0x7f19480a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f194e7fc640 1 -- 172.21.15.76:0/2326899267 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f19580cd280 con 0x7f195c0ffbc0 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f19615ea640 1 --2- 172.21.15.76:0/2399269291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593bc470 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f195c069580 tx=0x7f195002ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8af87d89543c84d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f19477fe640 1 -- 172.21.15.76:0/2399269291 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19500046c0 con 0x55fb593b7970 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f19477fe640 1 -- 172.21.15.76:0/2399269291 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1950004510 con 0x55fb593b7970 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.324+0000 7f19477fe640 1 -- 172.21.15.76:0/2399269291 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19500054a0 con 0x55fb593b7970 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.325+0000 7f1961deb640 1 --2- 172.21.15.76:0/2326899267 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f19480a1ac0 0x7f19480a3f80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55fb593b81d0 tx=0x7f1954007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.325+0000 7f1966423740 1 -- 172.21.15.76:0/2399269291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 msgr2=0x55fb593bc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.325+0000 7f1966423740 1 --2- 172.21.15.76:0/2399269291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593bc470 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f195c069580 tx=0x7f195002ca10 comp rx=0 tx=0).stop 2024-03-12T05:55:03.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.325+0000 7f1966423740 1 -- 172.21.15.76:0/2399269291 shutdown_connections 2024-03-12T05:55:03.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.325+0000 7f1966423740 1 --2- 172.21.15.76:0/2399269291 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb593bef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.325+0000 7f1966423740 1 --2- 172.21.15.76:0/2399269291 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593bc470 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.325+0000 7f1966423740 1 --2- 172.21.15.76:0/2399269291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 0x55fb593b73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.325+0000 7f1966423740 1 -- 172.21.15.76:0/2399269291 >> 172.21.15.76:0/2399269291 conn(0x55fb593b24f0 msgr2=0x55fb593b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.325+0000 7f1966423740 1 -- 172.21.15.76:0/2399269291 shutdown_connections 2024-03-12T05:55:03.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.325+0000 7f1966423740 1 -- 172.21.15.76:0/2399269291 wait complete. 2024-03-12T05:55:03.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.325+0000 7f1966423740 1 -- start start 2024-03-12T05:55:03.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.325+0000 7f1966423740 1 Processor -- start 2024-03-12T05:55:03.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1966423740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 0x55fb593c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1966423740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593c14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1966423740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb593c8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1966423740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb5931ec30 con 0x55fb593b7970 2024-03-12T05:55:03.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1966423740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb592c3d30 con 0x55fb593bcae0 2024-03-12T05:55:03.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1966423740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb58ff2310 con 0x55fb593b6f80 2024-03-12T05:55:03.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1966423740 10 client.0 _ll_register_callbacks cb 0x55fb592cbd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:03.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1960de9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593c14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1960de9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593c14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60368/0 (socket says 172.21.15.76:60368) 2024-03-12T05:55:03.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1960de9640 1 -- 172.21.15.76:0/1270107889 learned_addr learned my addr 172.21.15.76:0/1270107889 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f19615ea640 1 --2- 172.21.15.76:0/1270107889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 0x55fb593c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1961deb640 1 --2- 172.21.15.76:0/1270107889 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb593c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1960de9640 1 -- 172.21.15.76:0/1270107889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 msgr2=0x55fb593c0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1960de9640 1 --2- 172.21.15.76:0/1270107889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 0x55fb593c0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1960de9640 1 -- 172.21.15.76:0/1270107889 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 msgr2=0x55fb593c8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1960de9640 1 --2- 172.21.15.76:0/1270107889 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb593c8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1960de9640 1 -- 172.21.15.76:0/1270107889 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195c109ef0 con 0x55fb593b7970 2024-03-12T05:55:03.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f19615ea640 1 --2- 172.21.15.76:0/1270107889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 0x55fb593c0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:03.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1960de9640 1 --2- 172.21.15.76:0/1270107889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593c14d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1958096a70 tx=0x7f1958099be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb8bf8155685ac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1944ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:03.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1944ff9640 1 -- 172.21.15.76:0/1270107889 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195809a0a0 con 0x55fb593b7970 2024-03-12T05:55:03.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1944ff9640 1 -- 172.21.15.76:0/1270107889 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f195809e1f0 con 0x55fb593b7970 2024-03-12T05:55:03.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.326+0000 7f1966423740 1 -- 172.21.15.76:0/1270107889 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fb5901e6c0 con 0x55fb593b7970 2024-03-12T05:55:03.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.327+0000 7f1944ff9640 1 -- 172.21.15.76:0/1270107889 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19580a2840 con 0x55fb593b7970 2024-03-12T05:55:03.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.327+0000 7f1966423740 10 client.15084 fetch_fsmap learned FSMap version 21 2024-03-12T05:55:03.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.327+0000 7f1966423740 1 -- 172.21.15.76:0/1270107889 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55fb5931d2b0 con 0x55fb593b7970 2024-03-12T05:55:03.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.327+0000 7f1944ff9640 1 -- 172.21.15.76:0/1270107889 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55fb5901e6c0 con 0x55fb593b7970 2024-03-12T05:55:03.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.327+0000 7f1944ff9640 1 -- 172.21.15.76:0/1270107889 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 21) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1958016030 con 0x55fb593b7970 2024-03-12T05:55:03.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.327+0000 7f1966423740 10 client.15084 fetch_fsmap finished waiting for FSMap version 21 2024-03-12T05:55:03.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.327+0000 7f1966423740 10 client.15084 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-12T05:55:03.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.327+0000 7f1966423740 20 client.15084 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:03.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.327+0000 7f1966423740 1 --2- 172.21.15.76:0/1270107889 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55fb593cd9a0 0x55fb593ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.327+0000 7f1966423740 4 client.15084 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:55:03.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.327+0000 7f1966423740 1 -- 172.21.15.76:0/1270107889 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fb5906f270 con 0x55fb593cd9a0 2024-03-12T05:55:03.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.327+0000 7f19615ea640 1 --2- 172.21.15.76:0/1270107889 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55fb593cd9a0 0x55fb593ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.328+0000 7f1944ff9640 10 client.15084 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:55:03.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.328+0000 7f19615ea640 1 --2- 172.21.15.76:0/1270107889 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55fb593cd9a0 0x55fb593ccc30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3a27848b884071c6 server_cookie=e3c18eb4e1fd596e in_seq=0 out_seq=0 2024-03-12T05:55:03.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.328+0000 7f1944ff9640 1 -- 172.21.15.76:0/1270107889 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fb5906f270 con 0x55fb593cd9a0 2024-03-12T05:55:03.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.328+0000 7f1944ff9640 10 client.15084 handle_command_reply: tid=0 2024-03-12T05:55:03.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.328+0000 7f1966423740 1 client.15084 shutdown 2024-03-12T05:55:03.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.329+0000 7f1966423740 1 -- 172.21.15.76:0/1270107889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 msgr2=0x55fb593c14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.329+0000 7f1966423740 1 --2- 172.21.15.76:0/1270107889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593c14d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1958096a70 tx=0x7f1958099be0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.329+0000 7f1966423740 1 -- 172.21.15.76:0/1270107889 shutdown_connections 2024-03-12T05:55:03.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.329+0000 7f1966423740 1 --2- 172.21.15.76:0/1270107889 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55fb593cd9a0 0x55fb593ccc30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.329+0000 7f1966423740 1 --2- 172.21.15.76:0/1270107889 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb593c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.329+0000 7f1966423740 1 --2- 172.21.15.76:0/1270107889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593c14d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.329+0000 7f1944ff9640 0 client.15084 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:55:03.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.329+0000 7f1966423740 1 --2- 172.21.15.76:0/1270107889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 0x55fb593c0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.329+0000 7f1966423740 1 -- 172.21.15.76:0/1270107889 >> 172.21.15.76:0/1270107889 conn(0x55fb593b24f0 msgr2=0x55fb593b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.329+0000 7f1966423740 1 -- 172.21.15.76:0/1270107889 shutdown_connections 2024-03-12T05:55:03.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.329+0000 7f1966423740 1 -- 172.21.15.76:0/1270107889 wait complete. 2024-03-12T05:55:03.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.329+0000 7f1966423740 20 client.15084 trim_cache size 0 max 16384 2024-03-12T05:55:03.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.344+0000 7f1966423740 1 Processor -- start 2024-03-12T05:55:03.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.344+0000 7f1966423740 1 -- start start 2024-03-12T05:55:03.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.344+0000 7f1966423740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 0x55fb593b8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.344+0000 7f1966423740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593b9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.344+0000 7f1966423740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb593c84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.344+0000 7f1966423740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb593c0100 con 0x55fb593b7970 2024-03-12T05:55:03.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.344+0000 7f1966423740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb592a4490 con 0x55fb593bcae0 2024-03-12T05:55:03.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.344+0000 7f1966423740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb593bc640 con 0x55fb593b6f80 2024-03-12T05:55:03.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.345+0000 7f1960de9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593b9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.345+0000 7f1960de9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593b9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60376/0 (socket says 172.21.15.76:60376) 2024-03-12T05:55:03.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.345+0000 7f1960de9640 1 -- 172.21.15.76:0/240140195 learned_addr learned my addr 172.21.15.76:0/240140195 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.345+0000 7f1961deb640 1 --2- 172.21.15.76:0/240140195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb593c84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.345+0000 7f1960de9640 1 -- 172.21.15.76:0/240140195 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 msgr2=0x55fb593b8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.345+0000 7f1960de9640 1 --2- 172.21.15.76:0/240140195 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 0x55fb593b8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.345+0000 7f1960de9640 1 -- 172.21.15.76:0/240140195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 msgr2=0x55fb593c84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.345+0000 7f1960de9640 1 --2- 172.21.15.76:0/240140195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb593c84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.345+0000 7f1960de9640 1 -- 172.21.15.76:0/240140195 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb592b9a20 con 0x55fb593b7970 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.345+0000 7f1961deb640 1 --2- 172.21.15.76:0/240140195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb593c84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.345+0000 7f1960de9640 1 --2- 172.21.15.76:0/240140195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593b9330 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1958099da0 tx=0x7f1958099d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20fa6ca22dfaf1f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.345+0000 7f1946ffd640 1 -- 172.21.15.76:0/240140195 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958096340 con 0x55fb593b7970 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.345+0000 7f1946ffd640 1 -- 172.21.15.76:0/240140195 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19580b8070 con 0x55fb593b7970 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.345+0000 7f1946ffd640 1 -- 172.21.15.76:0/240140195 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19580cafb0 con 0x55fb593b7970 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 -- 172.21.15.76:0/240140195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 msgr2=0x55fb593b9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 --2- 172.21.15.76:0/240140195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593b9330 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1958099da0 tx=0x7f1958099d10 comp rx=0 tx=0).stop 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 -- 172.21.15.76:0/240140195 shutdown_connections 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 --2- 172.21.15.76:0/240140195 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb593c84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 --2- 172.21.15.76:0/240140195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b7970 0x55fb593b9330 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 --2- 172.21.15.76:0/240140195 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b6f80 0x55fb593b8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 -- 172.21.15.76:0/240140195 >> 172.21.15.76:0/240140195 conn(0x55fb593b24f0 msgr2=0x55fb593d5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 -- 172.21.15.76:0/240140195 shutdown_connections 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 -- 172.21.15.76:0/240140195 wait complete. 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 -- start start 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 Processor -- start 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b6f80 0x55fb593c5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b7970 0x55fb593c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb594235a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb593b2f40 con 0x55fb593b6f80 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb593bf100 con 0x55fb593bcae0 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb593c8710 con 0x55fb593b7970 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f1966423740 10 client.0 _ll_register_callbacks cb 0x7f1950005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f19615ea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b6f80 0x55fb593c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.346+0000 7f19615ea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b6f80 0x55fb593c5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60392/0 (socket says 172.21.15.76:60392) 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f19615ea640 1 -- 172.21.15.76:0/3991957117 learned_addr learned my addr 172.21.15.76:0/3991957117 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f1961deb640 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb594235a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f1960de9640 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b7970 0x55fb593c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f19615ea640 1 -- 172.21.15.76:0/3991957117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b7970 msgr2=0x55fb593c56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f19615ea640 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b7970 0x55fb593c56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f19615ea640 1 -- 172.21.15.76:0/3991957117 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 msgr2=0x55fb594235a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f19615ea640 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb594235a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f19615ea640 1 -- 172.21.15.76:0/3991957117 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fb5931d2b0 con 0x55fb593b6f80 2024-03-12T05:55:03.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f1961deb640 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb594235a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:03.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f1960de9640 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b7970 0x55fb593c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:03.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f19615ea640 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b6f80 0x55fb593c5180 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f195c069580 tx=0x7f195002cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc5f50053997861 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f1944ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:03.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f1944ff9640 1 -- 172.21.15.76:0/3991957117 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19500048c0 con 0x55fb593b6f80 2024-03-12T05:55:03.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f1944ff9640 1 -- 172.21.15.76:0/3991957117 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1950004a20 con 0x55fb593b6f80 2024-03-12T05:55:03.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f1966423740 1 -- 172.21.15.76:0/3991957117 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1950004510 con 0x55fb593b6f80 2024-03-12T05:55:03.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f1944ff9640 1 -- 172.21.15.76:0/3991957117 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1950050b70 con 0x55fb593b6f80 2024-03-12T05:55:03.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f1966423740 10 client.15090 fetch_fsmap learned FSMap version 21 2024-03-12T05:55:03.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f1966423740 1 -- 172.21.15.76:0/3991957117 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55fb592df780 con 0x55fb593b6f80 2024-03-12T05:55:03.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f1944ff9640 1 -- 172.21.15.76:0/3991957117 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1950004510 con 0x55fb593b6f80 2024-03-12T05:55:03.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f1944ff9640 1 -- 172.21.15.76:0/3991957117 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 21) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f195003c050 con 0x55fb593b6f80 2024-03-12T05:55:03.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.347+0000 7f1966423740 10 client.15090 fetch_fsmap finished waiting for FSMap version 21 2024-03-12T05:55:03.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1966423740 10 client.15090 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-12T05:55:03.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1966423740 20 client.15090 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:03.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1966423740 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55fb59428730 0x55fb59424b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1966423740 4 client.15090 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:55:03.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1966423740 1 -- 172.21.15.76:0/3991957117 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fb5905b610 con 0x55fb59428730 2024-03-12T05:55:03.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1960de9640 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55fb59428730 0x55fb59424b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1960de9640 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55fb59428730 0x55fb59424b70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=56b3be328a4a31a1 server_cookie=4e04f0fc94c0351d in_seq=0 out_seq=0 2024-03-12T05:55:03.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1944ff9640 10 client.15090 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:55:03.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1944ff9640 1 -- 172.21.15.76:0/3991957117 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55fb5905b610 con 0x55fb59428730 2024-03-12T05:55:03.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1944ff9640 10 client.15090 handle_command_reply: tid=0 2024-03-12T05:55:03.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1966423740 1 client.15090 shutdown 2024-03-12T05:55:03.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1966423740 1 -- 172.21.15.76:0/3991957117 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b6f80 msgr2=0x55fb593c5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1966423740 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b6f80 0x55fb593c5180 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f195c069580 tx=0x7f195002cd60 comp rx=0 tx=0).stop 2024-03-12T05:55:03.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1966423740 1 -- 172.21.15.76:0/3991957117 shutdown_connections 2024-03-12T05:55:03.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1966423740 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55fb59428730 0x55fb59424b70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1966423740 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb593bcae0 0x55fb594235a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1966423740 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb593b7970 0x55fb593c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1944ff9640 0 client.15090 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:55:03.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1966423740 1 --2- 172.21.15.76:0/3991957117 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb593b6f80 0x55fb593c5180 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.348+0000 7f1966423740 1 -- 172.21.15.76:0/3991957117 >> 172.21.15.76:0/3991957117 conn(0x55fb593b24f0 msgr2=0x55fb593b8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f1966423740 1 -- 172.21.15.76:0/3991957117 shutdown_connections 2024-03-12T05:55:03.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f1966423740 1 -- 172.21.15.76:0/3991957117 wait complete. 2024-03-12T05:55:03.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f1966423740 20 client.15090 trim_cache size 0 max 16384 2024-03-12T05:55:03.361 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:55:03.361 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "9f336de8-e033-11ee-95c8-87774f69a715", 2024-03-12T05:55:03.361 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 2, 2024-03-12T05:55:03.361 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24451, 2024-03-12T05:55:03.361 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-03-12T05:55:03.361 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-03-12T05:55:03.361 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-03-12T05:55:03.361 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 81.982997681000001, 2024-03-12T05:55:03.361 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 21, 2024-03-12T05:55:03.361 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 88, 2024-03-12T05:55:03.361 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 86, 2024-03-12T05:55:03.361 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 107.47592755300001 2024-03-12T05:55:03.361 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:55:03.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f196384e640 1 -- 172.21.15.76:0/2326899267 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f19480a1ac0 msgr2=0x7f19480a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f196384e640 1 --2- 172.21.15.76:0/2326899267 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f19480a1ac0 0x7f19480a3f80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55fb593b81d0 tx=0x7f1954007400 comp rx=0 tx=0).stop 2024-03-12T05:55:03.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f196384e640 1 -- 172.21.15.76:0/2326899267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f195c0ffbc0 msgr2=0x7f195c197750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f196384e640 1 --2- 172.21.15.76:0/2326899267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f195c0ffbc0 0x7f195c197750 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f195800c890 tx=0x7f195800cd50 comp rx=0 tx=0).stop 2024-03-12T05:55:03.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f196384e640 1 -- 172.21.15.76:0/2326899267 shutdown_connections 2024-03-12T05:55:03.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f196384e640 1 --2- 172.21.15.76:0/2326899267 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f19480a1ac0 0x7f19480a3f80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f196384e640 1 --2- 172.21.15.76:0/2326899267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195c102790 0x7f195c19e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f196384e640 1 --2- 172.21.15.76:0/2326899267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f195c0ffbc0 0x7f195c197750 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f196384e640 1 --2- 172.21.15.76:0/2326899267 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f195c0ff200 0x7f195c197210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f196384e640 1 -- 172.21.15.76:0/2326899267 >> 172.21.15.76:0/2326899267 conn(0x7f195c0fada0 msgr2=0x7f195c105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f196384e640 1 -- 172.21.15.76:0/2326899267 shutdown_connections 2024-03-12T05:55:03.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.349+0000 7f196384e640 1 -- 172.21.15.76:0/2326899267 wait complete. 2024-03-12T05:55:03.364 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-12T05:55:03.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:03 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3599220751' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:03.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:03 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3312617622' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:03.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:03 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2283398312' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:03.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:03 smithi082 ceph-mon[36382]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:55:03.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.464+0000 7f75d7c64640 1 -- 172.21.15.76:0/1380335404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75d0101190 msgr2=0x7f75d01015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.464+0000 7f75d7c64640 1 --2- 172.21.15.76:0/1380335404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75d0101190 0x7f75d01015b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009900 tx=0x7f75c002bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:03.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.464+0000 7f75d7c64640 1 -- 172.21.15.76:0/1380335404 shutdown_connections 2024-03-12T05:55:03.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.464+0000 7f75d7c64640 1 --2- 172.21.15.76:0/1380335404 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75d0102d40 0x7f75d01095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.464+0000 7f75d7c64640 1 --2- 172.21.15.76:0/1380335404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75d0102380 0x7f75d0102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.464+0000 7f75d7c64640 1 --2- 172.21.15.76:0/1380335404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75d0101190 0x7f75d01015b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.464+0000 7f75d7c64640 1 -- 172.21.15.76:0/1380335404 >> 172.21.15.76:0/1380335404 conn(0x7f75d00fc700 msgr2=0x7f75d00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.464+0000 7f75d7c64640 1 -- 172.21.15.76:0/1380335404 shutdown_connections 2024-03-12T05:55:03.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.464+0000 7f75d7c64640 1 -- 172.21.15.76:0/1380335404 wait complete. 2024-03-12T05:55:03.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d7c64640 1 Processor -- start 2024-03-12T05:55:03.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d7c64640 1 -- start start 2024-03-12T05:55:03.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d7c64640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75d0101190 0x7f75d0197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d7c64640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75d0102380 0x7f75d0197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d7c64640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75d0102d40 0x7f75d019e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d7c64640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d019ecf0 con 0x7f75d0102380 2024-03-12T05:55:03.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d7c64640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d019ee60 con 0x7f75d0102d40 2024-03-12T05:55:03.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d7c64640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d019f020 con 0x7f75d0101190 2024-03-12T05:55:03.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d51ff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75d0102380 0x7f75d0197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d51ff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75d0102380 0x7f75d0197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60400/0 (socket says 172.21.15.76:60400) 2024-03-12T05:55:03.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d51ff640 1 -- 172.21.15.76:0/2931256799 learned_addr learned my addr 172.21.15.76:0/2931256799 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d5a00640 1 --2- 172.21.15.76:0/2931256799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75d0101190 0x7f75d0197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d6201640 1 --2- 172.21.15.76:0/2931256799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75d0102d40 0x7f75d019e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d51ff640 1 -- 172.21.15.76:0/2931256799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75d0101190 msgr2=0x7f75d0197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d51ff640 1 --2- 172.21.15.76:0/2931256799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75d0101190 0x7f75d0197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d51ff640 1 -- 172.21.15.76:0/2931256799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75d0102d40 msgr2=0x7f75d019e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d51ff640 1 --2- 172.21.15.76:0/2931256799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75d0102d40 0x7f75d019e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d51ff640 1 -- 172.21.15.76:0/2931256799 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c0009560 con 0x7f75d0102380 2024-03-12T05:55:03.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.465+0000 7f75d51ff640 1 --2- 172.21.15.76:0/2931256799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75d0102380 0x7f75d0197740 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f75cc00c920 tx=0x7f75cc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.466+0000 7f75c6ffd640 1 -- 172.21.15.76:0/2931256799 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc007c70 con 0x7f75d0102380 2024-03-12T05:55:03.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.466+0000 7f75c6ffd640 1 -- 172.21.15.76:0/2931256799 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75cc007dd0 con 0x7f75d0102380 2024-03-12T05:55:03.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.466+0000 7f75d7c64640 1 -- 172.21.15.76:0/2931256799 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d019f300 con 0x7f75d0102380 2024-03-12T05:55:03.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.466+0000 7f75c6ffd640 1 -- 172.21.15.76:0/2931256799 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc018860 con 0x7f75d0102380 2024-03-12T05:55:03.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.466+0000 7f75d7c64640 1 -- 172.21.15.76:0/2931256799 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d019f860 con 0x7f75d0102380 2024-03-12T05:55:03.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.468+0000 7f75c6ffd640 1 -- 172.21.15.76:0/2931256799 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f75cc00fac0 con 0x7f75d0102380 2024-03-12T05:55:03.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.468+0000 7f75da839740 1 Processor -- start 2024-03-12T05:55:03.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.468+0000 7f75da839740 1 -- start start 2024-03-12T05:55:03.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.468+0000 7f75c6ffd640 1 --2- 172.21.15.76:0/2931256799 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75bc0a1b40 0x7f75bc0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75da839740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c09f80 0x555d50c0a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75da839740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c0a970 0x555d50c0f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75da839740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0fae0 0x555d50c11f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75da839740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d50bc5cd0 con 0x555d50c0fae0 2024-03-12T05:55:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75da839740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d50b78f20 con 0x555d50c09f80 2024-03-12T05:55:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75da839740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d50b76450 con 0x555d50c0a970 2024-03-12T05:55:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75d6201640 1 --2- 172.21.15.76:0/2931256799 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75bc0a1b40 0x7f75bc0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75c6ffd640 1 -- 172.21.15.76:0/2931256799 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f75cc0cd650 con 0x7f75d0102380 2024-03-12T05:55:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75d6201640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0fae0 0x555d50c11f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75d6201640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0fae0 0x555d50c11f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60416/0 (socket says 172.21.15.76:60416) 2024-03-12T05:55:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75d6201640 1 -- 172.21.15.76:0/743412630 learned_addr learned my addr 172.21.15.76:0/743412630 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75d51ff640 1 --2- 172.21.15.76:0/743412630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c0a970 0x555d50c0f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75d5a00640 1 --2- 172.21.15.76:0/743412630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c09f80 0x555d50c0a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75d6201640 1 -- 172.21.15.76:0/743412630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c0a970 msgr2=0x555d50c0f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75d6201640 1 --2- 172.21.15.76:0/743412630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c0a970 0x555d50c0f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75d6201640 1 -- 172.21.15.76:0/743412630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c09f80 msgr2=0x555d50c0a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75d6201640 1 --2- 172.21.15.76:0/743412630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c09f80 0x555d50c0a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75d6201640 1 -- 172.21.15.76:0/743412630 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d50b26f20 con 0x555d50c0fae0 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75d6201640 1 --2- 172.21.15.76:0/2931256799 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75bc0a1b40 0x7f75bc0a4000 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f75c8013b50 tx=0x7f75c800bd30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75d6201640 1 --2- 172.21.15.76:0/743412630 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0fae0 0x555d50c11f70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f75c8010860 tx=0x7f75c801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1132019d8f80ce0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75bbfff640 1 -- 172.21.15.76:0/743412630 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c8013270 con 0x555d50c0fae0 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75bbfff640 1 -- 172.21.15.76:0/743412630 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75c8019040 con 0x555d50c0fae0 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.469+0000 7f75bbfff640 1 -- 172.21.15.76:0/743412630 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c8004380 con 0x555d50c0fae0 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.470+0000 7f75da839740 1 -- 172.21.15.76:0/743412630 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0fae0 msgr2=0x555d50c11f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.470+0000 7f75da839740 1 --2- 172.21.15.76:0/743412630 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0fae0 0x555d50c11f70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f75c8010860 tx=0x7f75c801a000 comp rx=0 tx=0).stop 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.470+0000 7f75da839740 1 -- 172.21.15.76:0/743412630 shutdown_connections 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.470+0000 7f75da839740 1 --2- 172.21.15.76:0/743412630 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0fae0 0x555d50c11f70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.470+0000 7f75da839740 1 --2- 172.21.15.76:0/743412630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c0a970 0x555d50c0f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.470+0000 7f75da839740 1 --2- 172.21.15.76:0/743412630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c09f80 0x555d50c0a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.470+0000 7f75da839740 1 -- 172.21.15.76:0/743412630 >> 172.21.15.76:0/743412630 conn(0x555d50c054f0 msgr2=0x555d50c058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.470+0000 7f75da839740 1 -- 172.21.15.76:0/743412630 shutdown_connections 2024-03-12T05:55:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.470+0000 7f75da839740 1 -- 172.21.15.76:0/743412630 wait complete. 2024-03-12T05:55:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75da839740 1 -- start start 2024-03-12T05:55:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75da839740 1 Processor -- start 2024-03-12T05:55:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75da839740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c09f80 0x555d50c14160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75da839740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0a970 0x555d50c146a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75da839740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0fae0 0x555d50c1b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75da839740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d50b71c30 con 0x555d50c0fae0 2024-03-12T05:55:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75da839740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d50b16d30 con 0x555d50c0a970 2024-03-12T05:55:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75da839740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d50845310 con 0x555d50c09f80 2024-03-12T05:55:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75da839740 10 client.0 _ll_register_callbacks cb 0x555d50b1ed40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75d6201640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0fae0 0x555d50c1b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75d6201640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0fae0 0x555d50c1b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60426/0 (socket says 172.21.15.76:60426) 2024-03-12T05:55:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75d6201640 1 -- 172.21.15.76:0/127564194 learned_addr learned my addr 172.21.15.76:0/127564194 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75d51ff640 1 --2- 172.21.15.76:0/127564194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0a970 0x555d50c146a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75d5a00640 1 --2- 172.21.15.76:0/127564194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c09f80 0x555d50c14160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75d6201640 1 -- 172.21.15.76:0/127564194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c09f80 msgr2=0x555d50c14160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75d6201640 1 --2- 172.21.15.76:0/127564194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c09f80 0x555d50c14160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75d6201640 1 -- 172.21.15.76:0/127564194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0a970 msgr2=0x555d50c146a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75d6201640 1 --2- 172.21.15.76:0/127564194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0a970 0x555d50c146a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.471+0000 7f75d6201640 1 -- 172.21.15.76:0/127564194 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c8010b50 con 0x555d50c0fae0 2024-03-12T05:55:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75d6201640 1 --2- 172.21.15.76:0/127564194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0fae0 0x555d50c1b670 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f75c8014010 tx=0x7f75c8004830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=764dbad5791503cf server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75b97fa640 1 -- 172.21.15.76:0/127564194 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c800bac0 con 0x555d50c0fae0 2024-03-12T05:55:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75b97fa640 1 -- 172.21.15.76:0/127564194 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f75c8004c00 con 0x555d50c0fae0 2024-03-12T05:55:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75da839740 1 -- 172.21.15.76:0/127564194 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d508716c0 con 0x555d50c0fae0 2024-03-12T05:55:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75b97fa640 1 -- 172.21.15.76:0/127564194 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75c80132e0 con 0x555d50c0fae0 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75da839740 10 client.15102 fetch_fsmap learned FSMap version 21 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75da839740 1 -- 172.21.15.76:0/127564194 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x555d50b702b0 con 0x555d50c0fae0 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75b97fa640 1 -- 172.21.15.76:0/127564194 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555d508716c0 con 0x555d50c0fae0 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75b97fa640 1 -- 172.21.15.76:0/127564194 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 21) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f75c8038070 con 0x555d50c0fae0 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75da839740 10 client.15102 fetch_fsmap finished waiting for FSMap version 21 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75da839740 10 client.15102 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75da839740 20 client.15102 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75da839740 1 --2- 172.21.15.76:0/127564194 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x555d50c203c0 0x555d50c24e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75da839740 4 client.15102 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75da839740 1 -- 172.21.15.76:0/127564194 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d508c2270 con 0x555d50c203c0 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.472+0000 7f75d5a00640 1 --2- 172.21.15.76:0/127564194 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x555d50c203c0 0x555d50c24e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.473+0000 7f75b97fa640 10 client.15102 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.473+0000 7f75d5a00640 1 --2- 172.21.15.76:0/127564194 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x555d50c203c0 0x555d50c24e80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a4e4689d0b914ab6 server_cookie=713b43e68ee25085 in_seq=0 out_seq=0 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.473+0000 7f75b97fa640 1 -- 172.21.15.76:0/127564194 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555d508c2270 con 0x555d50c203c0 2024-03-12T05:55:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.474+0000 7f75b97fa640 10 client.15102 handle_command_reply: tid=0 2024-03-12T05:55:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.474+0000 7f75da839740 1 client.15102 shutdown 2024-03-12T05:55:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.474+0000 7f75da839740 1 -- 172.21.15.76:0/127564194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0fae0 msgr2=0x555d50c1b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.474+0000 7f75da839740 1 --2- 172.21.15.76:0/127564194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0fae0 0x555d50c1b670 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f75c8014010 tx=0x7f75c8004830 comp rx=0 tx=0).stop 2024-03-12T05:55:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.474+0000 7f75da839740 1 -- 172.21.15.76:0/127564194 shutdown_connections 2024-03-12T05:55:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.474+0000 7f75da839740 1 --2- 172.21.15.76:0/127564194 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x555d50c203c0 0x555d50c24e80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.474+0000 7f75da839740 1 --2- 172.21.15.76:0/127564194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0fae0 0x555d50c1b670 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.474+0000 7f75da839740 1 --2- 172.21.15.76:0/127564194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0a970 0x555d50c146a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.474+0000 7f75da839740 1 --2- 172.21.15.76:0/127564194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c09f80 0x555d50c14160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.474+0000 7f75b97fa640 0 client.15102 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:55:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.474+0000 7f75da839740 1 -- 172.21.15.76:0/127564194 >> 172.21.15.76:0/127564194 conn(0x555d50c054f0 msgr2=0x555d50c047d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.474+0000 7f75da839740 1 -- 172.21.15.76:0/127564194 shutdown_connections 2024-03-12T05:55:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.474+0000 7f75da839740 1 -- 172.21.15.76:0/127564194 wait complete. 2024-03-12T05:55:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.474+0000 7f75da839740 20 client.15102 trim_cache size 0 max 16384 2024-03-12T05:55:03.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:03 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3599220751' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:03.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:03 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3312617622' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:03.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:03 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2283398312' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:03.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:03 smithi088 ceph-mon[36585]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:55:03.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.489+0000 7f75da839740 1 Processor -- start 2024-03-12T05:55:03.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.489+0000 7f75da839740 1 -- start start 2024-03-12T05:55:03.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75da839740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c09f80 0x555d50c03f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75da839740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0a970 0x555d50c04480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75da839740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0fae0 0x555d50c28ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75da839740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d50af7490 con 0x555d50c0a970 2024-03-12T05:55:03.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75da839740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d50c1c2f0 con 0x555d50c0fae0 2024-03-12T05:55:03.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75da839740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d50c12100 con 0x555d50c09f80 2024-03-12T05:55:03.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75d51ff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0a970 0x555d50c04480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75d51ff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0a970 0x555d50c04480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60438/0 (socket says 172.21.15.76:60438) 2024-03-12T05:55:03.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75d51ff640 1 -- 172.21.15.76:0/2903111218 learned_addr learned my addr 172.21.15.76:0/2903111218 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75d6201640 1 --2- 172.21.15.76:0/2903111218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0fae0 0x555d50c28ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75d51ff640 1 -- 172.21.15.76:0/2903111218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c09f80 msgr2=0x555d50c03f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75d51ff640 1 --2- 172.21.15.76:0/2903111218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c09f80 0x555d50c03f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75d51ff640 1 -- 172.21.15.76:0/2903111218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0fae0 msgr2=0x555d50c28ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75d51ff640 1 --2- 172.21.15.76:0/2903111218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0fae0 0x555d50c28ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75d51ff640 1 -- 172.21.15.76:0/2903111218 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d50b0ca20 con 0x555d50c0a970 2024-03-12T05:55:03.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75d6201640 1 --2- 172.21.15.76:0/2903111218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0fae0 0x555d50c28ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:03.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75d51ff640 1 --2- 172.21.15.76:0/2903111218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0a970 0x555d50c04480 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f75cc09a400 tx=0x7f75cc096040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a1c614b31fe2f2c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75bb7fe640 1 -- 172.21.15.76:0/2903111218 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc09a470 con 0x555d50c0a970 2024-03-12T05:55:03.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75bb7fe640 1 -- 172.21.15.76:0/2903111218 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75cc09e5c0 con 0x555d50c0a970 2024-03-12T05:55:03.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.490+0000 7f75bb7fe640 1 -- 172.21.15.76:0/2903111218 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc086430 con 0x555d50c0a970 2024-03-12T05:55:03.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.491+0000 7f75da839740 1 -- 172.21.15.76:0/2903111218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0a970 msgr2=0x555d50c04480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.491+0000 7f75da839740 1 --2- 172.21.15.76:0/2903111218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0a970 0x555d50c04480 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f75cc09a400 tx=0x7f75cc096040 comp rx=0 tx=0).stop 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.491+0000 7f75da839740 1 -- 172.21.15.76:0/2903111218 shutdown_connections 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.491+0000 7f75da839740 1 --2- 172.21.15.76:0/2903111218 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0fae0 0x555d50c28ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.491+0000 7f75da839740 1 --2- 172.21.15.76:0/2903111218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c0a970 0x555d50c04480 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.491+0000 7f75da839740 1 --2- 172.21.15.76:0/2903111218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c09f80 0x555d50c03f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.491+0000 7f75da839740 1 -- 172.21.15.76:0/2903111218 >> 172.21.15.76:0/2903111218 conn(0x555d50c054f0 msgr2=0x555d50c0be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.491+0000 7f75da839740 1 -- 172.21.15.76:0/2903111218 shutdown_connections 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.491+0000 7f75da839740 1 -- 172.21.15.76:0/2903111218 wait complete. 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.491+0000 7f75da839740 1 -- start start 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.491+0000 7f75da839740 1 Processor -- start 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.491+0000 7f75da839740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c09f80 0x555d50c73160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.491+0000 7f75da839740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c0a970 0x555d50c736a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75da839740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0fae0 0x555d50c7a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75da839740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d50c1ff50 con 0x555d50c09f80 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75da839740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d50c7abb0 con 0x555d50c0fae0 2024-03-12T05:55:03.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75da839740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d50c7ad20 con 0x555d50c0a970 2024-03-12T05:55:03.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75da839740 10 client.0 _ll_register_callbacks cb 0x7f75c8013660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:03.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75d5a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c09f80 0x555d50c73160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75d5a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c09f80 0x555d50c73160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60452/0 (socket says 172.21.15.76:60452) 2024-03-12T05:55:03.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75d5a00640 1 -- 172.21.15.76:0/2004132247 learned_addr learned my addr 172.21.15.76:0/2004132247 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75d6201640 1 --2- 172.21.15.76:0/2004132247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0fae0 0x555d50c7a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75d51ff640 1 --2- 172.21.15.76:0/2004132247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c0a970 0x555d50c736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75d5a00640 1 -- 172.21.15.76:0/2004132247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c0a970 msgr2=0x555d50c736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75d5a00640 1 --2- 172.21.15.76:0/2004132247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c0a970 0x555d50c736a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75d5a00640 1 -- 172.21.15.76:0/2004132247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0fae0 msgr2=0x555d50c7a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75d5a00640 1 --2- 172.21.15.76:0/2004132247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0fae0 0x555d50c7a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75d5a00640 1 -- 172.21.15.76:0/2004132247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d019f860 con 0x555d50c09f80 2024-03-12T05:55:03.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75d6201640 1 --2- 172.21.15.76:0/2004132247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0fae0 0x555d50c7a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:03.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75d5a00640 1 --2- 172.21.15.76:0/2004132247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c09f80 0x555d50c73160 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009900 tx=0x7f75c002cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=876ba332b9a25a8c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:03.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75b97fa640 1 -- 172.21.15.76:0/2004132247 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c00047c0 con 0x555d50c09f80 2024-03-12T05:55:03.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75b97fa640 1 -- 172.21.15.76:0/2004132247 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f75c0045040 con 0x555d50c09f80 2024-03-12T05:55:03.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75da839740 1 -- 172.21.15.76:0/2004132247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75c8019040 con 0x555d50c09f80 2024-03-12T05:55:03.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75b97fa640 1 -- 172.21.15.76:0/2004132247 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75c0004920 con 0x555d50c09f80 2024-03-12T05:55:03.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75da839740 10 client.15108 fetch_fsmap learned FSMap version 21 2024-03-12T05:55:03.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75da839740 1 -- 172.21.15.76:0/2004132247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x555d50b32780 con 0x555d50c09f80 2024-03-12T05:55:03.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.492+0000 7f75b97fa640 1 -- 172.21.15.76:0/2004132247 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75c8019040 con 0x555d50c09f80 2024-03-12T05:55:03.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.493+0000 7f75b97fa640 1 -- 172.21.15.76:0/2004132247 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 21) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f75c003c040 con 0x555d50c09f80 2024-03-12T05:55:03.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.493+0000 7f75da839740 10 client.15108 fetch_fsmap finished waiting for FSMap version 21 2024-03-12T05:55:03.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.493+0000 7f75da839740 10 client.15108 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-12T05:55:03.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.493+0000 7f75da839740 20 client.15108 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:03.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.493+0000 7f75da839740 1 --2- 172.21.15.76:0/2004132247 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x555d50c800b0 0x555d50c7c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.493+0000 7f75da839740 4 client.15108 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.493+0000 7f75da839740 1 -- 172.21.15.76:0/2004132247 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555d508ae610 con 0x555d50c800b0 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.493+0000 7f75d51ff640 1 --2- 172.21.15.76:0/2004132247 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x555d50c800b0 0x555d50c7c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.493+0000 7f75b97fa640 10 client.15108 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.493+0000 7f75d51ff640 1 --2- 172.21.15.76:0/2004132247 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x555d50c800b0 0x555d50c7c4f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f0933fce643ae619 server_cookie=cdfb0b0c813bfae3 in_seq=0 out_seq=0 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.493+0000 7f75b97fa640 1 -- 172.21.15.76:0/2004132247 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x555d508ae610 con 0x555d50c800b0 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.493+0000 7f75b97fa640 10 client.15108 handle_command_reply: tid=0 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.493+0000 7f75da839740 1 client.15108 shutdown 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75da839740 1 -- 172.21.15.76:0/2004132247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c09f80 msgr2=0x555d50c73160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75da839740 1 --2- 172.21.15.76:0/2004132247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c09f80 0x555d50c73160 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009900 tx=0x7f75c002cb40 comp rx=0 tx=0).stop 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75da839740 1 -- 172.21.15.76:0/2004132247 shutdown_connections 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75da839740 1 --2- 172.21.15.76:0/2004132247 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x555d50c800b0 0x555d50c7c4f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75da839740 1 --2- 172.21.15.76:0/2004132247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555d50c0fae0 0x555d50c7a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75b97fa640 0 client.15108 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75da839740 1 --2- 172.21.15.76:0/2004132247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555d50c0a970 0x555d50c736a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75da839740 1 --2- 172.21.15.76:0/2004132247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555d50c09f80 0x555d50c73160 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75da839740 1 -- 172.21.15.76:0/2004132247 >> 172.21.15.76:0/2004132247 conn(0x555d50c054f0 msgr2=0x555d50c28ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75da839740 1 -- 172.21.15.76:0/2004132247 shutdown_connections 2024-03-12T05:55:03.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75da839740 1 -- 172.21.15.76:0/2004132247 wait complete. 2024-03-12T05:55:03.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75da839740 20 client.15108 trim_cache size 0 max 16384 2024-03-12T05:55:03.505 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:55:03.505 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "9f336de8-e033-11ee-95c8-87774f69a715", 2024-03-12T05:55:03.505 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 1, 2024-03-12T05:55:03.505 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24481, 2024-03-12T05:55:03.505 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-03-12T05:55:03.505 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-03-12T05:55:03.505 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-03-12T05:55:03.506 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 83.117031604999994, 2024-03-12T05:55:03.506 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 21, 2024-03-12T05:55:03.506 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 88, 2024-03-12T05:55:03.506 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 86, 2024-03-12T05:55:03.506 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 99.666240197999997 2024-03-12T05:55:03.506 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:55:03.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75d7c64640 1 -- 172.21.15.76:0/2931256799 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75bc0a1b40 msgr2=0x7f75bc0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75d7c64640 1 --2- 172.21.15.76:0/2931256799 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75bc0a1b40 0x7f75bc0a4000 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f75c8013b50 tx=0x7f75c800bd30 comp rx=0 tx=0).stop 2024-03-12T05:55:03.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75d7c64640 1 -- 172.21.15.76:0/2931256799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75d0102380 msgr2=0x7f75d0197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75d7c64640 1 --2- 172.21.15.76:0/2931256799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75d0102380 0x7f75d0197740 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f75cc00c920 tx=0x7f75cc00cde0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75d7c64640 1 -- 172.21.15.76:0/2931256799 shutdown_connections 2024-03-12T05:55:03.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75d7c64640 1 --2- 172.21.15.76:0/2931256799 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75bc0a1b40 0x7f75bc0a4000 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75d7c64640 1 --2- 172.21.15.76:0/2931256799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f75d0102d40 0x7f75d019e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75d7c64640 1 --2- 172.21.15.76:0/2931256799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75d0102380 0x7f75d0197740 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75d7c64640 1 --2- 172.21.15.76:0/2931256799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75d0101190 0x7f75d0197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75d7c64640 1 -- 172.21.15.76:0/2931256799 >> 172.21.15.76:0/2931256799 conn(0x7f75d00fc700 msgr2=0x7f75d0105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75d7c64640 1 -- 172.21.15.76:0/2931256799 shutdown_connections 2024-03-12T05:55:03.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.494+0000 7f75d7c64640 1 -- 172.21.15.76:0/2931256799 wait complete. 2024-03-12T05:55:03.509 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-12T05:55:03.628 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.608+0000 7fb81e6b5640 1 -- 172.21.15.76:0/3559836487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb818106f10 msgr2=0x7fb818109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.608+0000 7fb81e6b5640 1 --2- 172.21.15.76:0/3559836487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb818106f10 0x7fb818109300 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb810009ca0 tx=0x7fb81002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.608+0000 7fb81e6b5640 1 -- 172.21.15.76:0/3559836487 shutdown_connections 2024-03-12T05:55:03.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.608+0000 7fb81e6b5640 1 --2- 172.21.15.76:0/3559836487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb818106f10 0x7fb818109300 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.608+0000 7fb81e6b5640 1 --2- 172.21.15.76:0/3559836487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8181045e0 0x7fb8181069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.608+0000 7fb81e6b5640 1 --2- 172.21.15.76:0/3559836487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb818101cb0 0x7fb8181040a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.608+0000 7fb81e6b5640 1 -- 172.21.15.76:0/3559836487 >> 172.21.15.76:0/3559836487 conn(0x7fb8180fb5e0 msgr2=0x7fb8180fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.608+0000 7fb81e6b5640 1 -- 172.21.15.76:0/3559836487 shutdown_connections 2024-03-12T05:55:03.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.608+0000 7fb81e6b5640 1 -- 172.21.15.76:0/3559836487 wait complete. 2024-03-12T05:55:03.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.608+0000 7fb81e6b5640 1 Processor -- start 2024-03-12T05:55:03.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.608+0000 7fb81e6b5640 1 -- start start 2024-03-12T05:55:03.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb81e6b5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb818101cb0 0x7fb8181971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb81e6b5640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8181045e0 0x7fb8181976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb81e6b5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb818106f10 0x7fb81819e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb81e6b5640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81819eca0 con 0x7fb818101cb0 2024-03-12T05:55:03.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb81e6b5640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81819ee10 con 0x7fb8181045e0 2024-03-12T05:55:03.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb81e6b5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81819efd0 con 0x7fb818106f10 2024-03-12T05:55:03.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb817fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb818101cb0 0x7fb8181971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb817fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb818101cb0 0x7fb8181971b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60460/0 (socket says 172.21.15.76:60460) 2024-03-12T05:55:03.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb817fff640 1 -- 172.21.15.76:0/975933075 learned_addr learned my addr 172.21.15.76:0/975933075 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb8177fe640 1 --2- 172.21.15.76:0/975933075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8181045e0 0x7fb8181976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb81cc52640 1 --2- 172.21.15.76:0/975933075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb818106f10 0x7fb81819e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb817fff640 1 -- 172.21.15.76:0/975933075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb818106f10 msgr2=0x7fb81819e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb817fff640 1 --2- 172.21.15.76:0/975933075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb818106f10 0x7fb81819e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb817fff640 1 -- 172.21.15.76:0/975933075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8181045e0 msgr2=0x7fb8181976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb817fff640 1 --2- 172.21.15.76:0/975933075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8181045e0 0x7fb8181976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb817fff640 1 -- 172.21.15.76:0/975933075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb810009900 con 0x7fb818101cb0 2024-03-12T05:55:03.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb817fff640 1 --2- 172.21.15.76:0/975933075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb818101cb0 0x7fb8181971b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb80800c920 tx=0x7fb80800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb8157fa640 1 -- 172.21.15.76:0/975933075 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb808007bb0 con 0x7fb818101cb0 2024-03-12T05:55:03.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb81e6b5640 1 -- 172.21.15.76:0/975933075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81819f2b0 con 0x7fb818101cb0 2024-03-12T05:55:03.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb8157fa640 1 -- 172.21.15.76:0/975933075 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb808007d10 con 0x7fb818101cb0 2024-03-12T05:55:03.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb8157fa640 1 -- 172.21.15.76:0/975933075 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8080187c0 con 0x7fb818101cb0 2024-03-12T05:55:03.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.609+0000 7fb81e6b5640 1 -- 172.21.15.76:0/975933075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81819f7c0 con 0x7fb818101cb0 2024-03-12T05:55:03.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.611+0000 7fb8157fa640 1 -- 172.21.15.76:0/975933075 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fb80800f9c0 con 0x7fb818101cb0 2024-03-12T05:55:03.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb821363740 1 Processor -- start 2024-03-12T05:55:03.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb821363740 1 -- start start 2024-03-12T05:55:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb8157fa640 1 --2- 172.21.15.76:0/975933075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb8000a1ac0 0x7fb8000a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb821363740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b51f80 0x55d407b523a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb821363740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b52970 0x55d407b57470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb821363740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b57ae0 0x55d407b59f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb821363740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d407b0dcd0 con 0x55d407b52970 2024-03-12T05:55:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb821363740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d407ac0f20 con 0x55d407b57ae0 2024-03-12T05:55:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb821363740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d407abe450 con 0x55d407b51f80 2024-03-12T05:55:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb8177fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b52970 0x55d407b57470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb81cc52640 1 --2- 172.21.15.76:0/975933075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb8000a1ac0 0x7fb8000a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb8177fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b52970 0x55d407b57470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60466/0 (socket says 172.21.15.76:60466) 2024-03-12T05:55:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb8177fe640 1 -- 172.21.15.76:0/209935983 learned_addr learned my addr 172.21.15.76:0/209935983 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb8157fa640 1 -- 172.21.15.76:0/975933075 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb8080cd600 con 0x7fb818101cb0 2024-03-12T05:55:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb817fff640 1 --2- 172.21.15.76:0/209935983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b51f80 0x55d407b523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb81cc52640 1 --2- 172.21.15.76:0/209935983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b57ae0 0x55d407b59f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb8177fe640 1 -- 172.21.15.76:0/209935983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b51f80 msgr2=0x55d407b523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb8177fe640 1 --2- 172.21.15.76:0/209935983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b51f80 0x55d407b523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb8177fe640 1 -- 172.21.15.76:0/209935983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b57ae0 msgr2=0x55d407b59f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb8177fe640 1 --2- 172.21.15.76:0/209935983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b57ae0 0x55d407b59f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.612+0000 7fb8177fe640 1 -- 172.21.15.76:0/209935983 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d407a6ef20 con 0x55d407b52970 2024-03-12T05:55:03.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.613+0000 7fb81cc52640 1 --2- 172.21.15.76:0/975933075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb8000a1ac0 0x7fb8000a3f80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb810006010 tx=0x7fb810032ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.613+0000 7fb8177fe640 1 --2- 172.21.15.76:0/209935983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b52970 0x55d407b57470 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009a00 tx=0x7fb80c009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2484f768990aa677 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.613+0000 7fb7fe7fc640 1 -- 172.21.15.76:0/209935983 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c00eae0 con 0x55d407b52970 2024-03-12T05:55:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.613+0000 7fb7fe7fc640 1 -- 172.21.15.76:0/209935983 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb80c00ec40 con 0x55d407b52970 2024-03-12T05:55:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.613+0000 7fb7fe7fc640 1 -- 172.21.15.76:0/209935983 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c012630 con 0x55d407b52970 2024-03-12T05:55:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.613+0000 7fb821363740 1 -- 172.21.15.76:0/209935983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b52970 msgr2=0x55d407b57470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.613+0000 7fb821363740 1 --2- 172.21.15.76:0/209935983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b52970 0x55d407b57470 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009a00 tx=0x7fb80c009ec0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 -- 172.21.15.76:0/209935983 shutdown_connections 2024-03-12T05:55:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 --2- 172.21.15.76:0/209935983 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b57ae0 0x55d407b59f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 --2- 172.21.15.76:0/209935983 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b52970 0x55d407b57470 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 --2- 172.21.15.76:0/209935983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b51f80 0x55d407b523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 -- 172.21.15.76:0/209935983 >> 172.21.15.76:0/209935983 conn(0x55d407b4d4f0 msgr2=0x55d407b4d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 -- 172.21.15.76:0/209935983 shutdown_connections 2024-03-12T05:55:03.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 -- 172.21.15.76:0/209935983 wait complete. 2024-03-12T05:55:03.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 -- start start 2024-03-12T05:55:03.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 Processor -- start 2024-03-12T05:55:03.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b51f80 0x55d407b5c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 0x55d407b5c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b57ae0 0x55d407b636b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d407ab9c30 con 0x55d407b51f80 2024-03-12T05:55:03.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d407a5ed30 con 0x55d407b52970 2024-03-12T05:55:03.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40778d310 con 0x55d407b57ae0 2024-03-12T05:55:03.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb821363740 10 client.0 _ll_register_callbacks cb 0x55d407a66d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:03.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb817fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b51f80 0x55d407b5c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb817fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b51f80 0x55d407b5c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60476/0 (socket says 172.21.15.76:60476) 2024-03-12T05:55:03.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.614+0000 7fb817fff640 1 -- 172.21.15.76:0/791660717 learned_addr learned my addr 172.21.15.76:0/791660717 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb8177fe640 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 0x55d407b5c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb81cc52640 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b57ae0 0x55d407b636b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb817fff640 1 -- 172.21.15.76:0/791660717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b57ae0 msgr2=0x55d407b636b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb817fff640 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b57ae0 0x55d407b636b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb817fff640 1 -- 172.21.15.76:0/791660717 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 msgr2=0x55d407b5c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb817fff640 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 0x55d407b5c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb817fff640 1 -- 172.21.15.76:0/791660717 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80c009630 con 0x55d407b51f80 2024-03-12T05:55:03.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb81cc52640 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b57ae0 0x55d407b636b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:03.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb8177fe640 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 0x55d407b5c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:03.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb817fff640 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b51f80 0x55d407b5c130 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb808096880 tx=0x7fb80800fdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f96ec631c990a62 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb7f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:03.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb7f77fe640 1 -- 172.21.15.76:0/791660717 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8080b3150 con 0x55d407b51f80 2024-03-12T05:55:03.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb7f77fe640 1 -- 172.21.15.76:0/791660717 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb80809ab20 con 0x55d407b51f80 2024-03-12T05:55:03.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb821363740 1 -- 172.21.15.76:0/791660717 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4077b96c0 con 0x55d407b51f80 2024-03-12T05:55:03.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb7f77fe640 1 -- 172.21.15.76:0/791660717 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb808085070 con 0x55d407b51f80 2024-03-12T05:55:03.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb821363740 10 client.15120 fetch_fsmap learned FSMap version 21 2024-03-12T05:55:03.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb821363740 1 -- 172.21.15.76:0/791660717 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55d407ab82b0 con 0x55d407b51f80 2024-03-12T05:55:03.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb7f77fe640 1 -- 172.21.15.76:0/791660717 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d4077b96c0 con 0x55d407b51f80 2024-03-12T05:55:03.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb7f77fe640 1 -- 172.21.15.76:0/791660717 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 21) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb808016020 con 0x55d407b51f80 2024-03-12T05:55:03.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.615+0000 7fb821363740 10 client.15120 fetch_fsmap finished waiting for FSMap version 21 2024-03-12T05:55:03.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.616+0000 7fb821363740 10 client.15120 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-12T05:55:03.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.616+0000 7fb821363740 20 client.15120 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:03.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.616+0000 7fb821363740 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d407b68ab0 0x55d407b67d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.616+0000 7fb821363740 4 client.15120 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:55:03.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.616+0000 7fb821363740 1 -- 172.21.15.76:0/791660717 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d40780a270 con 0x55d407b68ab0 2024-03-12T05:55:03.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.616+0000 7fb8177fe640 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d407b68ab0 0x55d407b67d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.616+0000 7fb7f77fe640 10 client.15120 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:03.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.616+0000 7fb8177fe640 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d407b68ab0 0x55d407b67d30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2f33b6071b4f444 server_cookie=176acb7ceb0022ce in_seq=0 out_seq=0 2024-03-12T05:55:03.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.617+0000 7fb7f77fe640 1 -- 172.21.15.76:0/791660717 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d40780a270 con 0x55d407b68ab0 2024-03-12T05:55:03.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.617+0000 7fb7f77fe640 10 client.15120 handle_command_reply: tid=0 2024-03-12T05:55:03.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.617+0000 7fb821363740 1 client.15120 shutdown 2024-03-12T05:55:03.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.617+0000 7fb821363740 1 -- 172.21.15.76:0/791660717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b51f80 msgr2=0x55d407b5c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.617+0000 7fb821363740 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b51f80 0x55d407b5c130 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb808096880 tx=0x7fb80800fdc0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.617+0000 7fb821363740 1 -- 172.21.15.76:0/791660717 shutdown_connections 2024-03-12T05:55:03.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.617+0000 7fb821363740 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b57ae0 0x55d407b636b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.617+0000 7fb821363740 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 0x55d407b5c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.617+0000 7fb821363740 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d407b68ab0 0x55d407b67d30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.617+0000 7fb821363740 1 --2- 172.21.15.76:0/791660717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b51f80 0x55d407b5c130 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.617+0000 7fb7f77fe640 0 client.15120 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:03.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.617+0000 7fb821363740 1 -- 172.21.15.76:0/791660717 >> 172.21.15.76:0/791660717 conn(0x55d407b4d4f0 msgr2=0x55d407b4c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.617+0000 7fb821363740 1 -- 172.21.15.76:0/791660717 shutdown_connections 2024-03-12T05:55:03.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.618+0000 7fb821363740 1 -- 172.21.15.76:0/791660717 wait complete. 2024-03-12T05:55:03.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.618+0000 7fb821363740 20 client.15120 trim_cache size 0 max 16384 2024-03-12T05:55:03.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb821363740 1 Processor -- start 2024-03-12T05:55:03.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb821363740 1 -- start start 2024-03-12T05:55:03.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb821363740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b51f80 0x55d407b4d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb821363740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 0x55d407b63640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb821363740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b57ae0 0x55d407b5bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb821363740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d407a3f490 con 0x55d407b57ae0 2024-03-12T05:55:03.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb821363740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4077b96c0 con 0x55d407b52970 2024-03-12T05:55:03.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb821363740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d407ab2ad0 con 0x55d407b51f80 2024-03-12T05:55:03.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb81cc52640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b57ae0 0x55d407b5bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb81cc52640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b57ae0 0x55d407b5bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60478/0 (socket says 172.21.15.76:60478) 2024-03-12T05:55:03.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb81cc52640 1 -- 172.21.15.76:0/992601441 learned_addr learned my addr 172.21.15.76:0/992601441 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb8177fe640 1 --2- 172.21.15.76:0/992601441 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 0x55d407b63640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb81cc52640 1 -- 172.21.15.76:0/992601441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b51f80 msgr2=0x55d407b4d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb81cc52640 1 --2- 172.21.15.76:0/992601441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b51f80 0x55d407b4d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb81cc52640 1 -- 172.21.15.76:0/992601441 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 msgr2=0x55d407b63640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb81cc52640 1 --2- 172.21.15.76:0/992601441 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 0x55d407b63640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.633+0000 7fb81cc52640 1 -- 172.21.15.76:0/992601441 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d407a54a20 con 0x55d407b57ae0 2024-03-12T05:55:03.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.634+0000 7fb81cc52640 1 --2- 172.21.15.76:0/992601441 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b57ae0 0x55d407b5bd40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb8100325e0 tx=0x7fb810044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d249ee509e77984 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.634+0000 7fb7fdffb640 1 -- 172.21.15.76:0/992601441 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb810034c90 con 0x55d407b57ae0 2024-03-12T05:55:03.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.634+0000 7fb7fdffb640 1 -- 172.21.15.76:0/992601441 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb810034df0 con 0x55d407b57ae0 2024-03-12T05:55:03.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.634+0000 7fb7fdffb640 1 -- 172.21.15.76:0/992601441 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb810004570 con 0x55d407b57ae0 2024-03-12T05:55:03.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.634+0000 7fb821363740 1 -- 172.21.15.76:0/992601441 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b57ae0 msgr2=0x55d407b5bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.634+0000 7fb821363740 1 --2- 172.21.15.76:0/992601441 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b57ae0 0x55d407b5bd40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb8100325e0 tx=0x7fb810044000 comp rx=0 tx=0).stop 2024-03-12T05:55:03.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.634+0000 7fb821363740 1 -- 172.21.15.76:0/992601441 shutdown_connections 2024-03-12T05:55:03.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.634+0000 7fb821363740 1 --2- 172.21.15.76:0/992601441 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b57ae0 0x55d407b5bd40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.634+0000 7fb821363740 1 --2- 172.21.15.76:0/992601441 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 0x55d407b63640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.634+0000 7fb821363740 1 --2- 172.21.15.76:0/992601441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b51f80 0x55d407b4d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.634+0000 7fb821363740 1 -- 172.21.15.76:0/992601441 >> 172.21.15.76:0/992601441 conn(0x55d407b4d4f0 msgr2=0x55d407b53410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.634+0000 7fb821363740 1 -- 172.21.15.76:0/992601441 shutdown_connections 2024-03-12T05:55:03.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.634+0000 7fb821363740 1 -- 172.21.15.76:0/992601441 wait complete. 2024-03-12T05:55:03.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb821363740 1 -- start start 2024-03-12T05:55:03.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb821363740 1 Processor -- start 2024-03-12T05:55:03.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb821363740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b51f80 0x55d407b60bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb821363740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 0x55d407b61130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb821363740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b57ae0 0x55d407b62140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb821363740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d407b62680 con 0x55d407b51f80 2024-03-12T05:55:03.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb821363740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d407b627f0 con 0x55d407b52970 2024-03-12T05:55:03.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb821363740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d407b629b0 con 0x55d407b57ae0 2024-03-12T05:55:03.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb821363740 10 client.0 _ll_register_callbacks cb 0x7fb80c009420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:03.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb817fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b51f80 0x55d407b60bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb817fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b51f80 0x55d407b60bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60480/0 (socket says 172.21.15.76:60480) 2024-03-12T05:55:03.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb817fff640 1 -- 172.21.15.76:0/4240435323 learned_addr learned my addr 172.21.15.76:0/4240435323 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:03.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb8177fe640 1 --2- 172.21.15.76:0/4240435323 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 0x55d407b61130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb817fff640 1 -- 172.21.15.76:0/4240435323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b57ae0 msgr2=0x55d407b62140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb817fff640 1 --2- 172.21.15.76:0/4240435323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b57ae0 0x55d407b62140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb817fff640 1 -- 172.21.15.76:0/4240435323 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 msgr2=0x55d407b61130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:03.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb817fff640 1 --2- 172.21.15.76:0/4240435323 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 0x55d407b61130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb817fff640 1 -- 172.21.15.76:0/4240435323 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb818109ec0 con 0x55d407b51f80 2024-03-12T05:55:03.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb8177fe640 1 --2- 172.21.15.76:0/4240435323 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 0x55d407b61130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:03.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb817fff640 1 --2- 172.21.15.76:0/4240435323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b51f80 0x55d407b60bf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb8080b3be0 tx=0x7fb8080cb440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b648dc6bfbca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:03.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.635+0000 7fb7f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:03.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb7f77fe640 1 -- 172.21.15.76:0/4240435323 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8080850e0 con 0x55d407b51f80 2024-03-12T05:55:03.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb7f77fe640 1 -- 172.21.15.76:0/4240435323 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb80809ab20 con 0x55d407b51f80 2024-03-12T05:55:03.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb821363740 1 -- 172.21.15.76:0/4240435323 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d407b687c0 con 0x55d407b51f80 2024-03-12T05:55:03.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb7f77fe640 1 -- 172.21.15.76:0/4240435323 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8080b34b0 con 0x55d407b51f80 2024-03-12T05:55:03.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb821363740 10 client.15126 fetch_fsmap learned FSMap version 21 2024-03-12T05:55:03.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb821363740 1 -- 172.21.15.76:0/4240435323 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x55d407a7a780 con 0x55d407b51f80 2024-03-12T05:55:03.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb7f77fe640 1 -- 172.21.15.76:0/4240435323 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb80807d080 con 0x55d407b51f80 2024-03-12T05:55:03.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb7f77fe640 1 -- 172.21.15.76:0/4240435323 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 21) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb808016070 con 0x55d407b51f80 2024-03-12T05:55:03.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb821363740 10 client.15126 fetch_fsmap finished waiting for FSMap version 21 2024-03-12T05:55:03.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb821363740 10 client.15126 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-12T05:55:03.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb821363740 20 client.15126 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:03.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb821363740 1 --2- 172.21.15.76:0/4240435323 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d407bc4410 0x55d407bc0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:03.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb821363740 4 client.15126 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:55:03.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb821363740 1 -- 172.21.15.76:0/4240435323 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fb80c00ec40 con 0x55d407bc4410 2024-03-12T05:55:03.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.636+0000 7fb8177fe640 1 --2- 172.21.15.76:0/4240435323 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d407bc4410 0x55d407bc0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:03.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb7f77fe640 10 client.15126 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:03.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb8177fe640 1 --2- 172.21.15.76:0/4240435323 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d407bc4410 0x55d407bc0850 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61092c2f3e447e5d server_cookie=d13160c33ccd6a15 in_seq=0 out_seq=0 2024-03-12T05:55:03.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb7f77fe640 1 -- 172.21.15.76:0/4240435323 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x7fb80c00ec40 con 0x55d407bc4410 2024-03-12T05:55:03.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb7f77fe640 10 client.15126 handle_command_reply: tid=0 2024-03-12T05:55:03.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb821363740 1 client.15126 shutdown 2024-03-12T05:55:03.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb821363740 1 -- 172.21.15.76:0/4240435323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b51f80 msgr2=0x55d407b60bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb821363740 1 --2- 172.21.15.76:0/4240435323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b51f80 0x55d407b60bf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb8080b3be0 tx=0x7fb8080cb440 comp rx=0 tx=0).stop 2024-03-12T05:55:03.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb821363740 1 -- 172.21.15.76:0/4240435323 shutdown_connections 2024-03-12T05:55:03.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb821363740 1 --2- 172.21.15.76:0/4240435323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d407b57ae0 0x55d407b62140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb821363740 1 --2- 172.21.15.76:0/4240435323 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d407b52970 0x55d407b61130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb821363740 1 --2- 172.21.15.76:0/4240435323 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d407bc4410 0x55d407bc0850 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb821363740 1 --2- 172.21.15.76:0/4240435323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d407b51f80 0x55d407b60bf0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb7f77fe640 0 client.15126 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:03.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb821363740 1 -- 172.21.15.76:0/4240435323 >> 172.21.15.76:0/4240435323 conn(0x55d407b4d4f0 msgr2=0x55d407b4d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb821363740 1 -- 172.21.15.76:0/4240435323 shutdown_connections 2024-03-12T05:55:03.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb821363740 1 -- 172.21.15.76:0/4240435323 wait complete. 2024-03-12T05:55:03.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.637+0000 7fb821363740 20 client.15126 trim_cache size 0 max 16384 2024-03-12T05:55:03.664 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:55:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "9f336de8-e033-11ee-95c8-87774f69a715", 2024-03-12T05:55:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 0, 2024-03-12T05:55:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24518, 2024-03-12T05:55:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-03-12T05:55:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-03-12T05:55:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-03-12T05:55:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 85.275727852000003, 2024-03-12T05:55:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 21, 2024-03-12T05:55:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 88, 2024-03-12T05:55:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 86, 2024-03-12T05:55:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 91.491471109000003 2024-03-12T05:55:03.666 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:55:03.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.638+0000 7fb81e6b5640 1 -- 172.21.15.76:0/975933075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb8000a1ac0 msgr2=0x7fb8000a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.638+0000 7fb81e6b5640 1 --2- 172.21.15.76:0/975933075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb8000a1ac0 0x7fb8000a3f80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb810006010 tx=0x7fb810032ef0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.638+0000 7fb81e6b5640 1 -- 172.21.15.76:0/975933075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb818101cb0 msgr2=0x7fb8181971b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:03.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.638+0000 7fb81e6b5640 1 --2- 172.21.15.76:0/975933075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb818101cb0 0x7fb8181971b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb80800c920 tx=0x7fb80800cd20 comp rx=0 tx=0).stop 2024-03-12T05:55:03.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.638+0000 7fb81e6b5640 1 -- 172.21.15.76:0/975933075 shutdown_connections 2024-03-12T05:55:03.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.638+0000 7fb81e6b5640 1 --2- 172.21.15.76:0/975933075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb8000a1ac0 0x7fb8000a3f80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.638+0000 7fb81e6b5640 1 --2- 172.21.15.76:0/975933075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb818106f10 0x7fb81819e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.638+0000 7fb81e6b5640 1 --2- 172.21.15.76:0/975933075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8181045e0 0x7fb8181976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.638+0000 7fb81e6b5640 1 --2- 172.21.15.76:0/975933075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb818101cb0 0x7fb8181971b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:03.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.638+0000 7fb81e6b5640 1 -- 172.21.15.76:0/975933075 >> 172.21.15.76:0/975933075 conn(0x7fb8180fb5e0 msgr2=0x7fb8181025f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:03.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.638+0000 7fb81e6b5640 1 -- 172.21.15.76:0/975933075 shutdown_connections 2024-03-12T05:55:03.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:03.638+0000 7fb81e6b5640 1 -- 172.21.15.76:0/975933075 wait complete. 2024-03-12T05:55:03.668 INFO:tasks.cephadm:Setup complete, yielding 2024-03-12T05:55:03.669 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T05:55:03.683 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2024-03-12T05:55:03.684 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-12T05:55:03.923 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:55:04.305 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:04 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3434864429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:04.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:04 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3434864429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:04.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:04 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3434864429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:05.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:05 smithi082 ceph-mon[36382]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-12T05:55:05.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:05 smithi076 ceph-mon[32095]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-12T05:55:05.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:05 smithi088 ceph-mon[36585]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-12T05:55:05.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.677+0000 7f47b8e24700 1 -- 172.21.15.76:0/862085736 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47b4103840 msgr2=0x7f47b410a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:05.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.677+0000 7f47b8e24700 1 --2- 172.21.15.76:0/862085736 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47b4103840 0x7f47b410a0f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f47ac00b0d0 tx=0x7f47ac00b3e0 comp rx=0 tx=0).stop 2024-03-12T05:55:05.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.677+0000 7f47b8e24700 1 -- 172.21.15.76:0/862085736 shutdown_connections 2024-03-12T05:55:05.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.677+0000 7f47b8e24700 1 --2- 172.21.15.76:0/862085736 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47b4103840 0x7f47b410a0f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:05.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.677+0000 7f47b8e24700 1 --2- 172.21.15.76:0/862085736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47b4101ca0 0x7f47b41020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:05.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.677+0000 7f47b8e24700 1 --2- 172.21.15.76:0/862085736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47b4102ea0 0x7f47b4103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:05.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.677+0000 7f47b8e24700 1 -- 172.21.15.76:0/862085736 >> 172.21.15.76:0/862085736 conn(0x7f47b40fd280 msgr2=0x7f47b40ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:05.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.677+0000 7f47b8e24700 1 -- 172.21.15.76:0/862085736 shutdown_connections 2024-03-12T05:55:05.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.677+0000 7f47b8e24700 1 -- 172.21.15.76:0/862085736 wait complete. 2024-03-12T05:55:05.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.678+0000 7f47b8e24700 1 Processor -- start 2024-03-12T05:55:05.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.678+0000 7f47b8e24700 1 -- start start 2024-03-12T05:55:05.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.678+0000 7f47b8e24700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47b4101ca0 0x7f47b4197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:05.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.678+0000 7f47b8e24700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47b4102ea0 0x7f47b4198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:05.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.678+0000 7f47b8e24700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47b4103840 0x7f47b419f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:05.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.678+0000 7f47b8e24700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b419f8c0 con 0x7f47b4103840 2024-03-12T05:55:05.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.678+0000 7f47b8e24700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b419fa00 con 0x7f47b4101ca0 2024-03-12T05:55:05.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.678+0000 7f47b8e24700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b419fb90 con 0x7f47b4102ea0 2024-03-12T05:55:05.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.678+0000 7f47b2d9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47b4103840 0x7f47b419f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:05.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.678+0000 7f47b2d9d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47b4103840 0x7f47b419f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60498/0 (socket says 172.21.15.76:60498) 2024-03-12T05:55:05.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.678+0000 7f47b2d9d700 1 -- 172.21.15.76:0/1667285744 learned_addr learned my addr 172.21.15.76:0/1667285744 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:05.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.678+0000 7f47b259c700 1 --2- 172.21.15.76:0/1667285744 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47b4101ca0 0x7f47b4197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:05.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.679+0000 7f47b1d9b700 1 --2- 172.21.15.76:0/1667285744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47b4102ea0 0x7f47b4198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:05.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.679+0000 7f47b2d9d700 1 -- 172.21.15.76:0/1667285744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47b4102ea0 msgr2=0x7f47b4198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:05.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.679+0000 7f47b2d9d700 1 --2- 172.21.15.76:0/1667285744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47b4102ea0 0x7f47b4198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:05.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.679+0000 7f47b2d9d700 1 -- 172.21.15.76:0/1667285744 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47b4101ca0 msgr2=0x7f47b4197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:05.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.679+0000 7f47b2d9d700 1 --2- 172.21.15.76:0/1667285744 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47b4101ca0 0x7f47b4197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:05.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.679+0000 7f47b2d9d700 1 -- 172.21.15.76:0/1667285744 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ac009d00 con 0x7f47b4103840 2024-03-12T05:55:05.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.679+0000 7f47b2d9d700 1 --2- 172.21.15.76:0/1667285744 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47b4103840 0x7f47b419f2f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009380 tx=0x7f47ac0093b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:05.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.679+0000 7f47a37fe700 1 -- 172.21.15.76:0/1667285744 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac00e040 con 0x7f47b4103840 2024-03-12T05:55:05.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.679+0000 7f47a37fe700 1 -- 172.21.15.76:0/1667285744 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47ac01dd70 con 0x7f47b4103840 2024-03-12T05:55:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.679+0000 7f47b8e24700 1 -- 172.21.15.76:0/1667285744 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b419fe10 con 0x7f47b4103840 2024-03-12T05:55:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.679+0000 7f47a37fe700 1 -- 172.21.15.76:0/1667285744 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac01bda0 con 0x7f47b4103840 2024-03-12T05:55:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.679+0000 7f47b8e24700 1 -- 172.21.15.76:0/1667285744 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b41a0220 con 0x7f47b4103840 2024-03-12T05:55:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.680+0000 7f47b8e24700 1 -- 172.21.15.76:0/1667285744 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b4109000 con 0x7f47b4103840 2024-03-12T05:55:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.681+0000 7f47a37fe700 1 -- 172.21.15.76:0/1667285744 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7f47ac01d890 con 0x7f47b4103840 2024-03-12T05:55:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.682+0000 7f47a37fe700 1 --2- 172.21.15.76:0/1667285744 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f479c0a1b50 0x7f479c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.682+0000 7f47a37fe700 1 -- 172.21.15.76:0/1667285744 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f47ac0cf1c0 con 0x7f47b4103840 2024-03-12T05:55:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.682+0000 7f47b259c700 1 --2- 172.21.15.76:0/1667285744 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f479c0a1b50 0x7f479c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:05.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.682+0000 7f47b259c700 1 --2- 172.21.15.76:0/1667285744 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f479c0a1b50 0x7f479c0a4010 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f47b4102d00 tx=0x7f47a800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:05.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.686+0000 7f47a37fe700 1 -- 172.21.15.76:0/1667285744 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f47ac098310 con 0x7f47b4103840 2024-03-12T05:55:05.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.897+0000 7f47b8e24700 1 -- 172.21.15.76:0/1667285744 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f47b402d950 con 0x7f479c0a1b50 2024-03-12T05:55:05.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.898+0000 7f47a37fe700 1 -- 172.21.15.76:0/1667285744 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f47b402d950 con 0x7f479c0a1b50 2024-03-12T05:55:05.902 INFO:teuthology.orchestra.run.smithi076.stdout:Backend: cephadm 2024-03-12T05:55:05.902 INFO:teuthology.orchestra.run.smithi076.stdout:Available: Yes 2024-03-12T05:55:05.902 INFO:teuthology.orchestra.run.smithi076.stdout:Paused: No 2024-03-12T05:55:05.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.900+0000 7f47b8e24700 1 -- 172.21.15.76:0/1667285744 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f479c0a1b50 msgr2=0x7f479c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:05.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.900+0000 7f47b8e24700 1 --2- 172.21.15.76:0/1667285744 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f479c0a1b50 0x7f479c0a4010 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f47b4102d00 tx=0x7f47a800b410 comp rx=0 tx=0).stop 2024-03-12T05:55:05.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.900+0000 7f47b8e24700 1 -- 172.21.15.76:0/1667285744 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47b4103840 msgr2=0x7f47b419f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:05.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.900+0000 7f47b8e24700 1 --2- 172.21.15.76:0/1667285744 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47b4103840 0x7f47b419f2f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009380 tx=0x7f47ac0093b0 comp rx=0 tx=0).stop 2024-03-12T05:55:05.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.900+0000 7f47b8e24700 1 -- 172.21.15.76:0/1667285744 shutdown_connections 2024-03-12T05:55:05.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.900+0000 7f47b8e24700 1 --2- 172.21.15.76:0/1667285744 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f479c0a1b50 0x7f479c0a4010 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:05.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.900+0000 7f47b8e24700 1 --2- 172.21.15.76:0/1667285744 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f47b4103840 0x7f47b419f2f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:05.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.900+0000 7f47b8e24700 1 --2- 172.21.15.76:0/1667285744 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f47b4101ca0 0x7f47b4197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:05.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.900+0000 7f47b8e24700 1 --2- 172.21.15.76:0/1667285744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47b4102ea0 0x7f47b4198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:05.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.900+0000 7f47b8e24700 1 -- 172.21.15.76:0/1667285744 >> 172.21.15.76:0/1667285744 conn(0x7f47b40fd280 msgr2=0x7f47b4106140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:05.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.901+0000 7f47b8e24700 1 -- 172.21.15.76:0/1667285744 shutdown_connections 2024-03-12T05:55:05.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:05.901+0000 7f47b8e24700 1 -- 172.21.15.76:0/1667285744 wait complete. 2024-03-12T05:55:06.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:06 smithi082 ceph-mon[36382]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:06.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:06 smithi088 ceph-mon[36585]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:06.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:06 smithi076 ceph-mon[32095]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:06.568 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-12T05:55:06.821 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:55:07.211 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:07 smithi076 ceph-mon[32095]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:55:07.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:07 smithi082 ceph-mon[36382]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:55:07.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:07 smithi088 ceph-mon[36585]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:55:08.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.497+0000 7fab504ac700 1 -- 172.21.15.76:0/3296448577 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab48101dd0 msgr2=0x7fab48102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:08.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.497+0000 7fab504ac700 1 --2- 172.21.15.76:0/3296448577 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab48101dd0 0x7fab48102230 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fab380099c0 tx=0x7fab38009cd0 comp rx=0 tx=0).stop 2024-03-12T05:55:08.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 -- 172.21.15.76:0/3296448577 shutdown_connections 2024-03-12T05:55:08.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 --2- 172.21.15.76:0/3296448577 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab48102770 0x7fab48109020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:08.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 --2- 172.21.15.76:0/3296448577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab48100bd0 0x7fab48100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:08.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 --2- 172.21.15.76:0/3296448577 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab48101dd0 0x7fab48102230 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:08.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 -- 172.21.15.76:0/3296448577 >> 172.21.15.76:0/3296448577 conn(0x7fab480fc190 msgr2=0x7fab480fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:08.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 -- 172.21.15.76:0/3296448577 shutdown_connections 2024-03-12T05:55:08.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 -- 172.21.15.76:0/3296448577 wait complete. 2024-03-12T05:55:08.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 Processor -- start 2024-03-12T05:55:08.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 -- start start 2024-03-12T05:55:08.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab48100bd0 0x7fab48199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:08.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab48101dd0 0x7fab4819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:08.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab48102770 0x7fab481a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:08.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab481a1940 con 0x7fab48101dd0 2024-03-12T05:55:08.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab481a1a80 con 0x7fab48100bd0 2024-03-12T05:55:08.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab504ac700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab481a1c10 con 0x7fab48102770 2024-03-12T05:55:08.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab4da47700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab48101dd0 0x7fab4819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:08.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.498+0000 7fab4da47700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab48101dd0 0x7fab4819a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60502/0 (socket says 172.21.15.76:60502) 2024-03-12T05:55:08.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.499+0000 7fab4da47700 1 -- 172.21.15.76:0/1791444912 learned_addr learned my addr 172.21.15.76:0/1791444912 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:08.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.499+0000 7fab4ea49700 1 --2- 172.21.15.76:0/1791444912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab48102770 0x7fab481a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:08.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.499+0000 7fab4e248700 1 --2- 172.21.15.76:0/1791444912 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab48100bd0 0x7fab48199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:08.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.499+0000 7fab4da47700 1 -- 172.21.15.76:0/1791444912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab48102770 msgr2=0x7fab481a1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:08.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.499+0000 7fab4da47700 1 --2- 172.21.15.76:0/1791444912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab48102770 0x7fab481a1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:08.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.499+0000 7fab4da47700 1 -- 172.21.15.76:0/1791444912 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab48100bd0 msgr2=0x7fab48199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:08.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.499+0000 7fab4da47700 1 --2- 172.21.15.76:0/1791444912 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab48100bd0 0x7fab48199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:08.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.499+0000 7fab4da47700 1 -- 172.21.15.76:0/1791444912 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab380095f0 con 0x7fab48101dd0 2024-03-12T05:55:08.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.499+0000 7fab4da47700 1 --2- 172.21.15.76:0/1791444912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab48101dd0 0x7fab4819a2f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fab38005910 tx=0x7fab38005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:08.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.499+0000 7fab3f7fe700 1 -- 172.21.15.76:0/1791444912 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3801d070 con 0x7fab48101dd0 2024-03-12T05:55:08.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.499+0000 7fab504ac700 1 -- 172.21.15.76:0/1791444912 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab481a1e90 con 0x7fab48101dd0 2024-03-12T05:55:08.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.499+0000 7fab3f7fe700 1 -- 172.21.15.76:0/1791444912 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab38004050 con 0x7fab48101dd0 2024-03-12T05:55:08.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.499+0000 7fab3f7fe700 1 -- 172.21.15.76:0/1791444912 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab38022d90 con 0x7fab48101dd0 2024-03-12T05:55:08.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.499+0000 7fab504ac700 1 -- 172.21.15.76:0/1791444912 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab481a22a0 con 0x7fab48101dd0 2024-03-12T05:55:08.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.500+0000 7fab504ac700 1 -- 172.21.15.76:0/1791444912 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4810c510 con 0x7fab48101dd0 2024-03-12T05:55:08.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.501+0000 7fab3f7fe700 1 -- 172.21.15.76:0/1791444912 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fab38022470 con 0x7fab48101dd0 2024-03-12T05:55:08.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.502+0000 7fab3f7fe700 1 --2- 172.21.15.76:0/1791444912 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fab340a1b50 0x7fab340a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:08.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.502+0000 7fab4e248700 1 --2- 172.21.15.76:0/1791444912 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fab340a1b50 0x7fab340a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:08.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.502+0000 7fab3f7fe700 1 -- 172.21.15.76:0/1791444912 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fab380ce980 con 0x7fab48101dd0 2024-03-12T05:55:08.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.502+0000 7fab4e248700 1 --2- 172.21.15.76:0/1791444912 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fab340a1b50 0x7fab340a4010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fab44005950 tx=0x7fab440058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:08.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.505+0000 7fab3f7fe700 1 -- 172.21.15.76:0/1791444912 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fab38097ad0 con 0x7fab48101dd0 2024-03-12T05:55:08.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.710+0000 7fab504ac700 1 -- 172.21.15.76:0/1791444912 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fab480fe310 con 0x7fab340a1b50 2024-03-12T05:55:08.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.723+0000 7fab3f7fe700 1 -- 172.21.15.76:0/1791444912 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7fab480fe310 con 0x7fab340a1b50 2024-03-12T05:55:08.728 INFO:teuthology.orchestra.run.smithi076.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T05:55:08.728 INFO:teuthology.orchestra.run.smithi076.stdout:mds.a smithi076 running (99s) 16s ago 99s 15.2M - 19.0.0-1607-g94a53970 a6477a474d71 c5ac474888b8 2024-03-12T05:55:08.728 INFO:teuthology.orchestra.run.smithi076.stdout:mds.b smithi082 running (96s) 30s ago 96s 28.0M - 19.0.0-1607-g94a53970 a6477a474d71 1336ba7664a1 2024-03-12T05:55:08.728 INFO:teuthology.orchestra.run.smithi076.stdout:mds.c smithi088 running (110s) 30s ago 110s 15.7M - 19.0.0-1607-g94a53970 a6477a474d71 4550fbbd722a 2024-03-12T05:55:08.728 INFO:teuthology.orchestra.run.smithi076.stdout:mds.d smithi076 running (118s) 16s ago 117s 15.4M - 19.0.0-1607-g94a53970 a6477a474d71 89bf230cda25 2024-03-12T05:55:08.728 INFO:teuthology.orchestra.run.smithi076.stdout:mds.e smithi082 running (115s) 30s ago 115s 16.9M - 19.0.0-1607-g94a53970 a6477a474d71 862c71a1e1df 2024-03-12T05:55:08.728 INFO:teuthology.orchestra.run.smithi076.stdout:mds.f smithi088 running (113s) 30s ago 112s 19.8M - 19.0.0-1607-g94a53970 a6477a474d71 e1fdcaa76ae5 2024-03-12T05:55:08.728 INFO:teuthology.orchestra.run.smithi076.stdout:mds.g smithi076 running (102s) 16s ago 101s 15.2M - 19.0.0-1607-g94a53970 a6477a474d71 0c425e959729 2024-03-12T05:55:08.728 INFO:teuthology.orchestra.run.smithi076.stdout:mds.h smithi082 running (107s) 30s ago 107s 20.2M - 19.0.0-1607-g94a53970 a6477a474d71 e32cf20a5ee1 2024-03-12T05:55:08.728 INFO:teuthology.orchestra.run.smithi076.stdout:mds.i smithi088 running (105s) 30s ago 104s 16.7M - 19.0.0-1607-g94a53970 a6477a474d71 0f4ae941a28f 2024-03-12T05:55:08.728 INFO:teuthology.orchestra.run.smithi076.stdout:mds.j smithi076 running (2m) 16s ago 2m 12.9M - 19.0.0-1607-g94a53970 a6477a474d71 7656e4ec3d6a 2024-03-12T05:55:08.728 INFO:teuthology.orchestra.run.smithi076.stdout:mds.k smithi082 running (2m) 30s ago 2m 13.6M - 19.0.0-1607-g94a53970 a6477a474d71 3120ec94420a 2024-03-12T05:55:08.729 INFO:teuthology.orchestra.run.smithi076.stdout:mds.l smithi088 running (2m) 30s ago 2m 13.9M - 19.0.0-1607-g94a53970 a6477a474d71 5caf516650fa 2024-03-12T05:55:08.729 INFO:teuthology.orchestra.run.smithi076.stdout:mgr.x smithi076 *:9283,8765 running (9m) 16s ago 9m 512M - 19.0.0-1607-g94a53970 a6477a474d71 04d3f8c04590 2024-03-12T05:55:08.729 INFO:teuthology.orchestra.run.smithi076.stdout:mgr.y smithi082 *:8443,8765 running (7m) 30s ago 7m 453M - 19.0.0-1607-g94a53970 a6477a474d71 60fa68512286 2024-03-12T05:55:08.729 INFO:teuthology.orchestra.run.smithi076.stdout:mgr.z smithi088 *:8443,8765 running (7m) 30s ago 7m 450M - 19.0.0-1607-g94a53970 a6477a474d71 89c69d18a0a8 2024-03-12T05:55:08.729 INFO:teuthology.orchestra.run.smithi076.stdout:mon.a smithi076 running (9m) 16s ago 9m 50.8M 2048M 19.0.0-1607-g94a53970 a6477a474d71 9c347f27958b 2024-03-12T05:55:08.729 INFO:teuthology.orchestra.run.smithi076.stdout:mon.b smithi082 running (7m) 30s ago 7m 39.5M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e81f0cd76ab4 2024-03-12T05:55:08.729 INFO:teuthology.orchestra.run.smithi076.stdout:mon.c smithi088 running (7m) 30s ago 7m 38.2M 2048M 19.0.0-1607-g94a53970 a6477a474d71 9ca03031be8e 2024-03-12T05:55:08.729 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0 smithi076 running (6m) 16s ago 6m 58.9M 4272M 19.0.0-1607-g94a53970 a6477a474d71 0c7ed207da59 2024-03-12T05:55:08.729 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1 smithi082 running (6m) 30s ago 6m 55.5M 4270M 19.0.0-1607-g94a53970 a6477a474d71 85f29769d341 2024-03-12T05:55:08.729 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2 smithi088 running (6m) 30s ago 6m 50.2M 4270M 19.0.0-1607-g94a53970 a6477a474d71 87243c63e328 2024-03-12T05:55:08.729 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 smithi076 running (5m) 16s ago 5m 46.8M 4272M 19.0.0-1607-g94a53970 a6477a474d71 59da77640409 2024-03-12T05:55:08.729 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4 smithi082 running (5m) 30s ago 5m 52.1M 4270M 19.0.0-1607-g94a53970 a6477a474d71 23b934fbe8ee 2024-03-12T05:55:08.729 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 smithi088 running (4m) 30s ago 4m 46.6M 4270M 19.0.0-1607-g94a53970 a6477a474d71 36471a80adc5 2024-03-12T05:55:08.729 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6 smithi076 running (4m) 16s ago 4m 53.9M 4272M 19.0.0-1607-g94a53970 a6477a474d71 e4f91a3d2dca 2024-03-12T05:55:08.730 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7 smithi082 running (4m) 30s ago 4m 53.5M 4270M 19.0.0-1607-g94a53970 a6477a474d71 db57a3e3b6c0 2024-03-12T05:55:08.730 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8 smithi088 running (3m) 30s ago 3m 52.1M 4270M 19.0.0-1607-g94a53970 a6477a474d71 f4d6766e5324 2024-03-12T05:55:08.730 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9 smithi076 running (3m) 16s ago 3m 45.8M 4272M 19.0.0-1607-g94a53970 a6477a474d71 f2a23fff0117 2024-03-12T05:55:08.730 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10 smithi082 running (2m) 30s ago 2m 57.2M 4270M 19.0.0-1607-g94a53970 a6477a474d71 6a65aec96e6a 2024-03-12T05:55:08.730 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11 smithi088 running (2m) 30s ago 2m 51.1M 4270M 19.0.0-1607-g94a53970 a6477a474d71 bbe9c50879fa 2024-03-12T05:55:08.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.725+0000 7fab504ac700 1 -- 172.21.15.76:0/1791444912 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fab340a1b50 msgr2=0x7fab340a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:08.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.725+0000 7fab504ac700 1 --2- 172.21.15.76:0/1791444912 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fab340a1b50 0x7fab340a4010 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fab44005950 tx=0x7fab440058e0 comp rx=0 tx=0).stop 2024-03-12T05:55:08.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.725+0000 7fab504ac700 1 -- 172.21.15.76:0/1791444912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab48101dd0 msgr2=0x7fab4819a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:08.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.725+0000 7fab504ac700 1 --2- 172.21.15.76:0/1791444912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab48101dd0 0x7fab4819a2f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fab38005910 tx=0x7fab38005b00 comp rx=0 tx=0).stop 2024-03-12T05:55:08.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.725+0000 7fab504ac700 1 -- 172.21.15.76:0/1791444912 shutdown_connections 2024-03-12T05:55:08.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.725+0000 7fab504ac700 1 --2- 172.21.15.76:0/1791444912 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fab340a1b50 0x7fab340a4010 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:08.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.725+0000 7fab504ac700 1 --2- 172.21.15.76:0/1791444912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab48102770 0x7fab481a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:08.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.725+0000 7fab504ac700 1 --2- 172.21.15.76:0/1791444912 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab48100bd0 0x7fab48199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:08.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.725+0000 7fab504ac700 1 --2- 172.21.15.76:0/1791444912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab48101dd0 0x7fab4819a2f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:08.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.725+0000 7fab504ac700 1 -- 172.21.15.76:0/1791444912 >> 172.21.15.76:0/1791444912 conn(0x7fab480fc190 msgr2=0x7fab480fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:08.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.725+0000 7fab504ac700 1 -- 172.21.15.76:0/1791444912 shutdown_connections 2024-03-12T05:55:08.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:08.725+0000 7fab504ac700 1 -- 172.21.15.76:0/1791444912 wait complete. 2024-03-12T05:55:09.416 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-12T05:55:09.656 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:55:10.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:09 smithi076 ceph-mon[32095]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:10.051 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:09 smithi076 ceph-mon[32095]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:55:10.194 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:09 smithi082 ceph-mon[36382]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:10.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:09 smithi082 ceph-mon[36382]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:55:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:09 smithi088 ceph-mon[36585]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:10.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:09 smithi088 ceph-mon[36585]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:55:11.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.279+0000 7fd8702cb700 1 -- 172.21.15.76:0/2361088894 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd868100b90 msgr2=0x7fd868100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:11.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.279+0000 7fd8702cb700 1 --2- 172.21.15.76:0/2361088894 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd868100b90 0x7fd868100fb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd8580099c0 tx=0x7fd858009cd0 comp rx=0 tx=0).stop 2024-03-12T05:55:11.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.279+0000 7fd8702cb700 1 -- 172.21.15.76:0/2361088894 shutdown_connections 2024-03-12T05:55:11.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.279+0000 7fd8702cb700 1 --2- 172.21.15.76:0/2361088894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd868102730 0x7fd868108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:11.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.279+0000 7fd8702cb700 1 --2- 172.21.15.76:0/2361088894 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd868100b90 0x7fd868100fb0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:11.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.279+0000 7fd8702cb700 1 --2- 172.21.15.76:0/2361088894 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd868101d90 0x7fd8681021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:11.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.279+0000 7fd8702cb700 1 -- 172.21.15.76:0/2361088894 >> 172.21.15.76:0/2361088894 conn(0x7fd8680fc130 msgr2=0x7fd8680fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:11.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.279+0000 7fd8702cb700 1 -- 172.21.15.76:0/2361088894 shutdown_connections 2024-03-12T05:55:11.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.279+0000 7fd8702cb700 1 -- 172.21.15.76:0/2361088894 wait complete. 2024-03-12T05:55:11.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.280+0000 7fd8702cb700 1 Processor -- start 2024-03-12T05:55:11.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.280+0000 7fd8702cb700 1 -- start start 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.280+0000 7fd8702cb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd868100b90 0x7fd868197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.280+0000 7fd8702cb700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd868101d90 0x7fd868198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.280+0000 7fd8702cb700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd868102730 0x7fd86819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.280+0000 7fd8702cb700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86819f6d0 con 0x7fd868102730 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.280+0000 7fd8702cb700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86819f810 con 0x7fd868101d90 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.280+0000 7fd8702cb700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86819f9a0 con 0x7fd868100b90 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.280+0000 7fd86e868700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd868102730 0x7fd86819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.280+0000 7fd86e868700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd868102730 0x7fd86819f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:60514/0 (socket says 172.21.15.76:60514) 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.280+0000 7fd86e868700 1 -- 172.21.15.76:0/462071246 learned_addr learned my addr 172.21.15.76:0/462071246 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.280+0000 7fd86e067700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd868100b90 0x7fd868197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.280+0000 7fd86d866700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd868101d90 0x7fd868198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.281+0000 7fd86e868700 1 -- 172.21.15.76:0/462071246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd868100b90 msgr2=0x7fd868197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.281+0000 7fd86e868700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd868100b90 0x7fd868197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.281+0000 7fd86e868700 1 -- 172.21.15.76:0/462071246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd868101d90 msgr2=0x7fd868198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:11.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.281+0000 7fd86e868700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd868101d90 0x7fd868198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:11.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.281+0000 7fd86e868700 1 -- 172.21.15.76:0/462071246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8580095f0 con 0x7fd868102730 2024-03-12T05:55:11.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.281+0000 7fd86d866700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd868101d90 0x7fd868198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:11.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.281+0000 7fd86e868700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd868102730 0x7fd86819f100 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd860007da0 tx=0x7fd86000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:11.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.281+0000 7fd85f7fe700 1 -- 172.21.15.76:0/462071246 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86000ad00 con 0x7fd868102730 2024-03-12T05:55:11.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.281+0000 7fd8702cb700 1 -- 172.21.15.76:0/462071246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86819fc80 con 0x7fd868102730 2024-03-12T05:55:11.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.281+0000 7fd85f7fe700 1 -- 172.21.15.76:0/462071246 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd86000ae60 con 0x7fd868102730 2024-03-12T05:55:11.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.281+0000 7fd85f7fe700 1 -- 172.21.15.76:0/462071246 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860011710 con 0x7fd868102730 2024-03-12T05:55:11.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.281+0000 7fd8702cb700 1 -- 172.21.15.76:0/462071246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8681a0130 con 0x7fd868102730 2024-03-12T05:55:11.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.282+0000 7fd8702cb700 1 -- 172.21.15.76:0/462071246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd868068930 con 0x7fd868102730 2024-03-12T05:55:11.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.283+0000 7fd85f7fe700 1 -- 172.21.15.76:0/462071246 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) v1 ==== 136330+0+0 (secure 0 0 0) 0x7fd860012420 con 0x7fd868102730 2024-03-12T05:55:11.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.283+0000 7fd85f7fe700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd8540a1b50 0x7fd8540a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:11.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.284+0000 7fd86e067700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd8540a1b50 0x7fd8540a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:11.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.284+0000 7fd85f7fe700 1 -- 172.21.15.76:0/462071246 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd8600cd6f0 con 0x7fd868102730 2024-03-12T05:55:11.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.284+0000 7fd86e067700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd8540a1b50 0x7fd8540a4010 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd858006010 tx=0x7fd858000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:11.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.288+0000 7fd85f7fe700 1 -- 172.21.15.76:0/462071246 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd8600968c0 con 0x7fd868102730 2024-03-12T05:55:11.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.492+0000 7fd8702cb700 1 -- 172.21.15.76:0/462071246 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8680fe230 con 0x7fd8540a1b50 2024-03-12T05:55:11.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.498+0000 7fd85f7fe700 1 -- 172.21.15.76:0/462071246 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7fd8680fe230 con 0x7fd8540a1b50 2024-03-12T05:55:11.502 INFO:teuthology.orchestra.run.smithi076.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-12T05:55:11.502 INFO:teuthology.orchestra.run.smithi076.stdout:mds.all 12/12 33s ago 2m smithi076=a;smithi076=d;smithi076=g;smithi076=j;smithi082=b;smithi082=e;smithi082=h;smithi082=k;smithi088=c;smithi088=f;smithi088=i;smithi088=l;count:12 2024-03-12T05:55:11.503 INFO:teuthology.orchestra.run.smithi076.stdout:mgr 3/3 33s ago 7m smithi076=x;smithi082=y;smithi088=z;count:3 2024-03-12T05:55:11.503 INFO:teuthology.orchestra.run.smithi076.stdout:mon 3/3 33s ago 7m smithi076:172.21.15.76=a;smithi082:172.21.15.82=b;smithi088:172.21.15.88=c;count:3 2024-03-12T05:55:11.503 INFO:teuthology.orchestra.run.smithi076.stdout:osd 12 33s ago - 2024-03-12T05:55:11.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.501+0000 7fd8702cb700 1 -- 172.21.15.76:0/462071246 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd8540a1b50 msgr2=0x7fd8540a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.501+0000 7fd8702cb700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd8540a1b50 0x7fd8540a4010 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd858006010 tx=0x7fd858000bc0 comp rx=0 tx=0).stop 2024-03-12T05:55:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.501+0000 7fd8702cb700 1 -- 172.21.15.76:0/462071246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd868102730 msgr2=0x7fd86819f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.501+0000 7fd8702cb700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd868102730 0x7fd86819f100 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd860007da0 tx=0x7fd86000a340 comp rx=0 tx=0).stop 2024-03-12T05:55:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.501+0000 7fd8702cb700 1 -- 172.21.15.76:0/462071246 shutdown_connections 2024-03-12T05:55:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.501+0000 7fd8702cb700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fd8540a1b50 0x7fd8540a4010 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.501+0000 7fd8702cb700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd868102730 0x7fd86819f100 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.501+0000 7fd8702cb700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd868100b90 0x7fd868197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.501+0000 7fd8702cb700 1 --2- 172.21.15.76:0/462071246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd868101d90 0x7fd868198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.501+0000 7fd8702cb700 1 -- 172.21.15.76:0/462071246 >> 172.21.15.76:0/462071246 conn(0x7fd8680fc130 msgr2=0x7fd8680fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.501+0000 7fd8702cb700 1 -- 172.21.15.76:0/462071246 shutdown_connections 2024-03-12T05:55:11.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:11.501+0000 7fd8702cb700 1 -- 172.21.15.76:0/462071246 wait complete. 2024-03-12T05:55:12.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:11 smithi076 ceph-mon[32095]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:55:12.224 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-12T05:55:12.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:11 smithi088 ceph-mon[36585]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:55:12.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:11 smithi082 ceph-mon[36382]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:55:12.464 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:55:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:12 smithi088 ceph-mon[36585]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:13.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:12 smithi088 ceph-mon[36585]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-12T05:55:13.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:12 smithi076 ceph-mon[32095]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:13.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:12 smithi076 ceph-mon[32095]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-12T05:55:13.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:12 smithi082 ceph-mon[36382]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:13.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:12 smithi082 ceph-mon[36382]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-12T05:55:14.130 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:13 smithi076 ceph-mon[32095]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:55:14.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.154+0000 7fbc3846e700 1 -- 172.21.15.76:0/2429485137 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc30102ed0 msgr2=0x7fbc30103330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:14.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.154+0000 7fbc3846e700 1 --2- 172.21.15.76:0/2429485137 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc30102ed0 0x7fbc30103330 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0099c0 tx=0x7fbc2c009cd0 comp rx=0 tx=0).stop 2024-03-12T05:55:14.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.154+0000 7fbc3846e700 1 -- 172.21.15.76:0/2429485137 shutdown_connections 2024-03-12T05:55:14.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.154+0000 7fbc3846e700 1 --2- 172.21.15.76:0/2429485137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc30103870 0x7fbc3010a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:14.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.154+0000 7fbc3846e700 1 --2- 172.21.15.76:0/2429485137 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc30101cd0 0x7fbc301020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:14.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.154+0000 7fbc3846e700 1 --2- 172.21.15.76:0/2429485137 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc30102ed0 0x7fbc30103330 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:14.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.154+0000 7fbc3846e700 1 -- 172.21.15.76:0/2429485137 >> 172.21.15.76:0/2429485137 conn(0x7fbc300fd270 msgr2=0x7fbc300ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:14.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.154+0000 7fbc3846e700 1 -- 172.21.15.76:0/2429485137 shutdown_connections 2024-03-12T05:55:14.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.154+0000 7fbc3846e700 1 -- 172.21.15.76:0/2429485137 wait complete. 2024-03-12T05:55:14.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3846e700 1 Processor -- start 2024-03-12T05:55:14.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3846e700 1 -- start start 2024-03-12T05:55:14.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3846e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc30101cd0 0x7fbc30197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:14.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3846e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc30102ed0 0x7fbc30198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:14.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3846e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc30103870 0x7fbc3019f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:14.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3846e700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3019f7a0 con 0x7fbc30101cd0 2024-03-12T05:55:14.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3846e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3019f8e0 con 0x7fbc30103870 2024-03-12T05:55:14.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3846e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3019faa0 con 0x7fbc30102ed0 2024-03-12T05:55:14.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3620a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc30101cd0 0x7fbc30197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3620a700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc30101cd0 0x7fbc30197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49082/0 (socket says 172.21.15.76:49082) 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3620a700 1 -- 172.21.15.76:0/3101210080 learned_addr learned my addr 172.21.15.76:0/3101210080 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc36a0b700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc30103870 0x7fbc3019f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc35a09700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc30102ed0 0x7fbc30198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3620a700 1 -- 172.21.15.76:0/3101210080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc30102ed0 msgr2=0x7fbc30198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3620a700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc30102ed0 0x7fbc30198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3620a700 1 -- 172.21.15.76:0/3101210080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc30103870 msgr2=0x7fbc3019f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3620a700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc30103870 0x7fbc3019f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc3620a700 1 -- 172.21.15.76:0/3101210080 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c0095f0 con 0x7fbc30101cd0 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.155+0000 7fbc35a09700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc30102ed0 0x7fbc30198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.156+0000 7fbc3620a700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc30101cd0 0x7fbc30197c50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000c970 tx=0x7fbc2000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.156+0000 7fbc277fe700 1 -- 172.21.15.76:0/3101210080 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20004220 con 0x7fbc30101cd0 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.156+0000 7fbc277fe700 1 -- 172.21.15.76:0/3101210080 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc20007d80 con 0x7fbc30101cd0 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.156+0000 7fbc277fe700 1 -- 172.21.15.76:0/3101210080 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20003d00 con 0x7fbc30101cd0 2024-03-12T05:55:14.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.156+0000 7fbc3846e700 1 -- 172.21.15.76:0/3101210080 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc3019fd80 con 0x7fbc30101cd0 2024-03-12T05:55:14.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.156+0000 7fbc3846e700 1 -- 172.21.15.76:0/3101210080 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc301a0210 con 0x7fbc30101cd0 2024-03-12T05:55:14.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.157+0000 7fbc3846e700 1 -- 172.21.15.76:0/3101210080 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc30068930 con 0x7fbc30101cd0 2024-03-12T05:55:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.158+0000 7fbc277fe700 1 -- 172.21.15.76:0/3101210080 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbc200078a0 con 0x7fbc30101cd0 2024-03-12T05:55:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.158+0000 7fbc277fe700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbc1c0a1b40 0x7fbc1c0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.159+0000 7fbc35a09700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbc1c0a1b40 0x7fbc1c0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.159+0000 7fbc277fe700 1 -- 172.21.15.76:0/3101210080 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbc200cd300 con 0x7fbc30101cd0 2024-03-12T05:55:14.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.159+0000 7fbc35a09700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbc1c0a1b40 0x7fbc1c0a4000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c000c00 tx=0x7fbc2c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.162+0000 7fbc277fe700 1 -- 172.21.15.76:0/3101210080 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbc200965a0 con 0x7fbc30101cd0 2024-03-12T05:55:14.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:13 smithi088 ceph-mon[36585]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:55:14.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.366+0000 7fbc3846e700 1 -- 172.21.15.76:0/3101210080 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc3002d950 con 0x7fbc1c0a1b40 2024-03-12T05:55:14.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.368+0000 7fbc277fe700 1 -- 172.21.15.76:0/3101210080 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+187 (secure 0 0 0) 0x7fbc3002d950 con 0x7fbc1c0a1b40 2024-03-12T05:55:14.371 INFO:teuthology.orchestra.run.smithi076.stdout:HOST ADDR LABELS STATUS 2024-03-12T05:55:14.371 INFO:teuthology.orchestra.run.smithi076.stdout:smithi076 172.21.15.76 2024-03-12T05:55:14.371 INFO:teuthology.orchestra.run.smithi076.stdout:smithi082 172.21.15.82 2024-03-12T05:55:14.371 INFO:teuthology.orchestra.run.smithi076.stdout:smithi088 172.21.15.88 2024-03-12T05:55:14.371 INFO:teuthology.orchestra.run.smithi076.stdout:3 hosts in cluster 2024-03-12T05:55:14.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.370+0000 7fbc3846e700 1 -- 172.21.15.76:0/3101210080 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbc1c0a1b40 msgr2=0x7fbc1c0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:14.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.370+0000 7fbc3846e700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbc1c0a1b40 0x7fbc1c0a4000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c000c00 tx=0x7fbc2c005fb0 comp rx=0 tx=0).stop 2024-03-12T05:55:14.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.370+0000 7fbc3846e700 1 -- 172.21.15.76:0/3101210080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc30101cd0 msgr2=0x7fbc30197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:14.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.370+0000 7fbc3846e700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc30101cd0 0x7fbc30197c50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000c970 tx=0x7fbc2000cc80 comp rx=0 tx=0).stop 2024-03-12T05:55:14.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.370+0000 7fbc3846e700 1 -- 172.21.15.76:0/3101210080 shutdown_connections 2024-03-12T05:55:14.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.370+0000 7fbc3846e700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbc1c0a1b40 0x7fbc1c0a4000 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:14.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.370+0000 7fbc3846e700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc30103870 0x7fbc3019f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:14.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.370+0000 7fbc3846e700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc30101cd0 0x7fbc30197c50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:14.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.370+0000 7fbc3846e700 1 --2- 172.21.15.76:0/3101210080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc30102ed0 0x7fbc30198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:14.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.370+0000 7fbc3846e700 1 -- 172.21.15.76:0/3101210080 >> 172.21.15.76:0/3101210080 conn(0x7fbc300fd270 msgr2=0x7fbc30106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:14.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.370+0000 7fbc3846e700 1 -- 172.21.15.76:0/3101210080 shutdown_connections 2024-03-12T05:55:14.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:14.370+0000 7fbc3846e700 1 -- 172.21.15.76:0/3101210080 wait complete. 2024-03-12T05:55:14.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:13 smithi082 ceph-mon[36382]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-12T05:55:15.301 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-12T05:55:15.556 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:55:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:15 smithi088 ceph-mon[36585]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:16.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:15 smithi088 ceph-mon[36585]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-12T05:55:16.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:15 smithi076 ceph-mon[32095]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:16.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:15 smithi076 ceph-mon[32095]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-12T05:55:16.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:15 smithi082 ceph-mon[36382]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:16.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:15 smithi082 ceph-mon[36382]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-12T05:55:17.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.297+0000 7ff95c36b700 1 -- 172.21.15.76:0/3752250169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff954100bd0 msgr2=0x7ff954100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:17.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.297+0000 7ff95c36b700 1 --2- 172.21.15.76:0/3752250169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff954100bd0 0x7ff954100ff0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff9440099c0 tx=0x7ff944009cd0 comp rx=0 tx=0).stop 2024-03-12T05:55:17.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.297+0000 7ff95c36b700 1 -- 172.21.15.76:0/3752250169 shutdown_connections 2024-03-12T05:55:17.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.297+0000 7ff95c36b700 1 --2- 172.21.15.76:0/3752250169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff954102770 0x7ff954109020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.297+0000 7ff95c36b700 1 --2- 172.21.15.76:0/3752250169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff954100bd0 0x7ff954100ff0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.297+0000 7ff95c36b700 1 --2- 172.21.15.76:0/3752250169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff954101dd0 0x7ff954102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.297+0000 7ff95c36b700 1 -- 172.21.15.76:0/3752250169 >> 172.21.15.76:0/3752250169 conn(0x7ff9540fc190 msgr2=0x7ff9540fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.297+0000 7ff95c36b700 1 -- 172.21.15.76:0/3752250169 shutdown_connections 2024-03-12T05:55:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.297+0000 7ff95c36b700 1 -- 172.21.15.76:0/3752250169 wait complete. 2024-03-12T05:55:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95c36b700 1 Processor -- start 2024-03-12T05:55:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95c36b700 1 -- start start 2024-03-12T05:55:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95c36b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff954100bd0 0x7ff954197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95c36b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff954101dd0 0x7ff954198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95c36b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff954102770 0x7ff95419f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95c36b700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95419f8e0 con 0x7ff954100bd0 2024-03-12T05:55:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95c36b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95419fa20 con 0x7ff954101dd0 2024-03-12T05:55:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95c36b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95419fbb0 con 0x7ff954102770 2024-03-12T05:55:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95a107700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff954100bd0 0x7ff954197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95a107700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff954100bd0 0x7ff954197d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49094/0 (socket says 172.21.15.76:49094) 2024-03-12T05:55:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95a107700 1 -- 172.21.15.76:0/3466845263 learned_addr learned my addr 172.21.15.76:0/3466845263 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff959906700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff954101dd0 0x7ff954198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95a908700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff954102770 0x7ff95419f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95a107700 1 -- 172.21.15.76:0/3466845263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff954102770 msgr2=0x7ff95419f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95a107700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff954102770 0x7ff95419f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95a107700 1 -- 172.21.15.76:0/3466845263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff954101dd0 msgr2=0x7ff954198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95a107700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff954101dd0 0x7ff954198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95a107700 1 -- 172.21.15.76:0/3466845263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9440095f0 con 0x7ff954100bd0 2024-03-12T05:55:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95a908700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff954102770 0x7ff95419f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff95a107700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff954100bd0 0x7ff954197d50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff944003790 tx=0x7ff944004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.298+0000 7ff94b7fe700 1 -- 172.21.15.76:0/3466845263 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94401d070 con 0x7ff954100bd0 2024-03-12T05:55:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.299+0000 7ff94b7fe700 1 -- 172.21.15.76:0/3466845263 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff944022470 con 0x7ff954100bd0 2024-03-12T05:55:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.299+0000 7ff95c36b700 1 -- 172.21.15.76:0/3466845263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95419fe30 con 0x7ff954100bd0 2024-03-12T05:55:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.299+0000 7ff94b7fe700 1 -- 172.21.15.76:0/3466845263 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94400f6c0 con 0x7ff954100bd0 2024-03-12T05:55:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.299+0000 7ff95c36b700 1 -- 172.21.15.76:0/3466845263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9541a0290 con 0x7ff954100bd0 2024-03-12T05:55:17.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.300+0000 7ff95c36b700 1 -- 172.21.15.76:0/3466845263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff954068960 con 0x7ff954100bd0 2024-03-12T05:55:17.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.301+0000 7ff94b7fe700 1 -- 172.21.15.76:0/3466845263 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7ff9440225e0 con 0x7ff954100bd0 2024-03-12T05:55:17.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.301+0000 7ff94b7fe700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff9400a1ac0 0x7ff9400a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:17.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.301+0000 7ff959906700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff9400a1ac0 0x7ff9400a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:17.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.301+0000 7ff94b7fe700 1 -- 172.21.15.76:0/3466845263 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff9440ced60 con 0x7ff954100bd0 2024-03-12T05:55:17.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.302+0000 7ff959906700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff9400a1ac0 0x7ff9400a3f80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff950009a10 tx=0x7ff950009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:17.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.305+0000 7ff94b7fe700 1 -- 172.21.15.76:0/3466845263 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff944098010 con 0x7ff954100bd0 2024-03-12T05:55:17.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.509+0000 7ff95c36b700 1 -- 172.21.15.76:0/3466845263 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9540fe330 con 0x7ff9400a1ac0 2024-03-12T05:55:17.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.510+0000 7ff94b7fe700 1 -- 172.21.15.76:0/3466845263 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7ff9540fe330 con 0x7ff9400a1ac0 2024-03-12T05:55:17.514 INFO:teuthology.orchestra.run.smithi076.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-12T05:55:17.514 INFO:teuthology.orchestra.run.smithi076.stdout:smithi076 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400U4400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-12T05:55:17.514 INFO:teuthology.orchestra.run.smithi076.stdout:smithi082 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400SD400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-12T05:55:17.514 INFO:teuthology.orchestra.run.smithi076.stdout:smithi088 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400XD400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-12T05:55:17.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.513+0000 7ff95c36b700 1 -- 172.21.15.76:0/3466845263 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff9400a1ac0 msgr2=0x7ff9400a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:17.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.513+0000 7ff95c36b700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff9400a1ac0 0x7ff9400a3f80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff950009a10 tx=0x7ff950009190 comp rx=0 tx=0).stop 2024-03-12T05:55:17.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.513+0000 7ff95c36b700 1 -- 172.21.15.76:0/3466845263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff954100bd0 msgr2=0x7ff954197d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:17.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.513+0000 7ff95c36b700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff954100bd0 0x7ff954197d50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff944003790 tx=0x7ff944004730 comp rx=0 tx=0).stop 2024-03-12T05:55:17.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.513+0000 7ff95c36b700 1 -- 172.21.15.76:0/3466845263 shutdown_connections 2024-03-12T05:55:17.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.513+0000 7ff95c36b700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7ff9400a1ac0 0x7ff9400a3f80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:17.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.513+0000 7ff95c36b700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff954102770 0x7ff95419f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:17.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.513+0000 7ff95c36b700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff954100bd0 0x7ff954197d50 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:17.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.513+0000 7ff95c36b700 1 --2- 172.21.15.76:0/3466845263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff954101dd0 0x7ff954198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:17.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.513+0000 7ff95c36b700 1 -- 172.21.15.76:0/3466845263 >> 172.21.15.76:0/3466845263 conn(0x7ff9540fc190 msgr2=0x7ff9540fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:17.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.513+0000 7ff95c36b700 1 -- 172.21.15.76:0/3466845263 shutdown_connections 2024-03-12T05:55:17.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:17.513+0000 7ff95c36b700 1 -- 172.21.15.76:0/3466845263 wait complete. 2024-03-12T05:55:18.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:17 smithi076 ceph-mon[32095]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-12T05:55:18.202 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T05:55:18.212 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2024-03-12T05:55:18.212 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-12T05:55:18.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:17 smithi088 ceph-mon[36585]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-12T05:55:18.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:17 smithi082 ceph-mon[36382]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-12T05:55:18.456 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:55:19.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:18 smithi088 ceph-mon[36585]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:19.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:18 smithi076 ceph-mon[32095]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:19.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:18 smithi082 ceph-mon[36382]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:20.180 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:19 smithi076 ceph-mon[32095]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-12T05:55:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.168+0000 7f2370629700 1 -- 172.21.15.76:0/4185171441 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368105850 msgr2=0x7f2368105cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.168+0000 7f2370629700 1 --2- 172.21.15.76:0/4185171441 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368105850 0x7f2368105cd0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f236000b0a0 tx=0x7f236000b3b0 comp rx=0 tx=0).stop 2024-03-12T05:55:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.168+0000 7f2370629700 1 -- 172.21.15.76:0/4185171441 shutdown_connections 2024-03-12T05:55:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.168+0000 7f2370629700 1 --2- 172.21.15.76:0/4185171441 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368105850 0x7f2368105cd0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.168+0000 7f2370629700 1 --2- 172.21.15.76:0/4185171441 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2368100510 0x7f2368102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.168+0000 7f2370629700 1 --2- 172.21.15.76:0/4185171441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2368102e80 0x7f2368105310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.168+0000 7f2370629700 1 -- 172.21.15.76:0/4185171441 >> 172.21.15.76:0/4185171441 conn(0x7f23680fc150 msgr2=0x7f23680fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.168+0000 7f2370629700 1 -- 172.21.15.76:0/4185171441 shutdown_connections 2024-03-12T05:55:20.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.168+0000 7f2370629700 1 -- 172.21.15.76:0/4185171441 wait complete. 2024-03-12T05:55:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f2370629700 1 Processor -- start 2024-03-12T05:55:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f2370629700 1 -- start start 2024-03-12T05:55:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f2370629700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368100510 0x7f236819e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f2370629700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2368102e80 0x7f236819e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f2370629700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2368105850 0x7f23681a5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f2370629700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23681a5f20 con 0x7f2368100510 2024-03-12T05:55:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f2370629700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23681a6060 con 0x7f2368102e80 2024-03-12T05:55:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f2370629700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23681a61f0 con 0x7f2368105850 2024-03-12T05:55:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f236e3c5700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368100510 0x7f236819e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f236e3c5700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368100510 0x7f236819e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49102/0 (socket says 172.21.15.76:49102) 2024-03-12T05:55:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f236e3c5700 1 -- 172.21.15.76:0/4270285641 learned_addr learned my addr 172.21.15.76:0/4270285641 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f236dbc4700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2368102e80 0x7f236819e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f236ebc6700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2368105850 0x7f23681a5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:20.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f236e3c5700 1 -- 172.21.15.76:0/4270285641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2368105850 msgr2=0x7f23681a5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f236e3c5700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2368105850 0x7f23681a5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f236e3c5700 1 -- 172.21.15.76:0/4270285641 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2368102e80 msgr2=0x7f236819e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f236e3c5700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2368102e80 0x7f236819e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f236e3c5700 1 -- 172.21.15.76:0/4270285641 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2360009d00 con 0x7f2368100510 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f236dbc4700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2368102e80 0x7f236819e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.169+0000 7f236e3c5700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368100510 0x7f236819e390 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f236400b560 tx=0x7f236400b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.170+0000 7f235f7fe700 1 -- 172.21.15.76:0/4270285641 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364011830 con 0x7f2368100510 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.170+0000 7f235f7fe700 1 -- 172.21.15.76:0/4270285641 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2364011e70 con 0x7f2368100510 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.170+0000 7f2370629700 1 -- 172.21.15.76:0/4270285641 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23681a64d0 con 0x7f2368100510 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.170+0000 7f235f7fe700 1 -- 172.21.15.76:0/4270285641 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236400f2e0 con 0x7f2368100510 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.170+0000 7f2370629700 1 -- 172.21.15.76:0/4270285641 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23681a6a50 con 0x7f2368100510 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.171+0000 7f2370629700 1 -- 172.21.15.76:0/4270285641 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23680fdcd0 con 0x7f2368100510 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.172+0000 7f235f7fe700 1 -- 172.21.15.76:0/4270285641 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2364011990 con 0x7f2368100510 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.172+0000 7f235f7fe700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f23540a1ac0 0x7f23540a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.172+0000 7f236dbc4700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f23540a1ac0 0x7f23540a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.172+0000 7f235f7fe700 1 -- 172.21.15.76:0/4270285641 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f23640ccf50 con 0x7f2368100510 2024-03-12T05:55:20.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.173+0000 7f236dbc4700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f23540a1ac0 0x7f23540a3f80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2358005950 tx=0x7f23580058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:20.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.176+0000 7f235f7fe700 1 -- 172.21.15.76:0/4270285641 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2364096170 con 0x7f2368100510 2024-03-12T05:55:20.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:19 smithi088 ceph-mon[36585]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-12T05:55:20.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.416+0000 7f2370629700 1 -- 172.21.15.76:0/4270285641 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2368068930 con 0x7f2368100510 2024-03-12T05:55:20.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.417+0000 7f235f7fe700 1 -- 172.21.15.76:0/4270285641 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+2931 (secure 0 0 0) 0x7f2364015b90 con 0x7f2368100510 2024-03-12T05:55:20.421 INFO:teuthology.orchestra.run.smithi076.stdout:e21 2024-03-12T05:55:20.421 INFO:teuthology.orchestra.run.smithi076.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T05:55:20.421 INFO:teuthology.orchestra.run.smithi076.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T05:55:20.421 INFO:teuthology.orchestra.run.smithi076.stdout:legacy client fscid: 1 2024-03-12T05:55:20.421 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:20.422 INFO:teuthology.orchestra.run.smithi076.stdout:Filesystem 'cephfs' (1) 2024-03-12T05:55:20.422 INFO:teuthology.orchestra.run.smithi076.stdout:fs_name cephfs 2024-03-12T05:55:20.422 INFO:teuthology.orchestra.run.smithi076.stdout:epoch 21 2024-03-12T05:55:20.422 INFO:teuthology.orchestra.run.smithi076.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-12T05:55:20.422 INFO:teuthology.orchestra.run.smithi076.stdout:created 2024-03-12T05:53:37.535927+0000 2024-03-12T05:55:20.422 INFO:teuthology.orchestra.run.smithi076.stdout:modified 2024-03-12T05:53:42.384114+0000 2024-03-12T05:55:20.422 INFO:teuthology.orchestra.run.smithi076.stdout:tableserver 0 2024-03-12T05:55:20.423 INFO:teuthology.orchestra.run.smithi076.stdout:root 0 2024-03-12T05:55:20.423 INFO:teuthology.orchestra.run.smithi076.stdout:session_timeout 300 2024-03-12T05:55:20.423 INFO:teuthology.orchestra.run.smithi076.stdout:session_autoclose 300 2024-03-12T05:55:20.423 INFO:teuthology.orchestra.run.smithi076.stdout:max_file_size 1099511627776 2024-03-12T05:55:20.423 INFO:teuthology.orchestra.run.smithi076.stdout:max_xattr_size 65536 2024-03-12T05:55:20.423 INFO:teuthology.orchestra.run.smithi076.stdout:required_client_features {} 2024-03-12T05:55:20.423 INFO:teuthology.orchestra.run.smithi076.stdout:last_failure 0 2024-03-12T05:55:20.424 INFO:teuthology.orchestra.run.smithi076.stdout:last_failure_osd_epoch 0 2024-03-12T05:55:20.424 INFO:teuthology.orchestra.run.smithi076.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-12T05:55:20.424 INFO:teuthology.orchestra.run.smithi076.stdout:max_mds 3 2024-03-12T05:55:20.424 INFO:teuthology.orchestra.run.smithi076.stdout:in 0,1,2 2024-03-12T05:55:20.424 INFO:teuthology.orchestra.run.smithi076.stdout:up {0=24518,1=24481,2=24451} 2024-03-12T05:55:20.424 INFO:teuthology.orchestra.run.smithi076.stdout:failed 2024-03-12T05:55:20.424 INFO:teuthology.orchestra.run.smithi076.stdout:damaged 2024-03-12T05:55:20.425 INFO:teuthology.orchestra.run.smithi076.stdout:stopped 2024-03-12T05:55:20.425 INFO:teuthology.orchestra.run.smithi076.stdout:data_pools [3] 2024-03-12T05:55:20.425 INFO:teuthology.orchestra.run.smithi076.stdout:metadata_pool 2 2024-03-12T05:55:20.425 INFO:teuthology.orchestra.run.smithi076.stdout:inline_data disabled 2024-03-12T05:55:20.425 INFO:teuthology.orchestra.run.smithi076.stdout:balancer 2024-03-12T05:55:20.426 INFO:teuthology.orchestra.run.smithi076.stdout:bal_rank_mask -1 2024-03-12T05:55:20.426 INFO:teuthology.orchestra.run.smithi076.stdout:standby_count_wanted 0 2024-03-12T05:55:20.426 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.b{0:24518} state up:active seq 3 addr [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:55:20.426 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.h{0:24479} state up:standby-replay seq 1 addr [v2:172.21.15.82:6836/2523422510,v1:172.21.15.82:6837/2523422510] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:55:20.426 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.i{1:24481} state up:active seq 6 addr [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:55:20.426 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.c{1:24463} state up:standby-replay seq 1 addr [v2:172.21.15.88:6836/1311843051,v1:172.21.15.88:6837/1311843051] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:55:20.427 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.f{2:24451} state up:active seq 8 addr [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:55:20.427 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.e{2:24449} state up:standby-replay seq 1 addr [v2:172.21.15.82:6834/2138041661,v1:172.21.15.82:6835/2138041661] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:55:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:20.430 INFO:teuthology.orchestra.run.smithi076.stdout:Standby daemons: 2024-03-12T05:55:20.430 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:20.431 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.j{-1:14568} state up:standby seq 1 addr [v2:172.21.15.76:6834/4172787847,v1:172.21.15.76:6835/4172787847] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:55:20.431 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.d{-1:14586} state up:standby seq 1 addr [v2:172.21.15.76:6836/2194036306,v1:172.21.15.76:6837/2194036306] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:55:20.431 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.g{-1:14652} state up:standby seq 1 addr [v2:172.21.15.76:6838/1304487620,v1:172.21.15.76:6839/1304487620] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:55:20.431 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.a{-1:14670} state up:standby seq 1 addr [v2:172.21.15.76:6840/291567235,v1:172.21.15.76:6841/291567235] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:55:20.431 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.l{-1:24412} state up:standby seq 1 addr [v2:172.21.15.88:6832/3297889286,v1:172.21.15.88:6833/3297889286] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:55:20.431 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.k{-1:24419} state up:standby seq 1 addr [v2:172.21.15.82:6832/2359119459,v1:172.21.15.82:6833/2359119459] compat {c=[1],r=[1],i=[fff]}] 2024-03-12T05:55:20.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.419+0000 7f2370629700 1 -- 172.21.15.76:0/4270285641 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f23540a1ac0 msgr2=0x7f23540a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:20.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.419+0000 7f2370629700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f23540a1ac0 0x7f23540a3f80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2358005950 tx=0x7f23580058e0 comp rx=0 tx=0).stop 2024-03-12T05:55:20.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.419+0000 7f2370629700 1 -- 172.21.15.76:0/4270285641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368100510 msgr2=0x7f236819e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:20.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.419+0000 7f2370629700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368100510 0x7f236819e390 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f236400b560 tx=0x7f236400b920 comp rx=0 tx=0).stop 2024-03-12T05:55:20.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.419+0000 7f2370629700 1 -- 172.21.15.76:0/4270285641 shutdown_connections 2024-03-12T05:55:20.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.419+0000 7f2370629700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f23540a1ac0 0x7f23540a3f80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:20.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.419+0000 7f2370629700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2368105850 0x7f23681a5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:20.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.419+0000 7f2370629700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2368100510 0x7f236819e390 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:20.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.419+0000 7f2370629700 1 --2- 172.21.15.76:0/4270285641 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2368102e80 0x7f236819e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:20.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.419+0000 7f2370629700 1 -- 172.21.15.76:0/4270285641 >> 172.21.15.76:0/4270285641 conn(0x7f23680fc150 msgr2=0x7f23681011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:20.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.419+0000 7f2370629700 1 -- 172.21.15.76:0/4270285641 shutdown_connections 2024-03-12T05:55:20.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:20.419+0000 7f2370629700 1 -- 172.21.15.76:0/4270285641 wait complete. 2024-03-12T05:55:20.435 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 21 2024-03-12T05:55:20.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:19 smithi082 ceph-mon[36382]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-12T05:55:21.070 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-12T05:55:21.097 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:20 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4270285641' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-12T05:55:21.320 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:55:21.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:20 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4270285641' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-12T05:55:21.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:20 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4270285641' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-12T05:55:22.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:21 smithi076 ceph-mon[32095]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-12T05:55:22.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:21 smithi082 ceph-mon[36382]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-12T05:55:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:21 smithi088 ceph-mon[36585]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-12T05:55:23.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.130+0000 7fb8f2809700 1 -- 172.21.15.76:0/3988450307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8ec101e10 msgr2=0x7fb8ec102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:23.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.130+0000 7fb8f2809700 1 --2- 172.21.15.76:0/3988450307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8ec101e10 0x7fb8ec102270 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00099c0 tx=0x7fb8e0009cd0 comp rx=0 tx=0).stop 2024-03-12T05:55:23.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.130+0000 7fb8f2809700 1 -- 172.21.15.76:0/3988450307 shutdown_connections 2024-03-12T05:55:23.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.130+0000 7fb8f2809700 1 --2- 172.21.15.76:0/3988450307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8ec1027b0 0x7fb8ec109060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:23.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.130+0000 7fb8f2809700 1 --2- 172.21.15.76:0/3988450307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8ec100c10 0x7fb8ec101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:23.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.130+0000 7fb8f2809700 1 --2- 172.21.15.76:0/3988450307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8ec101e10 0x7fb8ec102270 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.130+0000 7fb8f2809700 1 -- 172.21.15.76:0/3988450307 >> 172.21.15.76:0/3988450307 conn(0x7fb8ec0fc1d0 msgr2=0x7fb8ec0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.130+0000 7fb8f2809700 1 -- 172.21.15.76:0/3988450307 shutdown_connections 2024-03-12T05:55:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.130+0000 7fb8f2809700 1 -- 172.21.15.76:0/3988450307 wait complete. 2024-03-12T05:55:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.131+0000 7fb8f2809700 1 Processor -- start 2024-03-12T05:55:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.131+0000 7fb8f2809700 1 -- start start 2024-03-12T05:55:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.131+0000 7fb8f2809700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8ec100c10 0x7fb8ec104ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.131+0000 7fb8f2809700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8ec101e10 0x7fb8ec1031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.131+0000 7fb8f2809700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8ec1027b0 0x7fb8ec103730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.131+0000 7fb8f2809700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec103eb0 con 0x7fb8ec101e10 2024-03-12T05:55:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.131+0000 7fb8f2809700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec104020 con 0x7fb8ec100c10 2024-03-12T05:55:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.131+0000 7fb8f2809700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec1a1cb0 con 0x7fb8ec1027b0 2024-03-12T05:55:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.131+0000 7fb8eb7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8ec101e10 0x7fb8ec1031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.131+0000 7fb8eb7fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8ec101e10 0x7fb8ec1031f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55238/0 (socket says 172.21.15.76:55238) 2024-03-12T05:55:23.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.131+0000 7fb8eb7fe700 1 -- 172.21.15.76:0/4274301920 learned_addr learned my addr 172.21.15.76:0/4274301920 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:23.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8ebfff700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8ec100c10 0x7fb8ec104ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:23.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8f0da6700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8ec1027b0 0x7fb8ec103730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:23.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8eb7fe700 1 -- 172.21.15.76:0/4274301920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8ec1027b0 msgr2=0x7fb8ec103730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:23.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8eb7fe700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8ec1027b0 0x7fb8ec103730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:23.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8eb7fe700 1 -- 172.21.15.76:0/4274301920 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8ec100c10 msgr2=0x7fb8ec104ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:23.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8eb7fe700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8ec100c10 0x7fb8ec104ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:23.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8eb7fe700 1 -- 172.21.15.76:0/4274301920 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e00095f0 con 0x7fb8ec101e10 2024-03-12T05:55:23.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8f0da6700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8ec1027b0 0x7fb8ec103730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:23.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8ebfff700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8ec100c10 0x7fb8ec104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:23.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8eb7fe700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8ec101e10 0x7fb8ec1031f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0003890 tx=0x7fb8e000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:23.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8e97fa700 1 -- 172.21.15.76:0/4274301920 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e001d070 con 0x7fb8ec101e10 2024-03-12T05:55:23.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8e97fa700 1 -- 172.21.15.76:0/4274301920 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb8e0003930 con 0x7fb8ec101e10 2024-03-12T05:55:23.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8f2809700 1 -- 172.21.15.76:0/4274301920 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ec1a1f30 con 0x7fb8ec101e10 2024-03-12T05:55:23.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8e97fa700 1 -- 172.21.15.76:0/4274301920 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e00056c0 con 0x7fb8ec101e10 2024-03-12T05:55:23.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.132+0000 7fb8f2809700 1 -- 172.21.15.76:0/4274301920 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ec1a2340 con 0x7fb8ec101e10 2024-03-12T05:55:23.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.133+0000 7fb8f2809700 1 -- 172.21.15.76:0/4274301920 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ec10c4e0 con 0x7fb8ec101e10 2024-03-12T05:55:23.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.134+0000 7fb8e97fa700 1 -- 172.21.15.76:0/4274301920 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fb8e0003aa0 con 0x7fb8ec101e10 2024-03-12T05:55:23.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.135+0000 7fb8e97fa700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb8d40a1ac0 0x7fb8d40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:23.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.135+0000 7fb8ebfff700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb8d40a1ac0 0x7fb8d40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:23.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.135+0000 7fb8e97fa700 1 -- 172.21.15.76:0/4274301920 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb8e00ce610 con 0x7fb8ec101e10 2024-03-12T05:55:23.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.135+0000 7fb8ebfff700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb8d40a1ac0 0x7fb8d40a3f80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec101c70 tx=0x7fb8dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:23.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.138+0000 7fb8e97fa700 1 -- 172.21.15.76:0/4274301920 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb8e00978c0 con 0x7fb8ec101e10 2024-03-12T05:55:23.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.337+0000 7fb8f2809700 1 -- 172.21.15.76:0/4274301920 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fb8ec10c6c0 con 0x7fb8ec101e10 2024-03-12T05:55:23.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.338+0000 7fb8e97fa700 1 -- 172.21.15.76:0/4274301920 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) v1 ==== 56+0+5244 (secure 0 0 0) 0x7fb8e009af80 con 0x7fb8ec101e10 2024-03-12T05:55:23.341 INFO:teuthology.orchestra.run.smithi076.stdout:epoch 88 2024-03-12T05:55:23.341 INFO:teuthology.orchestra.run.smithi076.stdout:fsid 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T05:55:23.342 INFO:teuthology.orchestra.run.smithi076.stdout:created 2024-03-12T05:45:46.455987+0000 2024-03-12T05:55:23.342 INFO:teuthology.orchestra.run.smithi076.stdout:modified 2024-03-12T05:54:42.007701+0000 2024-03-12T05:55:23.342 INFO:teuthology.orchestra.run.smithi076.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-12T05:55:23.342 INFO:teuthology.orchestra.run.smithi076.stdout:crush_version 26 2024-03-12T05:55:23.342 INFO:teuthology.orchestra.run.smithi076.stdout:full_ratio 0.95 2024-03-12T05:55:23.342 INFO:teuthology.orchestra.run.smithi076.stdout:backfillfull_ratio 0.9 2024-03-12T05:55:23.342 INFO:teuthology.orchestra.run.smithi076.stdout:nearfull_ratio 0.85 2024-03-12T05:55:23.342 INFO:teuthology.orchestra.run.smithi076.stdout:require_min_compat_client luminous 2024-03-12T05:55:23.342 INFO:teuthology.orchestra.run.smithi076.stdout:min_compat_client luminous 2024-03-12T05:55:23.342 INFO:teuthology.orchestra.run.smithi076.stdout:require_osd_release squid 2024-03-12T05:55:23.342 INFO:teuthology.orchestra.run.smithi076.stdout:stretch_mode_enabled false 2024-03-12T05:55:23.342 INFO:teuthology.orchestra.run.smithi076.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-12T05:55:23.342 INFO:teuthology.orchestra.run.smithi076.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-12T05:55:23.343 INFO:teuthology.orchestra.run.smithi076.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-12T05:55:23.343 INFO:teuthology.orchestra.run.smithi076.stdout:max_osd 12 2024-03-12T05:55:23.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6802/1797183805,v1:172.21.15.76:6803/1797183805] [v2:172.21.15.76:6804/1797183805,v1:172.21.15.76:6805/1797183805] exists,up 5faf2b7d-916a-4077-8da3-06bba2a002ab 2024-03-12T05:55:23.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6800/1086375488,v1:172.21.15.82:6801/1086375488] [v2:172.21.15.82:6802/1086375488,v1:172.21.15.82:6803/1086375488] exists,up 7a64f23f-579e-4114-8250-228b150a753b 2024-03-12T05:55:23.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.88:6800/1437946838,v1:172.21.15.88:6801/1437946838] [v2:172.21.15.88:6802/1437946838,v1:172.21.15.88:6803/1437946838] exists,up 621a410c-75b7-42c3-aa04-f4a3e188586b 2024-03-12T05:55:23.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 up in weight 1 up_from 25 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6810/159124618,v1:172.21.15.76:6811/159124618] [v2:172.21.15.76:6812/159124618,v1:172.21.15.76:6813/159124618] exists,up e843f131-6b3e-4148-90d2-0e44ea1b5511 2024-03-12T05:55:23.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4 up in weight 1 up_from 33 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6808/3036883628,v1:172.21.15.82:6809/3036883628] [v2:172.21.15.82:6810/3036883628,v1:172.21.15.82:6811/3036883628] exists,up f20ff3d5-82fd-4d8c-aac9-a3b538e12dff 2024-03-12T05:55:23.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.88:6808/2894543264,v1:172.21.15.88:6809/2894543264] [v2:172.21.15.88:6810/2894543264,v1:172.21.15.88:6811/2894543264] exists,up c9499144-0f35-47f2-89bc-ce4af1aa1b58 2024-03-12T05:55:23.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6 up in weight 1 up_from 46 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6818/4101985935,v1:172.21.15.76:6819/4101985935] [v2:172.21.15.76:6820/4101985935,v1:172.21.15.76:6821/4101985935] exists,up 1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4 2024-03-12T05:55:23.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7 up in weight 1 up_from 51 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6816/88004522,v1:172.21.15.82:6817/88004522] [v2:172.21.15.82:6818/88004522,v1:172.21.15.82:6819/88004522] exists,up 233f8b14-0d82-43d6-96f7-bc9a4ed524a9 2024-03-12T05:55:23.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.8 up in weight 1 up_from 60 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.88:6816/529149247,v1:172.21.15.88:6817/529149247] [v2:172.21.15.88:6818/529149247,v1:172.21.15.88:6819/529149247] exists,up 959d820e-13bb-4a2b-a9fc-11122a0be243 2024-03-12T05:55:23.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.9 up in weight 1 up_from 68 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6826/3035286191,v1:172.21.15.76:6827/3035286191] [v2:172.21.15.76:6828/3035286191,v1:172.21.15.76:6829/3035286191] exists,up bcb44da2-47f9-4ff3-bd96-f026626faea2 2024-03-12T05:55:23.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.10 up in weight 1 up_from 73 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6824/2430204327,v1:172.21.15.82:6825/2430204327] [v2:172.21.15.82:6826/2430204327,v1:172.21.15.82:6827/2430204327] exists,up c565e2eb-11bf-4889-b19e-11473f9ed6eb 2024-03-12T05:55:23.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.88:6824/3057841649,v1:172.21.15.88:6825/3057841649] [v2:172.21.15.88:6826/3057841649,v1:172.21.15.88:6827/3057841649] exists,up cd7c7012-e16e-4a96-b9e0-fb8d7a5af114 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 2.5 [8,11] 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 2.8 [2,11] 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 3.0 [4,8] 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 3.5 [3,9] 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 3.6 [4,7] 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 3.14 [4,6] 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 3.1d [5,9] 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 3.25 [5,9] 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6801/306877437 expires 2024-03-13T05:46:40.781750+0000 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6800/306877437 expires 2024-03-13T05:46:40.781750+0000 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/1321850785 expires 2024-03-13T05:46:40.781750+0000 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/241684208 expires 2024-03-13T05:46:40.781750+0000 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6800/4265639592 expires 2024-03-13T05:46:12.898913+0000 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/1776556257 expires 2024-03-13T05:46:12.898913+0000 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6801/4265639592 expires 2024-03-13T05:46:12.898913+0000 2024-03-12T05:55:23.344 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/3181569618 expires 2024-03-13T05:46:12.898913+0000 2024-03-12T05:55:23.345 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/2728891083 expires 2024-03-13T05:46:12.898913+0000 2024-03-12T05:55:23.345 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/3468109661 expires 2024-03-13T05:46:40.781750+0000 2024-03-12T05:55:23.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.340+0000 7fb8f2809700 1 -- 172.21.15.76:0/4274301920 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb8d40a1ac0 msgr2=0x7fb8d40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:23.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.340+0000 7fb8f2809700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb8d40a1ac0 0x7fb8d40a3f80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec101c70 tx=0x7fb8dc009260 comp rx=0 tx=0).stop 2024-03-12T05:55:23.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.340+0000 7fb8f2809700 1 -- 172.21.15.76:0/4274301920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8ec101e10 msgr2=0x7fb8ec1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:23.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.340+0000 7fb8f2809700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8ec101e10 0x7fb8ec1031f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0003890 tx=0x7fb8e000f740 comp rx=0 tx=0).stop 2024-03-12T05:55:23.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.340+0000 7fb8f2809700 1 -- 172.21.15.76:0/4274301920 shutdown_connections 2024-03-12T05:55:23.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.340+0000 7fb8f2809700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fb8d40a1ac0 0x7fb8d40a3f80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:23.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.340+0000 7fb8f2809700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8ec1027b0 0x7fb8ec103730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:23.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.340+0000 7fb8f2809700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8ec100c10 0x7fb8ec104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:23.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.340+0000 7fb8f2809700 1 --2- 172.21.15.76:0/4274301920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb8ec101e10 0x7fb8ec1031f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:23.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.340+0000 7fb8f2809700 1 -- 172.21.15.76:0/4274301920 >> 172.21.15.76:0/4274301920 conn(0x7fb8ec0fc1d0 msgr2=0x7fb8ec0fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:23.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.340+0000 7fb8f2809700 1 -- 172.21.15.76:0/4274301920 shutdown_connections 2024-03-12T05:55:23.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:23.340+0000 7fb8f2809700 1 -- 172.21.15.76:0/4274301920 wait complete. 2024-03-12T05:55:24.102 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:23 smithi076 ceph-mon[32095]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-12T05:55:24.102 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:23 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4274301920' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-12T05:55:24.128 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-12T05:55:24.139 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:55:24.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.269+0000 7f895feb5640 1 -- 172.21.15.76:0/1106464197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581000b0 msgr2=0x7f89581004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:24.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.269+0000 7f895feb5640 1 --2- 172.21.15.76:0/1106464197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581000b0 0x7f89581004d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8954009ca0 tx=0x7f895402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.269+0000 7f895feb5640 1 -- 172.21.15.76:0/1106464197 shutdown_connections 2024-03-12T05:55:24.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.269+0000 7f895feb5640 1 --2- 172.21.15.76:0/1106464197 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8958101c60 0x7f89581084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.269+0000 7f895feb5640 1 --2- 172.21.15.76:0/1106464197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89581012a0 0x7f8958101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.269+0000 7f895feb5640 1 --2- 172.21.15.76:0/1106464197 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581000b0 0x7f89581004d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.269+0000 7f895feb5640 1 -- 172.21.15.76:0/1106464197 >> 172.21.15.76:0/1106464197 conn(0x7f89580fb620 msgr2=0x7f89580fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:24.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.269+0000 7f895feb5640 1 -- 172.21.15.76:0/1106464197 shutdown_connections 2024-03-12T05:55:24.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.269+0000 7f895feb5640 1 -- 172.21.15.76:0/1106464197 wait complete. 2024-03-12T05:55:24.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.269+0000 7f895feb5640 1 Processor -- start 2024-03-12T05:55:24.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.269+0000 7f895feb5640 1 -- start start 2024-03-12T05:55:24.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895feb5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89581000b0 0x7f8958068f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:24.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895feb5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581012a0 0x7f895819c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895feb5640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8958101c60 0x7f895819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895feb5640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895819ee00 con 0x7f89581012a0 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895feb5640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895819ef70 con 0x7f8958101c60 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895feb5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895819f130 con 0x7f89581000b0 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895d450640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581012a0 0x7f895819c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895d450640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581012a0 0x7f895819c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55254/0 (socket says 172.21.15.76:55254) 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895d450640 1 -- 172.21.15.76:0/1058664786 learned_addr learned my addr 172.21.15.76:0/1058664786 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895d450640 1 -- 172.21.15.76:0/1058664786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89581000b0 msgr2=0x7f8958068f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895dc51640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89581000b0 0x7f8958068f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895d450640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89581000b0 0x7f8958068f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895d450640 1 -- 172.21.15.76:0/1058664786 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8958101c60 msgr2=0x7f895819e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895e452640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8958101c60 0x7f895819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895d450640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8958101c60 0x7f895819e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895d450640 1 -- 172.21.15.76:0/1058664786 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8954009900 con 0x7f89581012a0 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895dc51640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89581000b0 0x7f8958068f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:24.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895e452640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8958101c60 0x7f895819e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:24.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895d450640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581012a0 0x7f895819c100 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f894800b4a0 tx=0x7f894800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:24.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f894effd640 1 -- 172.21.15.76:0/1058664786 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8948010610 con 0x7f89581012a0 2024-03-12T05:55:24.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f894effd640 1 -- 172.21.15.76:0/1058664786 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8948010c50 con 0x7f89581012a0 2024-03-12T05:55:24.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895feb5640 1 -- 172.21.15.76:0/1058664786 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f895819f410 con 0x7f89581012a0 2024-03-12T05:55:24.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f894effd640 1 -- 172.21.15.76:0/1058664786 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8948017e20 con 0x7f89581012a0 2024-03-12T05:55:24.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.270+0000 7f895feb5640 1 -- 172.21.15.76:0/1058664786 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f895819f950 con 0x7f89581012a0 2024-03-12T05:55:24.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.271+0000 7f895feb5640 1 -- 172.21.15.76:0/1058664786 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89581073d0 con 0x7f89581012a0 2024-03-12T05:55:24.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.273+0000 7f894effd640 1 -- 172.21.15.76:0/1058664786 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8948017480 con 0x7f89581012a0 2024-03-12T05:55:24.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.273+0000 7f894effd640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f89440a1a30 0x7f89440a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:24.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.274+0000 7f895dc51640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f89440a1a30 0x7f89440a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:24.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.274+0000 7f894effd640 1 -- 172.21.15.76:0/1058664786 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f89480ccf50 con 0x7f89581012a0 2024-03-12T05:55:24.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.274+0000 7f895dc51640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f89440a1a30 0x7f89440a3ef0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8954032540 tx=0x7f8954005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:24.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.277+0000 7f894effd640 1 -- 172.21.15.76:0/1058664786 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8948096170 con 0x7f89581012a0 2024-03-12T05:55:24.286 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:23 smithi088 ceph-mon[36585]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-12T05:55:24.286 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:23 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4274301920' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-12T05:55:24.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:23 smithi082 ceph-mon[36382]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-12T05:55:24.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:23 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4274301920' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-12T05:55:24.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.451+0000 7f895feb5640 1 -- 172.21.15.76:0/1058664786 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f895810bb80 con 0x7f89581012a0 2024-03-12T05:55:24.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.452+0000 7f894effd640 1 -- 172.21.15.76:0/1058664786 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22541 (secure 0 0 0) 0x7f89480092b0 con 0x7f89581012a0 2024-03-12T05:55:24.456 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:24.456 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:54:42.007701+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:55:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.454+0000 7f895feb5640 1 -- 172.21.15.76:0/1058664786 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f89440a1a30 msgr2=0x7f89440a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.454+0000 7f895feb5640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f89440a1a30 0x7f89440a3ef0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8954032540 tx=0x7f8954005c50 comp rx=0 tx=0).stop 2024-03-12T05:55:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.454+0000 7f895feb5640 1 -- 172.21.15.76:0/1058664786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581012a0 msgr2=0x7f895819c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.454+0000 7f895feb5640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581012a0 0x7f895819c100 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f894800b4a0 tx=0x7f894800b960 comp rx=0 tx=0).stop 2024-03-12T05:55:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.454+0000 7f895feb5640 1 -- 172.21.15.76:0/1058664786 shutdown_connections 2024-03-12T05:55:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.454+0000 7f895feb5640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f89440a1a30 0x7f89440a3ef0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.454+0000 7f895feb5640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8958101c60 0x7f895819e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.454+0000 7f895feb5640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f89581012a0 0x7f895819c100 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.454+0000 7f895feb5640 1 --2- 172.21.15.76:0/1058664786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89581000b0 0x7f8958068f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.454+0000 7f895feb5640 1 -- 172.21.15.76:0/1058664786 >> 172.21.15.76:0/1058664786 conn(0x7f89580fb620 msgr2=0x7f89581045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.454+0000 7f895feb5640 1 -- 172.21.15.76:0/1058664786 shutdown_connections 2024-03-12T05:55:24.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.454+0000 7f895feb5640 1 -- 172.21.15.76:0/1058664786 wait complete. 2024-03-12T05:55:24.469 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:49:09.125536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:53:35.663129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:53:36.740826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:55:24.470 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:55:24.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.572+0000 7f6074113640 1 -- 172.21.15.76:0/4059396778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f606c100a10 msgr2=0x7f606c100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:24.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.572+0000 7f6074113640 1 --2- 172.21.15.76:0/4059396778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f606c100a10 0x7f606c100e90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6060009900 tx=0x7f606002bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:24.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.572+0000 7f6074113640 1 -- 172.21.15.76:0/4059396778 shutdown_connections 2024-03-12T05:55:24.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.572+0000 7f6074113640 1 --2- 172.21.15.76:0/4059396778 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f606c1013d0 0x7f606c107c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.572+0000 7f6074113640 1 --2- 172.21.15.76:0/4059396778 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f606c100a10 0x7f606c100e90 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.572+0000 7f6074113640 1 --2- 172.21.15.76:0/4059396778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f606c0ff820 0x7f606c0ffc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.572+0000 7f6074113640 1 -- 172.21.15.76:0/4059396778 >> 172.21.15.76:0/4059396778 conn(0x7f606c0fad90 msgr2=0x7f606c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.572+0000 7f6074113640 1 -- 172.21.15.76:0/4059396778 shutdown_connections 2024-03-12T05:55:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.572+0000 7f6074113640 1 -- 172.21.15.76:0/4059396778 wait complete. 2024-03-12T05:55:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.572+0000 7f6074113640 1 Processor -- start 2024-03-12T05:55:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.572+0000 7f6074113640 1 -- start start 2024-03-12T05:55:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6074113640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f606c0ff820 0x7f606c197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6074113640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f606c100a10 0x7f606c1978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6074113640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f606c1013d0 0x7f606c19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6074113640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c19ee80 con 0x7f606c0ff820 2024-03-12T05:55:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6074113640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c19eff0 con 0x7f606c1013d0 2024-03-12T05:55:24.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6074113640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c19f1b0 con 0x7f606c100a10 2024-03-12T05:55:24.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6071eaf640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f606c0ff820 0x7f606c197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:24.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6071eaf640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f606c0ff820 0x7f606c197390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55270/0 (socket says 172.21.15.76:55270) 2024-03-12T05:55:24.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6071eaf640 1 -- 172.21.15.76:0/2772199330 learned_addr learned my addr 172.21.15.76:0/2772199330 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:24.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f60716ae640 1 --2- 172.21.15.76:0/2772199330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f606c100a10 0x7f606c1978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:24.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f60726b0640 1 --2- 172.21.15.76:0/2772199330 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f606c1013d0 0x7f606c19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:24.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6071eaf640 1 -- 172.21.15.76:0/2772199330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f606c100a10 msgr2=0x7f606c1978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:24.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6071eaf640 1 --2- 172.21.15.76:0/2772199330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f606c100a10 0x7f606c1978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6071eaf640 1 -- 172.21.15.76:0/2772199330 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f606c1013d0 msgr2=0x7f606c19e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:24.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6071eaf640 1 --2- 172.21.15.76:0/2772199330 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f606c1013d0 0x7f606c19e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6071eaf640 1 -- 172.21.15.76:0/2772199330 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6060009560 con 0x7f606c0ff820 2024-03-12T05:55:24.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6071eaf640 1 --2- 172.21.15.76:0/2772199330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f606c0ff820 0x7f606c197390 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f606800b450 tx=0x7f606800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:24.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f605effd640 1 -- 172.21.15.76:0/2772199330 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068011890 con 0x7f606c0ff820 2024-03-12T05:55:24.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6074113640 1 -- 172.21.15.76:0/2772199330 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606c19f490 con 0x7f606c0ff820 2024-03-12T05:55:24.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f605effd640 1 -- 172.21.15.76:0/2772199330 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60680103e0 con 0x7f606c0ff820 2024-03-12T05:55:24.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f605effd640 1 -- 172.21.15.76:0/2772199330 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606800f580 con 0x7f606c0ff820 2024-03-12T05:55:24.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.573+0000 7f6074113640 1 -- 172.21.15.76:0/2772199330 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c19f9a0 con 0x7f606c0ff820 2024-03-12T05:55:24.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.574+0000 7f6074113640 1 -- 172.21.15.76:0/2772199330 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606c106bc0 con 0x7f606c0ff820 2024-03-12T05:55:24.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.575+0000 7f605effd640 1 -- 172.21.15.76:0/2772199330 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f60680108e0 con 0x7f606c0ff820 2024-03-12T05:55:24.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.576+0000 7f605effd640 1 --2- 172.21.15.76:0/2772199330 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f60580a1a30 0x7f60580a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:24.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.576+0000 7f60716ae640 1 --2- 172.21.15.76:0/2772199330 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f60580a1a30 0x7f60580a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:24.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.576+0000 7f605effd640 1 -- 172.21.15.76:0/2772199330 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f60680cd5d0 con 0x7f606c0ff820 2024-03-12T05:55:24.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.577+0000 7f60716ae640 1 --2- 172.21.15.76:0/2772199330 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f60580a1a30 0x7f60580a3ef0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6060006010 tx=0x7f6060005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:24.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.580+0000 7f605effd640 1 -- 172.21.15.76:0/2772199330 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6068096790 con 0x7f606c0ff820 2024-03-12T05:55:24.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.754+0000 7f6074113640 1 -- 172.21.15.76:0/2772199330 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f606c02cdd0 con 0x7f606c0ff820 2024-03-12T05:55:24.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.754+0000 7f605effd640 1 -- 172.21.15.76:0/2772199330 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6068099e50 con 0x7f606c0ff820 2024-03-12T05:55:24.757 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-12T05:55:24.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.756+0000 7f6074113640 1 -- 172.21.15.76:0/2772199330 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f60580a1a30 msgr2=0x7f60580a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:24.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.756+0000 7f6074113640 1 --2- 172.21.15.76:0/2772199330 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f60580a1a30 0x7f60580a3ef0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6060006010 tx=0x7f6060005d90 comp rx=0 tx=0).stop 2024-03-12T05:55:24.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.756+0000 7f6074113640 1 -- 172.21.15.76:0/2772199330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f606c0ff820 msgr2=0x7f606c197390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:24.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.756+0000 7f6074113640 1 --2- 172.21.15.76:0/2772199330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f606c0ff820 0x7f606c197390 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f606800b450 tx=0x7f606800b910 comp rx=0 tx=0).stop 2024-03-12T05:55:24.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.756+0000 7f6074113640 1 -- 172.21.15.76:0/2772199330 shutdown_connections 2024-03-12T05:55:24.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.756+0000 7f6074113640 1 --2- 172.21.15.76:0/2772199330 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f60580a1a30 0x7f60580a3ef0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.756+0000 7f6074113640 1 --2- 172.21.15.76:0/2772199330 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f606c1013d0 0x7f606c19e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.756+0000 7f6074113640 1 --2- 172.21.15.76:0/2772199330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f606c100a10 0x7f606c1978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.756+0000 7f6074113640 1 --2- 172.21.15.76:0/2772199330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f606c0ff820 0x7f606c197390 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.756+0000 7f6074113640 1 -- 172.21.15.76:0/2772199330 >> 172.21.15.76:0/2772199330 conn(0x7f606c0fad90 msgr2=0x7f606c103d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:24.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.756+0000 7f6074113640 1 -- 172.21.15.76:0/2772199330 shutdown_connections 2024-03-12T05:55:24.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.756+0000 7f6074113640 1 -- 172.21.15.76:0/2772199330 wait complete. 2024-03-12T05:55:24.771 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:55:24.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 -- 172.21.15.76:0/1441345223 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e2c101280 msgr2=0x7f0e2c101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 --2- 172.21.15.76:0/1441345223 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e2c101280 0x7f0e2c101700 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0e14009900 tx=0x7f0e1402bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 -- 172.21.15.76:0/1441345223 shutdown_connections 2024-03-12T05:55:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 --2- 172.21.15.76:0/1441345223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e2c101c40 0x7f0e2c1084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 --2- 172.21.15.76:0/1441345223 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e2c101280 0x7f0e2c101700 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 --2- 172.21.15.76:0/1441345223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e2c100090 0x7f0e2c1004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 -- 172.21.15.76:0/1441345223 >> 172.21.15.76:0/1441345223 conn(0x7f0e2c0fb5e0 msgr2=0x7f0e2c0fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 -- 172.21.15.76:0/1441345223 shutdown_connections 2024-03-12T05:55:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 -- 172.21.15.76:0/1441345223 wait complete. 2024-03-12T05:55:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 Processor -- start 2024-03-12T05:55:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 -- start start 2024-03-12T05:55:24.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e2c100090 0x7f0e2c197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:24.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e2c101280 0x7f0e2c1977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:24.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e2c101c40 0x7f0e2c19e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:24.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c19ed50 con 0x7f0e2c100090 2024-03-12T05:55:24.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c19eec0 con 0x7f0e2c101280 2024-03-12T05:55:24.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.872+0000 7f0e31478640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c19f080 con 0x7f0e2c101c40 2024-03-12T05:55:24.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e2affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e2c100090 0x7f0e2c197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:24.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e2affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e2c100090 0x7f0e2c197260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55284/0 (socket says 172.21.15.76:55284) 2024-03-12T05:55:24.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e2affd640 1 -- 172.21.15.76:0/183728147 learned_addr learned my addr 172.21.15.76:0/183728147 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:24.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e2a7fc640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e2c101280 0x7f0e2c1977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:24.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e2b7fe640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e2c101c40 0x7f0e2c19e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:24.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e2affd640 1 -- 172.21.15.76:0/183728147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e2c101c40 msgr2=0x7f0e2c19e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:24.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e2affd640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e2c101c40 0x7f0e2c19e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e2affd640 1 -- 172.21.15.76:0/183728147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e2c101280 msgr2=0x7f0e2c1977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:24.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e2affd640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e2c101280 0x7f0e2c1977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:24.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e2affd640 1 -- 172.21.15.76:0/183728147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e14009560 con 0x7f0e2c100090 2024-03-12T05:55:24.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e2a7fc640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e2c101280 0x7f0e2c1977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:24.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e2affd640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e2c100090 0x7f0e2c197260 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00d950 tx=0x7f0e1c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:24.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e23fff640 1 -- 172.21.15.76:0/183728147 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c009c30 con 0x7f0e2c100090 2024-03-12T05:55:24.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e23fff640 1 -- 172.21.15.76:0/183728147 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e1c009d90 con 0x7f0e2c100090 2024-03-12T05:55:24.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e31478640 1 -- 172.21.15.76:0/183728147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2c19f360 con 0x7f0e2c100090 2024-03-12T05:55:24.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e23fff640 1 -- 172.21.15.76:0/183728147 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c00f720 con 0x7f0e2c100090 2024-03-12T05:55:24.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.873+0000 7f0e31478640 1 -- 172.21.15.76:0/183728147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2c19f820 con 0x7f0e2c100090 2024-03-12T05:55:24.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.874+0000 7f0e31478640 1 -- 172.21.15.76:0/183728147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2c107400 con 0x7f0e2c100090 2024-03-12T05:55:24.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.876+0000 7f0e23fff640 1 -- 172.21.15.76:0/183728147 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0e1c010a60 con 0x7f0e2c100090 2024-03-12T05:55:24.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.876+0000 7f0e23fff640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0e180a1a30 0x7f0e180a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:24.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.877+0000 7f0e2a7fc640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0e180a1a30 0x7f0e180a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:24.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.877+0000 7f0e23fff640 1 -- 172.21.15.76:0/183728147 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f0e1c0cd1a0 con 0x7f0e2c100090 2024-03-12T05:55:24.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.877+0000 7f0e2a7fc640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0e180a1a30 0x7f0e180a3ef0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c198770 tx=0x7f0e14005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:24.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:24.880+0000 7f0e23fff640 1 -- 172.21.15.76:0/183728147 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0e1c0963c0 con 0x7f0e2c100090 2024-03-12T05:55:25.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.053+0000 7f0e31478640 1 -- 172.21.15.76:0/183728147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0e2c10cd50 con 0x7f0e2c100090 2024-03-12T05:55:25.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.053+0000 7f0e23fff640 1 -- 172.21.15.76:0/183728147 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0e1c021020 con 0x7f0e2c100090 2024-03-12T05:55:25.057 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-12T05:55:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.055+0000 7f0e31478640 1 -- 172.21.15.76:0/183728147 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0e180a1a30 msgr2=0x7f0e180a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.055+0000 7f0e31478640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0e180a1a30 0x7f0e180a3ef0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c198770 tx=0x7f0e14005be0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.055+0000 7f0e31478640 1 -- 172.21.15.76:0/183728147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e2c100090 msgr2=0x7f0e2c197260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.055+0000 7f0e31478640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e2c100090 0x7f0e2c197260 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00d950 tx=0x7f0e1c00de10 comp rx=0 tx=0).stop 2024-03-12T05:55:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.055+0000 7f0e31478640 1 -- 172.21.15.76:0/183728147 shutdown_connections 2024-03-12T05:55:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.056+0000 7f0e31478640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0e180a1a30 0x7f0e180a3ef0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.056+0000 7f0e31478640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e2c101c40 0x7f0e2c19e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.056+0000 7f0e31478640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0e2c101280 0x7f0e2c1977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.056+0000 7f0e31478640 1 --2- 172.21.15.76:0/183728147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0e2c100090 0x7f0e2c197260 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.056+0000 7f0e31478640 1 -- 172.21.15.76:0/183728147 >> 172.21.15.76:0/183728147 conn(0x7f0e2c0fb5e0 msgr2=0x7f0e2c1045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.056+0000 7f0e31478640 1 -- 172.21.15.76:0/183728147 shutdown_connections 2024-03-12T05:55:25.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.056+0000 7f0e31478640 1 -- 172.21.15.76:0/183728147 wait complete. 2024-03-12T05:55:25.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:24 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1058664786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:25.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:24 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2772199330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:25.070 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:55:25.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.170+0000 7f14354f4640 1 -- 172.21.15.76:0/2284251970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1430102380 msgr2=0x7f1430102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.170+0000 7f14354f4640 1 --2- 172.21.15.76:0/2284251970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1430102380 0x7f1430102800 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1420009900 tx=0x7f142002bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:25.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.170+0000 7f14354f4640 1 -- 172.21.15.76:0/2284251970 shutdown_connections 2024-03-12T05:55:25.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.170+0000 7f14354f4640 1 --2- 172.21.15.76:0/2284251970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1430102d40 0x7f14301095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.171+0000 7f14354f4640 1 --2- 172.21.15.76:0/2284251970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1430102380 0x7f1430102800 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.171+0000 7f14354f4640 1 --2- 172.21.15.76:0/2284251970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1430101190 0x7f14301015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.171+0000 7f14354f4640 1 -- 172.21.15.76:0/2284251970 >> 172.21.15.76:0/2284251970 conn(0x7f14300fc700 msgr2=0x7f14300feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:25.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.171+0000 7f14354f4640 1 -- 172.21.15.76:0/2284251970 shutdown_connections 2024-03-12T05:55:25.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.171+0000 7f14354f4640 1 -- 172.21.15.76:0/2284251970 wait complete. 2024-03-12T05:55:25.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.171+0000 7f14354f4640 1 Processor -- start 2024-03-12T05:55:25.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.171+0000 7f14354f4640 1 -- start start 2024-03-12T05:55:25.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.171+0000 7f14354f4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1430101190 0x7f1430197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:25.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.171+0000 7f14354f4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1430102380 0x7f1430197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:25.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.171+0000 7f14354f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1430102d40 0x7f143019e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:25.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.171+0000 7f14354f4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143019ed20 con 0x7f1430101190 2024-03-12T05:55:25.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.171+0000 7f14354f4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143019ee90 con 0x7f1430102380 2024-03-12T05:55:25.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.171+0000 7f14354f4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143019f050 con 0x7f1430102d40 2024-03-12T05:55:25.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f142effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1430101190 0x7f1430197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:25.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f142effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1430101190 0x7f1430197230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55298/0 (socket says 172.21.15.76:55298) 2024-03-12T05:55:25.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f142effd640 1 -- 172.21.15.76:0/548206906 learned_addr learned my addr 172.21.15.76:0/548206906 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:25.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f142e7fc640 1 --2- 172.21.15.76:0/548206906 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1430102380 0x7f1430197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:25.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f142f7fe640 1 --2- 172.21.15.76:0/548206906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1430102d40 0x7f143019e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:25.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f142effd640 1 -- 172.21.15.76:0/548206906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1430102d40 msgr2=0x7f143019e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f142effd640 1 --2- 172.21.15.76:0/548206906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1430102d40 0x7f143019e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f142effd640 1 -- 172.21.15.76:0/548206906 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1430102380 msgr2=0x7f1430197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f142effd640 1 --2- 172.21.15.76:0/548206906 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1430102380 0x7f1430197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f142effd640 1 -- 172.21.15.76:0/548206906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1420009560 con 0x7f1430101190 2024-03-12T05:55:25.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f142effd640 1 --2- 172.21.15.76:0/548206906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1430101190 0x7f1430197230 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f142400d770 tx=0x7f142400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:25.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f1417fff640 1 -- 172.21.15.76:0/548206906 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1424009c30 con 0x7f1430101190 2024-03-12T05:55:25.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f14354f4640 1 -- 172.21.15.76:0/548206906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143019f330 con 0x7f1430101190 2024-03-12T05:55:25.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f1417fff640 1 -- 172.21.15.76:0/548206906 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f142400de80 con 0x7f1430101190 2024-03-12T05:55:25.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f1417fff640 1 -- 172.21.15.76:0/548206906 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1424010460 con 0x7f1430101190 2024-03-12T05:55:25.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.172+0000 7f14354f4640 1 -- 172.21.15.76:0/548206906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f143019f810 con 0x7f1430101190 2024-03-12T05:55:25.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.173+0000 7f14354f4640 1 -- 172.21.15.76:0/548206906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1430108530 con 0x7f1430101190 2024-03-12T05:55:25.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.174+0000 7f1417fff640 1 -- 172.21.15.76:0/548206906 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f1424003f60 con 0x7f1430101190 2024-03-12T05:55:25.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.175+0000 7f1417fff640 1 --2- 172.21.15.76:0/548206906 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f14180a1a30 0x7f14180a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:25.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.175+0000 7f142e7fc640 1 --2- 172.21.15.76:0/548206906 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f14180a1a30 0x7f14180a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:25.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.175+0000 7f1417fff640 1 -- 172.21.15.76:0/548206906 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f14240cd100 con 0x7f1430101190 2024-03-12T05:55:25.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.175+0000 7f142e7fc640 1 --2- 172.21.15.76:0/548206906 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f14180a1a30 0x7f14180a3ef0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1420009230 tx=0x7f1420005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:25.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.179+0000 7f1417fff640 1 -- 172.21.15.76:0/548206906 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1424096320 con 0x7f1430101190 2024-03-12T05:55:25.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.354+0000 7f14354f4640 1 -- 172.21.15.76:0/548206906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f143002ce40 con 0x7f1430101190 2024-03-12T05:55:25.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.355+0000 7f1417fff640 1 -- 172.21.15.76:0/548206906 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1424020020 con 0x7f1430101190 2024-03-12T05:55:25.358 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-12T05:55:25.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.357+0000 7f14354f4640 1 -- 172.21.15.76:0/548206906 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f14180a1a30 msgr2=0x7f14180a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.357+0000 7f14354f4640 1 --2- 172.21.15.76:0/548206906 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f14180a1a30 0x7f14180a3ef0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1420009230 tx=0x7f1420005fd0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.357+0000 7f14354f4640 1 -- 172.21.15.76:0/548206906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1430101190 msgr2=0x7f1430197230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.357+0000 7f14354f4640 1 --2- 172.21.15.76:0/548206906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1430101190 0x7f1430197230 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f142400d770 tx=0x7f142400dc30 comp rx=0 tx=0).stop 2024-03-12T05:55:25.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.357+0000 7f14354f4640 1 -- 172.21.15.76:0/548206906 shutdown_connections 2024-03-12T05:55:25.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.357+0000 7f14354f4640 1 --2- 172.21.15.76:0/548206906 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f14180a1a30 0x7f14180a3ef0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.357+0000 7f14354f4640 1 --2- 172.21.15.76:0/548206906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1430102d40 0x7f143019e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.357+0000 7f14354f4640 1 --2- 172.21.15.76:0/548206906 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1430102380 0x7f1430197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.357+0000 7f14354f4640 1 --2- 172.21.15.76:0/548206906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1430101190 0x7f1430197230 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.357+0000 7f14354f4640 1 -- 172.21.15.76:0/548206906 >> 172.21.15.76:0/548206906 conn(0x7f14300fc700 msgr2=0x7f1430105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:25.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.357+0000 7f14354f4640 1 -- 172.21.15.76:0/548206906 shutdown_connections 2024-03-12T05:55:25.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.357+0000 7f14354f4640 1 -- 172.21.15.76:0/548206906 wait complete. 2024-03-12T05:55:25.372 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:55:25.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:24 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1058664786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:25.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:24 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2772199330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:25.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.470+0000 7fda2cce1640 1 -- 172.21.15.76:0/211703636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda281045d0 msgr2=0x7fda281069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.470+0000 7fda2cce1640 1 --2- 172.21.15.76:0/211703636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda281045d0 0x7fda281069c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fda18009900 tx=0x7fda1802bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:25.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.470+0000 7fda2cce1640 1 -- 172.21.15.76:0/211703636 shutdown_connections 2024-03-12T05:55:25.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.470+0000 7fda2cce1640 1 --2- 172.21.15.76:0/211703636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda28106f00 0x7fda28109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.470+0000 7fda2cce1640 1 --2- 172.21.15.76:0/211703636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda281045d0 0x7fda281069c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.470+0000 7fda2cce1640 1 --2- 172.21.15.76:0/211703636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda28101ca0 0x7fda28104090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.470+0000 7fda2cce1640 1 -- 172.21.15.76:0/211703636 >> 172.21.15.76:0/211703636 conn(0x7fda280fb690 msgr2=0x7fda280fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:25.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.470+0000 7fda2cce1640 1 -- 172.21.15.76:0/211703636 shutdown_connections 2024-03-12T05:55:25.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.470+0000 7fda2cce1640 1 -- 172.21.15.76:0/211703636 wait complete. 2024-03-12T05:55:25.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.470+0000 7fda2cce1640 1 Processor -- start 2024-03-12T05:55:25.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.470+0000 7fda2cce1640 1 -- start start 2024-03-12T05:55:25.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda2cce1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda28101ca0 0x7fda281972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:25.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda2cce1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda281045d0 0x7fda281977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:25.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda2cce1640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda28106f00 0x7fda2819e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:25.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda2cce1640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2819ed90 con 0x7fda281045d0 2024-03-12T05:55:25.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda2cce1640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2819ef00 con 0x7fda28106f00 2024-03-12T05:55:25.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda2cce1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2819f0c0 con 0x7fda28101ca0 2024-03-12T05:55:25.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda25d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda281045d0 0x7fda281977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:25.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda25d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda281045d0 0x7fda281977e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55330/0 (socket says 172.21.15.76:55330) 2024-03-12T05:55:25.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda25d9b640 1 -- 172.21.15.76:0/621035395 learned_addr learned my addr 172.21.15.76:0/621035395 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:25.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda2659c640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda28101ca0 0x7fda281972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:25.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda26d9d640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda28106f00 0x7fda2819e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:25.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda25d9b640 1 -- 172.21.15.76:0/621035395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda28101ca0 msgr2=0x7fda281972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda25d9b640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda28101ca0 0x7fda281972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda25d9b640 1 -- 172.21.15.76:0/621035395 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda28106f00 msgr2=0x7fda2819e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda25d9b640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda28106f00 0x7fda2819e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda25d9b640 1 -- 172.21.15.76:0/621035395 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda18009560 con 0x7fda281045d0 2024-03-12T05:55:25.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda2659c640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda28101ca0 0x7fda281972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:25.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda26d9d640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda28106f00 0x7fda2819e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:25.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda25d9b640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda281045d0 0x7fda281977e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fda18036040 tx=0x7fda180042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:25.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda1f7fe640 1 -- 172.21.15.76:0/621035395 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda180044e0 con 0x7fda281045d0 2024-03-12T05:55:25.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda2cce1640 1 -- 172.21.15.76:0/621035395 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2819f340 con 0x7fda281045d0 2024-03-12T05:55:25.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda1f7fe640 1 -- 172.21.15.76:0/621035395 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda180054e0 con 0x7fda281045d0 2024-03-12T05:55:25.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda1f7fe640 1 -- 172.21.15.76:0/621035395 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1803e450 con 0x7fda281045d0 2024-03-12T05:55:25.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.471+0000 7fda2cce1640 1 -- 172.21.15.76:0/621035395 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2819f8a0 con 0x7fda281045d0 2024-03-12T05:55:25.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.472+0000 7fda2cce1640 1 -- 172.21.15.76:0/621035395 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda280fd180 con 0x7fda281045d0 2024-03-12T05:55:25.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.473+0000 7fda1f7fe640 1 -- 172.21.15.76:0/621035395 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fda18003710 con 0x7fda281045d0 2024-03-12T05:55:25.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.474+0000 7fda1f7fe640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fda140a1a30 0x7fda140a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:25.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.474+0000 7fda1f7fe640 1 -- 172.21.15.76:0/621035395 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fda180efef0 con 0x7fda281045d0 2024-03-12T05:55:25.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.474+0000 7fda2659c640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fda140a1a30 0x7fda140a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:25.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.474+0000 7fda2659c640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fda140a1a30 0x7fda140a3ef0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fda100058c0 tx=0x7fda10005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:25.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.477+0000 7fda1f7fe640 1 -- 172.21.15.76:0/621035395 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fda180b9110 con 0x7fda281045d0 2024-03-12T05:55:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:24 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1058664786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:24 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2772199330' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:25.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.686+0000 7fda2cce1640 1 -- 172.21.15.76:0/621035395 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda2810cbc0 con 0x7fda281045d0 2024-03-12T05:55:25.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.687+0000 7fda1f7fe640 1 -- 172.21.15.76:0/621035395 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+11328 (secure 0 0 0) 0x7fda180bc7d0 con 0x7fda281045d0 2024-03-12T05:55:25.690 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:25.691 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:53:42.384114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:55:25.691 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 21 2024-03-12T05:55:25.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.689+0000 7fda2cce1640 1 -- 172.21.15.76:0/621035395 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fda140a1a30 msgr2=0x7fda140a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.689+0000 7fda2cce1640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fda140a1a30 0x7fda140a3ef0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fda100058c0 tx=0x7fda10005d30 comp rx=0 tx=0).stop 2024-03-12T05:55:25.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.689+0000 7fda2cce1640 1 -- 172.21.15.76:0/621035395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda281045d0 msgr2=0x7fda281977e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.689+0000 7fda2cce1640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda281045d0 0x7fda281977e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fda18036040 tx=0x7fda180042f0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.689+0000 7fda2cce1640 1 -- 172.21.15.76:0/621035395 shutdown_connections 2024-03-12T05:55:25.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.689+0000 7fda2cce1640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fda140a1a30 0x7fda140a3ef0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.689+0000 7fda2cce1640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda28106f00 0x7fda2819e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.689+0000 7fda2cce1640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda281045d0 0x7fda281977e0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.689+0000 7fda2cce1640 1 --2- 172.21.15.76:0/621035395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda28101ca0 0x7fda281972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.689+0000 7fda2cce1640 1 -- 172.21.15.76:0/621035395 >> 172.21.15.76:0/621035395 conn(0x7fda280fb690 msgr2=0x7fda281025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:25.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.689+0000 7fda2cce1640 1 -- 172.21.15.76:0/621035395 shutdown_connections 2024-03-12T05:55:25.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.689+0000 7fda2cce1640 1 -- 172.21.15.76:0/621035395 wait complete. 2024-03-12T05:55:25.704 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:55:25.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.802+0000 7faa8fde2640 1 -- 172.21.15.76:0/855204703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa88102400 msgr2=0x7faa88102880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.802+0000 7faa8fde2640 1 --2- 172.21.15.76:0/855204703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa88102400 0x7faa88102880 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7faa84009900 tx=0x7faa8402bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.802+0000 7faa8fde2640 1 -- 172.21.15.76:0/855204703 shutdown_connections 2024-03-12T05:55:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.802+0000 7faa8fde2640 1 --2- 172.21.15.76:0/855204703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa88102dc0 0x7faa88109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.802+0000 7faa8fde2640 1 --2- 172.21.15.76:0/855204703 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa88102400 0x7faa88102880 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.802+0000 7faa8fde2640 1 --2- 172.21.15.76:0/855204703 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa88101210 0x7faa88101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.802+0000 7faa8fde2640 1 -- 172.21.15.76:0/855204703 >> 172.21.15.76:0/855204703 conn(0x7faa880fc7a0 msgr2=0x7faa880febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.802+0000 7faa8fde2640 1 -- 172.21.15.76:0/855204703 shutdown_connections 2024-03-12T05:55:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.802+0000 7faa8fde2640 1 -- 172.21.15.76:0/855204703 wait complete. 2024-03-12T05:55:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.802+0000 7faa8fde2640 1 Processor -- start 2024-03-12T05:55:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.802+0000 7faa8fde2640 1 -- start start 2024-03-12T05:55:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8fde2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa88101210 0x7faa88074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8fde2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa88102400 0x7faa88073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8fde2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa88102dc0 0x7faa88073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8fde2640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa88074210 con 0x7faa88101210 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8fde2640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa88074380 con 0x7faa88102400 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8fde2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa881a11b0 con 0x7faa88102dc0 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8db7e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa88101210 0x7faa88074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8db7e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa88101210 0x7faa88074f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55340/0 (socket says 172.21.15.76:55340) 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8db7e640 1 -- 172.21.15.76:0/2952781629 learned_addr learned my addr 172.21.15.76:0/2952781629 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8d37d640 1 --2- 172.21.15.76:0/2952781629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa88102400 0x7faa88073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8db7e640 1 -- 172.21.15.76:0/2952781629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa88102dc0 msgr2=0x7faa88073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8db7e640 1 --2- 172.21.15.76:0/2952781629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa88102dc0 0x7faa88073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8db7e640 1 -- 172.21.15.76:0/2952781629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa88102400 msgr2=0x7faa88073570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8db7e640 1 --2- 172.21.15.76:0/2952781629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa88102400 0x7faa88073570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8db7e640 1 -- 172.21.15.76:0/2952781629 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa84009560 con 0x7faa88101210 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8db7e640 1 --2- 172.21.15.76:0/2952781629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa88101210 0x7faa88074f30 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faa7c00d980 tx=0x7faa7c00de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa7affd640 1 -- 172.21.15.76:0/2952781629 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c009c30 con 0x7faa88101210 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa7affd640 1 -- 172.21.15.76:0/2952781629 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa7c009d90 con 0x7faa88101210 2024-03-12T05:55:25.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8fde2640 1 -- 172.21.15.76:0/2952781629 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa881a1460 con 0x7faa88101210 2024-03-12T05:55:25.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa7affd640 1 -- 172.21.15.76:0/2952781629 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c00f7c0 con 0x7faa88101210 2024-03-12T05:55:25.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.803+0000 7faa8fde2640 1 -- 172.21.15.76:0/2952781629 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa881a1990 con 0x7faa88101210 2024-03-12T05:55:25.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.804+0000 7faa8fde2640 1 -- 172.21.15.76:0/2952781629 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa88108460 con 0x7faa88101210 2024-03-12T05:55:25.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.805+0000 7faa7affd640 1 -- 172.21.15.76:0/2952781629 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7faa7c010460 con 0x7faa88101210 2024-03-12T05:55:25.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.806+0000 7faa7affd640 1 --2- 172.21.15.76:0/2952781629 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faa740a1ab0 0x7faa740a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:25.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.806+0000 7faa8d37d640 1 --2- 172.21.15.76:0/2952781629 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faa740a1ab0 0x7faa740a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:25.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.806+0000 7faa7affd640 1 -- 172.21.15.76:0/2952781629 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7faa7c0cd680 con 0x7faa88101210 2024-03-12T05:55:25.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.806+0000 7faa8d37d640 1 --2- 172.21.15.76:0/2952781629 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faa740a1ab0 0x7faa740a3f70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7faa84006010 tx=0x7faa84000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:25.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.810+0000 7faa7affd640 1 -- 172.21.15.76:0/2952781629 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faa7c0968a0 con 0x7faa88101210 2024-03-12T05:55:25.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.985+0000 7faa8fde2640 1 -- 172.21.15.76:0/2952781629 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa8802ce70 con 0x7faa88101210 2024-03-12T05:55:25.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.987+0000 7faa7affd640 1 -- 172.21.15.76:0/2952781629 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22541 (secure 0 0 0) 0x7faa7c096070 con 0x7faa88101210 2024-03-12T05:55:25.990 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:25.990 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:54:42.007701+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:55:25.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.988+0000 7faa8fde2640 1 -- 172.21.15.76:0/2952781629 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faa740a1ab0 msgr2=0x7faa740a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.988+0000 7faa8fde2640 1 --2- 172.21.15.76:0/2952781629 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faa740a1ab0 0x7faa740a3f70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7faa84006010 tx=0x7faa84000bc0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.988+0000 7faa8fde2640 1 -- 172.21.15.76:0/2952781629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa88101210 msgr2=0x7faa88074f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:25.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.988+0000 7faa8fde2640 1 --2- 172.21.15.76:0/2952781629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa88101210 0x7faa88074f30 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faa7c00d980 tx=0x7faa7c00de40 comp rx=0 tx=0).stop 2024-03-12T05:55:25.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.989+0000 7faa8fde2640 1 -- 172.21.15.76:0/2952781629 shutdown_connections 2024-03-12T05:55:25.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.989+0000 7faa8fde2640 1 --2- 172.21.15.76:0/2952781629 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faa740a1ab0 0x7faa740a3f70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.989+0000 7faa8fde2640 1 --2- 172.21.15.76:0/2952781629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa88102dc0 0x7faa88073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.989+0000 7faa8fde2640 1 --2- 172.21.15.76:0/2952781629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa88102400 0x7faa88073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.989+0000 7faa8fde2640 1 --2- 172.21.15.76:0/2952781629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa88101210 0x7faa88074f30 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:25.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.989+0000 7faa8fde2640 1 -- 172.21.15.76:0/2952781629 >> 172.21.15.76:0/2952781629 conn(0x7faa880fc7a0 msgr2=0x7faa88105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:25.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.989+0000 7faa8fde2640 1 -- 172.21.15.76:0/2952781629 shutdown_connections 2024-03-12T05:55:25.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:25.989+0000 7faa8fde2640 1 -- 172.21.15.76:0/2952781629 wait complete. 2024-03-12T05:55:26.004 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:49:09.125536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:53:35.663129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:53:36.740826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:55:26.004 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:55:26.111 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.105+0000 7f8c4553a640 1 -- 172.21.15.76:0/2542905403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c40101290 msgr2=0x7f8c40101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.105+0000 7f8c4553a640 1 --2- 172.21.15.76:0/2542905403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c40101290 0x7f8c40101710 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8c28009900 tx=0x7f8c2802bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:26.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.105+0000 7f8c4553a640 1 -- 172.21.15.76:0/2542905403 shutdown_connections 2024-03-12T05:55:26.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.105+0000 7f8c4553a640 1 --2- 172.21.15.76:0/2542905403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c40101c50 0x7f8c401084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.105+0000 7f8c4553a640 1 --2- 172.21.15.76:0/2542905403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c40101290 0x7f8c40101710 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.105+0000 7f8c4553a640 1 --2- 172.21.15.76:0/2542905403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c401000a0 0x7f8c401004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.105+0000 7f8c4553a640 1 -- 172.21.15.76:0/2542905403 >> 172.21.15.76:0/2542905403 conn(0x7f8c400fb630 msgr2=0x7f8c400fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:26.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.106+0000 7f8c4553a640 1 -- 172.21.15.76:0/2542905403 shutdown_connections 2024-03-12T05:55:26.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.106+0000 7f8c4553a640 1 -- 172.21.15.76:0/2542905403 wait complete. 2024-03-12T05:55:26.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.106+0000 7f8c4553a640 1 Processor -- start 2024-03-12T05:55:26.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.106+0000 7f8c4553a640 1 -- start start 2024-03-12T05:55:26.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.106+0000 7f8c4553a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c401000a0 0x7f8c4006a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:26.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.106+0000 7f8c4553a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c40101290 0x7f8c4019e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:26.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.106+0000 7f8c4553a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c40101c50 0x7f8c401a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:26.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.106+0000 7f8c4553a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c401a0f40 con 0x7f8c40101290 2024-03-12T05:55:26.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.106+0000 7f8c4553a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c401a10b0 con 0x7f8c40101c50 2024-03-12T05:55:26.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.106+0000 7f8c4553a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c401a1270 con 0x7f8c401000a0 2024-03-12T05:55:26.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.106+0000 7f8c3e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c40101290 0x7f8c4019e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:26.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.106+0000 7f8c3e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c40101290 0x7f8c4019e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55362/0 (socket says 172.21.15.76:55362) 2024-03-12T05:55:26.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.106+0000 7f8c3e7fc640 1 -- 172.21.15.76:0/2780241754 learned_addr learned my addr 172.21.15.76:0/2780241754 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:26.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.107+0000 7f8c3e7fc640 1 -- 172.21.15.76:0/2780241754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c401000a0 msgr2=0x7f8c4006a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.107+0000 7f8c3f7fe640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c40101c50 0x7f8c401a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:26.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.107+0000 7f8c3effd640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c401000a0 0x7f8c4006a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:26.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.107+0000 7f8c3e7fc640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c401000a0 0x7f8c4006a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.107+0000 7f8c3e7fc640 1 -- 172.21.15.76:0/2780241754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c40101c50 msgr2=0x7f8c401a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.107+0000 7f8c3e7fc640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c40101c50 0x7f8c401a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.107+0000 7f8c3e7fc640 1 -- 172.21.15.76:0/2780241754 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c28009560 con 0x7f8c40101290 2024-03-12T05:55:26.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.107+0000 7f8c3effd640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c401000a0 0x7f8c4006a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:26.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.107+0000 7f8c3e7fc640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c40101290 0x7f8c4019e330 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8c28005ab0 tx=0x7f8c280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:26.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.107+0000 7f8c37fff640 1 -- 172.21.15.76:0/2780241754 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c280044e0 con 0x7f8c40101290 2024-03-12T05:55:26.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.107+0000 7f8c37fff640 1 -- 172.21.15.76:0/2780241754 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c280054e0 con 0x7f8c40101290 2024-03-12T05:55:26.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.107+0000 7f8c37fff640 1 -- 172.21.15.76:0/2780241754 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2803e450 con 0x7f8c40101290 2024-03-12T05:55:26.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.107+0000 7f8c4553a640 1 -- 172.21.15.76:0/2780241754 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c401a1490 con 0x7f8c40101290 2024-03-12T05:55:26.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.107+0000 7f8c4553a640 1 -- 172.21.15.76:0/2780241754 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c401a19a0 con 0x7f8c40101290 2024-03-12T05:55:26.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.108+0000 7f8c4553a640 1 -- 172.21.15.76:0/2780241754 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c401072f0 con 0x7f8c40101290 2024-03-12T05:55:26.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.109+0000 7f8c37fff640 1 -- 172.21.15.76:0/2780241754 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f8c280035f0 con 0x7f8c40101290 2024-03-12T05:55:26.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.109+0000 7f8c37fff640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8c2c0a1a60 0x7f8c2c0a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:26.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.110+0000 7f8c3effd640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8c2c0a1a60 0x7f8c2c0a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:26.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.110+0000 7f8c37fff640 1 -- 172.21.15.76:0/2780241754 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8c280ef9d0 con 0x7f8c40101290 2024-03-12T05:55:26.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.110+0000 7f8c3effd640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8c2c0a1a60 0x7f8c2c0a3f20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8c401010f0 tx=0x7f8c38005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:26.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.114+0000 7f8c37fff640 1 -- 172.21.15.76:0/2780241754 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8c280b8bf0 con 0x7f8c40101290 2024-03-12T05:55:26.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.287+0000 7f8c4553a640 1 -- 172.21.15.76:0/2780241754 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8c4002ce40 con 0x7f8c40101290 2024-03-12T05:55:26.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.287+0000 7f8c37fff640 1 -- 172.21.15.76:0/2780241754 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8c280bc2b0 con 0x7f8c40101290 2024-03-12T05:55:26.291 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-12T05:55:26.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.289+0000 7f8c4553a640 1 -- 172.21.15.76:0/2780241754 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8c2c0a1a60 msgr2=0x7f8c2c0a3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.289+0000 7f8c4553a640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8c2c0a1a60 0x7f8c2c0a3f20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8c401010f0 tx=0x7f8c38005d30 comp rx=0 tx=0).stop 2024-03-12T05:55:26.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.289+0000 7f8c4553a640 1 -- 172.21.15.76:0/2780241754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c40101290 msgr2=0x7f8c4019e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.289+0000 7f8c4553a640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c40101290 0x7f8c4019e330 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8c28005ab0 tx=0x7f8c280042f0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.289+0000 7f8c4553a640 1 -- 172.21.15.76:0/2780241754 shutdown_connections 2024-03-12T05:55:26.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.289+0000 7f8c4553a640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f8c2c0a1a60 0x7f8c2c0a3f20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.289+0000 7f8c4553a640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8c40101c50 0x7f8c401a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.289+0000 7f8c4553a640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c40101290 0x7f8c4019e330 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.289+0000 7f8c4553a640 1 --2- 172.21.15.76:0/2780241754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c401000a0 0x7f8c4006a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.289+0000 7f8c4553a640 1 -- 172.21.15.76:0/2780241754 >> 172.21.15.76:0/2780241754 conn(0x7f8c400fb630 msgr2=0x7f8c401044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:26.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.289+0000 7f8c4553a640 1 -- 172.21.15.76:0/2780241754 shutdown_connections 2024-03-12T05:55:26.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.289+0000 7f8c4553a640 1 -- 172.21.15.76:0/2780241754 wait complete. 2024-03-12T05:55:26.303 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:25 smithi076 ceph-mon[32095]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-12T05:55:26.303 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:25 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/183728147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:26.304 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:25 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/548206906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:26.304 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:25 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/621035395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:26.304 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:25 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2952781629' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:26.305 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:55:26.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 -- 172.21.15.76:0/1058996002 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64c8106ee0 msgr2=0x7f64c81092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 --2- 172.21.15.76:0/1058996002 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64c8106ee0 0x7f64c81092d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f64c0009ca0 tx=0x7f64c002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 -- 172.21.15.76:0/1058996002 shutdown_connections 2024-03-12T05:55:26.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 --2- 172.21.15.76:0/1058996002 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64c8106ee0 0x7f64c81092d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 --2- 172.21.15.76:0/1058996002 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64c81045b0 0x7f64c81069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 --2- 172.21.15.76:0/1058996002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64c8101c80 0x7f64c8104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 -- 172.21.15.76:0/1058996002 >> 172.21.15.76:0/1058996002 conn(0x7f64c80fb610 msgr2=0x7f64c80fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:26.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 -- 172.21.15.76:0/1058996002 shutdown_connections 2024-03-12T05:55:26.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 -- 172.21.15.76:0/1058996002 wait complete. 2024-03-12T05:55:26.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 Processor -- start 2024-03-12T05:55:26.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 -- start start 2024-03-12T05:55:26.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64c8101c80 0x7f64c8197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:26.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64c81045b0 0x7f64c8197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:26.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64c8106ee0 0x7f64c819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:26.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c819ecf0 con 0x7f64c8101c80 2024-03-12T05:55:26.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c819ee60 con 0x7f64c8106ee0 2024-03-12T05:55:26.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.406+0000 7f64cd95c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c819f020 con 0x7f64c81045b0 2024-03-12T05:55:26.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64c6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64c8101c80 0x7f64c8197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:26.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64c6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64c8101c80 0x7f64c8197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55372/0 (socket says 172.21.15.76:55372) 2024-03-12T05:55:26.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64c6ffd640 1 -- 172.21.15.76:0/2110554471 learned_addr learned my addr 172.21.15.76:0/2110554471 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:26.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64c77fe640 1 --2- 172.21.15.76:0/2110554471 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64c8106ee0 0x7f64c819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:26.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64c6ffd640 1 -- 172.21.15.76:0/2110554471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64c81045b0 msgr2=0x7f64c8197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64c6ffd640 1 --2- 172.21.15.76:0/2110554471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64c81045b0 0x7f64c8197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64c6ffd640 1 -- 172.21.15.76:0/2110554471 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64c8106ee0 msgr2=0x7f64c819e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64c6ffd640 1 --2- 172.21.15.76:0/2110554471 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64c8106ee0 0x7f64c819e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64c6ffd640 1 -- 172.21.15.76:0/2110554471 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c0009900 con 0x7f64c8101c80 2024-03-12T05:55:26.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64c6ffd640 1 --2- 172.21.15.76:0/2110554471 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64c8101c80 0x7f64c8197200 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f64b800b4a0 tx=0x7f64b800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:26.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64cc95a640 1 -- 172.21.15.76:0/2110554471 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b8010610 con 0x7f64c8101c80 2024-03-12T05:55:26.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64cc95a640 1 -- 172.21.15.76:0/2110554471 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64b8010c50 con 0x7f64c8101c80 2024-03-12T05:55:26.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64cd95c640 1 -- 172.21.15.76:0/2110554471 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c819f300 con 0x7f64c8101c80 2024-03-12T05:55:26.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64cc95a640 1 -- 172.21.15.76:0/2110554471 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b8017e20 con 0x7f64c8101c80 2024-03-12T05:55:26.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.407+0000 7f64cd95c640 1 -- 172.21.15.76:0/2110554471 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c819f810 con 0x7f64c8101c80 2024-03-12T05:55:26.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.408+0000 7f64cd95c640 1 -- 172.21.15.76:0/2110554471 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c80fd100 con 0x7f64c8101c80 2024-03-12T05:55:26.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.409+0000 7f64cc95a640 1 -- 172.21.15.76:0/2110554471 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f64b8017480 con 0x7f64c8101c80 2024-03-12T05:55:26.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.410+0000 7f64cc95a640 1 --2- 172.21.15.76:0/2110554471 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f64b40a1ab0 0x7f64b40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:26.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.410+0000 7f64bffff640 1 --2- 172.21.15.76:0/2110554471 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f64b40a1ab0 0x7f64b40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:26.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.410+0000 7f64cc95a640 1 -- 172.21.15.76:0/2110554471 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f64b80ccee0 con 0x7f64c8101c80 2024-03-12T05:55:26.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.410+0000 7f64bffff640 1 --2- 172.21.15.76:0/2110554471 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f64b40a1ab0 0x7f64b40a3f70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f64c8198710 tx=0x7f64b0005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:26.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.413+0000 7f64cc95a640 1 -- 172.21.15.76:0/2110554471 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f64b8096100 con 0x7f64c8101c80 2024-03-12T05:55:26.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:25 smithi082 ceph-mon[36382]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-12T05:55:26.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:25 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/183728147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:26.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:25 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/548206906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:26.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:25 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/621035395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:26.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:25 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2952781629' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:25 smithi088 ceph-mon[36585]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-12T05:55:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:25 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/183728147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:25 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/548206906' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:25 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/621035395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:25 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2952781629' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:26.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.585+0000 7f64cd95c640 1 -- 172.21.15.76:0/2110554471 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f64c802ce40 con 0x7f64c8101c80 2024-03-12T05:55:26.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.586+0000 7f64cc95a640 1 -- 172.21.15.76:0/2110554471 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f64b80092b0 con 0x7f64c8101c80 2024-03-12T05:55:26.589 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-12T05:55:26.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.588+0000 7f64cd95c640 1 -- 172.21.15.76:0/2110554471 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f64b40a1ab0 msgr2=0x7f64b40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.588+0000 7f64cd95c640 1 --2- 172.21.15.76:0/2110554471 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f64b40a1ab0 0x7f64b40a3f70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f64c8198710 tx=0x7f64b0005c70 comp rx=0 tx=0).stop 2024-03-12T05:55:26.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.588+0000 7f64cd95c640 1 -- 172.21.15.76:0/2110554471 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64c8101c80 msgr2=0x7f64c8197200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.588+0000 7f64cd95c640 1 --2- 172.21.15.76:0/2110554471 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64c8101c80 0x7f64c8197200 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f64b800b4a0 tx=0x7f64b800b960 comp rx=0 tx=0).stop 2024-03-12T05:55:26.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.588+0000 7f64cd95c640 1 -- 172.21.15.76:0/2110554471 shutdown_connections 2024-03-12T05:55:26.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.588+0000 7f64cd95c640 1 --2- 172.21.15.76:0/2110554471 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f64b40a1ab0 0x7f64b40a3f70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.588+0000 7f64cd95c640 1 --2- 172.21.15.76:0/2110554471 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f64c8106ee0 0x7f64c819e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.588+0000 7f64cd95c640 1 --2- 172.21.15.76:0/2110554471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64c81045b0 0x7f64c8197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.588+0000 7f64cd95c640 1 --2- 172.21.15.76:0/2110554471 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f64c8101c80 0x7f64c8197200 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.588+0000 7f64cd95c640 1 -- 172.21.15.76:0/2110554471 >> 172.21.15.76:0/2110554471 conn(0x7f64c80fb610 msgr2=0x7f64c81025c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:26.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.588+0000 7f64cd95c640 1 -- 172.21.15.76:0/2110554471 shutdown_connections 2024-03-12T05:55:26.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.588+0000 7f64cd95c640 1 -- 172.21.15.76:0/2110554471 wait complete. 2024-03-12T05:55:26.603 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:55:26.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.703+0000 7f21c70f4640 1 -- 172.21.15.76:0/672569528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c0101c50 msgr2=0x7f21c01084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.703+0000 7f21c70f4640 1 --2- 172.21.15.76:0/672569528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c0101c50 0x7f21c01084e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f21b8009ca0 tx=0x7f21b802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.703+0000 7f21c70f4640 1 -- 172.21.15.76:0/672569528 shutdown_connections 2024-03-12T05:55:26.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.703+0000 7f21c70f4640 1 --2- 172.21.15.76:0/672569528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c0101c50 0x7f21c01084e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.703+0000 7f21c70f4640 1 --2- 172.21.15.76:0/672569528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21c0101290 0x7f21c0101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.703+0000 7f21c70f4640 1 --2- 172.21.15.76:0/672569528 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21c01000a0 0x7f21c01004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.703+0000 7f21c70f4640 1 -- 172.21.15.76:0/672569528 >> 172.21.15.76:0/672569528 conn(0x7f21c00fb610 msgr2=0x7f21c00fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.703+0000 7f21c70f4640 1 -- 172.21.15.76:0/672569528 shutdown_connections 2024-03-12T05:55:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.703+0000 7f21c70f4640 1 -- 172.21.15.76:0/672569528 wait complete. 2024-03-12T05:55:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.703+0000 7f21c70f4640 1 Processor -- start 2024-03-12T05:55:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21c70f4640 1 -- start start 2024-03-12T05:55:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21c70f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21c01000a0 0x7f21c006a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21c70f4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c0101290 0x7f21c019bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:26.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21c70f4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21c0101c50 0x7f21c019e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21c70f4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c019ed00 con 0x7f21c0101290 2024-03-12T05:55:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21c70f4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c019ee70 con 0x7f21c0101c50 2024-03-12T05:55:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21c70f4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c019f030 con 0x7f21c01000a0 2024-03-12T05:55:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21bffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c0101290 0x7f21c019bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21bffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c0101290 0x7f21c019bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55388/0 (socket says 172.21.15.76:55388) 2024-03-12T05:55:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21bffff640 1 -- 172.21.15.76:0/3728741159 learned_addr learned my addr 172.21.15.76:0/3728741159 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21c5691640 1 --2- 172.21.15.76:0/3728741159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21c0101c50 0x7f21c019e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21c4e90640 1 --2- 172.21.15.76:0/3728741159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21c01000a0 0x7f21c006a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21bffff640 1 -- 172.21.15.76:0/3728741159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21c01000a0 msgr2=0x7f21c006a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21bffff640 1 --2- 172.21.15.76:0/3728741159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21c01000a0 0x7f21c006a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21bffff640 1 -- 172.21.15.76:0/3728741159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21c0101c50 msgr2=0x7f21c019e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21bffff640 1 --2- 172.21.15.76:0/3728741159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21c0101c50 0x7f21c019e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21bffff640 1 -- 172.21.15.76:0/3728741159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b8009900 con 0x7f21c0101290 2024-03-12T05:55:26.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21bffff640 1 --2- 172.21.15.76:0/3728741159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c0101290 0x7f21c019bf30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00e9f0 tx=0x7f21ac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21bdffb640 1 -- 172.21.15.76:0/3728741159 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac009720 con 0x7f21c0101290 2024-03-12T05:55:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21bdffb640 1 -- 172.21.15.76:0/3728741159 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21ac00cd80 con 0x7f21c0101290 2024-03-12T05:55:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21c70f4640 1 -- 172.21.15.76:0/3728741159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c019f310 con 0x7f21c0101290 2024-03-12T05:55:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.704+0000 7f21bdffb640 1 -- 172.21.15.76:0/3728741159 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac010690 con 0x7f21c0101290 2024-03-12T05:55:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.705+0000 7f21c70f4640 1 -- 172.21.15.76:0/3728741159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c019f7d0 con 0x7f21c0101290 2024-03-12T05:55:26.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.706+0000 7f21c70f4640 1 -- 172.21.15.76:0/3728741159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c01073c0 con 0x7f21c0101290 2024-03-12T05:55:26.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.706+0000 7f21bdffb640 1 -- 172.21.15.76:0/3728741159 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f21ac0046c0 con 0x7f21c0101290 2024-03-12T05:55:26.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.707+0000 7f21bdffb640 1 --2- 172.21.15.76:0/3728741159 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21b00a1a30 0x7f21b00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:26.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.707+0000 7f21c4e90640 1 --2- 172.21.15.76:0/3728741159 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21b00a1a30 0x7f21b00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:26.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.707+0000 7f21bdffb640 1 -- 172.21.15.76:0/3728741159 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f21ac014070 con 0x7f21c0101290 2024-03-12T05:55:26.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.707+0000 7f21c4e90640 1 --2- 172.21.15.76:0/3728741159 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21b00a1a30 0x7f21b00a3ef0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f21b4005fd0 tx=0x7f21b4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:26.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.711+0000 7f21bdffb640 1 -- 172.21.15.76:0/3728741159 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f21ac096a30 con 0x7f21c0101290 2024-03-12T05:55:26.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.883+0000 7f21c70f4640 1 -- 172.21.15.76:0/3728741159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f21c002cdd0 con 0x7f21c0101290 2024-03-12T05:55:26.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.884+0000 7f21bdffb640 1 -- 172.21.15.76:0/3728741159 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f21ac09a0f0 con 0x7f21c0101290 2024-03-12T05:55:26.888 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-12T05:55:26.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.886+0000 7f21c70f4640 1 -- 172.21.15.76:0/3728741159 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21b00a1a30 msgr2=0x7f21b00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.886+0000 7f21c70f4640 1 --2- 172.21.15.76:0/3728741159 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21b00a1a30 0x7f21b00a3ef0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f21b4005fd0 tx=0x7f21b4005850 comp rx=0 tx=0).stop 2024-03-12T05:55:26.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.886+0000 7f21c70f4640 1 -- 172.21.15.76:0/3728741159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c0101290 msgr2=0x7f21c019bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:26.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.886+0000 7f21c70f4640 1 --2- 172.21.15.76:0/3728741159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c0101290 0x7f21c019bf30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00e9f0 tx=0x7f21ac00edf0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.886+0000 7f21c70f4640 1 -- 172.21.15.76:0/3728741159 shutdown_connections 2024-03-12T05:55:26.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.886+0000 7f21c70f4640 1 --2- 172.21.15.76:0/3728741159 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21b00a1a30 0x7f21b00a3ef0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.886+0000 7f21c70f4640 1 --2- 172.21.15.76:0/3728741159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f21c0101c50 0x7f21c019e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.886+0000 7f21c70f4640 1 --2- 172.21.15.76:0/3728741159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f21c0101290 0x7f21c019bf30 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.886+0000 7f21c70f4640 1 --2- 172.21.15.76:0/3728741159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21c01000a0 0x7f21c006a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:26.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.886+0000 7f21c70f4640 1 -- 172.21.15.76:0/3728741159 >> 172.21.15.76:0/3728741159 conn(0x7f21c00fb610 msgr2=0x7f21c0104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:26.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.886+0000 7f21c70f4640 1 -- 172.21.15.76:0/3728741159 shutdown_connections 2024-03-12T05:55:26.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:26.886+0000 7f21c70f4640 1 -- 172.21.15.76:0/3728741159 wait complete. 2024-03-12T05:55:26.901 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:55:27.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.001+0000 7f49bade2640 1 -- 172.21.15.76:0/664251169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49b4101c60 msgr2=0x7f49b41084f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.001+0000 7f49bade2640 1 --2- 172.21.15.76:0/664251169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49b4101c60 0x7f49b41084f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f49ac009ca0 tx=0x7f49ac02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 -- 172.21.15.76:0/664251169 shutdown_connections 2024-03-12T05:55:27.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 --2- 172.21.15.76:0/664251169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49b4101c60 0x7f49b41084f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 --2- 172.21.15.76:0/664251169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49b41012a0 0x7f49b4101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 --2- 172.21.15.76:0/664251169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49b41000b0 0x7f49b41004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 -- 172.21.15.76:0/664251169 >> 172.21.15.76:0/664251169 conn(0x7f49b40fb620 msgr2=0x7f49b40fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 -- 172.21.15.76:0/664251169 shutdown_connections 2024-03-12T05:55:27.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 -- 172.21.15.76:0/664251169 wait complete. 2024-03-12T05:55:27.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 Processor -- start 2024-03-12T05:55:27.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 -- start start 2024-03-12T05:55:27.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49b41000b0 0x7f49b41971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49b41012a0 0x7f49b41976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49b4101c60 0x7f49b419e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b419eca0 con 0x7f49b4101c60 2024-03-12T05:55:27.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b419ee10 con 0x7f49b41000b0 2024-03-12T05:55:27.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49bade2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b419efd0 con 0x7f49b41012a0 2024-03-12T05:55:27.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49b937f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49b4101c60 0x7f49b419e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49b937f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49b4101c60 0x7f49b419e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55392/0 (socket says 172.21.15.76:55392) 2024-03-12T05:55:27.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.002+0000 7f49b937f640 1 -- 172.21.15.76:0/3419003900 learned_addr learned my addr 172.21.15.76:0/3419003900 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.003+0000 7f49b8b7e640 1 --2- 172.21.15.76:0/3419003900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49b41000b0 0x7f49b41971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.003+0000 7f49b3fff640 1 --2- 172.21.15.76:0/3419003900 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49b41012a0 0x7f49b41976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.003+0000 7f49b937f640 1 -- 172.21.15.76:0/3419003900 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49b41012a0 msgr2=0x7f49b41976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.003+0000 7f49b937f640 1 --2- 172.21.15.76:0/3419003900 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49b41012a0 0x7f49b41976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.003+0000 7f49b937f640 1 -- 172.21.15.76:0/3419003900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49b41000b0 msgr2=0x7f49b41971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.003+0000 7f49b937f640 1 --2- 172.21.15.76:0/3419003900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49b41000b0 0x7f49b41971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.003+0000 7f49b937f640 1 -- 172.21.15.76:0/3419003900 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac009900 con 0x7f49b4101c60 2024-03-12T05:55:27.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.003+0000 7f49b937f640 1 --2- 172.21.15.76:0/3419003900 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49b4101c60 0x7f49b419e760 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f49ac02c630 tx=0x7f49ac034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.003+0000 7f49b1ffb640 1 -- 172.21.15.76:0/3419003900 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ac032eb0 con 0x7f49b4101c60 2024-03-12T05:55:27.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.003+0000 7f49bade2640 1 -- 172.21.15.76:0/3419003900 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b419f250 con 0x7f49b4101c60 2024-03-12T05:55:27.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.003+0000 7f49b1ffb640 1 -- 172.21.15.76:0/3419003900 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49ac003e50 con 0x7f49b4101c60 2024-03-12T05:55:27.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.003+0000 7f49b1ffb640 1 -- 172.21.15.76:0/3419003900 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ac040450 con 0x7f49b4101c60 2024-03-12T05:55:27.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.003+0000 7f49bade2640 1 -- 172.21.15.76:0/3419003900 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b419f750 con 0x7f49b4101c60 2024-03-12T05:55:27.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.004+0000 7f49bade2640 1 -- 172.21.15.76:0/3419003900 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b41073d0 con 0x7f49b4101c60 2024-03-12T05:55:27.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.005+0000 7f49b1ffb640 1 -- 172.21.15.76:0/3419003900 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f49ac00a040 con 0x7f49b4101c60 2024-03-12T05:55:27.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.006+0000 7f49b1ffb640 1 --2- 172.21.15.76:0/3419003900 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f49a40a1a30 0x7f49a40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.006+0000 7f49b8b7e640 1 --2- 172.21.15.76:0/3419003900 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f49a40a1a30 0x7f49a40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.006+0000 7f49b1ffb640 1 -- 172.21.15.76:0/3419003900 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f49ac0f06b0 con 0x7f49b4101c60 2024-03-12T05:55:27.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.006+0000 7f49b8b7e640 1 --2- 172.21.15.76:0/3419003900 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f49a40a1a30 0x7f49a40a3ef0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f49b4101100 tx=0x7f49a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.009+0000 7f49b1ffb640 1 -- 172.21.15.76:0/3419003900 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f49ac0b98d0 con 0x7f49b4101c60 2024-03-12T05:55:27.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.217+0000 7f49bade2640 1 -- 172.21.15.76:0/3419003900 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49b402cdd0 con 0x7f49b4101c60 2024-03-12T05:55:27.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.218+0000 7f49b1ffb640 1 -- 172.21.15.76:0/3419003900 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+11328 (secure 0 0 0) 0x7f49ac0bcf90 con 0x7f49b4101c60 2024-03-12T05:55:27.222 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:27.222 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:53:42.384114+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:55:27.222 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 21 2024-03-12T05:55:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.220+0000 7f49bade2640 1 -- 172.21.15.76:0/3419003900 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f49a40a1a30 msgr2=0x7f49a40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.220+0000 7f49bade2640 1 --2- 172.21.15.76:0/3419003900 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f49a40a1a30 0x7f49a40a3ef0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f49b4101100 tx=0x7f49a8005f90 comp rx=0 tx=0).stop 2024-03-12T05:55:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.220+0000 7f49bade2640 1 -- 172.21.15.76:0/3419003900 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49b4101c60 msgr2=0x7f49b419e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.220+0000 7f49bade2640 1 --2- 172.21.15.76:0/3419003900 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49b4101c60 0x7f49b419e760 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f49ac02c630 tx=0x7f49ac034cc0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.220+0000 7f49bade2640 1 -- 172.21.15.76:0/3419003900 shutdown_connections 2024-03-12T05:55:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.220+0000 7f49bade2640 1 --2- 172.21.15.76:0/3419003900 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f49a40a1a30 0x7f49a40a3ef0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.220+0000 7f49bade2640 1 --2- 172.21.15.76:0/3419003900 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49b4101c60 0x7f49b419e760 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.220+0000 7f49bade2640 1 --2- 172.21.15.76:0/3419003900 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49b41012a0 0x7f49b41976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.220+0000 7f49bade2640 1 --2- 172.21.15.76:0/3419003900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49b41000b0 0x7f49b41971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.220+0000 7f49bade2640 1 -- 172.21.15.76:0/3419003900 >> 172.21.15.76:0/3419003900 conn(0x7f49b40fb620 msgr2=0x7f49b41045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.220+0000 7f49bade2640 1 -- 172.21.15.76:0/3419003900 shutdown_connections 2024-03-12T05:55:27.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.220+0000 7f49bade2640 1 -- 172.21.15.76:0/3419003900 wait complete. 2024-03-12T05:55:27.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:26 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2780241754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:27.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:26 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2110554471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:27.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:26 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3728741159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:27.235 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:55:27.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.332+0000 7f42a7b3a640 1 -- 172.21.15.76:0/427170614 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a0102810 msgr2=0x7f42a0102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.332+0000 7f42a7b3a640 1 --2- 172.21.15.76:0/427170614 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a0102810 0x7f42a0102c30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f429c009900 tx=0x7f429c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:27.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.332+0000 7f42a7b3a640 1 -- 172.21.15.76:0/427170614 shutdown_connections 2024-03-12T05:55:27.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.332+0000 7f42a7b3a640 1 --2- 172.21.15.76:0/427170614 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42a00ffc40 0x7f42a01097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.332+0000 7f42a7b3a640 1 --2- 172.21.15.76:0/427170614 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42a00ff280 0x7f42a00ff700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.332+0000 7f42a7b3a640 1 --2- 172.21.15.76:0/427170614 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a0102810 0x7f42a0102c30 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.332+0000 7f42a7b3a640 1 -- 172.21.15.76:0/427170614 >> 172.21.15.76:0/427170614 conn(0x7f42a00fae00 msgr2=0x7f42a00fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.332+0000 7f42a7b3a640 1 -- 172.21.15.76:0/427170614 shutdown_connections 2024-03-12T05:55:27.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.332+0000 7f42a7b3a640 1 -- 172.21.15.76:0/427170614 wait complete. 2024-03-12T05:55:27.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a7b3a640 1 Processor -- start 2024-03-12T05:55:27.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a7b3a640 1 -- start start 2024-03-12T05:55:27.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a7b3a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42a00ff280 0x7f42a0197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a7b3a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a00ffc40 0x7f42a01977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a7b3a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42a0102810 0x7f42a019e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a7b3a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a019ed70 con 0x7f42a00ffc40 2024-03-12T05:55:27.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a7b3a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a019eee0 con 0x7f42a00ff280 2024-03-12T05:55:27.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a7b3a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a019f0a0 con 0x7f42a0102810 2024-03-12T05:55:27.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a50d5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a00ffc40 0x7f42a01977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a50d5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a00ffc40 0x7f42a01977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55412/0 (socket says 172.21.15.76:55412) 2024-03-12T05:55:27.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a50d5640 1 -- 172.21.15.76:0/1911449943 learned_addr learned my addr 172.21.15.76:0/1911449943 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a58d6640 1 --2- 172.21.15.76:0/1911449943 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42a00ff280 0x7f42a0197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a60d7640 1 --2- 172.21.15.76:0/1911449943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42a0102810 0x7f42a019e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a50d5640 1 -- 172.21.15.76:0/1911449943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42a0102810 msgr2=0x7f42a019e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a50d5640 1 --2- 172.21.15.76:0/1911449943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42a0102810 0x7f42a019e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a50d5640 1 -- 172.21.15.76:0/1911449943 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42a00ff280 msgr2=0x7f42a0197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a50d5640 1 --2- 172.21.15.76:0/1911449943 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42a00ff280 0x7f42a0197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a50d5640 1 -- 172.21.15.76:0/1911449943 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c009560 con 0x7f42a00ffc40 2024-03-12T05:55:27.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a50d5640 1 --2- 172.21.15.76:0/1911449943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a00ffc40 0x7f42a01977c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f429000d950 tx=0x7f429000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f4296ffd640 1 -- 172.21.15.76:0/1911449943 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4290009c30 con 0x7f42a00ffc40 2024-03-12T05:55:27.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f4296ffd640 1 -- 172.21.15.76:0/1911449943 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4290009d90 con 0x7f42a00ffc40 2024-03-12T05:55:27.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f42a7b3a640 1 -- 172.21.15.76:0/1911449943 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a019f380 con 0x7f42a00ffc40 2024-03-12T05:55:27.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.333+0000 7f4296ffd640 1 -- 172.21.15.76:0/1911449943 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f429000f7e0 con 0x7f42a00ffc40 2024-03-12T05:55:27.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.334+0000 7f42a7b3a640 1 -- 172.21.15.76:0/1911449943 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a019f890 con 0x7f42a00ffc40 2024-03-12T05:55:27.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.335+0000 7f42a7b3a640 1 -- 172.21.15.76:0/1911449943 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a0108750 con 0x7f42a00ffc40 2024-03-12T05:55:27.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.336+0000 7f4296ffd640 1 -- 172.21.15.76:0/1911449943 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4290010ad0 con 0x7f42a00ffc40 2024-03-12T05:55:27.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.336+0000 7f4296ffd640 1 --2- 172.21.15.76:0/1911449943 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f428c0a1a30 0x7f428c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.336+0000 7f42a58d6640 1 --2- 172.21.15.76:0/1911449943 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f428c0a1a30 0x7f428c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.337+0000 7f4296ffd640 1 -- 172.21.15.76:0/1911449943 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f42900cd130 con 0x7f42a00ffc40 2024-03-12T05:55:27.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.337+0000 7f42a58d6640 1 --2- 172.21.15.76:0/1911449943 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f428c0a1a30 0x7f428c0a3ef0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f429c006010 tx=0x7f429c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.341+0000 7f4296ffd640 1 -- 172.21.15.76:0/1911449943 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4290096350 con 0x7f42a00ffc40 2024-03-12T05:55:27.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:26 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2780241754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:27.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:26 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2110554471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:27.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:26 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3728741159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:26 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2780241754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:26 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2110554471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:26 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3728741159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:27.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.516+0000 7f42a7b3a640 1 -- 172.21.15.76:0/1911449943 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42a002ce40 con 0x7f42a00ffc40 2024-03-12T05:55:27.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.517+0000 7f4296ffd640 1 -- 172.21.15.76:0/1911449943 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22541 (secure 0 0 0) 0x7f4290099a10 con 0x7f42a00ffc40 2024-03-12T05:55:27.521 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:27.521 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:54:42.007701+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:55:27.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.519+0000 7f42a7b3a640 1 -- 172.21.15.76:0/1911449943 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f428c0a1a30 msgr2=0x7f428c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.519+0000 7f42a7b3a640 1 --2- 172.21.15.76:0/1911449943 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f428c0a1a30 0x7f428c0a3ef0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f429c006010 tx=0x7f429c000bc0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.519+0000 7f42a7b3a640 1 -- 172.21.15.76:0/1911449943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a00ffc40 msgr2=0x7f42a01977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.519+0000 7f42a7b3a640 1 --2- 172.21.15.76:0/1911449943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a00ffc40 0x7f42a01977c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f429000d950 tx=0x7f429000de10 comp rx=0 tx=0).stop 2024-03-12T05:55:27.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.519+0000 7f42a7b3a640 1 -- 172.21.15.76:0/1911449943 shutdown_connections 2024-03-12T05:55:27.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.519+0000 7f42a7b3a640 1 --2- 172.21.15.76:0/1911449943 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f428c0a1a30 0x7f428c0a3ef0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.519+0000 7f42a7b3a640 1 --2- 172.21.15.76:0/1911449943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42a0102810 0x7f42a019e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.519+0000 7f42a7b3a640 1 --2- 172.21.15.76:0/1911449943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42a00ffc40 0x7f42a01977c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.519+0000 7f42a7b3a640 1 --2- 172.21.15.76:0/1911449943 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42a00ff280 0x7f42a0197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.519+0000 7f42a7b3a640 1 -- 172.21.15.76:0/1911449943 >> 172.21.15.76:0/1911449943 conn(0x7f42a00fae00 msgr2=0x7f42a0105920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.519+0000 7f42a7b3a640 1 -- 172.21.15.76:0/1911449943 shutdown_connections 2024-03-12T05:55:27.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.519+0000 7f42a7b3a640 1 -- 172.21.15.76:0/1911449943 wait complete. 2024-03-12T05:55:27.535 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-12T05:53:37.535927+0000', 'modified': '2024-03-12T05:53:42.384114+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24518, 'mds_1': 24481, 'mds_2': 24451}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24449': {'gid': 24449, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.82:6835/2138041661', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6834', 'nonce': 2138041661}, {'type': 'v1', 'addr': '172.21.15.82:6835', 'nonce': 2138041661}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.88:6835/3940245802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 3940245802}, {'type': 'v1', 'addr': '172.21.15.88:6835', 'nonce': 3940245802}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.88:6837/1311843051', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6836', 'nonce': 1311843051}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 1311843051}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24479': {'gid': 24479, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.82:6837/2523422510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6836', 'nonce': 2523422510}, {'type': 'v1', 'addr': '172.21.15.82:6837', 'nonce': 2523422510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24481': {'gid': 24481, 'name': 'i', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.88:6839/1164228359', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6838', 'nonce': 1164228359}, {'type': 'v1', 'addr': '172.21.15.88:6839', 'nonce': 1164228359}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24518': {'gid': 24518, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.82:6839/2242595718', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6838', 'nonce': 2242595718}, {'type': 'v1', 'addr': '172.21.15.82:6839', 'nonce': 2242595718}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-12T05:55:27.535 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-12T05:55:27.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-12T05:55:27.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.633+0000 7f9298240640 1 -- 172.21.15.76:0/3554824102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9290100090 msgr2=0x7f92901004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.633+0000 7f9298240640 1 --2- 172.21.15.76:0/3554824102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9290100090 0x7f92901004b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9288009ca0 tx=0x7f928802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.633+0000 7f9298240640 1 -- 172.21.15.76:0/3554824102 shutdown_connections 2024-03-12T05:55:27.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.633+0000 7f9298240640 1 --2- 172.21.15.76:0/3554824102 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9290101c40 0x7f92901084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.633+0000 7f9298240640 1 --2- 172.21.15.76:0/3554824102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9290101280 0x7f9290101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.633+0000 7f9298240640 1 --2- 172.21.15.76:0/3554824102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9290100090 0x7f92901004b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.633+0000 7f9298240640 1 -- 172.21.15.76:0/3554824102 >> 172.21.15.76:0/3554824102 conn(0x7f92900fb5e0 msgr2=0x7f92900fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.634+0000 7f9298240640 1 -- 172.21.15.76:0/3554824102 shutdown_connections 2024-03-12T05:55:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.634+0000 7f9298240640 1 -- 172.21.15.76:0/3554824102 wait complete. 2024-03-12T05:55:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.634+0000 7f9298240640 1 Processor -- start 2024-03-12T05:55:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.634+0000 7f9298240640 1 -- start start 2024-03-12T05:55:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.634+0000 7f9298240640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9290100090 0x7f9290196160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.634+0000 7f9298240640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9290101280 0x7f92901966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.634+0000 7f9298240640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9290101c40 0x7f929019d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.634+0000 7f9298240640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929019dc50 con 0x7f9290101280 2024-03-12T05:55:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.634+0000 7f9298240640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929019ddc0 con 0x7f9290100090 2024-03-12T05:55:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.634+0000 7f9298240640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929019df80 con 0x7f9290101c40 2024-03-12T05:55:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.634+0000 7f92957db640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9290101280 0x7f92901966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.634+0000 7f92957db640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9290101280 0x7f92901966a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55444/0 (socket says 172.21.15.76:55444) 2024-03-12T05:55:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.634+0000 7f92957db640 1 -- 172.21.15.76:0/1394022706 learned_addr learned my addr 172.21.15.76:0/1394022706 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.634+0000 7f9295fdc640 1 --2- 172.21.15.76:0/1394022706 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9290100090 0x7f9290196160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.635+0000 7f92957db640 1 -- 172.21.15.76:0/1394022706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9290101c40 msgr2=0x7f929019d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.635+0000 7f92957db640 1 --2- 172.21.15.76:0/1394022706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9290101c40 0x7f929019d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.635+0000 7f92957db640 1 -- 172.21.15.76:0/1394022706 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9290100090 msgr2=0x7f9290196160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.635+0000 7f92957db640 1 --2- 172.21.15.76:0/1394022706 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9290100090 0x7f9290196160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.635+0000 7f92957db640 1 -- 172.21.15.76:0/1394022706 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9288009900 con 0x7f9290101280 2024-03-12T05:55:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.635+0000 7f92957db640 1 --2- 172.21.15.76:0/1394022706 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9290101280 0x7f92901966a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f928000c970 tx=0x7f928000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.635+0000 7f9286ffd640 1 -- 172.21.15.76:0/1394022706 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9280007bb0 con 0x7f9290101280 2024-03-12T05:55:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.635+0000 7f9286ffd640 1 -- 172.21.15.76:0/1394022706 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9280007d10 con 0x7f9290101280 2024-03-12T05:55:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.635+0000 7f9298240640 1 -- 172.21.15.76:0/1394022706 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929019e260 con 0x7f9290101280 2024-03-12T05:55:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.635+0000 7f9286ffd640 1 -- 172.21.15.76:0/1394022706 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928000f450 con 0x7f9290101280 2024-03-12T05:55:27.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.635+0000 7f9298240640 1 -- 172.21.15.76:0/1394022706 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929019e770 con 0x7f9290101280 2024-03-12T05:55:27.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.637+0000 7f9286ffd640 1 -- 172.21.15.76:0/1394022706 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f9280004f70 con 0x7f9290101280 2024-03-12T05:55:27.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f929ae10740 1 Processor -- start 2024-03-12T05:55:27.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f929ae10740 1 -- start start 2024-03-12T05:55:27.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f929ae10740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4190 0x5624a4fc45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f929ae10740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4b80 0x5624a4fc9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f929ae10740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 0x5624a4fcc180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f929ae10740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a4f7dcd0 con 0x5624a4fc4190 2024-03-12T05:55:27.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f929ae10740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a4f30f20 con 0x5624a4fc4b80 2024-03-12T05:55:27.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f929ae10740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a4f2e450 con 0x5624a4fc9cf0 2024-03-12T05:55:27.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f92967dd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4190 0x5624a4fc45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f92967dd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4190 0x5624a4fc45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55456/0 (socket says 172.21.15.76:55456) 2024-03-12T05:55:27.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f92967dd640 1 -- 172.21.15.76:0/1697326060 learned_addr learned my addr 172.21.15.76:0/1697326060 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f9286ffd640 1 --2- 172.21.15.76:0/1394022706 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f927c0a1ab0 0x7f927c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f92957db640 1 --2- 172.21.15.76:0/1697326060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 0x5624a4fcc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f9295fdc640 1 --2- 172.21.15.76:0/1697326060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4b80 0x5624a4fc9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f92967dd640 1 --2- 172.21.15.76:0/1394022706 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f927c0a1ab0 0x7f927c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f92967dd640 1 -- 172.21.15.76:0/1697326060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 msgr2=0x5624a4fcc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f92967dd640 1 --2- 172.21.15.76:0/1697326060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 0x5624a4fcc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f92967dd640 1 -- 172.21.15.76:0/1697326060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4b80 msgr2=0x5624a4fc9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f92967dd640 1 --2- 172.21.15.76:0/1697326060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4b80 0x5624a4fc9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f92967dd640 1 -- 172.21.15.76:0/1697326060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624a4edef20 con 0x5624a4fc4190 2024-03-12T05:55:27.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f9286ffd640 1 -- 172.21.15.76:0/1394022706 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f92800cd0e0 con 0x7f9290101280 2024-03-12T05:55:27.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.638+0000 7f92957db640 1 --2- 172.21.15.76:0/1697326060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 0x5624a4fcc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:27.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.639+0000 7f92967dd640 1 --2- 172.21.15.76:0/1697326060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4190 0x5624a4fc45b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f928c014640 tx=0x7f928c014b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c6439f27ea0365f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.639+0000 7f927bfff640 1 -- 172.21.15.76:0/1697326060 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c01d830 con 0x5624a4fc4190 2024-03-12T05:55:27.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.639+0000 7f92967dd640 1 --2- 172.21.15.76:0/1394022706 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f927c0a1ab0 0x7f927c0a3f70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f928c01e000 tx=0x7f928c01b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.639+0000 7f927bfff640 1 -- 172.21.15.76:0/1697326060 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f928c01de70 con 0x5624a4fc4190 2024-03-12T05:55:27.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.639+0000 7f927bfff640 1 -- 172.21.15.76:0/1697326060 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c00ec30 con 0x5624a4fc4190 2024-03-12T05:55:27.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 -- 172.21.15.76:0/1697326060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4190 msgr2=0x5624a4fc45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 --2- 172.21.15.76:0/1697326060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4190 0x5624a4fc45b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f928c014640 tx=0x7f928c014b00 comp rx=0 tx=0).stop 2024-03-12T05:55:27.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 -- 172.21.15.76:0/1697326060 shutdown_connections 2024-03-12T05:55:27.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 --2- 172.21.15.76:0/1697326060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 0x5624a4fcc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 --2- 172.21.15.76:0/1697326060 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4b80 0x5624a4fc9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 --2- 172.21.15.76:0/1697326060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4190 0x5624a4fc45b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 -- 172.21.15.76:0/1697326060 >> 172.21.15.76:0/1697326060 conn(0x5624a4fbd4f0 msgr2=0x5624a4fbd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 -- 172.21.15.76:0/1697326060 shutdown_connections 2024-03-12T05:55:27.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 -- 172.21.15.76:0/1697326060 wait complete. 2024-03-12T05:55:27.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 -- start start 2024-03-12T05:55:27.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 Processor -- start 2024-03-12T05:55:27.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4190 0x5624a4fce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 0x5624a4fd58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a4f29c30 con 0x5624a4fc4b80 2024-03-12T05:55:27.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a4eced30 con 0x5624a4fc4190 2024-03-12T05:55:27.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.640+0000 7f929ae10740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a4bfd310 con 0x5624a4fc9cf0 2024-03-12T05:55:27.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f929ae10740 10 client.0 _ll_register_callbacks cb 0x5624a4ed6d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:27.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92957db640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92957db640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fce880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55464/0 (socket says 172.21.15.76:55464) 2024-03-12T05:55:27.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92957db640 1 -- 172.21.15.76:0/1938521403 learned_addr learned my addr 172.21.15.76:0/1938521403 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92967dd640 1 --2- 172.21.15.76:0/1938521403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 0x5624a4fd58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f9295fdc640 1 --2- 172.21.15.76:0/1938521403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4190 0x5624a4fce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92957db640 1 -- 172.21.15.76:0/1938521403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 msgr2=0x5624a4fd58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92957db640 1 --2- 172.21.15.76:0/1938521403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 0x5624a4fd58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92957db640 1 -- 172.21.15.76:0/1938521403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4190 msgr2=0x5624a4fce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92957db640 1 --2- 172.21.15.76:0/1938521403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4190 0x5624a4fce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92957db640 1 -- 172.21.15.76:0/1938521403 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928c00bda0 con 0x5624a4fc4b80 2024-03-12T05:55:27.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f9295fdc640 1 --2- 172.21.15.76:0/1938521403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4190 0x5624a4fce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:27.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92957db640 1 --2- 172.21.15.76:0/1938521403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fce880 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9280096a30 tx=0x7f9280069490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13977012a8203298 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92797fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:27.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92797fa640 1 -- 172.21.15.76:0/1938521403 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9280005420 con 0x5624a4fc4b80 2024-03-12T05:55:27.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92797fa640 1 -- 172.21.15.76:0/1938521403 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9280005600 con 0x5624a4fc4b80 2024-03-12T05:55:27.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f929ae10740 1 -- 172.21.15.76:0/1938521403 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624a4c296c0 con 0x5624a4fc4b80 2024-03-12T05:55:27.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92797fa640 1 -- 172.21.15.76:0/1938521403 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f928009a760 con 0x5624a4fc4b80 2024-03-12T05:55:27.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f929ae10740 10 client.15246 fetch_fsmap learned FSMap version 21 2024-03-12T05:55:27.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f929ae10740 1 -- 172.21.15.76:0/1938521403 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5624a4f282b0 con 0x5624a4fc4b80 2024-03-12T05:55:27.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.641+0000 7f92797fa640 1 -- 172.21.15.76:0/1938521403 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5624a4c296c0 con 0x5624a4fc4b80 2024-03-12T05:55:27.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.642+0000 7f92797fa640 1 -- 172.21.15.76:0/1938521403 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 21) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9280016030 con 0x5624a4fc4b80 2024-03-12T05:55:27.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.642+0000 7f929ae10740 10 client.15246 fetch_fsmap finished waiting for FSMap version 21 2024-03-12T05:55:27.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.642+0000 7f929ae10740 10 client.15246 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-12T05:55:27.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.642+0000 7f929ae10740 20 client.15246 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:27.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.642+0000 7f929ae10740 1 --2- 172.21.15.76:0/1938521403 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5624a4fda6b0 0x5624a4fdf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.642+0000 7f929ae10740 4 client.15246 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:55:27.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.642+0000 7f929ae10740 1 -- 172.21.15.76:0/1938521403 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624a4c7a270 con 0x5624a4fda6b0 2024-03-12T05:55:27.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.642+0000 7f9295fdc640 1 --2- 172.21.15.76:0/1938521403 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5624a4fda6b0 0x5624a4fdf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.642+0000 7f92797fa640 10 client.15246 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:55:27.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.642+0000 7f9295fdc640 1 --2- 172.21.15.76:0/1938521403 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5624a4fda6b0 0x5624a4fdf170 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c695f24c225dee7 server_cookie=891cd83264faae49 in_seq=0 out_seq=0 2024-03-12T05:55:27.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f92797fa640 1 -- 172.21.15.76:0/1938521403 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5624a4c7a270 con 0x5624a4fda6b0 2024-03-12T05:55:27.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f92797fa640 10 client.15246 handle_command_reply: tid=0 2024-03-12T05:55:27.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f929ae10740 1 client.15246 shutdown 2024-03-12T05:55:27.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f929ae10740 1 -- 172.21.15.76:0/1938521403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 msgr2=0x5624a4fce880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f929ae10740 1 --2- 172.21.15.76:0/1938521403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fce880 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9280096a30 tx=0x7f9280069490 comp rx=0 tx=0).stop 2024-03-12T05:55:27.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f929ae10740 1 -- 172.21.15.76:0/1938521403 shutdown_connections 2024-03-12T05:55:27.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f929ae10740 1 --2- 172.21.15.76:0/1938521403 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5624a4fda6b0 0x5624a4fdf170 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f929ae10740 1 --2- 172.21.15.76:0/1938521403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 0x5624a4fd58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f929ae10740 1 --2- 172.21.15.76:0/1938521403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fce880 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f929ae10740 1 --2- 172.21.15.76:0/1938521403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4190 0x5624a4fce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f92797fa640 0 client.15246 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:55:27.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f929ae10740 1 -- 172.21.15.76:0/1938521403 >> 172.21.15.76:0/1938521403 conn(0x5624a4fbd4f0 msgr2=0x5624a4fbc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f929ae10740 1 -- 172.21.15.76:0/1938521403 shutdown_connections 2024-03-12T05:55:27.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f929ae10740 1 -- 172.21.15.76:0/1938521403 wait complete. 2024-03-12T05:55:27.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.643+0000 7f929ae10740 20 client.15246 trim_cache size 0 max 16384 2024-03-12T05:55:27.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.658+0000 7f929ae10740 1 Processor -- start 2024-03-12T05:55:27.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f929ae10740 1 -- start start 2024-03-12T05:55:27.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f929ae10740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc4190 0x5624a4fc60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f929ae10740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fc65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f929ae10740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc9cf0 0x5624a4fd5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f929ae10740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a4eaf490 con 0x5624a4fc4b80 2024-03-12T05:55:27.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f929ae10740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a4fc98a0 con 0x5624a4fc9cf0 2024-03-12T05:55:27.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f929ae10740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a4fd6540 con 0x5624a4fc4190 2024-03-12T05:55:27.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f92957db640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fc65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f92957db640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fc65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55476/0 (socket says 172.21.15.76:55476) 2024-03-12T05:55:27.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f92957db640 1 -- 172.21.15.76:0/175945580 learned_addr learned my addr 172.21.15.76:0/175945580 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f92957db640 1 -- 172.21.15.76:0/175945580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc4190 msgr2=0x5624a4fc60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f92957db640 1 --2- 172.21.15.76:0/175945580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc4190 0x5624a4fc60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f92957db640 1 -- 172.21.15.76:0/175945580 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc9cf0 msgr2=0x5624a4fd5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f92957db640 1 --2- 172.21.15.76:0/175945580 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc9cf0 0x5624a4fd5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f92957db640 1 -- 172.21.15.76:0/175945580 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624a4ec4a20 con 0x5624a4fc4b80 2024-03-12T05:55:27.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f92957db640 1 --2- 172.21.15.76:0/175945580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fc65e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f92800058c0 tx=0x7f9280098570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37dab95b124e59c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f927b7fe640 1 -- 172.21.15.76:0/175945580 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92800b0080 con 0x5624a4fc4b80 2024-03-12T05:55:27.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f927b7fe640 1 -- 172.21.15.76:0/175945580 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f928000a4b0 con 0x5624a4fc4b80 2024-03-12T05:55:27.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.659+0000 7f927b7fe640 1 -- 172.21.15.76:0/175945580 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92800a25d0 con 0x5624a4fc4b80 2024-03-12T05:55:27.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.660+0000 7f929ae10740 1 -- 172.21.15.76:0/175945580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 msgr2=0x5624a4fc65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.660+0000 7f929ae10740 1 --2- 172.21.15.76:0/175945580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fc65e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f92800058c0 tx=0x7f9280098570 comp rx=0 tx=0).stop 2024-03-12T05:55:27.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.660+0000 7f929ae10740 1 -- 172.21.15.76:0/175945580 shutdown_connections 2024-03-12T05:55:27.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.660+0000 7f929ae10740 1 --2- 172.21.15.76:0/175945580 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc9cf0 0x5624a4fd5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.660+0000 7f929ae10740 1 --2- 172.21.15.76:0/175945580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fc65e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.660+0000 7f929ae10740 1 --2- 172.21.15.76:0/175945580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc4190 0x5624a4fc60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.660+0000 7f929ae10740 1 -- 172.21.15.76:0/175945580 >> 172.21.15.76:0/175945580 conn(0x5624a4fbd4f0 msgr2=0x5624a4fbd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.660+0000 7f929ae10740 1 -- 172.21.15.76:0/175945580 shutdown_connections 2024-03-12T05:55:27.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.660+0000 7f929ae10740 1 -- 172.21.15.76:0/175945580 wait complete. 2024-03-12T05:55:27.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.660+0000 7f929ae10740 1 -- start start 2024-03-12T05:55:27.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.660+0000 7f929ae10740 1 Processor -- start 2024-03-12T05:55:27.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.660+0000 7f929ae10740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4190 0x5624a4fd5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.660+0000 7f929ae10740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fd2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f929ae10740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 0x5624a4fd25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f929ae10740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a4fdaf20 con 0x5624a4fc4b80 2024-03-12T05:55:27.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f929ae10740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a4fd2b10 con 0x5624a4fc4190 2024-03-12T05:55:27.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f929ae10740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a4fd2cd0 con 0x5624a4fc9cf0 2024-03-12T05:55:27.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f929ae10740 10 client.0 _ll_register_callbacks cb 0x7f928c00b360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:27.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92957db640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fd2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92957db640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fd2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55484/0 (socket says 172.21.15.76:55484) 2024-03-12T05:55:27.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92957db640 1 -- 172.21.15.76:0/1904099422 learned_addr learned my addr 172.21.15.76:0/1904099422 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f9295fdc640 1 --2- 172.21.15.76:0/1904099422 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4190 0x5624a4fd5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92967dd640 1 --2- 172.21.15.76:0/1904099422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 0x5624a4fd25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92957db640 1 -- 172.21.15.76:0/1904099422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 msgr2=0x5624a4fd25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92957db640 1 --2- 172.21.15.76:0/1904099422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 0x5624a4fd25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92957db640 1 -- 172.21.15.76:0/1904099422 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4190 msgr2=0x5624a4fd5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92957db640 1 --2- 172.21.15.76:0/1904099422 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4190 0x5624a4fd5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92957db640 1 -- 172.21.15.76:0/1904099422 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624a4f282b0 con 0x5624a4fc4b80 2024-03-12T05:55:27.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92957db640 1 --2- 172.21.15.76:0/1904099422 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fd2090 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f928000c940 tx=0x7f9280094a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d35de056a2f285cd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92797fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:27.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92797fa640 1 -- 172.21.15.76:0/1904099422 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9280005600 con 0x5624a4fc4b80 2024-03-12T05:55:27.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92797fa640 1 -- 172.21.15.76:0/1904099422 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f928009a060 con 0x5624a4fc4b80 2024-03-12T05:55:27.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f929ae10740 1 -- 172.21.15.76:0/1904099422 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f928c01de70 con 0x5624a4fc4b80 2024-03-12T05:55:27.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92797fa640 1 -- 172.21.15.76:0/1904099422 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f928009e1b0 con 0x5624a4fc4b80 2024-03-12T05:55:27.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f929ae10740 10 client.15252 fetch_fsmap learned FSMap version 21 2024-03-12T05:55:27.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f929ae10740 1 -- 172.21.15.76:0/1904099422 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5624a4eea780 con 0x5624a4fc4b80 2024-03-12T05:55:27.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.661+0000 7f92797fa640 1 -- 172.21.15.76:0/1904099422 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f928007d080 con 0x5624a4fc4b80 2024-03-12T05:55:27.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.662+0000 7f92797fa640 1 -- 172.21.15.76:0/1904099422 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 21) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f928006d070 con 0x5624a4fc4b80 2024-03-12T05:55:27.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.662+0000 7f929ae10740 10 client.15252 fetch_fsmap finished waiting for FSMap version 21 2024-03-12T05:55:27.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.662+0000 7f929ae10740 10 client.15252 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-12T05:55:27.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.662+0000 7f929ae10740 20 client.15252 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:27.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.662+0000 7f929ae10740 1 --2- 172.21.15.76:0/1904099422 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5624a5035900 0x5624a5031ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.662+0000 7f929ae10740 4 client.15252 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:55:27.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.662+0000 7f929ae10740 1 -- 172.21.15.76:0/1904099422 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5624a4c66610 con 0x5624a5035900 2024-03-12T05:55:27.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.662+0000 7f9295fdc640 1 --2- 172.21.15.76:0/1904099422 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5624a5035900 0x5624a5031ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.662+0000 7f92797fa640 10 client.15252 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:55:27.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.662+0000 7f9295fdc640 1 --2- 172.21.15.76:0/1904099422 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5624a5035900 0x5624a5031ee0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=669c1ac18ad271ee server_cookie=10ebf053202e4400 in_seq=0 out_seq=0 2024-03-12T05:55:27.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.662+0000 7f92797fa640 1 -- 172.21.15.76:0/1904099422 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x5624a4c66610 con 0x5624a5035900 2024-03-12T05:55:27.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.662+0000 7f92797fa640 10 client.15252 handle_command_reply: tid=0 2024-03-12T05:55:27.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.662+0000 7f929ae10740 1 client.15252 shutdown 2024-03-12T05:55:27.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f929ae10740 1 -- 172.21.15.76:0/1904099422 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 msgr2=0x5624a4fd2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f929ae10740 1 --2- 172.21.15.76:0/1904099422 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fd2090 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f928000c940 tx=0x7f9280094a80 comp rx=0 tx=0).stop 2024-03-12T05:55:27.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f929ae10740 1 -- 172.21.15.76:0/1904099422 shutdown_connections 2024-03-12T05:55:27.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f929ae10740 1 --2- 172.21.15.76:0/1904099422 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5624a5035900 0x5624a5031ee0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f929ae10740 1 --2- 172.21.15.76:0/1904099422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a4fc9cf0 0x5624a4fd25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f929ae10740 1 --2- 172.21.15.76:0/1904099422 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5624a4fc4b80 0x5624a4fd2090 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f92797fa640 0 client.15252 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:55:27.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f929ae10740 1 --2- 172.21.15.76:0/1904099422 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5624a4fc4190 0x5624a4fd5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f929ae10740 1 -- 172.21.15.76:0/1904099422 >> 172.21.15.76:0/1904099422 conn(0x5624a4fbd4f0 msgr2=0x5624a4fc60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f929ae10740 1 -- 172.21.15.76:0/1904099422 shutdown_connections 2024-03-12T05:55:27.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f929ae10740 1 -- 172.21.15.76:0/1904099422 wait complete. 2024-03-12T05:55:27.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f929ae10740 20 client.15252 trim_cache size 0 max 16384 2024-03-12T05:55:27.674 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:55:27.674 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "9f336de8-e033-11ee-95c8-87774f69a715", 2024-03-12T05:55:27.674 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 2, 2024-03-12T05:55:27.674 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24451, 2024-03-12T05:55:27.674 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-03-12T05:55:27.674 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-03-12T05:55:27.674 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-03-12T05:55:27.674 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 106.297113522, 2024-03-12T05:55:27.674 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 21, 2024-03-12T05:55:27.674 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 88, 2024-03-12T05:55:27.674 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 86, 2024-03-12T05:55:27.674 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 131.790041102 2024-03-12T05:55:27.674 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:55:27.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f9298240640 1 -- 172.21.15.76:0/1394022706 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f927c0a1ab0 msgr2=0x7f927c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f9298240640 1 --2- 172.21.15.76:0/1394022706 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f927c0a1ab0 0x7f927c0a3f70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f928c01e000 tx=0x7f928c01b000 comp rx=0 tx=0).stop 2024-03-12T05:55:27.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f9298240640 1 -- 172.21.15.76:0/1394022706 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9290101280 msgr2=0x7f92901966a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f9298240640 1 --2- 172.21.15.76:0/1394022706 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9290101280 0x7f92901966a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f928000c970 tx=0x7f928000ce30 comp rx=0 tx=0).stop 2024-03-12T05:55:27.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f9298240640 1 -- 172.21.15.76:0/1394022706 shutdown_connections 2024-03-12T05:55:27.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f9298240640 1 --2- 172.21.15.76:0/1394022706 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f927c0a1ab0 0x7f927c0a3f70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f9298240640 1 --2- 172.21.15.76:0/1394022706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9290101c40 0x7f929019d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f9298240640 1 --2- 172.21.15.76:0/1394022706 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9290101280 0x7f92901966a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f9298240640 1 --2- 172.21.15.76:0/1394022706 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9290100090 0x7f9290196160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f9298240640 1 -- 172.21.15.76:0/1394022706 >> 172.21.15.76:0/1394022706 conn(0x7f92900fb5e0 msgr2=0x7f9290104650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f9298240640 1 -- 172.21.15.76:0/1394022706 shutdown_connections 2024-03-12T05:55:27.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.663+0000 7f9298240640 1 -- 172.21.15.76:0/1394022706 wait complete. 2024-03-12T05:55:27.677 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-12T05:55:27.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.777+0000 7fcf2a3e2640 1 -- 172.21.15.76:0/1479087164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf24101c50 msgr2=0x7fcf241084e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.777+0000 7fcf2a3e2640 1 --2- 172.21.15.76:0/1479087164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf24101c50 0x7fcf241084e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009ca0 tx=0x7fcf1c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.777+0000 7fcf2a3e2640 1 -- 172.21.15.76:0/1479087164 shutdown_connections 2024-03-12T05:55:27.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.777+0000 7fcf2a3e2640 1 --2- 172.21.15.76:0/1479087164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf24101c50 0x7fcf241084e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.777+0000 7fcf2a3e2640 1 --2- 172.21.15.76:0/1479087164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf24101290 0x7fcf24101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.777+0000 7fcf2a3e2640 1 --2- 172.21.15.76:0/1479087164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf241000a0 0x7fcf241004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.777+0000 7fcf2a3e2640 1 -- 172.21.15.76:0/1479087164 >> 172.21.15.76:0/1479087164 conn(0x7fcf240fb610 msgr2=0x7fcf240fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.777+0000 7fcf2a3e2640 1 -- 172.21.15.76:0/1479087164 shutdown_connections 2024-03-12T05:55:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.777+0000 7fcf2a3e2640 1 -- 172.21.15.76:0/1479087164 wait complete. 2024-03-12T05:55:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.778+0000 7fcf2a3e2640 1 Processor -- start 2024-03-12T05:55:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.778+0000 7fcf2a3e2640 1 -- start start 2024-03-12T05:55:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.778+0000 7fcf2a3e2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf241000a0 0x7fcf2410a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.778+0000 7fcf2a3e2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf24101290 0x7fcf24108fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.778+0000 7fcf2a3e2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf24101c50 0x7fcf241094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.778+0000 7fcf2a3e2640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf24109c40 con 0x7fcf24101c50 2024-03-12T05:55:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.778+0000 7fcf2a3e2640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf24109db0 con 0x7fcf24101290 2024-03-12T05:55:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.778+0000 7fcf2a3e2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf241a1270 con 0x7fcf241000a0 2024-03-12T05:55:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf2897f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf24101c50 0x7fcf241094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf2897f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf24101c50 0x7fcf241094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55512/0 (socket says 172.21.15.76:55512) 2024-03-12T05:55:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf2897f640 1 -- 172.21.15.76:0/3859241591 learned_addr learned my addr 172.21.15.76:0/3859241591 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf237fe640 1 --2- 172.21.15.76:0/3859241591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf24101290 0x7fcf24108fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf23fff640 1 --2- 172.21.15.76:0/3859241591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf241000a0 0x7fcf2410a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf2897f640 1 -- 172.21.15.76:0/3859241591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf241000a0 msgr2=0x7fcf2410a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf2897f640 1 --2- 172.21.15.76:0/3859241591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf241000a0 0x7fcf2410a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf2897f640 1 -- 172.21.15.76:0/3859241591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf24101290 msgr2=0x7fcf24108fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf2897f640 1 --2- 172.21.15.76:0/3859241591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf24101290 0x7fcf24108fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf2897f640 1 -- 172.21.15.76:0/3859241591 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c009900 con 0x7fcf24101c50 2024-03-12T05:55:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf2897f640 1 --2- 172.21.15.76:0/3859241591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf24101c50 0x7fcf241094e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0098d0 tx=0x7fcf1c034d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf217fa640 1 -- 172.21.15.76:0/3859241591 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c032eb0 con 0x7fcf24101c50 2024-03-12T05:55:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf2a3e2640 1 -- 172.21.15.76:0/3859241591 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf241a14c0 con 0x7fcf24101c50 2024-03-12T05:55:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf217fa640 1 -- 172.21.15.76:0/3859241591 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf1c003e50 con 0x7fcf24101c50 2024-03-12T05:55:27.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf217fa640 1 -- 172.21.15.76:0/3859241591 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c0406b0 con 0x7fcf24101c50 2024-03-12T05:55:27.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.779+0000 7fcf2a3e2640 1 -- 172.21.15.76:0/3859241591 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf241a1970 con 0x7fcf24101c50 2024-03-12T05:55:27.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.781+0000 7fcf217fa640 1 -- 172.21.15.76:0/3859241591 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fcf1c00a040 con 0x7fcf24101c50 2024-03-12T05:55:27.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.781+0000 7fcf2cfba740 1 Processor -- start 2024-03-12T05:55:27.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf2cfba740 1 -- start start 2024-03-12T05:55:27.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf2cfba740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c740af80 0x5635c740b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf2cfba740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7410470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf2cfba740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c7410ae0 0x5635c7412f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf2cfba740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635c73c6cd0 con 0x5635c740b970 2024-03-12T05:55:27.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf2cfba740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635c7379f20 con 0x5635c740af80 2024-03-12T05:55:27.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf2cfba740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635c7377450 con 0x5635c7410ae0 2024-03-12T05:55:27.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf23fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7410470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf23fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7410470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55526/0 (socket says 172.21.15.76:55526) 2024-03-12T05:55:27.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf23fff640 1 -- 172.21.15.76:0/2510311875 learned_addr learned my addr 172.21.15.76:0/2510311875 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf217fa640 1 --2- 172.21.15.76:0/3859241591 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcf0c0a1a30 0x7fcf0c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf237fe640 1 --2- 172.21.15.76:0/2510311875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c740af80 0x5635c740b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf237fe640 1 --2- 172.21.15.76:0/2510311875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c7410ae0 0x5635c7412f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf2897f640 1 --2- 172.21.15.76:0/3859241591 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcf0c0a1a30 0x7fcf0c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf23fff640 1 -- 172.21.15.76:0/2510311875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c7410ae0 msgr2=0x5635c7412f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf23fff640 1 --2- 172.21.15.76:0/2510311875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c7410ae0 0x5635c7412f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf217fa640 1 -- 172.21.15.76:0/3859241591 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcf1c0f0320 con 0x7fcf24101c50 2024-03-12T05:55:27.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf23fff640 1 -- 172.21.15.76:0/2510311875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c740af80 msgr2=0x5635c740b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf23fff640 1 --2- 172.21.15.76:0/2510311875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c740af80 0x5635c740b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf23fff640 1 -- 172.21.15.76:0/2510311875 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635c7327f20 con 0x5635c740b970 2024-03-12T05:55:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf23fff640 1 --2- 172.21.15.76:0/2510311875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7410470 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400cbc0 tx=0x7fcf1400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e479d7e464d5de8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf2897f640 1 --2- 172.21.15.76:0/3859241591 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcf0c0a1a30 0x7fcf0c0a3ef0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0ee1a0 tx=0x7fcf1c090630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.782+0000 7fcf0a7fc640 1 -- 172.21.15.76:0/2510311875 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14010630 con 0x5635c740b970 2024-03-12T05:55:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.783+0000 7fcf0a7fc640 1 -- 172.21.15.76:0/2510311875 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf1400eed0 con 0x5635c740b970 2024-03-12T05:55:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.783+0000 7fcf0a7fc640 1 -- 172.21.15.76:0/2510311875 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1401ab80 con 0x5635c740b970 2024-03-12T05:55:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.783+0000 7fcf2cfba740 1 -- 172.21.15.76:0/2510311875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 msgr2=0x5635c7410470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.783+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/2510311875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7410470 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400cbc0 tx=0x7fcf1400e720 comp rx=0 tx=0).stop 2024-03-12T05:55:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.783+0000 7fcf2cfba740 1 -- 172.21.15.76:0/2510311875 shutdown_connections 2024-03-12T05:55:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.783+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/2510311875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c7410ae0 0x5635c7412f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.783+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/2510311875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7410470 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.783+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/2510311875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c740af80 0x5635c740b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.783+0000 7fcf2cfba740 1 -- 172.21.15.76:0/2510311875 >> 172.21.15.76:0/2510311875 conn(0x5635c74064f0 msgr2=0x5635c74068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.783+0000 7fcf2cfba740 1 -- 172.21.15.76:0/2510311875 shutdown_connections 2024-03-12T05:55:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.783+0000 7fcf2cfba740 1 -- 172.21.15.76:0/2510311875 wait complete. 2024-03-12T05:55:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.783+0000 7fcf2cfba740 1 -- start start 2024-03-12T05:55:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf2cfba740 1 Processor -- start 2024-03-12T05:55:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf2cfba740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c740af80 0x5635c7415110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf2cfba740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7415650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf2cfba740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c7410ae0 0x5635c741c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf2cfba740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635c7372c30 con 0x5635c740b970 2024-03-12T05:55:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf2cfba740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635c7317d30 con 0x5635c7410ae0 2024-03-12T05:55:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf2cfba740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635c7046310 con 0x5635c740af80 2024-03-12T05:55:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf2cfba740 10 client.0 _ll_register_callbacks cb 0x5635c731fd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf23fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7415650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf23fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7415650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55534/0 (socket says 172.21.15.76:55534) 2024-03-12T05:55:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf23fff640 1 -- 172.21.15.76:0/4203359957 learned_addr learned my addr 172.21.15.76:0/4203359957 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf237fe640 1 --2- 172.21.15.76:0/4203359957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c7410ae0 0x5635c741c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf23fff640 1 -- 172.21.15.76:0/4203359957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c740af80 msgr2=0x5635c7415110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf23fff640 1 --2- 172.21.15.76:0/4203359957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c740af80 0x5635c7415110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf23fff640 1 -- 172.21.15.76:0/4203359957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c7410ae0 msgr2=0x5635c741c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf23fff640 1 --2- 172.21.15.76:0/4203359957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c7410ae0 0x5635c741c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf23fff640 1 -- 172.21.15.76:0/4203359957 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1400c7f0 con 0x5635c740b970 2024-03-12T05:55:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.784+0000 7fcf237fe640 1 --2- 172.21.15.76:0/4203359957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c7410ae0 0x5635c741c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf23fff640 1 --2- 172.21.15.76:0/4203359957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7415650 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fcf140058c0 tx=0x7fcf1401bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e439d2b58f911c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf037fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf037fe640 1 -- 172.21.15.76:0/4203359957 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14010840 con 0x5635c740b970 2024-03-12T05:55:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf037fe640 1 -- 172.21.15.76:0/4203359957 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcf140109a0 con 0x5635c740b970 2024-03-12T05:55:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf2cfba740 1 -- 172.21.15.76:0/4203359957 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5635c70726c0 con 0x5635c740b970 2024-03-12T05:55:27.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf037fe640 1 -- 172.21.15.76:0/4203359957 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf1401a520 con 0x5635c740b970 2024-03-12T05:55:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf2cfba740 10 client.15264 fetch_fsmap learned FSMap version 21 2024-03-12T05:55:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf2cfba740 1 -- 172.21.15.76:0/4203359957 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5635c73712b0 con 0x5635c740b970 2024-03-12T05:55:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf037fe640 1 -- 172.21.15.76:0/4203359957 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf14034020 con 0x5635c740b970 2024-03-12T05:55:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf037fe640 1 -- 172.21.15.76:0/4203359957 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 21) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcf14022070 con 0x5635c740b970 2024-03-12T05:55:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf2cfba740 10 client.15264 fetch_fsmap finished waiting for FSMap version 21 2024-03-12T05:55:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf2cfba740 10 client.15264 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-12T05:55:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf2cfba740 20 client.15264 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/4203359957 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5635c7421970 0x5635c7420c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf2cfba740 4 client.15264 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:55:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.785+0000 7fcf2cfba740 1 -- 172.21.15.76:0/4203359957 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5635c70c3270 con 0x5635c7421970 2024-03-12T05:55:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.786+0000 7fcf237fe640 1 --2- 172.21.15.76:0/4203359957 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5635c7421970 0x5635c7420c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.786+0000 7fcf037fe640 10 client.15264 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:55:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.786+0000 7fcf237fe640 1 --2- 172.21.15.76:0/4203359957 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5635c7421970 0x5635c7420c30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e8087e8deb3aff31 server_cookie=bf4e7cce7690909d in_seq=0 out_seq=0 2024-03-12T05:55:27.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf037fe640 1 -- 172.21.15.76:0/4203359957 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5635c70c3270 con 0x5635c7421970 2024-03-12T05:55:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf037fe640 10 client.15264 handle_command_reply: tid=0 2024-03-12T05:55:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf2cfba740 1 client.15264 shutdown 2024-03-12T05:55:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf2cfba740 1 -- 172.21.15.76:0/4203359957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 msgr2=0x5635c7415650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/4203359957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7415650 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fcf140058c0 tx=0x7fcf1401bf70 comp rx=0 tx=0).stop 2024-03-12T05:55:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf2cfba740 1 -- 172.21.15.76:0/4203359957 shutdown_connections 2024-03-12T05:55:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/4203359957 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5635c7421970 0x5635c7420c30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/4203359957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c7410ae0 0x5635c741c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf037fe640 0 client.15264 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:55:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/4203359957 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7415650 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/4203359957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c740af80 0x5635c7415110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf2cfba740 1 -- 172.21.15.76:0/4203359957 >> 172.21.15.76:0/4203359957 conn(0x5635c74064f0 msgr2=0x5635c74057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf2cfba740 1 -- 172.21.15.76:0/4203359957 shutdown_connections 2024-03-12T05:55:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf2cfba740 1 -- 172.21.15.76:0/4203359957 wait complete. 2024-03-12T05:55:27.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.787+0000 7fcf2cfba740 20 client.15264 trim_cache size 0 max 16384 2024-03-12T05:55:27.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf2cfba740 1 Processor -- start 2024-03-12T05:55:27.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf2cfba740 1 -- start start 2024-03-12T05:55:27.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf2cfba740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c740af80 0x5635c7404f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf2cfba740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c741c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf2cfba740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c7410ae0 0x5635c7405480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf2cfba740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635c72f8490 con 0x5635c740b970 2024-03-12T05:55:27.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf2cfba740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635c740c060 con 0x5635c7410ae0 2024-03-12T05:55:27.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf2cfba740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635c736bad0 con 0x5635c740af80 2024-03-12T05:55:27.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf23fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c741c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf23fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c741c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55550/0 (socket says 172.21.15.76:55550) 2024-03-12T05:55:27.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf23fff640 1 -- 172.21.15.76:0/1118840834 learned_addr learned my addr 172.21.15.76:0/1118840834 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf23fff640 1 -- 172.21.15.76:0/1118840834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c740af80 msgr2=0x5635c7404f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf23fff640 1 --2- 172.21.15.76:0/1118840834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c740af80 0x5635c7404f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf23fff640 1 -- 172.21.15.76:0/1118840834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c7410ae0 msgr2=0x5635c7405480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf237fe640 1 --2- 172.21.15.76:0/1118840834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c7410ae0 0x5635c7405480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf23fff640 1 --2- 172.21.15.76:0/1118840834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c7410ae0 0x5635c7405480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.803+0000 7fcf23fff640 1 -- 172.21.15.76:0/1118840834 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5635c730da20 con 0x5635c740b970 2024-03-12T05:55:27.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.804+0000 7fcf23fff640 1 --2- 172.21.15.76:0/1118840834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c741c650 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400eca0 tx=0x7fcf1400c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c841290d3affd01 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.804+0000 7fcf09ffb640 1 -- 172.21.15.76:0/1118840834 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14005270 con 0x5635c740b970 2024-03-12T05:55:27.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.804+0000 7fcf09ffb640 1 -- 172.21.15.76:0/1118840834 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf14034040 con 0x5635c740b970 2024-03-12T05:55:27.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.804+0000 7fcf09ffb640 1 -- 172.21.15.76:0/1118840834 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1401b360 con 0x5635c740b970 2024-03-12T05:55:27.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.804+0000 7fcf2cfba740 1 -- 172.21.15.76:0/1118840834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 msgr2=0x5635c741c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.804+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/1118840834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c741c650 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400eca0 tx=0x7fcf1400c4d0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.804+0000 7fcf2cfba740 1 -- 172.21.15.76:0/1118840834 shutdown_connections 2024-03-12T05:55:27.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.804+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/1118840834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c7410ae0 0x5635c7405480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.804+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/1118840834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c741c650 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.804+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/1118840834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c740af80 0x5635c7404f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.804+0000 7fcf2cfba740 1 -- 172.21.15.76:0/1118840834 >> 172.21.15.76:0/1118840834 conn(0x5635c74064f0 msgr2=0x5635c740cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.804+0000 7fcf2cfba740 1 -- 172.21.15.76:0/1118840834 shutdown_connections 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.804+0000 7fcf2cfba740 1 -- 172.21.15.76:0/1118840834 wait complete. 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf2cfba740 1 -- start start 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf2cfba740 1 Processor -- start 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf2cfba740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c740af80 0x5635c74196e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf2cfba740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7419c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf2cfba740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c7410ae0 0x5635c741abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf2cfba740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635c741b100 con 0x5635c740b970 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf2cfba740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635c741b270 con 0x5635c740af80 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf2cfba740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635c741b430 con 0x5635c7410ae0 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf2cfba740 10 client.0 _ll_register_callbacks cb 0x7fcf1400c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf23fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7419c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf23fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7419c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55562/0 (socket says 172.21.15.76:55562) 2024-03-12T05:55:27.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf23fff640 1 -- 172.21.15.76:0/2668460723 learned_addr learned my addr 172.21.15.76:0/2668460723 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf237fe640 1 --2- 172.21.15.76:0/2668460723 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c740af80 0x5635c74196e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf23fff640 1 -- 172.21.15.76:0/2668460723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c7410ae0 msgr2=0x5635c741abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf23fff640 1 --2- 172.21.15.76:0/2668460723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c7410ae0 0x5635c741abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf23fff640 1 -- 172.21.15.76:0/2668460723 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c740af80 msgr2=0x5635c74196e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf23fff640 1 --2- 172.21.15.76:0/2668460723 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c740af80 0x5635c74196e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf23fff640 1 -- 172.21.15.76:0/2668460723 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5635c73712b0 con 0x5635c740b970 2024-03-12T05:55:27.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf23fff640 1 --2- 172.21.15.76:0/2668460723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7419c20 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcf14030b00 tx=0x7fcf14005650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc8f4cddd5f9f7b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.805+0000 7fcf037fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:27.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf037fe640 1 -- 172.21.15.76:0/2668460723 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14004500 con 0x5635c740b970 2024-03-12T05:55:27.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf037fe640 1 -- 172.21.15.76:0/2668460723 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcf14030e40 con 0x5635c740b970 2024-03-12T05:55:27.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf2cfba740 1 -- 172.21.15.76:0/2668460723 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5635c7421680 con 0x5635c740b970 2024-03-12T05:55:27.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf037fe640 1 -- 172.21.15.76:0/2668460723 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf1401b0e0 con 0x5635c740b970 2024-03-12T05:55:27.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf2cfba740 10 client.15270 fetch_fsmap learned FSMap version 21 2024-03-12T05:55:27.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf2cfba740 1 -- 172.21.15.76:0/2668460723 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x5635c7333780 con 0x5635c740b970 2024-03-12T05:55:27.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf037fe640 1 -- 172.21.15.76:0/2668460723 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5635c7421680 con 0x5635c740b970 2024-03-12T05:55:27.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf037fe640 1 -- 172.21.15.76:0/2668460723 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 21) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcf1404c070 con 0x5635c740b970 2024-03-12T05:55:27.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf2cfba740 10 client.15270 fetch_fsmap finished waiting for FSMap version 21 2024-03-12T05:55:27.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf2cfba740 10 client.15270 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-12T05:55:27.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf2cfba740 20 client.15270 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:27.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/2668460723 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5635c747cc20 0x5635c7479580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf2cfba740 4 client.15270 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:55:27.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf2cfba740 1 -- 172.21.15.76:0/2668460723 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fcf1400eed0 con 0x5635c747cc20 2024-03-12T05:55:27.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.806+0000 7fcf237fe640 1 --2- 172.21.15.76:0/2668460723 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5635c747cc20 0x5635c7479580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf237fe640 1 --2- 172.21.15.76:0/2668460723 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5635c747cc20 0x5635c7479580 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1f0239701a098d7b server_cookie=b54c5d007baceacd in_seq=0 out_seq=0 2024-03-12T05:55:27.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf037fe640 10 client.15270 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:55:27.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf037fe640 1 -- 172.21.15.76:0/2668460723 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7fcf1400eed0 con 0x5635c747cc20 2024-03-12T05:55:27.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf037fe640 10 client.15270 handle_command_reply: tid=0 2024-03-12T05:55:27.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf2cfba740 1 client.15270 shutdown 2024-03-12T05:55:27.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf2cfba740 1 -- 172.21.15.76:0/2668460723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 msgr2=0x5635c7419c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/2668460723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7419c20 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcf14030b00 tx=0x7fcf14005650 comp rx=0 tx=0).stop 2024-03-12T05:55:27.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf2cfba740 1 -- 172.21.15.76:0/2668460723 shutdown_connections 2024-03-12T05:55:27.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/2668460723 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5635c747cc20 0x5635c7479580 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/2668460723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635c7410ae0 0x5635c741abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/2668460723 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5635c740b970 0x5635c7419c20 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf037fe640 0 client.15270 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:55:27.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf2cfba740 1 --2- 172.21.15.76:0/2668460723 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5635c740af80 0x5635c74196e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf2cfba740 1 -- 172.21.15.76:0/2668460723 >> 172.21.15.76:0/2668460723 conn(0x5635c74064f0 msgr2=0x5635c7409750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf2cfba740 1 -- 172.21.15.76:0/2668460723 shutdown_connections 2024-03-12T05:55:27.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf2cfba740 1 -- 172.21.15.76:0/2668460723 wait complete. 2024-03-12T05:55:27.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.807+0000 7fcf2cfba740 20 client.15270 trim_cache size 0 max 16384 2024-03-12T05:55:27.818 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:55:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "9f336de8-e033-11ee-95c8-87774f69a715", 2024-03-12T05:55:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 1, 2024-03-12T05:55:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24481, 2024-03-12T05:55:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-03-12T05:55:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-03-12T05:55:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-03-12T05:55:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 107.43058004300001, 2024-03-12T05:55:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 21, 2024-03-12T05:55:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 88, 2024-03-12T05:55:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 86, 2024-03-12T05:55:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 123.979788435 2024-03-12T05:55:27.818 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:55:27.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.808+0000 7fcf2a3e2640 1 -- 172.21.15.76:0/3859241591 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcf0c0a1a30 msgr2=0x7fcf0c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.808+0000 7fcf2a3e2640 1 --2- 172.21.15.76:0/3859241591 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcf0c0a1a30 0x7fcf0c0a3ef0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0ee1a0 tx=0x7fcf1c090630 comp rx=0 tx=0).stop 2024-03-12T05:55:27.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.808+0000 7fcf2a3e2640 1 -- 172.21.15.76:0/3859241591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf24101c50 msgr2=0x7fcf241094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.808+0000 7fcf2a3e2640 1 --2- 172.21.15.76:0/3859241591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf24101c50 0x7fcf241094e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0098d0 tx=0x7fcf1c034d40 comp rx=0 tx=0).stop 2024-03-12T05:55:27.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.808+0000 7fcf2a3e2640 1 -- 172.21.15.76:0/3859241591 shutdown_connections 2024-03-12T05:55:27.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.808+0000 7fcf2a3e2640 1 --2- 172.21.15.76:0/3859241591 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fcf0c0a1a30 0x7fcf0c0a3ef0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.808+0000 7fcf2a3e2640 1 --2- 172.21.15.76:0/3859241591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf24101c50 0x7fcf241094e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.808+0000 7fcf2a3e2640 1 --2- 172.21.15.76:0/3859241591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf24101290 0x7fcf24108fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.808+0000 7fcf2a3e2640 1 --2- 172.21.15.76:0/3859241591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf241000a0 0x7fcf2410a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.808+0000 7fcf2a3e2640 1 -- 172.21.15.76:0/3859241591 >> 172.21.15.76:0/3859241591 conn(0x7fcf240fb610 msgr2=0x7fcf24104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.808+0000 7fcf2a3e2640 1 -- 172.21.15.76:0/3859241591 shutdown_connections 2024-03-12T05:55:27.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.808+0000 7fcf2a3e2640 1 -- 172.21.15.76:0/3859241591 wait complete. 2024-03-12T05:55:27.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-12T05:55:27.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.921+0000 7efc78b2c640 1 -- 172.21.15.76:0/58750697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efc74106e80 msgr2=0x7efc741092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.921+0000 7efc78b2c640 1 --2- 172.21.15.76:0/58750697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efc74106e80 0x7efc741092d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009ca0 tx=0x7efc6c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.921+0000 7efc78b2c640 1 -- 172.21.15.76:0/58750697 shutdown_connections 2024-03-12T05:55:27.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.921+0000 7efc78b2c640 1 --2- 172.21.15.76:0/58750697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efc74106e80 0x7efc741092d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.921+0000 7efc78b2c640 1 --2- 172.21.15.76:0/58750697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc74104550 0x7efc74106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.921+0000 7efc78b2c640 1 --2- 172.21.15.76:0/58750697 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc74101c20 0x7efc74104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.921+0000 7efc78b2c640 1 -- 172.21.15.76:0/58750697 >> 172.21.15.76:0/58750697 conn(0x7efc740fb610 msgr2=0x7efc740fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.922+0000 7efc78b2c640 1 -- 172.21.15.76:0/58750697 shutdown_connections 2024-03-12T05:55:27.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.922+0000 7efc78b2c640 1 -- 172.21.15.76:0/58750697 wait complete. 2024-03-12T05:55:27.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.922+0000 7efc78b2c640 1 Processor -- start 2024-03-12T05:55:27.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.922+0000 7efc78b2c640 1 -- start start 2024-03-12T05:55:27.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.922+0000 7efc78b2c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efc74101c20 0x7efc74101590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.922+0000 7efc78b2c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc74104550 0x7efc740ffbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.922+0000 7efc78b2c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc74106e80 0x7efc74100110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.922+0000 7efc78b2c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7419b060 con 0x7efc74101c20 2024-03-12T05:55:27.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.922+0000 7efc78b2c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc74100650 con 0x7efc74104550 2024-03-12T05:55:27.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.922+0000 7efc78b2c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc74100810 con 0x7efc74106e80 2024-03-12T05:55:27.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.922+0000 7efc7259c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efc74101c20 0x7efc74101590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.922+0000 7efc7259c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efc74101c20 0x7efc74101590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55566/0 (socket says 172.21.15.76:55566) 2024-03-12T05:55:27.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.923+0000 7efc7259c640 1 -- 172.21.15.76:0/1733789729 learned_addr learned my addr 172.21.15.76:0/1733789729 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.923+0000 7efc72d9d640 1 --2- 172.21.15.76:0/1733789729 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc74106e80 0x7efc74100110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.923+0000 7efc71d9b640 1 --2- 172.21.15.76:0/1733789729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc74104550 0x7efc740ffbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.923+0000 7efc7259c640 1 -- 172.21.15.76:0/1733789729 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc74106e80 msgr2=0x7efc74100110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.923+0000 7efc7259c640 1 --2- 172.21.15.76:0/1733789729 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc74106e80 0x7efc74100110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.923+0000 7efc7259c640 1 -- 172.21.15.76:0/1733789729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc74104550 msgr2=0x7efc740ffbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.923+0000 7efc7259c640 1 --2- 172.21.15.76:0/1733789729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc74104550 0x7efc740ffbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.923+0000 7efc7259c640 1 -- 172.21.15.76:0/1733789729 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c009900 con 0x7efc74101c20 2024-03-12T05:55:27.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.923+0000 7efc7259c640 1 --2- 172.21.15.76:0/1733789729 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efc74101c20 0x7efc74101590 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efc6400c890 tx=0x7efc6400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.923+0000 7efc637fe640 1 -- 172.21.15.76:0/1733789729 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64004220 con 0x7efc74101c20 2024-03-12T05:55:27.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.923+0000 7efc78b2c640 1 -- 172.21.15.76:0/1733789729 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc741a5a50 con 0x7efc74101c20 2024-03-12T05:55:27.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.923+0000 7efc637fe640 1 -- 172.21.15.76:0/1733789729 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc64007e90 con 0x7efc74101c20 2024-03-12T05:55:27.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.923+0000 7efc637fe640 1 -- 172.21.15.76:0/1733789729 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64004a00 con 0x7efc74101c20 2024-03-12T05:55:27.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.923+0000 7efc78b2c640 1 -- 172.21.15.76:0/1733789729 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc741a5e70 con 0x7efc74101c20 2024-03-12T05:55:27.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.926+0000 7efc637fe640 1 -- 172.21.15.76:0/1733789729 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7efc640079b0 con 0x7efc74101c20 2024-03-12T05:55:27.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.926+0000 7efc7b7d9740 1 Processor -- start 2024-03-12T05:55:27.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.926+0000 7efc7b7d9740 1 -- start start 2024-03-12T05:55:27.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.926+0000 7efc7b7d9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e05f80 0x558112e063a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.926+0000 7efc7b7d9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e06970 0x558112e0b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.926+0000 7efc7b7d9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e0bae0 0x558112e0df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.926+0000 7efc7b7d9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558112dc1cd0 con 0x558112e05f80 2024-03-12T05:55:27.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.926+0000 7efc7b7d9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558112d74f20 con 0x558112e06970 2024-03-12T05:55:27.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.926+0000 7efc7b7d9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558112d72450 con 0x558112e0bae0 2024-03-12T05:55:27.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.926+0000 7efc72d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e05f80 0x558112e063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.926+0000 7efc72d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e05f80 0x558112e063a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55576/0 (socket says 172.21.15.76:55576) 2024-03-12T05:55:27.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.926+0000 7efc72d9d640 1 -- 172.21.15.76:0/395824072 learned_addr learned my addr 172.21.15.76:0/395824072 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.926+0000 7efc637fe640 1 --2- 172.21.15.76:0/1733789729 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7efc5c0a1a30 0x7efc5c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc71d9b640 1 --2- 172.21.15.76:0/395824072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e0bae0 0x558112e0df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc72d9d640 1 --2- 172.21.15.76:0/1733789729 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7efc5c0a1a30 0x7efc5c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc637fe640 1 -- 172.21.15.76:0/1733789729 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efc640cd1a0 con 0x7efc74101c20 2024-03-12T05:55:27.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc7259c640 1 --2- 172.21.15.76:0/395824072 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e06970 0x558112e0b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc72d9d640 1 -- 172.21.15.76:0/395824072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e0bae0 msgr2=0x558112e0df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc72d9d640 1 --2- 172.21.15.76:0/395824072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e0bae0 0x558112e0df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc72d9d640 1 -- 172.21.15.76:0/395824072 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e06970 msgr2=0x558112e0b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc72d9d640 1 --2- 172.21.15.76:0/395824072 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e06970 0x558112e0b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc72d9d640 1 -- 172.21.15.76:0/395824072 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558112d22f20 con 0x558112e05f80 2024-03-12T05:55:27.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc7259c640 1 --2- 172.21.15.76:0/395824072 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e06970 0x558112e0b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:27.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc72d9d640 1 --2- 172.21.15.76:0/395824072 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e05f80 0x558112e063a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009430 tx=0x7efc6c034710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44f45d28fa8a2ac server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc60ff9640 1 -- 172.21.15.76:0/395824072 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c032b30 con 0x558112e05f80 2024-03-12T05:55:27.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc60ff9640 1 -- 172.21.15.76:0/395824072 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc6c003e50 con 0x558112e05f80 2024-03-12T05:55:27.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc72d9d640 1 --2- 172.21.15.76:0/1733789729 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7efc5c0a1a30 0x7efc5c0a3ef0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efc6c04b000 tx=0x7efc6c04a000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.927+0000 7efc60ff9640 1 -- 172.21.15.76:0/395824072 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c004860 con 0x558112e05f80 2024-03-12T05:55:27.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.928+0000 7efc7b7d9740 1 -- 172.21.15.76:0/395824072 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e05f80 msgr2=0x558112e063a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.928+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/395824072 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e05f80 0x558112e063a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efc6c009430 tx=0x7efc6c034710 comp rx=0 tx=0).stop 2024-03-12T05:55:27.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.928+0000 7efc7b7d9740 1 -- 172.21.15.76:0/395824072 shutdown_connections 2024-03-12T05:55:27.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.928+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/395824072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e0bae0 0x558112e0df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.928+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/395824072 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e06970 0x558112e0b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.928+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/395824072 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e05f80 0x558112e063a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.928+0000 7efc7b7d9740 1 -- 172.21.15.76:0/395824072 >> 172.21.15.76:0/395824072 conn(0x558112e014f0 msgr2=0x558112e018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.928+0000 7efc7b7d9740 1 -- 172.21.15.76:0/395824072 shutdown_connections 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.928+0000 7efc7b7d9740 1 -- 172.21.15.76:0/395824072 wait complete. 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.928+0000 7efc7b7d9740 1 -- start start 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.928+0000 7efc7b7d9740 1 Processor -- start 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.928+0000 7efc7b7d9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e05f80 0x558112e10130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.928+0000 7efc7b7d9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e06970 0x558112e10670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc7b7d9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e0bae0 0x558112e176b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc7b7d9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558112d6dc30 con 0x558112e06970 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc7b7d9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558112d12d30 con 0x558112e0bae0 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc7b7d9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558112a41310 con 0x558112e05f80 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc7b7d9740 10 client.0 _ll_register_callbacks cb 0x558112d1ad40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc71d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e06970 0x558112e10670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc71d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e06970 0x558112e10670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55578/0 (socket says 172.21.15.76:55578) 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc71d9b640 1 -- 172.21.15.76:0/1981006643 learned_addr learned my addr 172.21.15.76:0/1981006643 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc7259c640 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e05f80 0x558112e10130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc72d9d640 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e0bae0 0x558112e176b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc71d9b640 1 -- 172.21.15.76:0/1981006643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e05f80 msgr2=0x558112e10130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc71d9b640 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e05f80 0x558112e10130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc71d9b640 1 -- 172.21.15.76:0/1981006643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e0bae0 msgr2=0x558112e176b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc71d9b640 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e0bae0 0x558112e176b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc71d9b640 1 -- 172.21.15.76:0/1981006643 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc74109eb0 con 0x558112e06970 2024-03-12T05:55:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc72d9d640 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e0bae0 0x558112e176b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc7259c640 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e05f80 0x558112e10130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc71d9b640 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e06970 0x558112e10670 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efc6800be80 tx=0x7efc68012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5820dc49ca3aec63 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc59ffb640 1 -- 172.21.15.76:0/1981006643 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc68005220 con 0x558112e06970 2024-03-12T05:55:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc59ffb640 1 -- 172.21.15.76:0/1981006643 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efc68005380 con 0x558112e06970 2024-03-12T05:55:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc7b7d9740 1 -- 172.21.15.76:0/1981006643 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558112a6d6c0 con 0x558112e06970 2024-03-12T05:55:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc59ffb640 1 -- 172.21.15.76:0/1981006643 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc68005540 con 0x558112e06970 2024-03-12T05:55:27.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc7b7d9740 10 client.15282 fetch_fsmap learned FSMap version 21 2024-03-12T05:55:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.929+0000 7efc7b7d9740 1 -- 172.21.15.76:0/1981006643 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x558112d6c2b0 con 0x558112e06970 2024-03-12T05:55:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.930+0000 7efc59ffb640 1 -- 172.21.15.76:0/1981006643 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558112a6d6c0 con 0x558112e06970 2024-03-12T05:55:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.930+0000 7efc59ffb640 1 -- 172.21.15.76:0/1981006643 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 21) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efc68010b40 con 0x558112e06970 2024-03-12T05:55:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.930+0000 7efc7b7d9740 10 client.15282 fetch_fsmap finished waiting for FSMap version 21 2024-03-12T05:55:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.930+0000 7efc7b7d9740 10 client.15282 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-12T05:55:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.930+0000 7efc7b7d9740 20 client.15282 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.930+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558112e1cae0 0x558112e1bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.930+0000 7efc7b7d9740 4 client.15282 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:55:27.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.930+0000 7efc7b7d9740 1 -- 172.21.15.76:0/1981006643 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558112abe270 con 0x558112e1cae0 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.930+0000 7efc7259c640 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558112e1cae0 0x558112e1bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.930+0000 7efc59ffb640 10 client.15282 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.930+0000 7efc7259c640 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558112e1cae0 0x558112e1bd00 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bad0b6034d7b3454 server_cookie=1834d5ae089e28c7 in_seq=0 out_seq=0 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc59ffb640 1 -- 172.21.15.76:0/1981006643 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558112abe270 con 0x558112e1cae0 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc59ffb640 10 client.15282 handle_command_reply: tid=0 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc7b7d9740 1 client.15282 shutdown 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc7b7d9740 1 -- 172.21.15.76:0/1981006643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e06970 msgr2=0x558112e10670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e06970 0x558112e10670 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efc6800be80 tx=0x7efc68012820 comp rx=0 tx=0).stop 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc7b7d9740 1 -- 172.21.15.76:0/1981006643 shutdown_connections 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e0bae0 0x558112e176b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558112e1cae0 0x558112e1bd00 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e06970 0x558112e10670 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/1981006643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e05f80 0x558112e10130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc59ffb640 0 client.15282 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:27.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc7b7d9740 1 -- 172.21.15.76:0/1981006643 >> 172.21.15.76:0/1981006643 conn(0x558112e014f0 msgr2=0x558112e007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc7b7d9740 1 -- 172.21.15.76:0/1981006643 shutdown_connections 2024-03-12T05:55:27.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc7b7d9740 1 -- 172.21.15.76:0/1981006643 wait complete. 2024-03-12T05:55:27.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.931+0000 7efc7b7d9740 20 client.15282 trim_cache size 0 max 16384 2024-03-12T05:55:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7b7d9740 1 Processor -- start 2024-03-12T05:55:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7b7d9740 1 -- start start 2024-03-12T05:55:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7b7d9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e05f80 0x558112e018d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7b7d9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e06970 0x558112e17670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7b7d9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e0bae0 0x558112e10130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7b7d9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558112cf3490 con 0x558112e05f80 2024-03-12T05:55:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7b7d9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558112e0f980 con 0x558112e06970 2024-03-12T05:55:27.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7b7d9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558112e0ff70 con 0x558112e0bae0 2024-03-12T05:55:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7259c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e05f80 0x558112e018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7259c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e05f80 0x558112e018d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55594/0 (socket says 172.21.15.76:55594) 2024-03-12T05:55:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7259c640 1 -- 172.21.15.76:0/3050363809 learned_addr learned my addr 172.21.15.76:0/3050363809 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc72d9d640 1 --2- 172.21.15.76:0/3050363809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e0bae0 0x558112e10130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7259c640 1 -- 172.21.15.76:0/3050363809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e0bae0 msgr2=0x558112e10130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7259c640 1 --2- 172.21.15.76:0/3050363809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e0bae0 0x558112e10130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7259c640 1 -- 172.21.15.76:0/3050363809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e06970 msgr2=0x558112e17670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7259c640 1 --2- 172.21.15.76:0/3050363809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e06970 0x558112e17670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7259c640 1 -- 172.21.15.76:0/3050363809 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558112d08a20 con 0x558112e05f80 2024-03-12T05:55:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc72d9d640 1 --2- 172.21.15.76:0/3050363809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e0bae0 0x558112e10130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc7259c640 1 --2- 172.21.15.76:0/3050363809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e05f80 0x558112e018d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7efc64096580 tx=0x7efc64000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d2541c49d6d41e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc5bfff640 1 -- 172.21.15.76:0/3050363809 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc640b6070 con 0x558112e05f80 2024-03-12T05:55:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc5bfff640 1 -- 172.21.15.76:0/3050363809 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc6409e270 con 0x558112e05f80 2024-03-12T05:55:27.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.947+0000 7efc5bfff640 1 -- 172.21.15.76:0/3050363809 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64004380 con 0x558112e05f80 2024-03-12T05:55:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.948+0000 7efc7b7d9740 1 -- 172.21.15.76:0/3050363809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e05f80 msgr2=0x558112e018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.948+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/3050363809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e05f80 0x558112e018d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7efc64096580 tx=0x7efc64000bc0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.948+0000 7efc7b7d9740 1 -- 172.21.15.76:0/3050363809 shutdown_connections 2024-03-12T05:55:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.948+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/3050363809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e0bae0 0x558112e10130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.948+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/3050363809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e06970 0x558112e17670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.948+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/3050363809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e05f80 0x558112e018d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.948+0000 7efc7b7d9740 1 -- 172.21.15.76:0/3050363809 >> 172.21.15.76:0/3050363809 conn(0x558112e014f0 msgr2=0x558112dfc660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.948+0000 7efc7b7d9740 1 -- 172.21.15.76:0/3050363809 shutdown_connections 2024-03-12T05:55:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.948+0000 7efc7b7d9740 1 -- 172.21.15.76:0/3050363809 wait complete. 2024-03-12T05:55:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.948+0000 7efc7b7d9740 1 -- start start 2024-03-12T05:55:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.948+0000 7efc7b7d9740 1 Processor -- start 2024-03-12T05:55:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc7b7d9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e05f80 0x558112e14b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc7b7d9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e06970 0x558112e150a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc7b7d9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e0bae0 0x558112e16060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc7b7d9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558112e165a0 con 0x558112e0bae0 2024-03-12T05:55:27.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc7b7d9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558112e16710 con 0x558112e05f80 2024-03-12T05:55:27.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc7b7d9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558112e168d0 con 0x558112e06970 2024-03-12T05:55:27.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc7b7d9740 10 client.0 _ll_register_callbacks cb 0x7efc6c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:27.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc72d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e0bae0 0x558112e16060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc72d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e0bae0 0x558112e16060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55598/0 (socket says 172.21.15.76:55598) 2024-03-12T05:55:27.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc72d9d640 1 -- 172.21.15.76:0/3503583594 learned_addr learned my addr 172.21.15.76:0/3503583594 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:27.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc7259c640 1 --2- 172.21.15.76:0/3503583594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e05f80 0x558112e14b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc71d9b640 1 --2- 172.21.15.76:0/3503583594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e06970 0x558112e150a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc72d9d640 1 -- 172.21.15.76:0/3503583594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e06970 msgr2=0x558112e150a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc72d9d640 1 --2- 172.21.15.76:0/3503583594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e06970 0x558112e150a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc72d9d640 1 -- 172.21.15.76:0/3503583594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e05f80 msgr2=0x558112e14b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:27.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc72d9d640 1 --2- 172.21.15.76:0/3503583594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e05f80 0x558112e14b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc72d9d640 1 -- 172.21.15.76:0/3503583594 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc741a5e70 con 0x558112e0bae0 2024-03-12T05:55:27.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc7259c640 1 --2- 172.21.15.76:0/3503583594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e05f80 0x558112e14b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:27.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc72d9d640 1 --2- 172.21.15.76:0/3503583594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e0bae0 0x558112e16060 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7efc6c02c700 tx=0x7efc6c032b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547611f7d55476ec server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:27.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:27.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc59ffb640 1 -- 172.21.15.76:0/3503583594 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c009540 con 0x558112e0bae0 2024-03-12T05:55:27.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc59ffb640 1 -- 172.21.15.76:0/3503583594 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efc6c049070 con 0x558112e0bae0 2024-03-12T05:55:27.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc7b7d9740 1 -- 172.21.15.76:0/3503583594 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efc6c003e50 con 0x558112e0bae0 2024-03-12T05:55:27.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc59ffb640 1 -- 172.21.15.76:0/3503583594 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc6c003970 con 0x558112e0bae0 2024-03-12T05:55:27.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc7b7d9740 10 client.15288 fetch_fsmap learned FSMap version 21 2024-03-12T05:55:27.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.949+0000 7efc7b7d9740 1 -- 172.21.15.76:0/3503583594 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) v3 -- 0x558112d2e780 con 0x558112e0bae0 2024-03-12T05:55:27.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.950+0000 7efc59ffb640 1 -- 172.21.15.76:0/3503583594 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc6c003e50 con 0x558112e0bae0 2024-03-12T05:55:27.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.950+0000 7efc59ffb640 1 -- 172.21.15.76:0/3503583594 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 21) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efc6c069070 con 0x558112e0bae0 2024-03-12T05:55:27.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.950+0000 7efc7b7d9740 10 client.15288 fetch_fsmap finished waiting for FSMap version 21 2024-03-12T05:55:27.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.950+0000 7efc7b7d9740 10 client.15288 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.950+0000 7efc7b7d9740 20 client.15288 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.950+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/3503583594 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558112e78430 0x558112e74950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.950+0000 7efc7b7d9740 4 client.15288 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.950+0000 7efc7b7d9740 1 -- 172.21.15.76:0/3503583594 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558112aaa610 con 0x558112e78430 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.950+0000 7efc7259c640 1 --2- 172.21.15.76:0/3503583594 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558112e78430 0x558112e74950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.950+0000 7efc7259c640 1 --2- 172.21.15.76:0/3503583594 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558112e78430 0x558112e74950 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=310abf86ad1e27b server_cookie=46623b649ccae758 in_seq=0 out_seq=0 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.950+0000 7efc59ffb640 10 client.15288 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc59ffb640 1 -- 172.21.15.76:0/3503583594 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x558112aaa610 con 0x558112e78430 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc59ffb640 10 client.15288 handle_command_reply: tid=0 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc7b7d9740 1 client.15288 shutdown 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc7b7d9740 1 -- 172.21.15.76:0/3503583594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e0bae0 msgr2=0x558112e16060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/3503583594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e0bae0 0x558112e16060 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7efc6c02c700 tx=0x7efc6c032b60 comp rx=0 tx=0).stop 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc7b7d9740 1 -- 172.21.15.76:0/3503583594 shutdown_connections 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/3503583594 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558112e78430 0x558112e74950 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/3503583594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558112e0bae0 0x558112e16060 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/3503583594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558112e06970 0x558112e150a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc7b7d9740 1 --2- 172.21.15.76:0/3503583594 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558112e05f80 0x558112e14b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc59ffb640 0 client.15288 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc7b7d9740 1 -- 172.21.15.76:0/3503583594 >> 172.21.15.76:0/3503583594 conn(0x558112e014f0 msgr2=0x558112e018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc7b7d9740 1 -- 172.21.15.76:0/3503583594 shutdown_connections 2024-03-12T05:55:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc7b7d9740 1 -- 172.21.15.76:0/3503583594 wait complete. 2024-03-12T05:55:27.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc7b7d9740 20 client.15288 trim_cache size 0 max 16384 2024-03-12T05:55:27.962 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:55:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_fsid": "9f336de8-e033-11ee-95c8-87774f69a715", 2024-03-12T05:55:27.962 INFO:teuthology.orchestra.run.smithi076.stdout: "whoami": 0, 2024-03-12T05:55:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "id": 24518, 2024-03-12T05:55:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "want_state": "up:active", 2024-03-12T05:55:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "state": "up:active", 2024-03-12T05:55:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "fs_name": "cephfs", 2024-03-12T05:55:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "rank_uptime": 109.589237563, 2024-03-12T05:55:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "mdsmap_epoch": 21, 2024-03-12T05:55:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch": 88, 2024-03-12T05:55:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_epoch_barrier": 86, 2024-03-12T05:55:27.963 INFO:teuthology.orchestra.run.smithi076.stdout: "uptime": 115.80498321100001 2024-03-12T05:55:27.963 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:55:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc78b2c640 1 -- 172.21.15.76:0/1733789729 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7efc5c0a1a30 msgr2=0x7efc5c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc78b2c640 1 --2- 172.21.15.76:0/1733789729 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7efc5c0a1a30 0x7efc5c0a3ef0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efc6c04b000 tx=0x7efc6c04a000 comp rx=0 tx=0).stop 2024-03-12T05:55:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc78b2c640 1 -- 172.21.15.76:0/1733789729 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efc74101c20 msgr2=0x7efc74101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc78b2c640 1 --2- 172.21.15.76:0/1733789729 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efc74101c20 0x7efc74101590 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efc6400c890 tx=0x7efc6400cd50 comp rx=0 tx=0).stop 2024-03-12T05:55:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc78b2c640 1 -- 172.21.15.76:0/1733789729 shutdown_connections 2024-03-12T05:55:27.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc78b2c640 1 --2- 172.21.15.76:0/1733789729 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7efc5c0a1a30 0x7efc5c0a3ef0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc78b2c640 1 --2- 172.21.15.76:0/1733789729 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc74106e80 0x7efc74100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc78b2c640 1 --2- 172.21.15.76:0/1733789729 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efc74104550 0x7efc740ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc78b2c640 1 --2- 172.21.15.76:0/1733789729 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efc74101c20 0x7efc74101590 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc78b2c640 1 -- 172.21.15.76:0/1733789729 >> 172.21.15.76:0/1733789729 conn(0x7efc740fb610 msgr2=0x7efc741024c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc78b2c640 1 -- 172.21.15.76:0/1733789729 shutdown_connections 2024-03-12T05:55:27.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:27.951+0000 7efc78b2c640 1 -- 172.21.15.76:0/1733789729 wait complete. 2024-03-12T05:55:27.966 INFO:teuthology.run_tasks:Running task kclient... 2024-03-12T05:55:27.990 INFO:tasks.kclient:Mounting kernel clients... 2024-03-12T05:55:27.990 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-12T05:55:27.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:27.990 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list 2024-03-12T05:55:28.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:28.006 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link delete ceph-brx 2024-03-12T05:55:28.071 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:28 smithi076 ceph-mon[32095]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T05:55:28.072 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:28 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3419003900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:28.072 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:28 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1911449943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:28.076 INFO:teuthology.orchestra.run.smithi076.stderr:Cannot find device "ceph-brx" 2024-03-12T05:55:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T05:55:28.077 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-12T05:55:28.077 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-12T05:55:28.077 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-12T05:55:28.078 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph fs ls 2024-03-12T05:55:28.346 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:28 smithi082 ceph-mon[36382]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T05:55:28.346 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:28 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3419003900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:28.346 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:28 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1911449943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:28.357 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T05:55:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:28 smithi088 ceph-mon[36585]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T05:55:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:28 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3419003900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:28 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1911449943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:30.047 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:30 smithi076 ceph-mon[32095]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:55:30.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.145+0000 7f755c8a0700 1 -- 172.21.15.76:0/2653509849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7558107030 msgr2=0x7f755810ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:30.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.145+0000 7f755c8a0700 1 --2- 172.21.15.76:0/2653509849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7558107030 0x7f755810ebb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f755000b0a0 tx=0x7f755000b3b0 comp rx=0 tx=0).stop 2024-03-12T05:55:30.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.146+0000 7f755c8a0700 1 -- 172.21.15.76:0/2653509849 shutdown_connections 2024-03-12T05:55:30.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.146+0000 7f755c8a0700 1 --2- 172.21.15.76:0/2653509849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7558107030 0x7f755810ebb0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:30.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.146+0000 7f755c8a0700 1 --2- 172.21.15.76:0/2653509849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7558100810 0x7f7558100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:30.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.146+0000 7f755c8a0700 1 --2- 172.21.15.76:0/2653509849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f75581011c0 0x7f7558106900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:30.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.146+0000 7f755c8a0700 1 -- 172.21.15.76:0/2653509849 >> 172.21.15.76:0/2653509849 conn(0x7f7558075eb0 msgr2=0x7f75580762c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:30.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.146+0000 7f755c8a0700 1 -- 172.21.15.76:0/2653509849 shutdown_connections 2024-03-12T05:55:30.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.146+0000 7f755c8a0700 1 -- 172.21.15.76:0/2653509849 wait complete. 2024-03-12T05:55:30.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.146+0000 7f755c8a0700 1 Processor -- start 2024-03-12T05:55:30.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.146+0000 7f755c8a0700 1 -- start start 2024-03-12T05:55:30.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f755c8a0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7558100810 0x7f75581980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:30.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f755c8a0700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75581011c0 0x7f7558198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:30.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f755c8a0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7558107030 0x7f755819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:30.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f755c8a0700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755819cf70 con 0x7f75581011c0 2024-03-12T05:55:30.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f755c8a0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755819d0e0 con 0x7f7558100810 2024-03-12T05:55:30.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f755c8a0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755819d2a0 con 0x7f7558107030 2024-03-12T05:55:30.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f7555d9b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75581011c0 0x7f7558198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:30.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f7555d9b700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75581011c0 0x7f7558198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55622/0 (socket says 172.21.15.76:55622) 2024-03-12T05:55:30.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f7555d9b700 1 -- 172.21.15.76:0/192587247 learned_addr learned my addr 172.21.15.76:0/192587247 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:30.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f755659c700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7558100810 0x7f75581980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:30.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f7556d9d700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7558107030 0x7f755819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:30.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f7555d9b700 1 -- 172.21.15.76:0/192587247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7558107030 msgr2=0x7f755819c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:30.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f7555d9b700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7558107030 0x7f755819c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:30.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f7555d9b700 1 -- 172.21.15.76:0/192587247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7558100810 msgr2=0x7f75581980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:30.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f7555d9b700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7558100810 0x7f75581980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:30.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f7555d9b700 1 -- 172.21.15.76:0/192587247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7550009d00 con 0x7f75581011c0 2024-03-12T05:55:30.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f7556d9d700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7558107030 0x7f755819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:30.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f755659c700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7558100810 0x7f75581980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:30.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f7555d9b700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75581011c0 0x7f7558198620 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f754c00b4d0 tx=0x7f754c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:30.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f75477fe700 1 -- 172.21.15.76:0/192587247 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c011830 con 0x7f75581011c0 2024-03-12T05:55:30.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f75477fe700 1 -- 172.21.15.76:0/192587247 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f754c011e70 con 0x7f75581011c0 2024-03-12T05:55:30.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f755c8a0700 1 -- 172.21.15.76:0/192587247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f755819d580 con 0x7f75581011c0 2024-03-12T05:55:30.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f75477fe700 1 -- 172.21.15.76:0/192587247 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c00f2e0 con 0x7f75581011c0 2024-03-12T05:55:30.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.147+0000 7f755c8a0700 1 -- 172.21.15.76:0/192587247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75581a4de0 con 0x7f75581011c0 2024-03-12T05:55:30.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.148+0000 7f755c8a0700 1 -- 172.21.15.76:0/192587247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755810a290 con 0x7f75581011c0 2024-03-12T05:55:30.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.149+0000 7f75477fe700 1 -- 172.21.15.76:0/192587247 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f754c0105a0 con 0x7f75581011c0 2024-03-12T05:55:30.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.150+0000 7f75477fe700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75400a1ac0 0x7f75400a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:30.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.150+0000 7f755659c700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75400a1ac0 0x7f75400a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:30.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.150+0000 7f75477fe700 1 -- 172.21.15.76:0/192587247 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f754c0ccf20 con 0x7f75581011c0 2024-03-12T05:55:30.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.150+0000 7f755659c700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75400a1ac0 0x7f75400a3f80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7548005fd0 tx=0x7f7548005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:30.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.154+0000 7f75477fe700 1 -- 172.21.15.76:0/192587247 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f754c096140 con 0x7f75581011c0 2024-03-12T05:55:30.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.396+0000 7f755c8a0700 1 -- 172.21.15.76:0/192587247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7558199450 con 0x7f75581011c0 2024-03-12T05:55:30.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.397+0000 7f75477fe700 1 -- 172.21.15.76:0/192587247 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) v1 ==== 53+0+73 (secure 0 0 0) 0x7f754c015060 con 0x7f75581011c0 2024-03-12T05:55:30.401 INFO:teuthology.orchestra.run.smithi076.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-12T05:55:30.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.399+0000 7f755c8a0700 1 -- 172.21.15.76:0/192587247 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75400a1ac0 msgr2=0x7f75400a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:30.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.399+0000 7f755c8a0700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75400a1ac0 0x7f75400a3f80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7548005fd0 tx=0x7f7548005e30 comp rx=0 tx=0).stop 2024-03-12T05:55:30.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.399+0000 7f755c8a0700 1 -- 172.21.15.76:0/192587247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75581011c0 msgr2=0x7f7558198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:30.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.399+0000 7f755c8a0700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75581011c0 0x7f7558198620 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f754c00b4d0 tx=0x7f754c00b890 comp rx=0 tx=0).stop 2024-03-12T05:55:30.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.399+0000 7f755c8a0700 1 -- 172.21.15.76:0/192587247 shutdown_connections 2024-03-12T05:55:30.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.399+0000 7f755c8a0700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f75400a1ac0 0x7f75400a3f80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:30.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.399+0000 7f755c8a0700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7558107030 0x7f755819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:30.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.399+0000 7f755c8a0700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7558100810 0x7f75581980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:30.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.399+0000 7f755c8a0700 1 --2- 172.21.15.76:0/192587247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75581011c0 0x7f7558198620 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:30.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.399+0000 7f755c8a0700 1 -- 172.21.15.76:0/192587247 >> 172.21.15.76:0/192587247 conn(0x7f7558075eb0 msgr2=0x7f75580fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:30.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.399+0000 7f755c8a0700 1 -- 172.21.15.76:0/192587247 shutdown_connections 2024-03-12T05:55:30.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:30.399+0000 7f755c8a0700 1 -- 172.21.15.76:0/192587247 wait complete. 2024-03-12T05:55:30.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:30 smithi082 ceph-mon[36382]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:55:30.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:30 smithi088 ceph-mon[36585]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:55:31.147 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-12T05:55:31.148 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-12T05:55:31.148 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi076.front.sepia.ceph.com 2024-03-12T05:55:31.148 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-12T05:55:31.148 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-12T05:55:31.148 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-12T05:55:31.148 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-12T05:55:31.148 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-12T05:55:31.148 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-12T05:55:31.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:31.148 DEBUG:teuthology.orchestra.run.smithi076:> ip addr 2024-03-12T05:55:31.165 INFO:teuthology.orchestra.run.smithi076.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-12T05:55:31.165 INFO:teuthology.orchestra.run.smithi076.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-12T05:55:31.165 INFO:teuthology.orchestra.run.smithi076.stdout: inet 127.0.0.1/8 scope host lo 2024-03-12T05:55:31.165 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft forever preferred_lft forever 2024-03-12T05:55:31.165 INFO:teuthology.orchestra.run.smithi076.stdout: inet6 ::1/128 scope host 2024-03-12T05:55:31.165 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft forever preferred_lft forever 2024-03-12T05:55:31.165 INFO:teuthology.orchestra.run.smithi076.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-12T05:55:31.165 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:d8:da:66 brd ff:ff:ff:ff:ff:ff 2024-03-12T05:55:31.165 INFO:teuthology.orchestra.run.smithi076.stdout: altname enp7s0f0 2024-03-12T05:55:31.165 INFO:teuthology.orchestra.run.smithi076.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-12T05:55:31.165 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:d8:da:67 brd ff:ff:ff:ff:ff:ff 2024-03-12T05:55:31.165 INFO:teuthology.orchestra.run.smithi076.stdout: altname enp7s0f1 2024-03-12T05:55:31.166 INFO:teuthology.orchestra.run.smithi076.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-12T05:55:31.166 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:8f:47:e0 brd ff:ff:ff:ff:ff:ff 2024-03-12T05:55:31.166 INFO:teuthology.orchestra.run.smithi076.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-12T05:55:31.166 INFO:teuthology.orchestra.run.smithi076.stdout: link/ether 0c:c4:7a:8f:47:e1 brd ff:ff:ff:ff:ff:ff 2024-03-12T05:55:31.166 INFO:teuthology.orchestra.run.smithi076.stdout: inet 172.21.15.76/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-12T05:55:31.166 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft 41870sec preferred_lft 41870sec 2024-03-12T05:55:31.166 INFO:teuthology.orchestra.run.smithi076.stdout: inet6 fe80::ec4:7aff:fe8f:47e1/64 scope link 2024-03-12T05:55:31.166 INFO:teuthology.orchestra.run.smithi076.stdout: valid_lft forever preferred_lft forever 2024-03-12T05:55:31.166 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-12T05:55:31.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:31.167 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec bash -c ' 2024-03-12T05:55:31.167 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-03-12T05:55:31.167 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link add name ceph-brx type bridge 2024-03-12T05:55:31.167 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip addr flush dev ceph-brx 2024-03-12T05:55:31.167 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link set ceph-brx up 2024-03-12T05:55:31.167 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-12T05:55:31.167 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-03-12T05:55:31.288 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:31 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/192587247' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-12T05:55:31.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:31.332 DEBUG:teuthology.orchestra.run.smithi076:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-12T05:55:31.362 INFO:teuthology.orchestra.run.smithi076.stdout:1 2024-03-12T05:55:31.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:31.362 DEBUG:teuthology.orchestra.run.smithi076:> route 2024-03-12T05:55:31.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:31 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/192587247' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-12T05:55:31.463 INFO:teuthology.orchestra.run.smithi076.stdout:Kernel IP routing table 2024-03-12T05:55:31.463 INFO:teuthology.orchestra.run.smithi076.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-12T05:55:31.463 INFO:teuthology.orchestra.run.smithi076.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-12T05:55:31.463 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-12T05:55:31.463 INFO:teuthology.orchestra.run.smithi076.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-12T05:55:31.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:31.464 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec bash -c ' 2024-03-12T05:55:31.464 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-03-12T05:55:31.464 DEBUG:teuthology.orchestra.run.smithi076:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-12T05:55:31.464 DEBUG:teuthology.orchestra.run.smithi076:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-12T05:55:31.464 DEBUG:teuthology.orchestra.run.smithi076:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-12T05:55:31.464 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-03-12T05:55:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:31 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/192587247' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-12T05:55:31.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:31.673 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list 2024-03-12T05:55:31.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:31.688 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list-id 2024-03-12T05:55:31.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:31.743 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec bash -c ' 2024-03-12T05:55:31.743 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-03-12T05:55:31.743 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-12T05:55:31.743 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-12T05:55:31.743 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-03-12T05:55:31.826 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-12T05:55:31.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:31.827 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec bash -c ' 2024-03-12T05:55:31.827 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-03-12T05:55:31.827 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-12T05:55:31.827 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-12T05:55:31.827 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-12T05:55:31.827 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-12T05:55:31.827 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-12T05:55:31.827 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-03-12T05:55:31.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:31.967 DEBUG:teuthology.orchestra.run.smithi076:> (cd / && exec bash -c ' 2024-03-12T05:55:31.967 DEBUG:teuthology.orchestra.run.smithi076:> set -e 2024-03-12T05:55:31.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link set brx.0 up 2024-03-12T05:55:31.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link set dev brx.0 master ceph-brx 2024-03-12T05:55:31.967 DEBUG:teuthology.orchestra.run.smithi076:> ') 2024-03-12T05:55:32.009 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-12T05:55:32.009 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-12T05:55:32.068 INFO:teuthology.orchestra.run.smithi076.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-12T05:55:32.069 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-12T05:55:32.069 DEBUG:teuthology.orchestra.run.smithi076:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-12T05:55:32.124 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-12T05:55:32.124 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-12T05:55:32.124 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/43d84004-5a04-4acb-a504-caeba624e5ab 2024-03-12T05:55:32.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:32.125 DEBUG:teuthology.orchestra.run.smithi076:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/43d84004-5a04-4acb-a504-caeba624e5ab /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-03-12T05:55:32.188 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:32 smithi076 ceph-mon[32095]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-12T05:55:32.208 INFO:teuthology.orchestra.run.smithi076.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-03-12T05:55:32.209 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync". 2024-03-12T05:55:32.324 INFO:teuthology.orchestra.run.smithi076.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-03-12T05:55:32.324 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync". 2024-03-12T05:55:32.324 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: resolved to: "172.21.15.76:6789,172.21.15.82:6789,172.21.15.88:6789" 2024-03-12T05:55:32.324 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: trying mount with new device syntax: 0@9f336de8-e033-11ee-95c8-87774f69a715.cephfs=/volumes/qa/sv_1/43d84004-5a04-4acb-a504-caeba624e5ab 2024-03-12T05:55:32.324 INFO:teuthology.orchestra.run.smithi076.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync,name=0,key=0,mon_addr=172.21.15.76:6789/172.21.15.82:6789/172.21.15.88:6789" will pass to kernel 2024-03-12T05:55:32.325 INFO:teuthology.orchestra.run.smithi076.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-12T05:55:32.325 INFO:teuthology.orchestra.run.smithi076.stdout: You just mounted a file system that supports labels which does not 2024-03-12T05:55:32.325 INFO:teuthology.orchestra.run.smithi076.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-12T05:55:32.325 INFO:teuthology.orchestra.run.smithi076.stdout: applications will generate AVC messages and not be allowed access to 2024-03-12T05:55:32.325 INFO:teuthology.orchestra.run.smithi076.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-12T05:55:32.326 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-12T05:55:32.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:32.326 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-12T05:55:32.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:32.398 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-12T05:55:32.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:32 smithi082 ceph-mon[36382]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-12T05:55:32.455 INFO:teuthology.orchestra.run.smithi076.stdout:client24878 2024-03-12T05:55:32.455 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:55:32.455 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-12T05:55:32.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:32 smithi088 ceph-mon[36585]: pgmap v343: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-12T05:55:32.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:32.512 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-12T05:55:32.567 INFO:teuthology.orchestra.run.smithi076.stdout:client24878 2024-03-12T05:55:32.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:32.568 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-12T05:55:32.623 INFO:teuthology.orchestra.run.smithi076.stdout:client24878 2024-03-12T05:55:32.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:32.624 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/9f336de8-e033-11ee-95c8-87774f69a715.client24878/status') 2024-03-12T05:55:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:instance: client.24878 (3)172.21.15.76:0/254026946 2024-03-12T05:55:32.691 INFO:teuthology.orchestra.run.smithi076.stdout:blocklisted: false 2024-03-12T05:55:32.692 INFO:teuthology.orchestra.run.smithi076.stderr:0+1 records in 2024-03-12T05:55:32.692 INFO:teuthology.orchestra.run.smithi076.stderr:0+1 records out 2024-03-12T05:55:32.692 INFO:teuthology.orchestra.run.smithi076.stderr:70 bytes copied, 2.1406e-05 s, 3.3 MB/s 2024-03-12T05:55:32.693 INFO:teuthology.run_tasks:Running task exec... 2024-03-12T05:55:32.703 INFO:teuthology.task.exec:Executing custom commands... 2024-03-12T05:55:32.703 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2024-03-12T05:55:32.704 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-12T05:55:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.833+0000 7f2b21815640 1 -- 172.21.15.76:0/4294751254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b1c1023c0 msgr2=0x7f2b1c102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.833+0000 7f2b21815640 1 --2- 172.21.15.76:0/4294751254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b1c1023c0 0x7f2b1c102840 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009900 tx=0x7f2b0402bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:32.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.833+0000 7f2b21815640 1 -- 172.21.15.76:0/4294751254 shutdown_connections 2024-03-12T05:55:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.833+0000 7f2b21815640 1 --2- 172.21.15.76:0/4294751254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b1c102d80 0x7f2b1c109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.833+0000 7f2b21815640 1 --2- 172.21.15.76:0/4294751254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b1c1023c0 0x7f2b1c102840 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.833+0000 7f2b21815640 1 --2- 172.21.15.76:0/4294751254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c1011d0 0x7f2b1c1015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.833+0000 7f2b21815640 1 -- 172.21.15.76:0/4294751254 >> 172.21.15.76:0/4294751254 conn(0x7f2b1c0fc740 msgr2=0x7f2b1c0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.833+0000 7f2b21815640 1 -- 172.21.15.76:0/4294751254 shutdown_connections 2024-03-12T05:55:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.833+0000 7f2b21815640 1 -- 172.21.15.76:0/4294751254 wait complete. 2024-03-12T05:55:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.834+0000 7f2b21815640 1 Processor -- start 2024-03-12T05:55:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.834+0000 7f2b21815640 1 -- start start 2024-03-12T05:55:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.834+0000 7f2b21815640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b1c1011d0 0x7f2b1c197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.834+0000 7f2b21815640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c1023c0 0x7f2b1c1977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.834+0000 7f2b21815640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b1c102d80 0x7f2b1c19e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.834+0000 7f2b21815640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c19ed70 con 0x7f2b1c102d80 2024-03-12T05:55:32.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.834+0000 7f2b21815640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c19eee0 con 0x7f2b1c1011d0 2024-03-12T05:55:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.834+0000 7f2b21815640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c19f0a0 con 0x7f2b1c1023c0 2024-03-12T05:55:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.834+0000 7f2b1b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b1c102d80 0x7f2b1c19e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.834+0000 7f2b1b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b1c102d80 0x7f2b1c19e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47232/0 (socket says 172.21.15.76:47232) 2024-03-12T05:55:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.834+0000 7f2b1b7fe640 1 -- 172.21.15.76:0/2059284615 learned_addr learned my addr 172.21.15.76:0/2059284615 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.834+0000 7f2b1affd640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b1c1011d0 0x7f2b1c197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.835+0000 7f2b13fff640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c1023c0 0x7f2b1c1977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.835+0000 7f2b1b7fe640 1 -- 172.21.15.76:0/2059284615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c1023c0 msgr2=0x7f2b1c1977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.835+0000 7f2b1b7fe640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c1023c0 0x7f2b1c1977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.835+0000 7f2b1b7fe640 1 -- 172.21.15.76:0/2059284615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b1c1011d0 msgr2=0x7f2b1c197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.835+0000 7f2b1b7fe640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b1c1011d0 0x7f2b1c197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:32.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.835+0000 7f2b1b7fe640 1 -- 172.21.15.76:0/2059284615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b04009560 con 0x7f2b1c102d80 2024-03-12T05:55:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.835+0000 7f2b13fff640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c1023c0 0x7f2b1c1977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.835+0000 7f2b1b7fe640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b1c102d80 0x7f2b1c19e830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009fd0 tx=0x7f2b1400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.835+0000 7f2b20813640 1 -- 172.21.15.76:0/2059284615 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1400bea0 con 0x7f2b1c102d80 2024-03-12T05:55:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.835+0000 7f2b21815640 1 -- 172.21.15.76:0/2059284615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1c19f380 con 0x7f2b1c102d80 2024-03-12T05:55:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.835+0000 7f2b20813640 1 -- 172.21.15.76:0/2059284615 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b14009940 con 0x7f2b1c102d80 2024-03-12T05:55:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.835+0000 7f2b20813640 1 -- 172.21.15.76:0/2059284615 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b140106c0 con 0x7f2b1c102d80 2024-03-12T05:55:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.835+0000 7f2b21815640 1 -- 172.21.15.76:0/2059284615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1c19f840 con 0x7f2b1c102d80 2024-03-12T05:55:32.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.836+0000 7f2b21815640 1 -- 172.21.15.76:0/2059284615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1c1085b0 con 0x7f2b1c102d80 2024-03-12T05:55:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.837+0000 7f2b20813640 1 -- 172.21.15.76:0/2059284615 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2b14004240 con 0x7f2b1c102d80 2024-03-12T05:55:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.837+0000 7f2b20813640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2b080a1a20 0x7f2b080a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.838+0000 7f2b1affd640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2b080a1a20 0x7f2b080a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.838+0000 7f2b20813640 1 -- 172.21.15.76:0/2059284615 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2b140cd110 con 0x7f2b1c102d80 2024-03-12T05:55:32.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.838+0000 7f2b1affd640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2b080a1a20 0x7f2b080a3ee0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c0058c0 tx=0x7f2b0c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:32.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:32.842+0000 7f2b20813640 1 -- 172.21.15.76:0/2059284615 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2b14096330 con 0x7f2b1c102d80 2024-03-12T05:55:33.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:33.054+0000 7f2b21815640 1 -- 172.21.15.76:0/2059284615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) v1 -- 0x7f2b1c02ce40 con 0x7f2b1c102d80 2024-03-12T05:55:34.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.020+0000 7f2b20813640 1 -- 172.21.15.76:0/2059284615 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v22) v1 ==== 117+0+0 (secure 0 0 0) 0x7f2b140999f0 con 0x7f2b1c102d80 2024-03-12T05:55:34.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.022+0000 7f2b21815640 1 -- 172.21.15.76:0/2059284615 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2b080a1a20 msgr2=0x7f2b080a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.022+0000 7f2b21815640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2b080a1a20 0x7f2b080a3ee0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c0058c0 tx=0x7f2b0c00a2d0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.023+0000 7f2b21815640 1 -- 172.21.15.76:0/2059284615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b1c102d80 msgr2=0x7f2b1c19e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.023+0000 7f2b21815640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b1c102d80 0x7f2b1c19e830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009fd0 tx=0x7f2b1400b260 comp rx=0 tx=0).stop 2024-03-12T05:55:34.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.023+0000 7f2b21815640 1 -- 172.21.15.76:0/2059284615 shutdown_connections 2024-03-12T05:55:34.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.023+0000 7f2b21815640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2b080a1a20 0x7f2b080a3ee0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.023+0000 7f2b21815640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b1c102d80 0x7f2b1c19e830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.023+0000 7f2b21815640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c1023c0 0x7f2b1c1977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.023+0000 7f2b21815640 1 --2- 172.21.15.76:0/2059284615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2b1c1011d0 0x7f2b1c197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.023+0000 7f2b21815640 1 -- 172.21.15.76:0/2059284615 >> 172.21.15.76:0/2059284615 conn(0x7f2b1c0fc740 msgr2=0x7f2b1c105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:34.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.023+0000 7f2b21815640 1 -- 172.21.15.76:0/2059284615 shutdown_connections 2024-03-12T05:55:34.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.023+0000 7f2b21815640 1 -- 172.21.15.76:0/2059284615 wait complete. 2024-03-12T05:55:34.038 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-12T05:55:34.049 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-12T05:55:34.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:55:34.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.161+0000 7f2984d1e640 1 -- 172.21.15.76:0/1842001134 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f29801000b0 msgr2=0x7f29801004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.161+0000 7f2984d1e640 1 --2- 172.21.15.76:0/1842001134 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f29801000b0 0x7f29801004d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2968009900 tx=0x7f296802bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:34.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.161+0000 7f2984d1e640 1 -- 172.21.15.76:0/1842001134 shutdown_connections 2024-03-12T05:55:34.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.161+0000 7f2984d1e640 1 --2- 172.21.15.76:0/1842001134 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2980101c60 0x7f29801084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.161+0000 7f2984d1e640 1 --2- 172.21.15.76:0/1842001134 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29801012a0 0x7f2980101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.161+0000 7f2984d1e640 1 --2- 172.21.15.76:0/1842001134 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f29801000b0 0x7f29801004d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.161+0000 7f2984d1e640 1 -- 172.21.15.76:0/1842001134 >> 172.21.15.76:0/1842001134 conn(0x7f29800fb640 msgr2=0x7f29800fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:34.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.161+0000 7f2984d1e640 1 -- 172.21.15.76:0/1842001134 shutdown_connections 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.161+0000 7f2984d1e640 1 -- 172.21.15.76:0/1842001134 wait complete. 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f2984d1e640 1 Processor -- start 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f2984d1e640 1 -- start start 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f2984d1e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29801000b0 0x7f29801971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f2984d1e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f29801012a0 0x7f2980197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f2984d1e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2980101c60 0x7f298019e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f2984d1e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298019ecd0 con 0x7f29801012a0 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f2984d1e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298019ee40 con 0x7f2980101c60 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f2984d1e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298019f000 con 0x7f29801000b0 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f297dd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f29801012a0 0x7f2980197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f297dd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f29801012a0 0x7f2980197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47240/0 (socket says 172.21.15.76:47240) 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f297dd9b640 1 -- 172.21.15.76:0/3782174111 learned_addr learned my addr 172.21.15.76:0/3782174111 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f297e59c640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29801000b0 0x7f29801971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f297ed9d640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2980101c60 0x7f298019e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:34.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f297dd9b640 1 -- 172.21.15.76:0/3782174111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29801000b0 msgr2=0x7f29801971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f297dd9b640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29801000b0 0x7f29801971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f297dd9b640 1 -- 172.21.15.76:0/3782174111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2980101c60 msgr2=0x7f298019e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f297dd9b640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2980101c60 0x7f298019e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f297dd9b640 1 -- 172.21.15.76:0/3782174111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2968009560 con 0x7f29801012a0 2024-03-12T05:55:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f297e59c640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29801000b0 0x7f29801971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f297ed9d640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2980101c60 0x7f298019e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f297dd9b640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f29801012a0 0x7f2980197720 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f297000c920 tx=0x7f297000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.162+0000 7f29777fe640 1 -- 172.21.15.76:0/3782174111 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970007c70 con 0x7f29801012a0 2024-03-12T05:55:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.163+0000 7f2984d1e640 1 -- 172.21.15.76:0/3782174111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298019f2e0 con 0x7f29801012a0 2024-03-12T05:55:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.163+0000 7f29777fe640 1 -- 172.21.15.76:0/3782174111 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2970007dd0 con 0x7f29801012a0 2024-03-12T05:55:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.163+0000 7f29777fe640 1 -- 172.21.15.76:0/3782174111 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970018860 con 0x7f29801012a0 2024-03-12T05:55:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.163+0000 7f2984d1e640 1 -- 172.21.15.76:0/3782174111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298019f7f0 con 0x7f29801012a0 2024-03-12T05:55:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.164+0000 7f2984d1e640 1 -- 172.21.15.76:0/3782174111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f298010bac0 con 0x7f29801012a0 2024-03-12T05:55:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.165+0000 7f29777fe640 1 -- 172.21.15.76:0/3782174111 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f297000fac0 con 0x7f29801012a0 2024-03-12T05:55:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.165+0000 7f29777fe640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f296c0a1a30 0x7f296c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.165+0000 7f297e59c640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f296c0a1a30 0x7f296c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.165+0000 7f29777fe640 1 -- 172.21.15.76:0/3782174111 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f29700cd5f0 con 0x7f29801012a0 2024-03-12T05:55:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.166+0000 7f297e59c640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f296c0a1a30 0x7f296c0a3ef0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2968036040 tx=0x7f2968000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:34.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.170+0000 7f29777fe640 1 -- 172.21.15.76:0/3782174111 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2970096810 con 0x7f29801012a0 2024-03-12T05:55:34.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.343+0000 7f2984d1e640 1 -- 172.21.15.76:0/3782174111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f298010bc50 con 0x7f29801012a0 2024-03-12T05:55:34.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.344+0000 7f29777fe640 1 -- 172.21.15.76:0/3782174111 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22541 (secure 0 0 0) 0x7f2970099ed0 con 0x7f29801012a0 2024-03-12T05:55:34.348 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:34.348 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:54:42.007701+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:55:34.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.346+0000 7f2984d1e640 1 -- 172.21.15.76:0/3782174111 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f296c0a1a30 msgr2=0x7f296c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.346+0000 7f2984d1e640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f296c0a1a30 0x7f296c0a3ef0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2968036040 tx=0x7f2968000bc0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.346+0000 7f2984d1e640 1 -- 172.21.15.76:0/3782174111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f29801012a0 msgr2=0x7f2980197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.346+0000 7f2984d1e640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f29801012a0 0x7f2980197720 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f297000c920 tx=0x7f297000cde0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.346+0000 7f2984d1e640 1 -- 172.21.15.76:0/3782174111 shutdown_connections 2024-03-12T05:55:34.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.346+0000 7f2984d1e640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f296c0a1a30 0x7f296c0a3ef0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.346+0000 7f2984d1e640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2980101c60 0x7f298019e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.346+0000 7f2984d1e640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f29801012a0 0x7f2980197720 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.346+0000 7f2984d1e640 1 --2- 172.21.15.76:0/3782174111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29801000b0 0x7f29801971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.346+0000 7f2984d1e640 1 -- 172.21.15.76:0/3782174111 >> 172.21.15.76:0/3782174111 conn(0x7f29800fb640 msgr2=0x7f29800fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:34.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.346+0000 7f2984d1e640 1 -- 172.21.15.76:0/3782174111 shutdown_connections 2024-03-12T05:55:34.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.346+0000 7f2984d1e640 1 -- 172.21.15.76:0/3782174111 wait complete. 2024-03-12T05:55:34.360 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:34 smithi076 ceph-mon[32095]: pgmap v344: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T05:55:34.360 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:34 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2059284615' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-12T05:55:34.362 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:49:09.125536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:53:35.663129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:53:36.740826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:55:34.362 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:55:34.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:34 smithi082 ceph-mon[36382]: pgmap v344: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T05:55:34.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:34 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2059284615' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-12T05:55:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 -- 172.21.15.76:0/2211209469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc024101290 msgr2=0x7fc024101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 --2- 172.21.15.76:0/2211209469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc024101290 0x7fc024101710 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc010009900 tx=0x7fc01002bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 -- 172.21.15.76:0/2211209469 shutdown_connections 2024-03-12T05:55:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 --2- 172.21.15.76:0/2211209469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc024101c50 0x7fc0241084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 --2- 172.21.15.76:0/2211209469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc024101290 0x7fc024101710 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 --2- 172.21.15.76:0/2211209469 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0241000a0 0x7fc0241004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 -- 172.21.15.76:0/2211209469 >> 172.21.15.76:0/2211209469 conn(0x7fc0240fb610 msgr2=0x7fc0240fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 -- 172.21.15.76:0/2211209469 shutdown_connections 2024-03-12T05:55:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 -- 172.21.15.76:0/2211209469 wait complete. 2024-03-12T05:55:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 Processor -- start 2024-03-12T05:55:34.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 -- start start 2024-03-12T05:55:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0241000a0 0x7fc024196120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc024101290 0x7fc024196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc024101c50 0x7fc02419d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02419dc10 con 0x7fc024101290 2024-03-12T05:55:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02419dd80 con 0x7fc0241000a0 2024-03-12T05:55:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.464+0000 7fc02aa64640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02419df40 con 0x7fc024101c50 2024-03-12T05:55:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc023fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc024101290 0x7fc024196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc023fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc024101290 0x7fc024196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47262/0 (socket says 172.21.15.76:47262) 2024-03-12T05:55:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc023fff640 1 -- 172.21.15.76:0/286202481 learned_addr learned my addr 172.21.15.76:0/286202481 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc029001640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc024101c50 0x7fc02419d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc028800640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0241000a0 0x7fc024196120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:34.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc023fff640 1 -- 172.21.15.76:0/286202481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc024101c50 msgr2=0x7fc02419d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc023fff640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc024101c50 0x7fc02419d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc023fff640 1 -- 172.21.15.76:0/286202481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0241000a0 msgr2=0x7fc024196120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc023fff640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0241000a0 0x7fc024196120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc023fff640 1 -- 172.21.15.76:0/286202481 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc010009560 con 0x7fc024101290 2024-03-12T05:55:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc029001640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc024101c50 0x7fc02419d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc028800640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0241000a0 0x7fc024196120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc023fff640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc024101290 0x7fc024196660 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc010006010 tx=0x7fc0100042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc021ffb640 1 -- 172.21.15.76:0/286202481 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0100044e0 con 0x7fc024101290 2024-03-12T05:55:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc02aa64640 1 -- 172.21.15.76:0/286202481 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02419e1c0 con 0x7fc024101290 2024-03-12T05:55:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc021ffb640 1 -- 172.21.15.76:0/286202481 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0100054e0 con 0x7fc024101290 2024-03-12T05:55:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc021ffb640 1 -- 172.21.15.76:0/286202481 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01003e450 con 0x7fc024101290 2024-03-12T05:55:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.465+0000 7fc02aa64640 1 -- 172.21.15.76:0/286202481 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02419e6d0 con 0x7fc024101290 2024-03-12T05:55:34.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.466+0000 7fc02aa64640 1 -- 172.21.15.76:0/286202481 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc024107440 con 0x7fc024101290 2024-03-12T05:55:34.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.467+0000 7fc021ffb640 1 -- 172.21.15.76:0/286202481 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fc0100035f0 con 0x7fc024101290 2024-03-12T05:55:34.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.468+0000 7fc021ffb640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc0140a1a30 0x7fc0140a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:34.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.468+0000 7fc028800640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc0140a1a30 0x7fc0140a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:34.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.468+0000 7fc021ffb640 1 -- 172.21.15.76:0/286202481 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc0100ef8a0 con 0x7fc024101290 2024-03-12T05:55:34.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.468+0000 7fc028800640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc0140a1a30 0x7fc0140a3ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc0241010f0 tx=0x7fc018005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:34.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.472+0000 7fc021ffb640 1 -- 172.21.15.76:0/286202481 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc0100b8ac0 con 0x7fc024101290 2024-03-12T05:55:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:34 smithi088 ceph-mon[36585]: pgmap v344: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T05:55:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:34 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2059284615' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-12T05:55:34.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.645+0000 7fc02aa64640 1 -- 172.21.15.76:0/286202481 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc02402ce40 con 0x7fc024101290 2024-03-12T05:55:34.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.645+0000 7fc021ffb640 1 -- 172.21.15.76:0/286202481 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc0100bc180 con 0x7fc024101290 2024-03-12T05:55:34.649 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-12T05:55:34.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.647+0000 7fc02aa64640 1 -- 172.21.15.76:0/286202481 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc0140a1a30 msgr2=0x7fc0140a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.647+0000 7fc02aa64640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc0140a1a30 0x7fc0140a3ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc0241010f0 tx=0x7fc018005e70 comp rx=0 tx=0).stop 2024-03-12T05:55:34.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.647+0000 7fc02aa64640 1 -- 172.21.15.76:0/286202481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc024101290 msgr2=0x7fc024196660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.647+0000 7fc02aa64640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc024101290 0x7fc024196660 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc010006010 tx=0x7fc0100042f0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.647+0000 7fc02aa64640 1 -- 172.21.15.76:0/286202481 shutdown_connections 2024-03-12T05:55:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.647+0000 7fc02aa64640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fc0140a1a30 0x7fc0140a3ef0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.647+0000 7fc02aa64640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc024101c50 0x7fc02419d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.647+0000 7fc02aa64640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc024101290 0x7fc024196660 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.647+0000 7fc02aa64640 1 --2- 172.21.15.76:0/286202481 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0241000a0 0x7fc024196120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.647+0000 7fc02aa64640 1 -- 172.21.15.76:0/286202481 >> 172.21.15.76:0/286202481 conn(0x7fc0240fb610 msgr2=0x7fc024104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.648+0000 7fc02aa64640 1 -- 172.21.15.76:0/286202481 shutdown_connections 2024-03-12T05:55:34.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.648+0000 7fc02aa64640 1 -- 172.21.15.76:0/286202481 wait complete. 2024-03-12T05:55:34.662 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:55:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.764+0000 7f5af0e55640 1 -- 172.21.15.76:0/435991973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aec1011d0 msgr2=0x7f5aec1015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.764+0000 7f5af0e55640 1 --2- 172.21.15.76:0/435991973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aec1011d0 0x7f5aec1015f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009ca0 tx=0x7f5adc02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 -- 172.21.15.76:0/435991973 shutdown_connections 2024-03-12T05:55:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 --2- 172.21.15.76:0/435991973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5aec102d80 0x7f5aec109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 --2- 172.21.15.76:0/435991973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5aec1023c0 0x7f5aec102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 --2- 172.21.15.76:0/435991973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aec1011d0 0x7f5aec1015f0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 -- 172.21.15.76:0/435991973 >> 172.21.15.76:0/435991973 conn(0x7f5aec0fc740 msgr2=0x7f5aec0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 -- 172.21.15.76:0/435991973 shutdown_connections 2024-03-12T05:55:34.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 -- 172.21.15.76:0/435991973 wait complete. 2024-03-12T05:55:34.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 Processor -- start 2024-03-12T05:55:34.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 -- start start 2024-03-12T05:55:34.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5aec1011d0 0x7f5aec1971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:34.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aec1023c0 0x7f5aec197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:34.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5aec102d80 0x7f5aec19e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:34.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec19ecd0 con 0x7f5aec1023c0 2024-03-12T05:55:34.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec19ee40 con 0x7f5aec1011d0 2024-03-12T05:55:34.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5af0e55640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec19f000 con 0x7f5aec102d80 2024-03-12T05:55:34.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.765+0000 7f5ae9d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aec1023c0 0x7f5aec197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:34.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5ae9d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aec1023c0 0x7f5aec197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47276/0 (socket says 172.21.15.76:47276) 2024-03-12T05:55:34.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5ae9d9b640 1 -- 172.21.15.76:0/135542702 learned_addr learned my addr 172.21.15.76:0/135542702 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:34.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5aead9d640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5aec102d80 0x7f5aec19e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:34.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5aea59c640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5aec1011d0 0x7f5aec1971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:34.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5ae9d9b640 1 -- 172.21.15.76:0/135542702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5aec102d80 msgr2=0x7f5aec19e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5ae9d9b640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5aec102d80 0x7f5aec19e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5ae9d9b640 1 -- 172.21.15.76:0/135542702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5aec1011d0 msgr2=0x7f5aec1971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5ae9d9b640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5aec1011d0 0x7f5aec1971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5ae9d9b640 1 -- 172.21.15.76:0/135542702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc009900 con 0x7f5aec1023c0 2024-03-12T05:55:34.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5aead9d640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5aec102d80 0x7f5aec19e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:34.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5ae9d9b640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aec1023c0 0x7f5aec197720 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5ae000c920 tx=0x7f5ae000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:34.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5adb7fe640 1 -- 172.21.15.76:0/135542702 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0007c70 con 0x7f5aec1023c0 2024-03-12T05:55:34.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5af0e55640 1 -- 172.21.15.76:0/135542702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec19f2e0 con 0x7f5aec1023c0 2024-03-12T05:55:34.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5adb7fe640 1 -- 172.21.15.76:0/135542702 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ae0007dd0 con 0x7f5aec1023c0 2024-03-12T05:55:34.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5adb7fe640 1 -- 172.21.15.76:0/135542702 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0018860 con 0x7f5aec1023c0 2024-03-12T05:55:34.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.766+0000 7f5af0e55640 1 -- 172.21.15.76:0/135542702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec19f7f0 con 0x7f5aec1023c0 2024-03-12T05:55:34.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.767+0000 7f5af0e55640 1 -- 172.21.15.76:0/135542702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aec108490 con 0x7f5aec1023c0 2024-03-12T05:55:34.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.768+0000 7f5adb7fe640 1 -- 172.21.15.76:0/135542702 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5ae000fac0 con 0x7f5aec1023c0 2024-03-12T05:55:34.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.769+0000 7f5adb7fe640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5ad40a1a30 0x7f5ad40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:34.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.769+0000 7f5aea59c640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5ad40a1a30 0x7f5ad40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:34.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.769+0000 7f5adb7fe640 1 -- 172.21.15.76:0/135542702 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5ae00cd5d0 con 0x7f5aec1023c0 2024-03-12T05:55:34.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.769+0000 7f5aea59c640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5ad40a1a30 0x7f5ad40a3ef0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5adc02c700 tx=0x7f5adc009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:34.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.773+0000 7f5adb7fe640 1 -- 172.21.15.76:0/135542702 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5ae00967f0 con 0x7f5aec1023c0 2024-03-12T05:55:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.946+0000 7f5af0e55640 1 -- 172.21.15.76:0/135542702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5aec02ce40 con 0x7f5aec1023c0 2024-03-12T05:55:34.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.946+0000 7f5adb7fe640 1 -- 172.21.15.76:0/135542702 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5ae0099eb0 con 0x7f5aec1023c0 2024-03-12T05:55:34.950 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-12T05:55:34.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.948+0000 7f5af0e55640 1 -- 172.21.15.76:0/135542702 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5ad40a1a30 msgr2=0x7f5ad40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.948+0000 7f5af0e55640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5ad40a1a30 0x7f5ad40a3ef0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5adc02c700 tx=0x7f5adc009800 comp rx=0 tx=0).stop 2024-03-12T05:55:34.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.948+0000 7f5af0e55640 1 -- 172.21.15.76:0/135542702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aec1023c0 msgr2=0x7f5aec197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:34.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.948+0000 7f5af0e55640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aec1023c0 0x7f5aec197720 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5ae000c920 tx=0x7f5ae000cde0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.948+0000 7f5af0e55640 1 -- 172.21.15.76:0/135542702 shutdown_connections 2024-03-12T05:55:34.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.948+0000 7f5af0e55640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5ad40a1a30 0x7f5ad40a3ef0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.948+0000 7f5af0e55640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5aec102d80 0x7f5aec19e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.948+0000 7f5af0e55640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aec1023c0 0x7f5aec197720 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.948+0000 7f5af0e55640 1 --2- 172.21.15.76:0/135542702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5aec1011d0 0x7f5aec1971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:34.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.948+0000 7f5af0e55640 1 -- 172.21.15.76:0/135542702 >> 172.21.15.76:0/135542702 conn(0x7f5aec0fc740 msgr2=0x7f5aec105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:34.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.948+0000 7f5af0e55640 1 -- 172.21.15.76:0/135542702 shutdown_connections 2024-03-12T05:55:34.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:34.948+0000 7f5af0e55640 1 -- 172.21.15.76:0/135542702 wait complete. 2024-03-12T05:55:34.963 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:55:35.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.066+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2767186292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f28101290 msgr2=0x7f0f28101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.066+0000 7f0f2cf4f640 1 --2- 172.21.15.76:0/2767186292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f28101290 0x7f0f28101710 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c009900 tx=0x7f0f1c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.066+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2767186292 shutdown_connections 2024-03-12T05:55:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.066+0000 7f0f2cf4f640 1 --2- 172.21.15.76:0/2767186292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f28101c50 0x7f0f281084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.066+0000 7f0f2cf4f640 1 --2- 172.21.15.76:0/2767186292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f28101290 0x7f0f28101710 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.066+0000 7f0f2cf4f640 1 --2- 172.21.15.76:0/2767186292 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f281000a0 0x7f0f281004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.066+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2767186292 >> 172.21.15.76:0/2767186292 conn(0x7f0f280fb610 msgr2=0x7f0f280fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.066+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2767186292 shutdown_connections 2024-03-12T05:55:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.066+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2767186292 wait complete. 2024-03-12T05:55:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.066+0000 7f0f2cf4f640 1 Processor -- start 2024-03-12T05:55:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.066+0000 7f0f2cf4f640 1 -- start start 2024-03-12T05:55:35.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f2cf4f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f281000a0 0x7f0f28197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f2cf4f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f28101290 0x7f0f28197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f2cf4f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f28101c50 0x7f0f2819e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f2cf4f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2819ed30 con 0x7f0f28101c50 2024-03-12T05:55:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f2cf4f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2819eea0 con 0x7f0f281000a0 2024-03-12T05:55:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f2cf4f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2819f060 con 0x7f0f28101290 2024-03-12T05:55:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f26d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f28101c50 0x7f0f2819e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f26d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f28101c50 0x7f0f2819e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47288/0 (socket says 172.21.15.76:47288) 2024-03-12T05:55:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f26d9d640 1 -- 172.21.15.76:0/2543234990 learned_addr learned my addr 172.21.15.76:0/2543234990 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f2659c640 1 --2- 172.21.15.76:0/2543234990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f281000a0 0x7f0f28197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f25d9b640 1 --2- 172.21.15.76:0/2543234990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f28101290 0x7f0f28197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f26d9d640 1 -- 172.21.15.76:0/2543234990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f28101290 msgr2=0x7f0f28197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f26d9d640 1 --2- 172.21.15.76:0/2543234990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f28101290 0x7f0f28197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f26d9d640 1 -- 172.21.15.76:0/2543234990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f281000a0 msgr2=0x7f0f28197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f26d9d640 1 --2- 172.21.15.76:0/2543234990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f281000a0 0x7f0f28197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f26d9d640 1 -- 172.21.15.76:0/2543234990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c009560 con 0x7f0f28101c50 2024-03-12T05:55:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f26d9d640 1 --2- 172.21.15.76:0/2543234990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f28101c50 0x7f0f2819e7f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000cad0 tx=0x7f0f2000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f177fe640 1 -- 172.21.15.76:0/2543234990 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2000ae20 con 0x7f0f28101c50 2024-03-12T05:55:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f177fe640 1 -- 172.21.15.76:0/2543234990 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f20005020 con 0x7f0f28101c50 2024-03-12T05:55:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2543234990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2819f340 con 0x7f0f28101c50 2024-03-12T05:55:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f177fe640 1 -- 172.21.15.76:0/2543234990 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2000f6b0 con 0x7f0f28101c50 2024-03-12T05:55:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.067+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2543234990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2819f850 con 0x7f0f28101c50 2024-03-12T05:55:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.068+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2543234990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f28107430 con 0x7f0f28101c50 2024-03-12T05:55:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.070+0000 7f0f177fe640 1 -- 172.21.15.76:0/2543234990 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f0f20005520 con 0x7f0f28101c50 2024-03-12T05:55:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.070+0000 7f0f177fe640 1 --2- 172.21.15.76:0/2543234990 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0f100a1a30 0x7f0f100a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.070+0000 7f0f177fe640 1 -- 172.21.15.76:0/2543234990 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f0f200cd770 con 0x7f0f28101c50 2024-03-12T05:55:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.070+0000 7f0f2659c640 1 --2- 172.21.15.76:0/2543234990 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0f100a1a30 0x7f0f100a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.071+0000 7f0f2659c640 1 --2- 172.21.15.76:0/2543234990 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0f100a1a30 0x7f0f100a3ef0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0f281010f0 tx=0x7f0f1800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:35.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.073+0000 7f0f177fe640 1 -- 172.21.15.76:0/2543234990 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0f20096990 con 0x7f0f28101c50 2024-03-12T05:55:35.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:35 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2059284615' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-12T05:55:35.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:35 smithi076 ceph-mon[32095]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:55:35.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:35 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3782174111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:35.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:35 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/286202481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:35.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:35 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/135542702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:35.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.247+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2543234990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0f2802ce40 con 0x7f0f28101c50 2024-03-12T05:55:35.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.248+0000 7f0f177fe640 1 -- 172.21.15.76:0/2543234990 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0f200b8e60 con 0x7f0f28101c50 2024-03-12T05:55:35.251 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-12T05:55:35.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.250+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2543234990 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0f100a1a30 msgr2=0x7f0f100a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.250+0000 7f0f2cf4f640 1 --2- 172.21.15.76:0/2543234990 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0f100a1a30 0x7f0f100a3ef0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0f281010f0 tx=0x7f0f1800b390 comp rx=0 tx=0).stop 2024-03-12T05:55:35.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.250+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2543234990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f28101c50 msgr2=0x7f0f2819e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.250+0000 7f0f2cf4f640 1 --2- 172.21.15.76:0/2543234990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f28101c50 0x7f0f2819e7f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000cad0 tx=0x7f0f2000cf90 comp rx=0 tx=0).stop 2024-03-12T05:55:35.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.250+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2543234990 shutdown_connections 2024-03-12T05:55:35.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.250+0000 7f0f2cf4f640 1 --2- 172.21.15.76:0/2543234990 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f0f100a1a30 0x7f0f100a3ef0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.250+0000 7f0f2cf4f640 1 --2- 172.21.15.76:0/2543234990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f28101c50 0x7f0f2819e7f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.250+0000 7f0f2cf4f640 1 --2- 172.21.15.76:0/2543234990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f28101290 0x7f0f28197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.250+0000 7f0f2cf4f640 1 --2- 172.21.15.76:0/2543234990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0f281000a0 0x7f0f28197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.250+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2543234990 >> 172.21.15.76:0/2543234990 conn(0x7f0f280fb610 msgr2=0x7f0f28104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:35.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.250+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2543234990 shutdown_connections 2024-03-12T05:55:35.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.250+0000 7f0f2cf4f640 1 -- 172.21.15.76:0/2543234990 wait complete. 2024-03-12T05:55:35.264 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:55:35.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.363+0000 7faa3384e640 1 -- 172.21.15.76:0/3015551610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa2c100d60 msgr2=0x7faa2c108600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.363+0000 7faa3384e640 1 --2- 172.21.15.76:0/3015551610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa2c100d60 0x7faa2c108600 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7faa24009ca0 tx=0x7faa2402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.363+0000 7faa3384e640 1 -- 172.21.15.76:0/3015551610 shutdown_connections 2024-03-12T05:55:35.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.363+0000 7faa3384e640 1 --2- 172.21.15.76:0/3015551610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa2c100d60 0x7faa2c108600 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.363+0000 7faa3384e640 1 --2- 172.21.15.76:0/3015551610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa2c1003a0 0x7faa2c100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.363+0000 7faa3384e640 1 --2- 172.21.15.76:0/3015551610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa2c101750 0x7faa2c101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.363+0000 7faa3384e640 1 -- 172.21.15.76:0/3015551610 >> 172.21.15.76:0/3015551610 conn(0x7faa2c0fae30 msgr2=0x7faa2c0fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:35.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.363+0000 7faa3384e640 1 -- 172.21.15.76:0/3015551610 shutdown_connections 2024-03-12T05:55:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.363+0000 7faa3384e640 1 -- 172.21.15.76:0/3015551610 wait complete. 2024-03-12T05:55:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa3384e640 1 Processor -- start 2024-03-12T05:55:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa3384e640 1 -- start start 2024-03-12T05:55:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa3384e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa2c1003a0 0x7faa2c197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa3384e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa2c100d60 0x7faa2c1976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa3384e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa2c101750 0x7faa2c19e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa3384e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c19ec80 con 0x7faa2c100d60 2024-03-12T05:55:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa3384e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c19edf0 con 0x7faa2c101750 2024-03-12T05:55:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa3384e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c19efb0 con 0x7faa2c1003a0 2024-03-12T05:55:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa30de9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa2c100d60 0x7faa2c1976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa30de9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa2c100d60 0x7faa2c1976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47296/0 (socket says 172.21.15.76:47296) 2024-03-12T05:55:35.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa30de9640 1 -- 172.21.15.76:0/3592910906 learned_addr learned my addr 172.21.15.76:0/3592910906 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa31deb640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa2c101750 0x7faa2c19e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa315ea640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa2c1003a0 0x7faa2c197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa30de9640 1 -- 172.21.15.76:0/3592910906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa2c1003a0 msgr2=0x7faa2c197190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa30de9640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa2c1003a0 0x7faa2c197190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa30de9640 1 -- 172.21.15.76:0/3592910906 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa2c101750 msgr2=0x7faa2c19e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa30de9640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa2c101750 0x7faa2c19e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa30de9640 1 -- 172.21.15.76:0/3592910906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa24009900 con 0x7faa2c100d60 2024-03-12T05:55:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa31deb640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa2c101750 0x7faa2c19e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa30de9640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa2c100d60 0x7faa2c1976d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7faa2800d770 tx=0x7faa2800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.364+0000 7faa227fc640 1 -- 172.21.15.76:0/3592910906 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa28009c30 con 0x7faa2c100d60 2024-03-12T05:55:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.365+0000 7faa227fc640 1 -- 172.21.15.76:0/3592910906 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa2800de80 con 0x7faa2c100d60 2024-03-12T05:55:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.365+0000 7faa3384e640 1 -- 172.21.15.76:0/3592910906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa2c19f290 con 0x7faa2c100d60 2024-03-12T05:55:35.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.365+0000 7faa227fc640 1 -- 172.21.15.76:0/3592910906 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa28010460 con 0x7faa2c100d60 2024-03-12T05:55:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.365+0000 7faa3384e640 1 -- 172.21.15.76:0/3592910906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2c19f7a0 con 0x7faa2c100d60 2024-03-12T05:55:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.366+0000 7faa3384e640 1 -- 172.21.15.76:0/3592910906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa2c107480 con 0x7faa2c100d60 2024-03-12T05:55:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.367+0000 7faa227fc640 1 -- 172.21.15.76:0/3592910906 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7faa28003f60 con 0x7faa2c100d60 2024-03-12T05:55:35.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.367+0000 7faa227fc640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faa180a1a30 0x7faa180a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.367+0000 7faa315ea640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faa180a1a30 0x7faa180a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.367+0000 7faa227fc640 1 -- 172.21.15.76:0/3592910906 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7faa280cd1e0 con 0x7faa2c100d60 2024-03-12T05:55:35.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.368+0000 7faa315ea640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faa180a1a30 0x7faa180a3ef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7faa1c005fd0 tx=0x7faa1c005ce0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:35.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.371+0000 7faa227fc640 1 -- 172.21.15.76:0/3592910906 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faa28096400 con 0x7faa2c100d60 2024-03-12T05:55:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:35 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2059284615' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-12T05:55:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:35 smithi082 ceph-mon[36382]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:55:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:35 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3782174111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:35 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/286202481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:35 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/135542702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:35.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:35 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2059284615' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-12T05:55:35.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:35 smithi088 ceph-mon[36585]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:55:35.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:35 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3782174111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:35.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:35 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/286202481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:35.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:35 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/135542702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.544+0000 7faa3384e640 1 -- 172.21.15.76:0/3592910906 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa2c10cc40 con 0x7faa2c100d60 2024-03-12T05:55:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.545+0000 7faa227fc640 1 -- 172.21.15.76:0/3592910906 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22541 (secure 0 0 0) 0x7faa28017020 con 0x7faa2c100d60 2024-03-12T05:55:35.549 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:35.549 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:54:42.007701+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:55:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.547+0000 7faa3384e640 1 -- 172.21.15.76:0/3592910906 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faa180a1a30 msgr2=0x7faa180a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.547+0000 7faa3384e640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faa180a1a30 0x7faa180a3ef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7faa1c005fd0 tx=0x7faa1c005ce0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.547+0000 7faa3384e640 1 -- 172.21.15.76:0/3592910906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa2c100d60 msgr2=0x7faa2c1976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.547+0000 7faa3384e640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa2c100d60 0x7faa2c1976d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7faa2800d770 tx=0x7faa2800dc30 comp rx=0 tx=0).stop 2024-03-12T05:55:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.547+0000 7faa3384e640 1 -- 172.21.15.76:0/3592910906 shutdown_connections 2024-03-12T05:55:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.547+0000 7faa3384e640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7faa180a1a30 0x7faa180a3ef0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.547+0000 7faa3384e640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa2c101750 0x7faa2c19e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.547+0000 7faa3384e640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa2c100d60 0x7faa2c1976d0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.547+0000 7faa3384e640 1 --2- 172.21.15.76:0/3592910906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa2c1003a0 0x7faa2c197190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.547+0000 7faa3384e640 1 -- 172.21.15.76:0/3592910906 >> 172.21.15.76:0/3592910906 conn(0x7faa2c0fae30 msgr2=0x7faa2c104650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.547+0000 7faa3384e640 1 -- 172.21.15.76:0/3592910906 shutdown_connections 2024-03-12T05:55:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.547+0000 7faa3384e640 1 -- 172.21.15.76:0/3592910906 wait complete. 2024-03-12T05:55:35.562 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:49:09.125536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:53:35.663129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:53:36.740826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:55:35.563 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:55:35.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.664+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1285593667 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a10101190 msgr2=0x7f4a101015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.664+0000 7f4a15ca7640 1 --2- 172.21.15.76:0/1285593667 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a10101190 0x7f4a101015b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009900 tx=0x7f4a0002bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.664+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1285593667 shutdown_connections 2024-03-12T05:55:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.664+0000 7f4a15ca7640 1 --2- 172.21.15.76:0/1285593667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a10102d40 0x7f4a101095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.664+0000 7f4a15ca7640 1 --2- 172.21.15.76:0/1285593667 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a10102380 0x7f4a10102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.664+0000 7f4a15ca7640 1 --2- 172.21.15.76:0/1285593667 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a10101190 0x7f4a101015b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.664+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1285593667 >> 172.21.15.76:0/1285593667 conn(0x7f4a100fc700 msgr2=0x7f4a100feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.664+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1285593667 shutdown_connections 2024-03-12T05:55:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.664+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1285593667 wait complete. 2024-03-12T05:55:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.664+0000 7f4a15ca7640 1 Processor -- start 2024-03-12T05:55:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a15ca7640 1 -- start start 2024-03-12T05:55:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a15ca7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a10101190 0x7f4a10199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a15ca7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a10102380 0x7f4a101999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a15ca7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a10102d40 0x7f4a101a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a15ca7640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a101a0f50 con 0x7f4a10102d40 2024-03-12T05:55:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a15ca7640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a101a10c0 con 0x7f4a10102380 2024-03-12T05:55:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a15ca7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a101a1280 con 0x7f4a10101190 2024-03-12T05:55:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a0ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a10102d40 0x7f4a101a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a0ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a10102d40 0x7f4a101a0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47312/0 (socket says 172.21.15.76:47312) 2024-03-12T05:55:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a0ffff640 1 -- 172.21.15.76:0/1355812568 learned_addr learned my addr 172.21.15.76:0/1355812568 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a0effd640 1 --2- 172.21.15.76:0/1355812568 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a10102380 0x7f4a101999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a0f7fe640 1 --2- 172.21.15.76:0/1355812568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a10101190 0x7f4a10199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a0ffff640 1 -- 172.21.15.76:0/1355812568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a10101190 msgr2=0x7f4a10199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a0ffff640 1 --2- 172.21.15.76:0/1355812568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a10101190 0x7f4a10199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a0ffff640 1 -- 172.21.15.76:0/1355812568 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a10102380 msgr2=0x7f4a101999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a0ffff640 1 --2- 172.21.15.76:0/1355812568 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a10102380 0x7f4a101999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a0ffff640 1 -- 172.21.15.76:0/1355812568 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a00009560 con 0x7f4a10102d40 2024-03-12T05:55:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.665+0000 7f4a0ffff640 1 --2- 172.21.15.76:0/1355812568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a10102d40 0x7f4a101a0a10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4a0800eaa0 tx=0x7f4a0800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.666+0000 7f4a0cff9640 1 -- 172.21.15.76:0/1355812568 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0800cdb0 con 0x7f4a10102d40 2024-03-12T05:55:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.666+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1355812568 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a101a1560 con 0x7f4a10102d40 2024-03-12T05:55:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.666+0000 7f4a0cff9640 1 -- 172.21.15.76:0/1355812568 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a08005020 con 0x7f4a10102d40 2024-03-12T05:55:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.666+0000 7f4a0cff9640 1 -- 172.21.15.76:0/1355812568 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a080116b0 con 0x7f4a10102d40 2024-03-12T05:55:35.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.666+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1355812568 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a101a1a20 con 0x7f4a10102d40 2024-03-12T05:55:35.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.667+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1355812568 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a1010cba0 con 0x7f4a10102d40 2024-03-12T05:55:35.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.668+0000 7f4a0cff9640 1 -- 172.21.15.76:0/1355812568 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f4a08005520 con 0x7f4a10102d40 2024-03-12T05:55:35.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.668+0000 7f4a0cff9640 1 --2- 172.21.15.76:0/1355812568 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f49f80a1a30 0x7f49f80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.668+0000 7f4a0cff9640 1 -- 172.21.15.76:0/1355812568 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4a080cd980 con 0x7f4a10102d40 2024-03-12T05:55:35.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.668+0000 7f4a0f7fe640 1 --2- 172.21.15.76:0/1355812568 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f49f80a1a30 0x7f49f80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.669+0000 7f4a0f7fe640 1 --2- 172.21.15.76:0/1355812568 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f49f80a1a30 0x7f49f80a3ef0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4a00005ab0 tx=0x7f4a00005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:35.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.672+0000 7f4a0cff9640 1 -- 172.21.15.76:0/1355812568 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4a08096c50 con 0x7f4a10102d40 2024-03-12T05:55:35.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.842+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1355812568 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4a101a1e60 con 0x7f4a10102d40 2024-03-12T05:55:35.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.842+0000 7f4a0cff9640 1 -- 172.21.15.76:0/1355812568 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4a0801f020 con 0x7f4a10102d40 2024-03-12T05:55:35.846 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-12T05:55:35.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.844+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1355812568 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f49f80a1a30 msgr2=0x7f49f80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.844+0000 7f4a15ca7640 1 --2- 172.21.15.76:0/1355812568 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f49f80a1a30 0x7f49f80a3ef0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4a00005ab0 tx=0x7f4a00005fd0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.844+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1355812568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a10102d40 msgr2=0x7f4a101a0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.844+0000 7f4a15ca7640 1 --2- 172.21.15.76:0/1355812568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a10102d40 0x7f4a101a0a10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4a0800eaa0 tx=0x7f4a0800ef60 comp rx=0 tx=0).stop 2024-03-12T05:55:35.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.844+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1355812568 shutdown_connections 2024-03-12T05:55:35.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.844+0000 7f4a15ca7640 1 --2- 172.21.15.76:0/1355812568 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f49f80a1a30 0x7f49f80a3ef0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.844+0000 7f4a15ca7640 1 --2- 172.21.15.76:0/1355812568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a10102d40 0x7f4a101a0a10 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.844+0000 7f4a15ca7640 1 --2- 172.21.15.76:0/1355812568 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a10102380 0x7f4a101999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.844+0000 7f4a15ca7640 1 --2- 172.21.15.76:0/1355812568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a10101190 0x7f4a10199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.844+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1355812568 >> 172.21.15.76:0/1355812568 conn(0x7f4a100fc700 msgr2=0x7f4a100733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:35.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.844+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1355812568 shutdown_connections 2024-03-12T05:55:35.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.844+0000 7f4a15ca7640 1 -- 172.21.15.76:0/1355812568 wait complete. 2024-03-12T05:55:35.859 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:55:35.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.960+0000 7f2126177640 1 -- 172.21.15.76:0/3551908051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120101c10 msgr2=0x7f2120104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.960+0000 7f2126177640 1 --2- 172.21.15.76:0/3551908051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120101c10 0x7f2120104000 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2110009900 tx=0x7f211002bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:35.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.960+0000 7f2126177640 1 -- 172.21.15.76:0/3551908051 shutdown_connections 2024-03-12T05:55:35.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.960+0000 7f2126177640 1 --2- 172.21.15.76:0/3551908051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2120106e70 0x7f21201092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.960+0000 7f2126177640 1 --2- 172.21.15.76:0/3551908051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2120104540 0x7f2120106930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.960+0000 7f2126177640 1 --2- 172.21.15.76:0/3551908051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120101c10 0x7f2120104000 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.960+0000 7f2126177640 1 -- 172.21.15.76:0/3551908051 >> 172.21.15.76:0/3551908051 conn(0x7f21200fb5e0 msgr2=0x7f21200fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:35.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.960+0000 7f2126177640 1 -- 172.21.15.76:0/3551908051 shutdown_connections 2024-03-12T05:55:35.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.960+0000 7f2126177640 1 -- 172.21.15.76:0/3551908051 wait complete. 2024-03-12T05:55:35.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.961+0000 7f2126177640 1 Processor -- start 2024-03-12T05:55:35.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.961+0000 7f2126177640 1 -- start start 2024-03-12T05:55:35.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.961+0000 7f2126177640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2120101c10 0x7f2120197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.961+0000 7f2126177640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104540 0x7f2120197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.961+0000 7f2126177640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2120106e70 0x7f212019e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.961+0000 7f2126177640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212019ed40 con 0x7f2120104540 2024-03-12T05:55:35.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.961+0000 7f2126177640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212019eeb0 con 0x7f2120106e70 2024-03-12T05:55:35.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.961+0000 7f2126177640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212019f070 con 0x7f2120101c10 2024-03-12T05:55:35.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.961+0000 7f211effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104540 0x7f2120197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.961+0000 7f211effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104540 0x7f2120197790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47326/0 (socket says 172.21.15.76:47326) 2024-03-12T05:55:35.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.961+0000 7f211effd640 1 -- 172.21.15.76:0/415113663 learned_addr learned my addr 172.21.15.76:0/415113663 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:35.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.961+0000 7f211ffff640 1 --2- 172.21.15.76:0/415113663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2120106e70 0x7f212019e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.962+0000 7f211f7fe640 1 --2- 172.21.15.76:0/415113663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2120101c10 0x7f2120197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.962+0000 7f211effd640 1 -- 172.21.15.76:0/415113663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2120101c10 msgr2=0x7f2120197250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.962+0000 7f211effd640 1 --2- 172.21.15.76:0/415113663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2120101c10 0x7f2120197250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.962+0000 7f211effd640 1 -- 172.21.15.76:0/415113663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2120106e70 msgr2=0x7f212019e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:35.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.962+0000 7f211effd640 1 --2- 172.21.15.76:0/415113663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2120106e70 0x7f212019e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:35.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.962+0000 7f211effd640 1 -- 172.21.15.76:0/415113663 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2110009560 con 0x7f2120104540 2024-03-12T05:55:35.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.962+0000 7f211effd640 1 --2- 172.21.15.76:0/415113663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104540 0x7f2120197790 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2114005fd0 tx=0x7f211400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:35.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.962+0000 7f211cff9640 1 -- 172.21.15.76:0/415113663 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2114009c30 con 0x7f2120104540 2024-03-12T05:55:35.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.962+0000 7f211cff9640 1 -- 172.21.15.76:0/415113663 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2114009d90 con 0x7f2120104540 2024-03-12T05:55:35.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.962+0000 7f2126177640 1 -- 172.21.15.76:0/415113663 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f212019f350 con 0x7f2120104540 2024-03-12T05:55:35.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.962+0000 7f211cff9640 1 -- 172.21.15.76:0/415113663 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2114010460 con 0x7f2120104540 2024-03-12T05:55:35.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.962+0000 7f2126177640 1 -- 172.21.15.76:0/415113663 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f212019f860 con 0x7f2120104540 2024-03-12T05:55:35.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.963+0000 7f2126177640 1 -- 172.21.15.76:0/415113663 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21200fd0d0 con 0x7f2120104540 2024-03-12T05:55:35.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.964+0000 7f211cff9640 1 -- 172.21.15.76:0/415113663 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2114003f20 con 0x7f2120104540 2024-03-12T05:55:35.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.965+0000 7f211cff9640 1 --2- 172.21.15.76:0/415113663 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21080a1a30 0x7f21080a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:35.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.965+0000 7f211f7fe640 1 --2- 172.21.15.76:0/415113663 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21080a1a30 0x7f21080a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:35.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.965+0000 7f211cff9640 1 -- 172.21.15.76:0/415113663 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f21140cd890 con 0x7f2120104540 2024-03-12T05:55:35.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.965+0000 7f211f7fe640 1 --2- 172.21.15.76:0/415113663 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21080a1a30 0x7f21080a3ef0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f211002c6a0 tx=0x7f2110005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:35.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:35.968+0000 7f211cff9640 1 -- 172.21.15.76:0/415113663 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2114096ab0 con 0x7f2120104540 2024-03-12T05:55:36.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.142+0000 7f2126177640 1 -- 172.21.15.76:0/415113663 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f212010cd30 con 0x7f2120104540 2024-03-12T05:55:36.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.143+0000 7f211cff9640 1 -- 172.21.15.76:0/415113663 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2114017020 con 0x7f2120104540 2024-03-12T05:55:36.146 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-12T05:55:36.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.145+0000 7f2126177640 1 -- 172.21.15.76:0/415113663 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21080a1a30 msgr2=0x7f21080a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.145+0000 7f2126177640 1 --2- 172.21.15.76:0/415113663 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21080a1a30 0x7f21080a3ef0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f211002c6a0 tx=0x7f2110005fd0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.145+0000 7f2126177640 1 -- 172.21.15.76:0/415113663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104540 msgr2=0x7f2120197790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.145+0000 7f2126177640 1 --2- 172.21.15.76:0/415113663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104540 0x7f2120197790 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2114005fd0 tx=0x7f211400dde0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.145+0000 7f2126177640 1 -- 172.21.15.76:0/415113663 shutdown_connections 2024-03-12T05:55:36.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.145+0000 7f2126177640 1 --2- 172.21.15.76:0/415113663 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f21080a1a30 0x7f21080a3ef0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.145+0000 7f2126177640 1 --2- 172.21.15.76:0/415113663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2120106e70 0x7f212019e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.145+0000 7f2126177640 1 --2- 172.21.15.76:0/415113663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104540 0x7f2120197790 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.145+0000 7f2126177640 1 --2- 172.21.15.76:0/415113663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2120101c10 0x7f2120197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.145+0000 7f2126177640 1 -- 172.21.15.76:0/415113663 >> 172.21.15.76:0/415113663 conn(0x7f21200fb5e0 msgr2=0x7f2120102630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:36.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.145+0000 7f2126177640 1 -- 172.21.15.76:0/415113663 shutdown_connections 2024-03-12T05:55:36.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.145+0000 7f2126177640 1 -- 172.21.15.76:0/415113663 wait complete. 2024-03-12T05:55:36.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:36 smithi076 ceph-mon[32095]: pgmap v345: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-12T05:55:36.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:36 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2543234990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:36.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:36 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3592910906' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:36.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:36 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1355812568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:36.161 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:55:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.265+0000 7f613225b640 1 -- 172.21.15.76:0/1840458521 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612c1023a0 msgr2=0x7f612c102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.265+0000 7f613225b640 1 --2- 172.21.15.76:0/1840458521 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612c1023a0 0x7f612c102820 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6114009900 tx=0x7f611402bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.265+0000 7f613225b640 1 -- 172.21.15.76:0/1840458521 shutdown_connections 2024-03-12T05:55:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.265+0000 7f613225b640 1 --2- 172.21.15.76:0/1840458521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612c102d60 0x7f612c1095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.265+0000 7f613225b640 1 --2- 172.21.15.76:0/1840458521 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612c1023a0 0x7f612c102820 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.265+0000 7f613225b640 1 --2- 172.21.15.76:0/1840458521 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f612c1011b0 0x7f612c1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.265+0000 7f613225b640 1 -- 172.21.15.76:0/1840458521 >> 172.21.15.76:0/1840458521 conn(0x7f612c0fc740 msgr2=0x7f612c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.265+0000 7f613225b640 1 -- 172.21.15.76:0/1840458521 shutdown_connections 2024-03-12T05:55:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.265+0000 7f613225b640 1 -- 172.21.15.76:0/1840458521 wait complete. 2024-03-12T05:55:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.265+0000 7f613225b640 1 Processor -- start 2024-03-12T05:55:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.265+0000 7f613225b640 1 -- start start 2024-03-12T05:55:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f613225b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f612c1011b0 0x7f612c197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f613225b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612c1023a0 0x7f612c1976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f613225b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612c102d60 0x7f612c19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f613225b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c19ec60 con 0x7f612c1023a0 2024-03-12T05:55:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f613225b640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c19edd0 con 0x7f612c1011b0 2024-03-12T05:55:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f613225b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c19ef90 con 0x7f612c102d60 2024-03-12T05:55:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f6122ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612c1023a0 0x7f612c1976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f6122ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612c1023a0 0x7f612c1976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47344/0 (socket says 172.21.15.76:47344) 2024-03-12T05:55:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f6122ffd640 1 -- 172.21.15.76:0/982607075 learned_addr learned my addr 172.21.15.76:0/982607075 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f612b7fe640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f612c1011b0 0x7f612c197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f612bfff640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612c102d60 0x7f612c19e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f6122ffd640 1 -- 172.21.15.76:0/982607075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612c102d60 msgr2=0x7f612c19e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f6122ffd640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612c102d60 0x7f612c19e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f6122ffd640 1 -- 172.21.15.76:0/982607075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f612c1011b0 msgr2=0x7f612c197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f6122ffd640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f612c1011b0 0x7f612c197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f6122ffd640 1 -- 172.21.15.76:0/982607075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6114009560 con 0x7f612c1023a0 2024-03-12T05:55:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f612b7fe640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f612c1011b0 0x7f612c197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f6122ffd640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612c1023a0 0x7f612c1976b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6114006010 tx=0x7f61140042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f61297fa640 1 -- 172.21.15.76:0/982607075 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61140044e0 con 0x7f612c1023a0 2024-03-12T05:55:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f61297fa640 1 -- 172.21.15.76:0/982607075 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61140054e0 con 0x7f612c1023a0 2024-03-12T05:55:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f613225b640 1 -- 172.21.15.76:0/982607075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612c19f210 con 0x7f612c1023a0 2024-03-12T05:55:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f61297fa640 1 -- 172.21.15.76:0/982607075 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611403e450 con 0x7f612c1023a0 2024-03-12T05:55:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.266+0000 7f613225b640 1 -- 172.21.15.76:0/982607075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612c19f720 con 0x7f612c1023a0 2024-03-12T05:55:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.267+0000 7f613225b640 1 -- 172.21.15.76:0/982607075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612c108470 con 0x7f612c1023a0 2024-03-12T05:55:36.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.268+0000 7f61297fa640 1 -- 172.21.15.76:0/982607075 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f61140035f0 con 0x7f612c1023a0 2024-03-12T05:55:36.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.269+0000 7f61297fa640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f61180a1a30 0x7f61180a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:36.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.269+0000 7f612b7fe640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f61180a1a30 0x7f61180a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:36.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.269+0000 7f61297fa640 1 -- 172.21.15.76:0/982607075 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f61140ef850 con 0x7f612c1023a0 2024-03-12T05:55:36.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.269+0000 7f612b7fe640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f61180a1a30 0x7f61180a3ef0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f612c102200 tx=0x7f611c00a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:36.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.273+0000 7f61297fa640 1 -- 172.21.15.76:0/982607075 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f61140b8a70 con 0x7f612c1023a0 2024-03-12T05:55:36.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:36 smithi082 ceph-mon[36382]: pgmap v345: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-12T05:55:36.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:36 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2543234990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:36.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:36 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3592910906' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:36.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:36 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1355812568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:36.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.449+0000 7f613225b640 1 -- 172.21.15.76:0/982607075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f612c02ce40 con 0x7f612c1023a0 2024-03-12T05:55:36.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.449+0000 7f61297fa640 1 -- 172.21.15.76:0/982607075 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f61140bc130 con 0x7f612c1023a0 2024-03-12T05:55:36.453 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-12T05:55:36.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.451+0000 7f613225b640 1 -- 172.21.15.76:0/982607075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f61180a1a30 msgr2=0x7f61180a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.451+0000 7f613225b640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f61180a1a30 0x7f61180a3ef0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f612c102200 tx=0x7f611c00a5e0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.451+0000 7f613225b640 1 -- 172.21.15.76:0/982607075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612c1023a0 msgr2=0x7f612c1976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.451+0000 7f613225b640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612c1023a0 0x7f612c1976b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6114006010 tx=0x7f61140042f0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.451+0000 7f613225b640 1 -- 172.21.15.76:0/982607075 shutdown_connections 2024-03-12T05:55:36.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.451+0000 7f613225b640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f61180a1a30 0x7f61180a3ef0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.451+0000 7f613225b640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612c102d60 0x7f612c19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.451+0000 7f613225b640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f612c1023a0 0x7f612c1976b0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.451+0000 7f613225b640 1 --2- 172.21.15.76:0/982607075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f612c1011b0 0x7f612c197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.451+0000 7f613225b640 1 -- 172.21.15.76:0/982607075 >> 172.21.15.76:0/982607075 conn(0x7f612c0fc740 msgr2=0x7f612c105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:36.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.452+0000 7f613225b640 1 -- 172.21.15.76:0/982607075 shutdown_connections 2024-03-12T05:55:36.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.452+0000 7f613225b640 1 -- 172.21.15.76:0/982607075 wait complete. 2024-03-12T05:55:36.466 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-12T05:55:36.466 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:55:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:36 smithi088 ceph-mon[36585]: pgmap v345: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-12T05:55:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:36 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2543234990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:36 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3592910906' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:36 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1355812568' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:36.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 -- 172.21.15.76:0/2823887645 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88d40fff90 msgr2=0x7f88d41003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 --2- 172.21.15.76:0/2823887645 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88d40fff90 0x7f88d41003b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f88c8009900 tx=0x7f88c802bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:36.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 -- 172.21.15.76:0/2823887645 shutdown_connections 2024-03-12T05:55:36.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 --2- 172.21.15.76:0/2823887645 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f88d4101b40 0x7f88d41083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 --2- 172.21.15.76:0/2823887645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d4101180 0x7f88d4101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 --2- 172.21.15.76:0/2823887645 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88d40fff90 0x7f88d41003b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 -- 172.21.15.76:0/2823887645 >> 172.21.15.76:0/2823887645 conn(0x7f88d40fb520 msgr2=0x7f88d40fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:36.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 -- 172.21.15.76:0/2823887645 shutdown_connections 2024-03-12T05:55:36.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 -- 172.21.15.76:0/2823887645 wait complete. 2024-03-12T05:55:36.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 Processor -- start 2024-03-12T05:55:36.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 -- start start 2024-03-12T05:55:36.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88d40fff90 0x7f88d4196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:36.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d4101180 0x7f88d41965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:36.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f88d4101b40 0x7f88d419d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:36.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d419db80 con 0x7f88d40fff90 2024-03-12T05:55:36.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d419dcf0 con 0x7f88d4101b40 2024-03-12T05:55:36.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d9da0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d419deb0 con 0x7f88d4101180 2024-03-12T05:55:36.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d8d9e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88d40fff90 0x7f88d4196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:36.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d8d9e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88d40fff90 0x7f88d4196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47358/0 (socket says 172.21.15.76:47358) 2024-03-12T05:55:36.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.565+0000 7f88d8d9e640 1 -- 172.21.15.76:0/1135088622 learned_addr learned my addr 172.21.15.76:0/1135088622 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:36.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.566+0000 7f88d3fff640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d4101180 0x7f88d41965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:36.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.566+0000 7f88d959f640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f88d4101b40 0x7f88d419d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:36.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.566+0000 7f88d8d9e640 1 -- 172.21.15.76:0/1135088622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d4101180 msgr2=0x7f88d41965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.566+0000 7f88d8d9e640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d4101180 0x7f88d41965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.566+0000 7f88d8d9e640 1 -- 172.21.15.76:0/1135088622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f88d4101b40 msgr2=0x7f88d419d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.566+0000 7f88d8d9e640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f88d4101b40 0x7f88d419d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.566+0000 7f88d8d9e640 1 -- 172.21.15.76:0/1135088622 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c8009560 con 0x7f88d40fff90 2024-03-12T05:55:36.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.566+0000 7f88d959f640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f88d4101b40 0x7f88d419d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:36.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.566+0000 7f88d8d9e640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88d40fff90 0x7f88d4196090 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f88c8000c00 tx=0x7f88c8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:36.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.566+0000 7f88d1ffb640 1 -- 172.21.15.76:0/1135088622 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c8004df0 con 0x7f88d40fff90 2024-03-12T05:55:36.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.566+0000 7f88d1ffb640 1 -- 172.21.15.76:0/1135088622 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88c8005430 con 0x7f88d40fff90 2024-03-12T05:55:36.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.566+0000 7f88d9da0640 1 -- 172.21.15.76:0/1135088622 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d419e130 con 0x7f88d40fff90 2024-03-12T05:55:36.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.566+0000 7f88d1ffb640 1 -- 172.21.15.76:0/1135088622 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c803e450 con 0x7f88d40fff90 2024-03-12T05:55:36.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.566+0000 7f88d9da0640 1 -- 172.21.15.76:0/1135088622 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d419e6c0 con 0x7f88d40fff90 2024-03-12T05:55:36.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.567+0000 7f88d9da0640 1 -- 172.21.15.76:0/1135088622 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d41072b0 con 0x7f88d40fff90 2024-03-12T05:55:36.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.568+0000 7f88d1ffb640 1 -- 172.21.15.76:0/1135088622 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f88c8003680 con 0x7f88d40fff90 2024-03-12T05:55:36.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.569+0000 7f88d1ffb640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f88c40a1a20 0x7f88c40a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:36.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.569+0000 7f88d3fff640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f88c40a1a20 0x7f88c40a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:36.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.569+0000 7f88d1ffb640 1 -- 172.21.15.76:0/1135088622 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f88c80ef9f0 con 0x7f88d40fff90 2024-03-12T05:55:36.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.569+0000 7f88d3fff640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f88c40a1a20 0x7f88c40a3ee0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f88d41975a0 tx=0x7f88c0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:36.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.573+0000 7f88d1ffb640 1 -- 172.21.15.76:0/1135088622 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f88c80b8bf0 con 0x7f88d40fff90 2024-03-12T05:55:36.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.783+0000 7f88d9da0640 1 -- 172.21.15.76:0/1135088622 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88d402cda0 con 0x7f88d40fff90 2024-03-12T05:55:36.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.784+0000 7f88d1ffb640 1 -- 172.21.15.76:0/1135088622 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11328 (secure 0 0 0) 0x7f88c80bc2b0 con 0x7f88d40fff90 2024-03-12T05:55:36.788 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:36.788 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:55:33.056004+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:55:36.788 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 22 2024-03-12T05:55:36.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.786+0000 7f88d9da0640 1 -- 172.21.15.76:0/1135088622 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f88c40a1a20 msgr2=0x7f88c40a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.786+0000 7f88d9da0640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f88c40a1a20 0x7f88c40a3ee0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f88d41975a0 tx=0x7f88c0008040 comp rx=0 tx=0).stop 2024-03-12T05:55:36.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.786+0000 7f88d9da0640 1 -- 172.21.15.76:0/1135088622 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88d40fff90 msgr2=0x7f88d4196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.786+0000 7f88d9da0640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88d40fff90 0x7f88d4196090 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f88c8000c00 tx=0x7f88c8004270 comp rx=0 tx=0).stop 2024-03-12T05:55:36.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.786+0000 7f88d9da0640 1 -- 172.21.15.76:0/1135088622 shutdown_connections 2024-03-12T05:55:36.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.786+0000 7f88d9da0640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f88c40a1a20 0x7f88c40a3ee0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.786+0000 7f88d9da0640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f88d4101b40 0x7f88d419d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.786+0000 7f88d9da0640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88d4101180 0x7f88d41965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.786+0000 7f88d9da0640 1 --2- 172.21.15.76:0/1135088622 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f88d40fff90 0x7f88d4196090 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.786+0000 7f88d9da0640 1 -- 172.21.15.76:0/1135088622 >> 172.21.15.76:0/1135088622 conn(0x7f88d40fb520 msgr2=0x7f88d4104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:36.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.786+0000 7f88d9da0640 1 -- 172.21.15.76:0/1135088622 shutdown_connections 2024-03-12T05:55:36.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.786+0000 7f88d9da0640 1 -- 172.21.15.76:0/1135088622 wait complete. 2024-03-12T05:55:36.801 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-12T05:55:36.801 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-12T05:55:36.802 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-12T05:55:36.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.900+0000 7f795fd2c640 1 -- 172.21.15.76:0/1048456224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7958100aa0 msgr2=0x7f7958100f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.900+0000 7f795fd2c640 1 --2- 172.21.15.76:0/1048456224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7958100aa0 0x7f7958100f20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7950009ca0 tx=0x7f795002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.900+0000 7f795fd2c640 1 -- 172.21.15.76:0/1048456224 shutdown_connections 2024-03-12T05:55:36.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.900+0000 7f795fd2c640 1 --2- 172.21.15.76:0/1048456224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7958100aa0 0x7f7958100f20 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.900+0000 7f795fd2c640 1 --2- 172.21.15.76:0/1048456224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7958069500 0x7f79580ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.900+0000 7f795fd2c640 1 --2- 172.21.15.76:0/1048456224 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7958069050 0x7f79580ff8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.900+0000 7f795fd2c640 1 -- 172.21.15.76:0/1048456224 >> 172.21.15.76:0/1048456224 conn(0x7f79580fae00 msgr2=0x7f79580fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:36.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.900+0000 7f795fd2c640 1 -- 172.21.15.76:0/1048456224 shutdown_connections 2024-03-12T05:55:36.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.900+0000 7f795fd2c640 1 -- 172.21.15.76:0/1048456224 wait complete. 2024-03-12T05:55:36.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795fd2c640 1 Processor -- start 2024-03-12T05:55:36.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795fd2c640 1 -- start start 2024-03-12T05:55:36.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795fd2c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7958069050 0x7f7958102850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:36.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795fd2c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7958069500 0x7f795819bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:36.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795fd2c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7958100aa0 0x7f795819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:36.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795fd2c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795819ed00 con 0x7f7958100aa0 2024-03-12T05:55:36.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795fd2c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795819ee70 con 0x7f7958069050 2024-03-12T05:55:36.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795fd2c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795819f030 con 0x7f7958069500 2024-03-12T05:55:36.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795e2c9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7958100aa0 0x7f795819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:36.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795e2c9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7958100aa0 0x7f795819e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47374/0 (socket says 172.21.15.76:47374) 2024-03-12T05:55:36.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795e2c9640 1 -- 172.21.15.76:0/2464527108 learned_addr learned my addr 172.21.15.76:0/2464527108 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:36.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795dac8640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7958069050 0x7f7958102850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:36.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795d2c7640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7958069500 0x7f795819bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:36.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795e2c9640 1 -- 172.21.15.76:0/2464527108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7958069500 msgr2=0x7f795819bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795e2c9640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7958069500 0x7f795819bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795e2c9640 1 -- 172.21.15.76:0/2464527108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7958069050 msgr2=0x7f7958102850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:36.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795e2c9640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7958069050 0x7f7958102850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:36.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795e2c9640 1 -- 172.21.15.76:0/2464527108 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7950009900 con 0x7f7958100aa0 2024-03-12T05:55:36.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795dac8640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7958069050 0x7f7958102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:36.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.901+0000 7f795e2c9640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7958100aa0 0x7f795819e570 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7950037040 tx=0x7f7950034d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:36.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.902+0000 7f794affd640 1 -- 172.21.15.76:0/2464527108 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7950003970 con 0x7f7958100aa0 2024-03-12T05:55:36.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.902+0000 7f794affd640 1 -- 172.21.15.76:0/2464527108 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7950004040 con 0x7f7958100aa0 2024-03-12T05:55:36.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.902+0000 7f795fd2c640 1 -- 172.21.15.76:0/2464527108 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795819f2b0 con 0x7f7958100aa0 2024-03-12T05:55:36.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.902+0000 7f794affd640 1 -- 172.21.15.76:0/2464527108 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79500406b0 con 0x7f7958100aa0 2024-03-12T05:55:36.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.902+0000 7f795fd2c640 1 -- 172.21.15.76:0/2464527108 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795819f7b0 con 0x7f7958100aa0 2024-03-12T05:55:36.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.903+0000 7f795fd2c640 1 -- 172.21.15.76:0/2464527108 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79581075d0 con 0x7f7958100aa0 2024-03-12T05:55:36.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.904+0000 7f794affd640 1 -- 172.21.15.76:0/2464527108 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f795000a040 con 0x7f7958100aa0 2024-03-12T05:55:36.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.904+0000 7f794affd640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f79440a1ab0 0x7f79440a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:36.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.904+0000 7f795dac8640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f79440a1ab0 0x7f79440a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:36.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.904+0000 7f794affd640 1 -- 172.21.15.76:0/2464527108 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f79500f0a20 con 0x7f7958100aa0 2024-03-12T05:55:36.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.905+0000 7f795dac8640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f79440a1ab0 0x7f79440a3f70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7958100900 tx=0x7f7954008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:36.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:36.907+0000 7f794affd640 1 -- 172.21.15.76:0/2464527108 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f79500b9c40 con 0x7f7958100aa0 2024-03-12T05:55:37.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.079+0000 7f795fd2c640 1 -- 172.21.15.76:0/2464527108 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f795802ce10 con 0x7f79440a1ab0 2024-03-12T05:55:37.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.086+0000 7f794affd640 1 -- 172.21.15.76:0/2464527108 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258839 (secure 0 0 0) 0x7f795802ce10 con 0x7f79440a1ab0 2024-03-12T05:55:37.093 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:37.094 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-03-12T05:55:37.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.089+0000 7f795fd2c640 1 -- 172.21.15.76:0/2464527108 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f79440a1ab0 msgr2=0x7f79440a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.089+0000 7f795fd2c640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f79440a1ab0 0x7f79440a3f70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7958100900 tx=0x7f7954008040 comp rx=0 tx=0).stop 2024-03-12T05:55:37.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.089+0000 7f795fd2c640 1 -- 172.21.15.76:0/2464527108 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7958100aa0 msgr2=0x7f795819e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.089+0000 7f795fd2c640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7958100aa0 0x7f795819e570 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7950037040 tx=0x7f7950034d20 comp rx=0 tx=0).stop 2024-03-12T05:55:37.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.089+0000 7f795fd2c640 1 -- 172.21.15.76:0/2464527108 shutdown_connections 2024-03-12T05:55:37.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.089+0000 7f795fd2c640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f79440a1ab0 0x7f79440a3f70 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.089+0000 7f795fd2c640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7958100aa0 0x7f795819e570 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.089+0000 7f795fd2c640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7958069500 0x7f795819bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.089+0000 7f795fd2c640 1 --2- 172.21.15.76:0/2464527108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7958069050 0x7f7958102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.089+0000 7f795fd2c640 1 -- 172.21.15.76:0/2464527108 >> 172.21.15.76:0/2464527108 conn(0x7f79580fae00 msgr2=0x7f79580fc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:37.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.089+0000 7f795fd2c640 1 -- 172.21.15.76:0/2464527108 shutdown_connections 2024-03-12T05:55:37.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.089+0000 7f795fd2c640 1 -- 172.21.15.76:0/2464527108 wait complete. 2024-03-12T05:55:37.103 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:37 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/415113663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:37.103 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:37 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/982607075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:37.103 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:37 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1135088622' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:37.104 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-03-12T05:55:36.892489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656655,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":845,"num_read_kb":542,"num_write":354,"num_write_kb":3059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180875,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":786,"ondisk_log_size":786,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":374940,"kb_used_data":13308,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566884,"statfs":{"total":1151940427776,"available":1151556489216,"internally_reserved":0,"allocated":13627392,"data_stored":7502289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":31,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002794"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904708+0000","last_change":"2024-03-12T05:53:38.207712+0000","last_active":"2024-03-12T05:55:10.904708+0000","last_peered":"2024-03-12T05:55:10.904708+0000","last_clean":"2024-03-12T05:55:10.904708+0000","last_became_active":"2024-03-12T05:53:38.207362+0000","last_became_peered":"2024-03-12T05:53:38.207362+0000","last_unstale":"2024-03-12T05:55:10.904708+0000","last_undegraded":"2024-03-12T05:55:10.904708+0000","last_fullsized":"2024-03-12T05:55:10.904708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:19.518959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.903191+0000","last_change":"2024-03-12T05:53:38.206877+0000","last_active":"2024-03-12T05:55:10.903191+0000","last_peered":"2024-03-12T05:55:10.903191+0000","last_clean":"2024-03-12T05:55:10.903191+0000","last_became_active":"2024-03-12T05:53:38.206468+0000","last_became_peered":"2024-03-12T05:53:38.206468+0000","last_unstale":"2024-03-12T05:55:10.903191+0000","last_undegraded":"2024-03-12T05:55:10.903191+0000","last_fullsized":"2024-03-12T05:55:10.903191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:02:45.138265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917190+0000","last_change":"2024-03-12T05:53:38.214979+0000","last_active":"2024-03-12T05:55:10.917190+0000","last_peered":"2024-03-12T05:55:10.917190+0000","last_clean":"2024-03-12T05:55:10.917190+0000","last_became_active":"2024-03-12T05:53:38.214580+0000","last_became_peered":"2024-03-12T05:53:38.214580+0000","last_unstale":"2024-03-12T05:55:10.917190+0000","last_undegraded":"2024-03-12T05:55:10.917190+0000","last_fullsized":"2024-03-12T05:55:10.917190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:39:58.989757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902953+0000","last_change":"2024-03-12T05:53:38.218645+0000","last_active":"2024-03-12T05:55:10.902953+0000","last_peered":"2024-03-12T05:55:10.902953+0000","last_clean":"2024-03-12T05:55:10.902953+0000","last_became_active":"2024-03-12T05:53:38.218254+0000","last_became_peered":"2024-03-12T05:53:38.218254+0000","last_unstale":"2024-03-12T05:55:10.902953+0000","last_undegraded":"2024-03-12T05:55:10.902953+0000","last_fullsized":"2024-03-12T05:55:10.902953+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:46:09.544989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.894649+0000","last_change":"2024-03-12T05:53:38.214588+0000","last_active":"2024-03-12T05:55:10.894649+0000","last_peered":"2024-03-12T05:55:10.894649+0000","last_clean":"2024-03-12T05:55:10.894649+0000","last_became_active":"2024-03-12T05:53:38.214192+0000","last_became_peered":"2024-03-12T05:53:38.214192+0000","last_unstale":"2024-03-12T05:55:10.894649+0000","last_undegraded":"2024-03-12T05:55:10.894649+0000","last_fullsized":"2024-03-12T05:55:10.894649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:54:04.880901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902295+0000","last_change":"2024-03-12T05:53:38.214315+0000","last_active":"2024-03-12T05:55:10.902295+0000","last_peered":"2024-03-12T05:55:10.902295+0000","last_clean":"2024-03-12T05:55:10.902295+0000","last_became_active":"2024-03-12T05:53:38.213881+0000","last_became_peered":"2024-03-12T05:53:38.213881+0000","last_unstale":"2024-03-12T05:55:10.902295+0000","last_undegraded":"2024-03-12T05:55:10.902295+0000","last_fullsized":"2024-03-12T05:55:10.902295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:11:32.834418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.908724+0000","last_change":"2024-03-12T05:53:38.214915+0000","last_active":"2024-03-12T05:55:10.908724+0000","last_peered":"2024-03-12T05:55:10.908724+0000","last_clean":"2024-03-12T05:55:10.908724+0000","last_became_active":"2024-03-12T05:53:38.214592+0000","last_became_peered":"2024-03-12T05:53:38.214592+0000","last_unstale":"2024-03-12T05:55:10.908724+0000","last_undegraded":"2024-03-12T05:55:10.908724+0000","last_fullsized":"2024-03-12T05:55:10.908724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:11.548585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.909127+0000","last_change":"2024-03-12T05:53:38.219201+0000","last_active":"2024-03-12T05:55:10.909127+0000","last_peered":"2024-03-12T05:55:10.909127+0000","last_clean":"2024-03-12T05:55:10.909127+0000","last_became_active":"2024-03-12T05:53:38.218833+0000","last_became_peered":"2024-03-12T05:53:38.218833+0000","last_unstale":"2024-03-12T05:55:10.909127+0000","last_undegraded":"2024-03-12T05:55:10.909127+0000","last_fullsized":"2024-03-12T05:55:10.909127+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:13:36.177325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.915355+0000","last_change":"2024-03-12T05:53:38.205126+0000","last_active":"2024-03-12T05:55:10.915355+0000","last_peered":"2024-03-12T05:55:10.915355+0000","last_clean":"2024-03-12T05:55:10.915355+0000","last_became_active":"2024-03-12T05:53:38.204478+0000","last_became_peered":"2024-03-12T05:53:38.204478+0000","last_unstale":"2024-03-12T05:55:10.915355+0000","last_undegraded":"2024-03-12T05:55:10.915355+0000","last_fullsized":"2024-03-12T05:55:10.915355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:25:56.475134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904502+0000","last_change":"2024-03-12T05:53:38.215186+0000","last_active":"2024-03-12T05:55:10.904502+0000","last_peered":"2024-03-12T05:55:10.904502+0000","last_clean":"2024-03-12T05:55:10.904502+0000","last_became_active":"2024-03-12T05:53:38.214832+0000","last_became_peered":"2024-03-12T05:53:38.214832+0000","last_unstale":"2024-03-12T05:55:10.904502+0000","last_undegraded":"2024-03-12T05:55:10.904502+0000","last_fullsized":"2024-03-12T05:55:10.904502+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:56:26.621990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918038+0000","last_change":"2024-03-12T05:53:38.222347+0000","last_active":"2024-03-12T05:55:10.918038+0000","last_peered":"2024-03-12T05:55:10.918038+0000","last_clean":"2024-03-12T05:55:10.918038+0000","last_became_active":"2024-03-12T05:53:38.221637+0000","last_became_peered":"2024-03-12T05:53:38.221637+0000","last_unstale":"2024-03-12T05:55:10.918038+0000","last_undegraded":"2024-03-12T05:55:10.918038+0000","last_fullsized":"2024-03-12T05:55:10.918038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:16:26.652673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.912936+0000","last_change":"2024-03-12T05:53:38.218687+0000","last_active":"2024-03-12T05:55:10.912936+0000","last_peered":"2024-03-12T05:55:10.912936+0000","last_clean":"2024-03-12T05:55:10.912936+0000","last_became_active":"2024-03-12T05:53:38.218393+0000","last_became_peered":"2024-03-12T05:53:38.218393+0000","last_unstale":"2024-03-12T05:55:10.912936+0000","last_undegraded":"2024-03-12T05:55:10.912936+0000","last_fullsized":"2024-03-12T05:55:10.912936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:41:39.918992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.900607+0000","last_change":"2024-03-12T05:53:38.207432+0000","last_active":"2024-03-12T05:55:10.900607+0000","last_peered":"2024-03-12T05:55:10.900607+0000","last_clean":"2024-03-12T05:55:10.900607+0000","last_became_active":"2024-03-12T05:53:38.207099+0000","last_became_peered":"2024-03-12T05:53:38.207099+0000","last_unstale":"2024-03-12T05:55:10.900607+0000","last_undegraded":"2024-03-12T05:55:10.900607+0000","last_fullsized":"2024-03-12T05:55:10.900607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:20:08.504723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.909220+0000","last_change":"2024-03-12T05:53:38.207086+0000","last_active":"2024-03-12T05:55:10.909220+0000","last_peered":"2024-03-12T05:55:10.909220+0000","last_clean":"2024-03-12T05:55:10.909220+0000","last_became_active":"2024-03-12T05:53:38.206714+0000","last_became_peered":"2024-03-12T05:53:38.206714+0000","last_unstale":"2024-03-12T05:55:10.909220+0000","last_undegraded":"2024-03-12T05:55:10.909220+0000","last_fullsized":"2024-03-12T05:55:10.909220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:07:13.012272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913790+0000","last_change":"2024-03-12T05:53:38.217179+0000","last_active":"2024-03-12T05:55:10.913790+0000","last_peered":"2024-03-12T05:55:10.913790+0000","last_clean":"2024-03-12T05:55:10.913790+0000","last_became_active":"2024-03-12T05:53:38.216820+0000","last_became_peered":"2024-03-12T05:53:38.216820+0000","last_unstale":"2024-03-12T05:55:10.913790+0000","last_undegraded":"2024-03-12T05:55:10.913790+0000","last_fullsized":"2024-03-12T05:55:10.913790+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:01:08.842611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918587+0000","last_change":"2024-03-12T05:53:38.220224+0000","last_active":"2024-03-12T05:55:10.918587+0000","last_peered":"2024-03-12T05:55:10.918587+0000","last_clean":"2024-03-12T05:55:10.918587+0000","last_became_active":"2024-03-12T05:53:38.219878+0000","last_became_peered":"2024-03-12T05:53:38.219878+0000","last_unstale":"2024-03-12T05:55:10.918587+0000","last_undegraded":"2024-03-12T05:55:10.918587+0000","last_fullsized":"2024-03-12T05:55:10.918587+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:53:16.545807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904735+0000","last_change":"2024-03-12T05:53:38.208524+0000","last_active":"2024-03-12T05:55:10.904735+0000","last_peered":"2024-03-12T05:55:10.904735+0000","last_clean":"2024-03-12T05:55:10.904735+0000","last_became_active":"2024-03-12T05:53:38.208209+0000","last_became_peered":"2024-03-12T05:53:38.208209+0000","last_unstale":"2024-03-12T05:55:10.904735+0000","last_undegraded":"2024-03-12T05:55:10.904735+0000","last_fullsized":"2024-03-12T05:55:10.904735+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:28:56.334066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917077+0000","last_change":"2024-03-12T05:53:38.208035+0000","last_active":"2024-03-12T05:55:10.917077+0000","last_peered":"2024-03-12T05:55:10.917077+0000","last_clean":"2024-03-12T05:55:10.917077+0000","last_became_active":"2024-03-12T05:53:38.207679+0000","last_became_peered":"2024-03-12T05:53:38.207679+0000","last_unstale":"2024-03-12T05:55:10.917077+0000","last_undegraded":"2024-03-12T05:55:10.917077+0000","last_fullsized":"2024-03-12T05:55:10.917077+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:40:20.620552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.890594+0000","last_change":"2024-03-12T05:53:38.207077+0000","last_active":"2024-03-12T05:55:10.890594+0000","last_peered":"2024-03-12T05:55:10.890594+0000","last_clean":"2024-03-12T05:55:10.890594+0000","last_became_active":"2024-03-12T05:53:38.206720+0000","last_became_peered":"2024-03-12T05:53:38.206720+0000","last_unstale":"2024-03-12T05:55:10.890594+0000","last_undegraded":"2024-03-12T05:55:10.890594+0000","last_fullsized":"2024-03-12T05:55:10.890594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:45:54.628394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013942+0000","last_change":"2024-03-12T05:53:37.201637+0000","last_active":"2024-03-12T05:54:42.013942+0000","last_peered":"2024-03-12T05:54:42.013942+0000","last_clean":"2024-03-12T05:54:42.013942+0000","last_became_active":"2024-03-12T05:53:37.201289+0000","last_became_peered":"2024-03-12T05:53:37.201289+0000","last_unstale":"2024-03-12T05:54:42.013942+0000","last_undegraded":"2024-03-12T05:54:42.013942+0000","last_fullsized":"2024-03-12T05:54:42.013942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:59:43.829570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902916+0000","last_change":"2024-03-12T05:53:38.218052+0000","last_active":"2024-03-12T05:55:10.902916+0000","last_peered":"2024-03-12T05:55:10.902916+0000","last_clean":"2024-03-12T05:55:10.902916+0000","last_became_active":"2024-03-12T05:53:38.217738+0000","last_became_peered":"2024-03-12T05:53:38.217738+0000","last_unstale":"2024-03-12T05:55:10.902916+0000","last_undegraded":"2024-03-12T05:55:10.902916+0000","last_fullsized":"2024-03-12T05:55:10.902916+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:59:40.599573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018614+0000","last_change":"2024-03-12T05:53:37.210321+0000","last_active":"2024-03-12T05:54:42.018614+0000","last_peered":"2024-03-12T05:54:42.018614+0000","last_clean":"2024-03-12T05:54:42.018614+0000","last_became_active":"2024-03-12T05:53:37.210018+0000","last_became_peered":"2024-03-12T05:53:37.210018+0000","last_unstale":"2024-03-12T05:54:42.018614+0000","last_undegraded":"2024-03-12T05:54:42.018614+0000","last_fullsized":"2024-03-12T05:54:42.018614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:13:45.433332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917328+0000","last_change":"2024-03-12T05:53:38.207801+0000","last_active":"2024-03-12T05:55:10.917328+0000","last_peered":"2024-03-12T05:55:10.917328+0000","last_clean":"2024-03-12T05:55:10.917328+0000","last_became_active":"2024-03-12T05:53:38.207467+0000","last_became_peered":"2024-03-12T05:53:38.207467+0000","last_unstale":"2024-03-12T05:55:10.917328+0000","last_undegraded":"2024-03-12T05:55:10.917328+0000","last_fullsized":"2024-03-12T05:55:10.917328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:56:14.469662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014569+0000","last_change":"2024-03-12T05:53:37.208288+0000","last_active":"2024-03-12T05:54:42.014569+0000","last_peered":"2024-03-12T05:54:42.014569+0000","last_clean":"2024-03-12T05:54:42.014569+0000","last_became_active":"2024-03-12T05:53:37.207929+0000","last_became_peered":"2024-03-12T05:53:37.207929+0000","last_unstale":"2024-03-12T05:54:42.014569+0000","last_undegraded":"2024-03-12T05:54:42.014569+0000","last_fullsized":"2024-03-12T05:54:42.014569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:10:52.719116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.916187+0000","last_change":"2024-03-12T05:53:38.207270+0000","last_active":"2024-03-12T05:55:10.916187+0000","last_peered":"2024-03-12T05:55:10.916187+0000","last_clean":"2024-03-12T05:55:10.916187+0000","last_became_active":"2024-03-12T05:53:38.205782+0000","last_became_peered":"2024-03-12T05:53:38.205782+0000","last_unstale":"2024-03-12T05:55:10.916187+0000","last_undegraded":"2024-03-12T05:55:10.916187+0000","last_fullsized":"2024-03-12T05:55:10.916187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:02:24.875444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014278+0000","last_change":"2024-03-12T05:53:37.203899+0000","last_active":"2024-03-12T05:54:42.014278+0000","last_peered":"2024-03-12T05:54:42.014278+0000","last_clean":"2024-03-12T05:54:42.014278+0000","last_became_active":"2024-03-12T05:53:37.203450+0000","last_became_peered":"2024-03-12T05:53:37.203450+0000","last_unstale":"2024-03-12T05:54:42.014278+0000","last_undegraded":"2024-03-12T05:54:42.014278+0000","last_fullsized":"2024-03-12T05:54:42.014278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:25.773826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917208+0000","last_change":"2024-03-12T05:54:42.480247+0000","last_active":"2024-03-12T05:55:10.917208+0000","last_peered":"2024-03-12T05:55:10.917208+0000","last_clean":"2024-03-12T05:55:10.917208+0000","last_became_active":"2024-03-12T05:54:42.479952+0000","last_became_peered":"2024-03-12T05:54:42.479952+0000","last_unstale":"2024-03-12T05:55:10.917208+0000","last_undegraded":"2024-03-12T05:55:10.917208+0000","last_fullsized":"2024-03-12T05:55:10.917208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:45:20.439878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":144,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:33.525060+0000","last_change":"2024-03-12T05:53:37.205401+0000","last_active":"2024-03-12T05:55:33.525060+0000","last_peered":"2024-03-12T05:55:33.525060+0000","last_clean":"2024-03-12T05:55:33.525060+0000","last_became_active":"2024-03-12T05:53:37.204993+0000","last_became_peered":"2024-03-12T05:53:37.204993+0000","last_unstale":"2024-03-12T05:55:33.525060+0000","last_undegraded":"2024-03-12T05:55:33.525060+0000","last_fullsized":"2024-03-12T05:55:33.525060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:56:06.177387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.901227+0000","last_change":"2024-03-12T05:53:38.210562+0000","last_active":"2024-03-12T05:55:10.901227+0000","last_peered":"2024-03-12T05:55:10.901227+0000","last_clean":"2024-03-12T05:55:10.901227+0000","last_became_active":"2024-03-12T05:53:38.210235+0000","last_became_peered":"2024-03-12T05:53:38.210235+0000","last_unstale":"2024-03-12T05:55:10.901227+0000","last_undegraded":"2024-03-12T05:55:10.901227+0000","last_fullsized":"2024-03-12T05:55:10.901227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:35.445812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014118+0000","last_change":"2024-03-12T05:53:37.204986+0000","last_active":"2024-03-12T05:54:42.014118+0000","last_peered":"2024-03-12T05:54:42.014118+0000","last_clean":"2024-03-12T05:54:42.014118+0000","last_became_active":"2024-03-12T05:53:37.204580+0000","last_became_peered":"2024-03-12T05:53:37.204580+0000","last_unstale":"2024-03-12T05:54:42.014118+0000","last_undegraded":"2024-03-12T05:54:42.014118+0000","last_fullsized":"2024-03-12T05:54:42.014118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:37:34.584395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.900872+0000","last_change":"2024-03-12T05:53:38.214948+0000","last_active":"2024-03-12T05:55:10.900872+0000","last_peered":"2024-03-12T05:55:10.900872+0000","last_clean":"2024-03-12T05:55:10.900872+0000","last_became_active":"2024-03-12T05:53:38.213748+0000","last_became_peered":"2024-03-12T05:53:38.213748+0000","last_unstale":"2024-03-12T05:55:10.900872+0000","last_undegraded":"2024-03-12T05:55:10.900872+0000","last_fullsized":"2024-03-12T05:55:10.900872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:12:45.994129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017009+0000","last_change":"2024-03-12T05:53:37.198167+0000","last_active":"2024-03-12T05:54:42.017009+0000","last_peered":"2024-03-12T05:54:42.017009+0000","last_clean":"2024-03-12T05:54:42.017009+0000","last_became_active":"2024-03-12T05:53:37.197808+0000","last_became_peered":"2024-03-12T05:53:37.197808+0000","last_unstale":"2024-03-12T05:54:42.017009+0000","last_undegraded":"2024-03-12T05:54:42.017009+0000","last_fullsized":"2024-03-12T05:54:42.017009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:40:19.498546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904933+0000","last_change":"2024-03-12T05:53:38.205464+0000","last_active":"2024-03-12T05:55:10.904933+0000","last_peered":"2024-03-12T05:55:10.904933+0000","last_clean":"2024-03-12T05:55:10.904933+0000","last_became_active":"2024-03-12T05:53:38.205129+0000","last_became_peered":"2024-03-12T05:53:38.205129+0000","last_unstale":"2024-03-12T05:55:10.904933+0000","last_undegraded":"2024-03-12T05:55:10.904933+0000","last_fullsized":"2024-03-12T05:55:10.904933+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:38:55.460862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017494+0000","last_change":"2024-03-12T05:53:37.201572+0000","last_active":"2024-03-12T05:54:42.017494+0000","last_peered":"2024-03-12T05:54:42.017494+0000","last_clean":"2024-03-12T05:54:42.017494+0000","last_became_active":"2024-03-12T05:53:37.201230+0000","last_became_peered":"2024-03-12T05:53:37.201230+0000","last_unstale":"2024-03-12T05:54:42.017494+0000","last_undegraded":"2024-03-12T05:54:42.017494+0000","last_fullsized":"2024-03-12T05:54:42.017494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:16:13.715459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.903437+0000","last_change":"2024-03-12T05:53:38.210696+0000","last_active":"2024-03-12T05:55:10.903437+0000","last_peered":"2024-03-12T05:55:10.903437+0000","last_clean":"2024-03-12T05:55:10.903437+0000","last_became_active":"2024-03-12T05:53:38.210325+0000","last_became_peered":"2024-03-12T05:53:38.210325+0000","last_unstale":"2024-03-12T05:55:10.903437+0000","last_undegraded":"2024-03-12T05:55:10.903437+0000","last_fullsized":"2024-03-12T05:55:10.903437+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:25:36.397314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:32.523269+0000","last_change":"2024-03-12T05:53:37.201712+0000","last_active":"2024-03-12T05:55:32.523269+0000","last_peered":"2024-03-12T05:55:32.523269+0000","last_clean":"2024-03-12T05:55:32.523269+0000","last_became_active":"2024-03-12T05:53:37.200794+0000","last_became_peered":"2024-03-12T05:53:37.200794+0000","last_unstale":"2024-03-12T05:55:32.523269+0000","last_undegraded":"2024-03-12T05:55:32.523269+0000","last_fullsized":"2024-03-12T05:55:32.523269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:43:29.673717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918225+0000","last_change":"2024-03-12T05:53:38.206647+0000","last_active":"2024-03-12T05:55:10.918225+0000","last_peered":"2024-03-12T05:55:10.918225+0000","last_clean":"2024-03-12T05:55:10.918225+0000","last_became_active":"2024-03-12T05:53:38.206254+0000","last_became_peered":"2024-03-12T05:53:38.206254+0000","last_unstale":"2024-03-12T05:55:10.918225+0000","last_undegraded":"2024-03-12T05:55:10.918225+0000","last_fullsized":"2024-03-12T05:55:10.918225+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:47:20.455360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687585+0000","last_change":"2024-03-12T05:53:37.208692+0000","last_active":"2024-03-12T05:54:42.687585+0000","last_peered":"2024-03-12T05:54:42.687585+0000","last_clean":"2024-03-12T05:54:42.687585+0000","last_became_active":"2024-03-12T05:53:37.208270+0000","last_became_peered":"2024-03-12T05:53:37.208270+0000","last_unstale":"2024-03-12T05:54:42.687585+0000","last_undegraded":"2024-03-12T05:54:42.687585+0000","last_fullsized":"2024-03-12T05:54:42.687585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:51:38.217999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913653+0000","last_change":"2024-03-12T05:53:38.218202+0000","last_active":"2024-03-12T05:55:10.913653+0000","last_peered":"2024-03-12T05:55:10.913653+0000","last_clean":"2024-03-12T05:55:10.913653+0000","last_became_active":"2024-03-12T05:53:38.217798+0000","last_became_peered":"2024-03-12T05:53:38.217798+0000","last_unstale":"2024-03-12T05:55:10.913653+0000","last_undegraded":"2024-03-12T05:55:10.913653+0000","last_fullsized":"2024-03-12T05:55:10.913653+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:09:29.422125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019722+0000","last_change":"2024-03-12T05:53:37.203252+0000","last_active":"2024-03-12T05:54:42.019722+0000","last_peered":"2024-03-12T05:54:42.019722+0000","last_clean":"2024-03-12T05:54:42.019722+0000","last_became_active":"2024-03-12T05:53:37.202916+0000","last_became_peered":"2024-03-12T05:53:37.202916+0000","last_unstale":"2024-03-12T05:54:42.019722+0000","last_undegraded":"2024-03-12T05:54:42.019722+0000","last_fullsized":"2024-03-12T05:54:42.019722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:09:34.629085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.916291+0000","last_change":"2024-03-12T05:54:42.021528+0000","last_active":"2024-03-12T05:55:10.916291+0000","last_peered":"2024-03-12T05:55:10.916291+0000","last_clean":"2024-03-12T05:55:10.916291+0000","last_became_active":"2024-03-12T05:54:42.021015+0000","last_became_peered":"2024-03-12T05:54:42.021015+0000","last_unstale":"2024-03-12T05:55:10.916291+0000","last_undegraded":"2024-03-12T05:55:10.916291+0000","last_fullsized":"2024-03-12T05:55:10.916291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:16.998017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":288,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:33.524254+0000","last_change":"2024-03-12T05:53:37.206271+0000","last_active":"2024-03-12T05:55:33.524254+0000","last_peered":"2024-03-12T05:55:33.524254+0000","last_clean":"2024-03-12T05:55:33.524254+0000","last_became_active":"2024-03-12T05:53:37.205920+0000","last_became_peered":"2024-03-12T05:53:37.205920+0000","last_unstale":"2024-03-12T05:55:33.524254+0000","last_undegraded":"2024-03-12T05:55:33.524254+0000","last_fullsized":"2024-03-12T05:55:33.524254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:30.140159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61603,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":236,"num_read_kb":178,"num_write":34,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.912256+0000","last_change":"2024-03-12T05:53:38.211397+0000","last_active":"2024-03-12T05:55:10.912256+0000","last_peered":"2024-03-12T05:55:10.912256+0000","last_clean":"2024-03-12T05:55:10.912256+0000","last_became_active":"2024-03-12T05:53:38.211045+0000","last_became_peered":"2024-03-12T05:53:38.211045+0000","last_unstale":"2024-03-12T05:55:10.912256+0000","last_undegraded":"2024-03-12T05:55:10.912256+0000","last_fullsized":"2024-03-12T05:55:10.912256+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:21:42.510436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019264+0000","last_change":"2024-03-12T05:53:37.210871+0000","last_active":"2024-03-12T05:54:42.019264+0000","last_peered":"2024-03-12T05:54:42.019264+0000","last_clean":"2024-03-12T05:54:42.019264+0000","last_became_active":"2024-03-12T05:53:37.210526+0000","last_became_peered":"2024-03-12T05:53:37.210526+0000","last_unstale":"2024-03-12T05:54:42.019264+0000","last_undegraded":"2024-03-12T05:54:42.019264+0000","last_fullsized":"2024-03-12T05:54:42.019264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:40:46.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.907189+0000","last_change":"2024-03-12T05:53:38.213751+0000","last_active":"2024-03-12T05:55:10.907189+0000","last_peered":"2024-03-12T05:55:10.907189+0000","last_clean":"2024-03-12T05:55:10.907189+0000","last_became_active":"2024-03-12T05:53:38.211558+0000","last_became_peered":"2024-03-12T05:53:38.211558+0000","last_unstale":"2024-03-12T05:55:10.907189+0000","last_undegraded":"2024-03-12T05:55:10.907189+0000","last_fullsized":"2024-03-12T05:55:10.907189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:40:44.194548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018330+0000","last_change":"2024-03-12T05:53:37.207368+0000","last_active":"2024-03-12T05:54:42.018330+0000","last_peered":"2024-03-12T05:54:42.018330+0000","last_clean":"2024-03-12T05:54:42.018330+0000","last_became_active":"2024-03-12T05:53:37.207066+0000","last_became_peered":"2024-03-12T05:53:37.207066+0000","last_unstale":"2024-03-12T05:54:42.018330+0000","last_undegraded":"2024-03-12T05:54:42.018330+0000","last_fullsized":"2024-03-12T05:54:42.018330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:28:58.155360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913837+0000","last_change":"2024-03-12T05:53:38.217079+0000","last_active":"2024-03-12T05:55:10.913837+0000","last_peered":"2024-03-12T05:55:10.913837+0000","last_clean":"2024-03-12T05:55:10.913837+0000","last_became_active":"2024-03-12T05:53:38.216757+0000","last_became_peered":"2024-03-12T05:53:38.216757+0000","last_unstale":"2024-03-12T05:55:10.913837+0000","last_undegraded":"2024-03-12T05:55:10.913837+0000","last_fullsized":"2024-03-12T05:55:10.913837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:48:13.125867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":140,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:34.555872+0000","last_change":"2024-03-12T05:53:37.202390+0000","last_active":"2024-03-12T05:55:34.555872+0000","last_peered":"2024-03-12T05:55:34.555872+0000","last_clean":"2024-03-12T05:55:34.555872+0000","last_became_active":"2024-03-12T05:53:37.202020+0000","last_became_peered":"2024-03-12T05:53:37.202020+0000","last_unstale":"2024-03-12T05:55:34.555872+0000","last_undegraded":"2024-03-12T05:55:34.555872+0000","last_fullsized":"2024-03-12T05:55:34.555872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:06:35.293227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":896,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.910544+0000","last_change":"2024-03-12T05:53:38.218464+0000","last_active":"2024-03-12T05:55:10.910544+0000","last_peered":"2024-03-12T05:55:10.910544+0000","last_clean":"2024-03-12T05:55:10.910544+0000","last_became_active":"2024-03-12T05:53:38.218136+0000","last_became_peered":"2024-03-12T05:53:38.218136+0000","last_unstale":"2024-03-12T05:55:10.910544+0000","last_undegraded":"2024-03-12T05:55:10.910544+0000","last_fullsized":"2024-03-12T05:55:10.910544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:31:05.409148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013820+0000","last_change":"2024-03-12T05:53:37.205161+0000","last_active":"2024-03-12T05:54:42.013820+0000","last_peered":"2024-03-12T05:54:42.013820+0000","last_clean":"2024-03-12T05:54:42.013820+0000","last_became_active":"2024-03-12T05:53:37.204799+0000","last_became_peered":"2024-03-12T05:53:37.204799+0000","last_unstale":"2024-03-12T05:54:42.013820+0000","last_undegraded":"2024-03-12T05:54:42.013820+0000","last_fullsized":"2024-03-12T05:54:42.013820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:29:32.065635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.900970+0000","last_change":"2024-03-12T05:53:38.216321+0000","last_active":"2024-03-12T05:55:10.900970+0000","last_peered":"2024-03-12T05:55:10.900970+0000","last_clean":"2024-03-12T05:55:10.900970+0000","last_became_active":"2024-03-12T05:53:38.215893+0000","last_became_peered":"2024-03-12T05:53:38.215893+0000","last_unstale":"2024-03-12T05:55:10.900970+0000","last_undegraded":"2024-03-12T05:55:10.900970+0000","last_fullsized":"2024-03-12T05:55:10.900970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:32:27.603930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014706+0000","last_change":"2024-03-12T05:53:37.203638+0000","last_active":"2024-03-12T05:54:42.014706+0000","last_peered":"2024-03-12T05:54:42.014706+0000","last_clean":"2024-03-12T05:54:42.014706+0000","last_became_active":"2024-03-12T05:53:37.203217+0000","last_became_peered":"2024-03-12T05:53:37.203217+0000","last_unstale":"2024-03-12T05:54:42.014706+0000","last_undegraded":"2024-03-12T05:54:42.014706+0000","last_fullsized":"2024-03-12T05:54:42.014706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:10.521197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904911+0000","last_change":"2024-03-12T05:53:38.210491+0000","last_active":"2024-03-12T05:55:10.904911+0000","last_peered":"2024-03-12T05:55:10.904911+0000","last_clean":"2024-03-12T05:55:10.904911+0000","last_became_active":"2024-03-12T05:53:38.210169+0000","last_became_peered":"2024-03-12T05:53:38.210169+0000","last_unstale":"2024-03-12T05:55:10.904911+0000","last_undegraded":"2024-03-12T05:55:10.904911+0000","last_fullsized":"2024-03-12T05:55:10.904911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:56:05.384834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013770+0000","last_change":"2024-03-12T05:53:37.204478+0000","last_active":"2024-03-12T05:54:42.013770+0000","last_peered":"2024-03-12T05:54:42.013770+0000","last_clean":"2024-03-12T05:54:42.013770+0000","last_became_active":"2024-03-12T05:53:37.204160+0000","last_became_peered":"2024-03-12T05:53:37.204160+0000","last_unstale":"2024-03-12T05:54:42.013770+0000","last_undegraded":"2024-03-12T05:54:42.013770+0000","last_fullsized":"2024-03-12T05:54:42.013770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:07.597731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.896102+0000","last_change":"2024-03-12T05:54:42.026040+0000","last_active":"2024-03-12T05:55:10.896102+0000","last_peered":"2024-03-12T05:55:10.896102+0000","last_clean":"2024-03-12T05:55:10.896102+0000","last_became_active":"2024-03-12T05:54:42.018230+0000","last_became_peered":"2024-03-12T05:54:42.018230+0000","last_unstale":"2024-03-12T05:55:10.896102+0000","last_undegraded":"2024-03-12T05:55:10.896102+0000","last_fullsized":"2024-03-12T05:55:10.896102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:32:28.703339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687385+0000","last_change":"2024-03-12T05:53:37.205983+0000","last_active":"2024-03-12T05:54:42.687385+0000","last_peered":"2024-03-12T05:54:42.687385+0000","last_clean":"2024-03-12T05:54:42.687385+0000","last_became_active":"2024-03-12T05:53:37.205629+0000","last_became_peered":"2024-03-12T05:53:37.205629+0000","last_unstale":"2024-03-12T05:54:42.687385+0000","last_undegraded":"2024-03-12T05:54:42.687385+0000","last_fullsized":"2024-03-12T05:54:42.687385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:37.928623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.908778+0000","last_change":"2024-03-12T05:53:38.213752+0000","last_active":"2024-03-12T05:55:10.908778+0000","last_peered":"2024-03-12T05:55:10.908778+0000","last_clean":"2024-03-12T05:55:10.908778+0000","last_became_active":"2024-03-12T05:53:38.213407+0000","last_became_peered":"2024-03-12T05:53:38.213407+0000","last_unstale":"2024-03-12T05:55:10.908778+0000","last_undegraded":"2024-03-12T05:55:10.908778+0000","last_fullsized":"2024-03-12T05:55:10.908778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:21:15.703455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.907245+0000","last_change":"2024-03-12T05:54:42.022196+0000","last_active":"2024-03-12T05:55:10.907245+0000","last_peered":"2024-03-12T05:55:10.907245+0000","last_clean":"2024-03-12T05:55:10.907245+0000","last_became_active":"2024-03-12T05:54:42.021774+0000","last_became_peered":"2024-03-12T05:54:42.021774+0000","last_unstale":"2024-03-12T05:55:10.907245+0000","last_undegraded":"2024-03-12T05:55:10.907245+0000","last_fullsized":"2024-03-12T05:55:10.907245+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:34:39.290359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:32.551476+0000","last_change":"2024-03-12T05:53:37.202782+0000","last_active":"2024-03-12T05:55:32.551476+0000","last_peered":"2024-03-12T05:55:32.551476+0000","last_clean":"2024-03-12T05:55:32.551476+0000","last_became_active":"2024-03-12T05:53:37.202439+0000","last_became_peered":"2024-03-12T05:53:37.202439+0000","last_unstale":"2024-03-12T05:55:32.551476+0000","last_undegraded":"2024-03-12T05:55:32.551476+0000","last_fullsized":"2024-03-12T05:55:32.551476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:07:18.939629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.914766+0000","last_change":"2024-03-12T05:53:38.217993+0000","last_active":"2024-03-12T05:55:10.914766+0000","last_peered":"2024-03-12T05:55:10.914766+0000","last_clean":"2024-03-12T05:55:10.914766+0000","last_became_active":"2024-03-12T05:53:38.217613+0000","last_became_peered":"2024-03-12T05:53:38.217613+0000","last_unstale":"2024-03-12T05:55:10.914766+0000","last_undegraded":"2024-03-12T05:55:10.914766+0000","last_fullsized":"2024-03-12T05:55:10.914766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:49:50.960004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.478908+0000","last_change":"2024-03-12T05:53:37.203662+0000","last_active":"2024-03-12T05:54:42.478908+0000","last_peered":"2024-03-12T05:54:42.478908+0000","last_clean":"2024-03-12T05:54:42.478908+0000","last_became_active":"2024-03-12T05:53:37.203309+0000","last_became_peered":"2024-03-12T05:53:37.203309+0000","last_unstale":"2024-03-12T05:54:42.478908+0000","last_undegraded":"2024-03-12T05:54:42.478908+0000","last_fullsized":"2024-03-12T05:54:42.478908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:47:24.603686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687337+0000","last_change":"2024-03-12T05:52:27.278524+0000","last_active":"2024-03-12T05:54:42.687337+0000","last_peered":"2024-03-12T05:54:42.687337+0000","last_clean":"2024-03-12T05:54:42.687337+0000","last_became_active":"2024-03-12T05:52:27.278010+0000","last_became_peered":"2024-03-12T05:52:27.278010+0000","last_unstale":"2024-03-12T05:54:42.687337+0000","last_undegraded":"2024-03-12T05:54:42.687337+0000","last_fullsized":"2024-03-12T05:54:42.687337+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:49:09.175725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:49:09.175725+0000","last_clean_scrub_stamp":"2024-03-12T05:49:09.175725+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:24:40.380724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.901203+0000","last_change":"2024-03-12T05:53:38.219451+0000","last_active":"2024-03-12T05:55:10.901203+0000","last_peered":"2024-03-12T05:55:10.901203+0000","last_clean":"2024-03-12T05:55:10.901203+0000","last_became_active":"2024-03-12T05:53:38.218840+0000","last_became_peered":"2024-03-12T05:53:38.218840+0000","last_unstale":"2024-03-12T05:55:10.901203+0000","last_undegraded":"2024-03-12T05:55:10.901203+0000","last_fullsized":"2024-03-12T05:55:10.901203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:05.595822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687398+0000","last_change":"2024-03-12T05:53:37.210352+0000","last_active":"2024-03-12T05:54:42.687398+0000","last_peered":"2024-03-12T05:54:42.687398+0000","last_clean":"2024-03-12T05:54:42.687398+0000","last_became_active":"2024-03-12T05:53:37.210002+0000","last_became_peered":"2024-03-12T05:53:37.210002+0000","last_unstale":"2024-03-12T05:54:42.687398+0000","last_undegraded":"2024-03-12T05:54:42.687398+0000","last_fullsized":"2024-03-12T05:54:42.687398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:49:52.368411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.022961+0000","last_change":"2024-03-12T05:54:42.022489+0000","last_active":"2024-03-12T05:54:42.022961+0000","last_peered":"2024-03-12T05:54:42.022961+0000","last_clean":"2024-03-12T05:54:42.022961+0000","last_became_active":"2024-03-12T05:54:42.016400+0000","last_became_peered":"2024-03-12T05:54:42.016400+0000","last_unstale":"2024-03-12T05:54:42.022961+0000","last_undegraded":"2024-03-12T05:54:42.022961+0000","last_fullsized":"2024-03-12T05:54:42.022961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:32:05.713420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.886986+0000","last_change":"2024-03-12T05:53:38.203951+0000","last_active":"2024-03-12T05:55:10.886986+0000","last_peered":"2024-03-12T05:55:10.886986+0000","last_clean":"2024-03-12T05:55:10.886986+0000","last_became_active":"2024-03-12T05:53:38.203617+0000","last_became_peered":"2024-03-12T05:53:38.203617+0000","last_unstale":"2024-03-12T05:55:10.886986+0000","last_undegraded":"2024-03-12T05:55:10.886986+0000","last_fullsized":"2024-03-12T05:55:10.886986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:19.138220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:11.379756+0000","last_change":"2024-03-12T05:53:37.203770+0000","last_active":"2024-03-12T05:55:11.379756+0000","last_peered":"2024-03-12T05:55:11.379756+0000","last_clean":"2024-03-12T05:55:11.379756+0000","last_became_active":"2024-03-12T05:53:37.203456+0000","last_became_peered":"2024-03-12T05:53:37.203456+0000","last_unstale":"2024-03-12T05:55:11.379756+0000","last_undegraded":"2024-03-12T05:55:11.379756+0000","last_fullsized":"2024-03-12T05:55:11.379756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:32.291536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.911027+0000","last_change":"2024-03-12T05:53:38.214872+0000","last_active":"2024-03-12T05:55:10.911027+0000","last_peered":"2024-03-12T05:55:10.911027+0000","last_clean":"2024-03-12T05:55:10.911027+0000","last_became_active":"2024-03-12T05:53:38.214139+0000","last_became_peered":"2024-03-12T05:53:38.214139+0000","last_unstale":"2024-03-12T05:55:10.911027+0000","last_undegraded":"2024-03-12T05:55:10.911027+0000","last_fullsized":"2024-03-12T05:55:10.911027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:20:21.808039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.479035+0000","last_change":"2024-03-12T05:53:37.203852+0000","last_active":"2024-03-12T05:54:42.479035+0000","last_peered":"2024-03-12T05:54:42.479035+0000","last_clean":"2024-03-12T05:54:42.479035+0000","last_became_active":"2024-03-12T05:53:37.203416+0000","last_became_peered":"2024-03-12T05:53:37.203416+0000","last_unstale":"2024-03-12T05:54:42.479035+0000","last_undegraded":"2024-03-12T05:54:42.479035+0000","last_fullsized":"2024-03-12T05:54:42.479035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:09:08.095957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.910705+0000","last_change":"2024-03-12T05:54:42.688395+0000","last_active":"2024-03-12T05:55:10.910705+0000","last_peered":"2024-03-12T05:55:10.910705+0000","last_clean":"2024-03-12T05:55:10.910705+0000","last_became_active":"2024-03-12T05:54:42.688094+0000","last_became_peered":"2024-03-12T05:54:42.688094+0000","last_unstale":"2024-03-12T05:55:10.910705+0000","last_undegraded":"2024-03-12T05:55:10.910705+0000","last_fullsized":"2024-03-12T05:55:10.910705+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:29:42.921726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.025218+0000","last_change":"2024-03-12T05:54:42.025218+0000","last_active":"2024-03-12T05:54:42.025218+0000","last_peered":"2024-03-12T05:54:42.025218+0000","last_clean":"2024-03-12T05:54:42.025218+0000","last_became_active":"2024-03-12T05:54:42.021022+0000","last_became_peered":"2024-03-12T05:54:42.021022+0000","last_unstale":"2024-03-12T05:54:42.025218+0000","last_undegraded":"2024-03-12T05:54:42.025218+0000","last_fullsized":"2024-03-12T05:54:42.025218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:20:57.232145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.874050+0000","last_change":"2024-03-12T05:53:38.207757+0000","last_active":"2024-03-12T05:55:10.874050+0000","last_peered":"2024-03-12T05:55:10.874050+0000","last_clean":"2024-03-12T05:55:10.874050+0000","last_became_active":"2024-03-12T05:53:38.207424+0000","last_became_peered":"2024-03-12T05:53:38.207424+0000","last_unstale":"2024-03-12T05:55:10.874050+0000","last_undegraded":"2024-03-12T05:55:10.874050+0000","last_fullsized":"2024-03-12T05:55:10.874050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:33:16.017907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016731+0000","last_change":"2024-03-12T05:53:37.208512+0000","last_active":"2024-03-12T05:54:42.016731+0000","last_peered":"2024-03-12T05:54:42.016731+0000","last_clean":"2024-03-12T05:54:42.016731+0000","last_became_active":"2024-03-12T05:53:37.207628+0000","last_became_peered":"2024-03-12T05:53:37.207628+0000","last_unstale":"2024-03-12T05:54:42.016731+0000","last_undegraded":"2024-03-12T05:54:42.016731+0000","last_fullsized":"2024-03-12T05:54:42.016731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:25:28.825779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.908590+0000","last_change":"2024-03-12T05:53:38.205654+0000","last_active":"2024-03-12T05:55:10.908590+0000","last_peered":"2024-03-12T05:55:10.908590+0000","last_clean":"2024-03-12T05:55:10.908590+0000","last_became_active":"2024-03-12T05:53:38.205325+0000","last_became_peered":"2024-03-12T05:53:38.205325+0000","last_unstale":"2024-03-12T05:55:10.908590+0000","last_undegraded":"2024-03-12T05:55:10.908590+0000","last_fullsized":"2024-03-12T05:55:10.908590+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:33:35.668625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019632+0000","last_change":"2024-03-12T05:53:37.211430+0000","last_active":"2024-03-12T05:54:42.019632+0000","last_peered":"2024-03-12T05:54:42.019632+0000","last_clean":"2024-03-12T05:54:42.019632+0000","last_became_active":"2024-03-12T05:53:37.210501+0000","last_became_peered":"2024-03-12T05:53:37.210501+0000","last_unstale":"2024-03-12T05:54:42.019632+0000","last_undegraded":"2024-03-12T05:54:42.019632+0000","last_fullsized":"2024-03-12T05:54:42.019632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:48:33.555134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902337+0000","last_change":"2024-03-12T05:53:38.212837+0000","last_active":"2024-03-12T05:55:10.902337+0000","last_peered":"2024-03-12T05:55:10.902337+0000","last_clean":"2024-03-12T05:55:10.902337+0000","last_became_active":"2024-03-12T05:53:38.212513+0000","last_became_peered":"2024-03-12T05:53:38.212513+0000","last_unstale":"2024-03-12T05:55:10.902337+0000","last_undegraded":"2024-03-12T05:55:10.902337+0000","last_fullsized":"2024-03-12T05:55:10.902337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:47:33.879954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014092+0000","last_change":"2024-03-12T05:53:37.207465+0000","last_active":"2024-03-12T05:54:42.014092+0000","last_peered":"2024-03-12T05:54:42.014092+0000","last_clean":"2024-03-12T05:54:42.014092+0000","last_became_active":"2024-03-12T05:53:37.207107+0000","last_became_peered":"2024-03-12T05:53:37.207107+0000","last_unstale":"2024-03-12T05:54:42.014092+0000","last_undegraded":"2024-03-12T05:54:42.014092+0000","last_fullsized":"2024-03-12T05:54:42.014092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:34.018856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917908+0000","last_change":"2024-03-12T05:53:38.221470+0000","last_active":"2024-03-12T05:55:10.917908+0000","last_peered":"2024-03-12T05:55:10.917908+0000","last_clean":"2024-03-12T05:55:10.917908+0000","last_became_active":"2024-03-12T05:53:38.221061+0000","last_became_peered":"2024-03-12T05:53:38.221061+0000","last_unstale":"2024-03-12T05:55:10.917908+0000","last_undegraded":"2024-03-12T05:55:10.917908+0000","last_fullsized":"2024-03-12T05:55:10.917908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:01:54.148877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019769+0000","last_change":"2024-03-12T05:53:37.203822+0000","last_active":"2024-03-12T05:54:42.019769+0000","last_peered":"2024-03-12T05:54:42.019769+0000","last_clean":"2024-03-12T05:54:42.019769+0000","last_became_active":"2024-03-12T05:53:37.203431+0000","last_became_peered":"2024-03-12T05:53:37.203431+0000","last_unstale":"2024-03-12T05:54:42.019769+0000","last_undegraded":"2024-03-12T05:54:42.019769+0000","last_fullsized":"2024-03-12T05:54:42.019769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:25:04.447261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918179+0000","last_change":"2024-03-12T05:53:38.222045+0000","last_active":"2024-03-12T05:55:10.918179+0000","last_peered":"2024-03-12T05:55:10.918179+0000","last_clean":"2024-03-12T05:55:10.918179+0000","last_became_active":"2024-03-12T05:53:38.220904+0000","last_became_peered":"2024-03-12T05:53:38.220904+0000","last_unstale":"2024-03-12T05:55:10.918179+0000","last_undegraded":"2024-03-12T05:55:10.918179+0000","last_fullsized":"2024-03-12T05:55:10.918179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:30:43.964599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019437+0000","last_change":"2024-03-12T05:53:37.206753+0000","last_active":"2024-03-12T05:54:42.019437+0000","last_peered":"2024-03-12T05:54:42.019437+0000","last_clean":"2024-03-12T05:54:42.019437+0000","last_became_active":"2024-03-12T05:53:37.206078+0000","last_became_peered":"2024-03-12T05:53:37.206078+0000","last_unstale":"2024-03-12T05:54:42.019437+0000","last_undegraded":"2024-03-12T05:54:42.019437+0000","last_fullsized":"2024-03-12T05:54:42.019437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:42:18.859528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.909834+0000","last_change":"2024-03-12T05:53:38.215671+0000","last_active":"2024-03-12T05:55:10.909834+0000","last_peered":"2024-03-12T05:55:10.909834+0000","last_clean":"2024-03-12T05:55:10.909834+0000","last_became_active":"2024-03-12T05:53:38.215309+0000","last_became_peered":"2024-03-12T05:53:38.215309+0000","last_unstale":"2024-03-12T05:55:10.909834+0000","last_undegraded":"2024-03-12T05:55:10.909834+0000","last_fullsized":"2024-03-12T05:55:10.909834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:50:12.176935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017343+0000","last_change":"2024-03-12T05:53:37.205627+0000","last_active":"2024-03-12T05:54:42.017343+0000","last_peered":"2024-03-12T05:54:42.017343+0000","last_clean":"2024-03-12T05:54:42.017343+0000","last_became_active":"2024-03-12T05:53:37.205289+0000","last_became_peered":"2024-03-12T05:53:37.205289+0000","last_unstale":"2024-03-12T05:54:42.017343+0000","last_undegraded":"2024-03-12T05:54:42.017343+0000","last_fullsized":"2024-03-12T05:54:42.017343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:56.682746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913506+0000","last_change":"2024-03-12T05:53:38.214376+0000","last_active":"2024-03-12T05:55:10.913506+0000","last_peered":"2024-03-12T05:55:10.913506+0000","last_clean":"2024-03-12T05:55:10.913506+0000","last_became_active":"2024-03-12T05:53:38.214011+0000","last_became_peered":"2024-03-12T05:53:38.214011+0000","last_unstale":"2024-03-12T05:55:10.913506+0000","last_undegraded":"2024-03-12T05:55:10.913506+0000","last_fullsized":"2024-03-12T05:55:10.913506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:32.605216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.908362+0000","last_change":"2024-03-12T05:53:38.205337+0000","last_active":"2024-03-12T05:55:10.908362+0000","last_peered":"2024-03-12T05:55:10.908362+0000","last_clean":"2024-03-12T05:55:10.908362+0000","last_became_active":"2024-03-12T05:53:38.205005+0000","last_became_peered":"2024-03-12T05:53:38.205005+0000","last_unstale":"2024-03-12T05:55:10.908362+0000","last_undegraded":"2024-03-12T05:55:10.908362+0000","last_fullsized":"2024-03-12T05:55:10.908362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:44:16.296370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.907222+0000","last_change":"2024-03-12T05:53:38.214694+0000","last_active":"2024-03-12T05:55:10.907222+0000","last_peered":"2024-03-12T05:55:10.907222+0000","last_clean":"2024-03-12T05:55:10.907222+0000","last_became_active":"2024-03-12T05:53:38.214285+0000","last_became_peered":"2024-03-12T05:53:38.214285+0000","last_unstale":"2024-03-12T05:55:10.907222+0000","last_undegraded":"2024-03-12T05:55:10.907222+0000","last_fullsized":"2024-03-12T05:55:10.907222+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:28:32.253332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902212+0000","last_change":"2024-03-12T05:53:38.213069+0000","last_active":"2024-03-12T05:55:10.902212+0000","last_peered":"2024-03-12T05:55:10.902212+0000","last_clean":"2024-03-12T05:55:10.902212+0000","last_became_active":"2024-03-12T05:53:38.212694+0000","last_became_peered":"2024-03-12T05:53:38.212694+0000","last_unstale":"2024-03-12T05:55:10.902212+0000","last_undegraded":"2024-03-12T05:55:10.902212+0000","last_fullsized":"2024-03-12T05:55:10.902212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:26:29.350292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913785+0000","last_change":"2024-03-12T05:53:38.212084+0000","last_active":"2024-03-12T05:55:10.913785+0000","last_peered":"2024-03-12T05:55:10.913785+0000","last_clean":"2024-03-12T05:55:10.913785+0000","last_became_active":"2024-03-12T05:53:38.211750+0000","last_became_peered":"2024-03-12T05:53:38.211750+0000","last_unstale":"2024-03-12T05:55:10.913785+0000","last_undegraded":"2024-03-12T05:55:10.913785+0000","last_fullsized":"2024-03-12T05:55:10.913785+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:05:02.099121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.899211+0000","last_change":"2024-03-12T05:53:38.212071+0000","last_active":"2024-03-12T05:55:10.899211+0000","last_peered":"2024-03-12T05:55:10.899211+0000","last_clean":"2024-03-12T05:55:10.899211+0000","last_became_active":"2024-03-12T05:53:38.211722+0000","last_became_peered":"2024-03-12T05:53:38.211722+0000","last_unstale":"2024-03-12T05:55:10.899211+0000","last_undegraded":"2024-03-12T05:55:10.899211+0000","last_fullsized":"2024-03-12T05:55:10.899211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:19:46.581837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.895503+0000","last_change":"2024-03-12T05:54:42.015717+0000","last_active":"2024-03-12T05:55:10.895503+0000","last_peered":"2024-03-12T05:55:10.895503+0000","last_clean":"2024-03-12T05:55:10.895503+0000","last_became_active":"2024-03-12T05:54:42.015266+0000","last_became_peered":"2024-03-12T05:54:42.015266+0000","last_unstale":"2024-03-12T05:55:10.895503+0000","last_undegraded":"2024-03-12T05:55:10.895503+0000","last_fullsized":"2024-03-12T05:55:10.895503+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:33:48.250458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.915527+0000","last_change":"2024-03-12T05:53:38.208018+0000","last_active":"2024-03-12T05:55:10.915527+0000","last_peered":"2024-03-12T05:55:10.915527+0000","last_clean":"2024-03-12T05:55:10.915527+0000","last_became_active":"2024-03-12T05:53:38.207682+0000","last_became_peered":"2024-03-12T05:53:38.207682+0000","last_unstale":"2024-03-12T05:55:10.915527+0000","last_undegraded":"2024-03-12T05:55:10.915527+0000","last_fullsized":"2024-03-12T05:55:10.915527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:56:10.903917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902615+0000","last_change":"2024-03-12T05:53:38.217540+0000","last_active":"2024-03-12T05:55:10.902615+0000","last_peered":"2024-03-12T05:55:10.902615+0000","last_clean":"2024-03-12T05:55:10.902615+0000","last_became_active":"2024-03-12T05:53:38.217144+0000","last_became_peered":"2024-03-12T05:53:38.217144+0000","last_unstale":"2024-03-12T05:55:10.902615+0000","last_undegraded":"2024-03-12T05:55:10.902615+0000","last_fullsized":"2024-03-12T05:55:10.902615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:46:18.905909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902398+0000","last_change":"2024-03-12T05:53:38.215006+0000","last_active":"2024-03-12T05:55:10.902398+0000","last_peered":"2024-03-12T05:55:10.902398+0000","last_clean":"2024-03-12T05:55:10.902398+0000","last_became_active":"2024-03-12T05:53:38.213756+0000","last_became_peered":"2024-03-12T05:53:38.213756+0000","last_unstale":"2024-03-12T05:55:10.902398+0000","last_undegraded":"2024-03-12T05:55:10.902398+0000","last_fullsized":"2024-03-12T05:55:10.902398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:14:57.642381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902674+0000","last_change":"2024-03-12T05:53:38.216096+0000","last_active":"2024-03-12T05:55:10.902674+0000","last_peered":"2024-03-12T05:55:10.902674+0000","last_clean":"2024-03-12T05:55:10.902674+0000","last_became_active":"2024-03-12T05:53:38.215153+0000","last_became_peered":"2024-03-12T05:53:38.215153+0000","last_unstale":"2024-03-12T05:55:10.902674+0000","last_undegraded":"2024-03-12T05:55:10.902674+0000","last_fullsized":"2024-03-12T05:55:10.902674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:41:03.893641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918844+0000","last_change":"2024-03-12T05:53:38.219165+0000","last_active":"2024-03-12T05:55:10.918844+0000","last_peered":"2024-03-12T05:55:10.918844+0000","last_clean":"2024-03-12T05:55:10.918844+0000","last_became_active":"2024-03-12T05:53:38.218579+0000","last_became_peered":"2024-03-12T05:53:38.218579+0000","last_unstale":"2024-03-12T05:55:10.918844+0000","last_undegraded":"2024-03-12T05:55:10.918844+0000","last_fullsized":"2024-03-12T05:55:10.918844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:06:37.531293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.907783+0000","last_change":"2024-03-12T05:53:38.212685+0000","last_active":"2024-03-12T05:55:10.907783+0000","last_peered":"2024-03-12T05:55:10.907783+0000","last_clean":"2024-03-12T05:55:10.907783+0000","last_became_active":"2024-03-12T05:53:38.211106+0000","last_became_peered":"2024-03-12T05:53:38.211106+0000","last_unstale":"2024-03-12T05:55:10.907783+0000","last_undegraded":"2024-03-12T05:55:10.907783+0000","last_fullsized":"2024-03-12T05:55:10.907783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:26:00.339270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904578+0000","last_change":"2024-03-12T05:53:38.218402+0000","last_active":"2024-03-12T05:55:10.904578+0000","last_peered":"2024-03-12T05:55:10.904578+0000","last_clean":"2024-03-12T05:55:10.904578+0000","last_became_active":"2024-03-12T05:53:38.217699+0000","last_became_peered":"2024-03-12T05:53:38.217699+0000","last_unstale":"2024-03-12T05:55:10.904578+0000","last_undegraded":"2024-03-12T05:55:10.904578+0000","last_fullsized":"2024-03-12T05:55:10.904578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:23:03.221710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66053,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":709,"num_read_kb":423,"num_write":171,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":198159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147,"ondisk_log_size":147,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351013,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":461631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612650,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32788,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712364,"statfs":{"total":95995035648,"available":95961460736,"internally_reserved":0,"allocated":1650688,"data_stored":1115001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776175,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":461962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037812,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":461743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32704,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712448,"statfs":{"total":95995035648,"available":95961546752,"internally_reserved":0,"allocated":1564672,"data_stored":1054144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":461234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":522630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":462505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":524512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":461428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":461979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1576960,"data_stored":1053520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T05:55:37.108 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-12T05:55:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.245+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/2706207149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb38102400 msgr2=0x7fbb38102880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.245+0000 7fbb3e3c7640 1 --2- 172.21.15.76:0/2706207149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb38102400 0x7fbb38102880 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009900 tx=0x7fbb2c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.245+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/2706207149 shutdown_connections 2024-03-12T05:55:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.245+0000 7fbb3e3c7640 1 --2- 172.21.15.76:0/2706207149 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb38102dc0 0x7fbb38109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.245+0000 7fbb3e3c7640 1 --2- 172.21.15.76:0/2706207149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb38102400 0x7fbb38102880 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.245+0000 7fbb3e3c7640 1 --2- 172.21.15.76:0/2706207149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb38101210 0x7fbb38101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.245+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/2706207149 >> 172.21.15.76:0/2706207149 conn(0x7fbb380fc7a0 msgr2=0x7fbb380febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.245+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/2706207149 shutdown_connections 2024-03-12T05:55:37.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.245+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/2706207149 wait complete. 2024-03-12T05:55:37.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.245+0000 7fbb3e3c7640 1 Processor -- start 2024-03-12T05:55:37.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb3e3c7640 1 -- start start 2024-03-12T05:55:37.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb3e3c7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb38101210 0x7fbb38197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:37.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb3e3c7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb38102400 0x7fbb38197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:37.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb3e3c7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb38102dc0 0x7fbb3819e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:37.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb3e3c7640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3819ed30 con 0x7fbb38101210 2024-03-12T05:55:37.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb3e3c7640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3819eea0 con 0x7fbb38102dc0 2024-03-12T05:55:37.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb3e3c7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3819f060 con 0x7fbb38102400 2024-03-12T05:55:37.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb38101210 0x7fbb38197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:37.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb38101210 0x7fbb38197240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47396/0 (socket says 172.21.15.76:47396) 2024-03-12T05:55:37.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb37fff640 1 -- 172.21.15.76:0/409297657 learned_addr learned my addr 172.21.15.76:0/409297657 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:37.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb377fe640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb38102400 0x7fbb38197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:37.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb3c964640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb38102dc0 0x7fbb3819e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:37.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb37fff640 1 -- 172.21.15.76:0/409297657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb38102400 msgr2=0x7fbb38197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb37fff640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb38102400 0x7fbb38197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb37fff640 1 -- 172.21.15.76:0/409297657 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb38102dc0 msgr2=0x7fbb3819e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb37fff640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb38102dc0 0x7fbb3819e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb37fff640 1 -- 172.21.15.76:0/409297657 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2c009560 con 0x7fbb38101210 2024-03-12T05:55:37.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb3c964640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb38102dc0 0x7fbb3819e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:37.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.246+0000 7fbb37fff640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb38101210 0x7fbb38197240 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800d950 tx=0x7fbb2800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:37.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.247+0000 7fbb357fa640 1 -- 172.21.15.76:0/409297657 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb28009c30 con 0x7fbb38101210 2024-03-12T05:55:37.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.247+0000 7fbb357fa640 1 -- 172.21.15.76:0/409297657 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb28009d90 con 0x7fbb38101210 2024-03-12T05:55:37.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.247+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/409297657 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb3819f340 con 0x7fbb38101210 2024-03-12T05:55:37.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.247+0000 7fbb357fa640 1 -- 172.21.15.76:0/409297657 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800f720 con 0x7fbb38101210 2024-03-12T05:55:37.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.247+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/409297657 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3819f800 con 0x7fbb38101210 2024-03-12T05:55:37.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.248+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/409297657 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb38108530 con 0x7fbb38101210 2024-03-12T05:55:37.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.249+0000 7fbb357fa640 1 -- 172.21.15.76:0/409297657 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fbb28010a60 con 0x7fbb38101210 2024-03-12T05:55:37.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.250+0000 7fbb357fa640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbb200a1a30 0x7fbb200a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:37.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.250+0000 7fbb377fe640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbb200a1a30 0x7fbb200a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:37.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.250+0000 7fbb357fa640 1 -- 172.21.15.76:0/409297657 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fbb280cd1a0 con 0x7fbb38101210 2024-03-12T05:55:37.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.250+0000 7fbb377fe640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbb200a1a30 0x7fbb200a3ef0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbb38198750 tx=0x7fbb2c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:37.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.253+0000 7fbb357fa640 1 -- 172.21.15.76:0/409297657 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbb280963c0 con 0x7fbb38101210 2024-03-12T05:55:37.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.427+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/409297657 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbb3802ceb0 con 0x7fbb200a1a30 2024-03-12T05:55:37.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.434+0000 7fbb357fa640 1 -- 172.21.15.76:0/409297657 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258839 (secure 0 0 0) 0x7fbb3802ceb0 con 0x7fbb200a1a30 2024-03-12T05:55:37.440 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:37.442 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-03-12T05:55:37.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.437+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/409297657 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbb200a1a30 msgr2=0x7fbb200a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.437+0000 7fbb3e3c7640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbb200a1a30 0x7fbb200a3ef0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbb38198750 tx=0x7fbb2c000bc0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.437+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/409297657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb38101210 msgr2=0x7fbb38197240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.437+0000 7fbb3e3c7640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb38101210 0x7fbb38197240 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800d950 tx=0x7fbb2800de10 comp rx=0 tx=0).stop 2024-03-12T05:55:37.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.437+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/409297657 shutdown_connections 2024-03-12T05:55:37.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.437+0000 7fbb3e3c7640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fbb200a1a30 0x7fbb200a3ef0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.437+0000 7fbb3e3c7640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbb38102dc0 0x7fbb3819e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.437+0000 7fbb3e3c7640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb38102400 0x7fbb38197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.437+0000 7fbb3e3c7640 1 --2- 172.21.15.76:0/409297657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb38101210 0x7fbb38197240 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.437+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/409297657 >> 172.21.15.76:0/409297657 conn(0x7fbb380fc7a0 msgr2=0x7fbb38105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:37.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.437+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/409297657 shutdown_connections 2024-03-12T05:55:37.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.437+0000 7fbb3e3c7640 1 -- 172.21.15.76:0/409297657 wait complete. 2024-03-12T05:55:37.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:37 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/415113663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:37.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:37 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/982607075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:37.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:37 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1135088622' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:37.453 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-03-12T05:55:36.892489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656655,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":845,"num_read_kb":542,"num_write":354,"num_write_kb":3059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180875,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":786,"ondisk_log_size":786,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":374940,"kb_used_data":13308,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566884,"statfs":{"total":1151940427776,"available":1151556489216,"internally_reserved":0,"allocated":13627392,"data_stored":7502289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":31,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002794"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904708+0000","last_change":"2024-03-12T05:53:38.207712+0000","last_active":"2024-03-12T05:55:10.904708+0000","last_peered":"2024-03-12T05:55:10.904708+0000","last_clean":"2024-03-12T05:55:10.904708+0000","last_became_active":"2024-03-12T05:53:38.207362+0000","last_became_peered":"2024-03-12T05:53:38.207362+0000","last_unstale":"2024-03-12T05:55:10.904708+0000","last_undegraded":"2024-03-12T05:55:10.904708+0000","last_fullsized":"2024-03-12T05:55:10.904708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:19.518959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.903191+0000","last_change":"2024-03-12T05:53:38.206877+0000","last_active":"2024-03-12T05:55:10.903191+0000","last_peered":"2024-03-12T05:55:10.903191+0000","last_clean":"2024-03-12T05:55:10.903191+0000","last_became_active":"2024-03-12T05:53:38.206468+0000","last_became_peered":"2024-03-12T05:53:38.206468+0000","last_unstale":"2024-03-12T05:55:10.903191+0000","last_undegraded":"2024-03-12T05:55:10.903191+0000","last_fullsized":"2024-03-12T05:55:10.903191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:02:45.138265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917190+0000","last_change":"2024-03-12T05:53:38.214979+0000","last_active":"2024-03-12T05:55:10.917190+0000","last_peered":"2024-03-12T05:55:10.917190+0000","last_clean":"2024-03-12T05:55:10.917190+0000","last_became_active":"2024-03-12T05:53:38.214580+0000","last_became_peered":"2024-03-12T05:53:38.214580+0000","last_unstale":"2024-03-12T05:55:10.917190+0000","last_undegraded":"2024-03-12T05:55:10.917190+0000","last_fullsized":"2024-03-12T05:55:10.917190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:39:58.989757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902953+0000","last_change":"2024-03-12T05:53:38.218645+0000","last_active":"2024-03-12T05:55:10.902953+0000","last_peered":"2024-03-12T05:55:10.902953+0000","last_clean":"2024-03-12T05:55:10.902953+0000","last_became_active":"2024-03-12T05:53:38.218254+0000","last_became_peered":"2024-03-12T05:53:38.218254+0000","last_unstale":"2024-03-12T05:55:10.902953+0000","last_undegraded":"2024-03-12T05:55:10.902953+0000","last_fullsized":"2024-03-12T05:55:10.902953+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:46:09.544989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.894649+0000","last_change":"2024-03-12T05:53:38.214588+0000","last_active":"2024-03-12T05:55:10.894649+0000","last_peered":"2024-03-12T05:55:10.894649+0000","last_clean":"2024-03-12T05:55:10.894649+0000","last_became_active":"2024-03-12T05:53:38.214192+0000","last_became_peered":"2024-03-12T05:53:38.214192+0000","last_unstale":"2024-03-12T05:55:10.894649+0000","last_undegraded":"2024-03-12T05:55:10.894649+0000","last_fullsized":"2024-03-12T05:55:10.894649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:54:04.880901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902295+0000","last_change":"2024-03-12T05:53:38.214315+0000","last_active":"2024-03-12T05:55:10.902295+0000","last_peered":"2024-03-12T05:55:10.902295+0000","last_clean":"2024-03-12T05:55:10.902295+0000","last_became_active":"2024-03-12T05:53:38.213881+0000","last_became_peered":"2024-03-12T05:53:38.213881+0000","last_unstale":"2024-03-12T05:55:10.902295+0000","last_undegraded":"2024-03-12T05:55:10.902295+0000","last_fullsized":"2024-03-12T05:55:10.902295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:11:32.834418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.908724+0000","last_change":"2024-03-12T05:53:38.214915+0000","last_active":"2024-03-12T05:55:10.908724+0000","last_peered":"2024-03-12T05:55:10.908724+0000","last_clean":"2024-03-12T05:55:10.908724+0000","last_became_active":"2024-03-12T05:53:38.214592+0000","last_became_peered":"2024-03-12T05:53:38.214592+0000","last_unstale":"2024-03-12T05:55:10.908724+0000","last_undegraded":"2024-03-12T05:55:10.908724+0000","last_fullsized":"2024-03-12T05:55:10.908724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:11.548585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.909127+0000","last_change":"2024-03-12T05:53:38.219201+0000","last_active":"2024-03-12T05:55:10.909127+0000","last_peered":"2024-03-12T05:55:10.909127+0000","last_clean":"2024-03-12T05:55:10.909127+0000","last_became_active":"2024-03-12T05:53:38.218833+0000","last_became_peered":"2024-03-12T05:53:38.218833+0000","last_unstale":"2024-03-12T05:55:10.909127+0000","last_undegraded":"2024-03-12T05:55:10.909127+0000","last_fullsized":"2024-03-12T05:55:10.909127+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:13:36.177325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.915355+0000","last_change":"2024-03-12T05:53:38.205126+0000","last_active":"2024-03-12T05:55:10.915355+0000","last_peered":"2024-03-12T05:55:10.915355+0000","last_clean":"2024-03-12T05:55:10.915355+0000","last_became_active":"2024-03-12T05:53:38.204478+0000","last_became_peered":"2024-03-12T05:53:38.204478+0000","last_unstale":"2024-03-12T05:55:10.915355+0000","last_undegraded":"2024-03-12T05:55:10.915355+0000","last_fullsized":"2024-03-12T05:55:10.915355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:25:56.475134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904502+0000","last_change":"2024-03-12T05:53:38.215186+0000","last_active":"2024-03-12T05:55:10.904502+0000","last_peered":"2024-03-12T05:55:10.904502+0000","last_clean":"2024-03-12T05:55:10.904502+0000","last_became_active":"2024-03-12T05:53:38.214832+0000","last_became_peered":"2024-03-12T05:53:38.214832+0000","last_unstale":"2024-03-12T05:55:10.904502+0000","last_undegraded":"2024-03-12T05:55:10.904502+0000","last_fullsized":"2024-03-12T05:55:10.904502+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:56:26.621990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918038+0000","last_change":"2024-03-12T05:53:38.222347+0000","last_active":"2024-03-12T05:55:10.918038+0000","last_peered":"2024-03-12T05:55:10.918038+0000","last_clean":"2024-03-12T05:55:10.918038+0000","last_became_active":"2024-03-12T05:53:38.221637+0000","last_became_peered":"2024-03-12T05:53:38.221637+0000","last_unstale":"2024-03-12T05:55:10.918038+0000","last_undegraded":"2024-03-12T05:55:10.918038+0000","last_fullsized":"2024-03-12T05:55:10.918038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:16:26.652673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.912936+0000","last_change":"2024-03-12T05:53:38.218687+0000","last_active":"2024-03-12T05:55:10.912936+0000","last_peered":"2024-03-12T05:55:10.912936+0000","last_clean":"2024-03-12T05:55:10.912936+0000","last_became_active":"2024-03-12T05:53:38.218393+0000","last_became_peered":"2024-03-12T05:53:38.218393+0000","last_unstale":"2024-03-12T05:55:10.912936+0000","last_undegraded":"2024-03-12T05:55:10.912936+0000","last_fullsized":"2024-03-12T05:55:10.912936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:41:39.918992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.900607+0000","last_change":"2024-03-12T05:53:38.207432+0000","last_active":"2024-03-12T05:55:10.900607+0000","last_peered":"2024-03-12T05:55:10.900607+0000","last_clean":"2024-03-12T05:55:10.900607+0000","last_became_active":"2024-03-12T05:53:38.207099+0000","last_became_peered":"2024-03-12T05:53:38.207099+0000","last_unstale":"2024-03-12T05:55:10.900607+0000","last_undegraded":"2024-03-12T05:55:10.900607+0000","last_fullsized":"2024-03-12T05:55:10.900607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:20:08.504723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.909220+0000","last_change":"2024-03-12T05:53:38.207086+0000","last_active":"2024-03-12T05:55:10.909220+0000","last_peered":"2024-03-12T05:55:10.909220+0000","last_clean":"2024-03-12T05:55:10.909220+0000","last_became_active":"2024-03-12T05:53:38.206714+0000","last_became_peered":"2024-03-12T05:53:38.206714+0000","last_unstale":"2024-03-12T05:55:10.909220+0000","last_undegraded":"2024-03-12T05:55:10.909220+0000","last_fullsized":"2024-03-12T05:55:10.909220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:07:13.012272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913790+0000","last_change":"2024-03-12T05:53:38.217179+0000","last_active":"2024-03-12T05:55:10.913790+0000","last_peered":"2024-03-12T05:55:10.913790+0000","last_clean":"2024-03-12T05:55:10.913790+0000","last_became_active":"2024-03-12T05:53:38.216820+0000","last_became_peered":"2024-03-12T05:53:38.216820+0000","last_unstale":"2024-03-12T05:55:10.913790+0000","last_undegraded":"2024-03-12T05:55:10.913790+0000","last_fullsized":"2024-03-12T05:55:10.913790+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:01:08.842611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918587+0000","last_change":"2024-03-12T05:53:38.220224+0000","last_active":"2024-03-12T05:55:10.918587+0000","last_peered":"2024-03-12T05:55:10.918587+0000","last_clean":"2024-03-12T05:55:10.918587+0000","last_became_active":"2024-03-12T05:53:38.219878+0000","last_became_peered":"2024-03-12T05:53:38.219878+0000","last_unstale":"2024-03-12T05:55:10.918587+0000","last_undegraded":"2024-03-12T05:55:10.918587+0000","last_fullsized":"2024-03-12T05:55:10.918587+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:53:16.545807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904735+0000","last_change":"2024-03-12T05:53:38.208524+0000","last_active":"2024-03-12T05:55:10.904735+0000","last_peered":"2024-03-12T05:55:10.904735+0000","last_clean":"2024-03-12T05:55:10.904735+0000","last_became_active":"2024-03-12T05:53:38.208209+0000","last_became_peered":"2024-03-12T05:53:38.208209+0000","last_unstale":"2024-03-12T05:55:10.904735+0000","last_undegraded":"2024-03-12T05:55:10.904735+0000","last_fullsized":"2024-03-12T05:55:10.904735+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:28:56.334066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917077+0000","last_change":"2024-03-12T05:53:38.208035+0000","last_active":"2024-03-12T05:55:10.917077+0000","last_peered":"2024-03-12T05:55:10.917077+0000","last_clean":"2024-03-12T05:55:10.917077+0000","last_became_active":"2024-03-12T05:53:38.207679+0000","last_became_peered":"2024-03-12T05:53:38.207679+0000","last_unstale":"2024-03-12T05:55:10.917077+0000","last_undegraded":"2024-03-12T05:55:10.917077+0000","last_fullsized":"2024-03-12T05:55:10.917077+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:40:20.620552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.890594+0000","last_change":"2024-03-12T05:53:38.207077+0000","last_active":"2024-03-12T05:55:10.890594+0000","last_peered":"2024-03-12T05:55:10.890594+0000","last_clean":"2024-03-12T05:55:10.890594+0000","last_became_active":"2024-03-12T05:53:38.206720+0000","last_became_peered":"2024-03-12T05:53:38.206720+0000","last_unstale":"2024-03-12T05:55:10.890594+0000","last_undegraded":"2024-03-12T05:55:10.890594+0000","last_fullsized":"2024-03-12T05:55:10.890594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:45:54.628394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013942+0000","last_change":"2024-03-12T05:53:37.201637+0000","last_active":"2024-03-12T05:54:42.013942+0000","last_peered":"2024-03-12T05:54:42.013942+0000","last_clean":"2024-03-12T05:54:42.013942+0000","last_became_active":"2024-03-12T05:53:37.201289+0000","last_became_peered":"2024-03-12T05:53:37.201289+0000","last_unstale":"2024-03-12T05:54:42.013942+0000","last_undegraded":"2024-03-12T05:54:42.013942+0000","last_fullsized":"2024-03-12T05:54:42.013942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:59:43.829570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902916+0000","last_change":"2024-03-12T05:53:38.218052+0000","last_active":"2024-03-12T05:55:10.902916+0000","last_peered":"2024-03-12T05:55:10.902916+0000","last_clean":"2024-03-12T05:55:10.902916+0000","last_became_active":"2024-03-12T05:53:38.217738+0000","last_became_peered":"2024-03-12T05:53:38.217738+0000","last_unstale":"2024-03-12T05:55:10.902916+0000","last_undegraded":"2024-03-12T05:55:10.902916+0000","last_fullsized":"2024-03-12T05:55:10.902916+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:59:40.599573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018614+0000","last_change":"2024-03-12T05:53:37.210321+0000","last_active":"2024-03-12T05:54:42.018614+0000","last_peered":"2024-03-12T05:54:42.018614+0000","last_clean":"2024-03-12T05:54:42.018614+0000","last_became_active":"2024-03-12T05:53:37.210018+0000","last_became_peered":"2024-03-12T05:53:37.210018+0000","last_unstale":"2024-03-12T05:54:42.018614+0000","last_undegraded":"2024-03-12T05:54:42.018614+0000","last_fullsized":"2024-03-12T05:54:42.018614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:13:45.433332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917328+0000","last_change":"2024-03-12T05:53:38.207801+0000","last_active":"2024-03-12T05:55:10.917328+0000","last_peered":"2024-03-12T05:55:10.917328+0000","last_clean":"2024-03-12T05:55:10.917328+0000","last_became_active":"2024-03-12T05:53:38.207467+0000","last_became_peered":"2024-03-12T05:53:38.207467+0000","last_unstale":"2024-03-12T05:55:10.917328+0000","last_undegraded":"2024-03-12T05:55:10.917328+0000","last_fullsized":"2024-03-12T05:55:10.917328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:56:14.469662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014569+0000","last_change":"2024-03-12T05:53:37.208288+0000","last_active":"2024-03-12T05:54:42.014569+0000","last_peered":"2024-03-12T05:54:42.014569+0000","last_clean":"2024-03-12T05:54:42.014569+0000","last_became_active":"2024-03-12T05:53:37.207929+0000","last_became_peered":"2024-03-12T05:53:37.207929+0000","last_unstale":"2024-03-12T05:54:42.014569+0000","last_undegraded":"2024-03-12T05:54:42.014569+0000","last_fullsized":"2024-03-12T05:54:42.014569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:10:52.719116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.916187+0000","last_change":"2024-03-12T05:53:38.207270+0000","last_active":"2024-03-12T05:55:10.916187+0000","last_peered":"2024-03-12T05:55:10.916187+0000","last_clean":"2024-03-12T05:55:10.916187+0000","last_became_active":"2024-03-12T05:53:38.205782+0000","last_became_peered":"2024-03-12T05:53:38.205782+0000","last_unstale":"2024-03-12T05:55:10.916187+0000","last_undegraded":"2024-03-12T05:55:10.916187+0000","last_fullsized":"2024-03-12T05:55:10.916187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:02:24.875444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014278+0000","last_change":"2024-03-12T05:53:37.203899+0000","last_active":"2024-03-12T05:54:42.014278+0000","last_peered":"2024-03-12T05:54:42.014278+0000","last_clean":"2024-03-12T05:54:42.014278+0000","last_became_active":"2024-03-12T05:53:37.203450+0000","last_became_peered":"2024-03-12T05:53:37.203450+0000","last_unstale":"2024-03-12T05:54:42.014278+0000","last_undegraded":"2024-03-12T05:54:42.014278+0000","last_fullsized":"2024-03-12T05:54:42.014278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:25.773826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917208+0000","last_change":"2024-03-12T05:54:42.480247+0000","last_active":"2024-03-12T05:55:10.917208+0000","last_peered":"2024-03-12T05:55:10.917208+0000","last_clean":"2024-03-12T05:55:10.917208+0000","last_became_active":"2024-03-12T05:54:42.479952+0000","last_became_peered":"2024-03-12T05:54:42.479952+0000","last_unstale":"2024-03-12T05:55:10.917208+0000","last_undegraded":"2024-03-12T05:55:10.917208+0000","last_fullsized":"2024-03-12T05:55:10.917208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:45:20.439878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":144,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:33.525060+0000","last_change":"2024-03-12T05:53:37.205401+0000","last_active":"2024-03-12T05:55:33.525060+0000","last_peered":"2024-03-12T05:55:33.525060+0000","last_clean":"2024-03-12T05:55:33.525060+0000","last_became_active":"2024-03-12T05:53:37.204993+0000","last_became_peered":"2024-03-12T05:53:37.204993+0000","last_unstale":"2024-03-12T05:55:33.525060+0000","last_undegraded":"2024-03-12T05:55:33.525060+0000","last_fullsized":"2024-03-12T05:55:33.525060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:56:06.177387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.901227+0000","last_change":"2024-03-12T05:53:38.210562+0000","last_active":"2024-03-12T05:55:10.901227+0000","last_peered":"2024-03-12T05:55:10.901227+0000","last_clean":"2024-03-12T05:55:10.901227+0000","last_became_active":"2024-03-12T05:53:38.210235+0000","last_became_peered":"2024-03-12T05:53:38.210235+0000","last_unstale":"2024-03-12T05:55:10.901227+0000","last_undegraded":"2024-03-12T05:55:10.901227+0000","last_fullsized":"2024-03-12T05:55:10.901227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:35.445812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014118+0000","last_change":"2024-03-12T05:53:37.204986+0000","last_active":"2024-03-12T05:54:42.014118+0000","last_peered":"2024-03-12T05:54:42.014118+0000","last_clean":"2024-03-12T05:54:42.014118+0000","last_became_active":"2024-03-12T05:53:37.204580+0000","last_became_peered":"2024-03-12T05:53:37.204580+0000","last_unstale":"2024-03-12T05:54:42.014118+0000","last_undegraded":"2024-03-12T05:54:42.014118+0000","last_fullsized":"2024-03-12T05:54:42.014118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:37:34.584395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.900872+0000","last_change":"2024-03-12T05:53:38.214948+0000","last_active":"2024-03-12T05:55:10.900872+0000","last_peered":"2024-03-12T05:55:10.900872+0000","last_clean":"2024-03-12T05:55:10.900872+0000","last_became_active":"2024-03-12T05:53:38.213748+0000","last_became_peered":"2024-03-12T05:53:38.213748+0000","last_unstale":"2024-03-12T05:55:10.900872+0000","last_undegraded":"2024-03-12T05:55:10.900872+0000","last_fullsized":"2024-03-12T05:55:10.900872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:12:45.994129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017009+0000","last_change":"2024-03-12T05:53:37.198167+0000","last_active":"2024-03-12T05:54:42.017009+0000","last_peered":"2024-03-12T05:54:42.017009+0000","last_clean":"2024-03-12T05:54:42.017009+0000","last_became_active":"2024-03-12T05:53:37.197808+0000","last_became_peered":"2024-03-12T05:53:37.197808+0000","last_unstale":"2024-03-12T05:54:42.017009+0000","last_undegraded":"2024-03-12T05:54:42.017009+0000","last_fullsized":"2024-03-12T05:54:42.017009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:40:19.498546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904933+0000","last_change":"2024-03-12T05:53:38.205464+0000","last_active":"2024-03-12T05:55:10.904933+0000","last_peered":"2024-03-12T05:55:10.904933+0000","last_clean":"2024-03-12T05:55:10.904933+0000","last_became_active":"2024-03-12T05:53:38.205129+0000","last_became_peered":"2024-03-12T05:53:38.205129+0000","last_unstale":"2024-03-12T05:55:10.904933+0000","last_undegraded":"2024-03-12T05:55:10.904933+0000","last_fullsized":"2024-03-12T05:55:10.904933+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:38:55.460862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017494+0000","last_change":"2024-03-12T05:53:37.201572+0000","last_active":"2024-03-12T05:54:42.017494+0000","last_peered":"2024-03-12T05:54:42.017494+0000","last_clean":"2024-03-12T05:54:42.017494+0000","last_became_active":"2024-03-12T05:53:37.201230+0000","last_became_peered":"2024-03-12T05:53:37.201230+0000","last_unstale":"2024-03-12T05:54:42.017494+0000","last_undegraded":"2024-03-12T05:54:42.017494+0000","last_fullsized":"2024-03-12T05:54:42.017494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:16:13.715459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.903437+0000","last_change":"2024-03-12T05:53:38.210696+0000","last_active":"2024-03-12T05:55:10.903437+0000","last_peered":"2024-03-12T05:55:10.903437+0000","last_clean":"2024-03-12T05:55:10.903437+0000","last_became_active":"2024-03-12T05:53:38.210325+0000","last_became_peered":"2024-03-12T05:53:38.210325+0000","last_unstale":"2024-03-12T05:55:10.903437+0000","last_undegraded":"2024-03-12T05:55:10.903437+0000","last_fullsized":"2024-03-12T05:55:10.903437+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:25:36.397314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:32.523269+0000","last_change":"2024-03-12T05:53:37.201712+0000","last_active":"2024-03-12T05:55:32.523269+0000","last_peered":"2024-03-12T05:55:32.523269+0000","last_clean":"2024-03-12T05:55:32.523269+0000","last_became_active":"2024-03-12T05:53:37.200794+0000","last_became_peered":"2024-03-12T05:53:37.200794+0000","last_unstale":"2024-03-12T05:55:32.523269+0000","last_undegraded":"2024-03-12T05:55:32.523269+0000","last_fullsized":"2024-03-12T05:55:32.523269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:43:29.673717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918225+0000","last_change":"2024-03-12T05:53:38.206647+0000","last_active":"2024-03-12T05:55:10.918225+0000","last_peered":"2024-03-12T05:55:10.918225+0000","last_clean":"2024-03-12T05:55:10.918225+0000","last_became_active":"2024-03-12T05:53:38.206254+0000","last_became_peered":"2024-03-12T05:53:38.206254+0000","last_unstale":"2024-03-12T05:55:10.918225+0000","last_undegraded":"2024-03-12T05:55:10.918225+0000","last_fullsized":"2024-03-12T05:55:10.918225+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:47:20.455360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687585+0000","last_change":"2024-03-12T05:53:37.208692+0000","last_active":"2024-03-12T05:54:42.687585+0000","last_peered":"2024-03-12T05:54:42.687585+0000","last_clean":"2024-03-12T05:54:42.687585+0000","last_became_active":"2024-03-12T05:53:37.208270+0000","last_became_peered":"2024-03-12T05:53:37.208270+0000","last_unstale":"2024-03-12T05:54:42.687585+0000","last_undegraded":"2024-03-12T05:54:42.687585+0000","last_fullsized":"2024-03-12T05:54:42.687585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:51:38.217999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913653+0000","last_change":"2024-03-12T05:53:38.218202+0000","last_active":"2024-03-12T05:55:10.913653+0000","last_peered":"2024-03-12T05:55:10.913653+0000","last_clean":"2024-03-12T05:55:10.913653+0000","last_became_active":"2024-03-12T05:53:38.217798+0000","last_became_peered":"2024-03-12T05:53:38.217798+0000","last_unstale":"2024-03-12T05:55:10.913653+0000","last_undegraded":"2024-03-12T05:55:10.913653+0000","last_fullsized":"2024-03-12T05:55:10.913653+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:09:29.422125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019722+0000","last_change":"2024-03-12T05:53:37.203252+0000","last_active":"2024-03-12T05:54:42.019722+0000","last_peered":"2024-03-12T05:54:42.019722+0000","last_clean":"2024-03-12T05:54:42.019722+0000","last_became_active":"2024-03-12T05:53:37.202916+0000","last_became_peered":"2024-03-12T05:53:37.202916+0000","last_unstale":"2024-03-12T05:54:42.019722+0000","last_undegraded":"2024-03-12T05:54:42.019722+0000","last_fullsized":"2024-03-12T05:54:42.019722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:09:34.629085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.916291+0000","last_change":"2024-03-12T05:54:42.021528+0000","last_active":"2024-03-12T05:55:10.916291+0000","last_peered":"2024-03-12T05:55:10.916291+0000","last_clean":"2024-03-12T05:55:10.916291+0000","last_became_active":"2024-03-12T05:54:42.021015+0000","last_became_peered":"2024-03-12T05:54:42.021015+0000","last_unstale":"2024-03-12T05:55:10.916291+0000","last_undegraded":"2024-03-12T05:55:10.916291+0000","last_fullsized":"2024-03-12T05:55:10.916291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:16.998017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":288,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:33.524254+0000","last_change":"2024-03-12T05:53:37.206271+0000","last_active":"2024-03-12T05:55:33.524254+0000","last_peered":"2024-03-12T05:55:33.524254+0000","last_clean":"2024-03-12T05:55:33.524254+0000","last_became_active":"2024-03-12T05:53:37.205920+0000","last_became_peered":"2024-03-12T05:53:37.205920+0000","last_unstale":"2024-03-12T05:55:33.524254+0000","last_undegraded":"2024-03-12T05:55:33.524254+0000","last_fullsized":"2024-03-12T05:55:33.524254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:30.140159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61603,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":236,"num_read_kb":178,"num_write":34,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.912256+0000","last_change":"2024-03-12T05:53:38.211397+0000","last_active":"2024-03-12T05:55:10.912256+0000","last_peered":"2024-03-12T05:55:10.912256+0000","last_clean":"2024-03-12T05:55:10.912256+0000","last_became_active":"2024-03-12T05:53:38.211045+0000","last_became_peered":"2024-03-12T05:53:38.211045+0000","last_unstale":"2024-03-12T05:55:10.912256+0000","last_undegraded":"2024-03-12T05:55:10.912256+0000","last_fullsized":"2024-03-12T05:55:10.912256+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:21:42.510436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019264+0000","last_change":"2024-03-12T05:53:37.210871+0000","last_active":"2024-03-12T05:54:42.019264+0000","last_peered":"2024-03-12T05:54:42.019264+0000","last_clean":"2024-03-12T05:54:42.019264+0000","last_became_active":"2024-03-12T05:53:37.210526+0000","last_became_peered":"2024-03-12T05:53:37.210526+0000","last_unstale":"2024-03-12T05:54:42.019264+0000","last_undegraded":"2024-03-12T05:54:42.019264+0000","last_fullsized":"2024-03-12T05:54:42.019264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:40:46.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.907189+0000","last_change":"2024-03-12T05:53:38.213751+0000","last_active":"2024-03-12T05:55:10.907189+0000","last_peered":"2024-03-12T05:55:10.907189+0000","last_clean":"2024-03-12T05:55:10.907189+0000","last_became_active":"2024-03-12T05:53:38.211558+0000","last_became_peered":"2024-03-12T05:53:38.211558+0000","last_unstale":"2024-03-12T05:55:10.907189+0000","last_undegraded":"2024-03-12T05:55:10.907189+0000","last_fullsized":"2024-03-12T05:55:10.907189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:40:44.194548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018330+0000","last_change":"2024-03-12T05:53:37.207368+0000","last_active":"2024-03-12T05:54:42.018330+0000","last_peered":"2024-03-12T05:54:42.018330+0000","last_clean":"2024-03-12T05:54:42.018330+0000","last_became_active":"2024-03-12T05:53:37.207066+0000","last_became_peered":"2024-03-12T05:53:37.207066+0000","last_unstale":"2024-03-12T05:54:42.018330+0000","last_undegraded":"2024-03-12T05:54:42.018330+0000","last_fullsized":"2024-03-12T05:54:42.018330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:28:58.155360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913837+0000","last_change":"2024-03-12T05:53:38.217079+0000","last_active":"2024-03-12T05:55:10.913837+0000","last_peered":"2024-03-12T05:55:10.913837+0000","last_clean":"2024-03-12T05:55:10.913837+0000","last_became_active":"2024-03-12T05:53:38.216757+0000","last_became_peered":"2024-03-12T05:53:38.216757+0000","last_unstale":"2024-03-12T05:55:10.913837+0000","last_undegraded":"2024-03-12T05:55:10.913837+0000","last_fullsized":"2024-03-12T05:55:10.913837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:48:13.125867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":140,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:34.555872+0000","last_change":"2024-03-12T05:53:37.202390+0000","last_active":"2024-03-12T05:55:34.555872+0000","last_peered":"2024-03-12T05:55:34.555872+0000","last_clean":"2024-03-12T05:55:34.555872+0000","last_became_active":"2024-03-12T05:53:37.202020+0000","last_became_peered":"2024-03-12T05:53:37.202020+0000","last_unstale":"2024-03-12T05:55:34.555872+0000","last_undegraded":"2024-03-12T05:55:34.555872+0000","last_fullsized":"2024-03-12T05:55:34.555872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:06:35.293227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":896,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.910544+0000","last_change":"2024-03-12T05:53:38.218464+0000","last_active":"2024-03-12T05:55:10.910544+0000","last_peered":"2024-03-12T05:55:10.910544+0000","last_clean":"2024-03-12T05:55:10.910544+0000","last_became_active":"2024-03-12T05:53:38.218136+0000","last_became_peered":"2024-03-12T05:53:38.218136+0000","last_unstale":"2024-03-12T05:55:10.910544+0000","last_undegraded":"2024-03-12T05:55:10.910544+0000","last_fullsized":"2024-03-12T05:55:10.910544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:31:05.409148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013820+0000","last_change":"2024-03-12T05:53:37.205161+0000","last_active":"2024-03-12T05:54:42.013820+0000","last_peered":"2024-03-12T05:54:42.013820+0000","last_clean":"2024-03-12T05:54:42.013820+0000","last_became_active":"2024-03-12T05:53:37.204799+0000","last_became_peered":"2024-03-12T05:53:37.204799+0000","last_unstale":"2024-03-12T05:54:42.013820+0000","last_undegraded":"2024-03-12T05:54:42.013820+0000","last_fullsized":"2024-03-12T05:54:42.013820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:29:32.065635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.900970+0000","last_change":"2024-03-12T05:53:38.216321+0000","last_active":"2024-03-12T05:55:10.900970+0000","last_peered":"2024-03-12T05:55:10.900970+0000","last_clean":"2024-03-12T05:55:10.900970+0000","last_became_active":"2024-03-12T05:53:38.215893+0000","last_became_peered":"2024-03-12T05:53:38.215893+0000","last_unstale":"2024-03-12T05:55:10.900970+0000","last_undegraded":"2024-03-12T05:55:10.900970+0000","last_fullsized":"2024-03-12T05:55:10.900970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:32:27.603930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014706+0000","last_change":"2024-03-12T05:53:37.203638+0000","last_active":"2024-03-12T05:54:42.014706+0000","last_peered":"2024-03-12T05:54:42.014706+0000","last_clean":"2024-03-12T05:54:42.014706+0000","last_became_active":"2024-03-12T05:53:37.203217+0000","last_became_peered":"2024-03-12T05:53:37.203217+0000","last_unstale":"2024-03-12T05:54:42.014706+0000","last_undegraded":"2024-03-12T05:54:42.014706+0000","last_fullsized":"2024-03-12T05:54:42.014706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:10.521197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904911+0000","last_change":"2024-03-12T05:53:38.210491+0000","last_active":"2024-03-12T05:55:10.904911+0000","last_peered":"2024-03-12T05:55:10.904911+0000","last_clean":"2024-03-12T05:55:10.904911+0000","last_became_active":"2024-03-12T05:53:38.210169+0000","last_became_peered":"2024-03-12T05:53:38.210169+0000","last_unstale":"2024-03-12T05:55:10.904911+0000","last_undegraded":"2024-03-12T05:55:10.904911+0000","last_fullsized":"2024-03-12T05:55:10.904911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:56:05.384834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013770+0000","last_change":"2024-03-12T05:53:37.204478+0000","last_active":"2024-03-12T05:54:42.013770+0000","last_peered":"2024-03-12T05:54:42.013770+0000","last_clean":"2024-03-12T05:54:42.013770+0000","last_became_active":"2024-03-12T05:53:37.204160+0000","last_became_peered":"2024-03-12T05:53:37.204160+0000","last_unstale":"2024-03-12T05:54:42.013770+0000","last_undegraded":"2024-03-12T05:54:42.013770+0000","last_fullsized":"2024-03-12T05:54:42.013770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:07.597731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.896102+0000","last_change":"2024-03-12T05:54:42.026040+0000","last_active":"2024-03-12T05:55:10.896102+0000","last_peered":"2024-03-12T05:55:10.896102+0000","last_clean":"2024-03-12T05:55:10.896102+0000","last_became_active":"2024-03-12T05:54:42.018230+0000","last_became_peered":"2024-03-12T05:54:42.018230+0000","last_unstale":"2024-03-12T05:55:10.896102+0000","last_undegraded":"2024-03-12T05:55:10.896102+0000","last_fullsized":"2024-03-12T05:55:10.896102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:32:28.703339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687385+0000","last_change":"2024-03-12T05:53:37.205983+0000","last_active":"2024-03-12T05:54:42.687385+0000","last_peered":"2024-03-12T05:54:42.687385+0000","last_clean":"2024-03-12T05:54:42.687385+0000","last_became_active":"2024-03-12T05:53:37.205629+0000","last_became_peered":"2024-03-12T05:53:37.205629+0000","last_unstale":"2024-03-12T05:54:42.687385+0000","last_undegraded":"2024-03-12T05:54:42.687385+0000","last_fullsized":"2024-03-12T05:54:42.687385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:37.928623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.908778+0000","last_change":"2024-03-12T05:53:38.213752+0000","last_active":"2024-03-12T05:55:10.908778+0000","last_peered":"2024-03-12T05:55:10.908778+0000","last_clean":"2024-03-12T05:55:10.908778+0000","last_became_active":"2024-03-12T05:53:38.213407+0000","last_became_peered":"2024-03-12T05:53:38.213407+0000","last_unstale":"2024-03-12T05:55:10.908778+0000","last_undegraded":"2024-03-12T05:55:10.908778+0000","last_fullsized":"2024-03-12T05:55:10.908778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:21:15.703455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.907245+0000","last_change":"2024-03-12T05:54:42.022196+0000","last_active":"2024-03-12T05:55:10.907245+0000","last_peered":"2024-03-12T05:55:10.907245+0000","last_clean":"2024-03-12T05:55:10.907245+0000","last_became_active":"2024-03-12T05:54:42.021774+0000","last_became_peered":"2024-03-12T05:54:42.021774+0000","last_unstale":"2024-03-12T05:55:10.907245+0000","last_undegraded":"2024-03-12T05:55:10.907245+0000","last_fullsized":"2024-03-12T05:55:10.907245+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:34:39.290359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:32.551476+0000","last_change":"2024-03-12T05:53:37.202782+0000","last_active":"2024-03-12T05:55:32.551476+0000","last_peered":"2024-03-12T05:55:32.551476+0000","last_clean":"2024-03-12T05:55:32.551476+0000","last_became_active":"2024-03-12T05:53:37.202439+0000","last_became_peered":"2024-03-12T05:53:37.202439+0000","last_unstale":"2024-03-12T05:55:32.551476+0000","last_undegraded":"2024-03-12T05:55:32.551476+0000","last_fullsized":"2024-03-12T05:55:32.551476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:07:18.939629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.914766+0000","last_change":"2024-03-12T05:53:38.217993+0000","last_active":"2024-03-12T05:55:10.914766+0000","last_peered":"2024-03-12T05:55:10.914766+0000","last_clean":"2024-03-12T05:55:10.914766+0000","last_became_active":"2024-03-12T05:53:38.217613+0000","last_became_peered":"2024-03-12T05:53:38.217613+0000","last_unstale":"2024-03-12T05:55:10.914766+0000","last_undegraded":"2024-03-12T05:55:10.914766+0000","last_fullsized":"2024-03-12T05:55:10.914766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:49:50.960004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.478908+0000","last_change":"2024-03-12T05:53:37.203662+0000","last_active":"2024-03-12T05:54:42.478908+0000","last_peered":"2024-03-12T05:54:42.478908+0000","last_clean":"2024-03-12T05:54:42.478908+0000","last_became_active":"2024-03-12T05:53:37.203309+0000","last_became_peered":"2024-03-12T05:53:37.203309+0000","last_unstale":"2024-03-12T05:54:42.478908+0000","last_undegraded":"2024-03-12T05:54:42.478908+0000","last_fullsized":"2024-03-12T05:54:42.478908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:47:24.603686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687337+0000","last_change":"2024-03-12T05:52:27.278524+0000","last_active":"2024-03-12T05:54:42.687337+0000","last_peered":"2024-03-12T05:54:42.687337+0000","last_clean":"2024-03-12T05:54:42.687337+0000","last_became_active":"2024-03-12T05:52:27.278010+0000","last_became_peered":"2024-03-12T05:52:27.278010+0000","last_unstale":"2024-03-12T05:54:42.687337+0000","last_undegraded":"2024-03-12T05:54:42.687337+0000","last_fullsized":"2024-03-12T05:54:42.687337+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:49:09.175725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:49:09.175725+0000","last_clean_scrub_stamp":"2024-03-12T05:49:09.175725+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:24:40.380724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.901203+0000","last_change":"2024-03-12T05:53:38.219451+0000","last_active":"2024-03-12T05:55:10.901203+0000","last_peered":"2024-03-12T05:55:10.901203+0000","last_clean":"2024-03-12T05:55:10.901203+0000","last_became_active":"2024-03-12T05:53:38.218840+0000","last_became_peered":"2024-03-12T05:53:38.218840+0000","last_unstale":"2024-03-12T05:55:10.901203+0000","last_undegraded":"2024-03-12T05:55:10.901203+0000","last_fullsized":"2024-03-12T05:55:10.901203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:05.595822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687398+0000","last_change":"2024-03-12T05:53:37.210352+0000","last_active":"2024-03-12T05:54:42.687398+0000","last_peered":"2024-03-12T05:54:42.687398+0000","last_clean":"2024-03-12T05:54:42.687398+0000","last_became_active":"2024-03-12T05:53:37.210002+0000","last_became_peered":"2024-03-12T05:53:37.210002+0000","last_unstale":"2024-03-12T05:54:42.687398+0000","last_undegraded":"2024-03-12T05:54:42.687398+0000","last_fullsized":"2024-03-12T05:54:42.687398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:49:52.368411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.022961+0000","last_change":"2024-03-12T05:54:42.022489+0000","last_active":"2024-03-12T05:54:42.022961+0000","last_peered":"2024-03-12T05:54:42.022961+0000","last_clean":"2024-03-12T05:54:42.022961+0000","last_became_active":"2024-03-12T05:54:42.016400+0000","last_became_peered":"2024-03-12T05:54:42.016400+0000","last_unstale":"2024-03-12T05:54:42.022961+0000","last_undegraded":"2024-03-12T05:54:42.022961+0000","last_fullsized":"2024-03-12T05:54:42.022961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:32:05.713420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.886986+0000","last_change":"2024-03-12T05:53:38.203951+0000","last_active":"2024-03-12T05:55:10.886986+0000","last_peered":"2024-03-12T05:55:10.886986+0000","last_clean":"2024-03-12T05:55:10.886986+0000","last_became_active":"2024-03-12T05:53:38.203617+0000","last_became_peered":"2024-03-12T05:53:38.203617+0000","last_unstale":"2024-03-12T05:55:10.886986+0000","last_undegraded":"2024-03-12T05:55:10.886986+0000","last_fullsized":"2024-03-12T05:55:10.886986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:19.138220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:11.379756+0000","last_change":"2024-03-12T05:53:37.203770+0000","last_active":"2024-03-12T05:55:11.379756+0000","last_peered":"2024-03-12T05:55:11.379756+0000","last_clean":"2024-03-12T05:55:11.379756+0000","last_became_active":"2024-03-12T05:53:37.203456+0000","last_became_peered":"2024-03-12T05:53:37.203456+0000","last_unstale":"2024-03-12T05:55:11.379756+0000","last_undegraded":"2024-03-12T05:55:11.379756+0000","last_fullsized":"2024-03-12T05:55:11.379756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:32.291536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.911027+0000","last_change":"2024-03-12T05:53:38.214872+0000","last_active":"2024-03-12T05:55:10.911027+0000","last_peered":"2024-03-12T05:55:10.911027+0000","last_clean":"2024-03-12T05:55:10.911027+0000","last_became_active":"2024-03-12T05:53:38.214139+0000","last_became_peered":"2024-03-12T05:53:38.214139+0000","last_unstale":"2024-03-12T05:55:10.911027+0000","last_undegraded":"2024-03-12T05:55:10.911027+0000","last_fullsized":"2024-03-12T05:55:10.911027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:20:21.808039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.479035+0000","last_change":"2024-03-12T05:53:37.203852+0000","last_active":"2024-03-12T05:54:42.479035+0000","last_peered":"2024-03-12T05:54:42.479035+0000","last_clean":"2024-03-12T05:54:42.479035+0000","last_became_active":"2024-03-12T05:53:37.203416+0000","last_became_peered":"2024-03-12T05:53:37.203416+0000","last_unstale":"2024-03-12T05:54:42.479035+0000","last_undegraded":"2024-03-12T05:54:42.479035+0000","last_fullsized":"2024-03-12T05:54:42.479035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:09:08.095957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.910705+0000","last_change":"2024-03-12T05:54:42.688395+0000","last_active":"2024-03-12T05:55:10.910705+0000","last_peered":"2024-03-12T05:55:10.910705+0000","last_clean":"2024-03-12T05:55:10.910705+0000","last_became_active":"2024-03-12T05:54:42.688094+0000","last_became_peered":"2024-03-12T05:54:42.688094+0000","last_unstale":"2024-03-12T05:55:10.910705+0000","last_undegraded":"2024-03-12T05:55:10.910705+0000","last_fullsized":"2024-03-12T05:55:10.910705+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:29:42.921726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.025218+0000","last_change":"2024-03-12T05:54:42.025218+0000","last_active":"2024-03-12T05:54:42.025218+0000","last_peered":"2024-03-12T05:54:42.025218+0000","last_clean":"2024-03-12T05:54:42.025218+0000","last_became_active":"2024-03-12T05:54:42.021022+0000","last_became_peered":"2024-03-12T05:54:42.021022+0000","last_unstale":"2024-03-12T05:54:42.025218+0000","last_undegraded":"2024-03-12T05:54:42.025218+0000","last_fullsized":"2024-03-12T05:54:42.025218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:20:57.232145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.874050+0000","last_change":"2024-03-12T05:53:38.207757+0000","last_active":"2024-03-12T05:55:10.874050+0000","last_peered":"2024-03-12T05:55:10.874050+0000","last_clean":"2024-03-12T05:55:10.874050+0000","last_became_active":"2024-03-12T05:53:38.207424+0000","last_became_peered":"2024-03-12T05:53:38.207424+0000","last_unstale":"2024-03-12T05:55:10.874050+0000","last_undegraded":"2024-03-12T05:55:10.874050+0000","last_fullsized":"2024-03-12T05:55:10.874050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:33:16.017907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016731+0000","last_change":"2024-03-12T05:53:37.208512+0000","last_active":"2024-03-12T05:54:42.016731+0000","last_peered":"2024-03-12T05:54:42.016731+0000","last_clean":"2024-03-12T05:54:42.016731+0000","last_became_active":"2024-03-12T05:53:37.207628+0000","last_became_peered":"2024-03-12T05:53:37.207628+0000","last_unstale":"2024-03-12T05:54:42.016731+0000","last_undegraded":"2024-03-12T05:54:42.016731+0000","last_fullsized":"2024-03-12T05:54:42.016731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:25:28.825779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.908590+0000","last_change":"2024-03-12T05:53:38.205654+0000","last_active":"2024-03-12T05:55:10.908590+0000","last_peered":"2024-03-12T05:55:10.908590+0000","last_clean":"2024-03-12T05:55:10.908590+0000","last_became_active":"2024-03-12T05:53:38.205325+0000","last_became_peered":"2024-03-12T05:53:38.205325+0000","last_unstale":"2024-03-12T05:55:10.908590+0000","last_undegraded":"2024-03-12T05:55:10.908590+0000","last_fullsized":"2024-03-12T05:55:10.908590+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:33:35.668625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019632+0000","last_change":"2024-03-12T05:53:37.211430+0000","last_active":"2024-03-12T05:54:42.019632+0000","last_peered":"2024-03-12T05:54:42.019632+0000","last_clean":"2024-03-12T05:54:42.019632+0000","last_became_active":"2024-03-12T05:53:37.210501+0000","last_became_peered":"2024-03-12T05:53:37.210501+0000","last_unstale":"2024-03-12T05:54:42.019632+0000","last_undegraded":"2024-03-12T05:54:42.019632+0000","last_fullsized":"2024-03-12T05:54:42.019632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:48:33.555134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902337+0000","last_change":"2024-03-12T05:53:38.212837+0000","last_active":"2024-03-12T05:55:10.902337+0000","last_peered":"2024-03-12T05:55:10.902337+0000","last_clean":"2024-03-12T05:55:10.902337+0000","last_became_active":"2024-03-12T05:53:38.212513+0000","last_became_peered":"2024-03-12T05:53:38.212513+0000","last_unstale":"2024-03-12T05:55:10.902337+0000","last_undegraded":"2024-03-12T05:55:10.902337+0000","last_fullsized":"2024-03-12T05:55:10.902337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:47:33.879954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014092+0000","last_change":"2024-03-12T05:53:37.207465+0000","last_active":"2024-03-12T05:54:42.014092+0000","last_peered":"2024-03-12T05:54:42.014092+0000","last_clean":"2024-03-12T05:54:42.014092+0000","last_became_active":"2024-03-12T05:53:37.207107+0000","last_became_peered":"2024-03-12T05:53:37.207107+0000","last_unstale":"2024-03-12T05:54:42.014092+0000","last_undegraded":"2024-03-12T05:54:42.014092+0000","last_fullsized":"2024-03-12T05:54:42.014092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:34.018856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917908+0000","last_change":"2024-03-12T05:53:38.221470+0000","last_active":"2024-03-12T05:55:10.917908+0000","last_peered":"2024-03-12T05:55:10.917908+0000","last_clean":"2024-03-12T05:55:10.917908+0000","last_became_active":"2024-03-12T05:53:38.221061+0000","last_became_peered":"2024-03-12T05:53:38.221061+0000","last_unstale":"2024-03-12T05:55:10.917908+0000","last_undegraded":"2024-03-12T05:55:10.917908+0000","last_fullsized":"2024-03-12T05:55:10.917908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:01:54.148877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019769+0000","last_change":"2024-03-12T05:53:37.203822+0000","last_active":"2024-03-12T05:54:42.019769+0000","last_peered":"2024-03-12T05:54:42.019769+0000","last_clean":"2024-03-12T05:54:42.019769+0000","last_became_active":"2024-03-12T05:53:37.203431+0000","last_became_peered":"2024-03-12T05:53:37.203431+0000","last_unstale":"2024-03-12T05:54:42.019769+0000","last_undegraded":"2024-03-12T05:54:42.019769+0000","last_fullsized":"2024-03-12T05:54:42.019769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:25:04.447261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918179+0000","last_change":"2024-03-12T05:53:38.222045+0000","last_active":"2024-03-12T05:55:10.918179+0000","last_peered":"2024-03-12T05:55:10.918179+0000","last_clean":"2024-03-12T05:55:10.918179+0000","last_became_active":"2024-03-12T05:53:38.220904+0000","last_became_peered":"2024-03-12T05:53:38.220904+0000","last_unstale":"2024-03-12T05:55:10.918179+0000","last_undegraded":"2024-03-12T05:55:10.918179+0000","last_fullsized":"2024-03-12T05:55:10.918179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:30:43.964599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019437+0000","last_change":"2024-03-12T05:53:37.206753+0000","last_active":"2024-03-12T05:54:42.019437+0000","last_peered":"2024-03-12T05:54:42.019437+0000","last_clean":"2024-03-12T05:54:42.019437+0000","last_became_active":"2024-03-12T05:53:37.206078+0000","last_became_peered":"2024-03-12T05:53:37.206078+0000","last_unstale":"2024-03-12T05:54:42.019437+0000","last_undegraded":"2024-03-12T05:54:42.019437+0000","last_fullsized":"2024-03-12T05:54:42.019437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:42:18.859528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.909834+0000","last_change":"2024-03-12T05:53:38.215671+0000","last_active":"2024-03-12T05:55:10.909834+0000","last_peered":"2024-03-12T05:55:10.909834+0000","last_clean":"2024-03-12T05:55:10.909834+0000","last_became_active":"2024-03-12T05:53:38.215309+0000","last_became_peered":"2024-03-12T05:53:38.215309+0000","last_unstale":"2024-03-12T05:55:10.909834+0000","last_undegraded":"2024-03-12T05:55:10.909834+0000","last_fullsized":"2024-03-12T05:55:10.909834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:50:12.176935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017343+0000","last_change":"2024-03-12T05:53:37.205627+0000","last_active":"2024-03-12T05:54:42.017343+0000","last_peered":"2024-03-12T05:54:42.017343+0000","last_clean":"2024-03-12T05:54:42.017343+0000","last_became_active":"2024-03-12T05:53:37.205289+0000","last_became_peered":"2024-03-12T05:53:37.205289+0000","last_unstale":"2024-03-12T05:54:42.017343+0000","last_undegraded":"2024-03-12T05:54:42.017343+0000","last_fullsized":"2024-03-12T05:54:42.017343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:56.682746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913506+0000","last_change":"2024-03-12T05:53:38.214376+0000","last_active":"2024-03-12T05:55:10.913506+0000","last_peered":"2024-03-12T05:55:10.913506+0000","last_clean":"2024-03-12T05:55:10.913506+0000","last_became_active":"2024-03-12T05:53:38.214011+0000","last_became_peered":"2024-03-12T05:53:38.214011+0000","last_unstale":"2024-03-12T05:55:10.913506+0000","last_undegraded":"2024-03-12T05:55:10.913506+0000","last_fullsized":"2024-03-12T05:55:10.913506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:32.605216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.908362+0000","last_change":"2024-03-12T05:53:38.205337+0000","last_active":"2024-03-12T05:55:10.908362+0000","last_peered":"2024-03-12T05:55:10.908362+0000","last_clean":"2024-03-12T05:55:10.908362+0000","last_became_active":"2024-03-12T05:53:38.205005+0000","last_became_peered":"2024-03-12T05:53:38.205005+0000","last_unstale":"2024-03-12T05:55:10.908362+0000","last_undegraded":"2024-03-12T05:55:10.908362+0000","last_fullsized":"2024-03-12T05:55:10.908362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:44:16.296370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.907222+0000","last_change":"2024-03-12T05:53:38.214694+0000","last_active":"2024-03-12T05:55:10.907222+0000","last_peered":"2024-03-12T05:55:10.907222+0000","last_clean":"2024-03-12T05:55:10.907222+0000","last_became_active":"2024-03-12T05:53:38.214285+0000","last_became_peered":"2024-03-12T05:53:38.214285+0000","last_unstale":"2024-03-12T05:55:10.907222+0000","last_undegraded":"2024-03-12T05:55:10.907222+0000","last_fullsized":"2024-03-12T05:55:10.907222+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:28:32.253332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902212+0000","last_change":"2024-03-12T05:53:38.213069+0000","last_active":"2024-03-12T05:55:10.902212+0000","last_peered":"2024-03-12T05:55:10.902212+0000","last_clean":"2024-03-12T05:55:10.902212+0000","last_became_active":"2024-03-12T05:53:38.212694+0000","last_became_peered":"2024-03-12T05:53:38.212694+0000","last_unstale":"2024-03-12T05:55:10.902212+0000","last_undegraded":"2024-03-12T05:55:10.902212+0000","last_fullsized":"2024-03-12T05:55:10.902212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:26:29.350292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913785+0000","last_change":"2024-03-12T05:53:38.212084+0000","last_active":"2024-03-12T05:55:10.913785+0000","last_peered":"2024-03-12T05:55:10.913785+0000","last_clean":"2024-03-12T05:55:10.913785+0000","last_became_active":"2024-03-12T05:53:38.211750+0000","last_became_peered":"2024-03-12T05:53:38.211750+0000","last_unstale":"2024-03-12T05:55:10.913785+0000","last_undegraded":"2024-03-12T05:55:10.913785+0000","last_fullsized":"2024-03-12T05:55:10.913785+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:05:02.099121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.899211+0000","last_change":"2024-03-12T05:53:38.212071+0000","last_active":"2024-03-12T05:55:10.899211+0000","last_peered":"2024-03-12T05:55:10.899211+0000","last_clean":"2024-03-12T05:55:10.899211+0000","last_became_active":"2024-03-12T05:53:38.211722+0000","last_became_peered":"2024-03-12T05:53:38.211722+0000","last_unstale":"2024-03-12T05:55:10.899211+0000","last_undegraded":"2024-03-12T05:55:10.899211+0000","last_fullsized":"2024-03-12T05:55:10.899211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:19:46.581837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.895503+0000","last_change":"2024-03-12T05:54:42.015717+0000","last_active":"2024-03-12T05:55:10.895503+0000","last_peered":"2024-03-12T05:55:10.895503+0000","last_clean":"2024-03-12T05:55:10.895503+0000","last_became_active":"2024-03-12T05:54:42.015266+0000","last_became_peered":"2024-03-12T05:54:42.015266+0000","last_unstale":"2024-03-12T05:55:10.895503+0000","last_undegraded":"2024-03-12T05:55:10.895503+0000","last_fullsized":"2024-03-12T05:55:10.895503+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:33:48.250458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.915527+0000","last_change":"2024-03-12T05:53:38.208018+0000","last_active":"2024-03-12T05:55:10.915527+0000","last_peered":"2024-03-12T05:55:10.915527+0000","last_clean":"2024-03-12T05:55:10.915527+0000","last_became_active":"2024-03-12T05:53:38.207682+0000","last_became_peered":"2024-03-12T05:53:38.207682+0000","last_unstale":"2024-03-12T05:55:10.915527+0000","last_undegraded":"2024-03-12T05:55:10.915527+0000","last_fullsized":"2024-03-12T05:55:10.915527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:56:10.903917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902615+0000","last_change":"2024-03-12T05:53:38.217540+0000","last_active":"2024-03-12T05:55:10.902615+0000","last_peered":"2024-03-12T05:55:10.902615+0000","last_clean":"2024-03-12T05:55:10.902615+0000","last_became_active":"2024-03-12T05:53:38.217144+0000","last_became_peered":"2024-03-12T05:53:38.217144+0000","last_unstale":"2024-03-12T05:55:10.902615+0000","last_undegraded":"2024-03-12T05:55:10.902615+0000","last_fullsized":"2024-03-12T05:55:10.902615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:46:18.905909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902398+0000","last_change":"2024-03-12T05:53:38.215006+0000","last_active":"2024-03-12T05:55:10.902398+0000","last_peered":"2024-03-12T05:55:10.902398+0000","last_clean":"2024-03-12T05:55:10.902398+0000","last_became_active":"2024-03-12T05:53:38.213756+0000","last_became_peered":"2024-03-12T05:53:38.213756+0000","last_unstale":"2024-03-12T05:55:10.902398+0000","last_undegraded":"2024-03-12T05:55:10.902398+0000","last_fullsized":"2024-03-12T05:55:10.902398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:14:57.642381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902674+0000","last_change":"2024-03-12T05:53:38.216096+0000","last_active":"2024-03-12T05:55:10.902674+0000","last_peered":"2024-03-12T05:55:10.902674+0000","last_clean":"2024-03-12T05:55:10.902674+0000","last_became_active":"2024-03-12T05:53:38.215153+0000","last_became_peered":"2024-03-12T05:53:38.215153+0000","last_unstale":"2024-03-12T05:55:10.902674+0000","last_undegraded":"2024-03-12T05:55:10.902674+0000","last_fullsized":"2024-03-12T05:55:10.902674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:41:03.893641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918844+0000","last_change":"2024-03-12T05:53:38.219165+0000","last_active":"2024-03-12T05:55:10.918844+0000","last_peered":"2024-03-12T05:55:10.918844+0000","last_clean":"2024-03-12T05:55:10.918844+0000","last_became_active":"2024-03-12T05:53:38.218579+0000","last_became_peered":"2024-03-12T05:53:38.218579+0000","last_unstale":"2024-03-12T05:55:10.918844+0000","last_undegraded":"2024-03-12T05:55:10.918844+0000","last_fullsized":"2024-03-12T05:55:10.918844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:06:37.531293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.907783+0000","last_change":"2024-03-12T05:53:38.212685+0000","last_active":"2024-03-12T05:55:10.907783+0000","last_peered":"2024-03-12T05:55:10.907783+0000","last_clean":"2024-03-12T05:55:10.907783+0000","last_became_active":"2024-03-12T05:53:38.211106+0000","last_became_peered":"2024-03-12T05:53:38.211106+0000","last_unstale":"2024-03-12T05:55:10.907783+0000","last_undegraded":"2024-03-12T05:55:10.907783+0000","last_fullsized":"2024-03-12T05:55:10.907783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:26:00.339270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904578+0000","last_change":"2024-03-12T05:53:38.218402+0000","last_active":"2024-03-12T05:55:10.904578+0000","last_peered":"2024-03-12T05:55:10.904578+0000","last_clean":"2024-03-12T05:55:10.904578+0000","last_became_active":"2024-03-12T05:53:38.217699+0000","last_became_peered":"2024-03-12T05:53:38.217699+0000","last_unstale":"2024-03-12T05:55:10.904578+0000","last_undegraded":"2024-03-12T05:55:10.904578+0000","last_fullsized":"2024-03-12T05:55:10.904578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:23:03.221710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66053,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":709,"num_read_kb":423,"num_write":171,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":198159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147,"ondisk_log_size":147,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351013,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":461631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612650,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32788,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712364,"statfs":{"total":95995035648,"available":95961460736,"internally_reserved":0,"allocated":1650688,"data_stored":1115001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776175,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":461962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037812,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":461743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32704,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712448,"statfs":{"total":95995035648,"available":95961546752,"internally_reserved":0,"allocated":1564672,"data_stored":1054144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":461234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":522630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":462505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":524512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":461428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":461979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1576960,"data_stored":1053520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T05:55:37.457 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-12T05:55:37.458 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-12T05:55:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:37 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/415113663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:37 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/982607075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:37.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:37 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1135088622' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:37.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.592+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1717777636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a44069050 msgr2=0x7f3a440ff8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.592+0000 7f3a4c14f640 1 --2- 172.21.15.76:0/1717777636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a44069050 0x7f3a440ff8b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009ca0 tx=0x7f3a4002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.592+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1717777636 shutdown_connections 2024-03-12T05:55:37.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.592+0000 7f3a4c14f640 1 --2- 172.21.15.76:0/1717777636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a44100aa0 0x7f3a44100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.592+0000 7f3a4c14f640 1 --2- 172.21.15.76:0/1717777636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a44069500 0x7f3a440ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.592+0000 7f3a4c14f640 1 --2- 172.21.15.76:0/1717777636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a44069050 0x7f3a440ff8b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.592+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1717777636 >> 172.21.15.76:0/1717777636 conn(0x7f3a440fae00 msgr2=0x7f3a440fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:37.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.593+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1717777636 shutdown_connections 2024-03-12T05:55:37.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.593+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1717777636 wait complete. 2024-03-12T05:55:37.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.593+0000 7f3a4c14f640 1 Processor -- start 2024-03-12T05:55:37.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.593+0000 7f3a4c14f640 1 -- start start 2024-03-12T05:55:37.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.593+0000 7f3a4c14f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a44069050 0x7f3a441960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:37.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.593+0000 7f3a4c14f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a44069500 0x7f3a44196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:37.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.593+0000 7f3a4c14f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a44100aa0 0x7f3a4419d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:37.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.593+0000 7f3a4c14f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4419dbc0 con 0x7f3a44069050 2024-03-12T05:55:37.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.593+0000 7f3a4c14f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4419dd30 con 0x7f3a44100aa0 2024-03-12T05:55:37.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.593+0000 7f3a4c14f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4419def0 con 0x7f3a44069500 2024-03-12T05:55:37.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.593+0000 7f3a49eeb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a44069050 0x7f3a441960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:37.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.593+0000 7f3a49eeb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a44069050 0x7f3a441960d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47414/0 (socket says 172.21.15.76:47414) 2024-03-12T05:55:37.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.593+0000 7f3a49eeb640 1 -- 172.21.15.76:0/1912884245 learned_addr learned my addr 172.21.15.76:0/1912884245 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:37.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.594+0000 7f3a49eeb640 1 -- 172.21.15.76:0/1912884245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a44069500 msgr2=0x7f3a44196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.594+0000 7f3a49eeb640 1 --2- 172.21.15.76:0/1912884245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a44069500 0x7f3a44196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.594+0000 7f3a49eeb640 1 -- 172.21.15.76:0/1912884245 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a44100aa0 msgr2=0x7f3a4419d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.594+0000 7f3a49eeb640 1 --2- 172.21.15.76:0/1912884245 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a44100aa0 0x7f3a4419d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.594+0000 7f3a49eeb640 1 -- 172.21.15.76:0/1912884245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a40009900 con 0x7f3a44069050 2024-03-12T05:55:37.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.594+0000 7f3a49eeb640 1 --2- 172.21.15.76:0/1912884245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a44069050 0x7f3a441960d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3a40037040 tx=0x7f3a40034d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:37.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.594+0000 7f3a3affd640 1 -- 172.21.15.76:0/1912884245 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40032eb0 con 0x7f3a44069050 2024-03-12T05:55:37.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.594+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1912884245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4419e170 con 0x7f3a44069050 2024-03-12T05:55:37.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.594+0000 7f3a3affd640 1 -- 172.21.15.76:0/1912884245 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a40003e50 con 0x7f3a44069050 2024-03-12T05:55:37.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.594+0000 7f3a3affd640 1 -- 172.21.15.76:0/1912884245 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40040470 con 0x7f3a44069050 2024-03-12T05:55:37.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.594+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1912884245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4419e620 con 0x7f3a44069050 2024-03-12T05:55:37.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.595+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1912884245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a44107610 con 0x7f3a44069050 2024-03-12T05:55:37.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.596+0000 7f3a3affd640 1 -- 172.21.15.76:0/1912884245 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f3a4000a040 con 0x7f3a44069050 2024-03-12T05:55:37.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.597+0000 7f3a3affd640 1 --2- 172.21.15.76:0/1912884245 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3a300a1a30 0x7f3a300a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:37.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.597+0000 7f3a496ea640 1 --2- 172.21.15.76:0/1912884245 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3a300a1a30 0x7f3a300a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:37.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.597+0000 7f3a3affd640 1 -- 172.21.15.76:0/1912884245 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3a400efff0 con 0x7f3a44069050 2024-03-12T05:55:37.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.597+0000 7f3a496ea640 1 --2- 172.21.15.76:0/1912884245 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3a300a1a30 0x7f3a300a3ef0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3a441975e0 tx=0x7f3a34005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:37.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.601+0000 7f3a3affd640 1 -- 172.21.15.76:0/1912884245 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3a400b9210 con 0x7f3a44069050 2024-03-12T05:55:37.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.772+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1912884245 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a4402cf90 con 0x7f3a300a1a30 2024-03-12T05:55:37.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.779+0000 7f3a3affd640 1 -- 172.21.15.76:0/1912884245 <== mgr.14152 v2:172.21.15.76:6800/1718825251 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258839 (secure 0 0 0) 0x7f3a4402cf90 con 0x7f3a300a1a30 2024-03-12T05:55:37.785 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:37.786 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2024-03-12T05:55:37.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.782+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1912884245 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3a300a1a30 msgr2=0x7f3a300a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.782+0000 7f3a4c14f640 1 --2- 172.21.15.76:0/1912884245 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3a300a1a30 0x7f3a300a3ef0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3a441975e0 tx=0x7f3a34005c70 comp rx=0 tx=0).stop 2024-03-12T05:55:37.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.782+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1912884245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a44069050 msgr2=0x7f3a441960d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.782+0000 7f3a4c14f640 1 --2- 172.21.15.76:0/1912884245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a44069050 0x7f3a441960d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3a40037040 tx=0x7f3a40034d40 comp rx=0 tx=0).stop 2024-03-12T05:55:37.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.782+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1912884245 shutdown_connections 2024-03-12T05:55:37.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.782+0000 7f3a4c14f640 1 --2- 172.21.15.76:0/1912884245 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f3a300a1a30 0x7f3a300a3ef0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.782+0000 7f3a4c14f640 1 --2- 172.21.15.76:0/1912884245 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a44100aa0 0x7f3a4419d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.782+0000 7f3a4c14f640 1 --2- 172.21.15.76:0/1912884245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a44069500 0x7f3a44196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.782+0000 7f3a4c14f640 1 --2- 172.21.15.76:0/1912884245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a44069050 0x7f3a441960d0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.782+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1912884245 >> 172.21.15.76:0/1912884245 conn(0x7f3a440fae00 msgr2=0x7f3a440fc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:37.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.783+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1912884245 shutdown_connections 2024-03-12T05:55:37.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.783+0000 7f3a4c14f640 1 -- 172.21.15.76:0/1912884245 wait complete. 2024-03-12T05:55:37.797 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-03-12T05:55:36.892489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656655,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":845,"num_read_kb":542,"num_write":354,"num_write_kb":3059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180875,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":786,"ondisk_log_size":786,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":374940,"kb_used_data":13308,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566884,"statfs":{"total":1151940427776,"available":1151556489216,"internally_reserved":0,"allocated":13627392,"data_stored":7502289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":31,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002794"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904708+0000","last_change":"2024-03-12T05:53:38.207712+0000","last_active":"2024-03-12T05:55:10.904708+0000","last_peered":"2024-03-12T05:55:10.904708+0000","last_clean":"2024-03-12T05:55:10.904708+0000","last_became_active":"2024-03-12T05:53:38.207362+0000","last_became_peered":"2024-03-12T05:53:38.207362+0000","last_unstale":"2024-03-12T05:55:10.904708+0000","last_undegraded":"2024-03-12T05:55:10.904708+0000","last_fullsized":"2024-03-12T05:55:10.904708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:19.518959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.903191+0000","last_change":"2024-03-12T05:53:38.206877+0000","last_active":"2024-03-12T05:55:10.903191+0000","last_peered":"2024-03-12T05:55:10.903191+0000","last_clean":"2024-03-12T05:55:10.903191+0000","last_became_active":"2024-03-12T05:53:38.206468+0000","last_became_peered":"2024-03-12T05:53:38.206468+0000","last_unstale":"2024-03-12T05:55:10.903191+0000","last_undegraded":"2024-03-12T05:55:10.903191+0000","last_fullsized":"2024-03-12T05:55:10.903191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:02:45.138265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917190+0000","last_change":"2024-03-12T05:53:38.214979+0000","last_active":"2024-03-12T05:55:10.917190+0000","last_peered":"2024-03-12T05:55:10.917190+0000","last_clean":"2024-03-12T05:55:10.917190+0000","last_became_active":"2024-03-12T05:53:38.214580+0000","last_became_peered":"2024-03-12T05:53:38.214580+0000","last_unstale":"2024-03-12T05:55:10.917190+0000","last_undegraded":"2024-03-12T05:55:10.917190+0000","last_fullsized":"2024-03-12T05:55:10.917190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:39:58.989757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902953+0000","last_change":"2024-03-12T05:53:38.218645+0000","last_active":"2024-03-12T05:55:10.902953+0000","last_peered":"2024-03-12T05:55:10.902953+0000","last_clean":"2024-03-12T05:55:10.902953+0000","last_became_active":"2024-03-12T05:53:38.218254+0000","last_became_peered":"2024-03-12T05:53:38.218254+0000","last_unstale":"2024-03-12T05:55:10.902953+0000","last_undegraded":"2024-03-12T05:55:10.902953+0000","last_fullsized":"2024-03-12T05:55:10.902953+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:46:09.544989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.894649+0000","last_change":"2024-03-12T05:53:38.214588+0000","last_active":"2024-03-12T05:55:10.894649+0000","last_peered":"2024-03-12T05:55:10.894649+0000","last_clean":"2024-03-12T05:55:10.894649+0000","last_became_active":"2024-03-12T05:53:38.214192+0000","last_became_peered":"2024-03-12T05:53:38.214192+0000","last_unstale":"2024-03-12T05:55:10.894649+0000","last_undegraded":"2024-03-12T05:55:10.894649+0000","last_fullsized":"2024-03-12T05:55:10.894649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:54:04.880901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902295+0000","last_change":"2024-03-12T05:53:38.214315+0000","last_active":"2024-03-12T05:55:10.902295+0000","last_peered":"2024-03-12T05:55:10.902295+0000","last_clean":"2024-03-12T05:55:10.902295+0000","last_became_active":"2024-03-12T05:53:38.213881+0000","last_became_peered":"2024-03-12T05:53:38.213881+0000","last_unstale":"2024-03-12T05:55:10.902295+0000","last_undegraded":"2024-03-12T05:55:10.902295+0000","last_fullsized":"2024-03-12T05:55:10.902295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:11:32.834418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.908724+0000","last_change":"2024-03-12T05:53:38.214915+0000","last_active":"2024-03-12T05:55:10.908724+0000","last_peered":"2024-03-12T05:55:10.908724+0000","last_clean":"2024-03-12T05:55:10.908724+0000","last_became_active":"2024-03-12T05:53:38.214592+0000","last_became_peered":"2024-03-12T05:53:38.214592+0000","last_unstale":"2024-03-12T05:55:10.908724+0000","last_undegraded":"2024-03-12T05:55:10.908724+0000","last_fullsized":"2024-03-12T05:55:10.908724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:11.548585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.909127+0000","last_change":"2024-03-12T05:53:38.219201+0000","last_active":"2024-03-12T05:55:10.909127+0000","last_peered":"2024-03-12T05:55:10.909127+0000","last_clean":"2024-03-12T05:55:10.909127+0000","last_became_active":"2024-03-12T05:53:38.218833+0000","last_became_peered":"2024-03-12T05:53:38.218833+0000","last_unstale":"2024-03-12T05:55:10.909127+0000","last_undegraded":"2024-03-12T05:55:10.909127+0000","last_fullsized":"2024-03-12T05:55:10.909127+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:13:36.177325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.915355+0000","last_change":"2024-03-12T05:53:38.205126+0000","last_active":"2024-03-12T05:55:10.915355+0000","last_peered":"2024-03-12T05:55:10.915355+0000","last_clean":"2024-03-12T05:55:10.915355+0000","last_became_active":"2024-03-12T05:53:38.204478+0000","last_became_peered":"2024-03-12T05:53:38.204478+0000","last_unstale":"2024-03-12T05:55:10.915355+0000","last_undegraded":"2024-03-12T05:55:10.915355+0000","last_fullsized":"2024-03-12T05:55:10.915355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:25:56.475134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904502+0000","last_change":"2024-03-12T05:53:38.215186+0000","last_active":"2024-03-12T05:55:10.904502+0000","last_peered":"2024-03-12T05:55:10.904502+0000","last_clean":"2024-03-12T05:55:10.904502+0000","last_became_active":"2024-03-12T05:53:38.214832+0000","last_became_peered":"2024-03-12T05:53:38.214832+0000","last_unstale":"2024-03-12T05:55:10.904502+0000","last_undegraded":"2024-03-12T05:55:10.904502+0000","last_fullsized":"2024-03-12T05:55:10.904502+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:56:26.621990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918038+0000","last_change":"2024-03-12T05:53:38.222347+0000","last_active":"2024-03-12T05:55:10.918038+0000","last_peered":"2024-03-12T05:55:10.918038+0000","last_clean":"2024-03-12T05:55:10.918038+0000","last_became_active":"2024-03-12T05:53:38.221637+0000","last_became_peered":"2024-03-12T05:53:38.221637+0000","last_unstale":"2024-03-12T05:55:10.918038+0000","last_undegraded":"2024-03-12T05:55:10.918038+0000","last_fullsized":"2024-03-12T05:55:10.918038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:16:26.652673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.912936+0000","last_change":"2024-03-12T05:53:38.218687+0000","last_active":"2024-03-12T05:55:10.912936+0000","last_peered":"2024-03-12T05:55:10.912936+0000","last_clean":"2024-03-12T05:55:10.912936+0000","last_became_active":"2024-03-12T05:53:38.218393+0000","last_became_peered":"2024-03-12T05:53:38.218393+0000","last_unstale":"2024-03-12T05:55:10.912936+0000","last_undegraded":"2024-03-12T05:55:10.912936+0000","last_fullsized":"2024-03-12T05:55:10.912936+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:41:39.918992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.900607+0000","last_change":"2024-03-12T05:53:38.207432+0000","last_active":"2024-03-12T05:55:10.900607+0000","last_peered":"2024-03-12T05:55:10.900607+0000","last_clean":"2024-03-12T05:55:10.900607+0000","last_became_active":"2024-03-12T05:53:38.207099+0000","last_became_peered":"2024-03-12T05:53:38.207099+0000","last_unstale":"2024-03-12T05:55:10.900607+0000","last_undegraded":"2024-03-12T05:55:10.900607+0000","last_fullsized":"2024-03-12T05:55:10.900607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:20:08.504723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.909220+0000","last_change":"2024-03-12T05:53:38.207086+0000","last_active":"2024-03-12T05:55:10.909220+0000","last_peered":"2024-03-12T05:55:10.909220+0000","last_clean":"2024-03-12T05:55:10.909220+0000","last_became_active":"2024-03-12T05:53:38.206714+0000","last_became_peered":"2024-03-12T05:53:38.206714+0000","last_unstale":"2024-03-12T05:55:10.909220+0000","last_undegraded":"2024-03-12T05:55:10.909220+0000","last_fullsized":"2024-03-12T05:55:10.909220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:07:13.012272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913790+0000","last_change":"2024-03-12T05:53:38.217179+0000","last_active":"2024-03-12T05:55:10.913790+0000","last_peered":"2024-03-12T05:55:10.913790+0000","last_clean":"2024-03-12T05:55:10.913790+0000","last_became_active":"2024-03-12T05:53:38.216820+0000","last_became_peered":"2024-03-12T05:53:38.216820+0000","last_unstale":"2024-03-12T05:55:10.913790+0000","last_undegraded":"2024-03-12T05:55:10.913790+0000","last_fullsized":"2024-03-12T05:55:10.913790+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:01:08.842611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918587+0000","last_change":"2024-03-12T05:53:38.220224+0000","last_active":"2024-03-12T05:55:10.918587+0000","last_peered":"2024-03-12T05:55:10.918587+0000","last_clean":"2024-03-12T05:55:10.918587+0000","last_became_active":"2024-03-12T05:53:38.219878+0000","last_became_peered":"2024-03-12T05:53:38.219878+0000","last_unstale":"2024-03-12T05:55:10.918587+0000","last_undegraded":"2024-03-12T05:55:10.918587+0000","last_fullsized":"2024-03-12T05:55:10.918587+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:53:16.545807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904735+0000","last_change":"2024-03-12T05:53:38.208524+0000","last_active":"2024-03-12T05:55:10.904735+0000","last_peered":"2024-03-12T05:55:10.904735+0000","last_clean":"2024-03-12T05:55:10.904735+0000","last_became_active":"2024-03-12T05:53:38.208209+0000","last_became_peered":"2024-03-12T05:53:38.208209+0000","last_unstale":"2024-03-12T05:55:10.904735+0000","last_undegraded":"2024-03-12T05:55:10.904735+0000","last_fullsized":"2024-03-12T05:55:10.904735+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:28:56.334066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917077+0000","last_change":"2024-03-12T05:53:38.208035+0000","last_active":"2024-03-12T05:55:10.917077+0000","last_peered":"2024-03-12T05:55:10.917077+0000","last_clean":"2024-03-12T05:55:10.917077+0000","last_became_active":"2024-03-12T05:53:38.207679+0000","last_became_peered":"2024-03-12T05:53:38.207679+0000","last_unstale":"2024-03-12T05:55:10.917077+0000","last_undegraded":"2024-03-12T05:55:10.917077+0000","last_fullsized":"2024-03-12T05:55:10.917077+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:40:20.620552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.890594+0000","last_change":"2024-03-12T05:53:38.207077+0000","last_active":"2024-03-12T05:55:10.890594+0000","last_peered":"2024-03-12T05:55:10.890594+0000","last_clean":"2024-03-12T05:55:10.890594+0000","last_became_active":"2024-03-12T05:53:38.206720+0000","last_became_peered":"2024-03-12T05:53:38.206720+0000","last_unstale":"2024-03-12T05:55:10.890594+0000","last_undegraded":"2024-03-12T05:55:10.890594+0000","last_fullsized":"2024-03-12T05:55:10.890594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:45:54.628394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013942+0000","last_change":"2024-03-12T05:53:37.201637+0000","last_active":"2024-03-12T05:54:42.013942+0000","last_peered":"2024-03-12T05:54:42.013942+0000","last_clean":"2024-03-12T05:54:42.013942+0000","last_became_active":"2024-03-12T05:53:37.201289+0000","last_became_peered":"2024-03-12T05:53:37.201289+0000","last_unstale":"2024-03-12T05:54:42.013942+0000","last_undegraded":"2024-03-12T05:54:42.013942+0000","last_fullsized":"2024-03-12T05:54:42.013942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:59:43.829570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902916+0000","last_change":"2024-03-12T05:53:38.218052+0000","last_active":"2024-03-12T05:55:10.902916+0000","last_peered":"2024-03-12T05:55:10.902916+0000","last_clean":"2024-03-12T05:55:10.902916+0000","last_became_active":"2024-03-12T05:53:38.217738+0000","last_became_peered":"2024-03-12T05:53:38.217738+0000","last_unstale":"2024-03-12T05:55:10.902916+0000","last_undegraded":"2024-03-12T05:55:10.902916+0000","last_fullsized":"2024-03-12T05:55:10.902916+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:59:40.599573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018614+0000","last_change":"2024-03-12T05:53:37.210321+0000","last_active":"2024-03-12T05:54:42.018614+0000","last_peered":"2024-03-12T05:54:42.018614+0000","last_clean":"2024-03-12T05:54:42.018614+0000","last_became_active":"2024-03-12T05:53:37.210018+0000","last_became_peered":"2024-03-12T05:53:37.210018+0000","last_unstale":"2024-03-12T05:54:42.018614+0000","last_undegraded":"2024-03-12T05:54:42.018614+0000","last_fullsized":"2024-03-12T05:54:42.018614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:13:45.433332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917328+0000","last_change":"2024-03-12T05:53:38.207801+0000","last_active":"2024-03-12T05:55:10.917328+0000","last_peered":"2024-03-12T05:55:10.917328+0000","last_clean":"2024-03-12T05:55:10.917328+0000","last_became_active":"2024-03-12T05:53:38.207467+0000","last_became_peered":"2024-03-12T05:53:38.207467+0000","last_unstale":"2024-03-12T05:55:10.917328+0000","last_undegraded":"2024-03-12T05:55:10.917328+0000","last_fullsized":"2024-03-12T05:55:10.917328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:56:14.469662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014569+0000","last_change":"2024-03-12T05:53:37.208288+0000","last_active":"2024-03-12T05:54:42.014569+0000","last_peered":"2024-03-12T05:54:42.014569+0000","last_clean":"2024-03-12T05:54:42.014569+0000","last_became_active":"2024-03-12T05:53:37.207929+0000","last_became_peered":"2024-03-12T05:53:37.207929+0000","last_unstale":"2024-03-12T05:54:42.014569+0000","last_undegraded":"2024-03-12T05:54:42.014569+0000","last_fullsized":"2024-03-12T05:54:42.014569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:10:52.719116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.916187+0000","last_change":"2024-03-12T05:53:38.207270+0000","last_active":"2024-03-12T05:55:10.916187+0000","last_peered":"2024-03-12T05:55:10.916187+0000","last_clean":"2024-03-12T05:55:10.916187+0000","last_became_active":"2024-03-12T05:53:38.205782+0000","last_became_peered":"2024-03-12T05:53:38.205782+0000","last_unstale":"2024-03-12T05:55:10.916187+0000","last_undegraded":"2024-03-12T05:55:10.916187+0000","last_fullsized":"2024-03-12T05:55:10.916187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:02:24.875444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014278+0000","last_change":"2024-03-12T05:53:37.203899+0000","last_active":"2024-03-12T05:54:42.014278+0000","last_peered":"2024-03-12T05:54:42.014278+0000","last_clean":"2024-03-12T05:54:42.014278+0000","last_became_active":"2024-03-12T05:53:37.203450+0000","last_became_peered":"2024-03-12T05:53:37.203450+0000","last_unstale":"2024-03-12T05:54:42.014278+0000","last_undegraded":"2024-03-12T05:54:42.014278+0000","last_fullsized":"2024-03-12T05:54:42.014278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:38:25.773826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917208+0000","last_change":"2024-03-12T05:54:42.480247+0000","last_active":"2024-03-12T05:55:10.917208+0000","last_peered":"2024-03-12T05:55:10.917208+0000","last_clean":"2024-03-12T05:55:10.917208+0000","last_became_active":"2024-03-12T05:54:42.479952+0000","last_became_peered":"2024-03-12T05:54:42.479952+0000","last_unstale":"2024-03-12T05:55:10.917208+0000","last_undegraded":"2024-03-12T05:55:10.917208+0000","last_fullsized":"2024-03-12T05:55:10.917208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:45:20.439878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":144,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:33.525060+0000","last_change":"2024-03-12T05:53:37.205401+0000","last_active":"2024-03-12T05:55:33.525060+0000","last_peered":"2024-03-12T05:55:33.525060+0000","last_clean":"2024-03-12T05:55:33.525060+0000","last_became_active":"2024-03-12T05:53:37.204993+0000","last_became_peered":"2024-03-12T05:53:37.204993+0000","last_unstale":"2024-03-12T05:55:33.525060+0000","last_undegraded":"2024-03-12T05:55:33.525060+0000","last_fullsized":"2024-03-12T05:55:33.525060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:56:06.177387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.901227+0000","last_change":"2024-03-12T05:53:38.210562+0000","last_active":"2024-03-12T05:55:10.901227+0000","last_peered":"2024-03-12T05:55:10.901227+0000","last_clean":"2024-03-12T05:55:10.901227+0000","last_became_active":"2024-03-12T05:53:38.210235+0000","last_became_peered":"2024-03-12T05:53:38.210235+0000","last_unstale":"2024-03-12T05:55:10.901227+0000","last_undegraded":"2024-03-12T05:55:10.901227+0000","last_fullsized":"2024-03-12T05:55:10.901227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:58:35.445812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014118+0000","last_change":"2024-03-12T05:53:37.204986+0000","last_active":"2024-03-12T05:54:42.014118+0000","last_peered":"2024-03-12T05:54:42.014118+0000","last_clean":"2024-03-12T05:54:42.014118+0000","last_became_active":"2024-03-12T05:53:37.204580+0000","last_became_peered":"2024-03-12T05:53:37.204580+0000","last_unstale":"2024-03-12T05:54:42.014118+0000","last_undegraded":"2024-03-12T05:54:42.014118+0000","last_fullsized":"2024-03-12T05:54:42.014118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:37:34.584395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.900872+0000","last_change":"2024-03-12T05:53:38.214948+0000","last_active":"2024-03-12T05:55:10.900872+0000","last_peered":"2024-03-12T05:55:10.900872+0000","last_clean":"2024-03-12T05:55:10.900872+0000","last_became_active":"2024-03-12T05:53:38.213748+0000","last_became_peered":"2024-03-12T05:53:38.213748+0000","last_unstale":"2024-03-12T05:55:10.900872+0000","last_undegraded":"2024-03-12T05:55:10.900872+0000","last_fullsized":"2024-03-12T05:55:10.900872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:12:45.994129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017009+0000","last_change":"2024-03-12T05:53:37.198167+0000","last_active":"2024-03-12T05:54:42.017009+0000","last_peered":"2024-03-12T05:54:42.017009+0000","last_clean":"2024-03-12T05:54:42.017009+0000","last_became_active":"2024-03-12T05:53:37.197808+0000","last_became_peered":"2024-03-12T05:53:37.197808+0000","last_unstale":"2024-03-12T05:54:42.017009+0000","last_undegraded":"2024-03-12T05:54:42.017009+0000","last_fullsized":"2024-03-12T05:54:42.017009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:40:19.498546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904933+0000","last_change":"2024-03-12T05:53:38.205464+0000","last_active":"2024-03-12T05:55:10.904933+0000","last_peered":"2024-03-12T05:55:10.904933+0000","last_clean":"2024-03-12T05:55:10.904933+0000","last_became_active":"2024-03-12T05:53:38.205129+0000","last_became_peered":"2024-03-12T05:53:38.205129+0000","last_unstale":"2024-03-12T05:55:10.904933+0000","last_undegraded":"2024-03-12T05:55:10.904933+0000","last_fullsized":"2024-03-12T05:55:10.904933+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:38:55.460862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017494+0000","last_change":"2024-03-12T05:53:37.201572+0000","last_active":"2024-03-12T05:54:42.017494+0000","last_peered":"2024-03-12T05:54:42.017494+0000","last_clean":"2024-03-12T05:54:42.017494+0000","last_became_active":"2024-03-12T05:53:37.201230+0000","last_became_peered":"2024-03-12T05:53:37.201230+0000","last_unstale":"2024-03-12T05:54:42.017494+0000","last_undegraded":"2024-03-12T05:54:42.017494+0000","last_fullsized":"2024-03-12T05:54:42.017494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:16:13.715459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.903437+0000","last_change":"2024-03-12T05:53:38.210696+0000","last_active":"2024-03-12T05:55:10.903437+0000","last_peered":"2024-03-12T05:55:10.903437+0000","last_clean":"2024-03-12T05:55:10.903437+0000","last_became_active":"2024-03-12T05:53:38.210325+0000","last_became_peered":"2024-03-12T05:53:38.210325+0000","last_unstale":"2024-03-12T05:55:10.903437+0000","last_undegraded":"2024-03-12T05:55:10.903437+0000","last_fullsized":"2024-03-12T05:55:10.903437+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:25:36.397314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:32.523269+0000","last_change":"2024-03-12T05:53:37.201712+0000","last_active":"2024-03-12T05:55:32.523269+0000","last_peered":"2024-03-12T05:55:32.523269+0000","last_clean":"2024-03-12T05:55:32.523269+0000","last_became_active":"2024-03-12T05:53:37.200794+0000","last_became_peered":"2024-03-12T05:53:37.200794+0000","last_unstale":"2024-03-12T05:55:32.523269+0000","last_undegraded":"2024-03-12T05:55:32.523269+0000","last_fullsized":"2024-03-12T05:55:32.523269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:43:29.673717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918225+0000","last_change":"2024-03-12T05:53:38.206647+0000","last_active":"2024-03-12T05:55:10.918225+0000","last_peered":"2024-03-12T05:55:10.918225+0000","last_clean":"2024-03-12T05:55:10.918225+0000","last_became_active":"2024-03-12T05:53:38.206254+0000","last_became_peered":"2024-03-12T05:53:38.206254+0000","last_unstale":"2024-03-12T05:55:10.918225+0000","last_undegraded":"2024-03-12T05:55:10.918225+0000","last_fullsized":"2024-03-12T05:55:10.918225+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:47:20.455360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687585+0000","last_change":"2024-03-12T05:53:37.208692+0000","last_active":"2024-03-12T05:54:42.687585+0000","last_peered":"2024-03-12T05:54:42.687585+0000","last_clean":"2024-03-12T05:54:42.687585+0000","last_became_active":"2024-03-12T05:53:37.208270+0000","last_became_peered":"2024-03-12T05:53:37.208270+0000","last_unstale":"2024-03-12T05:54:42.687585+0000","last_undegraded":"2024-03-12T05:54:42.687585+0000","last_fullsized":"2024-03-12T05:54:42.687585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:51:38.217999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913653+0000","last_change":"2024-03-12T05:53:38.218202+0000","last_active":"2024-03-12T05:55:10.913653+0000","last_peered":"2024-03-12T05:55:10.913653+0000","last_clean":"2024-03-12T05:55:10.913653+0000","last_became_active":"2024-03-12T05:53:38.217798+0000","last_became_peered":"2024-03-12T05:53:38.217798+0000","last_unstale":"2024-03-12T05:55:10.913653+0000","last_undegraded":"2024-03-12T05:55:10.913653+0000","last_fullsized":"2024-03-12T05:55:10.913653+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:09:29.422125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019722+0000","last_change":"2024-03-12T05:53:37.203252+0000","last_active":"2024-03-12T05:54:42.019722+0000","last_peered":"2024-03-12T05:54:42.019722+0000","last_clean":"2024-03-12T05:54:42.019722+0000","last_became_active":"2024-03-12T05:53:37.202916+0000","last_became_peered":"2024-03-12T05:53:37.202916+0000","last_unstale":"2024-03-12T05:54:42.019722+0000","last_undegraded":"2024-03-12T05:54:42.019722+0000","last_fullsized":"2024-03-12T05:54:42.019722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:09:34.629085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.916291+0000","last_change":"2024-03-12T05:54:42.021528+0000","last_active":"2024-03-12T05:55:10.916291+0000","last_peered":"2024-03-12T05:55:10.916291+0000","last_clean":"2024-03-12T05:55:10.916291+0000","last_became_active":"2024-03-12T05:54:42.021015+0000","last_became_peered":"2024-03-12T05:54:42.021015+0000","last_unstale":"2024-03-12T05:55:10.916291+0000","last_undegraded":"2024-03-12T05:55:10.916291+0000","last_fullsized":"2024-03-12T05:55:10.916291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:16.998017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":288,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:33.524254+0000","last_change":"2024-03-12T05:53:37.206271+0000","last_active":"2024-03-12T05:55:33.524254+0000","last_peered":"2024-03-12T05:55:33.524254+0000","last_clean":"2024-03-12T05:55:33.524254+0000","last_became_active":"2024-03-12T05:53:37.205920+0000","last_became_peered":"2024-03-12T05:53:37.205920+0000","last_unstale":"2024-03-12T05:55:33.524254+0000","last_undegraded":"2024-03-12T05:55:33.524254+0000","last_fullsized":"2024-03-12T05:55:33.524254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:30.140159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61603,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":236,"num_read_kb":178,"num_write":34,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.912256+0000","last_change":"2024-03-12T05:53:38.211397+0000","last_active":"2024-03-12T05:55:10.912256+0000","last_peered":"2024-03-12T05:55:10.912256+0000","last_clean":"2024-03-12T05:55:10.912256+0000","last_became_active":"2024-03-12T05:53:38.211045+0000","last_became_peered":"2024-03-12T05:53:38.211045+0000","last_unstale":"2024-03-12T05:55:10.912256+0000","last_undegraded":"2024-03-12T05:55:10.912256+0000","last_fullsized":"2024-03-12T05:55:10.912256+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:21:42.510436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019264+0000","last_change":"2024-03-12T05:53:37.210871+0000","last_active":"2024-03-12T05:54:42.019264+0000","last_peered":"2024-03-12T05:54:42.019264+0000","last_clean":"2024-03-12T05:54:42.019264+0000","last_became_active":"2024-03-12T05:53:37.210526+0000","last_became_peered":"2024-03-12T05:53:37.210526+0000","last_unstale":"2024-03-12T05:54:42.019264+0000","last_undegraded":"2024-03-12T05:54:42.019264+0000","last_fullsized":"2024-03-12T05:54:42.019264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:40:46.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.907189+0000","last_change":"2024-03-12T05:53:38.213751+0000","last_active":"2024-03-12T05:55:10.907189+0000","last_peered":"2024-03-12T05:55:10.907189+0000","last_clean":"2024-03-12T05:55:10.907189+0000","last_became_active":"2024-03-12T05:53:38.211558+0000","last_became_peered":"2024-03-12T05:53:38.211558+0000","last_unstale":"2024-03-12T05:55:10.907189+0000","last_undegraded":"2024-03-12T05:55:10.907189+0000","last_fullsized":"2024-03-12T05:55:10.907189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:40:44.194548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.018330+0000","last_change":"2024-03-12T05:53:37.207368+0000","last_active":"2024-03-12T05:54:42.018330+0000","last_peered":"2024-03-12T05:54:42.018330+0000","last_clean":"2024-03-12T05:54:42.018330+0000","last_became_active":"2024-03-12T05:53:37.207066+0000","last_became_peered":"2024-03-12T05:53:37.207066+0000","last_unstale":"2024-03-12T05:54:42.018330+0000","last_undegraded":"2024-03-12T05:54:42.018330+0000","last_fullsized":"2024-03-12T05:54:42.018330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:28:58.155360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913837+0000","last_change":"2024-03-12T05:53:38.217079+0000","last_active":"2024-03-12T05:55:10.913837+0000","last_peered":"2024-03-12T05:55:10.913837+0000","last_clean":"2024-03-12T05:55:10.913837+0000","last_became_active":"2024-03-12T05:53:38.216757+0000","last_became_peered":"2024-03-12T05:53:38.216757+0000","last_unstale":"2024-03-12T05:55:10.913837+0000","last_undegraded":"2024-03-12T05:55:10.913837+0000","last_fullsized":"2024-03-12T05:55:10.913837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:48:13.125867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'4","reported_seq":140,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:34.555872+0000","last_change":"2024-03-12T05:53:37.202390+0000","last_active":"2024-03-12T05:55:34.555872+0000","last_peered":"2024-03-12T05:55:34.555872+0000","last_clean":"2024-03-12T05:55:34.555872+0000","last_became_active":"2024-03-12T05:53:37.202020+0000","last_became_peered":"2024-03-12T05:53:37.202020+0000","last_unstale":"2024-03-12T05:55:34.555872+0000","last_undegraded":"2024-03-12T05:55:34.555872+0000","last_fullsized":"2024-03-12T05:55:34.555872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:06:35.293227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":896,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.910544+0000","last_change":"2024-03-12T05:53:38.218464+0000","last_active":"2024-03-12T05:55:10.910544+0000","last_peered":"2024-03-12T05:55:10.910544+0000","last_clean":"2024-03-12T05:55:10.910544+0000","last_became_active":"2024-03-12T05:53:38.218136+0000","last_became_peered":"2024-03-12T05:53:38.218136+0000","last_unstale":"2024-03-12T05:55:10.910544+0000","last_undegraded":"2024-03-12T05:55:10.910544+0000","last_fullsized":"2024-03-12T05:55:10.910544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:31:05.409148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013820+0000","last_change":"2024-03-12T05:53:37.205161+0000","last_active":"2024-03-12T05:54:42.013820+0000","last_peered":"2024-03-12T05:54:42.013820+0000","last_clean":"2024-03-12T05:54:42.013820+0000","last_became_active":"2024-03-12T05:53:37.204799+0000","last_became_peered":"2024-03-12T05:53:37.204799+0000","last_unstale":"2024-03-12T05:54:42.013820+0000","last_undegraded":"2024-03-12T05:54:42.013820+0000","last_fullsized":"2024-03-12T05:54:42.013820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:29:32.065635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.900970+0000","last_change":"2024-03-12T05:53:38.216321+0000","last_active":"2024-03-12T05:55:10.900970+0000","last_peered":"2024-03-12T05:55:10.900970+0000","last_clean":"2024-03-12T05:55:10.900970+0000","last_became_active":"2024-03-12T05:53:38.215893+0000","last_became_peered":"2024-03-12T05:53:38.215893+0000","last_unstale":"2024-03-12T05:55:10.900970+0000","last_undegraded":"2024-03-12T05:55:10.900970+0000","last_fullsized":"2024-03-12T05:55:10.900970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:32:27.603930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"88'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014706+0000","last_change":"2024-03-12T05:53:37.203638+0000","last_active":"2024-03-12T05:54:42.014706+0000","last_peered":"2024-03-12T05:54:42.014706+0000","last_clean":"2024-03-12T05:54:42.014706+0000","last_became_active":"2024-03-12T05:53:37.203217+0000","last_became_peered":"2024-03-12T05:53:37.203217+0000","last_unstale":"2024-03-12T05:54:42.014706+0000","last_undegraded":"2024-03-12T05:54:42.014706+0000","last_fullsized":"2024-03-12T05:54:42.014706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:17:10.521197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904911+0000","last_change":"2024-03-12T05:53:38.210491+0000","last_active":"2024-03-12T05:55:10.904911+0000","last_peered":"2024-03-12T05:55:10.904911+0000","last_clean":"2024-03-12T05:55:10.904911+0000","last_became_active":"2024-03-12T05:53:38.210169+0000","last_became_peered":"2024-03-12T05:53:38.210169+0000","last_unstale":"2024-03-12T05:55:10.904911+0000","last_undegraded":"2024-03-12T05:55:10.904911+0000","last_fullsized":"2024-03-12T05:55:10.904911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:56:05.384834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.013770+0000","last_change":"2024-03-12T05:53:37.204478+0000","last_active":"2024-03-12T05:54:42.013770+0000","last_peered":"2024-03-12T05:54:42.013770+0000","last_clean":"2024-03-12T05:54:42.013770+0000","last_became_active":"2024-03-12T05:53:37.204160+0000","last_became_peered":"2024-03-12T05:53:37.204160+0000","last_unstale":"2024-03-12T05:54:42.013770+0000","last_undegraded":"2024-03-12T05:54:42.013770+0000","last_fullsized":"2024-03-12T05:54:42.013770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:07:07.597731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.896102+0000","last_change":"2024-03-12T05:54:42.026040+0000","last_active":"2024-03-12T05:55:10.896102+0000","last_peered":"2024-03-12T05:55:10.896102+0000","last_clean":"2024-03-12T05:55:10.896102+0000","last_became_active":"2024-03-12T05:54:42.018230+0000","last_became_peered":"2024-03-12T05:54:42.018230+0000","last_unstale":"2024-03-12T05:55:10.896102+0000","last_undegraded":"2024-03-12T05:55:10.896102+0000","last_fullsized":"2024-03-12T05:55:10.896102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:32:28.703339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687385+0000","last_change":"2024-03-12T05:53:37.205983+0000","last_active":"2024-03-12T05:54:42.687385+0000","last_peered":"2024-03-12T05:54:42.687385+0000","last_clean":"2024-03-12T05:54:42.687385+0000","last_became_active":"2024-03-12T05:53:37.205629+0000","last_became_peered":"2024-03-12T05:53:37.205629+0000","last_unstale":"2024-03-12T05:54:42.687385+0000","last_undegraded":"2024-03-12T05:54:42.687385+0000","last_fullsized":"2024-03-12T05:54:42.687385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:37.928623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.908778+0000","last_change":"2024-03-12T05:53:38.213752+0000","last_active":"2024-03-12T05:55:10.908778+0000","last_peered":"2024-03-12T05:55:10.908778+0000","last_clean":"2024-03-12T05:55:10.908778+0000","last_became_active":"2024-03-12T05:53:38.213407+0000","last_became_peered":"2024-03-12T05:53:38.213407+0000","last_unstale":"2024-03-12T05:55:10.908778+0000","last_undegraded":"2024-03-12T05:55:10.908778+0000","last_fullsized":"2024-03-12T05:55:10.908778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:21:15.703455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.907245+0000","last_change":"2024-03-12T05:54:42.022196+0000","last_active":"2024-03-12T05:55:10.907245+0000","last_peered":"2024-03-12T05:55:10.907245+0000","last_clean":"2024-03-12T05:55:10.907245+0000","last_became_active":"2024-03-12T05:54:42.021774+0000","last_became_peered":"2024-03-12T05:54:42.021774+0000","last_unstale":"2024-03-12T05:55:10.907245+0000","last_undegraded":"2024-03-12T05:55:10.907245+0000","last_fullsized":"2024-03-12T05:55:10.907245+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:34:39.290359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:32.551476+0000","last_change":"2024-03-12T05:53:37.202782+0000","last_active":"2024-03-12T05:55:32.551476+0000","last_peered":"2024-03-12T05:55:32.551476+0000","last_clean":"2024-03-12T05:55:32.551476+0000","last_became_active":"2024-03-12T05:53:37.202439+0000","last_became_peered":"2024-03-12T05:53:37.202439+0000","last_unstale":"2024-03-12T05:55:32.551476+0000","last_undegraded":"2024-03-12T05:55:32.551476+0000","last_fullsized":"2024-03-12T05:55:32.551476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:07:18.939629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.914766+0000","last_change":"2024-03-12T05:53:38.217993+0000","last_active":"2024-03-12T05:55:10.914766+0000","last_peered":"2024-03-12T05:55:10.914766+0000","last_clean":"2024-03-12T05:55:10.914766+0000","last_became_active":"2024-03-12T05:53:38.217613+0000","last_became_peered":"2024-03-12T05:53:38.217613+0000","last_unstale":"2024-03-12T05:55:10.914766+0000","last_undegraded":"2024-03-12T05:55:10.914766+0000","last_fullsized":"2024-03-12T05:55:10.914766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:49:50.960004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.478908+0000","last_change":"2024-03-12T05:53:37.203662+0000","last_active":"2024-03-12T05:54:42.478908+0000","last_peered":"2024-03-12T05:54:42.478908+0000","last_clean":"2024-03-12T05:54:42.478908+0000","last_became_active":"2024-03-12T05:53:37.203309+0000","last_became_peered":"2024-03-12T05:53:37.203309+0000","last_unstale":"2024-03-12T05:54:42.478908+0000","last_undegraded":"2024-03-12T05:54:42.478908+0000","last_fullsized":"2024-03-12T05:54:42.478908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:47:24.603686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687337+0000","last_change":"2024-03-12T05:52:27.278524+0000","last_active":"2024-03-12T05:54:42.687337+0000","last_peered":"2024-03-12T05:54:42.687337+0000","last_clean":"2024-03-12T05:54:42.687337+0000","last_became_active":"2024-03-12T05:52:27.278010+0000","last_became_peered":"2024-03-12T05:52:27.278010+0000","last_unstale":"2024-03-12T05:54:42.687337+0000","last_undegraded":"2024-03-12T05:54:42.687337+0000","last_fullsized":"2024-03-12T05:54:42.687337+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:49:09.175725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:49:09.175725+0000","last_clean_scrub_stamp":"2024-03-12T05:49:09.175725+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:24:40.380724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.901203+0000","last_change":"2024-03-12T05:53:38.219451+0000","last_active":"2024-03-12T05:55:10.901203+0000","last_peered":"2024-03-12T05:55:10.901203+0000","last_clean":"2024-03-12T05:55:10.901203+0000","last_became_active":"2024-03-12T05:53:38.218840+0000","last_became_peered":"2024-03-12T05:53:38.218840+0000","last_unstale":"2024-03-12T05:55:10.901203+0000","last_undegraded":"2024-03-12T05:55:10.901203+0000","last_fullsized":"2024-03-12T05:55:10.901203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:05.595822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.687398+0000","last_change":"2024-03-12T05:53:37.210352+0000","last_active":"2024-03-12T05:54:42.687398+0000","last_peered":"2024-03-12T05:54:42.687398+0000","last_clean":"2024-03-12T05:54:42.687398+0000","last_became_active":"2024-03-12T05:53:37.210002+0000","last_became_peered":"2024-03-12T05:53:37.210002+0000","last_unstale":"2024-03-12T05:54:42.687398+0000","last_undegraded":"2024-03-12T05:54:42.687398+0000","last_fullsized":"2024-03-12T05:54:42.687398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:49:52.368411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.022961+0000","last_change":"2024-03-12T05:54:42.022489+0000","last_active":"2024-03-12T05:54:42.022961+0000","last_peered":"2024-03-12T05:54:42.022961+0000","last_clean":"2024-03-12T05:54:42.022961+0000","last_became_active":"2024-03-12T05:54:42.016400+0000","last_became_peered":"2024-03-12T05:54:42.016400+0000","last_unstale":"2024-03-12T05:54:42.022961+0000","last_undegraded":"2024-03-12T05:54:42.022961+0000","last_fullsized":"2024-03-12T05:54:42.022961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:32:05.713420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.886986+0000","last_change":"2024-03-12T05:53:38.203951+0000","last_active":"2024-03-12T05:55:10.886986+0000","last_peered":"2024-03-12T05:55:10.886986+0000","last_clean":"2024-03-12T05:55:10.886986+0000","last_became_active":"2024-03-12T05:53:38.203617+0000","last_became_peered":"2024-03-12T05:53:38.203617+0000","last_unstale":"2024-03-12T05:55:10.886986+0000","last_undegraded":"2024-03-12T05:55:10.886986+0000","last_fullsized":"2024-03-12T05:55:10.886986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:19.138220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'10","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:11.379756+0000","last_change":"2024-03-12T05:53:37.203770+0000","last_active":"2024-03-12T05:55:11.379756+0000","last_peered":"2024-03-12T05:55:11.379756+0000","last_clean":"2024-03-12T05:55:11.379756+0000","last_became_active":"2024-03-12T05:53:37.203456+0000","last_became_peered":"2024-03-12T05:53:37.203456+0000","last_unstale":"2024-03-12T05:55:11.379756+0000","last_undegraded":"2024-03-12T05:55:11.379756+0000","last_fullsized":"2024-03-12T05:55:11.379756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:32.291536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.911027+0000","last_change":"2024-03-12T05:53:38.214872+0000","last_active":"2024-03-12T05:55:10.911027+0000","last_peered":"2024-03-12T05:55:10.911027+0000","last_clean":"2024-03-12T05:55:10.911027+0000","last_became_active":"2024-03-12T05:53:38.214139+0000","last_became_peered":"2024-03-12T05:53:38.214139+0000","last_unstale":"2024-03-12T05:55:10.911027+0000","last_undegraded":"2024-03-12T05:55:10.911027+0000","last_fullsized":"2024-03-12T05:55:10.911027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:20:21.808039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.479035+0000","last_change":"2024-03-12T05:53:37.203852+0000","last_active":"2024-03-12T05:54:42.479035+0000","last_peered":"2024-03-12T05:54:42.479035+0000","last_clean":"2024-03-12T05:54:42.479035+0000","last_became_active":"2024-03-12T05:53:37.203416+0000","last_became_peered":"2024-03-12T05:53:37.203416+0000","last_unstale":"2024-03-12T05:54:42.479035+0000","last_undegraded":"2024-03-12T05:54:42.479035+0000","last_fullsized":"2024-03-12T05:54:42.479035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:09:08.095957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.910705+0000","last_change":"2024-03-12T05:54:42.688395+0000","last_active":"2024-03-12T05:55:10.910705+0000","last_peered":"2024-03-12T05:55:10.910705+0000","last_clean":"2024-03-12T05:55:10.910705+0000","last_became_active":"2024-03-12T05:54:42.688094+0000","last_became_peered":"2024-03-12T05:54:42.688094+0000","last_unstale":"2024-03-12T05:55:10.910705+0000","last_undegraded":"2024-03-12T05:55:10.910705+0000","last_fullsized":"2024-03-12T05:55:10.910705+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:29:42.921726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.025218+0000","last_change":"2024-03-12T05:54:42.025218+0000","last_active":"2024-03-12T05:54:42.025218+0000","last_peered":"2024-03-12T05:54:42.025218+0000","last_clean":"2024-03-12T05:54:42.025218+0000","last_became_active":"2024-03-12T05:54:42.021022+0000","last_became_peered":"2024-03-12T05:54:42.021022+0000","last_unstale":"2024-03-12T05:54:42.025218+0000","last_undegraded":"2024-03-12T05:54:42.025218+0000","last_fullsized":"2024-03-12T05:54:42.025218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:20:57.232145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.874050+0000","last_change":"2024-03-12T05:53:38.207757+0000","last_active":"2024-03-12T05:55:10.874050+0000","last_peered":"2024-03-12T05:55:10.874050+0000","last_clean":"2024-03-12T05:55:10.874050+0000","last_became_active":"2024-03-12T05:53:38.207424+0000","last_became_peered":"2024-03-12T05:53:38.207424+0000","last_unstale":"2024-03-12T05:55:10.874050+0000","last_undegraded":"2024-03-12T05:55:10.874050+0000","last_fullsized":"2024-03-12T05:55:10.874050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T11:33:16.017907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.016731+0000","last_change":"2024-03-12T05:53:37.208512+0000","last_active":"2024-03-12T05:54:42.016731+0000","last_peered":"2024-03-12T05:54:42.016731+0000","last_clean":"2024-03-12T05:54:42.016731+0000","last_became_active":"2024-03-12T05:53:37.207628+0000","last_became_peered":"2024-03-12T05:53:37.207628+0000","last_unstale":"2024-03-12T05:54:42.016731+0000","last_undegraded":"2024-03-12T05:54:42.016731+0000","last_fullsized":"2024-03-12T05:54:42.016731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:25:28.825779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.908590+0000","last_change":"2024-03-12T05:53:38.205654+0000","last_active":"2024-03-12T05:55:10.908590+0000","last_peered":"2024-03-12T05:55:10.908590+0000","last_clean":"2024-03-12T05:55:10.908590+0000","last_became_active":"2024-03-12T05:53:38.205325+0000","last_became_peered":"2024-03-12T05:53:38.205325+0000","last_unstale":"2024-03-12T05:55:10.908590+0000","last_undegraded":"2024-03-12T05:55:10.908590+0000","last_fullsized":"2024-03-12T05:55:10.908590+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:33:35.668625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019632+0000","last_change":"2024-03-12T05:53:37.211430+0000","last_active":"2024-03-12T05:54:42.019632+0000","last_peered":"2024-03-12T05:54:42.019632+0000","last_clean":"2024-03-12T05:54:42.019632+0000","last_became_active":"2024-03-12T05:53:37.210501+0000","last_became_peered":"2024-03-12T05:53:37.210501+0000","last_unstale":"2024-03-12T05:54:42.019632+0000","last_undegraded":"2024-03-12T05:54:42.019632+0000","last_fullsized":"2024-03-12T05:54:42.019632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:48:33.555134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902337+0000","last_change":"2024-03-12T05:53:38.212837+0000","last_active":"2024-03-12T05:55:10.902337+0000","last_peered":"2024-03-12T05:55:10.902337+0000","last_clean":"2024-03-12T05:55:10.902337+0000","last_became_active":"2024-03-12T05:53:38.212513+0000","last_became_peered":"2024-03-12T05:53:38.212513+0000","last_unstale":"2024-03-12T05:55:10.902337+0000","last_undegraded":"2024-03-12T05:55:10.902337+0000","last_fullsized":"2024-03-12T05:55:10.902337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:47:33.879954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.014092+0000","last_change":"2024-03-12T05:53:37.207465+0000","last_active":"2024-03-12T05:54:42.014092+0000","last_peered":"2024-03-12T05:54:42.014092+0000","last_clean":"2024-03-12T05:54:42.014092+0000","last_became_active":"2024-03-12T05:53:37.207107+0000","last_became_peered":"2024-03-12T05:53:37.207107+0000","last_unstale":"2024-03-12T05:54:42.014092+0000","last_undegraded":"2024-03-12T05:54:42.014092+0000","last_fullsized":"2024-03-12T05:54:42.014092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:05:34.018856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.917908+0000","last_change":"2024-03-12T05:53:38.221470+0000","last_active":"2024-03-12T05:55:10.917908+0000","last_peered":"2024-03-12T05:55:10.917908+0000","last_clean":"2024-03-12T05:55:10.917908+0000","last_became_active":"2024-03-12T05:53:38.221061+0000","last_became_peered":"2024-03-12T05:53:38.221061+0000","last_unstale":"2024-03-12T05:55:10.917908+0000","last_undegraded":"2024-03-12T05:55:10.917908+0000","last_fullsized":"2024-03-12T05:55:10.917908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:01:54.148877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019769+0000","last_change":"2024-03-12T05:53:37.203822+0000","last_active":"2024-03-12T05:54:42.019769+0000","last_peered":"2024-03-12T05:54:42.019769+0000","last_clean":"2024-03-12T05:54:42.019769+0000","last_became_active":"2024-03-12T05:53:37.203431+0000","last_became_peered":"2024-03-12T05:53:37.203431+0000","last_unstale":"2024-03-12T05:54:42.019769+0000","last_undegraded":"2024-03-12T05:54:42.019769+0000","last_fullsized":"2024-03-12T05:54:42.019769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:25:04.447261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918179+0000","last_change":"2024-03-12T05:53:38.222045+0000","last_active":"2024-03-12T05:55:10.918179+0000","last_peered":"2024-03-12T05:55:10.918179+0000","last_clean":"2024-03-12T05:55:10.918179+0000","last_became_active":"2024-03-12T05:53:38.220904+0000","last_became_peered":"2024-03-12T05:53:38.220904+0000","last_unstale":"2024-03-12T05:55:10.918179+0000","last_undegraded":"2024-03-12T05:55:10.918179+0000","last_fullsized":"2024-03-12T05:55:10.918179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T16:30:43.964599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.019437+0000","last_change":"2024-03-12T05:53:37.206753+0000","last_active":"2024-03-12T05:54:42.019437+0000","last_peered":"2024-03-12T05:54:42.019437+0000","last_clean":"2024-03-12T05:54:42.019437+0000","last_became_active":"2024-03-12T05:53:37.206078+0000","last_became_peered":"2024-03-12T05:53:37.206078+0000","last_unstale":"2024-03-12T05:54:42.019437+0000","last_undegraded":"2024-03-12T05:54:42.019437+0000","last_fullsized":"2024-03-12T05:54:42.019437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:42:18.859528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.909834+0000","last_change":"2024-03-12T05:53:38.215671+0000","last_active":"2024-03-12T05:55:10.909834+0000","last_peered":"2024-03-12T05:55:10.909834+0000","last_clean":"2024-03-12T05:55:10.909834+0000","last_became_active":"2024-03-12T05:53:38.215309+0000","last_became_peered":"2024-03-12T05:53:38.215309+0000","last_unstale":"2024-03-12T05:55:10.909834+0000","last_undegraded":"2024-03-12T05:55:10.909834+0000","last_fullsized":"2024-03-12T05:55:10.909834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:50:12.176935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:54:42.017343+0000","last_change":"2024-03-12T05:53:37.205627+0000","last_active":"2024-03-12T05:54:42.017343+0000","last_peered":"2024-03-12T05:54:42.017343+0000","last_clean":"2024-03-12T05:54:42.017343+0000","last_became_active":"2024-03-12T05:53:37.205289+0000","last_became_peered":"2024-03-12T05:53:37.205289+0000","last_unstale":"2024-03-12T05:54:42.017343+0000","last_undegraded":"2024-03-12T05:54:42.017343+0000","last_fullsized":"2024-03-12T05:54:42.017343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:36.183830+0000","last_clean_scrub_stamp":"2024-03-12T05:53:36.183830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:56.682746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913506+0000","last_change":"2024-03-12T05:53:38.214376+0000","last_active":"2024-03-12T05:55:10.913506+0000","last_peered":"2024-03-12T05:55:10.913506+0000","last_clean":"2024-03-12T05:55:10.913506+0000","last_became_active":"2024-03-12T05:53:38.214011+0000","last_became_peered":"2024-03-12T05:53:38.214011+0000","last_unstale":"2024-03-12T05:55:10.913506+0000","last_undegraded":"2024-03-12T05:55:10.913506+0000","last_fullsized":"2024-03-12T05:55:10.913506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:29:32.605216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.908362+0000","last_change":"2024-03-12T05:53:38.205337+0000","last_active":"2024-03-12T05:55:10.908362+0000","last_peered":"2024-03-12T05:55:10.908362+0000","last_clean":"2024-03-12T05:55:10.908362+0000","last_became_active":"2024-03-12T05:53:38.205005+0000","last_became_peered":"2024-03-12T05:53:38.205005+0000","last_unstale":"2024-03-12T05:55:10.908362+0000","last_undegraded":"2024-03-12T05:55:10.908362+0000","last_fullsized":"2024-03-12T05:55:10.908362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T12:44:16.296370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.907222+0000","last_change":"2024-03-12T05:53:38.214694+0000","last_active":"2024-03-12T05:55:10.907222+0000","last_peered":"2024-03-12T05:55:10.907222+0000","last_clean":"2024-03-12T05:55:10.907222+0000","last_became_active":"2024-03-12T05:53:38.214285+0000","last_became_peered":"2024-03-12T05:53:38.214285+0000","last_unstale":"2024-03-12T05:55:10.907222+0000","last_undegraded":"2024-03-12T05:55:10.907222+0000","last_fullsized":"2024-03-12T05:55:10.907222+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:28:32.253332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902212+0000","last_change":"2024-03-12T05:53:38.213069+0000","last_active":"2024-03-12T05:55:10.902212+0000","last_peered":"2024-03-12T05:55:10.902212+0000","last_clean":"2024-03-12T05:55:10.902212+0000","last_became_active":"2024-03-12T05:53:38.212694+0000","last_became_peered":"2024-03-12T05:53:38.212694+0000","last_unstale":"2024-03-12T05:55:10.902212+0000","last_undegraded":"2024-03-12T05:55:10.902212+0000","last_fullsized":"2024-03-12T05:55:10.902212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:26:29.350292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.913785+0000","last_change":"2024-03-12T05:53:38.212084+0000","last_active":"2024-03-12T05:55:10.913785+0000","last_peered":"2024-03-12T05:55:10.913785+0000","last_clean":"2024-03-12T05:55:10.913785+0000","last_became_active":"2024-03-12T05:53:38.211750+0000","last_became_peered":"2024-03-12T05:53:38.211750+0000","last_unstale":"2024-03-12T05:55:10.913785+0000","last_undegraded":"2024-03-12T05:55:10.913785+0000","last_fullsized":"2024-03-12T05:55:10.913785+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:05:02.099121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.899211+0000","last_change":"2024-03-12T05:53:38.212071+0000","last_active":"2024-03-12T05:55:10.899211+0000","last_peered":"2024-03-12T05:55:10.899211+0000","last_clean":"2024-03-12T05:55:10.899211+0000","last_became_active":"2024-03-12T05:53:38.211722+0000","last_became_peered":"2024-03-12T05:53:38.211722+0000","last_unstale":"2024-03-12T05:55:10.899211+0000","last_undegraded":"2024-03-12T05:55:10.899211+0000","last_fullsized":"2024-03-12T05:55:10.899211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:19:46.581837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.895503+0000","last_change":"2024-03-12T05:54:42.015717+0000","last_active":"2024-03-12T05:55:10.895503+0000","last_peered":"2024-03-12T05:55:10.895503+0000","last_clean":"2024-03-12T05:55:10.895503+0000","last_became_active":"2024-03-12T05:54:42.015266+0000","last_became_peered":"2024-03-12T05:54:42.015266+0000","last_unstale":"2024-03-12T05:55:10.895503+0000","last_undegraded":"2024-03-12T05:55:10.895503+0000","last_fullsized":"2024-03-12T05:55:10.895503+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:33:48.250458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.915527+0000","last_change":"2024-03-12T05:53:38.208018+0000","last_active":"2024-03-12T05:55:10.915527+0000","last_peered":"2024-03-12T05:55:10.915527+0000","last_clean":"2024-03-12T05:55:10.915527+0000","last_became_active":"2024-03-12T05:53:38.207682+0000","last_became_peered":"2024-03-12T05:53:38.207682+0000","last_unstale":"2024-03-12T05:55:10.915527+0000","last_undegraded":"2024-03-12T05:55:10.915527+0000","last_fullsized":"2024-03-12T05:55:10.915527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:56:10.903917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902615+0000","last_change":"2024-03-12T05:53:38.217540+0000","last_active":"2024-03-12T05:55:10.902615+0000","last_peered":"2024-03-12T05:55:10.902615+0000","last_clean":"2024-03-12T05:55:10.902615+0000","last_became_active":"2024-03-12T05:53:38.217144+0000","last_became_peered":"2024-03-12T05:53:38.217144+0000","last_unstale":"2024-03-12T05:55:10.902615+0000","last_undegraded":"2024-03-12T05:55:10.902615+0000","last_fullsized":"2024-03-12T05:55:10.902615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T15:46:18.905909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902398+0000","last_change":"2024-03-12T05:53:38.215006+0000","last_active":"2024-03-12T05:55:10.902398+0000","last_peered":"2024-03-12T05:55:10.902398+0000","last_clean":"2024-03-12T05:55:10.902398+0000","last_became_active":"2024-03-12T05:53:38.213756+0000","last_became_peered":"2024-03-12T05:53:38.213756+0000","last_unstale":"2024-03-12T05:55:10.902398+0000","last_undegraded":"2024-03-12T05:55:10.902398+0000","last_fullsized":"2024-03-12T05:55:10.902398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:14:57.642381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.902674+0000","last_change":"2024-03-12T05:53:38.216096+0000","last_active":"2024-03-12T05:55:10.902674+0000","last_peered":"2024-03-12T05:55:10.902674+0000","last_clean":"2024-03-12T05:55:10.902674+0000","last_became_active":"2024-03-12T05:53:38.215153+0000","last_became_peered":"2024-03-12T05:53:38.215153+0000","last_unstale":"2024-03-12T05:55:10.902674+0000","last_undegraded":"2024-03-12T05:55:10.902674+0000","last_fullsized":"2024-03-12T05:55:10.902674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:41:03.893641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.918844+0000","last_change":"2024-03-12T05:53:38.219165+0000","last_active":"2024-03-12T05:55:10.918844+0000","last_peered":"2024-03-12T05:55:10.918844+0000","last_clean":"2024-03-12T05:55:10.918844+0000","last_became_active":"2024-03-12T05:53:38.218579+0000","last_became_peered":"2024-03-12T05:53:38.218579+0000","last_unstale":"2024-03-12T05:55:10.918844+0000","last_undegraded":"2024-03-12T05:55:10.918844+0000","last_fullsized":"2024-03-12T05:55:10.918844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:06:37.531293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.907783+0000","last_change":"2024-03-12T05:53:38.212685+0000","last_active":"2024-03-12T05:55:10.907783+0000","last_peered":"2024-03-12T05:55:10.907783+0000","last_clean":"2024-03-12T05:55:10.907783+0000","last_became_active":"2024-03-12T05:53:38.211106+0000","last_became_peered":"2024-03-12T05:53:38.211106+0000","last_unstale":"2024-03-12T05:55:10.907783+0000","last_undegraded":"2024-03-12T05:55:10.907783+0000","last_fullsized":"2024-03-12T05:55:10.907783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T17:26:00.339270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-12T05:55:10.904578+0000","last_change":"2024-03-12T05:53:38.218402+0000","last_active":"2024-03-12T05:55:10.904578+0000","last_peered":"2024-03-12T05:55:10.904578+0000","last_clean":"2024-03-12T05:55:10.904578+0000","last_became_active":"2024-03-12T05:53:38.217699+0000","last_became_peered":"2024-03-12T05:53:38.217699+0000","last_unstale":"2024-03-12T05:55:10.904578+0000","last_undegraded":"2024-03-12T05:55:10.904578+0000","last_fullsized":"2024-03-12T05:55:10.904578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T05:53:37.185343+0000","last_clean_scrub_stamp":"2024-03-12T05:53:37.185343+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T13:23:03.221710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66053,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":709,"num_read_kb":423,"num_write":171,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":198159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147,"ondisk_log_size":147,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351013,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":461631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612650,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32788,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712364,"statfs":{"total":95995035648,"available":95961460736,"internally_reserved":0,"allocated":1650688,"data_stored":1115001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776175,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":461962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037812,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":461743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32704,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712448,"statfs":{"total":95995035648,"available":95961546752,"internally_reserved":0,"allocated":1564672,"data_stored":1054144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":461234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":522630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":462505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":524512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":461428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":461979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1576960,"data_stored":1053520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T05:55:37.801 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:55:37.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.935+0000 7fac6afbc640 1 -- 172.21.15.76:0/235068024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac64101740 msgr2=0x7fac64101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.935+0000 7fac6afbc640 1 --2- 172.21.15.76:0/235068024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac64101740 0x7fac64101b60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fac5c009ca0 tx=0x7fac5c02c140 comp rx=0 tx=0).stop 2024-03-12T05:55:37.941 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 -- 172.21.15.76:0/235068024 shutdown_connections 2024-03-12T05:55:37.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 --2- 172.21.15.76:0/235068024 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac64100cc0 0x7fac64108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 --2- 172.21.15.76:0/235068024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac64100300 0x7fac64100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 --2- 172.21.15.76:0/235068024 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac64101740 0x7fac64101b60 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 -- 172.21.15.76:0/235068024 >> 172.21.15.76:0/235068024 conn(0x7fac640fadb0 msgr2=0x7fac640fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:37.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 -- 172.21.15.76:0/235068024 shutdown_connections 2024-03-12T05:55:37.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 -- 172.21.15.76:0/235068024 wait complete. 2024-03-12T05:55:37.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 Processor -- start 2024-03-12T05:55:37.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 -- start start 2024-03-12T05:55:37.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac64100300 0x7fac640ffd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:37.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac64100cc0 0x7fac6419bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:37.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac64101740 0x7fac6419e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:37.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6419ed50 con 0x7fac64100cc0 2024-03-12T05:55:37.942 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6419eec0 con 0x7fac64101740 2024-03-12T05:55:37.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac6afbc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6419f080 con 0x7fac64100300 2024-03-12T05:55:37.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac63fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac64100cc0 0x7fac6419bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:37.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac63fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac64100cc0 0x7fac6419bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47422/0 (socket says 172.21.15.76:47422) 2024-03-12T05:55:37.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac63fff640 1 -- 172.21.15.76:0/406734737 learned_addr learned my addr 172.21.15.76:0/406734737 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:37.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.936+0000 7fac69559640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac64101740 0x7fac6419e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:37.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.937+0000 7fac68d58640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac64100300 0x7fac640ffd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:37.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.937+0000 7fac63fff640 1 -- 172.21.15.76:0/406734737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac64100300 msgr2=0x7fac640ffd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.937+0000 7fac63fff640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac64100300 0x7fac640ffd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.937+0000 7fac63fff640 1 -- 172.21.15.76:0/406734737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac64101740 msgr2=0x7fac6419e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:37.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.937+0000 7fac63fff640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac64101740 0x7fac6419e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:37.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.937+0000 7fac63fff640 1 -- 172.21.15.76:0/406734737 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c009900 con 0x7fac64100cc0 2024-03-12T05:55:37.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.937+0000 7fac68d58640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac64100300 0x7fac640ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:37.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.937+0000 7fac63fff640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac64100cc0 0x7fac6419bf80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fac5800c970 tx=0x7fac5800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:37.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.937+0000 7fac61ffb640 1 -- 172.21.15.76:0/406734737 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58004220 con 0x7fac64100cc0 2024-03-12T05:55:37.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.937+0000 7fac61ffb640 1 -- 172.21.15.76:0/406734737 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac58007e90 con 0x7fac64100cc0 2024-03-12T05:55:37.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.937+0000 7fac61ffb640 1 -- 172.21.15.76:0/406734737 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac58004a00 con 0x7fac64100cc0 2024-03-12T05:55:37.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.937+0000 7fac6afbc640 1 -- 172.21.15.76:0/406734737 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6419f360 con 0x7fac64100cc0 2024-03-12T05:55:37.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.937+0000 7fac6afbc640 1 -- 172.21.15.76:0/406734737 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6419f820 con 0x7fac64100cc0 2024-03-12T05:55:37.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.938+0000 7fac6afbc640 1 -- 172.21.15.76:0/406734737 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac64107490 con 0x7fac64100cc0 2024-03-12T05:55:37.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.939+0000 7fac61ffb640 1 -- 172.21.15.76:0/406734737 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fac580079b0 con 0x7fac64100cc0 2024-03-12T05:55:37.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.940+0000 7fac61ffb640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fac4c0a1a30 0x7fac4c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:37.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.940+0000 7fac61ffb640 1 -- 172.21.15.76:0/406734737 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fac580cd4c0 con 0x7fac64100cc0 2024-03-12T05:55:37.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.940+0000 7fac68d58640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fac4c0a1a30 0x7fac4c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:37.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.940+0000 7fac68d58640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fac4c0a1a30 0x7fac4c0a3ef0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fac5c009430 tx=0x7fac5c005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:37.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:37.944+0000 7fac61ffb640 1 -- 172.21.15.76:0/406734737 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fac580966e0 con 0x7fac64100cc0 2024-03-12T05:55:38.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.121+0000 7fac6afbc640 1 -- 172.21.15.76:0/406734737 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac6410bb10 con 0x7fac64100cc0 2024-03-12T05:55:38.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.123+0000 7fac61ffb640 1 -- 172.21.15.76:0/406734737 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22541 (secure 0 0 0) 0x7fac58099da0 con 0x7fac64100cc0 2024-03-12T05:55:38.127 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:38.127 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:54:42.007701+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:55:38.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.125+0000 7fac6afbc640 1 -- 172.21.15.76:0/406734737 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fac4c0a1a30 msgr2=0x7fac4c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.125+0000 7fac6afbc640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fac4c0a1a30 0x7fac4c0a3ef0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fac5c009430 tx=0x7fac5c005c50 comp rx=0 tx=0).stop 2024-03-12T05:55:38.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.125+0000 7fac6afbc640 1 -- 172.21.15.76:0/406734737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac64100cc0 msgr2=0x7fac6419bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.125+0000 7fac6afbc640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac64100cc0 0x7fac6419bf80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fac5800c970 tx=0x7fac5800ce30 comp rx=0 tx=0).stop 2024-03-12T05:55:38.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.125+0000 7fac6afbc640 1 -- 172.21.15.76:0/406734737 shutdown_connections 2024-03-12T05:55:38.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.125+0000 7fac6afbc640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fac4c0a1a30 0x7fac4c0a3ef0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.125+0000 7fac6afbc640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fac64101740 0x7fac6419e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.125+0000 7fac6afbc640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac64100cc0 0x7fac6419bf80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.125+0000 7fac6afbc640 1 --2- 172.21.15.76:0/406734737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac64100300 0x7fac640ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.125+0000 7fac6afbc640 1 -- 172.21.15.76:0/406734737 >> 172.21.15.76:0/406734737 conn(0x7fac640fadb0 msgr2=0x7fac64104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:38.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.126+0000 7fac6afbc640 1 -- 172.21.15.76:0/406734737 shutdown_connections 2024-03-12T05:55:38.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.126+0000 7fac6afbc640 1 -- 172.21.15.76:0/406734737 wait complete. 2024-03-12T05:55:38.139 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:38 smithi076 ceph-mon[32095]: pgmap v346: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-12T05:55:38.140 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:49:09.125536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:53:35.663129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:53:36.740826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T05:55:38.141 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T05:55:38.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.245+0000 7f2de9a2e640 1 -- 172.21.15.76:0/2681375032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2de41000a0 msgr2=0x7f2de41004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.245+0000 7f2de9a2e640 1 --2- 172.21.15.76:0/2681375032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2de41000a0 0x7f2de41004c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4009900 tx=0x7f2dd402bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:38.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.245+0000 7f2de9a2e640 1 -- 172.21.15.76:0/2681375032 shutdown_connections 2024-03-12T05:55:38.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.245+0000 7f2de9a2e640 1 --2- 172.21.15.76:0/2681375032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de4101c50 0x7f2de41084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.245+0000 7f2de9a2e640 1 --2- 172.21.15.76:0/2681375032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2de4101290 0x7f2de4101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.245+0000 7f2de9a2e640 1 --2- 172.21.15.76:0/2681375032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2de41000a0 0x7f2de41004c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.245+0000 7f2de9a2e640 1 -- 172.21.15.76:0/2681375032 >> 172.21.15.76:0/2681375032 conn(0x7f2de40fb610 msgr2=0x7f2de40fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:38.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.245+0000 7f2de9a2e640 1 -- 172.21.15.76:0/2681375032 shutdown_connections 2024-03-12T05:55:38.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.245+0000 7f2de9a2e640 1 -- 172.21.15.76:0/2681375032 wait complete. 2024-03-12T05:55:38.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.246+0000 7f2de9a2e640 1 Processor -- start 2024-03-12T05:55:38.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.246+0000 7f2de9a2e640 1 -- start start 2024-03-12T05:55:38.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.246+0000 7f2de9a2e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de41000a0 0x7f2de4068fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:38.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.246+0000 7f2de9a2e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2de4101290 0x7f2de419c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:38.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.246+0000 7f2de9a2e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2de4101c50 0x7f2de419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:38.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.246+0000 7f2de9a2e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de419ee70 con 0x7f2de4101c50 2024-03-12T05:55:38.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.246+0000 7f2de9a2e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de419efe0 con 0x7f2de4101290 2024-03-12T05:55:38.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.246+0000 7f2de9a2e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de419f1a0 con 0x7f2de41000a0 2024-03-12T05:55:38.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.246+0000 7f2de37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2de4101c50 0x7f2de419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:38.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.246+0000 7f2de37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2de4101c50 0x7f2de419e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47440/0 (socket says 172.21.15.76:47440) 2024-03-12T05:55:38.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.246+0000 7f2de37fe640 1 -- 172.21.15.76:0/1387269496 learned_addr learned my addr 172.21.15.76:0/1387269496 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:38.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.246+0000 7f2de2ffd640 1 --2- 172.21.15.76:0/1387269496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de41000a0 0x7f2de4068fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:38.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.246+0000 7f2de27fc640 1 --2- 172.21.15.76:0/1387269496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2de4101290 0x7f2de419c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:38.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.247+0000 7f2de37fe640 1 -- 172.21.15.76:0/1387269496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de41000a0 msgr2=0x7f2de4068fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.247+0000 7f2de37fe640 1 --2- 172.21.15.76:0/1387269496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de41000a0 0x7f2de4068fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.247+0000 7f2de37fe640 1 -- 172.21.15.76:0/1387269496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2de4101290 msgr2=0x7f2de419c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.247+0000 7f2de37fe640 1 --2- 172.21.15.76:0/1387269496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2de4101290 0x7f2de419c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.247+0000 7f2de37fe640 1 -- 172.21.15.76:0/1387269496 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd4009560 con 0x7f2de4101c50 2024-03-12T05:55:38.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.247+0000 7f2de37fe640 1 --2- 172.21.15.76:0/1387269496 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2de4101c50 0x7f2de419e690 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc00b970 tx=0x7f2ddc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:38.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.247+0000 7f2de8a2c640 1 -- 172.21.15.76:0/1387269496 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc00cb40 con 0x7f2de4101c50 2024-03-12T05:55:38.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.247+0000 7f2de8a2c640 1 -- 172.21.15.76:0/1387269496 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ddc00cca0 con 0x7f2de4101c50 2024-03-12T05:55:38.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.247+0000 7f2de9a2e640 1 -- 172.21.15.76:0/1387269496 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de419f420 con 0x7f2de4101c50 2024-03-12T05:55:38.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.247+0000 7f2de8a2c640 1 -- 172.21.15.76:0/1387269496 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc0077d0 con 0x7f2de4101c50 2024-03-12T05:55:38.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.247+0000 7f2de9a2e640 1 -- 172.21.15.76:0/1387269496 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de419f960 con 0x7f2de4101c50 2024-03-12T05:55:38.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.248+0000 7f2de9a2e640 1 -- 172.21.15.76:0/1387269496 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de4107490 con 0x7f2de4101c50 2024-03-12T05:55:38.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.249+0000 7f2de8a2c640 1 -- 172.21.15.76:0/1387269496 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f2ddc014440 con 0x7f2de4101c50 2024-03-12T05:55:38.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.250+0000 7f2de8a2c640 1 --2- 172.21.15.76:0/1387269496 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2dcc0a1a30 0x7f2dcc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:38.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.250+0000 7f2de2ffd640 1 --2- 172.21.15.76:0/1387269496 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2dcc0a1a30 0x7f2dcc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:38.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.250+0000 7f2de8a2c640 1 -- 172.21.15.76:0/1387269496 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2ddc0cccb0 con 0x7f2de4101c50 2024-03-12T05:55:38.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.250+0000 7f2de2ffd640 1 --2- 172.21.15.76:0/1387269496 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2dcc0a1a30 0x7f2dcc0a3ef0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4005ab0 tx=0x7f2dd4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:38.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.254+0000 7f2de8a2c640 1 -- 172.21.15.76:0/1387269496 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2ddc095f80 con 0x7f2de4101c50 2024-03-12T05:55:38.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.426+0000 7f2de9a2e640 1 -- 172.21.15.76:0/1387269496 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2de402cda0 con 0x7f2de4101c50 2024-03-12T05:55:38.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.426+0000 7f2de8a2c640 1 -- 172.21.15.76:0/1387269496 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2ddc0190a0 con 0x7f2de4101c50 2024-03-12T05:55:38.430 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-12T05:55:38.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.428+0000 7f2de9a2e640 1 -- 172.21.15.76:0/1387269496 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2dcc0a1a30 msgr2=0x7f2dcc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.428+0000 7f2de9a2e640 1 --2- 172.21.15.76:0/1387269496 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2dcc0a1a30 0x7f2dcc0a3ef0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4005ab0 tx=0x7f2dd4005fd0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.428+0000 7f2de9a2e640 1 -- 172.21.15.76:0/1387269496 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2de4101c50 msgr2=0x7f2de419e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.428+0000 7f2de9a2e640 1 --2- 172.21.15.76:0/1387269496 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2de4101c50 0x7f2de419e690 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc00b970 tx=0x7f2ddc00be30 comp rx=0 tx=0).stop 2024-03-12T05:55:38.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.428+0000 7f2de9a2e640 1 -- 172.21.15.76:0/1387269496 shutdown_connections 2024-03-12T05:55:38.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.428+0000 7f2de9a2e640 1 --2- 172.21.15.76:0/1387269496 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f2dcc0a1a30 0x7f2dcc0a3ef0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.428+0000 7f2de9a2e640 1 --2- 172.21.15.76:0/1387269496 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2de4101c50 0x7f2de419e690 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.428+0000 7f2de9a2e640 1 --2- 172.21.15.76:0/1387269496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2de4101290 0x7f2de419c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.428+0000 7f2de9a2e640 1 --2- 172.21.15.76:0/1387269496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de41000a0 0x7f2de4068fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.428+0000 7f2de9a2e640 1 -- 172.21.15.76:0/1387269496 >> 172.21.15.76:0/1387269496 conn(0x7f2de40fb610 msgr2=0x7f2de4104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:38.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.429+0000 7f2de9a2e640 1 -- 172.21.15.76:0/1387269496 shutdown_connections 2024-03-12T05:55:38.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.429+0000 7f2de9a2e640 1 -- 172.21.15.76:0/1387269496 wait complete. 2024-03-12T05:55:38.443 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T05:55:38.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:38 smithi082 ceph-mon[36382]: pgmap v346: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-12T05:55:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:38 smithi088 ceph-mon[36585]: pgmap v346: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-12T05:55:38.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.545+0000 7f5c03c49640 1 -- 172.21.15.76:0/1526611169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bfc101190 msgr2=0x7f5bfc1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.545+0000 7f5c03c49640 1 --2- 172.21.15.76:0/1526611169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bfc101190 0x7f5bfc1015b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009900 tx=0x7f5bf802bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:38.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.545+0000 7f5c03c49640 1 -- 172.21.15.76:0/1526611169 shutdown_connections 2024-03-12T05:55:38.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.545+0000 7f5c03c49640 1 --2- 172.21.15.76:0/1526611169 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5bfc102d40 0x7f5bfc1095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.545+0000 7f5c03c49640 1 --2- 172.21.15.76:0/1526611169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bfc102380 0x7f5bfc102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.545+0000 7f5c03c49640 1 --2- 172.21.15.76:0/1526611169 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bfc101190 0x7f5bfc1015b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.545+0000 7f5c03c49640 1 -- 172.21.15.76:0/1526611169 >> 172.21.15.76:0/1526611169 conn(0x7f5bfc0fc700 msgr2=0x7f5bfc0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:38.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.545+0000 7f5c03c49640 1 -- 172.21.15.76:0/1526611169 shutdown_connections 2024-03-12T05:55:38.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.545+0000 7f5c03c49640 1 -- 172.21.15.76:0/1526611169 wait complete. 2024-03-12T05:55:38.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.546+0000 7f5c03c49640 1 Processor -- start 2024-03-12T05:55:38.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.546+0000 7f5c03c49640 1 -- start start 2024-03-12T05:55:38.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.546+0000 7f5c03c49640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5bfc101190 0x7f5bfc197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:38.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.546+0000 7f5c03c49640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bfc102380 0x7f5bfc1977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:38.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.546+0000 7f5c03c49640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bfc102d40 0x7f5bfc19e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:38.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.546+0000 7f5c03c49640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc19ed50 con 0x7f5bfc102d40 2024-03-12T05:55:38.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.546+0000 7f5c03c49640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc19eec0 con 0x7f5bfc101190 2024-03-12T05:55:38.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.546+0000 7f5c03c49640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc19f080 con 0x7f5bfc102380 2024-03-12T05:55:38.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.546+0000 7f5c021e6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bfc102d40 0x7f5bfc19e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.546+0000 7f5c019e5640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5bfc101190 0x7f5bfc197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.546+0000 7f5c021e6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bfc102d40 0x7f5bfc19e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47464/0 (socket says 172.21.15.76:47464) 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.546+0000 7f5c021e6640 1 -- 172.21.15.76:0/1423197770 learned_addr learned my addr 172.21.15.76:0/1423197770 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.546+0000 7f5c011e4640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bfc102380 0x7f5bfc1977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.547+0000 7f5c021e6640 1 -- 172.21.15.76:0/1423197770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bfc102380 msgr2=0x7f5bfc1977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.547+0000 7f5c021e6640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bfc102380 0x7f5bfc1977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.547+0000 7f5c021e6640 1 -- 172.21.15.76:0/1423197770 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5bfc101190 msgr2=0x7f5bfc197260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.547+0000 7f5c021e6640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5bfc101190 0x7f5bfc197260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.547+0000 7f5c021e6640 1 -- 172.21.15.76:0/1423197770 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf8009560 con 0x7f5bfc102d40 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.547+0000 7f5c011e4640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bfc102380 0x7f5bfc1977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.547+0000 7f5c019e5640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5bfc101190 0x7f5bfc197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.547+0000 7f5c021e6640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bfc102d40 0x7f5bfc19e810 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400eb10 tx=0x7f5bf400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.547+0000 7f5bf2ffd640 1 -- 172.21.15.76:0/1423197770 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4004b20 con 0x7f5bfc102d40 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.547+0000 7f5c03c49640 1 -- 172.21.15.76:0/1423197770 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc19f360 con 0x7f5bfc102d40 2024-03-12T05:55:38.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.547+0000 7f5bf2ffd640 1 -- 172.21.15.76:0/1423197770 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bf4004c80 con 0x7f5bfc102d40 2024-03-12T05:55:38.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.547+0000 7f5bf2ffd640 1 -- 172.21.15.76:0/1423197770 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4010650 con 0x7f5bfc102d40 2024-03-12T05:55:38.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.547+0000 7f5c03c49640 1 -- 172.21.15.76:0/1423197770 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc19f820 con 0x7f5bfc102d40 2024-03-12T05:55:38.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.548+0000 7f5c03c49640 1 -- 172.21.15.76:0/1423197770 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bfc108550 con 0x7f5bfc102d40 2024-03-12T05:55:38.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.549+0000 7f5bf2ffd640 1 -- 172.21.15.76:0/1423197770 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f5bf4005020 con 0x7f5bfc102d40 2024-03-12T05:55:38.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.549+0000 7f5bf2ffd640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5be80a1a30 0x7f5be80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:38.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.550+0000 7f5c019e5640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5be80a1a30 0x7f5be80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:38.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.550+0000 7f5bf2ffd640 1 -- 172.21.15.76:0/1423197770 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5bf4014070 con 0x7f5bfc102d40 2024-03-12T05:55:38.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.550+0000 7f5c019e5640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5be80a1a30 0x7f5be80a3ef0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8005ab0 tx=0x7f5bf8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:38.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.554+0000 7f5bf2ffd640 1 -- 172.21.15.76:0/1423197770 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5bf4096c80 con 0x7f5bfc102d40 2024-03-12T05:55:38.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.728+0000 7f5c03c49640 1 -- 172.21.15.76:0/1423197770 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5bfc02ce40 con 0x7f5bfc102d40 2024-03-12T05:55:38.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.728+0000 7f5bf2ffd640 1 -- 172.21.15.76:0/1423197770 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5bf409a340 con 0x7f5bfc102d40 2024-03-12T05:55:38.732 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-12T05:55:38.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.730+0000 7f5c03c49640 1 -- 172.21.15.76:0/1423197770 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5be80a1a30 msgr2=0x7f5be80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.730+0000 7f5c03c49640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5be80a1a30 0x7f5be80a3ef0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8005ab0 tx=0x7f5bf8005fd0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.730+0000 7f5c03c49640 1 -- 172.21.15.76:0/1423197770 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bfc102d40 msgr2=0x7f5bfc19e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.730+0000 7f5c03c49640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bfc102d40 0x7f5bfc19e810 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400eb10 tx=0x7f5bf400c2d0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.730+0000 7f5c03c49640 1 -- 172.21.15.76:0/1423197770 shutdown_connections 2024-03-12T05:55:38.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.730+0000 7f5c03c49640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f5be80a1a30 0x7f5be80a3ef0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.730+0000 7f5c03c49640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5bfc102d40 0x7f5bfc19e810 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.730+0000 7f5c03c49640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bfc102380 0x7f5bfc1977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.730+0000 7f5c03c49640 1 --2- 172.21.15.76:0/1423197770 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5bfc101190 0x7f5bfc197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.730+0000 7f5c03c49640 1 -- 172.21.15.76:0/1423197770 >> 172.21.15.76:0/1423197770 conn(0x7f5bfc0fc700 msgr2=0x7f5bfc105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:38.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.730+0000 7f5c03c49640 1 -- 172.21.15.76:0/1423197770 shutdown_connections 2024-03-12T05:55:38.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.731+0000 7f5c03c49640 1 -- 172.21.15.76:0/1423197770 wait complete. 2024-03-12T05:55:38.745 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T05:55:38.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.846+0000 7f6706ad4640 1 -- 172.21.15.76:0/690607264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6700104610 msgr2=0x7f6700106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.846+0000 7f6706ad4640 1 --2- 172.21.15.76:0/690607264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6700104610 0x7f6700106a00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f66f4009900 tx=0x7f66f402bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:38.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 -- 172.21.15.76:0/690607264 shutdown_connections 2024-03-12T05:55:38.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 --2- 172.21.15.76:0/690607264 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6700106f40 0x7f6700109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 --2- 172.21.15.76:0/690607264 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6700104610 0x7f6700106a00 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 --2- 172.21.15.76:0/690607264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6700101ce0 0x7f67001040d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 -- 172.21.15.76:0/690607264 >> 172.21.15.76:0/690607264 conn(0x7f67000fb610 msgr2=0x7f67000fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:38.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 -- 172.21.15.76:0/690607264 shutdown_connections 2024-03-12T05:55:38.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 -- 172.21.15.76:0/690607264 wait complete. 2024-03-12T05:55:38.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 Processor -- start 2024-03-12T05:55:38.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 -- start start 2024-03-12T05:55:38.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6700101ce0 0x7f6700197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:38.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6700104610 0x7f67001977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:38.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6700106f40 0x7f670019e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:38.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670019ed80 con 0x7f6700104610 2024-03-12T05:55:38.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670019eef0 con 0x7f6700101ce0 2024-03-12T05:55:38.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f6706ad4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670019f0b0 con 0x7f6700106f40 2024-03-12T05:55:38.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f66fffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6700104610 0x7f67001977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:38.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f66fffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6700104610 0x7f67001977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47482/0 (socket says 172.21.15.76:47482) 2024-03-12T05:55:38.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.847+0000 7f66fffff640 1 -- 172.21.15.76:0/3010946415 learned_addr learned my addr 172.21.15.76:0/3010946415 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:38.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.848+0000 7f6704870640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6700101ce0 0x7f6700197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:38.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.848+0000 7f6705071640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6700106f40 0x7f670019e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:38.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.848+0000 7f66fffff640 1 -- 172.21.15.76:0/3010946415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6700106f40 msgr2=0x7f670019e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.848+0000 7f66fffff640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6700106f40 0x7f670019e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.848+0000 7f66fffff640 1 -- 172.21.15.76:0/3010946415 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6700101ce0 msgr2=0x7f6700197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:38.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.848+0000 7f66fffff640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6700101ce0 0x7f6700197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:38.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.848+0000 7f66fffff640 1 -- 172.21.15.76:0/3010946415 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f4009560 con 0x7f6700104610 2024-03-12T05:55:38.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.848+0000 7f6705071640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6700106f40 0x7f670019e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:38.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.848+0000 7f66fffff640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6700104610 0x7f67001977d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f66f4005ab0 tx=0x7f66f4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:38.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.848+0000 7f66fdffb640 1 -- 172.21.15.76:0/3010946415 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f4004aa0 con 0x7f6700104610 2024-03-12T05:55:38.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.848+0000 7f66fdffb640 1 -- 172.21.15.76:0/3010946415 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66f40052d0 con 0x7f6700104610 2024-03-12T05:55:38.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.848+0000 7f6706ad4640 1 -- 172.21.15.76:0/3010946415 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f670019f330 con 0x7f6700104610 2024-03-12T05:55:38.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.848+0000 7f66fdffb640 1 -- 172.21.15.76:0/3010946415 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f4003e80 con 0x7f6700104610 2024-03-12T05:55:38.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.848+0000 7f6706ad4640 1 -- 172.21.15.76:0/3010946415 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f670019f840 con 0x7f6700104610 2024-03-12T05:55:38.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.849+0000 7f6706ad4640 1 -- 172.21.15.76:0/3010946415 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67000fd100 con 0x7f6700104610 2024-03-12T05:55:38.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.850+0000 7f66fdffb640 1 -- 172.21.15.76:0/3010946415 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f66f4005000 con 0x7f6700104610 2024-03-12T05:55:38.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.851+0000 7f66fdffb640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f66e80a1a30 0x7f66e80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:38.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.851+0000 7f6704870640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f66e80a1a30 0x7f66e80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:38.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.851+0000 7f66fdffb640 1 -- 172.21.15.76:0/3010946415 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f66f40efec0 con 0x7f6700104610 2024-03-12T05:55:38.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.852+0000 7f6704870640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f66e80a1a30 0x7f66e80a3ef0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f66f0005fd0 tx=0x7f66f000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:38.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:38.855+0000 7f66fdffb640 1 -- 172.21.15.76:0/3010946415 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f66f40b90e0 con 0x7f6700104610 2024-03-12T05:55:39.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.027+0000 7f6706ad4640 1 -- 172.21.15.76:0/3010946415 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f670010ce10 con 0x7f6700104610 2024-03-12T05:55:39.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.028+0000 7f66fdffb640 1 -- 172.21.15.76:0/3010946415 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f66f40bc7a0 con 0x7f6700104610 2024-03-12T05:55:39.031 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-12T05:55:39.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.030+0000 7f6706ad4640 1 -- 172.21.15.76:0/3010946415 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f66e80a1a30 msgr2=0x7f66e80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.030+0000 7f6706ad4640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f66e80a1a30 0x7f66e80a3ef0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f66f0005fd0 tx=0x7f66f000b390 comp rx=0 tx=0).stop 2024-03-12T05:55:39.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.030+0000 7f6706ad4640 1 -- 172.21.15.76:0/3010946415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6700104610 msgr2=0x7f67001977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.030+0000 7f6706ad4640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6700104610 0x7f67001977d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f66f4005ab0 tx=0x7f66f4004270 comp rx=0 tx=0).stop 2024-03-12T05:55:39.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.030+0000 7f6706ad4640 1 -- 172.21.15.76:0/3010946415 shutdown_connections 2024-03-12T05:55:39.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.030+0000 7f6706ad4640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f66e80a1a30 0x7f66e80a3ef0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.030+0000 7f6706ad4640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6700106f40 0x7f670019e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.030+0000 7f6706ad4640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6700104610 0x7f67001977d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.030+0000 7f6706ad4640 1 --2- 172.21.15.76:0/3010946415 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6700101ce0 0x7f6700197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.030+0000 7f6706ad4640 1 -- 172.21.15.76:0/3010946415 >> 172.21.15.76:0/3010946415 conn(0x7f67000fb610 msgr2=0x7f67001025a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.030+0000 7f6706ad4640 1 -- 172.21.15.76:0/3010946415 shutdown_connections 2024-03-12T05:55:39.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.030+0000 7f6706ad4640 1 -- 172.21.15.76:0/3010946415 wait complete. 2024-03-12T05:55:39.045 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:55:39.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.142+0000 7f603c615640 1 -- 172.21.15.76:0/1756225419 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f60340ffc40 msgr2=0x7f60341097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.142+0000 7f603c615640 1 --2- 172.21.15.76:0/1756225419 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f60340ffc40 0x7f60341097f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f602c009ca0 tx=0x7f602c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.142+0000 7f603c615640 1 -- 172.21.15.76:0/1756225419 shutdown_connections 2024-03-12T05:55:39.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.142+0000 7f603c615640 1 --2- 172.21.15.76:0/1756225419 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f60340ffc40 0x7f60341097f0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.142+0000 7f603c615640 1 --2- 172.21.15.76:0/1756225419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60340ff280 0x7f60340ff700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.142+0000 7f603c615640 1 --2- 172.21.15.76:0/1756225419 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6034102810 0x7f6034102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.142+0000 7f603c615640 1 -- 172.21.15.76:0/1756225419 >> 172.21.15.76:0/1756225419 conn(0x7f60340fae00 msgr2=0x7f60340fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.142+0000 7f603c615640 1 -- 172.21.15.76:0/1756225419 shutdown_connections 2024-03-12T05:55:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.142+0000 7f603c615640 1 -- 172.21.15.76:0/1756225419 wait complete. 2024-03-12T05:55:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f603c615640 1 Processor -- start 2024-03-12T05:55:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f603c615640 1 -- start start 2024-03-12T05:55:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f603c615640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60340ff280 0x7f6034197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f603c615640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f60340ffc40 0x7f60341976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f603c615640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6034102810 0x7f603419e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f603c615640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603419ec80 con 0x7f60340ffc40 2024-03-12T05:55:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f603c615640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603419edf0 con 0x7f6034102810 2024-03-12T05:55:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f603c615640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603419efb0 con 0x7f60340ff280 2024-03-12T05:55:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f6039bb0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f60340ffc40 0x7f60341976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f6039bb0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f60340ffc40 0x7f60341976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47516/0 (socket says 172.21.15.76:47516) 2024-03-12T05:55:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f6039bb0640 1 -- 172.21.15.76:0/2703560982 learned_addr learned my addr 172.21.15.76:0/2703560982 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f603a3b1640 1 --2- 172.21.15.76:0/2703560982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60340ff280 0x7f6034197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f603abb2640 1 --2- 172.21.15.76:0/2703560982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6034102810 0x7f603419e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f6039bb0640 1 -- 172.21.15.76:0/2703560982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60340ff280 msgr2=0x7f6034197190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f6039bb0640 1 --2- 172.21.15.76:0/2703560982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60340ff280 0x7f6034197190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f6039bb0640 1 -- 172.21.15.76:0/2703560982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6034102810 msgr2=0x7f603419e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f6039bb0640 1 --2- 172.21.15.76:0/2703560982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6034102810 0x7f603419e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.143+0000 7f6039bb0640 1 -- 172.21.15.76:0/2703560982 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c009900 con 0x7f60340ffc40 2024-03-12T05:55:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.144+0000 7f6039bb0640 1 --2- 172.21.15.76:0/2703560982 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f60340ffc40 0x7f60341976d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f602400d770 tx=0x7f602400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.144+0000 7f602b7fe640 1 -- 172.21.15.76:0/2703560982 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6024009c30 con 0x7f60340ffc40 2024-03-12T05:55:39.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.144+0000 7f602b7fe640 1 -- 172.21.15.76:0/2703560982 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f602400de80 con 0x7f60340ffc40 2024-03-12T05:55:39.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.144+0000 7f603c615640 1 -- 172.21.15.76:0/2703560982 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603419f290 con 0x7f60340ffc40 2024-03-12T05:55:39.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.144+0000 7f602b7fe640 1 -- 172.21.15.76:0/2703560982 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6024004e80 con 0x7f60340ffc40 2024-03-12T05:55:39.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.144+0000 7f603c615640 1 -- 172.21.15.76:0/2703560982 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603419f750 con 0x7f60340ffc40 2024-03-12T05:55:39.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.145+0000 7f603c615640 1 -- 172.21.15.76:0/2703560982 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6034108600 con 0x7f60340ffc40 2024-03-12T05:55:39.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.146+0000 7f602b7fe640 1 -- 172.21.15.76:0/2703560982 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6024005020 con 0x7f60340ffc40 2024-03-12T05:55:39.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.146+0000 7f602b7fe640 1 --2- 172.21.15.76:0/2703560982 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f60200a1a30 0x7f60200a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.146+0000 7f603a3b1640 1 --2- 172.21.15.76:0/2703560982 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f60200a1a30 0x7f60200a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.147+0000 7f602b7fe640 1 -- 172.21.15.76:0/2703560982 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f60240ccee0 con 0x7f60340ffc40 2024-03-12T05:55:39.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.147+0000 7f603a3b1640 1 --2- 172.21.15.76:0/2703560982 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f60200a1a30 0x7f60200a3ef0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6030005fd0 tx=0x7f6030005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.151+0000 7f602b7fe640 1 -- 172.21.15.76:0/2703560982 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6024096100 con 0x7f60340ffc40 2024-03-12T05:55:39.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.359+0000 7f603c615640 1 -- 172.21.15.76:0/2703560982 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f603402ce40 con 0x7f60340ffc40 2024-03-12T05:55:39.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.360+0000 7f602b7fe640 1 -- 172.21.15.76:0/2703560982 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11328 (secure 0 0 0) 0x7f6024017090 con 0x7f60340ffc40 2024-03-12T05:55:39.363 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:39.363 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:55:33.056004+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:55:39.364 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 22 2024-03-12T05:55:39.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.362+0000 7f603c615640 1 -- 172.21.15.76:0/2703560982 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f60200a1a30 msgr2=0x7f60200a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.362+0000 7f603c615640 1 --2- 172.21.15.76:0/2703560982 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f60200a1a30 0x7f60200a3ef0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6030005fd0 tx=0x7f6030005850 comp rx=0 tx=0).stop 2024-03-12T05:55:39.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.362+0000 7f603c615640 1 -- 172.21.15.76:0/2703560982 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f60340ffc40 msgr2=0x7f60341976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.362+0000 7f603c615640 1 --2- 172.21.15.76:0/2703560982 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f60340ffc40 0x7f60341976d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f602400d770 tx=0x7f602400dc30 comp rx=0 tx=0).stop 2024-03-12T05:55:39.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.362+0000 7f603c615640 1 -- 172.21.15.76:0/2703560982 shutdown_connections 2024-03-12T05:55:39.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.362+0000 7f603c615640 1 --2- 172.21.15.76:0/2703560982 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f60200a1a30 0x7f60200a3ef0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.362+0000 7f603c615640 1 --2- 172.21.15.76:0/2703560982 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6034102810 0x7f603419e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.362+0000 7f603c615640 1 --2- 172.21.15.76:0/2703560982 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f60340ffc40 0x7f60341976d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.362+0000 7f603c615640 1 --2- 172.21.15.76:0/2703560982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60340ff280 0x7f6034197190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.362+0000 7f603c615640 1 -- 172.21.15.76:0/2703560982 >> 172.21.15.76:0/2703560982 conn(0x7f60340fae00 msgr2=0x7f60341057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.362+0000 7f603c615640 1 -- 172.21.15.76:0/2703560982 shutdown_connections 2024-03-12T05:55:39.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.362+0000 7f603c615640 1 -- 172.21.15.76:0/2703560982 wait complete. 2024-03-12T05:55:39.375 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:39 smithi076 ceph-mon[32095]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:55:39.375 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:39 smithi076 ceph-mon[32095]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:55:39.375 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:39 smithi076 ceph-mon[32095]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:55:39.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:39 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/406734737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:39.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:39 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1387269496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:39.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:39 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1423197770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:39.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:39 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3010946415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:39.377 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T05:55:39.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:39 smithi082 ceph-mon[36382]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:55:39.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:39 smithi082 ceph-mon[36382]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:55:39.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:39 smithi082 ceph-mon[36382]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:55:39.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:39 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/406734737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:39.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:39 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1387269496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:39.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:39 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1423197770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:39.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:39 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3010946415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:39.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.476+0000 7f955d3d7640 1 -- 172.21.15.76:0/17842679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9558104560 msgr2=0x7f9558106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.476+0000 7f955d3d7640 1 --2- 172.21.15.76:0/17842679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9558104560 0x7f9558106950 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f954c009900 tx=0x7f954c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:39.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.476+0000 7f955d3d7640 1 -- 172.21.15.76:0/17842679 shutdown_connections 2024-03-12T05:55:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.476+0000 7f955d3d7640 1 --2- 172.21.15.76:0/17842679 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9558106e90 0x7f95581092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.476+0000 7f955d3d7640 1 --2- 172.21.15.76:0/17842679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9558104560 0x7f9558106950 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.476+0000 7f955d3d7640 1 --2- 172.21.15.76:0/17842679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9558101c30 0x7f9558104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.476+0000 7f955d3d7640 1 -- 172.21.15.76:0/17842679 >> 172.21.15.76:0/17842679 conn(0x7f95580fb640 msgr2=0x7f95580fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.476+0000 7f955d3d7640 1 -- 172.21.15.76:0/17842679 shutdown_connections 2024-03-12T05:55:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.476+0000 7f955d3d7640 1 -- 172.21.15.76:0/17842679 wait complete. 2024-03-12T05:55:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.477+0000 7f955d3d7640 1 Processor -- start 2024-03-12T05:55:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.477+0000 7f955d3d7640 1 -- start start 2024-03-12T05:55:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.477+0000 7f955d3d7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9558101c30 0x7f9558197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.477+0000 7f955d3d7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9558104560 0x7f9558197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.477+0000 7f955d3d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9558106e90 0x7f955819e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.477+0000 7f955d3d7640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955819ed10 con 0x7f9558104560 2024-03-12T05:55:39.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.477+0000 7f955d3d7640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955819ee80 con 0x7f9558101c30 2024-03-12T05:55:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.477+0000 7f955d3d7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955819f040 con 0x7f9558106e90 2024-03-12T05:55:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.477+0000 7f95567fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9558104560 0x7f9558197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.477+0000 7f95567fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9558104560 0x7f9558197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47534/0 (socket says 172.21.15.76:47534) 2024-03-12T05:55:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.477+0000 7f95567fc640 1 -- 172.21.15.76:0/1366106791 learned_addr learned my addr 172.21.15.76:0/1366106791 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.477+0000 7f9556ffd640 1 --2- 172.21.15.76:0/1366106791 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9558101c30 0x7f9558197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.478+0000 7f95567fc640 1 -- 172.21.15.76:0/1366106791 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9558106e90 msgr2=0x7f955819e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.478+0000 7f95567fc640 1 --2- 172.21.15.76:0/1366106791 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9558106e90 0x7f955819e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.478+0000 7f95567fc640 1 -- 172.21.15.76:0/1366106791 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9558101c30 msgr2=0x7f9558197220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.478+0000 7f95567fc640 1 --2- 172.21.15.76:0/1366106791 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9558101c30 0x7f9558197220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.478+0000 7f95567fc640 1 -- 172.21.15.76:0/1366106791 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c009560 con 0x7f9558104560 2024-03-12T05:55:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.478+0000 7f95567fc640 1 --2- 172.21.15.76:0/1366106791 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9558104560 0x7f9558197760 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f954c000c00 tx=0x7f954c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.478+0000 7f953ffff640 1 -- 172.21.15.76:0/1366106791 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c004df0 con 0x7f9558104560 2024-03-12T05:55:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.478+0000 7f955d3d7640 1 -- 172.21.15.76:0/1366106791 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955819f2c0 con 0x7f9558104560 2024-03-12T05:55:39.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.478+0000 7f953ffff640 1 -- 172.21.15.76:0/1366106791 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f954c005430 con 0x7f9558104560 2024-03-12T05:55:39.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.478+0000 7f955d3d7640 1 -- 172.21.15.76:0/1366106791 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955819f820 con 0x7f9558104560 2024-03-12T05:55:39.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.478+0000 7f953ffff640 1 -- 172.21.15.76:0/1366106791 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954c03e450 con 0x7f9558104560 2024-03-12T05:55:39.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:39 smithi088 ceph-mon[36585]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:55:39.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:39 smithi088 ceph-mon[36585]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:55:39.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:39 smithi088 ceph-mon[36585]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T05:55:39.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:39 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/406734737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:39.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:39 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1387269496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T05:55:39.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:39 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1423197770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T05:55:39.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:39 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3010946415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T05:55:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.479+0000 7f955d3d7640 1 -- 172.21.15.76:0/1366106791 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95580fd130 con 0x7f9558104560 2024-03-12T05:55:39.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.480+0000 7f953ffff640 1 -- 172.21.15.76:0/1366106791 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f954c003680 con 0x7f9558104560 2024-03-12T05:55:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.481+0000 7f953ffff640 1 --2- 172.21.15.76:0/1366106791 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f95400a1a30 0x7f95400a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.481+0000 7f953ffff640 1 -- 172.21.15.76:0/1366106791 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f954c0ef690 con 0x7f9558104560 2024-03-12T05:55:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.481+0000 7f9556ffd640 1 --2- 172.21.15.76:0/1366106791 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f95400a1a30 0x7f95400a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.481+0000 7f9556ffd640 1 --2- 172.21.15.76:0/1366106791 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f95400a1a30 0x7f95400a3ef0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f954800bbe0 tx=0x7f954800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.485+0000 7f953ffff640 1 -- 172.21.15.76:0/1366106791 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f954c0b88b0 con 0x7f9558104560 2024-03-12T05:55:39.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.659+0000 7f955d3d7640 1 -- 172.21.15.76:0/1366106791 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f955802cdd0 con 0x7f9558104560 2024-03-12T05:55:39.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.661+0000 7f953ffff640 1 -- 172.21.15.76:0/1366106791 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22541 (secure 0 0 0) 0x7f954c0bbf70 con 0x7f9558104560 2024-03-12T05:55:39.664 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:55:39.665 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":88,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T05:54:42.007701+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T05:55:39.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.663+0000 7f955d3d7640 1 -- 172.21.15.76:0/1366106791 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f95400a1a30 msgr2=0x7f95400a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.663+0000 7f955d3d7640 1 --2- 172.21.15.76:0/1366106791 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f95400a1a30 0x7f95400a3ef0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f954800bbe0 tx=0x7f954800b390 comp rx=0 tx=0).stop 2024-03-12T05:55:39.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.663+0000 7f955d3d7640 1 -- 172.21.15.76:0/1366106791 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9558104560 msgr2=0x7f9558197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.663+0000 7f955d3d7640 1 --2- 172.21.15.76:0/1366106791 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9558104560 0x7f9558197760 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f954c000c00 tx=0x7f954c004270 comp rx=0 tx=0).stop 2024-03-12T05:55:39.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.663+0000 7f955d3d7640 1 -- 172.21.15.76:0/1366106791 shutdown_connections 2024-03-12T05:55:39.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.663+0000 7f955d3d7640 1 --2- 172.21.15.76:0/1366106791 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f95400a1a30 0x7f95400a3ef0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.663+0000 7f955d3d7640 1 --2- 172.21.15.76:0/1366106791 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9558106e90 0x7f955819e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.663+0000 7f955d3d7640 1 --2- 172.21.15.76:0/1366106791 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9558104560 0x7f9558197760 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.663+0000 7f955d3d7640 1 --2- 172.21.15.76:0/1366106791 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9558101c30 0x7f9558197220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.663+0000 7f955d3d7640 1 -- 172.21.15.76:0/1366106791 >> 172.21.15.76:0/1366106791 conn(0x7f95580fb640 msgr2=0x7f9558102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.663+0000 7f955d3d7640 1 -- 172.21.15.76:0/1366106791 shutdown_connections 2024-03-12T05:55:39.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.663+0000 7f955d3d7640 1 -- 172.21.15.76:0/1366106791 wait complete. 2024-03-12T05:55:39.679 DEBUG:tasks.fwd_scrub:Yielding 2024-03-12T05:55:39.679 INFO:teuthology.run_tasks:Running task exec... 2024-03-12T05:55:39.688 INFO:teuthology.task.exec:Executing custom commands... 2024-03-12T05:55:39.688 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2024-03-12T05:55:39.689 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-12T05:55:39.689 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-12T05:55:39.689 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:55:39.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:55:39.689 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:55:39.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.790+0000 7f6b5393a640 1 -- 172.21.15.76:0/1144466040 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4c101c90 msgr2=0x7f6b4c104080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.790+0000 7f6b5393a640 1 --2- 172.21.15.76:0/1144466040 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4c101c90 0x7f6b4c104080 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009900 tx=0x7f6b4402bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:39.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.790+0000 7f6b5393a640 1 -- 172.21.15.76:0/1144466040 shutdown_connections 2024-03-12T05:55:39.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.790+0000 7f6b5393a640 1 --2- 172.21.15.76:0/1144466040 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b4c106ef0 0x7f6b4c1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.790+0000 7f6b5393a640 1 --2- 172.21.15.76:0/1144466040 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b4c1045c0 0x7f6b4c1069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.790+0000 7f6b5393a640 1 --2- 172.21.15.76:0/1144466040 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4c101c90 0x7f6b4c104080 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.790+0000 7f6b5393a640 1 -- 172.21.15.76:0/1144466040 >> 172.21.15.76:0/1144466040 conn(0x7f6b4c0fb620 msgr2=0x7f6b4c0fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.790+0000 7f6b5393a640 1 -- 172.21.15.76:0/1144466040 shutdown_connections 2024-03-12T05:55:39.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.790+0000 7f6b5393a640 1 -- 172.21.15.76:0/1144466040 wait complete. 2024-03-12T05:55:39.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b5393a640 1 Processor -- start 2024-03-12T05:55:39.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b5393a640 1 -- start start 2024-03-12T05:55:39.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b5393a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4c101c90 0x7f6b4c197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b5393a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b4c1045c0 0x7f6b4c1977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b5393a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b4c106ef0 0x7f6b4c19e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b5393a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c19ed60 con 0x7f6b4c101c90 2024-03-12T05:55:39.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b5393a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c19eed0 con 0x7f6b4c106ef0 2024-03-12T05:55:39.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b5393a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c19f090 con 0x7f6b4c1045c0 2024-03-12T05:55:39.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b516d6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4c101c90 0x7f6b4c197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b50ed5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b4c1045c0 0x7f6b4c1977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b516d6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4c101c90 0x7f6b4c197270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47552/0 (socket says 172.21.15.76:47552) 2024-03-12T05:55:39.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b516d6640 1 -- 172.21.15.76:0/2113346888 learned_addr learned my addr 172.21.15.76:0/2113346888 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:39.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b51ed7640 1 --2- 172.21.15.76:0/2113346888 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b4c106ef0 0x7f6b4c19e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b516d6640 1 -- 172.21.15.76:0/2113346888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b4c1045c0 msgr2=0x7f6b4c1977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b516d6640 1 --2- 172.21.15.76:0/2113346888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b4c1045c0 0x7f6b4c1977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b516d6640 1 -- 172.21.15.76:0/2113346888 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b4c106ef0 msgr2=0x7f6b4c19e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b516d6640 1 --2- 172.21.15.76:0/2113346888 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b4c106ef0 0x7f6b4c19e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b516d6640 1 -- 172.21.15.76:0/2113346888 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b44009560 con 0x7f6b4c101c90 2024-03-12T05:55:39.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.791+0000 7f6b516d6640 1 --2- 172.21.15.76:0/2113346888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4c101c90 0x7f6b4c197270 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6b44006010 tx=0x7f6b440040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.792+0000 7f6b427fc640 1 -- 172.21.15.76:0/2113346888 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b440044c0 con 0x7f6b4c101c90 2024-03-12T05:55:39.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.792+0000 7f6b427fc640 1 -- 172.21.15.76:0/2113346888 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b440052d0 con 0x7f6b4c101c90 2024-03-12T05:55:39.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.792+0000 7f6b5393a640 1 -- 172.21.15.76:0/2113346888 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4c19f310 con 0x7f6b4c101c90 2024-03-12T05:55:39.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.792+0000 7f6b427fc640 1 -- 172.21.15.76:0/2113346888 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b44003e10 con 0x7f6b4c101c90 2024-03-12T05:55:39.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.792+0000 7f6b5393a640 1 -- 172.21.15.76:0/2113346888 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4c19f820 con 0x7f6b4c101c90 2024-03-12T05:55:39.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.793+0000 7f6b5393a640 1 -- 172.21.15.76:0/2113346888 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4c0fd110 con 0x7f6b4c101c90 2024-03-12T05:55:39.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.793+0000 7fa7c7695640 1 -- 172.21.15.76:0/105919474 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7c0101290 msgr2=0x7fa7c0101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.793+0000 7fa7c7695640 1 --2- 172.21.15.76:0/105919474 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7c0101290 0x7fa7c0101710 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009900 tx=0x7fa7b002bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:39.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.793+0000 7fa7c7695640 1 -- 172.21.15.76:0/105919474 shutdown_connections 2024-03-12T05:55:39.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.793+0000 7fa7c7695640 1 --2- 172.21.15.76:0/105919474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7c0101c50 0x7fa7c01084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.793+0000 7fa7c7695640 1 --2- 172.21.15.76:0/105919474 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7c0101290 0x7fa7c0101710 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.793+0000 7fa7c7695640 1 --2- 172.21.15.76:0/105919474 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7c01000a0 0x7fa7c01004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.793+0000 7fa7c7695640 1 -- 172.21.15.76:0/105919474 >> 172.21.15.76:0/105919474 conn(0x7fa7c00fb610 msgr2=0x7fa7c00fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.793+0000 7fa7c7695640 1 -- 172.21.15.76:0/105919474 shutdown_connections 2024-03-12T05:55:39.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.793+0000 7fa7c7695640 1 -- 172.21.15.76:0/105919474 wait complete. 2024-03-12T05:55:39.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7f6b427fc640 1 -- 172.21.15.76:0/2113346888 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f6b44005440 con 0x7f6b4c101c90 2024-03-12T05:55:39.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7fa7c7695640 1 Processor -- start 2024-03-12T05:55:39.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7fa7c7695640 1 -- start start 2024-03-12T05:55:39.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7fa7c7695640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7c01000a0 0x7fa7c01971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7fa7c7695640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7c0101290 0x7fa7c0197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7fa7c7695640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7c0101c50 0x7fa7c019e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7fa7c7695640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c019ecd0 con 0x7fa7c0101c50 2024-03-12T05:55:39.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7fa7c7695640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c019ee40 con 0x7fa7c01000a0 2024-03-12T05:55:39.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7fa7c7695640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c019f000 con 0x7fa7c0101290 2024-03-12T05:55:39.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7fa7c5c32640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7c0101c50 0x7fa7c019e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7fa7c5c32640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7c0101c50 0x7fa7c019e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47562/0 (socket says 172.21.15.76:47562) 2024-03-12T05:55:39.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7fa7c5c32640 1 -- 172.21.15.76:0/3270388088 learned_addr learned my addr 172.21.15.76:0/3270388088 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:39.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7fa7c4c30640 1 --2- 172.21.15.76:0/3270388088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7c0101290 0x7fa7c0197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7f6b427fc640 1 --2- 172.21.15.76:0/2113346888 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b380a1a20 0x7f6b380a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.794+0000 7fa7c5431640 1 --2- 172.21.15.76:0/3270388088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7c01000a0 0x7fa7c01971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.795+0000 7f6b50ed5640 1 --2- 172.21.15.76:0/2113346888 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b380a1a20 0x7f6b380a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.795+0000 7f6b427fc640 1 -- 172.21.15.76:0/2113346888 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6b440ef700 con 0x7f6b4c101c90 2024-03-12T05:55:39.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.795+0000 7fa7c5c32640 1 -- 172.21.15.76:0/3270388088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7c0101290 msgr2=0x7fa7c0197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.795+0000 7fa7c5c32640 1 --2- 172.21.15.76:0/3270388088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7c0101290 0x7fa7c0197720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.795+0000 7fa7c5c32640 1 -- 172.21.15.76:0/3270388088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7c01000a0 msgr2=0x7fa7c01971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.795+0000 7fa7c5c32640 1 --2- 172.21.15.76:0/3270388088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7c01000a0 0x7fa7c01971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.795+0000 7fa7c5c32640 1 -- 172.21.15.76:0/3270388088 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b0009560 con 0x7fa7c0101c50 2024-03-12T05:55:39.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.795+0000 7fa7c5c32640 1 --2- 172.21.15.76:0/3270388088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7c0101c50 0x7fa7c019e790 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa7b800eaa0 tx=0x7fa7b800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.795+0000 7f6b50ed5640 1 --2- 172.21.15.76:0/2113346888 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b380a1a20 0x7f6b380a3ee0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c198780 tx=0x7f6b3c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.799+0000 7fa7b67fc640 1 -- 172.21.15.76:0/3270388088 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b800cdb0 con 0x7fa7c0101c50 2024-03-12T05:55:39.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.799+0000 7fa7b67fc640 1 -- 172.21.15.76:0/3270388088 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7b8005020 con 0x7fa7c0101c50 2024-03-12T05:55:39.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.799+0000 7fa7b67fc640 1 -- 172.21.15.76:0/3270388088 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80116b0 con 0x7fa7c0101c50 2024-03-12T05:55:39.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.799+0000 7fa7c7695640 1 -- 172.21.15.76:0/3270388088 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c019f2e0 con 0x7fa7c0101c50 2024-03-12T05:55:39.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.799+0000 7f6b427fc640 1 -- 172.21.15.76:0/2113346888 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6b440b8920 con 0x7f6b4c101c90 2024-03-12T05:55:39.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.799+0000 7fa7c7695640 1 -- 172.21.15.76:0/3270388088 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c019f770 con 0x7fa7c0101c50 2024-03-12T05:55:39.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.801+0000 7fa7b67fc640 1 -- 172.21.15.76:0/3270388088 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7fa7b8005520 con 0x7fa7c0101c50 2024-03-12T05:55:39.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.802+0000 7fa7b67fc640 1 --2- 172.21.15.76:0/3270388088 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa7ac0a1a30 0x7fa7ac0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.802+0000 7fa7b67fc640 1 -- 172.21.15.76:0/3270388088 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa7b80cd550 con 0x7fa7c0101c50 2024-03-12T05:55:39.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.802+0000 7fa7c5431640 1 --2- 172.21.15.76:0/3270388088 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa7ac0a1a30 0x7fa7ac0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.802+0000 7fa7ca26a740 1 Processor -- start 2024-03-12T05:55:39.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.802+0000 7fa7ca26a740 1 -- start start 2024-03-12T05:55:39.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.802+0000 7fa7c5431640 1 --2- 172.21.15.76:0/3270388088 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa7ac0a1a30 0x7fa7ac0a3ef0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa7c01010f0 tx=0x7fa7bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.802+0000 7fa7ca26a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a064c0 0x555e85a068e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.802+0000 7fa7ca26a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a07690 0x555e85a07b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.802+0000 7fa7ca26a740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a0e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.802+0000 7fa7ca26a740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e85975f90 con 0x555e85a08050 2024-03-12T05:55:39.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.802+0000 7fa7ca26a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e859734c0 con 0x555e85a064c0 2024-03-12T05:55:39.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.802+0000 7fa7ca26a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e8596eca0 con 0x555e85a07690 2024-03-12T05:55:39.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.803+0000 7fa7c5c32640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a0e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.803+0000 7fa7c5c32640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a0e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47578/0 (socket says 172.21.15.76:47578) 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.803+0000 7fa7c4c30640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a07690 0x555e85a07b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.803+0000 7fa7c5c32640 1 -- 172.21.15.76:0/3965876574 learned_addr learned my addr 172.21.15.76:0/3965876574 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.803+0000 7fa7c5431640 1 --2- 172.21.15.76:0/3965876574 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a064c0 0x555e85a068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.803+0000 7fa7c5c32640 1 -- 172.21.15.76:0/3965876574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a07690 msgr2=0x555e85a07b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.803+0000 7fa7c5c32640 1 --2- 172.21.15.76:0/3965876574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a07690 0x555e85a07b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.803+0000 7fa7c5c32640 1 -- 172.21.15.76:0/3965876574 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a064c0 msgr2=0x555e85a068e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.803+0000 7fa7c5c32640 1 --2- 172.21.15.76:0/3965876574 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a064c0 0x555e85a068e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.803+0000 7fa7c5c32640 1 -- 172.21.15.76:0/3965876574 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e859310b0 con 0x555e85a08050 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.803+0000 7fa7c5c32640 1 --2- 172.21.15.76:0/3965876574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a0e8e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa7b809a510 tx=0x7fa7b809a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6717d3317df712cd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.803+0000 7fa7ab7fe640 1 -- 172.21.15.76:0/3965876574 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80ba0d0 con 0x555e85a08050 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.803+0000 7fa7ab7fe640 1 -- 172.21.15.76:0/3965876574 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7b8098880 con 0x555e85a08050 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.803+0000 7fa7ab7fe640 1 -- 172.21.15.76:0/3965876574 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80cd110 con 0x555e85a08050 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.804+0000 7fa7ca26a740 1 -- 172.21.15.76:0/3965876574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 msgr2=0x555e85a0e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.804+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/3965876574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a0e8e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa7b809a510 tx=0x7fa7b809a640 comp rx=0 tx=0).stop 2024-03-12T05:55:39.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.804+0000 7fa7ca26a740 1 -- 172.21.15.76:0/3965876574 shutdown_connections 2024-03-12T05:55:39.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.804+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/3965876574 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a0e8e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.804+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/3965876574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a07690 0x555e85a07b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.804+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/3965876574 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a064c0 0x555e85a068e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.804+0000 7fa7ca26a740 1 -- 172.21.15.76:0/3965876574 >> 172.21.15.76:0/3965876574 conn(0x555e85a016a0 msgr2=0x555e85a01aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.804+0000 7fa7ca26a740 1 -- 172.21.15.76:0/3965876574 shutdown_connections 2024-03-12T05:55:39.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.804+0000 7fa7ca26a740 1 -- 172.21.15.76:0/3965876574 wait complete. 2024-03-12T05:55:39.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.804+0000 7fa7ca26a740 1 -- start start 2024-03-12T05:55:39.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.804+0000 7fa7ca26a740 1 Processor -- start 2024-03-12T05:55:39.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7ca26a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a064c0 0x555e85a10730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7ca26a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a07690 0x555e85a10c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7ca26a740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a17cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7ca26a740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e85913f40 con 0x555e85a08050 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7ca26a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e856424b0 con 0x555e85a07690 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7ca26a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e858f4670 con 0x555e85a064c0 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7ca26a740 10 client.0 _ll_register_callbacks cb 0x555e85961f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7c4c30640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a07690 0x555e85a10c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7c4c30640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a07690 0x555e85a10c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:51924/0 (socket says 172.21.15.76:51924) 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7c4c30640 1 -- 172.21.15.76:0/367962719 learned_addr learned my addr 172.21.15.76:0/367962719 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7c5c32640 1 --2- 172.21.15.76:0/367962719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a17cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7c5431640 1 --2- 172.21.15.76:0/367962719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a064c0 0x555e85a10730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7c4c30640 1 -- 172.21.15.76:0/367962719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a064c0 msgr2=0x555e85a10730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7c4c30640 1 --2- 172.21.15.76:0/367962719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a064c0 0x555e85a10730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7c4c30640 1 -- 172.21.15.76:0/367962719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 msgr2=0x555e85a17cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7c4c30640 1 --2- 172.21.15.76:0/367962719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a17cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7c4c30640 1 -- 172.21.15.76:0/367962719 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c019f770 con 0x555e85a07690 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.805+0000 7fa7c5c32640 1 --2- 172.21.15.76:0/367962719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a17cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:55:39.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7c4c30640 1 --2- 172.21.15.76:0/367962719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a07690 0x555e85a10c70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009900 tx=0x7fa7b0004b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a61e5bce0dc3ab22 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7a8ff9640 1 -- 172.21.15.76:0/367962719 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b0004390 con 0x555e85a07690 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7ca26a740 1 -- 172.21.15.76:0/367962719 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e85948f40 con 0x555e85a07690 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7a8ff9640 1 -- 172.21.15.76:0/367962719 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa7b0004df0 con 0x555e85a07690 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7a8ff9640 1 -- 172.21.15.76:0/367962719 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7b00044f0 con 0x555e85a07690 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7ca26a740 10 client.24980 fetch_fsmap learned FSMap version 22 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7ca26a740 1 -- 172.21.15.76:0/367962719 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x555e8596d320 con 0x555e85a07690 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7a8ff9640 1 -- 172.21.15.76:0/367962719 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555e85948f40 con 0x555e85a07690 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7a8ff9640 1 -- 172.21.15.76:0/367962719 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa7b0053070 con 0x555e85a07690 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7ca26a740 10 client.24980 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7ca26a740 10 client.24980 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7ca26a740 20 client.24980 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/367962719 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x555e85a1c9b0 0x555e85a1edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7ca26a740 4 client.24980 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:55:39.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.806+0000 7fa7ca26a740 1 -- 172.21.15.76:0/367962719 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e85965310 con 0x555e85a1c9b0 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.807+0000 7fa7c5431640 1 --2- 172.21.15.76:0/367962719 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x555e85a1c9b0 0x555e85a1edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.807+0000 7fa7a8ff9640 10 client.24980 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.807+0000 7fa7c5431640 1 --2- 172.21.15.76:0/367962719 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x555e85a1c9b0 0x555e85a1edd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82b86788264ad6f1 server_cookie=63f6c2b8aafa619d in_seq=0 out_seq=0 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7a8ff9640 1 -- 172.21.15.76:0/367962719 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555e85965310 con 0x555e85a1c9b0 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7a8ff9640 10 client.24980 handle_command_reply: tid=0 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7ca26a740 1 client.24980 shutdown 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7ca26a740 1 -- 172.21.15.76:0/367962719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a07690 msgr2=0x555e85a10c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/367962719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a07690 0x555e85a10c70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009900 tx=0x7fa7b0004b70 comp rx=0 tx=0).stop 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7ca26a740 1 -- 172.21.15.76:0/367962719 shutdown_connections 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/367962719 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x555e85a1c9b0 0x555e85a1edd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/367962719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a17cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/367962719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a07690 0x555e85a10c70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/367962719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a064c0 0x555e85a10730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7a8ff9640 0 client.24980 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:39.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7ca26a740 1 -- 172.21.15.76:0/367962719 >> 172.21.15.76:0/367962719 conn(0x555e85a016a0 msgr2=0x555e85a008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7ca26a740 1 -- 172.21.15.76:0/367962719 shutdown_connections 2024-03-12T05:55:39.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7ca26a740 1 -- 172.21.15.76:0/367962719 wait complete. 2024-03-12T05:55:39.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.808+0000 7fa7ca26a740 20 client.24980 trim_cache size 0 max 16384 2024-03-12T05:55:39.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.823+0000 7fa7ca26a740 1 Processor -- start 2024-03-12T05:55:39.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.823+0000 7fa7ca26a740 1 -- start start 2024-03-12T05:55:39.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.823+0000 7fa7ca26a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a064c0 0x555e85a0b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.823+0000 7fa7ca26a740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a07690 0x555e85a0b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.823+0000 7fa7ca26a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a08050 0x555e85a17c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.823+0000 7fa7ca26a740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e85a0f6e0 con 0x555e85a07690 2024-03-12T05:55:39.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.823+0000 7fa7ca26a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e85a01ee0 con 0x555e85a08050 2024-03-12T05:55:39.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.823+0000 7fa7ca26a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e859f6350 con 0x555e85a064c0 2024-03-12T05:55:39.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.823+0000 7fa7c4c30640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a07690 0x555e85a0b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.823+0000 7fa7c4c30640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a07690 0x555e85a0b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47596/0 (socket says 172.21.15.76:47596) 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.823+0000 7fa7c4c30640 1 -- 172.21.15.76:0/670280216 learned_addr learned my addr 172.21.15.76:0/670280216 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.824+0000 7fa7c5c32640 1 --2- 172.21.15.76:0/670280216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a08050 0x555e85a17c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.824+0000 7fa7c5431640 1 --2- 172.21.15.76:0/670280216 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a064c0 0x555e85a0b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.824+0000 7fa7c4c30640 1 -- 172.21.15.76:0/670280216 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a064c0 msgr2=0x555e85a0b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.824+0000 7fa7c4c30640 1 --2- 172.21.15.76:0/670280216 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a064c0 0x555e85a0b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.824+0000 7fa7c4c30640 1 -- 172.21.15.76:0/670280216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a08050 msgr2=0x555e85a17c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.824+0000 7fa7c4c30640 1 --2- 172.21.15.76:0/670280216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a08050 0x555e85a17c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.824+0000 7fa7c4c30640 1 -- 172.21.15.76:0/670280216 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e85909c00 con 0x555e85a07690 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.824+0000 7fa7c5431640 1 --2- 172.21.15.76:0/670280216 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a064c0 0x555e85a0b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.824+0000 7fa7c5c32640 1 --2- 172.21.15.76:0/670280216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a08050 0x555e85a17c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.824+0000 7fa7c4c30640 1 --2- 172.21.15.76:0/670280216 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a07690 0x555e85a0b770 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009850 tx=0x7fa7b002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5dfbfb8b205e33 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.824+0000 7fa7aaffd640 1 -- 172.21.15.76:0/670280216 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b0034440 con 0x555e85a07690 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.824+0000 7fa7aaffd640 1 -- 172.21.15.76:0/670280216 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7b0050070 con 0x555e85a07690 2024-03-12T05:55:39.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.824+0000 7fa7aaffd640 1 -- 172.21.15.76:0/670280216 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b0034c50 con 0x555e85a07690 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 -- 172.21.15.76:0/670280216 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a07690 msgr2=0x555e85a0b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/670280216 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a07690 0x555e85a0b770 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009850 tx=0x7fa7b002bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 -- 172.21.15.76:0/670280216 shutdown_connections 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/670280216 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a08050 0x555e85a17c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/670280216 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a07690 0x555e85a0b770 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/670280216 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a064c0 0x555e85a0b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 -- 172.21.15.76:0/670280216 >> 172.21.15.76:0/670280216 conn(0x555e85a016a0 msgr2=0x555e85a10730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 -- 172.21.15.76:0/670280216 shutdown_connections 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 -- 172.21.15.76:0/670280216 wait complete. 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 -- start start 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 Processor -- start 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a064c0 0x555e85a16560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a07690 0x555e85a16aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a16fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e85a72ea0 con 0x555e85a08050 2024-03-12T05:55:39.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e85a73010 con 0x555e85a064c0 2024-03-12T05:55:39.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e85a731d0 con 0x555e85a07690 2024-03-12T05:55:39.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.825+0000 7fa7ca26a740 10 client.0 _ll_register_callbacks cb 0x7fa7b8096820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:39.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7c5c32640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a16fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7c5c32640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a16fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47608/0 (socket says 172.21.15.76:47608) 2024-03-12T05:55:39.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7c5c32640 1 -- 172.21.15.76:0/4249534074 learned_addr learned my addr 172.21.15.76:0/4249534074 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:39.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7c4c30640 1 --2- 172.21.15.76:0/4249534074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a07690 0x555e85a16aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7c5431640 1 --2- 172.21.15.76:0/4249534074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a064c0 0x555e85a16560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7c5c32640 1 -- 172.21.15.76:0/4249534074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a07690 msgr2=0x555e85a16aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7c5c32640 1 --2- 172.21.15.76:0/4249534074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a07690 0x555e85a16aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7c5c32640 1 -- 172.21.15.76:0/4249534074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a064c0 msgr2=0x555e85a16560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7c5c32640 1 --2- 172.21.15.76:0/4249534074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a064c0 0x555e85a16560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7c5c32640 1 -- 172.21.15.76:0/4249534074 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e8596d320 con 0x555e85a08050 2024-03-12T05:55:39.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7c5c32640 1 --2- 172.21.15.76:0/4249534074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a16fe0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8096f50 tx=0x7fa7b800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1366b1cea8ac66be server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:39.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7a8ff9640 1 -- 172.21.15.76:0/4249534074 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b80ba0d0 con 0x555e85a08050 2024-03-12T05:55:39.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7a8ff9640 1 -- 172.21.15.76:0/4249534074 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa7b80cd110 con 0x555e85a08050 2024-03-12T05:55:39.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7ca26a740 1 -- 172.21.15.76:0/4249534074 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e85994900 con 0x555e85a08050 2024-03-12T05:55:39.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7a8ff9640 1 -- 172.21.15.76:0/4249534074 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7b80cd2d0 con 0x555e85a08050 2024-03-12T05:55:39.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7ca26a740 10 client.15432 fetch_fsmap learned FSMap version 22 2024-03-12T05:55:39.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.826+0000 7fa7ca26a740 1 -- 172.21.15.76:0/4249534074 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x555e8592f910 con 0x555e85a08050 2024-03-12T05:55:39.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.827+0000 7fa7a8ff9640 1 -- 172.21.15.76:0/4249534074 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555e85994900 con 0x555e85a08050 2024-03-12T05:55:39.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.827+0000 7fa7a8ff9640 1 -- 172.21.15.76:0/4249534074 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa7b8018030 con 0x555e85a08050 2024-03-12T05:55:39.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.827+0000 7fa7ca26a740 10 client.15432 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:55:39.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.827+0000 7fa7ca26a740 10 client.15432 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:55:39.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.827+0000 7fa7ca26a740 20 client.15432 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:39.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.827+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/4249534074 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x555e85a747e0 0x555e85a76bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.827+0000 7fa7ca26a740 4 client.15432 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:55:39.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.827+0000 7fa7ca26a740 1 -- 172.21.15.76:0/4249534074 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x555e8566e860 con 0x555e85a747e0 2024-03-12T05:55:39.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.827+0000 7fa7c5431640 1 --2- 172.21.15.76:0/4249534074 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x555e85a747e0 0x555e85a76bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.827+0000 7fa7a8ff9640 10 client.15432 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:39.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.827+0000 7fa7c5431640 1 --2- 172.21.15.76:0/4249534074 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x555e85a747e0 0x555e85a76bd0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb5fe10b19e371c0 server_cookie=263638564a80f55a in_seq=0 out_seq=0 2024-03-12T05:55:39.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.828+0000 7fa7a8ff9640 1 -- 172.21.15.76:0/4249534074 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555e8566e860 con 0x555e85a747e0 2024-03-12T05:55:39.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.828+0000 7fa7a8ff9640 10 client.15432 handle_command_reply: tid=0 2024-03-12T05:55:39.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.828+0000 7fa7ca26a740 1 client.15432 shutdown 2024-03-12T05:55:39.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.828+0000 7fa7ca26a740 1 -- 172.21.15.76:0/4249534074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 msgr2=0x555e85a16fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.828+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/4249534074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a16fe0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8096f50 tx=0x7fa7b800c800 comp rx=0 tx=0).stop 2024-03-12T05:55:39.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7ca26a740 1 -- 172.21.15.76:0/4249534074 shutdown_connections 2024-03-12T05:55:39.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/4249534074 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x555e85a747e0 0x555e85a76bd0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/4249534074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555e85a08050 0x555e85a16fe0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/4249534074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e85a07690 0x555e85a16aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7a8ff9640 0 client.15432 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:39.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7ca26a740 1 --2- 172.21.15.76:0/4249534074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555e85a064c0 0x555e85a16560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7ca26a740 1 -- 172.21.15.76:0/4249534074 >> 172.21.15.76:0/4249534074 conn(0x555e85a016a0 msgr2=0x555e85a106d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7ca26a740 1 -- 172.21.15.76:0/4249534074 shutdown_connections 2024-03-12T05:55:39.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7ca26a740 1 -- 172.21.15.76:0/4249534074 wait complete. 2024-03-12T05:55:39.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7ca26a740 20 client.15432 trim_cache size 0 max 16384 2024-03-12T05:55:39.840 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:55:39.840 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T05:55:39.840 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "edc08182-5ec8-475f-b666-36458eb2853d", 2024-03-12T05:55:39.840 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T05:55:39.840 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:55:39.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7c7695640 1 -- 172.21.15.76:0/3270388088 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa7ac0a1a30 msgr2=0x7fa7ac0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7c7695640 1 --2- 172.21.15.76:0/3270388088 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa7ac0a1a30 0x7fa7ac0a3ef0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa7c01010f0 tx=0x7fa7bc005f90 comp rx=0 tx=0).stop 2024-03-12T05:55:39.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7c7695640 1 -- 172.21.15.76:0/3270388088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7c0101c50 msgr2=0x7fa7c019e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7c7695640 1 --2- 172.21.15.76:0/3270388088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7c0101c50 0x7fa7c019e790 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa7b800eaa0 tx=0x7fa7b800ef60 comp rx=0 tx=0).stop 2024-03-12T05:55:39.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7c7695640 1 -- 172.21.15.76:0/3270388088 shutdown_connections 2024-03-12T05:55:39.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7c7695640 1 --2- 172.21.15.76:0/3270388088 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fa7ac0a1a30 0x7fa7ac0a3ef0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.829+0000 7fa7c7695640 1 --2- 172.21.15.76:0/3270388088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7c0101c50 0x7fa7c019e790 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.830+0000 7fa7c7695640 1 --2- 172.21.15.76:0/3270388088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7c0101290 0x7fa7c0197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.830+0000 7fa7c7695640 1 --2- 172.21.15.76:0/3270388088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa7c01000a0 0x7fa7c01971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.830+0000 7fa7c7695640 1 -- 172.21.15.76:0/3270388088 >> 172.21.15.76:0/3270388088 conn(0x7fa7c00fb610 msgr2=0x7fa7c0068450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.830+0000 7fa7c7695640 1 -- 172.21.15.76:0/3270388088 shutdown_connections 2024-03-12T05:55:39.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.830+0000 7fa7c7695640 1 -- 172.21.15.76:0/3270388088 wait complete. 2024-03-12T05:55:39.844 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:55:39.844 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:55:39.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.947+0000 7f57ee681640 1 -- 172.21.15.76:0/3537855876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57e8100080 msgr2=0x7f57e81004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.947+0000 7f57ee681640 1 --2- 172.21.15.76:0/3537855876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57e8100080 0x7f57e81004a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f57d8009900 tx=0x7f57d802bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:39.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.947+0000 7f57ee681640 1 -- 172.21.15.76:0/3537855876 shutdown_connections 2024-03-12T05:55:39.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.947+0000 7f57ee681640 1 --2- 172.21.15.76:0/3537855876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57e8101c30 0x7f57e81084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.947+0000 7f57ee681640 1 --2- 172.21.15.76:0/3537855876 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57e8101270 0x7f57e81016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.947+0000 7f57ee681640 1 --2- 172.21.15.76:0/3537855876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57e8100080 0x7f57e81004a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.947+0000 7f57ee681640 1 -- 172.21.15.76:0/3537855876 >> 172.21.15.76:0/3537855876 conn(0x7f57e80fb5b0 msgr2=0x7f57e80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.947+0000 7f57ee681640 1 -- 172.21.15.76:0/3537855876 shutdown_connections 2024-03-12T05:55:39.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.947+0000 7f57ee681640 1 -- 172.21.15.76:0/3537855876 wait complete. 2024-03-12T05:55:39.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.947+0000 7f57ee681640 1 Processor -- start 2024-03-12T05:55:39.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.947+0000 7f57ee681640 1 -- start start 2024-03-12T05:55:39.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ee681640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57e8100080 0x7f57e8068fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ee681640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57e8101270 0x7f57e819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ee681640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57e8101c30 0x7f57e819e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ee681640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e819ed40 con 0x7f57e8101c30 2024-03-12T05:55:39.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ee681640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e819eeb0 con 0x7f57e8101270 2024-03-12T05:55:39.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ee681640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e819f070 con 0x7f57e8100080 2024-03-12T05:55:39.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ecc1e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57e8101c30 0x7f57e819e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ecc1e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57e8101c30 0x7f57e819e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47636/0 (socket says 172.21.15.76:47636) 2024-03-12T05:55:39.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ecc1e640 1 -- 172.21.15.76:0/42367593 learned_addr learned my addr 172.21.15.76:0/42367593 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:39.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57e7fff640 1 --2- 172.21.15.76:0/42367593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57e8100080 0x7f57e8068fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57e77fe640 1 --2- 172.21.15.76:0/42367593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57e8101270 0x7f57e819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ecc1e640 1 -- 172.21.15.76:0/42367593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57e8100080 msgr2=0x7f57e8068fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ecc1e640 1 --2- 172.21.15.76:0/42367593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57e8100080 0x7f57e8068fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ecc1e640 1 -- 172.21.15.76:0/42367593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57e8101270 msgr2=0x7f57e819bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ecc1e640 1 --2- 172.21.15.76:0/42367593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57e8101270 0x7f57e819bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ecc1e640 1 -- 172.21.15.76:0/42367593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d8009560 con 0x7f57e8101c30 2024-03-12T05:55:39.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ecc1e640 1 --2- 172.21.15.76:0/42367593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57e8101c30 0x7f57e819e560 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f57e000eb10 tx=0x7f57e000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57e57fa640 1 -- 172.21.15.76:0/42367593 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e0004b20 con 0x7f57e8101c30 2024-03-12T05:55:39.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57e57fa640 1 -- 172.21.15.76:0/42367593 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57e0004c80 con 0x7f57e8101c30 2024-03-12T05:55:39.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ee681640 1 -- 172.21.15.76:0/42367593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e819f2f0 con 0x7f57e8101c30 2024-03-12T05:55:39.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57e57fa640 1 -- 172.21.15.76:0/42367593 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e0010650 con 0x7f57e8101c30 2024-03-12T05:55:39.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.948+0000 7f57ee681640 1 -- 172.21.15.76:0/42367593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e819f800 con 0x7f57e8101c30 2024-03-12T05:55:39.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57e57fa640 1 -- 172.21.15.76:0/42367593 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 18) v1 ==== 136279+0+0 (secure 0 0 0) 0x7f57e0005020 con 0x7f57e8101c30 2024-03-12T05:55:39.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57f124d740 1 Processor -- start 2024-03-12T05:55:39.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57f124d740 1 -- start start 2024-03-12T05:55:39.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57f124d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc762d0 0x5558dcc766f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57f124d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc7b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57f124d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc7be30 0x5558dcc7e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57f124d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558dcbe6280 con 0x5558dcc762d0 2024-03-12T05:55:39.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57f124d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558dcbe37b0 con 0x5558dcc76cc0 2024-03-12T05:55:39.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57f124d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558dcbdef90 con 0x5558dcc7be30 2024-03-12T05:55:39.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57e57fa640 1 --2- 172.21.15.76:0/42367593 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f57d00a1a30 0x7f57d00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc762d0 0x5558dcc766f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc762d0 0x5558dcc766f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47648/0 (socket says 172.21.15.76:47648) 2024-03-12T05:55:39.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57e77fe640 1 -- 172.21.15.76:0/97832431 learned_addr learned my addr 172.21.15.76:0/97832431 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:39.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57ecc1e640 1 --2- 172.21.15.76:0/42367593 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f57d00a1a30 0x7f57d00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57e7fff640 1 --2- 172.21.15.76:0/97832431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc7b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.951+0000 7f57e77fe640 1 --2- 172.21.15.76:0/97832431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc7be30 0x5558dcc7e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.952+0000 7f57e57fa640 1 -- 172.21.15.76:0/42367593 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f57e0014070 con 0x7f57e8101c30 2024-03-12T05:55:39.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.952+0000 7f57e77fe640 1 -- 172.21.15.76:0/97832431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc7be30 msgr2=0x5558dcc7e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.952+0000 7f57e77fe640 1 --2- 172.21.15.76:0/97832431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc7be30 0x5558dcc7e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.952+0000 7f57e77fe640 1 -- 172.21.15.76:0/97832431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 msgr2=0x5558dcc7b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.952+0000 7f57e77fe640 1 --2- 172.21.15.76:0/97832431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc7b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.952+0000 7f57e77fe640 1 -- 172.21.15.76:0/97832431 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558dcb9d570 con 0x5558dcc762d0 2024-03-12T05:55:39.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.952+0000 7f57e77fe640 1 --2- 172.21.15.76:0/97832431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc762d0 0x5558dcc766f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f57dc005cf0 tx=0x7f57dc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d1f0aa04da511e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.952+0000 7f57ecc1e640 1 --2- 172.21.15.76:0/42367593 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f57d00a1a30 0x7f57d00a3ef0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f57e009a5c0 tx=0x7f57e006dbc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.952+0000 7f57ce7fc640 1 -- 172.21.15.76:0/97832431 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc004030 con 0x5558dcc762d0 2024-03-12T05:55:39.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.952+0000 7f57ce7fc640 1 -- 172.21.15.76:0/97832431 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57dc005500 con 0x5558dcc762d0 2024-03-12T05:55:39.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.952+0000 7f57ce7fc640 1 -- 172.21.15.76:0/97832431 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc004af0 con 0x5558dcc762d0 2024-03-12T05:55:39.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 -- 172.21.15.76:0/97832431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc762d0 msgr2=0x5558dcc766f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 --2- 172.21.15.76:0/97832431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc762d0 0x5558dcc766f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f57dc005cf0 tx=0x7f57dc00b840 comp rx=0 tx=0).stop 2024-03-12T05:55:39.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 -- 172.21.15.76:0/97832431 shutdown_connections 2024-03-12T05:55:39.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 --2- 172.21.15.76:0/97832431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc7be30 0x5558dcc7e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 --2- 172.21.15.76:0/97832431 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc7b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 --2- 172.21.15.76:0/97832431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc762d0 0x5558dcc766f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 -- 172.21.15.76:0/97832431 >> 172.21.15.76:0/97832431 conn(0x5558dcc717f0 msgr2=0x5558dcc71bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 -- 172.21.15.76:0/97832431 shutdown_connections 2024-03-12T05:55:39.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 -- 172.21.15.76:0/97832431 wait complete. 2024-03-12T05:55:39.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 -- start start 2024-03-12T05:55:39.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 Processor -- start 2024-03-12T05:55:39.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc762d0 0x5558dcc80280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc807c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc7be30 0x5558dcc87800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558dc8b2430 con 0x5558dcc7be30 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558dc86ee00 con 0x5558dcc76cc0 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.953+0000 7f57f124d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558dcb63fd0 con 0x5558dcc762d0 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57f124d740 10 client.0 _ll_register_callbacks cb 0x5558dcbb9fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc7be30 0x5558dcc87800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc7be30 0x5558dcc87800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47660/0 (socket says 172.21.15.76:47660) 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57e77fe640 1 -- 172.21.15.76:0/1699245908 learned_addr learned my addr 172.21.15.76:0/1699245908 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57e7fff640 1 --2- 172.21.15.76:0/1699245908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc807c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57e77fe640 1 -- 172.21.15.76:0/1699245908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc762d0 msgr2=0x5558dcc80280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57e77fe640 1 --2- 172.21.15.76:0/1699245908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc762d0 0x5558dcc80280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57e77fe640 1 -- 172.21.15.76:0/1699245908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 msgr2=0x5558dcc807c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57e77fe640 1 --2- 172.21.15.76:0/1699245908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc807c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57e77fe640 1 -- 172.21.15.76:0/1699245908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57dc00ba90 con 0x5558dcc7be30 2024-03-12T05:55:39.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57e77fe640 1 --2- 172.21.15.76:0/1699245908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc7be30 0x5558dcc87800 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0057c0 tx=0x7f57dc012210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=407beb2f0d2c5154 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:39.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57c77fe640 1 -- 172.21.15.76:0/1699245908 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc004a90 con 0x5558dcc7be30 2024-03-12T05:55:39.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57c77fe640 1 -- 172.21.15.76:0/1699245908 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f57dc004bf0 con 0x5558dcc7be30 2024-03-12T05:55:39.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57f124d740 1 -- 172.21.15.76:0/1699245908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5558dc8de7e0 con 0x5558dcc7be30 2024-03-12T05:55:39.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57c77fe640 1 -- 172.21.15.76:0/1699245908 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57dc004db0 con 0x5558dcc7be30 2024-03-12T05:55:39.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57f124d740 10 client.15444 fetch_fsmap learned FSMap version 22 2024-03-12T05:55:39.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.954+0000 7f57f124d740 1 -- 172.21.15.76:0/1699245908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5558dcbdd610 con 0x5558dcc7be30 2024-03-12T05:55:39.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.955+0000 7f57c77fe640 1 -- 172.21.15.76:0/1699245908 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5558dc8de7e0 con 0x5558dcc7be30 2024-03-12T05:55:39.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.955+0000 7f57c77fe640 1 -- 172.21.15.76:0/1699245908 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f57dc01b4f0 con 0x5558dcc7be30 2024-03-12T05:55:39.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.955+0000 7f57f124d740 10 client.15444 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:55:39.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.955+0000 7f57f124d740 10 client.15444 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:55:39.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.955+0000 7f57f124d740 20 client.15444 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:39.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.955+0000 7f57f124d740 1 --2- 172.21.15.76:0/1699245908 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5558dcc8c2f0 0x5558dcc8e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.955+0000 7f57f124d740 4 client.15444 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.955+0000 7f57f124d740 1 -- 172.21.15.76:0/1699245908 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558dc91b720 con 0x5558dcc8c2f0 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.955+0000 7f57e7fff640 1 --2- 172.21.15.76:0/1699245908 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5558dcc8c2f0 0x5558dcc8e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57e7fff640 1 --2- 172.21.15.76:0/1699245908 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5558dcc8c2f0 0x5558dcc8e6e0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d8b306dc405b059 server_cookie=54e0b51c05a30bc1 in_seq=0 out_seq=0 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57c77fe640 10 client.15444 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57c77fe640 1 -- 172.21.15.76:0/1699245908 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5558dc91b720 con 0x5558dcc8c2f0 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57c77fe640 10 client.15444 handle_command_reply: tid=0 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57f124d740 1 client.15444 shutdown 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57f124d740 1 -- 172.21.15.76:0/1699245908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc7be30 msgr2=0x5558dcc87800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57f124d740 1 --2- 172.21.15.76:0/1699245908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc7be30 0x5558dcc87800 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0057c0 tx=0x7f57dc012210 comp rx=0 tx=0).stop 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57f124d740 1 -- 172.21.15.76:0/1699245908 shutdown_connections 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57f124d740 1 --2- 172.21.15.76:0/1699245908 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5558dcc8c2f0 0x5558dcc8e6e0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57f124d740 1 --2- 172.21.15.76:0/1699245908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc7be30 0x5558dcc87800 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57f124d740 1 --2- 172.21.15.76:0/1699245908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc807c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57c77fe640 0 client.15444 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:39.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57f124d740 1 --2- 172.21.15.76:0/1699245908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc762d0 0x5558dcc80280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57f124d740 1 -- 172.21.15.76:0/1699245908 >> 172.21.15.76:0/1699245908 conn(0x5558dcc717f0 msgr2=0x5558dcc70a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57f124d740 1 -- 172.21.15.76:0/1699245908 shutdown_connections 2024-03-12T05:55:39.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.956+0000 7f57f124d740 1 -- 172.21.15.76:0/1699245908 wait complete. 2024-03-12T05:55:39.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.957+0000 7f57f124d740 20 client.15444 trim_cache size 0 max 16384 2024-03-12T05:55:39.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.971+0000 7f57f124d740 1 Processor -- start 2024-03-12T05:55:39.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.971+0000 7f57f124d740 1 -- start start 2024-03-12T05:55:39.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.971+0000 7f57f124d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc762d0 0x5558dcc71bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.971+0000 7f57f124d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc87790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.971+0000 7f57f124d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc7be30 0x5558dcc80280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.971+0000 7f57f124d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558dcc6d320 con 0x5558dcc762d0 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.971+0000 7f57f124d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558dcc7e980 con 0x5558dcc76cc0 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.971+0000 7f57f124d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558dcc6be60 con 0x5558dcc7be30 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.971+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc762d0 0x5558dcc71bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.971+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc762d0 0x5558dcc71bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47664/0 (socket says 172.21.15.76:47664) 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.971+0000 7f57e77fe640 1 -- 172.21.15.76:0/2725800687 learned_addr learned my addr 172.21.15.76:0/2725800687 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.972+0000 7f57e7fff640 1 --2- 172.21.15.76:0/2725800687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc87790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.972+0000 7f57e77fe640 1 -- 172.21.15.76:0/2725800687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc7be30 msgr2=0x5558dcc80280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.972+0000 7f57e77fe640 1 --2- 172.21.15.76:0/2725800687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc7be30 0x5558dcc80280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.972+0000 7f57e77fe640 1 -- 172.21.15.76:0/2725800687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 msgr2=0x5558dcc87790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.972+0000 7f57e77fe640 1 --2- 172.21.15.76:0/2725800687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc87790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.972+0000 7f57e77fe640 1 -- 172.21.15.76:0/2725800687 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558dcb79ac0 con 0x5558dcc762d0 2024-03-12T05:55:39.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.972+0000 7f57e7fff640 1 --2- 172.21.15.76:0/2725800687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc87790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:55:39.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.972+0000 7f57e77fe640 1 --2- 172.21.15.76:0/2725800687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc762d0 0x5558dcc71bd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f57dc004800 tx=0x7f57dc005c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad3b859988bc260 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.972+0000 7f57cdffb640 1 -- 172.21.15.76:0/2725800687 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc01bea0 con 0x5558dcc762d0 2024-03-12T05:55:39.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.972+0000 7f57cdffb640 1 -- 172.21.15.76:0/2725800687 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57dc021030 con 0x5558dcc762d0 2024-03-12T05:55:39.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.972+0000 7f57cdffb640 1 -- 172.21.15.76:0/2725800687 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc004a90 con 0x5558dcc762d0 2024-03-12T05:55:39.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 -- 172.21.15.76:0/2725800687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc762d0 msgr2=0x5558dcc71bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 --2- 172.21.15.76:0/2725800687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc762d0 0x5558dcc71bd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f57dc004800 tx=0x7f57dc005c10 comp rx=0 tx=0).stop 2024-03-12T05:55:39.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 -- 172.21.15.76:0/2725800687 shutdown_connections 2024-03-12T05:55:39.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 --2- 172.21.15.76:0/2725800687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc7be30 0x5558dcc80280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 --2- 172.21.15.76:0/2725800687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc87790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 --2- 172.21.15.76:0/2725800687 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc762d0 0x5558dcc71bd0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 -- 172.21.15.76:0/2725800687 >> 172.21.15.76:0/2725800687 conn(0x5558dcc717f0 msgr2=0x5558dcc7b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 -- 172.21.15.76:0/2725800687 shutdown_connections 2024-03-12T05:55:39.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 -- 172.21.15.76:0/2725800687 wait complete. 2024-03-12T05:55:39.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 -- start start 2024-03-12T05:55:39.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 Processor -- start 2024-03-12T05:55:39.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc762d0 0x5558dcc87280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc877c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc7be30 0x5558dcc85890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558dcc85f90 con 0x5558dcc7be30 2024-03-12T05:55:39.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558dcc86100 con 0x5558dcc76cc0 2024-03-12T05:55:39.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558dcc862c0 con 0x5558dcc762d0 2024-03-12T05:55:39.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.973+0000 7f57f124d740 10 client.0 _ll_register_callbacks cb 0x7f57dc00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:55:39.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc7be30 0x5558dcc85890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc7be30 0x5558dcc85890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47676/0 (socket says 172.21.15.76:47676) 2024-03-12T05:55:39.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57e77fe640 1 -- 172.21.15.76:0/157328849 learned_addr learned my addr 172.21.15.76:0/157328849 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:39.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57e77fe640 1 --2- 172.21.15.76:0/157328849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc762d0 0x5558dcc87280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57e7fff640 1 --2- 172.21.15.76:0/157328849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc877c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57e77fe640 1 -- 172.21.15.76:0/157328849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc762d0 msgr2=0x5558dcc87280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57e77fe640 1 --2- 172.21.15.76:0/157328849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc762d0 0x5558dcc87280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57e77fe640 1 -- 172.21.15.76:0/157328849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 msgr2=0x5558dcc877c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:55:39.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57e77fe640 1 --2- 172.21.15.76:0/157328849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc877c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57e77fe640 1 -- 172.21.15.76:0/157328849 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558dcbdd610 con 0x5558dcc7be30 2024-03-12T05:55:39.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57e7fff640 1 --2- 172.21.15.76:0/157328849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc877c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:55:39.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57e77fe640 1 --2- 172.21.15.76:0/157328849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc7be30 0x5558dcc85890 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f57dc01b900 tx=0x7f57dc005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7f42e8900a177b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:39.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:55:39.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57c77fe640 1 -- 172.21.15.76:0/157328849 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc01bea0 con 0x5558dcc7be30 2024-03-12T05:55:39.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57c77fe640 1 -- 172.21.15.76:0/157328849 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f57dc000e40 con 0x5558dcc7be30 2024-03-12T05:55:39.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57f124d740 1 -- 172.21.15.76:0/157328849 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57dc005500 con 0x5558dcc7be30 2024-03-12T05:55:39.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57c77fe640 1 -- 172.21.15.76:0/157328849 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57dc004e80 con 0x5558dcc7be30 2024-03-12T05:55:39.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57f124d740 10 client.15450 fetch_fsmap learned FSMap version 22 2024-03-12T05:55:39.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.974+0000 7f57f124d740 1 -- 172.21.15.76:0/157328849 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5558dcb9f6a0 con 0x5558dcc7be30 2024-03-12T05:55:39.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.975+0000 7f57c77fe640 1 -- 172.21.15.76:0/157328849 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57dc005500 con 0x5558dcc7be30 2024-03-12T05:55:39.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.975+0000 7f57c77fe640 1 -- 172.21.15.76:0/157328849 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f57dc003710 con 0x5558dcc7be30 2024-03-12T05:55:39.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.975+0000 7f57f124d740 10 client.15450 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:55:39.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.975+0000 7f57f124d740 10 client.15450 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:55:39.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.975+0000 7f57f124d740 20 client.15450 populate_metadata read hostname 'smithi076' 2024-03-12T05:55:39.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.975+0000 7f57f124d740 1 --2- 172.21.15.76:0/157328849 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5558dcce3890 0x5558dcce5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:39.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.975+0000 7f57f124d740 4 client.15450 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.975+0000 7f57f124d740 1 -- 172.21.15.76:0/157328849 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5558dc96a890 con 0x5558dcce3890 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.975+0000 7f57e7fff640 1 --2- 172.21.15.76:0/157328849 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5558dcce3890 0x5558dcce5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.975+0000 7f57c77fe640 10 client.15450 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.975+0000 7f57e7fff640 1 --2- 172.21.15.76:0/157328849 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5558dcce3890 0x5558dcce5cb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b645ee8ff94d6dc8 server_cookie=a4be0220da05f511 in_seq=0 out_seq=0 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.975+0000 7f57c77fe640 1 -- 172.21.15.76:0/157328849 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5558dc96a890 con 0x5558dcce3890 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.975+0000 7f57c77fe640 10 client.15450 handle_command_reply: tid=0 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.975+0000 7f57f124d740 1 client.15450 shutdown 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57f124d740 1 -- 172.21.15.76:0/157328849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc7be30 msgr2=0x5558dcc85890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57f124d740 1 --2- 172.21.15.76:0/157328849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc7be30 0x5558dcc85890 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f57dc01b900 tx=0x7f57dc005670 comp rx=0 tx=0).stop 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57f124d740 1 -- 172.21.15.76:0/157328849 shutdown_connections 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57f124d740 1 --2- 172.21.15.76:0/157328849 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5558dcce3890 0x5558dcce5cb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57f124d740 1 --2- 172.21.15.76:0/157328849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5558dcc7be30 0x5558dcc85890 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57f124d740 1 --2- 172.21.15.76:0/157328849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5558dcc76cc0 0x5558dcc877c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57c77fe640 0 client.15450 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:55:39.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57f124d740 1 --2- 172.21.15.76:0/157328849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558dcc762d0 0x5558dcc87280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57f124d740 1 -- 172.21.15.76:0/157328849 >> 172.21.15.76:0/157328849 conn(0x5558dcc717f0 msgr2=0x5558dcc71bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57f124d740 1 -- 172.21.15.76:0/157328849 shutdown_connections 2024-03-12T05:55:39.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57f124d740 1 -- 172.21.15.76:0/157328849 wait complete. 2024-03-12T05:55:39.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57f124d740 20 client.15450 trim_cache size 0 max 16384 2024-03-12T05:55:39.986 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:55:39.986 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:55:39.986 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T05:55:39.986 INFO:teuthology.orchestra.run.smithi076.stdout: "edc08182-5ec8-475f-b666-36458eb2853d": { 2024-03-12T05:55:39.986 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T05:55:39.987 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "edc08182-5ec8-475f-b666-36458eb2853d", 2024-03-12T05:55:39.987 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T05:55:39.987 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:55:39.987 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:55:39.987 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:55:39.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57ee681640 1 -- 172.21.15.76:0/42367593 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f57d00a1a30 msgr2=0x7f57d00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57ee681640 1 --2- 172.21.15.76:0/42367593 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f57d00a1a30 0x7f57d00a3ef0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f57e009a5c0 tx=0x7f57e006dbc0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57ee681640 1 -- 172.21.15.76:0/42367593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57e8101c30 msgr2=0x7f57e819e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:39.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57ee681640 1 --2- 172.21.15.76:0/42367593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57e8101c30 0x7f57e819e560 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f57e000eb10 tx=0x7f57e000c2d0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57ee681640 1 -- 172.21.15.76:0/42367593 shutdown_connections 2024-03-12T05:55:39.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57ee681640 1 --2- 172.21.15.76:0/42367593 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f57d00a1a30 0x7f57d00a3ef0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57ee681640 1 --2- 172.21.15.76:0/42367593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57e8101c30 0x7f57e819e560 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57ee681640 1 --2- 172.21.15.76:0/42367593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57e8101270 0x7f57e819bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57ee681640 1 --2- 172.21.15.76:0/42367593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57e8100080 0x7f57e8068fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:39.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.976+0000 7f57ee681640 1 -- 172.21.15.76:0/42367593 >> 172.21.15.76:0/42367593 conn(0x7f57e80fb5b0 msgr2=0x7f57e81045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:39.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.977+0000 7f57ee681640 1 -- 172.21.15.76:0/42367593 shutdown_connections 2024-03-12T05:55:39.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:39.977+0000 7f57ee681640 1 -- 172.21.15.76:0/42367593 wait complete. 2024-03-12T05:55:39.991 INFO:tasks.cephfs.filesystem:scrub status for tag:edc08182-5ec8-475f-b666-36458eb2853d - {'path': '/', 'tag': 'edc08182-5ec8-475f-b666-36458eb2853d', 'options': 'recursive,force'} 2024-03-12T05:55:40.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.034+0000 7f6b5393a640 1 -- 172.21.15.76:0/2113346888 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f6b4c02cdd0 con 0x7f6b4c101c90 2024-03-12T05:55:40.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.047+0000 7f6b427fc640 1 -- 172.21.15.76:0/2113346888 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f6b440bbfe0 con 0x7f6b4c101c90 2024-03-12T05:55:40.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.050+0000 7f6b5393a640 1 -- 172.21.15.76:0/2113346888 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b380a1a20 msgr2=0x7f6b380a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.050+0000 7f6b5393a640 1 --2- 172.21.15.76:0/2113346888 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b380a1a20 0x7f6b380a3ee0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c198780 tx=0x7f6b3c00b390 comp rx=0 tx=0).stop 2024-03-12T05:55:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.050+0000 7f6b5393a640 1 -- 172.21.15.76:0/2113346888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4c101c90 msgr2=0x7f6b4c197270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.050+0000 7f6b5393a640 1 --2- 172.21.15.76:0/2113346888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4c101c90 0x7f6b4c197270 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6b44006010 tx=0x7f6b440040c0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.050+0000 7f6b5393a640 1 -- 172.21.15.76:0/2113346888 shutdown_connections 2024-03-12T05:55:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.050+0000 7f6b5393a640 1 --2- 172.21.15.76:0/2113346888 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f6b380a1a20 0x7f6b380a3ee0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.050+0000 7f6b5393a640 1 --2- 172.21.15.76:0/2113346888 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6b4c106ef0 0x7f6b4c19e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.050+0000 7f6b5393a640 1 --2- 172.21.15.76:0/2113346888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b4c1045c0 0x7f6b4c1977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.050+0000 7f6b5393a640 1 --2- 172.21.15.76:0/2113346888 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4c101c90 0x7f6b4c197270 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.050+0000 7f6b5393a640 1 -- 172.21.15.76:0/2113346888 >> 172.21.15.76:0/2113346888 conn(0x7f6b4c0fb620 msgr2=0x7f6b4c102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:40.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.051+0000 7f6b5393a640 1 -- 172.21.15.76:0/2113346888 shutdown_connections 2024-03-12T05:55:40.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.052+0000 7f6b5393a640 1 -- 172.21.15.76:0/2113346888 wait complete. 2024-03-12T05:55:40.071 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-12T05:55:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.177+0000 7fba7ef48640 1 -- 172.21.15.76:0/2473171856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba78101c60 msgr2=0x7fba781084f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.177+0000 7fba7ef48640 1 --2- 172.21.15.76:0/2473171856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba78101c60 0x7fba781084f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fba68009900 tx=0x7fba6802bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.177+0000 7fba7ef48640 1 -- 172.21.15.76:0/2473171856 shutdown_connections 2024-03-12T05:55:40.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.177+0000 7fba7ef48640 1 --2- 172.21.15.76:0/2473171856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba78101c60 0x7fba781084f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.177+0000 7fba7ef48640 1 --2- 172.21.15.76:0/2473171856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba781012a0 0x7fba78101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.177+0000 7fba7ef48640 1 --2- 172.21.15.76:0/2473171856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba781000b0 0x7fba781004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.177+0000 7fba7ef48640 1 -- 172.21.15.76:0/2473171856 >> 172.21.15.76:0/2473171856 conn(0x7fba780fb620 msgr2=0x7fba780fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.177+0000 7fba7ef48640 1 -- 172.21.15.76:0/2473171856 shutdown_connections 2024-03-12T05:55:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.177+0000 7fba7ef48640 1 -- 172.21.15.76:0/2473171856 wait complete. 2024-03-12T05:55:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba7ef48640 1 Processor -- start 2024-03-12T05:55:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba7ef48640 1 -- start start 2024-03-12T05:55:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba7ef48640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba781000b0 0x7fba78196150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba7ef48640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba781012a0 0x7fba78196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba7ef48640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba78101c60 0x7fba7819d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba7ef48640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7819dc40 con 0x7fba781012a0 2024-03-12T05:55:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba7ef48640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7819ddb0 con 0x7fba781000b0 2024-03-12T05:55:40.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba7ef48640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7819df70 con 0x7fba78101c60 2024-03-12T05:55:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba77fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba781012a0 0x7fba78196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba77fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba781012a0 0x7fba78196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47706/0 (socket says 172.21.15.76:47706) 2024-03-12T05:55:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba77fff640 1 -- 172.21.15.76:0/3260081633 learned_addr learned my addr 172.21.15.76:0/3260081633 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba7d4e5640 1 --2- 172.21.15.76:0/3260081633 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba78101c60 0x7fba7819d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba7cce4640 1 --2- 172.21.15.76:0/3260081633 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba781000b0 0x7fba78196150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba77fff640 1 -- 172.21.15.76:0/3260081633 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba78101c60 msgr2=0x7fba7819d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba77fff640 1 --2- 172.21.15.76:0/3260081633 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba78101c60 0x7fba7819d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba77fff640 1 -- 172.21.15.76:0/3260081633 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba781000b0 msgr2=0x7fba78196150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba77fff640 1 --2- 172.21.15.76:0/3260081633 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba781000b0 0x7fba78196150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba77fff640 1 -- 172.21.15.76:0/3260081633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c009490 con 0x7fba781012a0 2024-03-12T05:55:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.178+0000 7fba77fff640 1 --2- 172.21.15.76:0/3260081633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba781012a0 0x7fba78196690 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fba6c00ea60 tx=0x7fba6c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.179+0000 7fba75ffb640 1 -- 172.21.15.76:0/3260081633 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c00cde0 con 0x7fba781012a0 2024-03-12T05:55:40.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.179+0000 7fba75ffb640 1 -- 172.21.15.76:0/3260081633 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba6c0035f0 con 0x7fba781012a0 2024-03-12T05:55:40.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.179+0000 7fba7ef48640 1 -- 172.21.15.76:0/3260081633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba68009560 con 0x7fba781012a0 2024-03-12T05:55:40.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.179+0000 7fba75ffb640 1 -- 172.21.15.76:0/3260081633 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c010690 con 0x7fba781012a0 2024-03-12T05:55:40.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.179+0000 7fba7ef48640 1 -- 172.21.15.76:0/3260081633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7819e620 con 0x7fba781012a0 2024-03-12T05:55:40.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.180+0000 7fba7ef48640 1 -- 172.21.15.76:0/3260081633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba78107370 con 0x7fba781012a0 2024-03-12T05:55:40.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.181+0000 7fba75ffb640 1 -- 172.21.15.76:0/3260081633 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 19) v1 ==== 136296+0+0 (secure 0 0 0) 0x7fba6c003af0 con 0x7fba781012a0 2024-03-12T05:55:40.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.182+0000 7fba75ffb640 1 --2- 172.21.15.76:0/3260081633 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fba600a1af0 0x7fba600a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:40.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.182+0000 7fba7cce4640 1 -- 172.21.15.76:0/3260081633 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fba600a1af0 msgr2=0x7fba600a3fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/1718825251 2024-03-12T05:55:40.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.182+0000 7fba75ffb640 1 -- 172.21.15.76:0/3260081633 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fba6c014070 con 0x7fba781012a0 2024-03-12T05:55:40.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.182+0000 7fba7cce4640 1 --2- 172.21.15.76:0/3260081633 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fba600a1af0 0x7fba600a3fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:55:40.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.186+0000 7fba75ffb640 1 -- 172.21.15.76:0/3260081633 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fba6c096a80 con 0x7fba781012a0 2024-03-12T05:55:40.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.367+0000 7fba7ef48640 1 -- 172.21.15.76:0/3260081633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fba7810bb10 con 0x7fba781012a0 2024-03-12T05:55:40.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.371+0000 7fba75ffb640 1 -- 172.21.15.76:0/3260081633 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7fba6c09a140 con 0x7fba781012a0 2024-03-12T05:55:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.373+0000 7fba7ef48640 1 -- 172.21.15.76:0/3260081633 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fba600a1af0 msgr2=0x7fba600a3fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:55:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.373+0000 7fba7ef48640 1 --2- 172.21.15.76:0/3260081633 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fba600a1af0 0x7fba600a3fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.373+0000 7fba7ef48640 1 -- 172.21.15.76:0/3260081633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba781012a0 msgr2=0x7fba78196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.373+0000 7fba7ef48640 1 --2- 172.21.15.76:0/3260081633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba781012a0 0x7fba78196690 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fba6c00ea60 tx=0x7fba6c00ef20 comp rx=0 tx=0).stop 2024-03-12T05:55:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.373+0000 7fba7ef48640 1 -- 172.21.15.76:0/3260081633 shutdown_connections 2024-03-12T05:55:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.373+0000 7fba7ef48640 1 --2- 172.21.15.76:0/3260081633 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fba600a1af0 0x7fba600a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.373+0000 7fba7ef48640 1 --2- 172.21.15.76:0/3260081633 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba78101c60 0x7fba7819d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.373+0000 7fba7ef48640 1 --2- 172.21.15.76:0/3260081633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba781012a0 0x7fba78196690 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.373+0000 7fba7ef48640 1 --2- 172.21.15.76:0/3260081633 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba781000b0 0x7fba78196150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.373+0000 7fba7ef48640 1 -- 172.21.15.76:0/3260081633 >> 172.21.15.76:0/3260081633 conn(0x7fba780fb620 msgr2=0x7fba78104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.373+0000 7fba7ef48640 1 -- 172.21.15.76:0/3260081633 shutdown_connections 2024-03-12T05:55:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.373+0000 7fba7ef48640 1 -- 172.21.15.76:0/3260081633 wait complete. 2024-03-12T05:55:40.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:40 smithi076 ceph-mon[32095]: pgmap v347: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-12T05:55:40.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:40 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2703560982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:40.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:40 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1366106791' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:40.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:40 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2113346888' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-12T05:55:40.387 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:40 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: ignoring --setuser ceph since I am not root 2024-03-12T05:55:40.387 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:40 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: ignoring --setgroup ceph since I am not root 2024-03-12T05:55:40.388 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-12T05:55:40.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:40 smithi082 ceph-mon[36382]: pgmap v347: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-12T05:55:40.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:40 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2703560982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:40.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:40 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1366106791' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:40.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:40 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2113346888' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-12T05:55:40.445 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:40 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: ignoring --setuser ceph since I am not root 2024-03-12T05:55:40.445 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:40 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: ignoring --setgroup ceph since I am not root 2024-03-12T05:55:40.445 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:40 smithi082 ceph-mgr[37764]: -- 172.21.15.82:0/3030886059 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564ebfd581a0 con 0x564ebfd2ec00 2024-03-12T05:55:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:40 smithi088 ceph-mon[36585]: pgmap v347: 97 pgs: 97 active+clean; 641 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-12T05:55:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:40 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2703560982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:55:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:40 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1366106791' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T05:55:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:40 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2113346888' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-12T05:55:40.484 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:40 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: ignoring --setuser ceph since I am not root 2024-03-12T05:55:40.484 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:40 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: ignoring --setgroup ceph since I am not root 2024-03-12T05:55:40.484 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:40 smithi088 ceph-mgr[38431]: -- 172.21.15.88:0/1894881198 <== mon.1 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5618d1d7c1a0 con 0x5618d1d52c00 2024-03-12T05:55:40.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.489+0000 7f31640f4640 1 -- 172.21.15.76:0/3413084058 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f315c104560 msgr2=0x7f315c106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:40.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.489+0000 7f31640f4640 1 --2- 172.21.15.76:0/3413084058 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f315c104560 0x7f315c106950 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f314c009900 tx=0x7f314c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:40.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.489+0000 7f31640f4640 1 -- 172.21.15.76:0/3413084058 shutdown_connections 2024-03-12T05:55:40.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.489+0000 7f31640f4640 1 --2- 172.21.15.76:0/3413084058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f315c106e90 0x7f315c1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.489+0000 7f31640f4640 1 --2- 172.21.15.76:0/3413084058 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f315c104560 0x7f315c106950 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.489+0000 7f31640f4640 1 --2- 172.21.15.76:0/3413084058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f315c101c30 0x7f315c104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.489+0000 7f31640f4640 1 -- 172.21.15.76:0/3413084058 >> 172.21.15.76:0/3413084058 conn(0x7f315c0fb620 msgr2=0x7f315c0fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.489+0000 7f31640f4640 1 -- 172.21.15.76:0/3413084058 shutdown_connections 2024-03-12T05:55:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.489+0000 7f31640f4640 1 -- 172.21.15.76:0/3413084058 wait complete. 2024-03-12T05:55:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.489+0000 7f31640f4640 1 Processor -- start 2024-03-12T05:55:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.489+0000 7f31640f4640 1 -- start start 2024-03-12T05:55:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f31640f4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f315c101c30 0x7f315c197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f31640f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f315c104560 0x7f315c1977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f31640f4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f315c106e90 0x7f315c19e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f31640f4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315c19ed50 con 0x7f315c101c30 2024-03-12T05:55:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f31640f4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315c19eec0 con 0x7f315c106e90 2024-03-12T05:55:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f31640f4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315c19f080 con 0x7f315c104560 2024-03-12T05:55:40.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f3161e90640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f315c101c30 0x7f315c197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:40.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f3161e90640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f315c101c30 0x7f315c197260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47716/0 (socket says 172.21.15.76:47716) 2024-03-12T05:55:40.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f3161e90640 1 -- 172.21.15.76:0/26119701 learned_addr learned my addr 172.21.15.76:0/26119701 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:40.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f316168f640 1 --2- 172.21.15.76:0/26119701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f315c104560 0x7f315c1977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:40.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f3162691640 1 --2- 172.21.15.76:0/26119701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f315c106e90 0x7f315c19e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:40.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f3161e90640 1 -- 172.21.15.76:0/26119701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f315c104560 msgr2=0x7f315c1977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:40.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f3161e90640 1 --2- 172.21.15.76:0/26119701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f315c104560 0x7f315c1977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f3161e90640 1 -- 172.21.15.76:0/26119701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f315c106e90 msgr2=0x7f315c19e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:40.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f3161e90640 1 --2- 172.21.15.76:0/26119701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f315c106e90 0x7f315c19e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f3161e90640 1 -- 172.21.15.76:0/26119701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c009560 con 0x7f315c101c30 2024-03-12T05:55:40.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f3161e90640 1 --2- 172.21.15.76:0/26119701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f315c101c30 0x7f315c197260 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f315800e9f0 tx=0x7f315800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:40.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f3152ffd640 1 -- 172.21.15.76:0/26119701 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158009700 con 0x7f315c101c30 2024-03-12T05:55:40.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f3152ffd640 1 -- 172.21.15.76:0/26119701 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f315800cd80 con 0x7f315c101c30 2024-03-12T05:55:40.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f31640f4640 1 -- 172.21.15.76:0/26119701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f315c19f360 con 0x7f315c101c30 2024-03-12T05:55:40.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f3152ffd640 1 -- 172.21.15.76:0/26119701 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158018920 con 0x7f315c101c30 2024-03-12T05:55:40.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.490+0000 7f31640f4640 1 -- 172.21.15.76:0/26119701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f315c19f840 con 0x7f315c101c30 2024-03-12T05:55:40.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.492+0000 7f31640f4640 1 -- 172.21.15.76:0/26119701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315c04f730 con 0x7f315c101c30 2024-03-12T05:55:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.492+0000 7f3152ffd640 1 -- 172.21.15.76:0/26119701 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 19) v1 ==== 136296+0+0 (secure 0 0 0) 0x7f3158018a80 con 0x7f315c101c30 2024-03-12T05:55:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.493+0000 7f3152ffd640 1 --2- 172.21.15.76:0/26119701 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f31480a1a70 0x7f31480a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.493+0000 7f316168f640 1 -- 172.21.15.76:0/26119701 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f31480a1a70 msgr2=0x7f31480a3f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/1718825251 2024-03-12T05:55:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.493+0000 7f3152ffd640 1 -- 172.21.15.76:0/26119701 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3158014070 con 0x7f315c101c30 2024-03-12T05:55:40.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.493+0000 7f316168f640 1 --2- 172.21.15.76:0/26119701 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f31480a1a70 0x7f31480a3f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:55:40.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.497+0000 7f3152ffd640 1 -- 172.21.15.76:0/26119701 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3158052d20 con 0x7f315c101c30 2024-03-12T05:55:40.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.694+0000 7f316168f640 1 -- 172.21.15.76:0/26119701 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f31480a1a70 msgr2=0x7f31480a3f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/1718825251 2024-03-12T05:55:40.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.694+0000 7f316168f640 1 --2- 172.21.15.76:0/26119701 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f31480a1a70 0x7f31480a3f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T05:55:40.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.700+0000 7f31640f4640 1 -- 172.21.15.76:0/26119701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f315c0fd110 con 0x7f315c101c30 2024-03-12T05:55:40.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.704+0000 7f3152ffd640 1 -- 172.21.15.76:0/26119701 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v32)=0 v32) v1 ==== 133+0+0 (secure 0 0 0) 0x7f315c0fd110 con 0x7f315c101c30 2024-03-12T05:55:40.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.706+0000 7f31640f4640 1 -- 172.21.15.76:0/26119701 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f31480a1a70 msgr2=0x7f31480a3f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:55:40.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.706+0000 7f31640f4640 1 --2- 172.21.15.76:0/26119701 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f31480a1a70 0x7f31480a3f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.706+0000 7f31640f4640 1 -- 172.21.15.76:0/26119701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f315c101c30 msgr2=0x7f315c197260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:40.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.706+0000 7f31640f4640 1 --2- 172.21.15.76:0/26119701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f315c101c30 0x7f315c197260 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f315800e9f0 tx=0x7f315800eeb0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.706+0000 7f31640f4640 1 -- 172.21.15.76:0/26119701 shutdown_connections 2024-03-12T05:55:40.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.706+0000 7f31640f4640 1 --2- 172.21.15.76:0/26119701 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7f31480a1a70 0x7f31480a3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.706+0000 7f31640f4640 1 --2- 172.21.15.76:0/26119701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f315c106e90 0x7f315c19e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.706+0000 7f31640f4640 1 --2- 172.21.15.76:0/26119701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f315c104560 0x7f315c1977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.706+0000 7f31640f4640 1 --2- 172.21.15.76:0/26119701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f315c101c30 0x7f315c197260 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.706+0000 7f31640f4640 1 -- 172.21.15.76:0/26119701 >> 172.21.15.76:0/26119701 conn(0x7f315c0fb620 msgr2=0x7f315c1027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:40.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.706+0000 7f31640f4640 1 -- 172.21.15.76:0/26119701 shutdown_connections 2024-03-12T05:55:40.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.706+0000 7f31640f4640 1 -- 172.21.15.76:0/26119701 wait complete. 2024-03-12T05:55:40.721 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-12T05:55:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.823+0000 7fdf3540f640 1 -- 172.21.15.76:0/1668130115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf30104590 msgr2=0x7fdf30106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.823+0000 7fdf3540f640 1 --2- 172.21.15.76:0/1668130115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf30104590 0x7fdf30106980 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdf20009900 tx=0x7fdf2002bc30 comp rx=0 tx=0).stop 2024-03-12T05:55:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.823+0000 7fdf3540f640 1 -- 172.21.15.76:0/1668130115 shutdown_connections 2024-03-12T05:55:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.823+0000 7fdf3540f640 1 --2- 172.21.15.76:0/1668130115 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf30106ec0 0x7fdf30109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.823+0000 7fdf3540f640 1 --2- 172.21.15.76:0/1668130115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf30104590 0x7fdf30106980 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.823+0000 7fdf3540f640 1 --2- 172.21.15.76:0/1668130115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf30101c60 0x7fdf30104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.823+0000 7fdf3540f640 1 -- 172.21.15.76:0/1668130115 >> 172.21.15.76:0/1668130115 conn(0x7fdf300fb690 msgr2=0x7fdf300fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.824+0000 7fdf3540f640 1 -- 172.21.15.76:0/1668130115 shutdown_connections 2024-03-12T05:55:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.824+0000 7fdf3540f640 1 -- 172.21.15.76:0/1668130115 wait complete. 2024-03-12T05:55:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.824+0000 7fdf3540f640 1 Processor -- start 2024-03-12T05:55:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.824+0000 7fdf3540f640 1 -- start start 2024-03-12T05:55:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.824+0000 7fdf3540f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf30101c60 0x7fdf301972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.824+0000 7fdf3540f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf30104590 0x7fdf301977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.824+0000 7fdf3540f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf30106ec0 0x7fdf3019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.824+0000 7fdf3540f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3019ed90 con 0x7fdf30101c60 2024-03-12T05:55:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.824+0000 7fdf3540f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3019ef00 con 0x7fdf30106ec0 2024-03-12T05:55:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.824+0000 7fdf3540f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3019f0c0 con 0x7fdf30104590 2024-03-12T05:55:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf2effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf30101c60 0x7fdf301972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:40.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf2effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf30101c60 0x7fdf301972a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47732/0 (socket says 172.21.15.76:47732) 2024-03-12T05:55:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf2effd640 1 -- 172.21.15.76:0/1470858315 learned_addr learned my addr 172.21.15.76:0/1470858315 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf2f7fe640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf30106ec0 0x7fdf3019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf2e7fc640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf30104590 0x7fdf301977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf2effd640 1 -- 172.21.15.76:0/1470858315 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf30104590 msgr2=0x7fdf301977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf2effd640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf30104590 0x7fdf301977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf2effd640 1 -- 172.21.15.76:0/1470858315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf30106ec0 msgr2=0x7fdf3019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf2effd640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf30106ec0 0x7fdf3019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf2effd640 1 -- 172.21.15.76:0/1470858315 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf20009560 con 0x7fdf30101c60 2024-03-12T05:55:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf2f7fe640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf30106ec0 0x7fdf3019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:55:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf2effd640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf30101c60 0x7fdf301972a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fdf2400e9f0 tx=0x7fdf2400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf17fff640 1 -- 172.21.15.76:0/1470858315 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24009720 con 0x7fdf30101c60 2024-03-12T05:55:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf3540f640 1 -- 172.21.15.76:0/1470858315 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3019f3a0 con 0x7fdf30101c60 2024-03-12T05:55:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf17fff640 1 -- 172.21.15.76:0/1470858315 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf2400cd80 con 0x7fdf30101c60 2024-03-12T05:55:40.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf17fff640 1 -- 172.21.15.76:0/1470858315 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24010690 con 0x7fdf30101c60 2024-03-12T05:55:40.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.825+0000 7fdf3540f640 1 -- 172.21.15.76:0/1470858315 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3019f830 con 0x7fdf30101c60 2024-03-12T05:55:40.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.826+0000 7fdf3540f640 1 -- 172.21.15.76:0/1470858315 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf300fd180 con 0x7fdf30101c60 2024-03-12T05:55:40.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.827+0000 7fdf17fff640 1 -- 172.21.15.76:0/1470858315 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 19) v1 ==== 136296+0+0 (secure 0 0 0) 0x7fdf240046c0 con 0x7fdf30101c60 2024-03-12T05:55:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.828+0000 7fdf17fff640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdf180a1a70 0x7fdf180a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.828+0000 7fdf2e7fc640 1 -- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdf180a1a70 msgr2=0x7fdf180a3f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/1718825251 2024-03-12T05:55:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.828+0000 7fdf17fff640 1 -- 172.21.15.76:0/1470858315 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdf24014070 con 0x7fdf30101c60 2024-03-12T05:55:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.828+0000 7fdf2e7fc640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdf180a1a70 0x7fdf180a3f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T05:55:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:40.832+0000 7fdf17fff640 1 -- 172.21.15.76:0/1470858315 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdf240969e0 con 0x7fdf30101c60 2024-03-12T05:55:40.944 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:40 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:40.610+0000 7f01d9c6a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:55:40.945 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:40 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:40.703+0000 7f01d9c6a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:55:40.984 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:40 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:40.616+0000 7f5162cf8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:55:40.984 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:40 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:40.711+0000 7f5162cf8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:55:40.987 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:40 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:40.657+0000 7fca7ec4c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T05:55:40.988 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:40 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:40.757+0000 7fca7ec4c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T05:55:41.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:41.029+0000 7fdf2e7fc640 1 -- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdf180a1a70 msgr2=0x7fdf180a3f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/1718825251 2024-03-12T05:55:41.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:41.029+0000 7fdf2e7fc640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdf180a1a70 0x7fdf180a3f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T05:55:41.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:41.032+0000 7fdf3540f640 1 -- 172.21.15.76:0/1470858315 --> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf3002ceb0 con 0x7fdf180a1a70 2024-03-12T05:55:41.334 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:41 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T05:55:41.334 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:41 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:55:41.334 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:41 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:55:41.334 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:41 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:55:41.334 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:41 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2113346888' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-12T05:55:41.334 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:41 smithi082 ceph-mon[36382]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-03-12T05:55:41.334 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:41 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3260081633' entity='client.admin' 2024-03-12T05:55:41.334 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:41 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/26119701' entity='client.admin' 2024-03-12T05:55:41.346 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:41 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T05:55:41.347 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:41 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:55:41.347 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:41 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:55:41.347 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:41 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:55:41.347 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:41 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2113346888' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-12T05:55:41.347 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:41 smithi088 ceph-mon[36585]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-03-12T05:55:41.347 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:41 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3260081633' entity='client.admin' 2024-03-12T05:55:41.347 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:41 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/26119701' entity='client.admin' 2024-03-12T05:55:41.375 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:41 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T05:55:41.375 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:41 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:55:41.375 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:41 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:55:41.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:41 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:55:41.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:41 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2113346888' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-12T05:55:41.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:41 smithi076 ceph-mon[32095]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-03-12T05:55:41.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:41 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3260081633' entity='client.admin' 2024-03-12T05:55:41.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:41 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/26119701' entity='client.admin' 2024-03-12T05:55:41.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:41.430+0000 7fdf2e7fc640 1 -- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdf180a1a70 msgr2=0x7fdf180a3f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/1718825251 2024-03-12T05:55:41.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:41.430+0000 7fdf2e7fc640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdf180a1a70 0x7fdf180a3f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-12T05:55:41.645 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:41 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:41.406+0000 7fca7ec4c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:55:41.645 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:41 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:41.500+0000 7fca7ec4c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:55:41.645 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:41 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:41.640+0000 7fca7ec4c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:55:41.659 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:41.330+0000 7f01d9c6a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:55:41.659 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:41.421+0000 7f01d9c6a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:55:41.659 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:41.561+0000 7f01d9c6a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:55:41.659 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:41.655+0000 7f01d9c6a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:55:41.675 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:41 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:41.343+0000 7f5162cf8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T05:55:41.675 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:41 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:41.436+0000 7f5162cf8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T05:55:41.675 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:41 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:41.577+0000 7f5162cf8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T05:55:41.944 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:41.737+0000 7f01d9c6a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:55:41.984 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:41 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:41.671+0000 7f5162cf8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:55:41.984 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:41 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:41.754+0000 7f5162cf8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:55:41.987 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:41 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:41.736+0000 7fca7ec4c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T05:55:41.988 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:41 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:41.819+0000 7fca7ec4c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T05:55:42.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:42.231+0000 7fdf2e7fc640 1 -- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdf180a1a70 msgr2=0x7fdf180a3f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/1718825251 2024-03-12T05:55:42.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:42.231+0000 7fdf2e7fc640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdf180a1a70 0x7fdf180a3f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-12T05:55:42.286 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:42 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:55:42.286 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:41.995+0000 7f01d9c6a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:55:42.286 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:42.102+0000 7f01d9c6a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:55:42.286 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:42.186+0000 7f01d9c6a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:55:42.286 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:42.281+0000 7f01d9c6a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:55:42.313 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:42 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:55:42.313 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:42 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:42.017+0000 7f5162cf8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:55:42.313 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:42 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:42.125+0000 7f5162cf8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:55:42.313 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:42 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:42.212+0000 7f5162cf8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:55:42.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:42 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:55:42.377 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:42 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:42.082+0000 7fca7ec4c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T05:55:42.377 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:42 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:42.189+0000 7fca7ec4c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T05:55:42.377 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:42 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:42.277+0000 7fca7ec4c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T05:55:42.377 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:42 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:42.372+0000 7fca7ec4c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:55:42.655 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:42 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:42.556+0000 7fca7ec4c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:55:42.694 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:42.454+0000 7f01d9c6a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:55:42.734 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:42 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:42.309+0000 7f5162cf8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T05:55:42.734 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:42 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:42.486+0000 7f5162cf8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T05:55:43.695 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:43 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:43.399+0000 7f01d9c6a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:55:43.695 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:43 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:43.527+0000 7f01d9c6a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:55:43.734 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:43 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:43.441+0000 7f5162cf8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:55:43.734 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:43 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:43.571+0000 7f5162cf8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:55:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:43.832+0000 7fdf2e7fc640 1 -- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdf180a1a70 msgr2=0x7fdf180a3f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.76:6800/1718825251 2024-03-12T05:55:43.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:43.832+0000 7fdf2e7fc640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdf180a1a70 0x7fdf180a3f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-12T05:55:43.988 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:43 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:43.552+0000 7fca7ec4c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T05:55:43.988 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:43 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:43.688+0000 7fca7ec4c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T05:55:44.137 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:43 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:43.853+0000 7f01d9c6a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:55:44.137 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:43 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:43.941+0000 7f01d9c6a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:55:44.137 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:44 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:44.133+0000 7f01d9c6a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:55:44.192 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:43 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:43.905+0000 7f5162cf8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:55:44.192 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:43 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:43.995+0000 7f5162cf8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:55:44.306 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:44 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:44.016+0000 7fca7ec4c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T05:55:44.306 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:44 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:44.106+0000 7fca7ec4c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T05:55:44.444 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:44 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:44.315+0000 7f01d9c6a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:55:44.484 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:44 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:44.188+0000 7f5162cf8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:55:44.484 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:44 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:44.373+0000 7f5162cf8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:55:44.738 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:44 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:44.302+0000 7fca7ec4c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:55:44.738 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:44 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:44.489+0000 7fca7ec4c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T05:55:44.852 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:44 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:44.594+0000 7f01d9c6a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:55:44.934 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:44 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:44.657+0000 7f5162cf8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:55:45.039 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:44 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:44.779+0000 7fca7ec4c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T05:55:45.195 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:44 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:44.848+0000 7f01d9c6a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:55:45.195 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:45 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:45.079+0000 7f01d9c6a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:55:45.234 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:44 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:44.914+0000 7f5162cf8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:55:45.234 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:45 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:45.149+0000 7f5162cf8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:55:45.488 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:45 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:45.035+0000 7fca7ec4c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T05:55:45.488 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:45 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:45.272+0000 7fca7ec4c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T05:55:46.194 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:45 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:45.869+0000 7f01d9c6a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:55:46.234 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:45 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:45.943+0000 7f5162cf8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:55:46.488 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:46 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:46.075+0000 7fca7ec4c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T05:55:46.694 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:46 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:46.438+0000 7f01d9c6a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:55:46.695 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:46 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:46.534+0000 7f01d9c6a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:55:46.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:46.821+0000 7fdf17fff640 1 -- 172.21.15.76:0/1470858315 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 20) v1 ==== 136296+0+0 (secure 0 0 0) 0x7fdf24096db0 con 0x7fdf30101c60 2024-03-12T05:55:46.828 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:46 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:46.512+0000 7f5162cf8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:55:46.828 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:46 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:46.609+0000 7f5162cf8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:55:46.970 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:46 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:46.644+0000 7fca7ec4c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T05:55:46.971 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:46 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:46.743+0000 7fca7ec4c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T05:55:46.971 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:46 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:46.967+0000 7fca7ec4c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:55:46.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:46 smithi076 ceph-mon[32095]: Standby manager daemon y restarted 2024-03-12T05:55:46.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:46 smithi076 ceph-mon[32095]: Standby manager daemon y started 2024-03-12T05:55:46.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:46 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.82:0/3941643842' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T05:55:46.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:46 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.82:0/3941643842' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:55:46.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:46 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.82:0/3941643842' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T05:55:46.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:46 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.82:0/3941643842' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:55:46.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:46.990+0000 7fdf17fff640 1 -- 172.21.15.76:0/1470858315 <== mon.0 v2:172.21.15.76:3300/0 8 ==== mgrmap(e 21) v1 ==== 135964+0+0 (secure 0 0 0) 0x7fdf240cb320 con 0x7fdf30101c60 2024-03-12T05:55:46.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:46.991+0000 7fdf17fff640 1 -- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdf180a1a70 msgr2=0x7fdf180a3f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T05:55:46.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:46.991+0000 7fdf17fff640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/1718825251,v1:172.21.15.76:6801/1718825251] conn(0x7fdf180a1a70 0x7fdf180a3f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:47.136 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 05:55:46 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y[37759]: 2024-03-12T05:55:46.748+0000 7f01d9c6a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:55:47.136 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:46 smithi082 ceph-mon[36382]: Standby manager daemon y restarted 2024-03-12T05:55:47.136 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:46 smithi082 ceph-mon[36382]: Standby manager daemon y started 2024-03-12T05:55:47.136 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:46 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.82:0/3941643842' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T05:55:47.136 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:46 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.82:0/3941643842' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:55:47.137 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:46 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.82:0/3941643842' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T05:55:47.137 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:46 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.82:0/3941643842' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:55:47.137 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 05:55:46 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z[38408]: 2024-03-12T05:55:46.827+0000 7f5162cf8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T05:55:47.137 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:46 smithi088 ceph-mon[36585]: Standby manager daemon y restarted 2024-03-12T05:55:47.137 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:46 smithi088 ceph-mon[36585]: Standby manager daemon y started 2024-03-12T05:55:47.137 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:46 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.82:0/3941643842' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T05:55:47.137 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:46 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.82:0/3941643842' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:55:47.138 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:46 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.82:0/3941643842' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T05:55:47.138 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:46 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.82:0/3941643842' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:55:47.488 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:47.231+0000 7fc9e0679700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:55:47.488 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:47.231+0000 7fc9e0679700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:55:47.488 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:47.231+0000 7fc9e0679700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:55:47.488 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:47.231+0000 7fc9e0679700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:55:47.488 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:47.231+0000 7fc9e0679700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:55:47.489 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:47.232+0000 7fc9dd673700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:55:47.489 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:47.232+0000 7fc9dd673700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:55:47.489 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:47.232+0000 7fc9dd673700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:55:47.489 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:47.232+0000 7fc9dd673700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:55:47.489 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:55:47.232+0000 7fc9dd673700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:55:47.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:47.995+0000 7fdf17fff640 1 -- 172.21.15.76:0/1470858315 <== mon.0 v2:172.21.15.76:3300/0 9 ==== mgrmap(e 22) v1 ==== 136091+0+0 (secure 0 0 0) 0x7fdf240a2a50 con 0x7fdf30101c60 2024-03-12T05:55:47.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:47.996+0000 7fdf17fff640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdf18034670 0x7fdf18034a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:48.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:47.996+0000 7fdf17fff640 1 -- 172.21.15.76:0/1470858315 --> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf3002ceb0 con 0x7fdf18034670 2024-03-12T05:55:48.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:47.998+0000 7fdf2e7fc640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdf18034670 0x7fdf18034a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:48.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.001+0000 7fdf2e7fc640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdf18034670 0x7fdf18034a50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdf20005ab0 tx=0x7fdf20005fd0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:48.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.065+0000 7fdf17fff640 1 -- 172.21.15.76:0/1470858315 <== mgr.15465 v2:172.21.15.76:6800/513615067 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fdf3002ceb0 con 0x7fdf18034670 2024-03-12T05:55:48.069 INFO:teuthology.orchestra.run.smithi076.stdout:Schedule set for path / 2024-03-12T05:55:48.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.067+0000 7fdf3540f640 1 -- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdf18034670 msgr2=0x7fdf18034a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.067+0000 7fdf3540f640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdf18034670 0x7fdf18034a50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdf20005ab0 tx=0x7fdf20005fd0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.067+0000 7fdf3540f640 1 -- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf30101c60 msgr2=0x7fdf301972a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.067+0000 7fdf3540f640 1 --2- 172.21.15.76:0/1470858315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf30101c60 0x7fdf301972a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fdf2400e9f0 tx=0x7fdf2400edf0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.067+0000 7fdf2f7fe640 1 -- 172.21.15.76:0/1470858315 reap_dead start 2024-03-12T05:55:48.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.067+0000 7fdf3540f640 1 -- 172.21.15.76:0/1470858315 shutdown_connections 2024-03-12T05:55:48.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.067+0000 7fdf3540f640 1 -- 172.21.15.76:0/1470858315 >> 172.21.15.76:0/1470858315 conn(0x7fdf300fb690 msgr2=0x7fdf30102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:48.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.067+0000 7fdf3540f640 1 -- 172.21.15.76:0/1470858315 shutdown_connections 2024-03-12T05:55:48.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.067+0000 7fdf3540f640 1 -- 172.21.15.76:0/1470858315 wait complete. 2024-03-12T05:55:48.081 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-12T05:55:48.081 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: Standby manager daemon z restarted 2024-03-12T05:55:48.081 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: Standby manager daemon z started 2024-03-12T05:55:48.081 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.88:0/3344976465' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T05:55:48.081 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.88:0/3344976465' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:55:48.081 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.88:0/3344976465' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T05:55:48.082 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.? 172.21.15.88:0/3344976465' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:55:48.082 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: Active manager daemon x restarted 2024-03-12T05:55:48.082 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: Activating manager daemon x 2024-03-12T05:55:48.082 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: osdmap e89: 12 total, 12 up, 12 in 2024-03-12T05:55:48.082 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: mgrmap e21: x(active, starting, since 0.0123215s), standbys: z, y 2024-03-12T05:55:48.082 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:55:48.082 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:55:48.082 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:55:48.082 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-12T05:55:48.082 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-12T05:55:48.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:55:48.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:55:48.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:55:48.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:55:48.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:55:48.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:55:48.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:55:48.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:55:48.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:55:48.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:55:48.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T05:55:48.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T05:55:48.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T05:55:48.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:55:48.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:55:48.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:55:48.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:55:48.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:55:48.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:55:48.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:55:48.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:55:48.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:55:48.085 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:55:48.085 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:55:48.085 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:55:48.085 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T05:55:48.085 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T05:55:48.085 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T05:55:48.085 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: Manager daemon x is now available 2024-03-12T05:55:48.085 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:55:48.085 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:55:48.085 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T05:55:48.085 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:47 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T05:55:48.086 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-12T05:55:48.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.189+0000 7f4d2f876640 1 -- 172.21.15.76:0/3693525562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d28106ec0 msgr2=0x7f4d28109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.189+0000 7f4d2f876640 1 --2- 172.21.15.76:0/3693525562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d28106ec0 0x7f4d28109310 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4d20009ca0 tx=0x7f4d2002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.189+0000 7f4d2f876640 1 -- 172.21.15.76:0/3693525562 shutdown_connections 2024-03-12T05:55:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.189+0000 7f4d2f876640 1 --2- 172.21.15.76:0/3693525562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d28106ec0 0x7f4d28109310 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.189+0000 7f4d2f876640 1 --2- 172.21.15.76:0/3693525562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4d28104590 0x7f4d28106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.189+0000 7f4d2f876640 1 --2- 172.21.15.76:0/3693525562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d28101c60 0x7f4d28104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.189+0000 7f4d2f876640 1 -- 172.21.15.76:0/3693525562 >> 172.21.15.76:0/3693525562 conn(0x7f4d280fb650 msgr2=0x7f4d280fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.189+0000 7f4d2f876640 1 -- 172.21.15.76:0/3693525562 shutdown_connections 2024-03-12T05:55:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.189+0000 7f4d2f876640 1 -- 172.21.15.76:0/3693525562 wait complete. 2024-03-12T05:55:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2f876640 1 Processor -- start 2024-03-12T05:55:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2f876640 1 -- start start 2024-03-12T05:55:48.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2f876640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d28101c60 0x7f4d28197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:48.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2f876640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4d28104590 0x7f4d281977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:48.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2f876640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d28106ec0 0x7f4d2819e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:48.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2f876640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2819ed30 con 0x7f4d28106ec0 2024-03-12T05:55:48.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2f876640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2819eea0 con 0x7f4d28104590 2024-03-12T05:55:48.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2f876640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2819f060 con 0x7f4d28101c60 2024-03-12T05:55:48.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2de13640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d28106ec0 0x7f4d2819e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:48.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2de13640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d28106ec0 0x7f4d2819e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48752/0 (socket says 172.21.15.76:48752) 2024-03-12T05:55:48.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2de13640 1 -- 172.21.15.76:0/2241511674 learned_addr learned my addr 172.21.15.76:0/2241511674 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:48.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2d612640 1 --2- 172.21.15.76:0/2241511674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d28101c60 0x7f4d28197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:48.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2ce11640 1 --2- 172.21.15.76:0/2241511674 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4d28104590 0x7f4d281977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:48.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2de13640 1 -- 172.21.15.76:0/2241511674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d28101c60 msgr2=0x7f4d28197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2de13640 1 --2- 172.21.15.76:0/2241511674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d28101c60 0x7f4d28197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2de13640 1 -- 172.21.15.76:0/2241511674 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4d28104590 msgr2=0x7f4d281977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2de13640 1 --2- 172.21.15.76:0/2241511674 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4d28104590 0x7f4d281977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.190+0000 7f4d2de13640 1 -- 172.21.15.76:0/2241511674 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d18009490 con 0x7f4d28106ec0 2024-03-12T05:55:48.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.191+0000 7f4d2de13640 1 --2- 172.21.15.76:0/2241511674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d28106ec0 0x7f4d2819e7f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4d20037040 tx=0x7f4d20009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:48.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.191+0000 7f4d1e7fc640 1 -- 172.21.15.76:0/2241511674 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d200329b0 con 0x7f4d28106ec0 2024-03-12T05:55:48.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.191+0000 7f4d1e7fc640 1 -- 172.21.15.76:0/2241511674 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d20003e50 con 0x7f4d28106ec0 2024-03-12T05:55:48.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.191+0000 7f4d2f876640 1 -- 172.21.15.76:0/2241511674 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d20009900 con 0x7f4d28106ec0 2024-03-12T05:55:48.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.191+0000 7f4d1e7fc640 1 -- 172.21.15.76:0/2241511674 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d20040450 con 0x7f4d28106ec0 2024-03-12T05:55:48.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.191+0000 7f4d2f876640 1 -- 172.21.15.76:0/2241511674 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2819f650 con 0x7f4d28106ec0 2024-03-12T05:55:48.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.192+0000 7f4d2f876640 1 -- 172.21.15.76:0/2241511674 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d280fd140 con 0x7f4d28106ec0 2024-03-12T05:55:48.202 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-12T05:55:48.202 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: Standby manager daemon z restarted 2024-03-12T05:55:48.202 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: Standby manager daemon z started 2024-03-12T05:55:48.202 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.88:0/3344976465' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T05:55:48.202 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.88:0/3344976465' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:55:48.203 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.88:0/3344976465' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T05:55:48.203 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.? 172.21.15.88:0/3344976465' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:55:48.203 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: Active manager daemon x restarted 2024-03-12T05:55:48.203 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: Activating manager daemon x 2024-03-12T05:55:48.203 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: osdmap e89: 12 total, 12 up, 12 in 2024-03-12T05:55:48.203 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: mgrmap e21: x(active, starting, since 0.0123215s), standbys: z, y 2024-03-12T05:55:48.203 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:55:48.203 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:55:48.203 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:55:48.203 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-12T05:55:48.203 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-12T05:55:48.203 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:55:48.203 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:55:48.203 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:55:48.204 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:55:48.204 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:55:48.204 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:55:48.204 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:55:48.204 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:55:48.204 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:55:48.204 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:55:48.204 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T05:55:48.204 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T05:55:48.204 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T05:55:48.204 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:55:48.204 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:55:48.204 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:55:48.205 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:55:48.205 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:55:48.205 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:55:48.205 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:55:48.205 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:55:48.205 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:55:48.205 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:55:48.205 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:55:48.205 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:55:48.205 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T05:55:48.205 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T05:55:48.205 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T05:55:48.205 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: Manager daemon x is now available 2024-03-12T05:55:48.206 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:55:48.206 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:55:48.206 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T05:55:48.206 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:47 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T05:55:48.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-12T05:55:48.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: Standby manager daemon z restarted 2024-03-12T05:55:48.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: Standby manager daemon z started 2024-03-12T05:55:48.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.88:0/3344976465' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-12T05:55:48.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.88:0/3344976465' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T05:55:48.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.88:0/3344976465' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-12T05:55:48.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.? 172.21.15.88:0/3344976465' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T05:55:48.206 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: Active manager daemon x restarted 2024-03-12T05:55:48.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: Activating manager daemon x 2024-03-12T05:55:48.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: osdmap e89: 12 total, 12 up, 12 in 2024-03-12T05:55:48.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: mgrmap e21: x(active, starting, since 0.0123215s), standbys: z, y 2024-03-12T05:55:48.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T05:55:48.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T05:55:48.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T05:55:48.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-12T05:55:48.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-12T05:55:48.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-12T05:55:48.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-12T05:55:48.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-12T05:55:48.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-12T05:55:48.207 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T05:55:48.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T05:55:48.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T05:55:48.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T05:55:48.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T05:55:48.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T05:55:48.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T05:55:48.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-12T05:55:48.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T05:55:48.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T05:55:48.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T05:55:48.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T05:55:48.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T05:55:48.208 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T05:55:48.209 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T05:55:48.209 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T05:55:48.209 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T05:55:48.209 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-12T05:55:48.209 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-12T05:55:48.209 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-12T05:55:48.209 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-12T05:55:48.209 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T05:55:48.209 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T05:55:48.209 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T05:55:48.209 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: Manager daemon x is now available 2024-03-12T05:55:48.209 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:55:48.209 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:55:48.210 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T05:55:48.210 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:47 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T05:55:48.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.193+0000 7f4d1e7fc640 1 -- 172.21.15.76:0/2241511674 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 22) v1 ==== 136091+0+0 (secure 0 0 0) 0x7f4d2000a040 con 0x7f4d28106ec0 2024-03-12T05:55:48.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.194+0000 7f4d1e7fc640 1 --2- 172.21.15.76:0/2241511674 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4d140a1890 0x7f4d140a3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:48.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.194+0000 7f4d2d612640 1 --2- 172.21.15.76:0/2241511674 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4d140a1890 0x7f4d140a3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:48.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.194+0000 7f4d1e7fc640 1 -- 172.21.15.76:0/2241511674 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4d200efe60 con 0x7f4d28106ec0 2024-03-12T05:55:48.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.194+0000 7f4d2d612640 1 --2- 172.21.15.76:0/2241511674 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4d140a1890 0x7f4d140a3d50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009bd0 tx=0x7f4d24005f90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:48.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.198+0000 7f4d1e7fc640 1 -- 172.21.15.76:0/2241511674 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4d200b9300 con 0x7f4d28106ec0 2024-03-12T05:55:48.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.405+0000 7f4d2f876640 1 -- 172.21.15.76:0/2241511674 --> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d2802cf50 con 0x7f4d140a1890 2024-03-12T05:55:48.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.460+0000 7f4d1e7fc640 1 -- 172.21.15.76:0/2241511674 <== mgr.15465 v2:172.21.15.76:6800/513615067 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f4d2802cf50 con 0x7f4d140a1890 2024-03-12T05:55:48.463 INFO:teuthology.orchestra.run.smithi076.stdout:Retention added to path / 2024-03-12T05:55:48.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.462+0000 7f4d2f876640 1 -- 172.21.15.76:0/2241511674 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4d140a1890 msgr2=0x7f4d140a3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.462+0000 7f4d2f876640 1 --2- 172.21.15.76:0/2241511674 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4d140a1890 0x7f4d140a3d50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009bd0 tx=0x7f4d24005f90 comp rx=0 tx=0).stop 2024-03-12T05:55:48.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.462+0000 7f4d2f876640 1 -- 172.21.15.76:0/2241511674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d28106ec0 msgr2=0x7f4d2819e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.462+0000 7f4d2f876640 1 --2- 172.21.15.76:0/2241511674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d28106ec0 0x7f4d2819e7f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4d20037040 tx=0x7f4d20009360 comp rx=0 tx=0).stop 2024-03-12T05:55:48.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.462+0000 7f4d2f876640 1 -- 172.21.15.76:0/2241511674 shutdown_connections 2024-03-12T05:55:48.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.462+0000 7f4d2f876640 1 --2- 172.21.15.76:0/2241511674 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4d140a1890 0x7f4d140a3d50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.462+0000 7f4d2f876640 1 --2- 172.21.15.76:0/2241511674 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d28106ec0 0x7f4d2819e7f0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.462+0000 7f4d2f876640 1 --2- 172.21.15.76:0/2241511674 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4d28104590 0x7f4d281977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.462+0000 7f4d2f876640 1 --2- 172.21.15.76:0/2241511674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d28101c60 0x7f4d28197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.462+0000 7f4d2f876640 1 -- 172.21.15.76:0/2241511674 >> 172.21.15.76:0/2241511674 conn(0x7f4d280fb650 msgr2=0x7f4d28102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:48.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.462+0000 7f4d2f876640 1 -- 172.21.15.76:0/2241511674 shutdown_connections 2024-03-12T05:55:48.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.462+0000 7f4d2f876640 1 -- 172.21.15.76:0/2241511674 wait complete. 2024-03-12T05:55:48.477 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-12T05:55:48.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.578+0000 7f8f8c12c640 1 -- 172.21.15.76:0/3310393786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f84101c70 msgr2=0x7f8f84108500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.578+0000 7f8f8c12c640 1 --2- 172.21.15.76:0/3310393786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f84101c70 0x7f8f84108500 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c009ca0 tx=0x7f8f7c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.578+0000 7f8f8c12c640 1 -- 172.21.15.76:0/3310393786 shutdown_connections 2024-03-12T05:55:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.578+0000 7f8f8c12c640 1 --2- 172.21.15.76:0/3310393786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f84101c70 0x7f8f84108500 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.578+0000 7f8f8c12c640 1 --2- 172.21.15.76:0/3310393786 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f841012b0 0x7f8f84101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.578+0000 7f8f8c12c640 1 --2- 172.21.15.76:0/3310393786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f841000c0 0x7f8f841004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.578+0000 7f8f8c12c640 1 -- 172.21.15.76:0/3310393786 >> 172.21.15.76:0/3310393786 conn(0x7f8f840fb650 msgr2=0x7f8f840fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.578+0000 7f8f8c12c640 1 -- 172.21.15.76:0/3310393786 shutdown_connections 2024-03-12T05:55:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.578+0000 7f8f8c12c640 1 -- 172.21.15.76:0/3310393786 wait complete. 2024-03-12T05:55:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.578+0000 7f8f8c12c640 1 Processor -- start 2024-03-12T05:55:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.578+0000 7f8f8c12c640 1 -- start start 2024-03-12T05:55:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f8c12c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f841000c0 0x7f8f8406a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f8c12c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f841012b0 0x7f8f8419e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f8c12c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f84101c70 0x7f8f841a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f8c12c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f841a0ea0 con 0x7f8f841012b0 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f8c12c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f841a1010 con 0x7f8f84101c70 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f8c12c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f841a11d0 con 0x7f8f841000c0 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f896c7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f841012b0 0x7f8f8419e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f896c7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f841012b0 0x7f8f8419e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48774/0 (socket says 172.21.15.76:48774) 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f896c7640 1 -- 172.21.15.76:0/2042916939 learned_addr learned my addr 172.21.15.76:0/2042916939 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f8a6c9640 1 --2- 172.21.15.76:0/2042916939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f84101c70 0x7f8f841a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f89ec8640 1 --2- 172.21.15.76:0/2042916939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f841000c0 0x7f8f8406a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f896c7640 1 -- 172.21.15.76:0/2042916939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f841000c0 msgr2=0x7f8f8406a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f896c7640 1 --2- 172.21.15.76:0/2042916939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f841000c0 0x7f8f8406a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f896c7640 1 -- 172.21.15.76:0/2042916939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f84101c70 msgr2=0x7f8f841a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f896c7640 1 --2- 172.21.15.76:0/2042916939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f84101c70 0x7f8f841a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f896c7640 1 -- 172.21.15.76:0/2042916939 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c009900 con 0x7f8f841012b0 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f896c7640 1 --2- 172.21.15.76:0/2042916939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f841012b0 0x7f8f8419e290 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8f8000c920 tx=0x7f8f8000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f76ffd640 1 -- 172.21.15.76:0/2042916939 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f80007c70 con 0x7f8f841012b0 2024-03-12T05:55:48.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f8c12c640 1 -- 172.21.15.76:0/2042916939 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f841a1450 con 0x7f8f841012b0 2024-03-12T05:55:48.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f76ffd640 1 -- 172.21.15.76:0/2042916939 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f80007dd0 con 0x7f8f841012b0 2024-03-12T05:55:48.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f76ffd640 1 -- 172.21.15.76:0/2042916939 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f80018860 con 0x7f8f841012b0 2024-03-12T05:55:48.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.579+0000 7f8f8c12c640 1 -- 172.21.15.76:0/2042916939 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f841a1960 con 0x7f8f841012b0 2024-03-12T05:55:48.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.581+0000 7f8f8c12c640 1 -- 172.21.15.76:0/2042916939 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f84107310 con 0x7f8f841012b0 2024-03-12T05:55:48.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.581+0000 7f8f76ffd640 1 -- 172.21.15.76:0/2042916939 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 22) v1 ==== 136091+0+0 (secure 0 0 0) 0x7f8f8000fac0 con 0x7f8f841012b0 2024-03-12T05:55:48.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.582+0000 7f8f76ffd640 1 --2- 172.21.15.76:0/2042916939 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8f700a1850 0x7f8f700a3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:48.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.582+0000 7f8f76ffd640 1 -- 172.21.15.76:0/2042916939 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8f800cd120 con 0x7f8f841012b0 2024-03-12T05:55:48.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.582+0000 7f8f89ec8640 1 --2- 172.21.15.76:0/2042916939 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8f700a1850 0x7f8f700a3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:48.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.582+0000 7f8f89ec8640 1 --2- 172.21.15.76:0/2042916939 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8f700a1850 0x7f8f700a3d10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8f78005fd0 tx=0x7f8f7800b410 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:48.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.586+0000 7f8f76ffd640 1 -- 172.21.15.76:0/2042916939 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8f80096530 con 0x7f8f841012b0 2024-03-12T05:55:48.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.793+0000 7f8f8c12c640 1 -- 172.21.15.76:0/2042916939 --> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f8402ceb0 con 0x7f8f700a1850 2024-03-12T05:55:48.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.800+0000 7f8f76ffd640 1 -- 172.21.15.76:0/2042916939 <== mgr.15465 v2:172.21.15.76:6800/513615067 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7f8f8402ceb0 con 0x7f8f700a1850 2024-03-12T05:55:48.804 INFO:teuthology.orchestra.run.smithi076.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-12T00:00:00", "created": "2024-03-12T05:55:48", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-12T05:55:48.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.802+0000 7f8f8c12c640 1 -- 172.21.15.76:0/2042916939 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8f700a1850 msgr2=0x7f8f700a3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.802+0000 7f8f8c12c640 1 --2- 172.21.15.76:0/2042916939 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8f700a1850 0x7f8f700a3d10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8f78005fd0 tx=0x7f8f7800b410 comp rx=0 tx=0).stop 2024-03-12T05:55:48.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.802+0000 7f8f8c12c640 1 -- 172.21.15.76:0/2042916939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f841012b0 msgr2=0x7f8f8419e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.802+0000 7f8f8c12c640 1 --2- 172.21.15.76:0/2042916939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f841012b0 0x7f8f8419e290 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8f8000c920 tx=0x7f8f8000cde0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.802+0000 7f8f8c12c640 1 -- 172.21.15.76:0/2042916939 shutdown_connections 2024-03-12T05:55:48.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.802+0000 7f8f8c12c640 1 --2- 172.21.15.76:0/2042916939 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8f700a1850 0x7f8f700a3d10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.802+0000 7f8f8c12c640 1 --2- 172.21.15.76:0/2042916939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f84101c70 0x7f8f841a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.802+0000 7f8f8c12c640 1 --2- 172.21.15.76:0/2042916939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f841012b0 0x7f8f8419e290 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.802+0000 7f8f8c12c640 1 --2- 172.21.15.76:0/2042916939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f841000c0 0x7f8f8406a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.802+0000 7f8f8c12c640 1 -- 172.21.15.76:0/2042916939 >> 172.21.15.76:0/2042916939 conn(0x7f8f840fb650 msgr2=0x7f8f841044e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:48.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.802+0000 7f8f8c12c640 1 -- 172.21.15.76:0/2042916939 shutdown_connections 2024-03-12T05:55:48.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.802+0000 7f8f8c12c640 1 -- 172.21.15.76:0/2042916939 wait complete. 2024-03-12T05:55:48.818 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-12T05:55:48.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.940+0000 7fe73665b640 1 -- 172.21.15.76:0/1431774146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe730102da0 msgr2=0x7fe730109630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.940+0000 7fe73665b640 1 --2- 172.21.15.76:0/1431774146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe730102da0 0x7fe730109630 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe728009ca0 tx=0x7fe72802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.940+0000 7fe73665b640 1 -- 172.21.15.76:0/1431774146 shutdown_connections 2024-03-12T05:55:48.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.940+0000 7fe73665b640 1 --2- 172.21.15.76:0/1431774146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe730102da0 0x7fe730109630 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.940+0000 7fe73665b640 1 --2- 172.21.15.76:0/1431774146 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7301023e0 0x7fe730102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.940+0000 7fe73665b640 1 --2- 172.21.15.76:0/1431774146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7301011f0 0x7fe730101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.940+0000 7fe73665b640 1 -- 172.21.15.76:0/1431774146 >> 172.21.15.76:0/1431774146 conn(0x7fe7300fc760 msgr2=0x7fe7300feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:48.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.940+0000 7fe73665b640 1 -- 172.21.15.76:0/1431774146 shutdown_connections 2024-03-12T05:55:48.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.940+0000 7fe73665b640 1 -- 172.21.15.76:0/1431774146 wait complete. 2024-03-12T05:55:48.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.940+0000 7fe73665b640 1 Processor -- start 2024-03-12T05:55:48.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe73665b640 1 -- start start 2024-03-12T05:55:48.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe73665b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe7301011f0 0x7fe730197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:48.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe73665b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7301023e0 0x7fe730197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:48.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe73665b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe730102da0 0x7fe73019e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:48.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe73665b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73019ed20 con 0x7fe7301011f0 2024-03-12T05:55:48.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe73665b640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73019ee90 con 0x7fe7301023e0 2024-03-12T05:55:48.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe73665b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73019f050 con 0x7fe730102da0 2024-03-12T05:55:48.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe72ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe7301011f0 0x7fe730197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:48.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe72ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe7301011f0 0x7fe730197230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48790/0 (socket says 172.21.15.76:48790) 2024-03-12T05:55:48.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe72ffff640 1 -- 172.21.15.76:0/1940995615 learned_addr learned my addr 172.21.15.76:0/1940995615 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:55:48.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe734bf8640 1 --2- 172.21.15.76:0/1940995615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe730102da0 0x7fe73019e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:48.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe72f7fe640 1 --2- 172.21.15.76:0/1940995615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7301023e0 0x7fe730197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:48.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe72ffff640 1 -- 172.21.15.76:0/1940995615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe730102da0 msgr2=0x7fe73019e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe72ffff640 1 --2- 172.21.15.76:0/1940995615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe730102da0 0x7fe73019e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe72ffff640 1 -- 172.21.15.76:0/1940995615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7301023e0 msgr2=0x7fe730197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:48.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe72ffff640 1 --2- 172.21.15.76:0/1940995615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7301023e0 0x7fe730197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:48.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.941+0000 7fe72ffff640 1 -- 172.21.15.76:0/1940995615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe728009900 con 0x7fe7301011f0 2024-03-12T05:55:48.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.942+0000 7fe72ffff640 1 --2- 172.21.15.76:0/1940995615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe7301011f0 0x7fe730197230 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe72000e9c0 tx=0x7fe72000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:48.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.942+0000 7fe72d7fa640 1 -- 172.21.15.76:0/1940995615 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7200096e0 con 0x7fe7301011f0 2024-03-12T05:55:48.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.942+0000 7fe72d7fa640 1 -- 172.21.15.76:0/1940995615 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe72000cd80 con 0x7fe7301011f0 2024-03-12T05:55:48.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.942+0000 7fe73665b640 1 -- 172.21.15.76:0/1940995615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe73019f330 con 0x7fe7301011f0 2024-03-12T05:55:48.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.942+0000 7fe72d7fa640 1 -- 172.21.15.76:0/1940995615 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe720010430 con 0x7fe7301011f0 2024-03-12T05:55:48.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.942+0000 7fe73665b640 1 -- 172.21.15.76:0/1940995615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73019f7f0 con 0x7fe7301011f0 2024-03-12T05:55:48.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.943+0000 7fe73665b640 1 -- 172.21.15.76:0/1940995615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe730108510 con 0x7fe7301011f0 2024-03-12T05:55:48.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.944+0000 7fe72d7fa640 1 -- 172.21.15.76:0/1940995615 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 22) v1 ==== 136091+0+0 (secure 0 0 0) 0x7fe720003680 con 0x7fe7301011f0 2024-03-12T05:55:48.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.945+0000 7fe72d7fa640 1 --2- 172.21.15.76:0/1940995615 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe7180a1810 0x7fe7180a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:55:48.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.945+0000 7fe72f7fe640 1 --2- 172.21.15.76:0/1940995615 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe7180a1810 0x7fe7180a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:55:48.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.945+0000 7fe72d7fa640 1 -- 172.21.15.76:0/1940995615 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe720014070 con 0x7fe7301011f0 2024-03-12T05:55:48.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.945+0000 7fe72f7fe640 1 --2- 172.21.15.76:0/1940995615 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe7180a1810 0x7fe7180a3cd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe730198740 tx=0x7fe724005f90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:55:48.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:48.948+0000 7fe72d7fa640 1 -- 172.21.15.76:0/1940995615 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe720096980 con 0x7fe7301011f0 2024-03-12T05:55:49.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:49.184+0000 7fe73665b640 1 -- 172.21.15.76:0/1940995615 --> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fe73002ceb0 con 0x7fe7180a1810 2024-03-12T05:55:49.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:49.191+0000 7fe72d7fa640 1 -- 172.21.15.76:0/1940995615 <== mgr.15465 v2:172.21.15.76:6800/513615067 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fe73002ceb0 con 0x7fe7180a1810 2024-03-12T05:55:49.194 INFO:teuthology.orchestra.run.smithi076.stdout:/ 1m 6m3h 2024-03-12T05:55:49.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:49.193+0000 7fe73665b640 1 -- 172.21.15.76:0/1940995615 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe7180a1810 msgr2=0x7fe7180a3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:49.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:49.193+0000 7fe73665b640 1 --2- 172.21.15.76:0/1940995615 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe7180a1810 0x7fe7180a3cd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe730198740 tx=0x7fe724005f90 comp rx=0 tx=0).stop 2024-03-12T05:55:49.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:49.193+0000 7fe73665b640 1 -- 172.21.15.76:0/1940995615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe7301011f0 msgr2=0x7fe730197230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:55:49.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:49.193+0000 7fe73665b640 1 --2- 172.21.15.76:0/1940995615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe7301011f0 0x7fe730197230 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe72000e9c0 tx=0x7fe72000ee80 comp rx=0 tx=0).stop 2024-03-12T05:55:49.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:49.193+0000 7fe73665b640 1 -- 172.21.15.76:0/1940995615 shutdown_connections 2024-03-12T05:55:49.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:49.193+0000 7fe73665b640 1 --2- 172.21.15.76:0/1940995615 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe7180a1810 0x7fe7180a3cd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:49.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:49.193+0000 7fe73665b640 1 --2- 172.21.15.76:0/1940995615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe730102da0 0x7fe73019e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:49.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:49.193+0000 7fe73665b640 1 --2- 172.21.15.76:0/1940995615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe7301023e0 0x7fe730197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:49.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:49.193+0000 7fe73665b640 1 --2- 172.21.15.76:0/1940995615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe7301011f0 0x7fe730197230 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:55:49.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:49.193+0000 7fe73665b640 1 -- 172.21.15.76:0/1940995615 >> 172.21.15.76:0/1940995615 conn(0x7fe7300fc760 msgr2=0x7fe7301056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:55:49.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:49.194+0000 7fe73665b640 1 -- 172.21.15.76:0/1940995615 shutdown_connections 2024-03-12T05:55:49.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:55:49.194+0000 7fe73665b640 1 -- 172.21.15.76:0/1940995615 wait complete. 2024-03-12T05:55:49.208 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:48 smithi076 ceph-mon[32095]: [12/Mar/2024:05:55:47] ENGINE Bus STARTING 2024-03-12T05:55:49.208 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:48 smithi076 ceph-mon[32095]: [12/Mar/2024:05:55:47] ENGINE Serving on https://172.21.15.76:7150 2024-03-12T05:55:49.208 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:48 smithi076 ceph-mon[32095]: [12/Mar/2024:05:55:47] ENGINE Serving on http://172.21.15.76:8765 2024-03-12T05:55:49.208 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:48 smithi076 ceph-mon[32095]: [12/Mar/2024:05:55:47] ENGINE Bus STARTED 2024-03-12T05:55:49.208 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:48 smithi076 ceph-mon[32095]: mgrmap e22: x(active, since 1.0187s), standbys: z, y 2024-03-12T05:55:49.208 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:48 smithi076 ceph-mon[32095]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:49.208 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:48 smithi076 ceph-mon[32095]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:55:49.209 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-12T05:55:49.237 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-12T05:55:49.248 INFO:teuthology.run_tasks:Running task workunit... 2024-03-12T05:55:49.280 INFO:tasks.workunit:Pulling workunits from ref b6a4997be2b8bb74411268cc42753f29ae9a78ad 2024-03-12T05:55:49.281 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-12T05:55:49.281 INFO:tasks.workunit:timeout=3h 2024-03-12T05:55:49.281 INFO:tasks.workunit:cleanup=True 2024-03-12T05:55:49.281 DEBUG:teuthology.orchestra.run.smithi076:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-12T05:55:49.298 INFO:teuthology.orchestra.run.smithi076.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-12T05:55:49.298 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-12T05:55:49.298 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-12T05:55:49.298 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T05:55:49.298 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-12T05:55:49.298 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-03-12 05:53:43.808747972 +0000 2024-03-12T05:55:49.298 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-03-12 05:53:43.808747972 +0000 2024-03-12T05:55:49.299 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-03-12 05:55:32.388249855 +0000 2024-03-12T05:55:49.299 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-03-12 05:53:43.808747972 +0000 2024-03-12T05:55:49.299 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-12T05:55:49.299 DEBUG:teuthology.orchestra.run.smithi076:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-12T05:55:49.375 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b6a4997be2b8bb74411268cc42753f29ae9a78ad 2024-03-12T05:55:49.431 INFO:tasks.workunit.client.0.smithi076.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-12T05:55:49.444 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:48 smithi082 ceph-mon[36382]: [12/Mar/2024:05:55:47] ENGINE Bus STARTING 2024-03-12T05:55:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:48 smithi082 ceph-mon[36382]: [12/Mar/2024:05:55:47] ENGINE Serving on https://172.21.15.76:7150 2024-03-12T05:55:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:48 smithi082 ceph-mon[36382]: [12/Mar/2024:05:55:47] ENGINE Serving on http://172.21.15.76:8765 2024-03-12T05:55:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:48 smithi082 ceph-mon[36382]: [12/Mar/2024:05:55:47] ENGINE Bus STARTED 2024-03-12T05:55:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:48 smithi082 ceph-mon[36382]: mgrmap e22: x(active, since 1.0187s), standbys: z, y 2024-03-12T05:55:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:48 smithi082 ceph-mon[36382]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:48 smithi082 ceph-mon[36382]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:55:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:48 smithi088 ceph-mon[36585]: [12/Mar/2024:05:55:47] ENGINE Bus STARTING 2024-03-12T05:55:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:48 smithi088 ceph-mon[36585]: [12/Mar/2024:05:55:47] ENGINE Serving on https://172.21.15.76:7150 2024-03-12T05:55:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:48 smithi088 ceph-mon[36585]: [12/Mar/2024:05:55:47] ENGINE Serving on http://172.21.15.76:8765 2024-03-12T05:55:49.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:48 smithi088 ceph-mon[36585]: [12/Mar/2024:05:55:47] ENGINE Bus STARTED 2024-03-12T05:55:49.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:48 smithi088 ceph-mon[36585]: mgrmap e22: x(active, since 1.0187s), standbys: z, y 2024-03-12T05:55:49.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:48 smithi088 ceph-mon[36585]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:49.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:48 smithi088 ceph-mon[36585]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:55:50.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:50 smithi082 ceph-mon[36382]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:50.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:50 smithi082 ceph-mon[36382]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:50.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:50 smithi082 ceph-mon[36382]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:55:50.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:50 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:50 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:50 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:50 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:50 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:50 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:50 smithi088 ceph-mon[36585]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:50 smithi088 ceph-mon[36585]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:50 smithi088 ceph-mon[36585]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:55:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:50 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:50 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:50 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:50 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:50 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:50 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:50 smithi076 ceph-mon[32095]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:50.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:50 smithi076 ceph-mon[32095]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:50.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:50 smithi076 ceph-mon[32095]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:55:50.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:50 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:50 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:50 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:50 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:50 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:50.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:50 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:51.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: mgrmap e23: x(active, since 3s), standbys: z, y 2024-03-12T05:55:51.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:55:51.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:51 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:55:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:51.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: mgrmap e23: x(active, since 3s), standbys: z, y 2024-03-12T05:55:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:55:51.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:51 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:55:51.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T05:55:51.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: mgrmap e23: x(active, since 3s), standbys: z, y 2024-03-12T05:55:51.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:51.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T05:55:51.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:55:51.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:51 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:55:52.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi082 to 174.1M 2024-03-12T05:55:52.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Unable to set osd_memory_target on smithi082 to 182611251: error parsing value: Value '182611251' is below minimum 939524096 2024-03-12T05:55:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi088 to 174.1M 2024-03-12T05:55:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Unable to set osd_memory_target on smithi088 to 182634188: error parsing value: Value '182634188' is below minimum 939524096 2024-03-12T05:55:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-03-12T05:55:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Unable to set osd_memory_target on smithi076 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-03-12T05:55:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Updating smithi076:/etc/ceph/ceph.conf 2024-03-12T05:55:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Updating smithi082:/etc/ceph/ceph.conf 2024-03-12T05:55:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Updating smithi088:/etc/ceph/ceph.conf 2024-03-12T05:55:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Updating smithi082:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:55:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Updating smithi088:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:55:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Updating smithi076:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:55:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:55:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:55:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:55:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:55:52.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Updating smithi082:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:55:52.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Updating smithi088:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:55:52.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: Updating smithi076:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:55:52.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-03-12T05:55:52.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:55:52.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:55:52.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:55:52.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:52 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi082 to 174.1M 2024-03-12T05:55:52.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Unable to set osd_memory_target on smithi082 to 182611251: error parsing value: Value '182611251' is below minimum 939524096 2024-03-12T05:55:52.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi088 to 174.1M 2024-03-12T05:55:52.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Unable to set osd_memory_target on smithi088 to 182634188: error parsing value: Value '182634188' is below minimum 939524096 2024-03-12T05:55:52.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-03-12T05:55:52.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Unable to set osd_memory_target on smithi076 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-03-12T05:55:52.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Updating smithi076:/etc/ceph/ceph.conf 2024-03-12T05:55:52.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Updating smithi082:/etc/ceph/ceph.conf 2024-03-12T05:55:52.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Updating smithi088:/etc/ceph/ceph.conf 2024-03-12T05:55:52.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Updating smithi082:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:55:52.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Updating smithi088:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Updating smithi076:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Updating smithi082:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Updating smithi088:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: Updating smithi076:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:55:52.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:55:52.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:55:52.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:52 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi082 to 174.1M 2024-03-12T05:55:52.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Unable to set osd_memory_target on smithi082 to 182611251: error parsing value: Value '182611251' is below minimum 939524096 2024-03-12T05:55:52.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi088 to 174.1M 2024-03-12T05:55:52.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Unable to set osd_memory_target on smithi088 to 182634188: error parsing value: Value '182634188' is below minimum 939524096 2024-03-12T05:55:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-03-12T05:55:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Unable to set osd_memory_target on smithi076 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-03-12T05:55:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Updating smithi076:/etc/ceph/ceph.conf 2024-03-12T05:55:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Updating smithi082:/etc/ceph/ceph.conf 2024-03-12T05:55:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Updating smithi088:/etc/ceph/ceph.conf 2024-03-12T05:55:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Updating smithi082:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:55:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Updating smithi088:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:55:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Updating smithi076:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T05:55:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:55:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:55:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-03-12T05:55:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-12T05:55:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Updating smithi082:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:55:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Updating smithi088:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:55:52.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: Updating smithi076:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T05:55:52.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-03-12T05:55:52.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:52.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:55:52.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:55:52.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:55:52.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:52 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:55:54.287 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:54 smithi088 ceph-mon[36585]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 66 op/s 2024-03-12T05:55:54.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:54 smithi082 ceph-mon[36382]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 66 op/s 2024-03-12T05:55:54.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:54 smithi076 ceph-mon[32095]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 66 op/s 2024-03-12T05:55:56.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:56 smithi082 ceph-mon[36382]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-12T05:55:56.475 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:56 smithi088 ceph-mon[36585]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-12T05:55:56.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:56 smithi076 ceph-mon[32095]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-12T05:55:58.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:55:58 smithi082 ceph-mon[36382]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-12T05:55:58.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:55:58 smithi088 ceph-mon[36585]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-12T05:55:58.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:55:58 smithi076 ceph-mon[32095]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-12T05:56:00.440 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:00 smithi082 ceph-mon[36382]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-12T05:56:00.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:00 smithi076 ceph-mon[32095]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-12T05:56:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:00 smithi088 ceph-mon[36585]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-12T05:56:00.738 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:56:00 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:56:00.471+0000 7fc9bf277700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:56:00.738 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:56:00 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:56:00.471+0000 7fc9bf277700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:56:00.739 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:56:00 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:56:00.471+0000 7fc9bf277700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:56:00.739 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:56:00 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:56:00.471+0000 7fc9bf277700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:56:00.739 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 05:56:00 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x[32328]: 2024-03-12T05:56:00.471+0000 7fc9bf277700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-12T05:56:02.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:02 smithi082 ceph-mon[36382]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-12T05:56:02.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:02 smithi088 ceph-mon[36585]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-12T05:56:02.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:02 smithi076 ceph-mon[32095]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-12T05:56:03.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:03 smithi082 ceph-mon[36382]: mgrmap e25: x(active, since 15s), standbys: z, y 2024-03-12T05:56:03.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:03 smithi088 ceph-mon[36585]: mgrmap e25: x(active, since 15s), standbys: z, y 2024-03-12T05:56:03.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:03 smithi076 ceph-mon[32095]: mgrmap e25: x(active, since 15s), standbys: z, y 2024-03-12T05:56:04.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:04 smithi082 ceph-mon[36382]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 88 KiB/s wr, 60 op/s 2024-03-12T05:56:04.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:04 smithi088 ceph-mon[36585]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 88 KiB/s wr, 60 op/s 2024-03-12T05:56:04.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:04 smithi076 ceph-mon[32095]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 88 KiB/s wr, 60 op/s 2024-03-12T05:56:06.234 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:06 smithi088 ceph-mon[36585]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 62 KiB/s wr, 37 op/s 2024-03-12T05:56:06.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:06 smithi082 ceph-mon[36382]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 62 KiB/s wr, 37 op/s 2024-03-12T05:56:06.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:06 smithi076 ceph-mon[32095]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 62 KiB/s wr, 37 op/s 2024-03-12T05:56:08.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:08 smithi082 ceph-mon[36382]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-12T05:56:08.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:08 smithi088 ceph-mon[36585]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-12T05:56:08.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:08 smithi076 ceph-mon[32095]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-12T05:56:09.991 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:56:09.992 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:56:10.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.105+0000 7f58485d0640 1 -- 172.21.15.76:0/2771247807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58400ffcb0 msgr2=0x7f5840100090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.105+0000 7f58485d0640 1 --2- 172.21.15.76:0/2771247807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58400ffcb0 0x7f5840100090 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f583c009930 tx=0x7f583c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:56:10.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.105+0000 7f58485d0640 1 -- 172.21.15.76:0/2771247807 shutdown_connections 2024-03-12T05:56:10.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.105+0000 7f58485d0640 1 --2- 172.21.15.76:0/2771247807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5840106530 0x7f584010e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.105+0000 7f58485d0640 1 --2- 172.21.15.76:0/2771247807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5840100660 0x7f5840105e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.105+0000 7f58485d0640 1 --2- 172.21.15.76:0/2771247807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58400ffcb0 0x7f5840100090 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.105+0000 7f58485d0640 1 -- 172.21.15.76:0/2771247807 >> 172.21.15.76:0/2771247807 conn(0x7f5840075340 msgr2=0x7f5840075750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.105+0000 7f58485d0640 1 -- 172.21.15.76:0/2771247807 shutdown_connections 2024-03-12T05:56:10.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.105+0000 7f58485d0640 1 -- 172.21.15.76:0/2771247807 wait complete. 2024-03-12T05:56:10.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f58485d0640 1 Processor -- start 2024-03-12T05:56:10.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f58485d0640 1 -- start start 2024-03-12T05:56:10.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f58485d0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58400ffcb0 0x7f584019d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f58485d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5840106530 0x7f584019dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f58485d0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5840197660 0x7f5840197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f58485d0640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5840198130 con 0x7f5840197660 2024-03-12T05:56:10.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f58485d0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58401982a0 con 0x7f58400ffcb0 2024-03-12T05:56:10.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f58485d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5840198460 con 0x7f5840106530 2024-03-12T05:56:10.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f5846b6d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5840197660 0x7f5840197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f5846b6d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5840197660 0x7f5840197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48112/0 (socket says 172.21.15.76:48112) 2024-03-12T05:56:10.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f5846b6d640 1 -- 172.21.15.76:0/3868212920 learned_addr learned my addr 172.21.15.76:0/3868212920 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f584636c640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58400ffcb0 0x7f584019d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f5845b6b640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5840106530 0x7f584019dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f5846b6d640 1 -- 172.21.15.76:0/3868212920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5840106530 msgr2=0x7f584019dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f5846b6d640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5840106530 0x7f584019dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f5846b6d640 1 -- 172.21.15.76:0/3868212920 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58400ffcb0 msgr2=0x7f584019d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f5846b6d640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58400ffcb0 0x7f584019d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f5846b6d640 1 -- 172.21.15.76:0/3868212920 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c009560 con 0x7f5840197660 2024-03-12T05:56:10.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.106+0000 7f584636c640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58400ffcb0 0x7f584019d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:10.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.107+0000 7f5846b6d640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5840197660 0x7f5840197ae0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f583800b970 tx=0x7f583800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.107+0000 7f58377fe640 1 -- 172.21.15.76:0/3868212920 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583800cb40 con 0x7f5840197660 2024-03-12T05:56:10.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.107+0000 7f58377fe640 1 -- 172.21.15.76:0/3868212920 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f583800cca0 con 0x7f5840197660 2024-03-12T05:56:10.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.107+0000 7f58377fe640 1 -- 172.21.15.76:0/3868212920 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58380077d0 con 0x7f5840197660 2024-03-12T05:56:10.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.107+0000 7f58485d0640 1 -- 172.21.15.76:0/3868212920 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58401a4400 con 0x7f5840197660 2024-03-12T05:56:10.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.107+0000 7f58485d0640 1 -- 172.21.15.76:0/3868212920 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58401a4960 con 0x7f5840197660 2024-03-12T05:56:10.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.109+0000 7f58377fe640 1 -- 172.21.15.76:0/3868212920 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f5838014440 con 0x7f5840197660 2024-03-12T05:56:10.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f584b1a5740 1 Processor -- start 2024-03-12T05:56:10.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f584b1a5740 1 -- start start 2024-03-12T05:56:10.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f584b1a5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb43690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f584b1a5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 0x55d54bb4c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f584b1a5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb4c7b0 0x55d54bb4eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f58377fe640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f582c0a1e10 0x7f582c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f584b1a5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54bab2280 con 0x55d54bb432b0 2024-03-12T05:56:10.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f584b1a5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54baaf7b0 con 0x55d54bb4c7b0 2024-03-12T05:56:10.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f584b1a5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54baaaf90 con 0x55d54bb43bd0 2024-03-12T05:56:10.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f5846b6d640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f582c0a1e10 0x7f582c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f5845b6b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb43690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f5845b6b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb43690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48116/0 (socket says 172.21.15.76:48116) 2024-03-12T05:56:10.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f5845b6b640 1 -- 172.21.15.76:0/1680899167 learned_addr learned my addr 172.21.15.76:0/1680899167 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f584636c640 1 --2- 172.21.15.76:0/1680899167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 0x55d54bb4c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f58377fe640 1 -- 172.21.15.76:0/3868212920 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f58380cd190 con 0x7f5840197660 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.110+0000 7f5845b6b640 1 --2- 172.21.15.76:0/1680899167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb4c7b0 0x55d54bb4eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.111+0000 7f5845b6b640 1 -- 172.21.15.76:0/1680899167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 msgr2=0x55d54bb4c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.111+0000 7f5845b6b640 1 --2- 172.21.15.76:0/1680899167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 0x55d54bb4c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.111+0000 7f5845b6b640 1 -- 172.21.15.76:0/1680899167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb4c7b0 msgr2=0x55d54bb4eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.111+0000 7f5845b6b640 1 --2- 172.21.15.76:0/1680899167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb4c7b0 0x55d54bb4eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.111+0000 7f5845b6b640 1 -- 172.21.15.76:0/1680899167 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d54ba69570 con 0x55d54bb432b0 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.111+0000 7f5846b6d640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f582c0a1e10 0x7f582c0a42d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f58380ca900 tx=0x7f5838004b50 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.111+0000 7f5845b6b640 1 --2- 172.21.15.76:0/1680899167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb43690 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5830013500 tx=0x7f583000b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e16900d663cb7935 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.111+0000 7f5834bfe640 1 -- 172.21.15.76:0/1680899167 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583001e410 con 0x55d54bb432b0 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.111+0000 7f5834bfe640 1 -- 172.21.15.76:0/1680899167 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5830011920 con 0x55d54bb432b0 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.111+0000 7f5834bfe640 1 -- 172.21.15.76:0/1680899167 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583001d670 con 0x55d54bb432b0 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 -- 172.21.15.76:0/1680899167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 msgr2=0x55d54bb43690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 --2- 172.21.15.76:0/1680899167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb43690 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5830013500 tx=0x7f583000b5e0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 -- 172.21.15.76:0/1680899167 shutdown_connections 2024-03-12T05:56:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 --2- 172.21.15.76:0/1680899167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb4c7b0 0x55d54bb4eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 --2- 172.21.15.76:0/1680899167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 0x55d54bb4c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 --2- 172.21.15.76:0/1680899167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb43690 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 -- 172.21.15.76:0/1680899167 >> 172.21.15.76:0/1680899167 conn(0x55d54bb3d990 msgr2=0x55d54bb3dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 -- 172.21.15.76:0/1680899167 shutdown_connections 2024-03-12T05:56:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 -- 172.21.15.76:0/1680899167 wait complete. 2024-03-12T05:56:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 -- start start 2024-03-12T05:56:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 Processor -- start 2024-03-12T05:56:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb4c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 0x55d54bb49530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb4c7b0 0x55d54bb49b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54b77e430 con 0x55d54bb432b0 2024-03-12T05:56:10.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54b73ae00 con 0x55d54bb4c7b0 2024-03-12T05:56:10.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54ba2ffd0 con 0x55d54bb43bd0 2024-03-12T05:56:10.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.112+0000 7f584b1a5740 10 client.0 _ll_register_callbacks cb 0x55d54ba85fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:10.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f5845b6b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb4c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f5845b6b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb4c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48124/0 (socket says 172.21.15.76:48124) 2024-03-12T05:56:10.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f5845b6b640 1 -- 172.21.15.76:0/3532680172 learned_addr learned my addr 172.21.15.76:0/3532680172 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f584636c640 1 --2- 172.21.15.76:0/3532680172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 0x55d54bb49530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f5845b6b640 1 -- 172.21.15.76:0/3532680172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 msgr2=0x55d54bb49530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f5845b6b640 1 --2- 172.21.15.76:0/3532680172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 0x55d54bb49530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f5845b6b640 1 -- 172.21.15.76:0/3532680172 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb4c7b0 msgr2=0x55d54bb49b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f5845b6b640 1 --2- 172.21.15.76:0/3532680172 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb4c7b0 0x55d54bb49b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f5845b6b640 1 -- 172.21.15.76:0/3532680172 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583000bd10 con 0x55d54bb432b0 2024-03-12T05:56:10.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f5845b6b640 1 --2- 172.21.15.76:0/3532680172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb4c130 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5830005890 tx=0x7f5830011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b195f54c91e1d0f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f5829ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:10.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f5829ffb640 1 -- 172.21.15.76:0/3532680172 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830005040 con 0x55d54bb432b0 2024-03-12T05:56:10.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f5829ffb640 1 -- 172.21.15.76:0/3532680172 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f58300051a0 con 0x55d54bb432b0 2024-03-12T05:56:10.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f584b1a5740 1 -- 172.21.15.76:0/3532680172 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d54b7aa7e0 con 0x55d54bb432b0 2024-03-12T05:56:10.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f5829ffb640 1 -- 172.21.15.76:0/3532680172 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5830005360 con 0x55d54bb432b0 2024-03-12T05:56:10.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f584b1a5740 10 client.15564 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:10.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f584b1a5740 1 -- 172.21.15.76:0/3532680172 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55d54baa9610 con 0x55d54bb432b0 2024-03-12T05:56:10.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.113+0000 7f5829ffb640 1 -- 172.21.15.76:0/3532680172 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d54b7aa7e0 con 0x55d54bb432b0 2024-03-12T05:56:10.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.114+0000 7f5829ffb640 1 -- 172.21.15.76:0/3532680172 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f583001d4e0 con 0x55d54bb432b0 2024-03-12T05:56:10.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.114+0000 7f584b1a5740 10 client.15564 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:10.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.114+0000 7f584b1a5740 10 client.15564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:10.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.114+0000 7f584b1a5740 20 client.15564 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:10.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.114+0000 7f584b1a5740 1 --2- 172.21.15.76:0/3532680172 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d54bb5c800 0x55d54bb5ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.114+0000 7f584b1a5740 4 client.15564 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.114+0000 7f584b1a5740 1 -- 172.21.15.76:0/3532680172 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d54b7e7720 con 0x55d54bb5c800 2024-03-12T05:56:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.114+0000 7f584636c640 1 --2- 172.21.15.76:0/3532680172 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d54bb5c800 0x55d54bb5ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.114+0000 7f584636c640 1 --2- 172.21.15.76:0/3532680172 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d54bb5c800 0x55d54bb5ec20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebd82b184a8d80fd server_cookie=f1322471e128a87f in_seq=0 out_seq=0 2024-03-12T05:56:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.114+0000 7f5829ffb640 10 client.15564 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f5829ffb640 1 -- 172.21.15.76:0/3532680172 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d54b7e7720 con 0x55d54bb5c800 2024-03-12T05:56:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f5829ffb640 10 client.15564 handle_command_reply: tid=0 2024-03-12T05:56:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f584b1a5740 1 client.15564 shutdown 2024-03-12T05:56:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f584b1a5740 1 -- 172.21.15.76:0/3532680172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 msgr2=0x55d54bb4c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f584b1a5740 1 --2- 172.21.15.76:0/3532680172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb4c130 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5830005890 tx=0x7f5830011a90 comp rx=0 tx=0).stop 2024-03-12T05:56:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f584b1a5740 1 -- 172.21.15.76:0/3532680172 shutdown_connections 2024-03-12T05:56:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f584b1a5740 1 --2- 172.21.15.76:0/3532680172 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb4c7b0 0x55d54bb49b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f584b1a5740 1 --2- 172.21.15.76:0/3532680172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 0x55d54bb49530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f584b1a5740 1 --2- 172.21.15.76:0/3532680172 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d54bb5c800 0x55d54bb5ec20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f584b1a5740 1 --2- 172.21.15.76:0/3532680172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb4c130 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f5829ffb640 0 client.15564 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:10.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f584b1a5740 1 -- 172.21.15.76:0/3532680172 >> 172.21.15.76:0/3532680172 conn(0x55d54bb3d990 msgr2=0x55d54bb3f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f584b1a5740 1 -- 172.21.15.76:0/3532680172 shutdown_connections 2024-03-12T05:56:10.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f584b1a5740 1 -- 172.21.15.76:0/3532680172 wait complete. 2024-03-12T05:56:10.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.115+0000 7f584b1a5740 20 client.15564 trim_cache size 0 max 16384 2024-03-12T05:56:10.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.130+0000 7f584b1a5740 1 Processor -- start 2024-03-12T05:56:10.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.130+0000 7f584b1a5740 1 -- start start 2024-03-12T05:56:10.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.130+0000 7f584b1a5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb432b0 0x55d54bb494f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.130+0000 7f584b1a5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb43bd0 0x55d54bb4bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.130+0000 7f584b1a5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb4c7b0 0x55d54bb447b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.130+0000 7f584b1a5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54bb42f10 con 0x55d54bb4c7b0 2024-03-12T05:56:10.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.130+0000 7f584b1a5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54bb442e0 con 0x55d54bb43bd0 2024-03-12T05:56:10.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.130+0000 7f584b1a5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54bb49300 con 0x55d54bb432b0 2024-03-12T05:56:10.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.130+0000 7f5845b6b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb4c7b0 0x55d54bb447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.130+0000 7f5845b6b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb4c7b0 0x55d54bb447b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48128/0 (socket says 172.21.15.76:48128) 2024-03-12T05:56:10.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.130+0000 7f5845b6b640 1 -- 172.21.15.76:0/2969484693 learned_addr learned my addr 172.21.15.76:0/2969484693 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f5845b6b640 1 -- 172.21.15.76:0/2969484693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb432b0 msgr2=0x55d54bb494f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f5845b6b640 1 --2- 172.21.15.76:0/2969484693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb432b0 0x55d54bb494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f5845b6b640 1 -- 172.21.15.76:0/2969484693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb43bd0 msgr2=0x55d54bb4bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f5845b6b640 1 --2- 172.21.15.76:0/2969484693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb43bd0 0x55d54bb4bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f5845b6b640 1 -- 172.21.15.76:0/2969484693 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d54ba45ac0 con 0x55d54bb4c7b0 2024-03-12T05:56:10.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f5845b6b640 1 --2- 172.21.15.76:0/2969484693 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb4c7b0 0x55d54bb447b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f58300118f0 tx=0x7f583000b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d3841b61bfb078 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f582bfff640 1 -- 172.21.15.76:0/2969484693 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58300114f0 con 0x55d54bb4c7b0 2024-03-12T05:56:10.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f582bfff640 1 -- 172.21.15.76:0/2969484693 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5830025070 con 0x55d54bb4c7b0 2024-03-12T05:56:10.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f582bfff640 1 -- 172.21.15.76:0/2969484693 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583001d3c0 con 0x55d54bb4c7b0 2024-03-12T05:56:10.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f584b1a5740 1 -- 172.21.15.76:0/2969484693 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb4c7b0 msgr2=0x55d54bb447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f584b1a5740 1 --2- 172.21.15.76:0/2969484693 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb4c7b0 0x55d54bb447b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f58300118f0 tx=0x7f583000b5e0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f584b1a5740 1 -- 172.21.15.76:0/2969484693 shutdown_connections 2024-03-12T05:56:10.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f584b1a5740 1 --2- 172.21.15.76:0/2969484693 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb4c7b0 0x55d54bb447b0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f584b1a5740 1 --2- 172.21.15.76:0/2969484693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb43bd0 0x55d54bb4bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f584b1a5740 1 --2- 172.21.15.76:0/2969484693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb432b0 0x55d54bb494f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.131+0000 7f584b1a5740 1 -- 172.21.15.76:0/2969484693 >> 172.21.15.76:0/2969484693 conn(0x55d54bb3d990 msgr2=0x55d54bb63580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f584b1a5740 1 -- 172.21.15.76:0/2969484693 shutdown_connections 2024-03-12T05:56:10.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f584b1a5740 1 -- 172.21.15.76:0/2969484693 wait complete. 2024-03-12T05:56:10.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f584b1a5740 1 -- start start 2024-03-12T05:56:10.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f584b1a5740 1 Processor -- start 2024-03-12T05:56:10.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f584b1a5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f584b1a5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 0x55d54bb4fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f584b1a5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb4c7b0 0x55d54bb501f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f584b1a5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54bb507b0 con 0x55d54bb432b0 2024-03-12T05:56:10.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f584b1a5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54bb50920 con 0x55d54bb4c7b0 2024-03-12T05:56:10.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f584b1a5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54bb50ae0 con 0x55d54bb43bd0 2024-03-12T05:56:10.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f584b1a5740 10 client.0 _ll_register_callbacks cb 0x7f583000b490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:10.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f5845b6b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f5845b6b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48132/0 (socket says 172.21.15.76:48132) 2024-03-12T05:56:10.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f5845b6b640 1 -- 172.21.15.76:0/252971806 learned_addr learned my addr 172.21.15.76:0/252971806 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f5845b6b640 1 -- 172.21.15.76:0/252971806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 msgr2=0x55d54bb4fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f584636c640 1 --2- 172.21.15.76:0/252971806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 0x55d54bb4fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f5845b6b640 1 --2- 172.21.15.76:0/252971806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 0x55d54bb4fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f5845b6b640 1 -- 172.21.15.76:0/252971806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb4c7b0 msgr2=0x55d54bb501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f5845b6b640 1 --2- 172.21.15.76:0/252971806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb4c7b0 0x55d54bb501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.132+0000 7f5845b6b640 1 -- 172.21.15.76:0/252971806 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d54baa9610 con 0x55d54bb432b0 2024-03-12T05:56:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f5845b6b640 1 --2- 172.21.15.76:0/252971806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb538e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5830000c00 tx=0x7f5830005c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55825cbc6d5bc4c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f5829ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f5829ffb640 1 -- 172.21.15.76:0/252971806 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830011a90 con 0x55d54bb432b0 2024-03-12T05:56:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f5829ffb640 1 -- 172.21.15.76:0/252971806 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5830000e40 con 0x55d54bb432b0 2024-03-12T05:56:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f584b1a5740 1 -- 172.21.15.76:0/252971806 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5830011920 con 0x55d54bb432b0 2024-03-12T05:56:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f5829ffb640 1 -- 172.21.15.76:0/252971806 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f583001de20 con 0x55d54bb432b0 2024-03-12T05:56:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f584b1a5740 10 client.15570 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f584b1a5740 1 -- 172.21.15.76:0/252971806 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55d54ba6b6a0 con 0x55d54bb432b0 2024-03-12T05:56:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f5829ffb640 1 -- 172.21.15.76:0/252971806 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5830011920 con 0x55d54bb432b0 2024-03-12T05:56:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f5829ffb640 1 -- 172.21.15.76:0/252971806 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f583003c930 con 0x55d54bb432b0 2024-03-12T05:56:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f584b1a5740 10 client.15570 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f584b1a5740 10 client.15570 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f584b1a5740 20 client.15570 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:10.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f584b1a5740 1 --2- 172.21.15.76:0/252971806 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d54bbb3e50 0x55d54bbb6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f584b1a5740 4 client.15570 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.133+0000 7f584b1a5740 1 -- 172.21.15.76:0/252971806 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d54b836890 con 0x55d54bbb3e50 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f584636c640 1 --2- 172.21.15.76:0/252971806 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d54bbb3e50 0x55d54bbb6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f5829ffb640 10 client.15570 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f584636c640 1 --2- 172.21.15.76:0/252971806 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d54bbb3e50 0x55d54bbb6240 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b7f6bdf67094d6f server_cookie=3bad6a097d6443c2 in_seq=0 out_seq=0 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f5829ffb640 1 -- 172.21.15.76:0/252971806 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d54b836890 con 0x55d54bbb3e50 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f5829ffb640 10 client.15570 handle_command_reply: tid=0 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f584b1a5740 1 client.15570 shutdown 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f584b1a5740 1 -- 172.21.15.76:0/252971806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 msgr2=0x55d54bb538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f584b1a5740 1 --2- 172.21.15.76:0/252971806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb538e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5830000c00 tx=0x7f5830005c20 comp rx=0 tx=0).stop 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f584b1a5740 1 -- 172.21.15.76:0/252971806 shutdown_connections 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f584b1a5740 1 --2- 172.21.15.76:0/252971806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d54bb4c7b0 0x55d54bb501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f584b1a5740 1 --2- 172.21.15.76:0/252971806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d54bb43bd0 0x55d54bb4fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f584b1a5740 1 --2- 172.21.15.76:0/252971806 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d54bbb3e50 0x55d54bbb6240 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f584b1a5740 1 --2- 172.21.15.76:0/252971806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d54bb432b0 0x55d54bb538e0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f584b1a5740 1 -- 172.21.15.76:0/252971806 >> 172.21.15.76:0/252971806 conn(0x55d54bb3d990 msgr2=0x55d54bb447b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f5829ffb640 0 client.15570 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:10.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f584b1a5740 1 -- 172.21.15.76:0/252971806 shutdown_connections 2024-03-12T05:56:10.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f584b1a5740 1 -- 172.21.15.76:0/252971806 wait complete. 2024-03-12T05:56:10.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.134+0000 7f584b1a5740 20 client.15570 trim_cache size 0 max 16384 2024-03-12T05:56:10.145 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:56:10.145 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T05:56:10.145 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T05:56:10.145 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:56:10.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.135+0000 7f58485d0640 1 -- 172.21.15.76:0/3868212920 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f582c0a1e10 msgr2=0x7f582c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.135+0000 7f58485d0640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f582c0a1e10 0x7f582c0a42d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f58380ca900 tx=0x7f5838004b50 comp rx=0 tx=0).stop 2024-03-12T05:56:10.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.135+0000 7f58485d0640 1 -- 172.21.15.76:0/3868212920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5840197660 msgr2=0x7f5840197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.135+0000 7f58485d0640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5840197660 0x7f5840197ae0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f583800b970 tx=0x7f583800be30 comp rx=0 tx=0).stop 2024-03-12T05:56:10.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.135+0000 7f58485d0640 1 -- 172.21.15.76:0/3868212920 shutdown_connections 2024-03-12T05:56:10.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.135+0000 7f58485d0640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f582c0a1e10 0x7f582c0a42d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.135+0000 7f58485d0640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5840197660 0x7f5840197ae0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.135+0000 7f58485d0640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5840106530 0x7f584019dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.135+0000 7f58485d0640 1 --2- 172.21.15.76:0/3868212920 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58400ffcb0 0x7f584019d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.135+0000 7f58485d0640 1 -- 172.21.15.76:0/3868212920 >> 172.21.15.76:0/3868212920 conn(0x7f5840075340 msgr2=0x7f58400fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.135+0000 7f58485d0640 1 -- 172.21.15.76:0/3868212920 shutdown_connections 2024-03-12T05:56:10.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.135+0000 7f58485d0640 1 -- 172.21.15.76:0/3868212920 wait complete. 2024-03-12T05:56:10.149 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:56:10.149 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:56:10.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:10 smithi082 ceph-mon[36382]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-12T05:56:10.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.250+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/792854959 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e18100c50 msgr2=0x7f9e18104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.250+0000 7f9e1ec0e640 1 --2- 172.21.15.76:0/792854959 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e18100c50 0x7f9e18104c90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009930 tx=0x7f9e0402bc30 comp rx=0 tx=0).stop 2024-03-12T05:56:10.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.250+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/792854959 shutdown_connections 2024-03-12T05:56:10.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.250+0000 7f9e1ec0e640 1 --2- 172.21.15.76:0/792854959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e18105370 0x7f9e1810cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.250+0000 7f9e1ec0e640 1 --2- 172.21.15.76:0/792854959 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e18100c50 0x7f9e18104c90 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.250+0000 7f9e1ec0e640 1 --2- 172.21.15.76:0/792854959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e181002a0 0x7f9e18100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.250+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/792854959 >> 172.21.15.76:0/792854959 conn(0x7f9e180772c0 msgr2=0x7f9e180776d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.250+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/792854959 shutdown_connections 2024-03-12T05:56:10.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.250+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/792854959 wait complete. 2024-03-12T05:56:10.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.250+0000 7f9e1ec0e640 1 Processor -- start 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e1ec0e640 1 -- start start 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e1ec0e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e181002a0 0x7f9e1819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e1ec0e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e18105370 0x7f9e1819d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e1ec0e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e18197650 0x7f9e18197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e1ec0e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e18198040 con 0x7f9e18105370 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e1ec0e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e181981b0 con 0x7f9e18197650 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e1ec0e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e18198370 con 0x7f9e181002a0 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e17fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e18105370 0x7f9e1819d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e17fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e18105370 0x7f9e1819d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48142/0 (socket says 172.21.15.76:48142) 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e17fff640 1 -- 172.21.15.76:0/2963663923 learned_addr learned my addr 172.21.15.76:0/2963663923 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e1d1ab640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e18197650 0x7f9e18197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e1c9aa640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e181002a0 0x7f9e1819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e17fff640 1 -- 172.21.15.76:0/2963663923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e181002a0 msgr2=0x7f9e1819d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e17fff640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e181002a0 0x7f9e1819d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e17fff640 1 -- 172.21.15.76:0/2963663923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e18197650 msgr2=0x7f9e18197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e17fff640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e18197650 0x7f9e18197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e17fff640 1 -- 172.21.15.76:0/2963663923 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e04009560 con 0x7f9e18105370 2024-03-12T05:56:10.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e1c9aa640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e181002a0 0x7f9e1819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:10.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e17fff640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e18105370 0x7f9e1819d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009230 tx=0x7f9e04004c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e15ffb640 1 -- 172.21.15.76:0/2963663923 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e040043d0 con 0x7f9e18105370 2024-03-12T05:56:10.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.251+0000 7f9e15ffb640 1 -- 172.21.15.76:0/2963663923 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e040052d0 con 0x7f9e18105370 2024-03-12T05:56:10.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.252+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/2963663923 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e181a4310 con 0x7f9e18105370 2024-03-12T05:56:10.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.252+0000 7f9e15ffb640 1 -- 172.21.15.76:0/2963663923 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04003d80 con 0x7f9e18105370 2024-03-12T05:56:10.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.252+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/2963663923 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e181a47c0 con 0x7f9e18105370 2024-03-12T05:56:10.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.253+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/2963663923 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e18101770 con 0x7f9e18105370 2024-03-12T05:56:10.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.254+0000 7f9e15ffb640 1 -- 172.21.15.76:0/2963663923 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f9e04005440 con 0x7f9e18105370 2024-03-12T05:56:10.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.254+0000 7f9e15ffb640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9e080a1cc0 0x7f9e080a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.254+0000 7f9e15ffb640 1 -- 172.21.15.76:0/2963663923 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9e040f0200 con 0x7f9e18105370 2024-03-12T05:56:10.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.254+0000 7f9e1c9aa640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9e080a1cc0 0x7f9e080a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.255+0000 7f9e1c9aa640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9e080a1cc0 0x7f9e080a4180 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c009c20 tx=0x7f9e0c005f90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.258+0000 7f9e15ffb640 1 -- 172.21.15.76:0/2963663923 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9e040b8fc0 con 0x7f9e18105370 2024-03-12T05:56:10.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.467+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/2963663923 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e18108660 con 0x7f9e18105370 2024-03-12T05:56:10.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.468+0000 7f9e15ffb640 1 -- 172.21.15.76:0/2963663923 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11328 (secure 0 0 0) 0x7f9e04043090 con 0x7f9e18105370 2024-03-12T05:56:10.472 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:56:10.472 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:55:33.056004+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:56:10.472 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 22 2024-03-12T05:56:10.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.470+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/2963663923 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9e080a1cc0 msgr2=0x7f9e080a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.470+0000 7f9e1ec0e640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9e080a1cc0 0x7f9e080a4180 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c009c20 tx=0x7f9e0c005f90 comp rx=0 tx=0).stop 2024-03-12T05:56:10.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.470+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/2963663923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e18105370 msgr2=0x7f9e1819d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.470+0000 7f9e1ec0e640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e18105370 0x7f9e1819d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009230 tx=0x7f9e04004c00 comp rx=0 tx=0).stop 2024-03-12T05:56:10.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.470+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/2963663923 shutdown_connections 2024-03-12T05:56:10.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.470+0000 7f9e1ec0e640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9e080a1cc0 0x7f9e080a4180 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.470+0000 7f9e1ec0e640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e18197650 0x7f9e18197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.470+0000 7f9e1ec0e640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e18105370 0x7f9e1819d990 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.470+0000 7f9e1ec0e640 1 --2- 172.21.15.76:0/2963663923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e181002a0 0x7f9e1819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.470+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/2963663923 >> 172.21.15.76:0/2963663923 conn(0x7f9e180772c0 msgr2=0x7f9e180fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.470+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/2963663923 shutdown_connections 2024-03-12T05:56:10.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.470+0000 7f9e1ec0e640 1 -- 172.21.15.76:0/2963663923 wait complete. 2024-03-12T05:56:10.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:10 smithi076 ceph-mon[32095]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-12T05:56:10.485 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:56:10.485 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:56:10.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:10 smithi088 ceph-mon[36585]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-12T05:56:10.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.585+0000 7f3a7ed0c640 1 -- 172.21.15.76:0/4256720048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a78100c40 msgr2=0x7f3a78104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.585+0000 7f3a7ed0c640 1 --2- 172.21.15.76:0/4256720048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a78100c40 0x7f3a78104c80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009930 tx=0x7f3a6802bc30 comp rx=0 tx=0).stop 2024-03-12T05:56:10.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.585+0000 7f3a7ed0c640 1 -- 172.21.15.76:0/4256720048 shutdown_connections 2024-03-12T05:56:10.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.585+0000 7f3a7ed0c640 1 --2- 172.21.15.76:0/4256720048 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a78105360 0x7f3a7810cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.585+0000 7f3a7ed0c640 1 --2- 172.21.15.76:0/4256720048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a78100c40 0x7f3a78104c80 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.585+0000 7f3a7ed0c640 1 --2- 172.21.15.76:0/4256720048 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a78100290 0x7f3a78100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.585+0000 7f3a7ed0c640 1 -- 172.21.15.76:0/4256720048 >> 172.21.15.76:0/4256720048 conn(0x7f3a78077230 msgr2=0x7f3a78077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.586+0000 7f3a7ed0c640 1 -- 172.21.15.76:0/4256720048 shutdown_connections 2024-03-12T05:56:10.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.586+0000 7f3a7ed0c640 1 -- 172.21.15.76:0/4256720048 wait complete. 2024-03-12T05:56:10.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.586+0000 7f3a7ed0c640 1 Processor -- start 2024-03-12T05:56:10.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.586+0000 7f3a7ed0c640 1 -- start start 2024-03-12T05:56:10.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.586+0000 7f3a7ed0c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a78100290 0x7f3a78198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.586+0000 7f3a7ed0c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a78105360 0x7f3a78198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.586+0000 7f3a7ed0c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a78199110 0x7f3a781a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.586+0000 7f3a7ed0c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a781a3d60 con 0x7f3a78105360 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.586+0000 7f3a7ed0c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a781a3ed0 con 0x7f3a78100290 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.586+0000 7f3a7ed0c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a781a4090 con 0x7f3a78199110 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.586+0000 7f3a77fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a78105360 0x7f3a78198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.586+0000 7f3a77fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a78105360 0x7f3a78198b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48162/0 (socket says 172.21.15.76:48162) 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.586+0000 7f3a77fff640 1 -- 172.21.15.76:0/587148129 learned_addr learned my addr 172.21.15.76:0/587148129 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.587+0000 7f3a7d2a9640 1 --2- 172.21.15.76:0/587148129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a78199110 0x7f3a781a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.587+0000 7f3a7caa8640 1 --2- 172.21.15.76:0/587148129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a78100290 0x7f3a78198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.587+0000 7f3a77fff640 1 -- 172.21.15.76:0/587148129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a78199110 msgr2=0x7f3a781a36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.587+0000 7f3a77fff640 1 --2- 172.21.15.76:0/587148129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a78199110 0x7f3a781a36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.587+0000 7f3a77fff640 1 -- 172.21.15.76:0/587148129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a78100290 msgr2=0x7f3a78198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.587+0000 7f3a77fff640 1 --2- 172.21.15.76:0/587148129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a78100290 0x7f3a78198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.587+0000 7f3a77fff640 1 -- 172.21.15.76:0/587148129 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a68009560 con 0x7f3a78105360 2024-03-12T05:56:10.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.587+0000 7f3a77fff640 1 --2- 172.21.15.76:0/587148129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a78105360 0x7f3a78198b80 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009230 tx=0x7f3a68004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.587+0000 7f3a75ffb640 1 -- 172.21.15.76:0/587148129 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6802cbc0 con 0x7f3a78105360 2024-03-12T05:56:10.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.587+0000 7f3a75ffb640 1 -- 172.21.15.76:0/587148129 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a680052d0 con 0x7f3a78105360 2024-03-12T05:56:10.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.587+0000 7f3a7ed0c640 1 -- 172.21.15.76:0/587148129 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a781a4310 con 0x7f3a78105360 2024-03-12T05:56:10.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.587+0000 7f3a75ffb640 1 -- 172.21.15.76:0/587148129 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68003e30 con 0x7f3a78105360 2024-03-12T05:56:10.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.587+0000 7f3a7ed0c640 1 -- 172.21.15.76:0/587148129 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a781a4820 con 0x7f3a78105360 2024-03-12T05:56:10.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.589+0000 7f3a75ffb640 1 -- 172.21.15.76:0/587148129 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f3a68005440 con 0x7f3a78105360 2024-03-12T05:56:10.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a819bf740 1 Processor -- start 2024-03-12T05:56:10.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a819bf740 1 -- start start 2024-03-12T05:56:10.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a819bf740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131250 0x55e9a1131630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a819bf740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131b70 0x55e9a113a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a819bf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a113a750 0x55e9a113cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a819bf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9a10ec960 con 0x55e9a113a750 2024-03-12T05:56:10.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a819bf740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9a10a0310 con 0x55e9a1131b70 2024-03-12T05:56:10.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a819bf740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9a109d840 con 0x55e9a1131250 2024-03-12T05:56:10.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a75ffb640 1 --2- 172.21.15.76:0/587148129 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3a640a1cc0 0x7f3a640a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a77fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a113a750 0x55e9a113cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a7d2a9640 1 --2- 172.21.15.76:0/587148129 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3a640a1cc0 0x7f3a640a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a77fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a113a750 0x55e9a113cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48174/0 (socket says 172.21.15.76:48174) 2024-03-12T05:56:10.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a77fff640 1 -- 172.21.15.76:0/3619106702 learned_addr learned my addr 172.21.15.76:0/3619106702 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a7caa8640 1 --2- 172.21.15.76:0/3619106702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131b70 0x55e9a113a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a7d2a9640 1 --2- 172.21.15.76:0/3619106702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131250 0x55e9a1131630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a75ffb640 1 -- 172.21.15.76:0/587148129 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3a680efe00 con 0x7f3a78105360 2024-03-12T05:56:10.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a77fff640 1 -- 172.21.15.76:0/3619106702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131250 msgr2=0x55e9a1131630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a77fff640 1 --2- 172.21.15.76:0/3619106702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131250 0x55e9a1131630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a77fff640 1 -- 172.21.15.76:0/3619106702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131b70 msgr2=0x55e9a113a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a77fff640 1 --2- 172.21.15.76:0/3619106702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131b70 0x55e9a113a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.590+0000 7f3a77fff640 1 -- 172.21.15.76:0/3619106702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e9a105b060 con 0x55e9a113a750 2024-03-12T05:56:10.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.591+0000 7f3a7d2a9640 1 --2- 172.21.15.76:0/587148129 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3a640a1cc0 0x7f3a640a4180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009fd0 tx=0x7f3a70009d80 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.591+0000 7f3a77fff640 1 --2- 172.21.15.76:0/3619106702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a113a750 0x55e9a113cb40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3a680bc8b0 tx=0x7f3a68009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91753b758952b136 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.591+0000 7f3a6effd640 1 -- 172.21.15.76:0/3619106702 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a680bc920 con 0x55e9a113a750 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.591+0000 7f3a6effd640 1 -- 172.21.15.76:0/3619106702 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a680c0a70 con 0x55e9a113a750 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.591+0000 7f3a6effd640 1 -- 172.21.15.76:0/3619106702 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a680c4e90 con 0x55e9a113a750 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.591+0000 7f3a819bf740 1 -- 172.21.15.76:0/3619106702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a113a750 msgr2=0x55e9a113cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.591+0000 7f3a819bf740 1 --2- 172.21.15.76:0/3619106702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a113a750 0x55e9a113cb40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3a680bc8b0 tx=0x7f3a68009260 comp rx=0 tx=0).stop 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.591+0000 7f3a819bf740 1 -- 172.21.15.76:0/3619106702 shutdown_connections 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.591+0000 7f3a819bf740 1 --2- 172.21.15.76:0/3619106702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a113a750 0x55e9a113cb40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.591+0000 7f3a819bf740 1 --2- 172.21.15.76:0/3619106702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131b70 0x55e9a113a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.591+0000 7f3a819bf740 1 --2- 172.21.15.76:0/3619106702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131250 0x55e9a1131630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.591+0000 7f3a819bf740 1 -- 172.21.15.76:0/3619106702 >> 172.21.15.76:0/3619106702 conn(0x55e9a112b980 msgr2=0x55e9a112bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.591+0000 7f3a819bf740 1 -- 172.21.15.76:0/3619106702 shutdown_connections 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.591+0000 7f3a819bf740 1 -- 172.21.15.76:0/3619106702 wait complete. 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.592+0000 7f3a819bf740 1 -- start start 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.592+0000 7f3a819bf740 1 Processor -- start 2024-03-12T05:56:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.592+0000 7f3a819bf740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131250 0x55e9a11376e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.592+0000 7f3a819bf740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131b70 0x55e9a1137c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.592+0000 7f3a819bf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a113a750 0x55e9a1138160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.592+0000 7f3a819bf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9a1099020 con 0x55e9a113a750 2024-03-12T05:56:10.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.592+0000 7f3a819bf740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9a0d6c430 con 0x55e9a1131250 2024-03-12T05:56:10.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.592+0000 7f3a819bf740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9a0d28e00 con 0x55e9a1131b70 2024-03-12T05:56:10.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.592+0000 7f3a819bf740 10 client.0 _ll_register_callbacks cb 0x55e9a1040cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:10.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.592+0000 7f3a7d2a9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a113a750 0x55e9a1138160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.592+0000 7f3a7d2a9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a113a750 0x55e9a1138160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48190/0 (socket says 172.21.15.76:48190) 2024-03-12T05:56:10.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.592+0000 7f3a7d2a9640 1 -- 172.21.15.76:0/3291873823 learned_addr learned my addr 172.21.15.76:0/3291873823 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.592+0000 7f3a7caa8640 1 --2- 172.21.15.76:0/3291873823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131250 0x55e9a11376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a77fff640 1 --2- 172.21.15.76:0/3291873823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131b70 0x55e9a1137c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a7d2a9640 1 -- 172.21.15.76:0/3291873823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131b70 msgr2=0x55e9a1137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a7d2a9640 1 --2- 172.21.15.76:0/3291873823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131b70 0x55e9a1137c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a7d2a9640 1 -- 172.21.15.76:0/3291873823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131250 msgr2=0x55e9a11376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a7d2a9640 1 --2- 172.21.15.76:0/3291873823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131250 0x55e9a11376e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a7d2a9640 1 -- 172.21.15.76:0/3291873823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a781a4820 con 0x55e9a113a750 2024-03-12T05:56:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a7d2a9640 1 --2- 172.21.15.76:0/3291873823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a113a750 0x55e9a1138160 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3a70010dc0 tx=0x7f3a70009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464d70a1730681e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a57fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a57fff640 1 -- 172.21.15.76:0/3291873823 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a70004530 con 0x55e9a113a750 2024-03-12T05:56:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a57fff640 1 -- 172.21.15.76:0/3291873823 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3a70004690 con 0x55e9a113a750 2024-03-12T05:56:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a819bf740 1 -- 172.21.15.76:0/3291873823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e9a0d987e0 con 0x55e9a113a750 2024-03-12T05:56:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a57fff640 1 -- 172.21.15.76:0/3291873823 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a70004850 con 0x55e9a113a750 2024-03-12T05:56:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a819bf740 10 client.15588 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a819bf740 1 -- 172.21.15.76:0/3291873823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55e9a10976a0 con 0x55e9a113a750 2024-03-12T05:56:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a57fff640 1 -- 172.21.15.76:0/3291873823 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e9a0d987e0 con 0x55e9a113a750 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a57fff640 1 -- 172.21.15.76:0/3291873823 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3a70026030 con 0x55e9a113a750 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a819bf740 10 client.15588 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a819bf740 10 client.15588 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.593+0000 7f3a819bf740 20 client.15588 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.594+0000 7f3a819bf740 1 --2- 172.21.15.76:0/3291873823 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e9a114a820 0x55e9a114cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.594+0000 7f3a819bf740 4 client.15588 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.594+0000 7f3a819bf740 1 -- 172.21.15.76:0/3291873823 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e9a0dd5720 con 0x55e9a114a820 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.594+0000 7f3a7caa8640 1 --2- 172.21.15.76:0/3291873823 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e9a114a820 0x55e9a114cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.594+0000 7f3a7caa8640 1 --2- 172.21.15.76:0/3291873823 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e9a114a820 0x55e9a114cc40 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8837bec666bf1026 server_cookie=5f08d314fc92ae8 in_seq=0 out_seq=0 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.594+0000 7f3a57fff640 10 client.15588 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a57fff640 1 -- 172.21.15.76:0/3291873823 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e9a0dd5720 con 0x55e9a114a820 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a57fff640 10 client.15588 handle_command_reply: tid=0 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a819bf740 1 client.15588 shutdown 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a819bf740 1 -- 172.21.15.76:0/3291873823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a113a750 msgr2=0x55e9a1138160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a819bf740 1 --2- 172.21.15.76:0/3291873823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a113a750 0x55e9a1138160 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3a70010dc0 tx=0x7f3a70009550 comp rx=0 tx=0).stop 2024-03-12T05:56:10.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a819bf740 1 -- 172.21.15.76:0/3291873823 shutdown_connections 2024-03-12T05:56:10.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a819bf740 1 --2- 172.21.15.76:0/3291873823 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e9a114a820 0x55e9a114cc40 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a819bf740 1 --2- 172.21.15.76:0/3291873823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a113a750 0x55e9a1138160 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a819bf740 1 --2- 172.21.15.76:0/3291873823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131b70 0x55e9a1137c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a57fff640 0 client.15588 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:56:10.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a819bf740 1 --2- 172.21.15.76:0/3291873823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131250 0x55e9a11376e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a819bf740 1 -- 172.21.15.76:0/3291873823 >> 172.21.15.76:0/3291873823 conn(0x55e9a112b980 msgr2=0x55e9a112d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a819bf740 1 -- 172.21.15.76:0/3291873823 shutdown_connections 2024-03-12T05:56:10.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a819bf740 1 -- 172.21.15.76:0/3291873823 wait complete. 2024-03-12T05:56:10.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.595+0000 7f3a819bf740 20 client.15588 trim_cache size 0 max 16384 2024-03-12T05:56:10.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.606+0000 7f3a819bf740 1 Processor -- start 2024-03-12T05:56:10.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a819bf740 1 -- start start 2024-03-12T05:56:10.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a819bf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a1131250 0x55e9a1139e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a819bf740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131b70 0x55e9a1137f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a819bf740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a113a750 0x55e9a1134940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a819bf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9a101e130 con 0x55e9a1131250 2024-03-12T05:56:10.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a819bf740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9a114a650 con 0x55e9a1131b70 2024-03-12T05:56:10.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a819bf740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9a11332e0 con 0x55e9a113a750 2024-03-12T05:56:10.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a7caa8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a1131250 0x55e9a1139e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a7caa8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a1131250 0x55e9a1139e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48192/0 (socket says 172.21.15.76:48192) 2024-03-12T05:56:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a7caa8640 1 -- 172.21.15.76:0/829991295 learned_addr learned my addr 172.21.15.76:0/829991295 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a7d2a9640 1 --2- 172.21.15.76:0/829991295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a113a750 0x55e9a1134940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a7caa8640 1 -- 172.21.15.76:0/829991295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a113a750 msgr2=0x55e9a1134940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a7caa8640 1 --2- 172.21.15.76:0/829991295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a113a750 0x55e9a1134940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a77fff640 1 --2- 172.21.15.76:0/829991295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131b70 0x55e9a1137f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a7caa8640 1 -- 172.21.15.76:0/829991295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131b70 msgr2=0x55e9a1137f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a7caa8640 1 --2- 172.21.15.76:0/829991295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131b70 0x55e9a1137f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a7caa8640 1 -- 172.21.15.76:0/829991295 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e9a1033c20 con 0x55e9a1131250 2024-03-12T05:56:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a7caa8640 1 --2- 172.21.15.76:0/829991295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a1131250 0x55e9a1139e10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x55e9a114d540 tx=0x7f3a6000f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402f98f7e90a267a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.607+0000 7f3a6e7fc640 1 -- 172.21.15.76:0/829991295 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a60016040 con 0x55e9a1131250 2024-03-12T05:56:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.608+0000 7f3a6e7fc640 1 -- 172.21.15.76:0/829991295 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a6001f070 con 0x55e9a1131250 2024-03-12T05:56:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.608+0000 7f3a6e7fc640 1 -- 172.21.15.76:0/829991295 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6000ed40 con 0x55e9a1131250 2024-03-12T05:56:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.608+0000 7f3a819bf740 1 -- 172.21.15.76:0/829991295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a1131250 msgr2=0x55e9a1139e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.608+0000 7f3a819bf740 1 --2- 172.21.15.76:0/829991295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a1131250 0x55e9a1139e10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x55e9a114d540 tx=0x7f3a6000f7f0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.608+0000 7f3a819bf740 1 -- 172.21.15.76:0/829991295 shutdown_connections 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.608+0000 7f3a819bf740 1 --2- 172.21.15.76:0/829991295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a113a750 0x55e9a1134940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.608+0000 7f3a819bf740 1 --2- 172.21.15.76:0/829991295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a1131b70 0x55e9a1137f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.608+0000 7f3a819bf740 1 --2- 172.21.15.76:0/829991295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a1131250 0x55e9a1139e10 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.608+0000 7f3a819bf740 1 -- 172.21.15.76:0/829991295 >> 172.21.15.76:0/829991295 conn(0x55e9a112b980 msgr2=0x55e9a11515c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.608+0000 7f3a819bf740 1 -- 172.21.15.76:0/829991295 shutdown_connections 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.608+0000 7f3a819bf740 1 -- 172.21.15.76:0/829991295 wait complete. 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.608+0000 7f3a819bf740 1 -- start start 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a819bf740 1 Processor -- start 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a819bf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a1131250 0x55e9a1140030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a819bf740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131b70 0x55e9a1140570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a819bf740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a113a750 0x55e9a1140ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a819bf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9a11a0d80 con 0x55e9a1131250 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a819bf740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9a11a0ef0 con 0x55e9a113a750 2024-03-12T05:56:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a819bf740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9a11a10b0 con 0x55e9a1131b70 2024-03-12T05:56:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a819bf740 10 client.0 _ll_register_callbacks cb 0x7f3a700124d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a7caa8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a1131250 0x55e9a1140030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a7caa8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a1131250 0x55e9a1140030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48200/0 (socket says 172.21.15.76:48200) 2024-03-12T05:56:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a7caa8640 1 -- 172.21.15.76:0/2409353290 learned_addr learned my addr 172.21.15.76:0/2409353290 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a77fff640 1 --2- 172.21.15.76:0/2409353290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131b70 0x55e9a1140570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a7d2a9640 1 --2- 172.21.15.76:0/2409353290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a113a750 0x55e9a1140ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a7caa8640 1 -- 172.21.15.76:0/2409353290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131b70 msgr2=0x55e9a1140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a7caa8640 1 --2- 172.21.15.76:0/2409353290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131b70 0x55e9a1140570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a7caa8640 1 -- 172.21.15.76:0/2409353290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a113a750 msgr2=0x55e9a1140ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a7caa8640 1 --2- 172.21.15.76:0/2409353290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a113a750 0x55e9a1140ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a7caa8640 1 -- 172.21.15.76:0/2409353290 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6000bcb0 con 0x55e9a1131250 2024-03-12T05:56:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.609+0000 7f3a7caa8640 1 --2- 172.21.15.76:0/2409353290 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a1131250 0x55e9a1140030 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3a60000c00 tx=0x7f3a6000e660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef5ce1f8387f0159 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a57fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:10.619 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a57fff640 1 -- 172.21.15.76:0/2409353290 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a60016040 con 0x55e9a1131250 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a57fff640 1 -- 172.21.15.76:0/2409353290 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3a6000ebc0 con 0x55e9a1131250 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a819bf740 1 -- 172.21.15.76:0/2409353290 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a680c0a70 con 0x55e9a1131250 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a57fff640 1 -- 172.21.15.76:0/2409353290 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a6000ee00 con 0x55e9a1131250 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a819bf740 10 client.15594 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a819bf740 1 -- 172.21.15.76:0/2409353290 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55e9a10598c0 con 0x55e9a1131250 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a57fff640 1 -- 172.21.15.76:0/2409353290 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a680c0a70 con 0x55e9a1131250 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a57fff640 1 -- 172.21.15.76:0/2409353290 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3a60004700 con 0x55e9a1131250 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a819bf740 10 client.15594 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a819bf740 10 client.15594 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a819bf740 20 client.15594 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a819bf740 1 --2- 172.21.15.76:0/2409353290 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e9a11a2800 0x55e9a11a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a819bf740 4 client.15594 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a819bf740 1 -- 172.21.15.76:0/2409353290 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e9a0e24890 con 0x55e9a11a2800 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.610+0000 7f3a77fff640 1 --2- 172.21.15.76:0/2409353290 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e9a11a2800 0x55e9a11a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a77fff640 1 --2- 172.21.15.76:0/2409353290 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e9a11a2800 0x55e9a11a4c20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d644e19f2f69360 server_cookie=68b9af3c4a5a0f02 in_seq=0 out_seq=0 2024-03-12T05:56:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a57fff640 10 client.15594 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:56:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a57fff640 1 -- 172.21.15.76:0/2409353290 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e9a0e24890 con 0x55e9a11a2800 2024-03-12T05:56:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a57fff640 10 client.15594 handle_command_reply: tid=0 2024-03-12T05:56:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a819bf740 1 client.15594 shutdown 2024-03-12T05:56:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a819bf740 1 -- 172.21.15.76:0/2409353290 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a1131250 msgr2=0x55e9a1140030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a819bf740 1 --2- 172.21.15.76:0/2409353290 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a1131250 0x55e9a1140030 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3a60000c00 tx=0x7f3a6000e660 comp rx=0 tx=0).stop 2024-03-12T05:56:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a819bf740 1 -- 172.21.15.76:0/2409353290 shutdown_connections 2024-03-12T05:56:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a819bf740 1 --2- 172.21.15.76:0/2409353290 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e9a11a2800 0x55e9a11a4c20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a819bf740 1 --2- 172.21.15.76:0/2409353290 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e9a113a750 0x55e9a1140ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a819bf740 1 --2- 172.21.15.76:0/2409353290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e9a1131b70 0x55e9a1140570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a57fff640 0 client.15594 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:56:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a819bf740 1 --2- 172.21.15.76:0/2409353290 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e9a1131250 0x55e9a1140030 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a819bf740 1 -- 172.21.15.76:0/2409353290 >> 172.21.15.76:0/2409353290 conn(0x55e9a112b980 msgr2=0x55e9a112a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a819bf740 1 -- 172.21.15.76:0/2409353290 shutdown_connections 2024-03-12T05:56:10.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a819bf740 1 -- 172.21.15.76:0/2409353290 wait complete. 2024-03-12T05:56:10.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.611+0000 7f3a819bf740 20 client.15594 trim_cache size 0 max 16384 2024-03-12T05:56:10.622 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:56:10.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.612+0000 7f3a7ed0c640 1 -- 172.21.15.76:0/587148129 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3a640a1cc0 msgr2=0x7f3a640a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.612+0000 7f3a7ed0c640 1 --2- 172.21.15.76:0/587148129 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3a640a1cc0 0x7f3a640a4180 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009fd0 tx=0x7f3a70009d80 comp rx=0 tx=0).stop 2024-03-12T05:56:10.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.612+0000 7f3a7ed0c640 1 -- 172.21.15.76:0/587148129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a78105360 msgr2=0x7f3a78198b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.612+0000 7f3a7ed0c640 1 --2- 172.21.15.76:0/587148129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a78105360 0x7f3a78198b80 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009230 tx=0x7f3a68004270 comp rx=0 tx=0).stop 2024-03-12T05:56:10.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.612+0000 7f3a7ed0c640 1 -- 172.21.15.76:0/587148129 shutdown_connections 2024-03-12T05:56:10.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.612+0000 7f3a7ed0c640 1 --2- 172.21.15.76:0/587148129 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3a640a1cc0 0x7f3a640a4180 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.612+0000 7f3a7ed0c640 1 --2- 172.21.15.76:0/587148129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a78199110 0x7f3a781a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.612+0000 7f3a7ed0c640 1 --2- 172.21.15.76:0/587148129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a78105360 0x7f3a78198b80 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.612+0000 7f3a7ed0c640 1 --2- 172.21.15.76:0/587148129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a78100290 0x7f3a78198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.612+0000 7f3a7ed0c640 1 -- 172.21.15.76:0/587148129 >> 172.21.15.76:0/587148129 conn(0x7f3a78077230 msgr2=0x7f3a780fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.612+0000 7f3a7ed0c640 1 -- 172.21.15.76:0/587148129 shutdown_connections 2024-03-12T05:56:10.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.612+0000 7f3a7ed0c640 1 -- 172.21.15.76:0/587148129 wait complete. 2024-03-12T05:56:10.626 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:56:10.626 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:56:10.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.728+0000 7f9927393640 1 -- 172.21.15.76:0/1827107785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9920100c40 msgr2=0x7f9920104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.728+0000 7f9927393640 1 --2- 172.21.15.76:0/1827107785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9920100c40 0x7f9920104c80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f991c009930 tx=0x7f991c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:56:10.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.728+0000 7f9927393640 1 -- 172.21.15.76:0/1827107785 shutdown_connections 2024-03-12T05:56:10.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.728+0000 7f9927393640 1 --2- 172.21.15.76:0/1827107785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9920105360 0x7f992010cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.728+0000 7f9927393640 1 --2- 172.21.15.76:0/1827107785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9920100c40 0x7f9920104c80 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.728+0000 7f9927393640 1 --2- 172.21.15.76:0/1827107785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9920100290 0x7f9920100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.728+0000 7f9927393640 1 -- 172.21.15.76:0/1827107785 >> 172.21.15.76:0/1827107785 conn(0x7f9920077230 msgr2=0x7f9920077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.728+0000 7f9927393640 1 -- 172.21.15.76:0/1827107785 shutdown_connections 2024-03-12T05:56:10.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.728+0000 7f9927393640 1 -- 172.21.15.76:0/1827107785 wait complete. 2024-03-12T05:56:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.728+0000 7f9927393640 1 Processor -- start 2024-03-12T05:56:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.728+0000 7f9927393640 1 -- start start 2024-03-12T05:56:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9927393640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9920100290 0x7f992019d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9927393640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9920105360 0x7f992019dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9927393640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9920197630 0x7f9920197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9927393640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9920198100 con 0x7f9920197630 2024-03-12T05:56:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9927393640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9920198270 con 0x7f9920105360 2024-03-12T05:56:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9927393640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9920198430 con 0x7f9920100290 2024-03-12T05:56:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9925930640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9920197630 0x7f9920197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9925930640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9920197630 0x7f9920197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48212/0 (socket says 172.21.15.76:48212) 2024-03-12T05:56:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9925930640 1 -- 172.21.15.76:0/2273686654 learned_addr learned my addr 172.21.15.76:0/2273686654 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f992512f640 1 --2- 172.21.15.76:0/2273686654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9920100290 0x7f992019d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f992492e640 1 --2- 172.21.15.76:0/2273686654 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9920105360 0x7f992019dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9925930640 1 -- 172.21.15.76:0/2273686654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9920100290 msgr2=0x7f992019d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9925930640 1 --2- 172.21.15.76:0/2273686654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9920100290 0x7f992019d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9925930640 1 -- 172.21.15.76:0/2273686654 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9920105360 msgr2=0x7f992019dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9925930640 1 --2- 172.21.15.76:0/2273686654 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9920105360 0x7f992019dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9925930640 1 -- 172.21.15.76:0/2273686654 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c009560 con 0x7f9920197630 2024-03-12T05:56:10.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9925930640 1 --2- 172.21.15.76:0/2273686654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9920197630 0x7f9920197ab0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f991800e960 tx=0x7f991800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f99167fc640 1 -- 172.21.15.76:0/2273686654 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918004d20 con 0x7f9920197630 2024-03-12T05:56:10.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9927393640 1 -- 172.21.15.76:0/2273686654 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99201a43d0 con 0x7f9920197630 2024-03-12T05:56:10.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f99167fc640 1 -- 172.21.15.76:0/2273686654 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9918004e80 con 0x7f9920197630 2024-03-12T05:56:10.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f99167fc640 1 -- 172.21.15.76:0/2273686654 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99180116d0 con 0x7f9920197630 2024-03-12T05:56:10.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.729+0000 7f9927393640 1 -- 172.21.15.76:0/2273686654 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99201a4860 con 0x7f9920197630 2024-03-12T05:56:10.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.731+0000 7f99167fc640 1 -- 172.21.15.76:0/2273686654 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f9918005340 con 0x7f9920197630 2024-03-12T05:56:10.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f9929f6d740 1 Processor -- start 2024-03-12T05:56:10.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f9929f6d740 1 -- start start 2024-03-12T05:56:10.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f9929f6d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 0x55e7e3ff0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f9929f6d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ffd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f9929f6d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ffdb60 0x55e7e3ffff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f9929f6d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e3fab960 con 0x55e7e3ff08a0 2024-03-12T05:56:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f9929f6d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e3f5f310 con 0x55e7e3ffdb60 2024-03-12T05:56:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f9929f6d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e3f5c840 con 0x55e7e3feff80 2024-03-12T05:56:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f99167fc640 1 --2- 172.21.15.76:0/2273686654 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f990c0a1d40 0x7f990c0a4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f992512f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ffd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f992512f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ffd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48226/0 (socket says 172.21.15.76:48226) 2024-03-12T05:56:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f992512f640 1 -- 172.21.15.76:0/4045657411 learned_addr learned my addr 172.21.15.76:0/4045657411 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f9925930640 1 --2- 172.21.15.76:0/2273686654 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f990c0a1d40 0x7f990c0a4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f992492e640 1 --2- 172.21.15.76:0/4045657411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 0x55e7e3ff0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f992492e640 1 --2- 172.21.15.76:0/4045657411 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ffdb60 0x55e7e3ffff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.732+0000 7f99167fc640 1 -- 172.21.15.76:0/2273686654 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f99180cdc50 con 0x7f9920197630 2024-03-12T05:56:10.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.733+0000 7f992512f640 1 -- 172.21.15.76:0/4045657411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 msgr2=0x55e7e3ff0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.733+0000 7f992512f640 1 --2- 172.21.15.76:0/4045657411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 0x55e7e3ff0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.733+0000 7f992512f640 1 -- 172.21.15.76:0/4045657411 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ffdb60 msgr2=0x55e7e3ffff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.733+0000 7f992512f640 1 --2- 172.21.15.76:0/4045657411 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ffdb60 0x55e7e3ffff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.733+0000 7f992512f640 1 -- 172.21.15.76:0/4045657411 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7e3f1a060 con 0x55e7e3ff08a0 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.733+0000 7f9925930640 1 --2- 172.21.15.76:0/2273686654 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f990c0a1d40 0x7f990c0a4200 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f99180cb250 tx=0x7f99180ba040 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.733+0000 7f992512f640 1 --2- 172.21.15.76:0/4045657411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ffd620 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f991000bd10 tx=0x7f99100107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856a67d683e71be6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.733+0000 7f990b7fe640 1 -- 172.21.15.76:0/4045657411 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9910004030 con 0x55e7e3ff08a0 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.733+0000 7f990b7fe640 1 -- 172.21.15.76:0/4045657411 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9910015070 con 0x55e7e3ff08a0 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.733+0000 7f990b7fe640 1 -- 172.21.15.76:0/4045657411 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99100116a0 con 0x55e7e3ff08a0 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.734+0000 7f9929f6d740 1 -- 172.21.15.76:0/4045657411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ff08a0 msgr2=0x55e7e3ffd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.734+0000 7f9929f6d740 1 --2- 172.21.15.76:0/4045657411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ffd620 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f991000bd10 tx=0x7f99100107a0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.734+0000 7f9929f6d740 1 -- 172.21.15.76:0/4045657411 shutdown_connections 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.734+0000 7f9929f6d740 1 --2- 172.21.15.76:0/4045657411 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ffdb60 0x55e7e3ffff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.734+0000 7f9929f6d740 1 --2- 172.21.15.76:0/4045657411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ffd620 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.734+0000 7f9929f6d740 1 --2- 172.21.15.76:0/4045657411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 0x55e7e3ff0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.734+0000 7f9929f6d740 1 -- 172.21.15.76:0/4045657411 >> 172.21.15.76:0/4045657411 conn(0x55e7e3fea980 msgr2=0x55e7e3fead60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.734+0000 7f9929f6d740 1 -- 172.21.15.76:0/4045657411 shutdown_connections 2024-03-12T05:56:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.734+0000 7f9929f6d740 1 -- 172.21.15.76:0/4045657411 wait complete. 2024-03-12T05:56:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.734+0000 7f9929f6d740 1 -- start start 2024-03-12T05:56:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.734+0000 7f9929f6d740 1 Processor -- start 2024-03-12T05:56:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f9929f6d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 0x55e7e3ffabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f9929f6d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ffb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f9929f6d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ffdb60 0x55e7e3ffb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f9929f6d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e3f58020 con 0x55e7e3ffdb60 2024-03-12T05:56:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f9929f6d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e3c2b430 con 0x55e7e3ff08a0 2024-03-12T05:56:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f9929f6d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e3be7e00 con 0x55e7e3feff80 2024-03-12T05:56:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f9929f6d740 10 client.0 _ll_register_callbacks cb 0x55e7e3effcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f992492e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ffdb60 0x55e7e3ffb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f992492e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ffdb60 0x55e7e3ffb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48242/0 (socket says 172.21.15.76:48242) 2024-03-12T05:56:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f992492e640 1 -- 172.21.15.76:0/2711333395 learned_addr learned my addr 172.21.15.76:0/2711333395 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f992492e640 1 --2- 172.21.15.76:0/2711333395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 0x55e7e3ffabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f992512f640 1 --2- 172.21.15.76:0/2711333395 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ffb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f992492e640 1 -- 172.21.15.76:0/2711333395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 msgr2=0x55e7e3ffabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f992492e640 1 --2- 172.21.15.76:0/2711333395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 0x55e7e3ffabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f992492e640 1 -- 172.21.15.76:0/2711333395 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 msgr2=0x55e7e3ffb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f992492e640 1 --2- 172.21.15.76:0/2711333395 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ffb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f992492e640 1 -- 172.21.15.76:0/2711333395 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f991000b940 con 0x55e7e3ffdb60 2024-03-12T05:56:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f992492e640 1 --2- 172.21.15.76:0/2711333395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ffdb60 0x55e7e3ffb630 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f991c009930 tx=0x7f991c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9fea1b9105f3673 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f9908ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f9908ff9640 1 -- 172.21.15.76:0/2711333395 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c005200 con 0x55e7e3ffdb60 2024-03-12T05:56:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f9908ff9640 1 -- 172.21.15.76:0/2711333395 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f991c005360 con 0x55e7e3ffdb60 2024-03-12T05:56:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.735+0000 7f9929f6d740 1 -- 172.21.15.76:0/2711333395 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e7e3c577e0 con 0x55e7e3ffdb60 2024-03-12T05:56:10.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.736+0000 7f9908ff9640 1 -- 172.21.15.76:0/2711333395 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f991c0044f0 con 0x55e7e3ffdb60 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.736+0000 7f9929f6d740 10 client.15606 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.736+0000 7f9929f6d740 1 -- 172.21.15.76:0/2711333395 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55e7e3f566a0 con 0x55e7e3ffdb60 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.736+0000 7f9908ff9640 1 -- 172.21.15.76:0/2711333395 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e7e3c577e0 con 0x55e7e3ffdb60 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.736+0000 7f9908ff9640 1 -- 172.21.15.76:0/2711333395 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f991c043450 con 0x55e7e3ffdb60 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.736+0000 7f9929f6d740 10 client.15606 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.736+0000 7f9929f6d740 10 client.15606 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.736+0000 7f9929f6d740 20 client.15606 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.736+0000 7f9929f6d740 1 --2- 172.21.15.76:0/2711333395 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e7e400def0 0x55e7e4010310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.736+0000 7f9929f6d740 4 client.15606 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.736+0000 7f9929f6d740 1 -- 172.21.15.76:0/2711333395 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e7e3c94720 con 0x55e7e400def0 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.736+0000 7f992512f640 1 --2- 172.21.15.76:0/2711333395 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e7e400def0 0x55e7e4010310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.737+0000 7f9908ff9640 10 client.15606 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.737+0000 7f992512f640 1 --2- 172.21.15.76:0/2711333395 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e7e400def0 0x55e7e4010310 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=19cdecda55baa2c8 server_cookie=75fd41f1b3f5a73a in_seq=0 out_seq=0 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.737+0000 7f9908ff9640 1 -- 172.21.15.76:0/2711333395 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e7e3c94720 con 0x55e7e400def0 2024-03-12T05:56:10.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.737+0000 7f9908ff9640 10 client.15606 handle_command_reply: tid=0 2024-03-12T05:56:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.737+0000 7f9929f6d740 1 client.15606 shutdown 2024-03-12T05:56:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.737+0000 7f9929f6d740 1 -- 172.21.15.76:0/2711333395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ffdb60 msgr2=0x55e7e3ffb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.737+0000 7f9929f6d740 1 --2- 172.21.15.76:0/2711333395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ffdb60 0x55e7e3ffb630 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f991c009930 tx=0x7f991c009260 comp rx=0 tx=0).stop 2024-03-12T05:56:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.737+0000 7f9929f6d740 1 -- 172.21.15.76:0/2711333395 shutdown_connections 2024-03-12T05:56:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.737+0000 7f9929f6d740 1 --2- 172.21.15.76:0/2711333395 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e7e400def0 0x55e7e4010310 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.737+0000 7f9929f6d740 1 --2- 172.21.15.76:0/2711333395 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ffdb60 0x55e7e3ffb630 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.737+0000 7f9929f6d740 1 --2- 172.21.15.76:0/2711333395 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ffb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.738+0000 7f9929f6d740 1 --2- 172.21.15.76:0/2711333395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 0x55e7e3ffabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.738+0000 7f9908ff9640 0 client.15606 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:56:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.738+0000 7f9929f6d740 1 -- 172.21.15.76:0/2711333395 >> 172.21.15.76:0/2711333395 conn(0x55e7e3fea980 msgr2=0x55e7e3fec6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.738+0000 7f9929f6d740 1 -- 172.21.15.76:0/2711333395 shutdown_connections 2024-03-12T05:56:10.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.738+0000 7f9929f6d740 1 -- 172.21.15.76:0/2711333395 wait complete. 2024-03-12T05:56:10.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.738+0000 7f9929f6d740 20 client.15606 trim_cache size 0 max 16384 2024-03-12T05:56:10.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.749+0000 7f9929f6d740 1 Processor -- start 2024-03-12T05:56:10.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.749+0000 7f9929f6d740 1 -- start start 2024-03-12T05:56:10.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.749+0000 7f9929f6d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3feff80 0x55e7e3ffd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.749+0000 7f9929f6d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ff7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.749+0000 7f9929f6d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3ffdb60 0x55e7e3ff8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.749+0000 7f9929f6d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e3edd130 con 0x55e7e3feff80 2024-03-12T05:56:10.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.749+0000 7f9929f6d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e400aaa0 con 0x55e7e3ff08a0 2024-03-12T05:56:10.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.749+0000 7f9929f6d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e3fe65c0 con 0x55e7e3ffdb60 2024-03-12T05:56:10.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.750+0000 7f992492e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3feff80 0x55e7e3ffd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.750+0000 7f992492e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3feff80 0x55e7e3ffd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48254/0 (socket says 172.21.15.76:48254) 2024-03-12T05:56:10.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.750+0000 7f992492e640 1 -- 172.21.15.76:0/467917439 learned_addr learned my addr 172.21.15.76:0/467917439 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.750+0000 7f992492e640 1 -- 172.21.15.76:0/467917439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3ffdb60 msgr2=0x55e7e3ff8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.750+0000 7f992492e640 1 --2- 172.21.15.76:0/467917439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3ffdb60 0x55e7e3ff8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.750+0000 7f992512f640 1 --2- 172.21.15.76:0/467917439 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ff7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.750+0000 7f992492e640 1 -- 172.21.15.76:0/467917439 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 msgr2=0x55e7e3ff7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.750+0000 7f992492e640 1 --2- 172.21.15.76:0/467917439 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ff7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.750+0000 7f992492e640 1 -- 172.21.15.76:0/467917439 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7e3ef2c20 con 0x55e7e3feff80 2024-03-12T05:56:10.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.750+0000 7f992512f640 1 --2- 172.21.15.76:0/467917439 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ff7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:56:10.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.750+0000 7f992492e640 1 --2- 172.21.15.76:0/467917439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3feff80 0x55e7e3ffd2e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f991c02c170 tx=0x7f991c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0f6e108bcc59b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.750+0000 7f990affd640 1 -- 172.21.15.76:0/467917439 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c047070 con 0x55e7e3feff80 2024-03-12T05:56:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.750+0000 7f990affd640 1 -- 172.21.15.76:0/467917439 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f991c04a070 con 0x55e7e3feff80 2024-03-12T05:56:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.750+0000 7f990affd640 1 -- 172.21.15.76:0/467917439 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c043c30 con 0x55e7e3feff80 2024-03-12T05:56:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 -- 172.21.15.76:0/467917439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3feff80 msgr2=0x55e7e3ffd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 --2- 172.21.15.76:0/467917439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3feff80 0x55e7e3ffd2e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f991c02c170 tx=0x7f991c005fd0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 -- 172.21.15.76:0/467917439 shutdown_connections 2024-03-12T05:56:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 --2- 172.21.15.76:0/467917439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3ffdb60 0x55e7e3ff8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 --2- 172.21.15.76:0/467917439 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 0x55e7e3ff7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 --2- 172.21.15.76:0/467917439 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3feff80 0x55e7e3ffd2e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 -- 172.21.15.76:0/467917439 >> 172.21.15.76:0/467917439 conn(0x55e7e3fea980 msgr2=0x55e7e3ff5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 -- 172.21.15.76:0/467917439 shutdown_connections 2024-03-12T05:56:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 -- 172.21.15.76:0/467917439 wait complete. 2024-03-12T05:56:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 -- start start 2024-03-12T05:56:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 Processor -- start 2024-03-12T05:56:10.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 0x55e7e40037e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 0x55e7e4003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ffdb60 0x55e7e4064440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e4004530 con 0x55e7e3ffdb60 2024-03-12T05:56:10.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e40046a0 con 0x55e7e3ff08a0 2024-03-12T05:56:10.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7e4064a00 con 0x55e7e3feff80 2024-03-12T05:56:10.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.751+0000 7f9929f6d740 10 client.0 _ll_register_callbacks cb 0x7f991000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:10.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f992492e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ffdb60 0x55e7e4064440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f992492e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ffdb60 0x55e7e4064440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48260/0 (socket says 172.21.15.76:48260) 2024-03-12T05:56:10.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f992492e640 1 -- 172.21.15.76:0/3222652443 learned_addr learned my addr 172.21.15.76:0/3222652443 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f992492e640 1 --2- 172.21.15.76:0/3222652443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 0x55e7e40037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f992512f640 1 --2- 172.21.15.76:0/3222652443 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 0x55e7e4003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f992492e640 1 -- 172.21.15.76:0/3222652443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 msgr2=0x55e7e40037e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f992492e640 1 --2- 172.21.15.76:0/3222652443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 0x55e7e40037e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f992492e640 1 -- 172.21.15.76:0/3222652443 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 msgr2=0x55e7e4003d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f992492e640 1 --2- 172.21.15.76:0/3222652443 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 0x55e7e4003d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f992492e640 1 -- 172.21.15.76:0/3222652443 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e7e3f566a0 con 0x55e7e3ffdb60 2024-03-12T05:56:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f992492e640 1 --2- 172.21.15.76:0/3222652443 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ffdb60 0x55e7e4064440 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f991c043550 tx=0x7f991c043450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2a141fb67a0712 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f9908ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f9908ff9640 1 -- 172.21.15.76:0/3222652443 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c047040 con 0x55e7e3ffdb60 2024-03-12T05:56:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f9908ff9640 1 -- 172.21.15.76:0/3222652443 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f991c000e40 con 0x55e7e3ffdb60 2024-03-12T05:56:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f9929f6d740 1 -- 172.21.15.76:0/3222652443 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9910015070 con 0x55e7e3ffdb60 2024-03-12T05:56:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f9908ff9640 1 -- 172.21.15.76:0/3222652443 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f991c005200 con 0x55e7e3ffdb60 2024-03-12T05:56:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f9929f6d740 10 client.15612 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f9929f6d740 1 -- 172.21.15.76:0/3222652443 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55e7e3f188c0 con 0x55e7e3ffdb60 2024-03-12T05:56:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.752+0000 7f9908ff9640 1 -- 172.21.15.76:0/3222652443 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9910015070 con 0x55e7e3ffdb60 2024-03-12T05:56:10.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.753+0000 7f9908ff9640 1 -- 172.21.15.76:0/3222652443 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f991c05a430 con 0x55e7e3ffdb60 2024-03-12T05:56:10.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.753+0000 7f9929f6d740 10 client.15612 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:10.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.753+0000 7f9929f6d740 10 client.15612 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:56:10.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.753+0000 7f9929f6d740 20 client.15612 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:10.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.753+0000 7f9929f6d740 1 --2- 172.21.15.76:0/3222652443 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e7e4065eb0 0x55e7e40682d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.753+0000 7f9929f6d740 4 client.15612 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:56:10.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.753+0000 7f9929f6d740 1 -- 172.21.15.76:0/3222652443 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e7e3ce3890 con 0x55e7e4065eb0 2024-03-12T05:56:10.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.753+0000 7f992512f640 1 --2- 172.21.15.76:0/3222652443 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e7e4065eb0 0x55e7e40682d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.753+0000 7f9908ff9640 10 client.15612 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:56:10.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.753+0000 7f992512f640 1 --2- 172.21.15.76:0/3222652443 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e7e4065eb0 0x55e7e40682d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8b5bf6a557ce56d5 server_cookie=daebef1a83911e35 in_seq=0 out_seq=0 2024-03-12T05:56:10.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.753+0000 7f9908ff9640 1 -- 172.21.15.76:0/3222652443 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e7e3ce3890 con 0x55e7e4065eb0 2024-03-12T05:56:10.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.753+0000 7f9908ff9640 10 client.15612 handle_command_reply: tid=0 2024-03-12T05:56:10.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.753+0000 7f9929f6d740 1 client.15612 shutdown 2024-03-12T05:56:10.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9929f6d740 1 -- 172.21.15.76:0/3222652443 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ffdb60 msgr2=0x55e7e4064440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9929f6d740 1 --2- 172.21.15.76:0/3222652443 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ffdb60 0x55e7e4064440 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f991c043550 tx=0x7f991c043450 comp rx=0 tx=0).stop 2024-03-12T05:56:10.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9929f6d740 1 -- 172.21.15.76:0/3222652443 shutdown_connections 2024-03-12T05:56:10.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9929f6d740 1 --2- 172.21.15.76:0/3222652443 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e7e4065eb0 0x55e7e40682d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9929f6d740 1 --2- 172.21.15.76:0/3222652443 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7e3ffdb60 0x55e7e4064440 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9929f6d740 1 --2- 172.21.15.76:0/3222652443 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7e3ff08a0 0x55e7e4003d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9908ff9640 0 client.15612 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:56:10.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9929f6d740 1 --2- 172.21.15.76:0/3222652443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7e3feff80 0x55e7e40037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9929f6d740 1 -- 172.21.15.76:0/3222652443 >> 172.21.15.76:0/3222652443 conn(0x55e7e3fea980 msgr2=0x55e7e3fe92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9929f6d740 1 -- 172.21.15.76:0/3222652443 shutdown_connections 2024-03-12T05:56:10.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9929f6d740 1 -- 172.21.15.76:0/3222652443 wait complete. 2024-03-12T05:56:10.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9929f6d740 20 client.15612 trim_cache size 0 max 16384 2024-03-12T05:56:10.766 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:56:10.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9927393640 1 -- 172.21.15.76:0/2273686654 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f990c0a1d40 msgr2=0x7f990c0a4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9927393640 1 --2- 172.21.15.76:0/2273686654 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f990c0a1d40 0x7f990c0a4200 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f99180cb250 tx=0x7f99180ba040 comp rx=0 tx=0).stop 2024-03-12T05:56:10.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9927393640 1 -- 172.21.15.76:0/2273686654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9920197630 msgr2=0x7f9920197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9927393640 1 --2- 172.21.15.76:0/2273686654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9920197630 0x7f9920197ab0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f991800e960 tx=0x7f991800ee20 comp rx=0 tx=0).stop 2024-03-12T05:56:10.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9927393640 1 -- 172.21.15.76:0/2273686654 shutdown_connections 2024-03-12T05:56:10.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9927393640 1 --2- 172.21.15.76:0/2273686654 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f990c0a1d40 0x7f990c0a4200 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9927393640 1 --2- 172.21.15.76:0/2273686654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9920197630 0x7f9920197ab0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9927393640 1 --2- 172.21.15.76:0/2273686654 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9920105360 0x7f992019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9927393640 1 --2- 172.21.15.76:0/2273686654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9920100290 0x7f992019d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9927393640 1 -- 172.21.15.76:0/2273686654 >> 172.21.15.76:0/2273686654 conn(0x7f9920077230 msgr2=0x7f99200687f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9927393640 1 -- 172.21.15.76:0/2273686654 shutdown_connections 2024-03-12T05:56:10.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.754+0000 7f9927393640 1 -- 172.21.15.76:0/2273686654 wait complete. 2024-03-12T05:56:10.768 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:56:10.769 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:56:10.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.868+0000 7efe2e58a640 1 -- 172.21.15.76:0/2677678818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe28106450 msgr2=0x7efe2810e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.868+0000 7efe2e58a640 1 --2- 172.21.15.76:0/2677678818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe28106450 0x7efe2810e020 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7efe20009cd0 tx=0x7efe2002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.868+0000 7efe2e58a640 1 -- 172.21.15.76:0/2677678818 shutdown_connections 2024-03-12T05:56:10.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.868+0000 7efe2e58a640 1 --2- 172.21.15.76:0/2677678818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe28106450 0x7efe2810e020 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.868+0000 7efe2e58a640 1 --2- 172.21.15.76:0/2677678818 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe28101d30 0x7efe28105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.868+0000 7efe2e58a640 1 --2- 172.21.15.76:0/2677678818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe28101380 0x7efe28101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.868+0000 7efe2e58a640 1 -- 172.21.15.76:0/2677678818 >> 172.21.15.76:0/2677678818 conn(0x7efe280fcc20 msgr2=0x7efe280ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.868+0000 7efe2e58a640 1 -- 172.21.15.76:0/2677678818 shutdown_connections 2024-03-12T05:56:10.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.868+0000 7efe2e58a640 1 -- 172.21.15.76:0/2677678818 wait complete. 2024-03-12T05:56:10.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.869+0000 7efe2e58a640 1 Processor -- start 2024-03-12T05:56:10.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.869+0000 7efe2e58a640 1 -- start start 2024-03-12T05:56:10.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.869+0000 7efe2e58a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe28101380 0x7efe2819d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.869+0000 7efe2e58a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe28106450 0x7efe2819d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.869+0000 7efe2e58a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe281975e0 0x7efe28197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.869+0000 7efe2e58a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe28197fd0 con 0x7efe28106450 2024-03-12T05:56:10.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.869+0000 7efe2e58a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe28198140 con 0x7efe281975e0 2024-03-12T05:56:10.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.869+0000 7efe2e58a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe28198300 con 0x7efe28101380 2024-03-12T05:56:10.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.869+0000 7efe277fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe28106450 0x7efe2819d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.869+0000 7efe277fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe28106450 0x7efe2819d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48282/0 (socket says 172.21.15.76:48282) 2024-03-12T05:56:10.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.869+0000 7efe277fe640 1 -- 172.21.15.76:0/3217993721 learned_addr learned my addr 172.21.15.76:0/3217993721 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.869+0000 7efe2cb27640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe281975e0 0x7efe28197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.870+0000 7efe27fff640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe28101380 0x7efe2819d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.870+0000 7efe277fe640 1 -- 172.21.15.76:0/3217993721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe28101380 msgr2=0x7efe2819d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.870+0000 7efe277fe640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe28101380 0x7efe2819d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.870+0000 7efe277fe640 1 -- 172.21.15.76:0/3217993721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe281975e0 msgr2=0x7efe28197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.870+0000 7efe277fe640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe281975e0 0x7efe28197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.870+0000 7efe277fe640 1 -- 172.21.15.76:0/3217993721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe20009900 con 0x7efe28106450 2024-03-12T05:56:10.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.870+0000 7efe2cb27640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe281975e0 0x7efe28197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:10.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.870+0000 7efe277fe640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe28106450 0x7efe2819d970 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7efe1c00d770 tx=0x7efe1c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.870+0000 7efe257fa640 1 -- 172.21.15.76:0/3217993721 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c009c30 con 0x7efe28106450 2024-03-12T05:56:10.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.870+0000 7efe257fa640 1 -- 172.21.15.76:0/3217993721 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe1c00de80 con 0x7efe28106450 2024-03-12T05:56:10.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.870+0000 7efe257fa640 1 -- 172.21.15.76:0/3217993721 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c00f6b0 con 0x7efe28106450 2024-03-12T05:56:10.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.870+0000 7efe2e58a640 1 -- 172.21.15.76:0/3217993721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe281a42f0 con 0x7efe28106450 2024-03-12T05:56:10.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.870+0000 7efe2e58a640 1 -- 172.21.15.76:0/3217993721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe281a4800 con 0x7efe28106450 2024-03-12T05:56:10.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.872+0000 7efe257fa640 1 -- 172.21.15.76:0/3217993721 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7efe1c010460 con 0x7efe28106450 2024-03-12T05:56:10.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe31235740 1 Processor -- start 2024-03-12T05:56:10.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe31235740 1 -- start start 2024-03-12T05:56:10.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe257fa640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efe100a1cc0 0x7efe100a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe31235740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407cf5060 0x55c407cf5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe31235740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5a10 0x55c407d02620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe31235740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407d02b60 0x55c407d04f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe31235740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c407cb0960 con 0x55c407cf5a10 2024-03-12T05:56:10.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe31235740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c407c64310 con 0x55c407cf5060 2024-03-12T05:56:10.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe31235740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c407c61840 con 0x55c407d02b60 2024-03-12T05:56:10.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe2cb27640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efe100a1cc0 0x7efe100a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe257fa640 1 -- 172.21.15.76:0/3217993721 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efe1c0cd840 con 0x7efe28106450 2024-03-12T05:56:10.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe277fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5a10 0x55c407d02620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe277fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5a10 0x55c407d02620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48296/0 (socket says 172.21.15.76:48296) 2024-03-12T05:56:10.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe277fe640 1 -- 172.21.15.76:0/2814077462 learned_addr learned my addr 172.21.15.76:0/2814077462 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe27fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407cf5060 0x55c407cf5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe277fe640 1 -- 172.21.15.76:0/2814077462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407d02b60 msgr2=0x55c407d04f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe277fe640 1 --2- 172.21.15.76:0/2814077462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407d02b60 0x55c407d04f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe277fe640 1 -- 172.21.15.76:0/2814077462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407cf5060 msgr2=0x55c407cf5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe277fe640 1 --2- 172.21.15.76:0/2814077462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407cf5060 0x55c407cf5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe277fe640 1 -- 172.21.15.76:0/2814077462 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c407c1f060 con 0x55c407cf5a10 2024-03-12T05:56:10.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe27fff640 1 --2- 172.21.15.76:0/2814077462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407cf5060 0x55c407cf5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:10.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe2cb27640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efe100a1cc0 0x7efe100a4180 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efe20006010 tx=0x7efe20032f50 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe277fe640 1 --2- 172.21.15.76:0/2814077462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5a10 0x55c407d02620 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7efe1c009610 tx=0x7efe1c09a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c83d99b168a3eb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.873+0000 7efe0e7fc640 1 -- 172.21.15.76:0/2814077462 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c09e160 con 0x55c407cf5a10 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.874+0000 7efe0e7fc640 1 -- 172.21.15.76:0/2814077462 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe1c0a29f0 con 0x55c407cf5a10 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.874+0000 7efe0e7fc640 1 -- 172.21.15.76:0/2814077462 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c086350 con 0x55c407cf5a10 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.874+0000 7efe31235740 1 -- 172.21.15.76:0/2814077462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5a10 msgr2=0x55c407d02620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.874+0000 7efe31235740 1 --2- 172.21.15.76:0/2814077462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5a10 0x55c407d02620 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7efe1c009610 tx=0x7efe1c09a950 comp rx=0 tx=0).stop 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.874+0000 7efe31235740 1 -- 172.21.15.76:0/2814077462 shutdown_connections 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.874+0000 7efe31235740 1 --2- 172.21.15.76:0/2814077462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407d02b60 0x55c407d04f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.874+0000 7efe31235740 1 --2- 172.21.15.76:0/2814077462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5a10 0x55c407d02620 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.874+0000 7efe31235740 1 --2- 172.21.15.76:0/2814077462 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407cf5060 0x55c407cf5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.874+0000 7efe31235740 1 -- 172.21.15.76:0/2814077462 >> 172.21.15.76:0/2814077462 conn(0x55c407cef980 msgr2=0x55c407cefd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.874+0000 7efe31235740 1 -- 172.21.15.76:0/2814077462 shutdown_connections 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.874+0000 7efe31235740 1 -- 172.21.15.76:0/2814077462 wait complete. 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe31235740 1 -- start start 2024-03-12T05:56:10.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe31235740 1 Processor -- start 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe31235740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5060 0x55c407cffc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe31235740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407cf5a10 0x55c407d00150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe31235740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407d02b60 0x55c407d00690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe31235740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c407c5d020 con 0x55c407d02b60 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe31235740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c407930430 con 0x55c407cf5a10 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe31235740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4078ece00 con 0x55c407cf5060 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe31235740 10 client.0 _ll_register_callbacks cb 0x55c407c04cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe2cb27640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407d02b60 0x55c407d00690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe2cb27640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407d02b60 0x55c407d00690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48306/0 (socket says 172.21.15.76:48306) 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe2cb27640 1 -- 172.21.15.76:0/1670348999 learned_addr learned my addr 172.21.15.76:0/1670348999 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe277fe640 1 --2- 172.21.15.76:0/1670348999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407cf5a10 0x55c407d00150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe27fff640 1 --2- 172.21.15.76:0/1670348999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5060 0x55c407cffc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe2cb27640 1 -- 172.21.15.76:0/1670348999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5060 msgr2=0x55c407cffc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe2cb27640 1 --2- 172.21.15.76:0/1670348999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5060 0x55c407cffc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe2cb27640 1 -- 172.21.15.76:0/1670348999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407cf5a10 msgr2=0x55c407d00150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe2cb27640 1 --2- 172.21.15.76:0/1670348999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407cf5a10 0x55c407d00150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe2cb27640 1 -- 172.21.15.76:0/1670348999 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe281a4800 con 0x55c407d02b60 2024-03-12T05:56:10.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.875+0000 7efe277fe640 1 --2- 172.21.15.76:0/1670348999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407cf5a10 0x55c407d00150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:10.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe2cb27640 1 --2- 172.21.15.76:0/1670348999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407d02b60 0x55c407d00690 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7efe20036000 tx=0x7efe200095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=496812edf9c694b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe077fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:10.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe077fe640 1 -- 172.21.15.76:0/1670348999 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20003b70 con 0x55c407d02b60 2024-03-12T05:56:10.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe31235740 1 -- 172.21.15.76:0/1670348999 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c40795c7e0 con 0x55c407d02b60 2024-03-12T05:56:10.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe077fe640 1 -- 172.21.15.76:0/1670348999 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efe20003cd0 con 0x55c407d02b60 2024-03-12T05:56:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe077fe640 1 -- 172.21.15.76:0/1670348999 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe20003e90 con 0x55c407d02b60 2024-03-12T05:56:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe31235740 10 client.15624 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe31235740 1 -- 172.21.15.76:0/1670348999 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55c407c5b6a0 con 0x55c407d02b60 2024-03-12T05:56:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe077fe640 1 -- 172.21.15.76:0/1670348999 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c40795c7e0 con 0x55c407d02b60 2024-03-12T05:56:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe077fe640 1 -- 172.21.15.76:0/1670348999 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efe20040680 con 0x55c407d02b60 2024-03-12T05:56:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe31235740 10 client.15624 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe31235740 10 client.15624 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe31235740 20 client.15624 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe31235740 1 --2- 172.21.15.76:0/1670348999 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c407d12fe0 0x55c407d153d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe31235740 4 client.15624 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.876+0000 7efe31235740 1 -- 172.21.15.76:0/1670348999 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c407999720 con 0x55c407d12fe0 2024-03-12T05:56:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.877+0000 7efe27fff640 1 --2- 172.21.15.76:0/1670348999 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c407d12fe0 0x55c407d153d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.877+0000 7efe077fe640 10 client.15624 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:10.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.877+0000 7efe27fff640 1 --2- 172.21.15.76:0/1670348999 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c407d12fe0 0x55c407d153d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c80391a224915c7b server_cookie=43d722f03312358f in_seq=0 out_seq=0 2024-03-12T05:56:10.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe077fe640 1 -- 172.21.15.76:0/1670348999 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c407999720 con 0x55c407d12fe0 2024-03-12T05:56:10.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe077fe640 10 client.15624 handle_command_reply: tid=0 2024-03-12T05:56:10.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe31235740 1 client.15624 shutdown 2024-03-12T05:56:10.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe31235740 1 -- 172.21.15.76:0/1670348999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407d02b60 msgr2=0x55c407d00690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe31235740 1 --2- 172.21.15.76:0/1670348999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407d02b60 0x55c407d00690 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7efe20036000 tx=0x7efe200095c0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe31235740 1 -- 172.21.15.76:0/1670348999 shutdown_connections 2024-03-12T05:56:10.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe31235740 1 --2- 172.21.15.76:0/1670348999 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c407d12fe0 0x55c407d153d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe31235740 1 --2- 172.21.15.76:0/1670348999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407d02b60 0x55c407d00690 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe077fe640 0 client.15624 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:10.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe31235740 1 --2- 172.21.15.76:0/1670348999 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407cf5a10 0x55c407d00150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe31235740 1 --2- 172.21.15.76:0/1670348999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5060 0x55c407cffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe31235740 1 -- 172.21.15.76:0/1670348999 >> 172.21.15.76:0/1670348999 conn(0x55c407cef980 msgr2=0x55c407cf16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe31235740 1 -- 172.21.15.76:0/1670348999 shutdown_connections 2024-03-12T05:56:10.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe31235740 1 -- 172.21.15.76:0/1670348999 wait complete. 2024-03-12T05:56:10.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.878+0000 7efe31235740 20 client.15624 trim_cache size 0 max 16384 2024-03-12T05:56:10.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.889+0000 7efe31235740 1 Processor -- start 2024-03-12T05:56:10.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.889+0000 7efe31235740 1 -- start start 2024-03-12T05:56:10.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe31235740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5060 0x55c407d02340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe31235740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5a10 0x55c407d004b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe31235740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407d02b60 0x55c407cfce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe31235740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c407be2130 con 0x55c407cf5a10 2024-03-12T05:56:10.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe31235740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c407ce1a40 con 0x55c407d02b60 2024-03-12T05:56:10.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe31235740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c407cf6020 con 0x55c407cf5060 2024-03-12T05:56:10.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe277fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5a10 0x55c407d004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe277fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5a10 0x55c407d004b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48310/0 (socket says 172.21.15.76:48310) 2024-03-12T05:56:10.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe277fe640 1 -- 172.21.15.76:0/2745147629 learned_addr learned my addr 172.21.15.76:0/2745147629 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe27fff640 1 --2- 172.21.15.76:0/2745147629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5060 0x55c407d02340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe2cb27640 1 --2- 172.21.15.76:0/2745147629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407d02b60 0x55c407cfce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe277fe640 1 -- 172.21.15.76:0/2745147629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5060 msgr2=0x55c407d02340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe277fe640 1 --2- 172.21.15.76:0/2745147629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5060 0x55c407d02340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe277fe640 1 -- 172.21.15.76:0/2745147629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407d02b60 msgr2=0x55c407cfce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe277fe640 1 --2- 172.21.15.76:0/2745147629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407d02b60 0x55c407cfce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe277fe640 1 -- 172.21.15.76:0/2745147629 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c407bf7c20 con 0x55c407cf5a10 2024-03-12T05:56:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe277fe640 1 --2- 172.21.15.76:0/2745147629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5a10 0x55c407d004b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7efe1c09ab00 tx=0x7efe1c010db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22bad21223809620 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe0dffb640 1 -- 172.21.15.76:0/2745147629 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c0cad80 con 0x55c407cf5a10 2024-03-12T05:56:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe0dffb640 1 -- 172.21.15.76:0/2745147629 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe1c09e160 con 0x55c407cf5a10 2024-03-12T05:56:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.890+0000 7efe0dffb640 1 -- 172.21.15.76:0/2745147629 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c093eb0 con 0x55c407cf5a10 2024-03-12T05:56:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.891+0000 7efe31235740 1 -- 172.21.15.76:0/2745147629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5a10 msgr2=0x55c407d004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.891+0000 7efe31235740 1 --2- 172.21.15.76:0/2745147629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5a10 0x55c407d004b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7efe1c09ab00 tx=0x7efe1c010db0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.891+0000 7efe31235740 1 -- 172.21.15.76:0/2745147629 shutdown_connections 2024-03-12T05:56:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.891+0000 7efe31235740 1 --2- 172.21.15.76:0/2745147629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407d02b60 0x55c407cfce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.891+0000 7efe31235740 1 --2- 172.21.15.76:0/2745147629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5a10 0x55c407d004b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.891+0000 7efe31235740 1 --2- 172.21.15.76:0/2745147629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5060 0x55c407d02340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.891+0000 7efe31235740 1 -- 172.21.15.76:0/2745147629 >> 172.21.15.76:0/2745147629 conn(0x55c407cef980 msgr2=0x55c407cfaf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.891+0000 7efe31235740 1 -- 172.21.15.76:0/2745147629 shutdown_connections 2024-03-12T05:56:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.891+0000 7efe31235740 1 -- 172.21.15.76:0/2745147629 wait complete. 2024-03-12T05:56:10.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.891+0000 7efe31235740 1 -- start start 2024-03-12T05:56:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.891+0000 7efe31235740 1 Processor -- start 2024-03-12T05:56:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe31235740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5060 0x55c407d08580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe31235740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5a10 0x55c407d08ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe31235740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407d02b60 0x55c407d09000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe31235740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c407d69540 con 0x55c407cf5060 2024-03-12T05:56:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe31235740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c407d696b0 con 0x55c407d02b60 2024-03-12T05:56:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe31235740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c407d69870 con 0x55c407cf5a10 2024-03-12T05:56:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe31235740 10 client.0 _ll_register_callbacks cb 0x7efe1c0964f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe27fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5060 0x55c407d08580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe27fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5060 0x55c407d08580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48326/0 (socket says 172.21.15.76:48326) 2024-03-12T05:56:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe27fff640 1 -- 172.21.15.76:0/2517421854 learned_addr learned my addr 172.21.15.76:0/2517421854 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe2cb27640 1 --2- 172.21.15.76:0/2517421854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407d02b60 0x55c407d09000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe277fe640 1 --2- 172.21.15.76:0/2517421854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5a10 0x55c407d08ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe27fff640 1 -- 172.21.15.76:0/2517421854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5a10 msgr2=0x55c407d08ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe27fff640 1 --2- 172.21.15.76:0/2517421854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5a10 0x55c407d08ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe27fff640 1 -- 172.21.15.76:0/2517421854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407d02b60 msgr2=0x55c407d09000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:10.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe27fff640 1 --2- 172.21.15.76:0/2517421854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407d02b60 0x55c407d09000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe27fff640 1 -- 172.21.15.76:0/2517421854 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c407c1f060 con 0x55c407cf5060 2024-03-12T05:56:10.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe2cb27640 1 --2- 172.21.15.76:0/2517421854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407d02b60 0x55c407d09000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:10.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe27fff640 1 --2- 172.21.15.76:0/2517421854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5060 0x55c407d08580 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7efe1800bcd0 tx=0x7efe1801c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a84ff3e193b2b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:10.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe077fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:10.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe077fe640 1 -- 172.21.15.76:0/2517421854 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18017040 con 0x55c407cf5060 2024-03-12T05:56:10.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe077fe640 1 -- 172.21.15.76:0/2517421854 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efe1801c8d0 con 0x55c407cf5060 2024-03-12T05:56:10.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.892+0000 7efe31235740 1 -- 172.21.15.76:0/2517421854 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efe1c0a29f0 con 0x55c407cf5060 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.893+0000 7efe077fe640 1 -- 172.21.15.76:0/2517421854 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe18011d80 con 0x55c407cf5060 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.893+0000 7efe31235740 10 client.15630 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.893+0000 7efe31235740 1 -- 172.21.15.76:0/2517421854 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55c407c1d8c0 con 0x55c407cf5060 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.893+0000 7efe077fe640 1 -- 172.21.15.76:0/2517421854 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe1c0a29f0 con 0x55c407cf5060 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.893+0000 7efe077fe640 1 -- 172.21.15.76:0/2517421854 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efe18010630 con 0x55c407cf5060 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.893+0000 7efe31235740 10 client.15630 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.893+0000 7efe31235740 10 client.15630 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.893+0000 7efe31235740 20 client.15630 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.893+0000 7efe31235740 1 --2- 172.21.15.76:0/2517421854 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c407d6b100 0x55c407d6d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.893+0000 7efe31235740 4 client.15630 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.893+0000 7efe31235740 1 -- 172.21.15.76:0/2517421854 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c4079e8890 con 0x55c407d6b100 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.893+0000 7efe277fe640 1 --2- 172.21.15.76:0/2517421854 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c407d6b100 0x55c407d6d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.893+0000 7efe077fe640 10 client.15630 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.893+0000 7efe277fe640 1 --2- 172.21.15.76:0/2517421854 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c407d6b100 0x55c407d6d520 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b53c9a4c6691cf9 server_cookie=99b1610db1f2a13e in_seq=0 out_seq=0 2024-03-12T05:56:10.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe077fe640 1 -- 172.21.15.76:0/2517421854 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c4079e8890 con 0x55c407d6b100 2024-03-12T05:56:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe077fe640 10 client.15630 handle_command_reply: tid=0 2024-03-12T05:56:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe31235740 1 client.15630 shutdown 2024-03-12T05:56:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe31235740 1 -- 172.21.15.76:0/2517421854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5060 msgr2=0x55c407d08580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe31235740 1 --2- 172.21.15.76:0/2517421854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5060 0x55c407d08580 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7efe1800bcd0 tx=0x7efe1801c680 comp rx=0 tx=0).stop 2024-03-12T05:56:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe31235740 1 -- 172.21.15.76:0/2517421854 shutdown_connections 2024-03-12T05:56:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe31235740 1 --2- 172.21.15.76:0/2517421854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c407d02b60 0x55c407d09000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe31235740 1 --2- 172.21.15.76:0/2517421854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c407cf5a10 0x55c407d08ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe31235740 1 --2- 172.21.15.76:0/2517421854 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c407d6b100 0x55c407d6d520 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe31235740 1 --2- 172.21.15.76:0/2517421854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c407cf5060 0x55c407d08580 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe077fe640 0 client.15630 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe31235740 1 -- 172.21.15.76:0/2517421854 >> 172.21.15.76:0/2517421854 conn(0x55c407cef980 msgr2=0x55c407cee2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe31235740 1 -- 172.21.15.76:0/2517421854 shutdown_connections 2024-03-12T05:56:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe31235740 1 -- 172.21.15.76:0/2517421854 wait complete. 2024-03-12T05:56:10.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe31235740 20 client.15630 trim_cache size 0 max 16384 2024-03-12T05:56:10.910 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:56:10.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe2e58a640 1 -- 172.21.15.76:0/3217993721 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efe100a1cc0 msgr2=0x7efe100a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe2e58a640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efe100a1cc0 0x7efe100a4180 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efe20006010 tx=0x7efe20032f50 comp rx=0 tx=0).stop 2024-03-12T05:56:10.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe2e58a640 1 -- 172.21.15.76:0/3217993721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe28106450 msgr2=0x7efe2819d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:10.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.894+0000 7efe2e58a640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe28106450 0x7efe2819d970 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7efe1c00d770 tx=0x7efe1c00dc30 comp rx=0 tx=0).stop 2024-03-12T05:56:10.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.895+0000 7efe2e58a640 1 -- 172.21.15.76:0/3217993721 shutdown_connections 2024-03-12T05:56:10.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.895+0000 7efe2e58a640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efe100a1cc0 0x7efe100a4180 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.895+0000 7efe2e58a640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe281975e0 0x7efe28197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.895+0000 7efe2e58a640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe28106450 0x7efe2819d970 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.895+0000 7efe2e58a640 1 --2- 172.21.15.76:0/3217993721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe28101380 0x7efe2819d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:10.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.895+0000 7efe2e58a640 1 -- 172.21.15.76:0/3217993721 >> 172.21.15.76:0/3217993721 conn(0x7efe280fcc20 msgr2=0x7efe280fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:10.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.895+0000 7efe2e58a640 1 -- 172.21.15.76:0/3217993721 shutdown_connections 2024-03-12T05:56:10.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:10.895+0000 7efe2e58a640 1 -- 172.21.15.76:0/3217993721 wait complete. 2024-03-12T05:56:11.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:11 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2963663923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:56:11.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:11 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2963663923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:56:11.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:11 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2963663923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:56:11.913 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:56:11.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:56:11.914 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:56:12.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.023+0000 7f772464e640 1 -- 172.21.15.76:0/860627001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771c101d30 msgr2=0x7f771c105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.023+0000 7f772464e640 1 --2- 172.21.15.76:0/860627001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771c101d30 0x7f771c105d70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7710009930 tx=0x7f771002bc30 comp rx=0 tx=0).stop 2024-03-12T05:56:12.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.023+0000 7f772464e640 1 -- 172.21.15.76:0/860627001 shutdown_connections 2024-03-12T05:56:12.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.023+0000 7f772464e640 1 --2- 172.21.15.76:0/860627001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c106450 0x7f771c10e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.023+0000 7f772464e640 1 --2- 172.21.15.76:0/860627001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771c101d30 0x7f771c105d70 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.023+0000 7f772464e640 1 --2- 172.21.15.76:0/860627001 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f771c101380 0x7f771c101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.023+0000 7f772464e640 1 -- 172.21.15.76:0/860627001 >> 172.21.15.76:0/860627001 conn(0x7f771c0fcc20 msgr2=0x7f771c0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.023+0000 7f772464e640 1 -- 172.21.15.76:0/860627001 shutdown_connections 2024-03-12T05:56:12.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.023+0000 7f772464e640 1 -- 172.21.15.76:0/860627001 wait complete. 2024-03-12T05:56:12.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f772464e640 1 Processor -- start 2024-03-12T05:56:12.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f772464e640 1 -- start start 2024-03-12T05:56:12.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f772464e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c101380 0x7f771c19d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f772464e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f771c106450 0x7f771c19dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f772464e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771c197650 0x7f771c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f772464e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c198120 con 0x7f771c197650 2024-03-12T05:56:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f772464e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c198290 con 0x7f771c106450 2024-03-12T05:56:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f772464e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c198450 con 0x7f771c101380 2024-03-12T05:56:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f7722beb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771c197650 0x7f771c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f7722beb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771c197650 0x7f771c197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57222/0 (socket says 172.21.15.76:57222) 2024-03-12T05:56:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f7722beb640 1 -- 172.21.15.76:0/1818313611 learned_addr learned my addr 172.21.15.76:0/1818313611 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f7721be9640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f771c106450 0x7f771c19dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f77223ea640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c101380 0x7f771c19d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f7722beb640 1 -- 172.21.15.76:0/1818313611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c101380 msgr2=0x7f771c19d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f7722beb640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c101380 0x7f771c19d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f7722beb640 1 -- 172.21.15.76:0/1818313611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f771c106450 msgr2=0x7f771c19dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f7722beb640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f771c106450 0x7f771c19dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f7722beb640 1 -- 172.21.15.76:0/1818313611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7710009560 con 0x7f771c197650 2024-03-12T05:56:12.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.024+0000 7f77223ea640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c101380 0x7f771c19d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:56:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.025+0000 7f7722beb640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771c197650 0x7f771c197ad0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f770c00d950 tx=0x7f770c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.025+0000 7f771b7fe640 1 -- 172.21.15.76:0/1818313611 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c009a30 con 0x7f771c197650 2024-03-12T05:56:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.025+0000 7f771b7fe640 1 -- 172.21.15.76:0/1818313611 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f770c009b90 con 0x7f771c197650 2024-03-12T05:56:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.025+0000 7f772464e640 1 -- 172.21.15.76:0/1818313611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c1a43f0 con 0x7f771c197650 2024-03-12T05:56:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.025+0000 7f771b7fe640 1 -- 172.21.15.76:0/1818313611 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c003d60 con 0x7f771c197650 2024-03-12T05:56:12.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.025+0000 7f772464e640 1 -- 172.21.15.76:0/1818313611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c1a48d0 con 0x7f771c197650 2024-03-12T05:56:12.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.027+0000 7f771b7fe640 1 -- 172.21.15.76:0/1818313611 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f770c009d00 con 0x7f771c197650 2024-03-12T05:56:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.027+0000 7f7727224740 1 Processor -- start 2024-03-12T05:56:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.027+0000 7f7727224740 1 -- start start 2024-03-12T05:56:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f771b7fe640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f77040a1d40 0x7f77040a4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f7727224740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696380 0x5574da696760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f7727224740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da696d30 0x5574da69f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f7727224740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da69f8b0 0x5574da6a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f7727224740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574da604f90 con 0x5574da696d30 2024-03-12T05:56:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f7727224740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574da6024c0 con 0x5574da696380 2024-03-12T05:56:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f7727224740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574da5fdca0 con 0x5574da69f8b0 2024-03-12T05:56:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f7721be9640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f77040a1d40 0x7f77040a4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f771b7fe640 1 -- 172.21.15.76:0/1818313611 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f770c0cd440 con 0x7f771c197650 2024-03-12T05:56:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f7721be9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da696d30 0x5574da69f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f77223ea640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696380 0x5574da696760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f77223ea640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696380 0x5574da696760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:45264/0 (socket says 172.21.15.76:45264) 2024-03-12T05:56:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f7721be9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da696d30 0x5574da69f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57224/0 (socket says 172.21.15.76:57224) 2024-03-12T05:56:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f77223ea640 1 -- 172.21.15.76:0/2643217720 learned_addr learned my addr 172.21.15.76:0/2643217720 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f7722beb640 1 --2- 172.21.15.76:0/2643217720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da69f8b0 0x5574da6a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f77223ea640 1 -- 172.21.15.76:0/2643217720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da69f8b0 msgr2=0x5574da6a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f77223ea640 1 --2- 172.21.15.76:0/2643217720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da69f8b0 0x5574da6a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f77223ea640 1 -- 172.21.15.76:0/2643217720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da696d30 msgr2=0x5574da69f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f77223ea640 1 --2- 172.21.15.76:0/2643217720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da696d30 0x5574da69f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f77223ea640 1 -- 172.21.15.76:0/2643217720 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574da5c00b0 con 0x5574da696380 2024-03-12T05:56:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f7721be9640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f77040a1d40 0x7f77040a4200 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7710000c00 tx=0x7f7710004750 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.028+0000 7f77223ea640 1 --2- 172.21.15.76:0/2643217720 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696380 0x5574da696760 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7714009f20 tx=0x7f771400f7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=884bd70eb4faed0a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.029+0000 7f7718bfe640 1 -- 172.21.15.76:0/2643217720 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714003b70 con 0x5574da696380 2024-03-12T05:56:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.029+0000 7f7718bfe640 1 -- 172.21.15.76:0/2643217720 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7714003cd0 con 0x5574da696380 2024-03-12T05:56:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.029+0000 7f7718bfe640 1 -- 172.21.15.76:0/2643217720 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714011820 con 0x5574da696380 2024-03-12T05:56:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.029+0000 7f7727224740 1 -- 172.21.15.76:0/2643217720 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696380 msgr2=0x5574da696760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.029+0000 7f7727224740 1 --2- 172.21.15.76:0/2643217720 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696380 0x5574da696760 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7714009f20 tx=0x7f771400f7f0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.029+0000 7f7727224740 1 -- 172.21.15.76:0/2643217720 shutdown_connections 2024-03-12T05:56:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.029+0000 7f7727224740 1 --2- 172.21.15.76:0/2643217720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da69f8b0 0x5574da6a1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.029+0000 7f7727224740 1 --2- 172.21.15.76:0/2643217720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da696d30 0x5574da69f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.029+0000 7f7727224740 1 --2- 172.21.15.76:0/2643217720 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696380 0x5574da696760 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.029+0000 7f7727224740 1 -- 172.21.15.76:0/2643217720 >> 172.21.15.76:0/2643217720 conn(0x5574da690840 msgr2=0x5574da690c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f7727224740 1 -- 172.21.15.76:0/2643217720 shutdown_connections 2024-03-12T05:56:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f7727224740 1 -- 172.21.15.76:0/2643217720 wait complete. 2024-03-12T05:56:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f7727224740 1 -- start start 2024-03-12T05:56:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f7727224740 1 Processor -- start 2024-03-12T05:56:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f7727224740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696380 0x5574da69c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f7727224740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da696d30 0x5574da69cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f7727224740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da69f8b0 0x5574da69d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f7727224740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574da5a2f40 con 0x5574da69f8b0 2024-03-12T05:56:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f7727224740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574da2d14b0 con 0x5574da696380 2024-03-12T05:56:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f7727224740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574da583670 con 0x5574da696d30 2024-03-12T05:56:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f7727224740 10 client.0 _ll_register_callbacks cb 0x5574da5f0f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f7722beb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da69f8b0 0x5574da69d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f7722beb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da69f8b0 0x5574da69d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57236/0 (socket says 172.21.15.76:57236) 2024-03-12T05:56:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f7722beb640 1 -- 172.21.15.76:0/1375597667 learned_addr learned my addr 172.21.15.76:0/1375597667 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.030+0000 7f77223ea640 1 --2- 172.21.15.76:0/1375597667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696380 0x5574da69c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7722beb640 1 -- 172.21.15.76:0/1375597667 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da696d30 msgr2=0x5574da69cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7721be9640 1 --2- 172.21.15.76:0/1375597667 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da696d30 0x5574da69cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7722beb640 1 --2- 172.21.15.76:0/1375597667 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da696d30 0x5574da69cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7722beb640 1 -- 172.21.15.76:0/1375597667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696380 msgr2=0x5574da69c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7722beb640 1 --2- 172.21.15.76:0/1375597667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696380 0x5574da69c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7722beb640 1 -- 172.21.15.76:0/1375597667 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7714009b50 con 0x5574da69f8b0 2024-03-12T05:56:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f77223ea640 1 --2- 172.21.15.76:0/1375597667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696380 0x5574da69c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7722beb640 1 --2- 172.21.15.76:0/1375597667 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da69f8b0 0x5574da69d330 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f770c099b50 tx=0x7f770c0998b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d16a354ce1fd368e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7701ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7701ffb640 1 -- 172.21.15.76:0/1375597667 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c0adea0 con 0x5574da69f8b0 2024-03-12T05:56:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7701ffb640 1 -- 172.21.15.76:0/1375597667 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f770c09dfa0 con 0x5574da69f8b0 2024-03-12T05:56:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7727224740 1 -- 172.21.15.76:0/1375597667 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5574da5d7f40 con 0x5574da69f8b0 2024-03-12T05:56:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7701ffb640 1 -- 172.21.15.76:0/1375597667 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f770c083f50 con 0x5574da69f8b0 2024-03-12T05:56:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7727224740 10 client.15642 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7727224740 1 -- 172.21.15.76:0/1375597667 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5574da5fc320 con 0x5574da69f8b0 2024-03-12T05:56:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7701ffb640 1 -- 172.21.15.76:0/1375597667 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5574da5d7f40 con 0x5574da69f8b0 2024-03-12T05:56:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7701ffb640 1 -- 172.21.15.76:0/1375597667 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f770c0cae40 con 0x5574da69f8b0 2024-03-12T05:56:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7727224740 10 client.15642 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7727224740 10 client.15642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7727224740 20 client.15642 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7727224740 1 --2- 172.21.15.76:0/1375597667 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5574da6afd70 0x5574da6b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7727224740 4 client.15642 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.031+0000 7f7727224740 1 -- 172.21.15.76:0/1375597667 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5574da5f4310 con 0x5574da6afd70 2024-03-12T05:56:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.032+0000 7f77223ea640 1 --2- 172.21.15.76:0/1375597667 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5574da6afd70 0x5574da6b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.032+0000 7f7701ffb640 10 client.15642 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.032+0000 7f77223ea640 1 --2- 172.21.15.76:0/1375597667 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5574da6afd70 0x5574da6b21a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28504fd440113bee server_cookie=dc371dc05c2b8d85 in_seq=0 out_seq=0 2024-03-12T05:56:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7701ffb640 1 -- 172.21.15.76:0/1375597667 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5574da5f4310 con 0x5574da6afd70 2024-03-12T05:56:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7701ffb640 10 client.15642 handle_command_reply: tid=0 2024-03-12T05:56:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7727224740 1 client.15642 shutdown 2024-03-12T05:56:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7727224740 1 -- 172.21.15.76:0/1375597667 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da69f8b0 msgr2=0x5574da69d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7727224740 1 --2- 172.21.15.76:0/1375597667 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da69f8b0 0x5574da69d330 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f770c099b50 tx=0x7f770c0998b0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7727224740 1 -- 172.21.15.76:0/1375597667 shutdown_connections 2024-03-12T05:56:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7727224740 1 --2- 172.21.15.76:0/1375597667 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5574da6afd70 0x5574da6b21a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7727224740 1 --2- 172.21.15.76:0/1375597667 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da69f8b0 0x5574da69d330 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7701ffb640 0 client.15642 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7727224740 1 --2- 172.21.15.76:0/1375597667 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da696d30 0x5574da69cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7727224740 1 --2- 172.21.15.76:0/1375597667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696380 0x5574da69c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7727224740 1 -- 172.21.15.76:0/1375597667 >> 172.21.15.76:0/1375597667 conn(0x5574da690840 msgr2=0x5574da6927a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7727224740 1 -- 172.21.15.76:0/1375597667 shutdown_connections 2024-03-12T05:56:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7727224740 1 -- 172.21.15.76:0/1375597667 wait complete. 2024-03-12T05:56:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.033+0000 7f7727224740 20 client.15642 trim_cache size 0 max 16384 2024-03-12T05:56:12.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.048+0000 7f7727224740 1 Processor -- start 2024-03-12T05:56:12.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.048+0000 7f7727224740 1 -- start start 2024-03-12T05:56:12.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.048+0000 7f7727224740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da696380 0x5574da697c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.048+0000 7f7727224740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696d30 0x5574da698140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.048+0000 7f7727224740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da69f8b0 0x5574da69ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.048+0000 7f7727224740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574da69dd40 con 0x5574da69f8b0 2024-03-12T05:56:12.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.048+0000 7f7727224740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574da6900d0 con 0x5574da696d30 2024-03-12T05:56:12.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.048+0000 7f7727224740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574da690370 con 0x5574da696380 2024-03-12T05:56:12.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.048+0000 7f7722beb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da69f8b0 0x5574da69ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.048+0000 7f7722beb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da69f8b0 0x5574da69ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57246/0 (socket says 172.21.15.76:57246) 2024-03-12T05:56:12.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.048+0000 7f7722beb640 1 -- 172.21.15.76:0/58604381 learned_addr learned my addr 172.21.15.76:0/58604381 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f77223ea640 1 --2- 172.21.15.76:0/58604381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da696380 0x5574da697c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f7721be9640 1 --2- 172.21.15.76:0/58604381 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696d30 0x5574da698140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f7722beb640 1 -- 172.21.15.76:0/58604381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da696380 msgr2=0x5574da697c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f7722beb640 1 --2- 172.21.15.76:0/58604381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da696380 0x5574da697c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f7722beb640 1 -- 172.21.15.76:0/58604381 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696d30 msgr2=0x5574da698140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f7722beb640 1 --2- 172.21.15.76:0/58604381 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696d30 0x5574da698140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f7722beb640 1 -- 172.21.15.76:0/58604381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574da598c00 con 0x5574da69f8b0 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f77223ea640 1 --2- 172.21.15.76:0/58604381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da696380 0x5574da697c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f7721be9640 1 --2- 172.21.15.76:0/58604381 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696d30 0x5574da698140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f7722beb640 1 --2- 172.21.15.76:0/58604381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da69f8b0 0x5574da69ce20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f770c099920 tx=0x7f770c0960f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f227ee9d6ea95c8e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f7703fff640 1 -- 172.21.15.76:0/58604381 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c0adea0 con 0x5574da69f8b0 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f7703fff640 1 -- 172.21.15.76:0/58604381 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f770c099990 con 0x5574da69f8b0 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f7703fff640 1 -- 172.21.15.76:0/58604381 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c0b83e0 con 0x5574da69f8b0 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f7727224740 1 -- 172.21.15.76:0/58604381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da69f8b0 msgr2=0x5574da69ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.049+0000 7f7727224740 1 --2- 172.21.15.76:0/58604381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da69f8b0 0x5574da69ce20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f770c099920 tx=0x7f770c0960f0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 -- 172.21.15.76:0/58604381 shutdown_connections 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 --2- 172.21.15.76:0/58604381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da69f8b0 0x5574da69ce20 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 --2- 172.21.15.76:0/58604381 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696d30 0x5574da698140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 --2- 172.21.15.76:0/58604381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da696380 0x5574da697c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 -- 172.21.15.76:0/58604381 >> 172.21.15.76:0/58604381 conn(0x5574da690840 msgr2=0x5574da697110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 -- 172.21.15.76:0/58604381 shutdown_connections 2024-03-12T05:56:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 -- 172.21.15.76:0/58604381 wait complete. 2024-03-12T05:56:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 -- start start 2024-03-12T05:56:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 Processor -- start 2024-03-12T05:56:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da696380 0x5574da6a5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696d30 0x5574da6a5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da69f8b0 0x5574da6a5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574da70a6a0 con 0x5574da696380 2024-03-12T05:56:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574da70a810 con 0x5574da696d30 2024-03-12T05:56:12.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574da70a9d0 con 0x5574da69f8b0 2024-03-12T05:56:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f7727224740 10 client.0 _ll_register_callbacks cb 0x7f77140096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f77223ea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da696380 0x5574da6a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f77223ea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da696380 0x5574da6a5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57248/0 (socket says 172.21.15.76:57248) 2024-03-12T05:56:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f77223ea640 1 -- 172.21.15.76:0/3026918731 learned_addr learned my addr 172.21.15.76:0/3026918731 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f77223ea640 1 -- 172.21.15.76:0/3026918731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da69f8b0 msgr2=0x5574da6a5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f77223ea640 1 --2- 172.21.15.76:0/3026918731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da69f8b0 0x5574da6a5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f77223ea640 1 -- 172.21.15.76:0/3026918731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696d30 msgr2=0x5574da6a5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f77223ea640 1 --2- 172.21.15.76:0/3026918731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696d30 0x5574da6a5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.050+0000 7f77223ea640 1 -- 172.21.15.76:0/3026918731 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574da5fc320 con 0x5574da696380 2024-03-12T05:56:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f77223ea640 1 --2- 172.21.15.76:0/3026918731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da696380 0x5574da6a5410 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7714006010 tx=0x7f7714011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a564cac8e2e9ee4c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7701ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7701ffb640 1 -- 172.21.15.76:0/3026918731 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714004e10 con 0x5574da696380 2024-03-12T05:56:12.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7701ffb640 1 -- 172.21.15.76:0/3026918731 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7714015070 con 0x5574da696380 2024-03-12T05:56:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7727224740 1 -- 172.21.15.76:0/3026918731 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5574da623900 con 0x5574da696380 2024-03-12T05:56:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7701ffb640 1 -- 172.21.15.76:0/3026918731 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7714004820 con 0x5574da696380 2024-03-12T05:56:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7727224740 10 client.15648 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7727224740 1 -- 172.21.15.76:0/3026918731 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5574da5be910 con 0x5574da696380 2024-03-12T05:56:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7701ffb640 1 -- 172.21.15.76:0/3026918731 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7714036020 con 0x5574da696380 2024-03-12T05:56:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7701ffb640 1 -- 172.21.15.76:0/3026918731 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7714005680 con 0x5574da696380 2024-03-12T05:56:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7727224740 10 client.15648 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7727224740 10 client.15648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7727224740 20 client.15648 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7727224740 1 --2- 172.21.15.76:0/3026918731 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5574da70c1e0 0x5574da70e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7727224740 4 client.15648 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:56:12.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7727224740 1 -- 172.21.15.76:0/3026918731 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5574da2fd860 con 0x5574da70c1e0 2024-03-12T05:56:12.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.051+0000 7f7721be9640 1 --2- 172.21.15.76:0/3026918731 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5574da70c1e0 0x5574da70e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.052+0000 7f7701ffb640 10 client.15648 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:12.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.052+0000 7f7721be9640 1 --2- 172.21.15.76:0/3026918731 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5574da70c1e0 0x5574da70e600 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4551ba0a5bd43297 server_cookie=b14cc4e588f51de5 in_seq=0 out_seq=0 2024-03-12T05:56:12.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7701ffb640 1 -- 172.21.15.76:0/3026918731 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5574da2fd860 con 0x5574da70c1e0 2024-03-12T05:56:12.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7701ffb640 10 client.15648 handle_command_reply: tid=0 2024-03-12T05:56:12.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7727224740 1 client.15648 shutdown 2024-03-12T05:56:12.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7727224740 1 -- 172.21.15.76:0/3026918731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da696380 msgr2=0x5574da6a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7727224740 1 --2- 172.21.15.76:0/3026918731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da696380 0x5574da6a5410 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7714006010 tx=0x7f7714011440 comp rx=0 tx=0).stop 2024-03-12T05:56:12.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7727224740 1 -- 172.21.15.76:0/3026918731 shutdown_connections 2024-03-12T05:56:12.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7727224740 1 --2- 172.21.15.76:0/3026918731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574da69f8b0 0x5574da6a5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7727224740 1 --2- 172.21.15.76:0/3026918731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574da696d30 0x5574da6a5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7727224740 1 --2- 172.21.15.76:0/3026918731 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5574da70c1e0 0x5574da70e600 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7727224740 1 --2- 172.21.15.76:0/3026918731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574da696380 0x5574da6a5410 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7727224740 1 -- 172.21.15.76:0/3026918731 >> 172.21.15.76:0/3026918731 conn(0x5574da690840 msgr2=0x5574da697c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7701ffb640 0 client.15648 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:12.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7727224740 1 -- 172.21.15.76:0/3026918731 shutdown_connections 2024-03-12T05:56:12.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7727224740 1 -- 172.21.15.76:0/3026918731 wait complete. 2024-03-12T05:56:12.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.053+0000 7f7727224740 20 client.15648 trim_cache size 0 max 16384 2024-03-12T05:56:12.064 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:56:12.065 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T05:56:12.065 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "a9b92593-d785-441e-a5df-72974968c6e7", 2024-03-12T05:56:12.065 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T05:56:12.065 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:56:12.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.054+0000 7f772464e640 1 -- 172.21.15.76:0/1818313611 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f77040a1d40 msgr2=0x7f77040a4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.054+0000 7f772464e640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f77040a1d40 0x7f77040a4200 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7710000c00 tx=0x7f7710004750 comp rx=0 tx=0).stop 2024-03-12T05:56:12.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.054+0000 7f772464e640 1 -- 172.21.15.76:0/1818313611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771c197650 msgr2=0x7f771c197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.054+0000 7f772464e640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771c197650 0x7f771c197ad0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f770c00d950 tx=0x7f770c00de10 comp rx=0 tx=0).stop 2024-03-12T05:56:12.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.054+0000 7f772464e640 1 -- 172.21.15.76:0/1818313611 shutdown_connections 2024-03-12T05:56:12.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.054+0000 7f772464e640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f77040a1d40 0x7f77040a4200 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.054+0000 7f772464e640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771c197650 0x7f771c197ad0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.054+0000 7f772464e640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f771c106450 0x7f771c19dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.054+0000 7f772464e640 1 --2- 172.21.15.76:0/1818313611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f771c101380 0x7f771c19d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.054+0000 7f772464e640 1 -- 172.21.15.76:0/1818313611 >> 172.21.15.76:0/1818313611 conn(0x7f771c0fcc20 msgr2=0x7f771c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.054+0000 7f772464e640 1 -- 172.21.15.76:0/1818313611 shutdown_connections 2024-03-12T05:56:12.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.054+0000 7f772464e640 1 -- 172.21.15.76:0/1818313611 wait complete. 2024-03-12T05:56:12.068 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:56:12.068 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:56:12.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.168+0000 7f691121c640 1 -- 172.21.15.76:0/915195151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690c105340 msgr2=0x7f690c10cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.168+0000 7f691121c640 1 --2- 172.21.15.76:0/915195151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690c105340 0x7f690c10cf10 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009930 tx=0x7f68fc02bc30 comp rx=0 tx=0).stop 2024-03-12T05:56:12.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.168+0000 7f691121c640 1 -- 172.21.15.76:0/915195151 shutdown_connections 2024-03-12T05:56:12.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.168+0000 7f691121c640 1 --2- 172.21.15.76:0/915195151 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690c105340 0x7f690c10cf10 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.168+0000 7f691121c640 1 --2- 172.21.15.76:0/915195151 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f690c100c30 0x7f690c104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.168+0000 7f691121c640 1 --2- 172.21.15.76:0/915195151 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f690c100280 0x7f690c100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.168+0000 7f691121c640 1 -- 172.21.15.76:0/915195151 >> 172.21.15.76:0/915195151 conn(0x7f690c077120 msgr2=0x7f690c077530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.168+0000 7f691121c640 1 -- 172.21.15.76:0/915195151 shutdown_connections 2024-03-12T05:56:12.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.168+0000 7f691121c640 1 -- 172.21.15.76:0/915195151 wait complete. 2024-03-12T05:56:12.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f691121c640 1 Processor -- start 2024-03-12T05:56:12.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f691121c640 1 -- start start 2024-03-12T05:56:12.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f691121c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690c100280 0x7f690c19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f691121c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f690c105340 0x7f690c19fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f691121c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f690c199740 0x7f690c199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f691121c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c19a210 con 0x7f690c100280 2024-03-12T05:56:12.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f691121c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c19a380 con 0x7f690c105340 2024-03-12T05:56:12.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f691121c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c19a540 con 0x7f690c199740 2024-03-12T05:56:12.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f690ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690c100280 0x7f690c19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f690ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690c100280 0x7f690c19f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57264/0 (socket says 172.21.15.76:57264) 2024-03-12T05:56:12.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f690ad9d640 1 -- 172.21.15.76:0/3074072260 learned_addr learned my addr 172.21.15.76:0/3074072260 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f690a59c640 1 --2- 172.21.15.76:0/3074072260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f690c105340 0x7f690c19fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f690b59e640 1 --2- 172.21.15.76:0/3074072260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f690c199740 0x7f690c199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f690ad9d640 1 -- 172.21.15.76:0/3074072260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f690c199740 msgr2=0x7f690c199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f690ad9d640 1 --2- 172.21.15.76:0/3074072260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f690c199740 0x7f690c199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f690ad9d640 1 -- 172.21.15.76:0/3074072260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f690c105340 msgr2=0x7f690c19fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f690ad9d640 1 --2- 172.21.15.76:0/3074072260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f690c105340 0x7f690c19fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.176 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f690ad9d640 1 -- 172.21.15.76:0/3074072260 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68fc009560 con 0x7f690c100280 2024-03-12T05:56:12.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.169+0000 7f690ad9d640 1 --2- 172.21.15.76:0/3074072260 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690c100280 0x7f690c19f630 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f690400d770 tx=0x7f690400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.170+0000 7f68f3fff640 1 -- 172.21.15.76:0/3074072260 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6904009c30 con 0x7f690c100280 2024-03-12T05:56:12.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.170+0000 7f68f3fff640 1 -- 172.21.15.76:0/3074072260 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f690400de80 con 0x7f690c100280 2024-03-12T05:56:12.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.170+0000 7f691121c640 1 -- 172.21.15.76:0/3074072260 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f690c1a6490 con 0x7f690c100280 2024-03-12T05:56:12.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.170+0000 7f68f3fff640 1 -- 172.21.15.76:0/3074072260 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6904004e80 con 0x7f690c100280 2024-03-12T05:56:12.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.170+0000 7f691121c640 1 -- 172.21.15.76:0/3074072260 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f690c1a6920 con 0x7f690c100280 2024-03-12T05:56:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f68f3fff640 1 -- 172.21.15.76:0/3074072260 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f6904005020 con 0x7f690c100280 2024-03-12T05:56:12.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f6913df2740 1 Processor -- start 2024-03-12T05:56:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f6913df2740 1 -- start start 2024-03-12T05:56:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f6913df2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b32b0 0x5600381b3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f6913df2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b3bd0 0x5600381bc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f6913df2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381beba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f6913df2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560038122280 con 0x5600381b3bd0 2024-03-12T05:56:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f6913df2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56003811f7b0 con 0x5600381b32b0 2024-03-12T05:56:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f6913df2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56003811af90 con 0x5600381bc7b0 2024-03-12T05:56:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f690ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b3bd0 0x5600381bc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f690ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b3bd0 0x5600381bc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57278/0 (socket says 172.21.15.76:57278) 2024-03-12T05:56:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f690ad9d640 1 -- 172.21.15.76:0/1334582713 learned_addr learned my addr 172.21.15.76:0/1334582713 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f690b59e640 1 --2- 172.21.15.76:0/1334582713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b32b0 0x5600381b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f690a59c640 1 --2- 172.21.15.76:0/1334582713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381beba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f68f3fff640 1 --2- 172.21.15.76:0/3074072260 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f68f40a1cc0 0x7f68f40a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f690ad9d640 1 -- 172.21.15.76:0/1334582713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 msgr2=0x5600381beba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f690ad9d640 1 --2- 172.21.15.76:0/1334582713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381beba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f690ad9d640 1 -- 172.21.15.76:0/1334582713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b32b0 msgr2=0x5600381b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f690ad9d640 1 --2- 172.21.15.76:0/1334582713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b32b0 0x5600381b3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f690ad9d640 1 -- 172.21.15.76:0/1334582713 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600380d9570 con 0x5600381b3bd0 2024-03-12T05:56:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f690b59e640 1 --2- 172.21.15.76:0/3074072260 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f68f40a1cc0 0x7f68f40a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.173+0000 7f68f3fff640 1 -- 172.21.15.76:0/3074072260 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f69040cd510 con 0x7f690c100280 2024-03-12T05:56:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.174+0000 7f690ad9d640 1 --2- 172.21.15.76:0/1334582713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b3bd0 0x5600381bc270 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f69040cab40 tx=0x7f690408f590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5250baa41ad93a70 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.174+0000 7f690b59e640 1 --2- 172.21.15.76:0/3074072260 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f68f40a1cc0 0x7f68f40a4180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f68fc006010 tx=0x7f68fc009260 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.174+0000 7f68f17fa640 1 -- 172.21.15.76:0/1334582713 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6904099c90 con 0x5600381b3bd0 2024-03-12T05:56:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.174+0000 7f68f17fa640 1 -- 172.21.15.76:0/1334582713 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6904099a00 con 0x5600381b3bd0 2024-03-12T05:56:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.174+0000 7f68f17fa640 1 -- 172.21.15.76:0/1334582713 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69040a2660 con 0x5600381b3bd0 2024-03-12T05:56:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.174+0000 7f6913df2740 1 -- 172.21.15.76:0/1334582713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b3bd0 msgr2=0x5600381bc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.174+0000 7f6913df2740 1 --2- 172.21.15.76:0/1334582713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b3bd0 0x5600381bc270 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f69040cab40 tx=0x7f690408f590 comp rx=0 tx=0).stop 2024-03-12T05:56:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.174+0000 7f6913df2740 1 -- 172.21.15.76:0/1334582713 shutdown_connections 2024-03-12T05:56:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.174+0000 7f6913df2740 1 --2- 172.21.15.76:0/1334582713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381beba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.174+0000 7f6913df2740 1 --2- 172.21.15.76:0/1334582713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b3bd0 0x5600381bc270 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.174+0000 7f6913df2740 1 --2- 172.21.15.76:0/1334582713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b32b0 0x5600381b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.174+0000 7f6913df2740 1 -- 172.21.15.76:0/1334582713 >> 172.21.15.76:0/1334582713 conn(0x5600381ad990 msgr2=0x5600381add70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f6913df2740 1 -- 172.21.15.76:0/1334582713 shutdown_connections 2024-03-12T05:56:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f6913df2740 1 -- 172.21.15.76:0/1334582713 wait complete. 2024-03-12T05:56:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f6913df2740 1 -- start start 2024-03-12T05:56:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f6913df2740 1 Processor -- start 2024-03-12T05:56:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f6913df2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b32b0 0x5600381bc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f6913df2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b3bd0 0x5600381b9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f6913df2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381b9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f6913df2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560037dee430 con 0x5600381b3bd0 2024-03-12T05:56:12.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f6913df2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560037daae00 con 0x5600381b32b0 2024-03-12T05:56:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f6913df2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56003809ffd0 con 0x5600381bc7b0 2024-03-12T05:56:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f6913df2740 10 client.0 _ll_register_callbacks cb 0x5600380f5fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f690a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b3bd0 0x5600381b9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f690a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b3bd0 0x5600381b9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57280/0 (socket says 172.21.15.76:57280) 2024-03-12T05:56:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f690a59c640 1 -- 172.21.15.76:0/2879938608 learned_addr learned my addr 172.21.15.76:0/2879938608 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f690ad9d640 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b32b0 0x5600381bc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f690b59e640 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381b9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f690a59c640 1 -- 172.21.15.76:0/2879938608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 msgr2=0x5600381b9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f690a59c640 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381b9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.175+0000 7f690a59c640 1 -- 172.21.15.76:0/2879938608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b32b0 msgr2=0x5600381bc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f690a59c640 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b32b0 0x5600381bc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f690a59c640 1 -- 172.21.15.76:0/2879938608 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f690c1a6920 con 0x5600381b3bd0 2024-03-12T05:56:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f690ad9d640 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b32b0 0x5600381bc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:12.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f690b59e640 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381b9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f690a59c640 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b3bd0 0x5600381b9590 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6900011a60 tx=0x7f6900011f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ea38b9a4bda914 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f68eeff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f68eeff5640 1 -- 172.21.15.76:0/2879938608 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900004b20 con 0x5600381b3bd0 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f68eeff5640 1 -- 172.21.15.76:0/2879938608 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f6900004c80 con 0x5600381b3bd0 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f6913df2740 1 -- 172.21.15.76:0/2879938608 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560037e1a7e0 con 0x5600381b3bd0 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f68eeff5640 1 -- 172.21.15.76:0/2879938608 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6900004e40 con 0x5600381b3bd0 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f6913df2740 10 client.15660 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f6913df2740 1 -- 172.21.15.76:0/2879938608 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x560038119610 con 0x5600381b3bd0 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f68eeff5640 1 -- 172.21.15.76:0/2879938608 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6900005500 con 0x5600381b3bd0 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f68eeff5640 1 -- 172.21.15.76:0/2879938608 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f690001b850 con 0x5600381b3bd0 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f6913df2740 10 client.15660 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f6913df2740 10 client.15660 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f6913df2740 20 client.15660 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f6913df2740 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5600381cc9b0 0x5600381cee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f6913df2740 4 client.15660 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.176+0000 7f6913df2740 1 -- 172.21.15.76:0/2879938608 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560037e1a7e0 con 0x5600381cc9b0 2024-03-12T05:56:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.177+0000 7f690ad9d640 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5600381cc9b0 0x5600381cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.177+0000 7f68eeff5640 10 client.15660 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.177+0000 7f690ad9d640 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5600381cc9b0 0x5600381cee30 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4e5ecfa94484 server_cookie=77424ac7656f76be in_seq=0 out_seq=0 2024-03-12T05:56:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f68eeff5640 1 -- 172.21.15.76:0/2879938608 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560037e1a7e0 con 0x5600381cc9b0 2024-03-12T05:56:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f68eeff5640 10 client.15660 handle_command_reply: tid=0 2024-03-12T05:56:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f6913df2740 1 client.15660 shutdown 2024-03-12T05:56:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f6913df2740 1 -- 172.21.15.76:0/2879938608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b3bd0 msgr2=0x5600381b9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f6913df2740 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b3bd0 0x5600381b9590 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6900011a60 tx=0x7f6900011f20 comp rx=0 tx=0).stop 2024-03-12T05:56:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f6913df2740 1 -- 172.21.15.76:0/2879938608 shutdown_connections 2024-03-12T05:56:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f6913df2740 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381b9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f6913df2740 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5600381cc9b0 0x5600381cee30 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f6913df2740 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b3bd0 0x5600381b9590 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f68eeff5640 0 client.15660 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:12.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f6913df2740 1 --2- 172.21.15.76:0/2879938608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b32b0 0x5600381bc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f6913df2740 1 -- 172.21.15.76:0/2879938608 >> 172.21.15.76:0/2879938608 conn(0x5600381ad990 msgr2=0x5600381af730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f6913df2740 1 -- 172.21.15.76:0/2879938608 shutdown_connections 2024-03-12T05:56:12.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f6913df2740 1 -- 172.21.15.76:0/2879938608 wait complete. 2024-03-12T05:56:12.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.178+0000 7f6913df2740 20 client.15660 trim_cache size 0 max 16384 2024-03-12T05:56:12.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f6913df2740 1 Processor -- start 2024-03-12T05:56:12.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f6913df2740 1 -- start start 2024-03-12T05:56:12.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f6913df2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b32b0 0x5600381ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f6913df2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b3bd0 0x5600381bbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f6913df2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381bc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f6913df2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600381d4060 con 0x5600381b32b0 2024-03-12T05:56:12.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f6913df2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600381b7ae0 con 0x5600381b3bd0 2024-03-12T05:56:12.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f6913df2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600381b39c0 con 0x5600381bc7b0 2024-03-12T05:56:12.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f690ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b32b0 0x5600381ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f690ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b32b0 0x5600381ceda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57292/0 (socket says 172.21.15.76:57292) 2024-03-12T05:56:12.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f690ad9d640 1 -- 172.21.15.76:0/2663671153 learned_addr learned my addr 172.21.15.76:0/2663671153 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f690a59c640 1 --2- 172.21.15.76:0/2663671153 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b3bd0 0x5600381bbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f690ad9d640 1 -- 172.21.15.76:0/2663671153 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 msgr2=0x5600381bc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f690ad9d640 1 --2- 172.21.15.76:0/2663671153 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381bc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f690ad9d640 1 -- 172.21.15.76:0/2663671153 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b3bd0 msgr2=0x5600381bbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f690ad9d640 1 --2- 172.21.15.76:0/2663671153 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b3bd0 0x5600381bbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f690ad9d640 1 -- 172.21.15.76:0/2663671153 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600380b5ac0 con 0x5600381b32b0 2024-03-12T05:56:12.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f690a59c640 1 --2- 172.21.15.76:0/2663671153 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b3bd0 0x5600381bbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:56:12.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f690ad9d640 1 --2- 172.21.15.76:0/2663671153 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b32b0 0x5600381ceda0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6904096890 tx=0x7f6904099db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e96d201c51184ac server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f68f0ff9640 1 -- 172.21.15.76:0/2663671153 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690409a5c0 con 0x5600381b32b0 2024-03-12T05:56:12.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f68f0ff9640 1 -- 172.21.15.76:0/2663671153 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69040a2660 con 0x5600381b32b0 2024-03-12T05:56:12.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.193+0000 7f68f0ff9640 1 -- 172.21.15.76:0/2663671153 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6904095c90 con 0x5600381b32b0 2024-03-12T05:56:12.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.194+0000 7f6913df2740 1 -- 172.21.15.76:0/2663671153 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b32b0 msgr2=0x5600381ceda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.194+0000 7f6913df2740 1 --2- 172.21.15.76:0/2663671153 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b32b0 0x5600381ceda0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6904096890 tx=0x7f6904099db0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.194+0000 7f6913df2740 1 -- 172.21.15.76:0/2663671153 shutdown_connections 2024-03-12T05:56:12.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.194+0000 7f6913df2740 1 --2- 172.21.15.76:0/2663671153 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381bc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.194+0000 7f6913df2740 1 --2- 172.21.15.76:0/2663671153 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b3bd0 0x5600381bbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.194+0000 7f6913df2740 1 --2- 172.21.15.76:0/2663671153 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b32b0 0x5600381ceda0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.194+0000 7f6913df2740 1 -- 172.21.15.76:0/2663671153 >> 172.21.15.76:0/2663671153 conn(0x5600381ad990 msgr2=0x5600381b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.194+0000 7f6913df2740 1 -- 172.21.15.76:0/2663671153 shutdown_connections 2024-03-12T05:56:12.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.194+0000 7f6913df2740 1 -- 172.21.15.76:0/2663671153 wait complete. 2024-03-12T05:56:12.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.194+0000 7f6913df2740 1 -- start start 2024-03-12T05:56:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.194+0000 7f6913df2740 1 Processor -- start 2024-03-12T05:56:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f6913df2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b32b0 0x5600381c3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f6913df2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b3bd0 0x5600381bfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f6913df2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381c0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f6913df2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600381c0650 con 0x5600381b32b0 2024-03-12T05:56:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f6913df2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600381c07c0 con 0x5600381b3bd0 2024-03-12T05:56:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f6913df2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600381c0980 con 0x5600381bc7b0 2024-03-12T05:56:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f6913df2740 10 client.0 _ll_register_callbacks cb 0x7f69040d0e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f690ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b32b0 0x5600381c3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f690ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b32b0 0x5600381c3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57306/0 (socket says 172.21.15.76:57306) 2024-03-12T05:56:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f690ad9d640 1 -- 172.21.15.76:0/1047512990 learned_addr learned my addr 172.21.15.76:0/1047512990 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f690a59c640 1 --2- 172.21.15.76:0/1047512990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b3bd0 0x5600381bfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f690b59e640 1 --2- 172.21.15.76:0/1047512990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381c0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f690ad9d640 1 -- 172.21.15.76:0/1047512990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 msgr2=0x5600381c0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f690ad9d640 1 --2- 172.21.15.76:0/1047512990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381c0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f690ad9d640 1 -- 172.21.15.76:0/1047512990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b3bd0 msgr2=0x5600381bfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f690ad9d640 1 --2- 172.21.15.76:0/1047512990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b3bd0 0x5600381bfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f690ad9d640 1 -- 172.21.15.76:0/1047512990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600380d9570 con 0x5600381b32b0 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f690ad9d640 1 --2- 172.21.15.76:0/1047512990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b32b0 0x5600381c3720 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6904098170 tx=0x7f69040cd7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddfce1c112a2baa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f68eeff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f68eeff5640 1 -- 172.21.15.76:0/1047512990 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690409a5c0 con 0x5600381b32b0 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f68eeff5640 1 -- 172.21.15.76:0/1047512990 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f69040c6e40 con 0x5600381b32b0 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f6913df2740 1 -- 172.21.15.76:0/1047512990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6904099a00 con 0x5600381b32b0 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f68eeff5640 1 -- 172.21.15.76:0/1047512990 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69040981c0 con 0x5600381b32b0 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f6913df2740 10 client.15666 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.195+0000 7f6913df2740 1 -- 172.21.15.76:0/1047512990 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5600380db6a0 con 0x5600381b32b0 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.196+0000 7f68eeff5640 1 -- 172.21.15.76:0/1047512990 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6904099a00 con 0x5600381b32b0 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.196+0000 7f68eeff5640 1 -- 172.21.15.76:0/1047512990 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f690409a240 con 0x5600381b32b0 2024-03-12T05:56:12.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.196+0000 7f6913df2740 10 client.15666 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.196+0000 7f6913df2740 10 client.15666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.196+0000 7f6913df2740 20 client.15666 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.196+0000 7f6913df2740 1 --2- 172.21.15.76:0/1047512990 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560038223f90 0x5600382263b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.196+0000 7f6913df2740 4 client.15666 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:56:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.196+0000 7f6913df2740 1 -- 172.21.15.76:0/1047512990 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560037e57720 con 0x560038223f90 2024-03-12T05:56:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.196+0000 7f690a59c640 1 --2- 172.21.15.76:0/1047512990 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560038223f90 0x5600382263b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.196+0000 7f690a59c640 1 --2- 172.21.15.76:0/1047512990 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560038223f90 0x5600382263b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17dc95b4c66087c server_cookie=2b53eaa8f756ff98 in_seq=0 out_seq=0 2024-03-12T05:56:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.196+0000 7f68eeff5640 10 client.15666 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.196+0000 7f68eeff5640 1 -- 172.21.15.76:0/1047512990 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560037e57720 con 0x560038223f90 2024-03-12T05:56:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.196+0000 7f68eeff5640 10 client.15666 handle_command_reply: tid=0 2024-03-12T05:56:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.196+0000 7f6913df2740 1 client.15666 shutdown 2024-03-12T05:56:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f6913df2740 1 -- 172.21.15.76:0/1047512990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b32b0 msgr2=0x5600381c3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f6913df2740 1 --2- 172.21.15.76:0/1047512990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b32b0 0x5600381c3720 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6904098170 tx=0x7f69040cd7d0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f6913df2740 1 -- 172.21.15.76:0/1047512990 shutdown_connections 2024-03-12T05:56:12.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f6913df2740 1 --2- 172.21.15.76:0/1047512990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600381bc7b0 0x5600381c0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f6913df2740 1 --2- 172.21.15.76:0/1047512990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5600381b3bd0 0x5600381bfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f6913df2740 1 --2- 172.21.15.76:0/1047512990 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560038223f90 0x5600382263b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f6913df2740 1 --2- 172.21.15.76:0/1047512990 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5600381b32b0 0x5600381c3720 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f68eeff5640 0 client.15666 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:12.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f6913df2740 1 -- 172.21.15.76:0/1047512990 >> 172.21.15.76:0/1047512990 conn(0x5600381ad990 msgr2=0x5600381b8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f6913df2740 1 -- 172.21.15.76:0/1047512990 shutdown_connections 2024-03-12T05:56:12.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f6913df2740 1 -- 172.21.15.76:0/1047512990 wait complete. 2024-03-12T05:56:12.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f6913df2740 20 client.15666 trim_cache size 0 max 16384 2024-03-12T05:56:12.209 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:56:12.209 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T05:56:12.210 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T05:56:12.210 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:56:12.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f691121c640 1 -- 172.21.15.76:0/3074072260 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f68f40a1cc0 msgr2=0x7f68f40a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f691121c640 1 --2- 172.21.15.76:0/3074072260 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f68f40a1cc0 0x7f68f40a4180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f68fc006010 tx=0x7f68fc009260 comp rx=0 tx=0).stop 2024-03-12T05:56:12.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f691121c640 1 -- 172.21.15.76:0/3074072260 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690c100280 msgr2=0x7f690c19f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f691121c640 1 --2- 172.21.15.76:0/3074072260 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690c100280 0x7f690c19f630 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f690400d770 tx=0x7f690400dc30 comp rx=0 tx=0).stop 2024-03-12T05:56:12.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f691121c640 1 -- 172.21.15.76:0/3074072260 shutdown_connections 2024-03-12T05:56:12.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f691121c640 1 --2- 172.21.15.76:0/3074072260 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f68f40a1cc0 0x7f68f40a4180 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f691121c640 1 --2- 172.21.15.76:0/3074072260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f690c199740 0x7f690c199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f691121c640 1 --2- 172.21.15.76:0/3074072260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f690c105340 0x7f690c19fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f691121c640 1 --2- 172.21.15.76:0/3074072260 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690c100280 0x7f690c19f630 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f691121c640 1 -- 172.21.15.76:0/3074072260 >> 172.21.15.76:0/3074072260 conn(0x7f690c077120 msgr2=0x7f690c10bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f691121c640 1 -- 172.21.15.76:0/3074072260 shutdown_connections 2024-03-12T05:56:12.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.197+0000 7f691121c640 1 -- 172.21.15.76:0/3074072260 wait complete. 2024-03-12T05:56:12.212 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:12 smithi076 ceph-mon[32095]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-12T05:56:12.213 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:56:12.213 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:56:12.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.314+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/1624528038 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c48105370 msgr2=0x7f3c4810cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.314+0000 7f3c4fa2e640 1 --2- 172.21.15.76:0/1624528038 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c48105370 0x7f3c4810cf40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3c40009cd0 tx=0x7f3c4002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.314+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/1624528038 shutdown_connections 2024-03-12T05:56:12.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.314+0000 7f3c4fa2e640 1 --2- 172.21.15.76:0/1624528038 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c48105370 0x7f3c4810cf40 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.314+0000 7f3c4fa2e640 1 --2- 172.21.15.76:0/1624528038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3c48100c50 0x7f3c48104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.314+0000 7f3c4fa2e640 1 --2- 172.21.15.76:0/1624528038 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c481002a0 0x7f3c48100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.314+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/1624528038 >> 172.21.15.76:0/1624528038 conn(0x7f3c48077200 msgr2=0x7f3c48077610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.314+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/1624528038 shutdown_connections 2024-03-12T05:56:12.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.314+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/1624528038 wait complete. 2024-03-12T05:56:12.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.314+0000 7f3c4fa2e640 1 Processor -- start 2024-03-12T05:56:12.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.314+0000 7f3c4fa2e640 1 -- start start 2024-03-12T05:56:12.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.314+0000 7f3c4fa2e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3c481002a0 0x7f3c481963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4fa2e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c48105370 0x7f3c48196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4fa2e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c48196e50 0x7f3c481a5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4fa2e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c481a5f90 con 0x7f3c48196e50 2024-03-12T05:56:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4fa2e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c481a60d0 con 0x7f3c481002a0 2024-03-12T05:56:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4fa2e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c481a6210 con 0x7f3c48105370 2024-03-12T05:56:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4dfcb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c48196e50 0x7f3c481a5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4dfcb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c48196e50 0x7f3c481a5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57332/0 (socket says 172.21.15.76:57332) 2024-03-12T05:56:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4dfcb640 1 -- 172.21.15.76:0/4272162475 learned_addr learned my addr 172.21.15.76:0/4272162475 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4cfc9640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c48105370 0x7f3c48196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4d7ca640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3c481002a0 0x7f3c481963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4dfcb640 1 -- 172.21.15.76:0/4272162475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c48105370 msgr2=0x7f3c48196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4dfcb640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c48105370 0x7f3c48196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4dfcb640 1 -- 172.21.15.76:0/4272162475 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3c481002a0 msgr2=0x7f3c481963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4dfcb640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3c481002a0 0x7f3c481963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4dfcb640 1 -- 172.21.15.76:0/4272162475 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c40009900 con 0x7f3c48196e50 2024-03-12T05:56:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4d7ca640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3c481002a0 0x7f3c481963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4dfcb640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c48196e50 0x7f3c481a5a50 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3c40004af0 tx=0x7f3c40034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c3e7fc640 1 -- 172.21.15.76:0/4272162475 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c400328c0 con 0x7f3c48196e50 2024-03-12T05:56:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c3e7fc640 1 -- 172.21.15.76:0/4272162475 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c40003e50 con 0x7f3c48196e50 2024-03-12T05:56:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/4272162475 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c481a6490 con 0x7f3c48196e50 2024-03-12T05:56:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c3e7fc640 1 -- 172.21.15.76:0/4272162475 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40040450 con 0x7f3c48196e50 2024-03-12T05:56:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.315+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/4272162475 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c481a6940 con 0x7f3c48196e50 2024-03-12T05:56:12.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.316+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/4272162475 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c48068000 con 0x7f3c48196e50 2024-03-12T05:56:12.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.318+0000 7f3c3e7fc640 1 -- 172.21.15.76:0/4272162475 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f3c4000a040 con 0x7f3c48196e50 2024-03-12T05:56:12.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.319+0000 7f3c3e7fc640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3c340a1cc0 0x7f3c340a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.319+0000 7f3c3e7fc640 1 -- 172.21.15.76:0/4272162475 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3c400f0a90 con 0x7f3c48196e50 2024-03-12T05:56:12.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.319+0000 7f3c4d7ca640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3c340a1cc0 0x7f3c340a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.319+0000 7f3c4d7ca640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3c340a1cc0 0x7f3c340a4180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3c440058c0 tx=0x7f3c44005850 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.322+0000 7f3c3e7fc640 1 -- 172.21.15.76:0/4272162475 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3c400b9920 con 0x7f3c48196e50 2024-03-12T05:56:12.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:12 smithi082 ceph-mon[36382]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-12T05:56:12.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:12 smithi088 ceph-mon[36585]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-12T05:56:12.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.539+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/4272162475 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c48108660 con 0x7f3c48196e50 2024-03-12T05:56:12.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.540+0000 7f3c3e7fc640 1 -- 172.21.15.76:0/4272162475 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11328 (secure 0 0 0) 0x7f3c400bcfe0 con 0x7f3c48196e50 2024-03-12T05:56:12.545 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:56:12.545 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:55:33.056004+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:56:12.545 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 22 2024-03-12T05:56:12.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.542+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/4272162475 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3c340a1cc0 msgr2=0x7f3c340a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.542+0000 7f3c4fa2e640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3c340a1cc0 0x7f3c340a4180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3c440058c0 tx=0x7f3c44005850 comp rx=0 tx=0).stop 2024-03-12T05:56:12.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.542+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/4272162475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c48196e50 msgr2=0x7f3c481a5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.542+0000 7f3c4fa2e640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c48196e50 0x7f3c481a5a50 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3c40004af0 tx=0x7f3c40034cc0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.542+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/4272162475 shutdown_connections 2024-03-12T05:56:12.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.542+0000 7f3c4fa2e640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3c340a1cc0 0x7f3c340a4180 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.542+0000 7f3c4fa2e640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c48196e50 0x7f3c481a5a50 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.542+0000 7f3c4fa2e640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c48105370 0x7f3c48196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.542+0000 7f3c4fa2e640 1 --2- 172.21.15.76:0/4272162475 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3c481002a0 0x7f3c481963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.542+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/4272162475 >> 172.21.15.76:0/4272162475 conn(0x7f3c48077200 msgr2=0x7f3c480fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.543+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/4272162475 shutdown_connections 2024-03-12T05:56:12.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.543+0000 7f3c4fa2e640 1 -- 172.21.15.76:0/4272162475 wait complete. 2024-03-12T05:56:12.558 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:56:12.559 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:56:12.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.662+0000 7fcc9ab48640 1 -- 172.21.15.76:0/667033223 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc94106450 msgr2=0x7fcc9410e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.662+0000 7fcc9ab48640 1 --2- 172.21.15.76:0/667033223 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc94106450 0x7fcc9410e020 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c009cd0 tx=0x7fcc8c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.662+0000 7fcc9ab48640 1 -- 172.21.15.76:0/667033223 shutdown_connections 2024-03-12T05:56:12.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.662+0000 7fcc9ab48640 1 --2- 172.21.15.76:0/667033223 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc94106450 0x7fcc9410e020 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.662+0000 7fcc9ab48640 1 --2- 172.21.15.76:0/667033223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc94101d30 0x7fcc94105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.662+0000 7fcc9ab48640 1 --2- 172.21.15.76:0/667033223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc94101380 0x7fcc94101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.662+0000 7fcc9ab48640 1 -- 172.21.15.76:0/667033223 >> 172.21.15.76:0/667033223 conn(0x7fcc940fcc00 msgr2=0x7fcc940ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.662+0000 7fcc9ab48640 1 -- 172.21.15.76:0/667033223 shutdown_connections 2024-03-12T05:56:12.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.662+0000 7fcc9ab48640 1 -- 172.21.15.76:0/667033223 wait complete. 2024-03-12T05:56:12.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.662+0000 7fcc9ab48640 1 Processor -- start 2024-03-12T05:56:12.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.662+0000 7fcc9ab48640 1 -- start start 2024-03-12T05:56:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc9ab48640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc94101380 0x7fcc9419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc9ab48640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc94106450 0x7fcc9419fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc9ab48640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc941997e0 0x7fcc94199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc9ab48640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9419a290 con 0x7fcc94106450 2024-03-12T05:56:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc9ab48640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9419a400 con 0x7fcc941997e0 2024-03-12T05:56:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc9ab48640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9419a570 con 0x7fcc94101380 2024-03-12T05:56:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc93fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc94106450 0x7fcc9419fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc93fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc94106450 0x7fcc9419fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57346/0 (socket says 172.21.15.76:57346) 2024-03-12T05:56:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc93fff640 1 -- 172.21.15.76:0/3916265485 learned_addr learned my addr 172.21.15.76:0/3916265485 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc990e5640 1 --2- 172.21.15.76:0/3916265485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc941997e0 0x7fcc94199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc988e4640 1 --2- 172.21.15.76:0/3916265485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc94101380 0x7fcc9419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc93fff640 1 -- 172.21.15.76:0/3916265485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc94101380 msgr2=0x7fcc9419f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc93fff640 1 --2- 172.21.15.76:0/3916265485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc94101380 0x7fcc9419f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc93fff640 1 -- 172.21.15.76:0/3916265485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc941997e0 msgr2=0x7fcc94199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc93fff640 1 --2- 172.21.15.76:0/3916265485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc941997e0 0x7fcc94199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc93fff640 1 -- 172.21.15.76:0/3916265485 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8c009900 con 0x7fcc94106450 2024-03-12T05:56:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc93fff640 1 --2- 172.21.15.76:0/3916265485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc94106450 0x7fcc9419fbb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fcc8000d770 tx=0x7fcc8000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc91ffb640 1 -- 172.21.15.76:0/3916265485 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80009c30 con 0x7fcc94106450 2024-03-12T05:56:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc9ab48640 1 -- 172.21.15.76:0/3916265485 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc941a6520 con 0x7fcc94106450 2024-03-12T05:56:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc91ffb640 1 -- 172.21.15.76:0/3916265485 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc8000de80 con 0x7fcc94106450 2024-03-12T05:56:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.663+0000 7fcc91ffb640 1 -- 172.21.15.76:0/3916265485 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8000f6b0 con 0x7fcc94106450 2024-03-12T05:56:12.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.664+0000 7fcc9ab48640 1 -- 172.21.15.76:0/3916265485 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc941a69e0 con 0x7fcc94106450 2024-03-12T05:56:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.666+0000 7fcc91ffb640 1 -- 172.21.15.76:0/3916265485 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fcc80010460 con 0x7fcc94106450 2024-03-12T05:56:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.666+0000 7fcc9d7f6740 1 Processor -- start 2024-03-12T05:56:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.666+0000 7fcc9d7f6740 1 -- start start 2024-03-12T05:56:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.666+0000 7fcc9d7f6740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556ead250 0x559556ead630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.666+0000 7fcc9d7f6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 0x559556eb6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.666+0000 7fcc9d7f6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 0x559556eb8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.666+0000 7fcc9d7f6740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x559556e68960 con 0x559556ead250 2024-03-12T05:56:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.666+0000 7fcc9d7f6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559556e1c310 con 0x559556eadb70 2024-03-12T05:56:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.666+0000 7fcc9d7f6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559556e19840 con 0x559556eb6750 2024-03-12T05:56:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc990e5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556ead250 0x559556ead630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc91ffb640 1 --2- 172.21.15.76:0/3916265485 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcc840a1cc0 0x7fcc840a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc990e5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556ead250 0x559556ead630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57360/0 (socket says 172.21.15.76:57360) 2024-03-12T05:56:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc990e5640 1 -- 172.21.15.76:0/1063670444 learned_addr learned my addr 172.21.15.76:0/1063670444 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc988e4640 1 --2- 172.21.15.76:0/1063670444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 0x559556eb6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc990e5640 1 --2- 172.21.15.76:0/3916265485 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcc840a1cc0 0x7fcc840a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc91ffb640 1 -- 172.21.15.76:0/3916265485 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcc800cd7b0 con 0x7fcc94106450 2024-03-12T05:56:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc93fff640 1 --2- 172.21.15.76:0/1063670444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 0x559556eb8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc990e5640 1 -- 172.21.15.76:0/1063670444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 msgr2=0x559556eb8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc990e5640 1 --2- 172.21.15.76:0/1063670444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 0x559556eb8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc990e5640 1 -- 172.21.15.76:0/1063670444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 msgr2=0x559556eb6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc990e5640 1 --2- 172.21.15.76:0/1063670444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 0x559556eb6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc990e5640 1 -- 172.21.15.76:0/1063670444 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559556dd7060 con 0x559556ead250 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc990e5640 1 --2- 172.21.15.76:0/1063670444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556ead250 0x559556ead630 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c02c700 tx=0x7fcc8c0345f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9e6cf64d8e6335 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc7effd640 1 -- 172.21.15.76:0/1063670444 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c0328b0 con 0x559556ead250 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc7effd640 1 -- 172.21.15.76:0/1063670444 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc8c003e50 con 0x559556ead250 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.667+0000 7fcc7effd640 1 -- 172.21.15.76:0/1063670444 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c0044a0 con 0x559556ead250 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.668+0000 7fcc990e5640 1 --2- 172.21.15.76:0/3916265485 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcc840a1cc0 0x7fcc840a4180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c04f000 tx=0x7fcc8c04b000 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.668+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/1063670444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556ead250 msgr2=0x559556ead630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.668+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/1063670444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556ead250 0x559556ead630 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c02c700 tx=0x7fcc8c0345f0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.668+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/1063670444 shutdown_connections 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.668+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/1063670444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 0x559556eb8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.668+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/1063670444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 0x559556eb6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.668+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/1063670444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556ead250 0x559556ead630 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.668+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/1063670444 >> 172.21.15.76:0/1063670444 conn(0x559556ea7980 msgr2=0x559556ea7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.668+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/1063670444 shutdown_connections 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.668+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/1063670444 wait complete. 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc9d7f6740 1 -- start start 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc9d7f6740 1 Processor -- start 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc9d7f6740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556ead250 0x559556eb3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc9d7f6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 0x559556eb3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc9d7f6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 0x559556eb41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc9d7f6740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x559556e15020 con 0x559556ead250 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc9d7f6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559556ae8430 con 0x559556eadb70 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc9d7f6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559556aa4e00 con 0x559556eb6750 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc9d7f6740 10 client.0 _ll_register_callbacks cb 0x559556dbccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc988e4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556ead250 0x559556eb3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc988e4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556ead250 0x559556eb3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57372/0 (socket says 172.21.15.76:57372) 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc988e4640 1 -- 172.21.15.76:0/3801587274 learned_addr learned my addr 172.21.15.76:0/3801587274 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc990e5640 1 --2- 172.21.15.76:0/3801587274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 0x559556eb41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc93fff640 1 --2- 172.21.15.76:0/3801587274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 0x559556eb3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc988e4640 1 -- 172.21.15.76:0/3801587274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 msgr2=0x559556eb41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.669+0000 7fcc988e4640 1 --2- 172.21.15.76:0/3801587274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 0x559556eb41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc988e4640 1 -- 172.21.15.76:0/3801587274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 msgr2=0x559556eb3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc988e4640 1 --2- 172.21.15.76:0/3801587274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 0x559556eb3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc988e4640 1 -- 172.21.15.76:0/3801587274 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9410e830 con 0x559556ead250 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc988e4640 1 --2- 172.21.15.76:0/3801587274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556ead250 0x559556eb3760 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcc88011760 tx=0x7fcc88011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d669faf43611149a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc73fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc73fff640 1 -- 172.21.15.76:0/3801587274 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8800fe10 con 0x559556ead250 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc73fff640 1 -- 172.21.15.76:0/3801587274 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcc880193f0 con 0x559556ead250 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3801587274 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559556b147e0 con 0x559556ead250 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc73fff640 1 -- 172.21.15.76:0/3801587274 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc880195b0 con 0x559556ead250 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc9d7f6740 10 client.15684 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3801587274 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x559556e136a0 con 0x559556ead250 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc73fff640 1 -- 172.21.15.76:0/3801587274 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559556b147e0 con 0x559556ead250 2024-03-12T05:56:12.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc73fff640 1 -- 172.21.15.76:0/3801587274 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcc880214a0 con 0x559556ead250 2024-03-12T05:56:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc9d7f6740 10 client.15684 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc9d7f6740 10 client.15684 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:56:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc9d7f6740 20 client.15684 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3801587274 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559556ec6c30 0x559556ec9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc9d7f6740 4 client.15684 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.670+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3801587274 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559556b51720 con 0x559556ec6c30 2024-03-12T05:56:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.671+0000 7fcc93fff640 1 --2- 172.21.15.76:0/3801587274 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559556ec6c30 0x559556ec9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.671+0000 7fcc73fff640 10 client.15684 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:56:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.671+0000 7fcc93fff640 1 --2- 172.21.15.76:0/3801587274 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559556ec6c30 0x559556ec9050 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=684ce7a580c659a server_cookie=8bf2944916029332 in_seq=0 out_seq=0 2024-03-12T05:56:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc73fff640 1 -- 172.21.15.76:0/3801587274 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559556b51720 con 0x559556ec6c30 2024-03-12T05:56:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc73fff640 10 client.15684 handle_command_reply: tid=0 2024-03-12T05:56:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc9d7f6740 1 client.15684 shutdown 2024-03-12T05:56:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3801587274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556ead250 msgr2=0x559556eb3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3801587274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556ead250 0x559556eb3760 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcc88011760 tx=0x7fcc88011c20 comp rx=0 tx=0).stop 2024-03-12T05:56:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3801587274 shutdown_connections 2024-03-12T05:56:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3801587274 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559556ec6c30 0x559556ec9050 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3801587274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 0x559556eb41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3801587274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 0x559556eb3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc73fff640 0 client.15684 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:56:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3801587274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556ead250 0x559556eb3760 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3801587274 >> 172.21.15.76:0/3801587274 conn(0x559556ea7980 msgr2=0x559556ea96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3801587274 shutdown_connections 2024-03-12T05:56:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3801587274 wait complete. 2024-03-12T05:56:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.672+0000 7fcc9d7f6740 20 client.15684 trim_cache size 0 max 16384 2024-03-12T05:56:12.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.683+0000 7fcc9d7f6740 1 Processor -- start 2024-03-12T05:56:12.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc9d7f6740 1 -- start start 2024-03-12T05:56:12.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc9d7f6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556ead250 0x559556eb5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc9d7f6740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556eadb70 0x559556eb4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc9d7f6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 0x559556eaea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc9d7f6740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x559556d9a130 con 0x559556eadb70 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc9d7f6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559556ea3310 con 0x559556ead250 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc9d7f6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559556eb1ce0 con 0x559556eb6750 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc93fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556eadb70 0x559556eb4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc93fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556eadb70 0x559556eb4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57380/0 (socket says 172.21.15.76:57380) 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc93fff640 1 -- 172.21.15.76:0/3723888717 learned_addr learned my addr 172.21.15.76:0/3723888717 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc990e5640 1 --2- 172.21.15.76:0/3723888717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 0x559556eaea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc988e4640 1 --2- 172.21.15.76:0/3723888717 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556ead250 0x559556eb5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc93fff640 1 -- 172.21.15.76:0/3723888717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 msgr2=0x559556eaea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc93fff640 1 --2- 172.21.15.76:0/3723888717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 0x559556eaea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc93fff640 1 -- 172.21.15.76:0/3723888717 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556ead250 msgr2=0x559556eb5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc93fff640 1 --2- 172.21.15.76:0/3723888717 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556ead250 0x559556eb5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc93fff640 1 -- 172.21.15.76:0/3723888717 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559556dafc20 con 0x559556eadb70 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc93fff640 1 --2- 172.21.15.76:0/3723888717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556eadb70 0x559556eb4000 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcc80010d60 tx=0x7fcc80096530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420890de789d0626 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.684+0000 7fcc7e7fc640 1 -- 172.21.15.76:0/3723888717 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc800a2940 con 0x559556eadb70 2024-03-12T05:56:12.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.685+0000 7fcc7e7fc640 1 -- 172.21.15.76:0/3723888717 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc80009d90 con 0x559556eadb70 2024-03-12T05:56:12.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.685+0000 7fcc7e7fc640 1 -- 172.21.15.76:0/3723888717 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc800cb070 con 0x559556eadb70 2024-03-12T05:56:12.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.685+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3723888717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556eadb70 msgr2=0x559556eb4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.685+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3723888717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556eadb70 0x559556eb4000 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcc80010d60 tx=0x7fcc80096530 comp rx=0 tx=0).stop 2024-03-12T05:56:12.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.685+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3723888717 shutdown_connections 2024-03-12T05:56:12.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.685+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3723888717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556eb6750 0x559556eaea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.685+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3723888717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556eadb70 0x559556eb4000 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.685+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3723888717 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556ead250 0x559556eb5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.685+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3723888717 >> 172.21.15.76:0/3723888717 conn(0x559556ea7980 msgr2=0x559556ea2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.685+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3723888717 shutdown_connections 2024-03-12T05:56:12.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.685+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3723888717 wait complete. 2024-03-12T05:56:12.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc9d7f6740 1 -- start start 2024-03-12T05:56:12.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc9d7f6740 1 Processor -- start 2024-03-12T05:56:12.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc9d7f6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556ead250 0x559556ebc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc9d7f6740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 0x559556ebc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc9d7f6740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556eb6750 0x559556ebcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc9d7f6740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x559556f1d160 con 0x559556eb6750 2024-03-12T05:56:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc9d7f6740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559556f1d2a0 con 0x559556eadb70 2024-03-12T05:56:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc9d7f6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559556f1d3e0 con 0x559556ead250 2024-03-12T05:56:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc9d7f6740 10 client.0 _ll_register_callbacks cb 0x7fcc8c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc990e5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556eb6750 0x559556ebcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc990e5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556eb6750 0x559556ebcaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57390/0 (socket says 172.21.15.76:57390) 2024-03-12T05:56:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc990e5640 1 -- 172.21.15.76:0/3731357541 learned_addr learned my addr 172.21.15.76:0/3731357541 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc988e4640 1 --2- 172.21.15.76:0/3731357541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556ead250 0x559556ebc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc990e5640 1 -- 172.21.15.76:0/3731357541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556ead250 msgr2=0x559556ebc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc990e5640 1 --2- 172.21.15.76:0/3731357541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556ead250 0x559556ebc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc990e5640 1 -- 172.21.15.76:0/3731357541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 msgr2=0x559556ebc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc93fff640 1 --2- 172.21.15.76:0/3731357541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 0x559556ebc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc990e5640 1 --2- 172.21.15.76:0/3731357541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 0x559556ebc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.686+0000 7fcc990e5640 1 -- 172.21.15.76:0/3731357541 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc941a69e0 con 0x559556eb6750 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc990e5640 1 --2- 172.21.15.76:0/3731357541 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556eb6750 0x559556ebcaa0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c02c6d0 tx=0x7fcc8c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae5381689f5ea1b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc73fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc73fff640 1 -- 172.21.15.76:0/3731357541 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c05c070 con 0x559556eb6750 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc73fff640 1 -- 172.21.15.76:0/3731357541 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcc8c04a070 con 0x559556eb6750 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3731357541 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc8c003e50 con 0x559556eb6750 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc73fff640 1 -- 172.21.15.76:0/3731357541 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc8c009540 con 0x559556eb6750 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc9d7f6740 10 client.15690 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3731357541 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x559556dd58c0 con 0x559556eb6750 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc73fff640 1 -- 172.21.15.76:0/3731357541 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc8c06e020 con 0x559556eb6750 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc73fff640 1 -- 172.21.15.76:0/3731357541 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcc8c03e050 con 0x559556eb6750 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc9d7f6740 10 client.15690 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc9d7f6740 10 client.15690 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:56:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc9d7f6740 20 client.15690 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3731357541 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559556f1e930 0x559556f20d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc9d7f6740 4 client.15690 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:56:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.687+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3731357541 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559556ba0890 con 0x559556f1e930 2024-03-12T05:56:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc988e4640 1 --2- 172.21.15.76:0/3731357541 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559556f1e930 0x559556f20d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc73fff640 10 client.15690 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:56:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc988e4640 1 --2- 172.21.15.76:0/3731357541 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559556f1e930 0x559556f20d50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d50344183ef0c57 server_cookie=38b15bdc254720b1 in_seq=0 out_seq=0 2024-03-12T05:56:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc73fff640 1 -- 172.21.15.76:0/3731357541 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559556ba0890 con 0x559556f1e930 2024-03-12T05:56:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc73fff640 10 client.15690 handle_command_reply: tid=0 2024-03-12T05:56:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc9d7f6740 1 client.15690 shutdown 2024-03-12T05:56:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3731357541 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556eb6750 msgr2=0x559556ebcaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3731357541 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556eb6750 0x559556ebcaa0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c02c6d0 tx=0x7fcc8c004040 comp rx=0 tx=0).stop 2024-03-12T05:56:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3731357541 shutdown_connections 2024-03-12T05:56:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3731357541 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559556f1e930 0x559556f20d50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3731357541 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559556eb6750 0x559556ebcaa0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc73fff640 0 client.15690 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:56:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3731357541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559556eadb70 0x559556ebc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc9d7f6740 1 --2- 172.21.15.76:0/3731357541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559556ead250 0x559556ebc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.688+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3731357541 >> 172.21.15.76:0/3731357541 conn(0x559556ea7980 msgr2=0x559556eaea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3731357541 shutdown_connections 2024-03-12T05:56:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9d7f6740 1 -- 172.21.15.76:0/3731357541 wait complete. 2024-03-12T05:56:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9d7f6740 20 client.15690 trim_cache size 0 max 16384 2024-03-12T05:56:12.699 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:56:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9ab48640 1 -- 172.21.15.76:0/3916265485 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcc840a1cc0 msgr2=0x7fcc840a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9ab48640 1 --2- 172.21.15.76:0/3916265485 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcc840a1cc0 0x7fcc840a4180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c04f000 tx=0x7fcc8c04b000 comp rx=0 tx=0).stop 2024-03-12T05:56:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9ab48640 1 -- 172.21.15.76:0/3916265485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc94106450 msgr2=0x7fcc9419fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9ab48640 1 --2- 172.21.15.76:0/3916265485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc94106450 0x7fcc9419fbb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fcc8000d770 tx=0x7fcc8000dc30 comp rx=0 tx=0).stop 2024-03-12T05:56:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9ab48640 1 -- 172.21.15.76:0/3916265485 shutdown_connections 2024-03-12T05:56:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9ab48640 1 --2- 172.21.15.76:0/3916265485 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcc840a1cc0 0x7fcc840a4180 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9ab48640 1 --2- 172.21.15.76:0/3916265485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc941997e0 0x7fcc94199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9ab48640 1 --2- 172.21.15.76:0/3916265485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc94106450 0x7fcc9419fbb0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9ab48640 1 --2- 172.21.15.76:0/3916265485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc94101380 0x7fcc9419f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9ab48640 1 -- 172.21.15.76:0/3916265485 >> 172.21.15.76:0/3916265485 conn(0x7fcc940fcc00 msgr2=0x7fcc940fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9ab48640 1 -- 172.21.15.76:0/3916265485 shutdown_connections 2024-03-12T05:56:12.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.689+0000 7fcc9ab48640 1 -- 172.21.15.76:0/3916265485 wait complete. 2024-03-12T05:56:12.705 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:56:12.705 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:56:12.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.806+0000 7fa0e2415640 1 -- 172.21.15.76:0/1929089278 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0dc1023a0 msgr2=0x7fa0dc102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.806+0000 7fa0e2415640 1 --2- 172.21.15.76:0/1929089278 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0dc1023a0 0x7fa0dc102780 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc009900 tx=0x7fa0cc02bcd0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.806+0000 7fa0e2415640 1 -- 172.21.15.76:0/1929089278 shutdown_connections 2024-03-12T05:56:12.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.806+0000 7fa0e2415640 1 --2- 172.21.15.76:0/1929089278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0dc10b770 0x7fa0dc10dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.806+0000 7fa0e2415640 1 --2- 172.21.15.76:0/1929089278 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa0dc102cc0 0x7fa0dc10b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.806+0000 7fa0e2415640 1 --2- 172.21.15.76:0/1929089278 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0dc1023a0 0x7fa0dc102780 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.806+0000 7fa0e2415640 1 -- 172.21.15.76:0/1929089278 >> 172.21.15.76:0/1929089278 conn(0x7fa0dc077230 msgr2=0x7fa0dc077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.806+0000 7fa0e2415640 1 -- 172.21.15.76:0/1929089278 shutdown_connections 2024-03-12T05:56:12.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.806+0000 7fa0e2415640 1 -- 172.21.15.76:0/1929089278 wait complete. 2024-03-12T05:56:12.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.807+0000 7fa0e2415640 1 Processor -- start 2024-03-12T05:56:12.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.807+0000 7fa0e2415640 1 -- start start 2024-03-12T05:56:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.807+0000 7fa0e2415640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa0dc1023a0 0x7fa0dc19d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.807+0000 7fa0e2415640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0dc102cc0 0x7fa0dc19d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.807+0000 7fa0e2415640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0dc10b770 0x7fa0dc197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.807+0000 7fa0e2415640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc197c60 con 0x7fa0dc102cc0 2024-03-12T05:56:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.807+0000 7fa0e2415640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc197dd0 con 0x7fa0dc1023a0 2024-03-12T05:56:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.807+0000 7fa0e2415640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc197f90 con 0x7fa0dc10b770 2024-03-12T05:56:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.807+0000 7fa0db7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0dc102cc0 0x7fa0dc19d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.807+0000 7fa0db7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0dc102cc0 0x7fa0dc19d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57418/0 (socket says 172.21.15.76:57418) 2024-03-12T05:56:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.807+0000 7fa0db7fe640 1 -- 172.21.15.76:0/385168508 learned_addr learned my addr 172.21.15.76:0/385168508 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.807+0000 7fa0e09b2640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0dc10b770 0x7fa0dc197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.807+0000 7fa0dbfff640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa0dc1023a0 0x7fa0dc19d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.808+0000 7fa0db7fe640 1 -- 172.21.15.76:0/385168508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0dc10b770 msgr2=0x7fa0dc197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.808+0000 7fa0db7fe640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0dc10b770 0x7fa0dc197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.808+0000 7fa0db7fe640 1 -- 172.21.15.76:0/385168508 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa0dc1023a0 msgr2=0x7fa0dc19d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.808+0000 7fa0db7fe640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa0dc1023a0 0x7fa0dc19d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.808+0000 7fa0db7fe640 1 -- 172.21.15.76:0/385168508 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0cc009560 con 0x7fa0dc102cc0 2024-03-12T05:56:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.808+0000 7fa0dbfff640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa0dc1023a0 0x7fa0dc19d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.808+0000 7fa0db7fe640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0dc102cc0 0x7fa0dc19d950 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000d950 tx=0x7fa0d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.808+0000 7fa0d97fa640 1 -- 172.21.15.76:0/385168508 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0009a30 con 0x7fa0dc102cc0 2024-03-12T05:56:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.808+0000 7fa0d97fa640 1 -- 172.21.15.76:0/385168508 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0d0009b90 con 0x7fa0dc102cc0 2024-03-12T05:56:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.808+0000 7fa0e2415640 1 -- 172.21.15.76:0/385168508 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0dc198270 con 0x7fa0dc102cc0 2024-03-12T05:56:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.808+0000 7fa0d97fa640 1 -- 172.21.15.76:0/385168508 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0004a00 con 0x7fa0dc102cc0 2024-03-12T05:56:12.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.808+0000 7fa0e2415640 1 -- 172.21.15.76:0/385168508 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0dc1a44b0 con 0x7fa0dc102cc0 2024-03-12T05:56:12.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.810+0000 7fa0d97fa640 1 -- 172.21.15.76:0/385168508 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fa0d0009d00 con 0x7fa0dc102cc0 2024-03-12T05:56:12.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e4feb740 1 Processor -- start 2024-03-12T05:56:12.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e4feb740 1 -- start start 2024-03-12T05:56:12.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e4feb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7250 0x55dd654b7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e4feb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654b7b70 0x55dd654c0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e4feb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654c0750 0x55dd654c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e4feb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65472960 con 0x55dd654c0750 2024-03-12T05:56:12.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e4feb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65426310 con 0x55dd654b7250 2024-03-12T05:56:12.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e4feb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65423840 con 0x55dd654b7b70 2024-03-12T05:56:12.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0db7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654c0750 0x55dd654c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0d97fa640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa0c40a1cc0 0x7fa0c40a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0db7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654c0750 0x55dd654c2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57422/0 (socket says 172.21.15.76:57422) 2024-03-12T05:56:12.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0db7fe640 1 -- 172.21.15.76:0/1087812201 learned_addr learned my addr 172.21.15.76:0/1087812201 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e09b2640 1 --2- 172.21.15.76:0/1087812201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7250 0x55dd654b7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e09b2640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa0c40a1cc0 0x7fa0c40a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0dbfff640 1 --2- 172.21.15.76:0/1087812201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654b7b70 0x55dd654c0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0d97fa640 1 -- 172.21.15.76:0/385168508 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa0d00cd460 con 0x7fa0dc102cc0 2024-03-12T05:56:12.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e09b2640 1 -- 172.21.15.76:0/1087812201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654b7b70 msgr2=0x55dd654c0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e09b2640 1 --2- 172.21.15.76:0/1087812201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654b7b70 0x55dd654c0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e09b2640 1 -- 172.21.15.76:0/1087812201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654c0750 msgr2=0x55dd654c2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e09b2640 1 --2- 172.21.15.76:0/1087812201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654c0750 0x55dd654c2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0e09b2640 1 -- 172.21.15.76:0/1087812201 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd653e1060 con 0x55dd654b7250 2024-03-12T05:56:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.811+0000 7fa0db7fe640 1 --2- 172.21.15.76:0/1087812201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654c0750 0x55dd654c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.812+0000 7fa0e09b2640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa0c40a1cc0 0x7fa0c40a4180 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa0d400afd0 tx=0x7fa0d4016000 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.812+0000 7fa0e09b2640 1 --2- 172.21.15.76:0/1087812201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7250 0x55dd654b7630 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4014db0 tx=0x7fa0d40117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e023a0ed05bdc5d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.812+0000 7fa0c27fc640 1 -- 172.21.15.76:0/1087812201 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d400cdf0 con 0x55dd654b7250 2024-03-12T05:56:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.812+0000 7fa0c27fc640 1 -- 172.21.15.76:0/1087812201 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0d401a040 con 0x55dd654b7250 2024-03-12T05:56:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.812+0000 7fa0c27fc640 1 -- 172.21.15.76:0/1087812201 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4004770 con 0x55dd654b7250 2024-03-12T05:56:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.813+0000 7fa0e4feb740 1 -- 172.21.15.76:0/1087812201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7250 msgr2=0x55dd654b7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.813+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/1087812201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7250 0x55dd654b7630 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4014db0 tx=0x7fa0d40117b0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.813+0000 7fa0e4feb740 1 -- 172.21.15.76:0/1087812201 shutdown_connections 2024-03-12T05:56:12.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.813+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/1087812201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654c0750 0x55dd654c2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.813+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/1087812201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654b7b70 0x55dd654c0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.813+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/1087812201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7250 0x55dd654b7630 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.813+0000 7fa0e4feb740 1 -- 172.21.15.76:0/1087812201 >> 172.21.15.76:0/1087812201 conn(0x55dd654b1980 msgr2=0x55dd654b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.813+0000 7fa0e4feb740 1 -- 172.21.15.76:0/1087812201 shutdown_connections 2024-03-12T05:56:12.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.813+0000 7fa0e4feb740 1 -- 172.21.15.76:0/1087812201 wait complete. 2024-03-12T05:56:12.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.813+0000 7fa0e4feb740 1 -- start start 2024-03-12T05:56:12.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.813+0000 7fa0e4feb740 1 Processor -- start 2024-03-12T05:56:12.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.813+0000 7fa0e4feb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7250 0x55dd654bd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0e4feb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7b70 0x55dd654bdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0e4feb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654c0750 0x55dd654be1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0e4feb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd6541f020 con 0x55dd654b7b70 2024-03-12T05:56:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0e4feb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd650f2430 con 0x55dd654b7250 2024-03-12T05:56:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0e4feb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd650aee00 con 0x55dd654c0750 2024-03-12T05:56:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0e4feb740 10 client.0 _ll_register_callbacks cb 0x55dd653c6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0db7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7b70 0x55dd654bdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0db7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7b70 0x55dd654bdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57424/0 (socket says 172.21.15.76:57424) 2024-03-12T05:56:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0db7fe640 1 -- 172.21.15.76:0/2487887267 learned_addr learned my addr 172.21.15.76:0/2487887267 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0dbfff640 1 --2- 172.21.15.76:0/2487887267 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7250 0x55dd654bd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0e09b2640 1 --2- 172.21.15.76:0/2487887267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654c0750 0x55dd654be1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0db7fe640 1 -- 172.21.15.76:0/2487887267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654c0750 msgr2=0x55dd654be1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0db7fe640 1 --2- 172.21.15.76:0/2487887267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654c0750 0x55dd654be1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0db7fe640 1 -- 172.21.15.76:0/2487887267 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7250 msgr2=0x55dd654bd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0db7fe640 1 --2- 172.21.15.76:0/2487887267 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7250 0x55dd654bd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0db7fe640 1 -- 172.21.15.76:0/2487887267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d4014990 con 0x55dd654b7b70 2024-03-12T05:56:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0db7fe640 1 --2- 172.21.15.76:0/2487887267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7b70 0x55dd654bdc70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0000c00 tx=0x7fa0d0099850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a2cb2385320e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0bb7fe640 1 -- 172.21.15.76:0/2487887267 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d00caf40 con 0x55dd654b7b70 2024-03-12T05:56:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0bb7fe640 1 -- 172.21.15.76:0/2487887267 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa0d009dfc0 con 0x55dd654b7b70 2024-03-12T05:56:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0e4feb740 1 -- 172.21.15.76:0/2487887267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dd6511e7e0 con 0x55dd654b7b70 2024-03-12T05:56:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0bb7fe640 1 -- 172.21.15.76:0/2487887267 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0d00a2610 con 0x55dd654b7b70 2024-03-12T05:56:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0e4feb740 10 client.15702 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:12.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.814+0000 7fa0e4feb740 1 -- 172.21.15.76:0/2487887267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55dd6541d6a0 con 0x55dd654b7b70 2024-03-12T05:56:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.815+0000 7fa0bb7fe640 1 -- 172.21.15.76:0/2487887267 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0d00b6080 con 0x55dd654b7b70 2024-03-12T05:56:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.815+0000 7fa0bb7fe640 1 -- 172.21.15.76:0/2487887267 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa0d00b8120 con 0x55dd654b7b70 2024-03-12T05:56:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.815+0000 7fa0e4feb740 10 client.15702 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:12.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.815+0000 7fa0e4feb740 10 client.15702 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:56:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.815+0000 7fa0e4feb740 20 client.15702 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.815+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/2487887267 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55dd654d0b00 0x55dd654d2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.815+0000 7fa0e4feb740 4 client.15702 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.815+0000 7fa0e4feb740 1 -- 172.21.15.76:0/2487887267 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd6515b720 con 0x55dd654d0b00 2024-03-12T05:56:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.815+0000 7fa0dbfff640 1 --2- 172.21.15.76:0/2487887267 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55dd654d0b00 0x55dd654d2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.816+0000 7fa0dbfff640 1 --2- 172.21.15.76:0/2487887267 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55dd654d0b00 0x55dd654d2f20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=460859f98475fd0e server_cookie=3b9673fab6d7b7fa in_seq=0 out_seq=0 2024-03-12T05:56:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.816+0000 7fa0bb7fe640 10 client.15702 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:56:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.816+0000 7fa0bb7fe640 1 -- 172.21.15.76:0/2487887267 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55dd6515b720 con 0x55dd654d0b00 2024-03-12T05:56:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.816+0000 7fa0bb7fe640 10 client.15702 handle_command_reply: tid=0 2024-03-12T05:56:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.816+0000 7fa0e4feb740 1 client.15702 shutdown 2024-03-12T05:56:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.816+0000 7fa0e4feb740 1 -- 172.21.15.76:0/2487887267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7b70 msgr2=0x55dd654bdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.816+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/2487887267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7b70 0x55dd654bdc70 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0000c00 tx=0x7fa0d0099850 comp rx=0 tx=0).stop 2024-03-12T05:56:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.817+0000 7fa0e4feb740 1 -- 172.21.15.76:0/2487887267 shutdown_connections 2024-03-12T05:56:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.817+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/2487887267 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55dd654d0b00 0x55dd654d2f20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.817+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/2487887267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654c0750 0x55dd654be1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.817+0000 7fa0bb7fe640 0 client.15702 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:56:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.817+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/2487887267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7b70 0x55dd654bdc70 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.817+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/2487887267 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7250 0x55dd654bd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.817+0000 7fa0e4feb740 1 -- 172.21.15.76:0/2487887267 >> 172.21.15.76:0/2487887267 conn(0x55dd654b1980 msgr2=0x55dd654b36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.817+0000 7fa0e4feb740 1 -- 172.21.15.76:0/2487887267 shutdown_connections 2024-03-12T05:56:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.817+0000 7fa0e4feb740 1 -- 172.21.15.76:0/2487887267 wait complete. 2024-03-12T05:56:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.817+0000 7fa0e4feb740 20 client.15702 trim_cache size 0 max 16384 2024-03-12T05:56:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.828+0000 7fa0e4feb740 1 Processor -- start 2024-03-12T05:56:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.828+0000 7fa0e4feb740 1 -- start start 2024-03-12T05:56:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.828+0000 7fa0e4feb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7250 0x55dd654bfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.828+0000 7fa0e4feb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7b70 0x55dd654bdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.828+0000 7fa0e4feb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654c0750 0x55dd654b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.828+0000 7fa0e4feb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd653a4130 con 0x55dd654b7250 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.828+0000 7fa0e4feb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd654b6c20 con 0x55dd654b7b70 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.828+0000 7fa0e4feb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd654b6570 con 0x55dd654c0750 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.829+0000 7fa0dbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7250 0x55dd654bfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.829+0000 7fa0dbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7250 0x55dd654bfe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57426/0 (socket says 172.21.15.76:57426) 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.829+0000 7fa0dbfff640 1 -- 172.21.15.76:0/3553649745 learned_addr learned my addr 172.21.15.76:0/3553649745 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.829+0000 7fa0dbfff640 1 -- 172.21.15.76:0/3553649745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654c0750 msgr2=0x55dd654b8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.829+0000 7fa0dbfff640 1 --2- 172.21.15.76:0/3553649745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654c0750 0x55dd654b8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.829+0000 7fa0dbfff640 1 -- 172.21.15.76:0/3553649745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7b70 msgr2=0x55dd654bdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.829+0000 7fa0dbfff640 1 --2- 172.21.15.76:0/3553649745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7b70 0x55dd654bdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.829+0000 7fa0dbfff640 1 -- 172.21.15.76:0/3553649745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd653b9c20 con 0x55dd654b7250 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.829+0000 7fa0dbfff640 1 --2- 172.21.15.76:0/3553649745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7250 0x55dd654bfe60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc02c210 tx=0x7fa0cc02c290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f78be19e25976c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.829+0000 7fa0c1ffb640 1 -- 172.21.15.76:0/3553649745 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc044040 con 0x55dd654b7250 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.829+0000 7fa0c1ffb640 1 -- 172.21.15.76:0/3553649745 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0cc046040 con 0x55dd654b7250 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.829+0000 7fa0c1ffb640 1 -- 172.21.15.76:0/3553649745 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc004df0 con 0x55dd654b7250 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.830+0000 7fa0e4feb740 1 -- 172.21.15.76:0/3553649745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7250 msgr2=0x55dd654bfe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.830+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/3553649745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7250 0x55dd654bfe60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc02c210 tx=0x7fa0cc02c290 comp rx=0 tx=0).stop 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.830+0000 7fa0e4feb740 1 -- 172.21.15.76:0/3553649745 shutdown_connections 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.830+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/3553649745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654c0750 0x55dd654b8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.830+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/3553649745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7b70 0x55dd654bdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.830+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/3553649745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7250 0x55dd654bfe60 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.830+0000 7fa0e4feb740 1 -- 172.21.15.76:0/3553649745 >> 172.21.15.76:0/3553649745 conn(0x55dd654b1980 msgr2=0x55dd654aca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.830+0000 7fa0e4feb740 1 -- 172.21.15.76:0/3553649745 shutdown_connections 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.830+0000 7fa0e4feb740 1 -- 172.21.15.76:0/3553649745 wait complete. 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.830+0000 7fa0e4feb740 1 -- start start 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.830+0000 7fa0e4feb740 1 Processor -- start 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0e4feb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7250 0x55dd654c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0e4feb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7b70 0x55dd654c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0e4feb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654c0750 0x55dd65527030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0e4feb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd654c7210 con 0x55dd654b7250 2024-03-12T05:56:12.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0e4feb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd65527570 con 0x55dd654b7b70 2024-03-12T05:56:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0e4feb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd655276e0 con 0x55dd654c0750 2024-03-12T05:56:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0e4feb740 10 client.0 _ll_register_callbacks cb 0x7fa0d400ade0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0dbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7250 0x55dd654c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0dbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7250 0x55dd654c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57440/0 (socket says 172.21.15.76:57440) 2024-03-12T05:56:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0dbfff640 1 -- 172.21.15.76:0/536410091 learned_addr learned my addr 172.21.15.76:0/536410091 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0db7fe640 1 --2- 172.21.15.76:0/536410091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7b70 0x55dd654c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0e09b2640 1 --2- 172.21.15.76:0/536410091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654c0750 0x55dd65527030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0dbfff640 1 -- 172.21.15.76:0/536410091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654c0750 msgr2=0x55dd65527030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0dbfff640 1 --2- 172.21.15.76:0/536410091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654c0750 0x55dd65527030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0dbfff640 1 -- 172.21.15.76:0/536410091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7b70 msgr2=0x55dd654c6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0dbfff640 1 --2- 172.21.15.76:0/536410091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7b70 0x55dd654c6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0dbfff640 1 -- 172.21.15.76:0/536410091 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0dc10e820 con 0x55dd654b7250 2024-03-12T05:56:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0dbfff640 1 --2- 172.21.15.76:0/536410091 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7250 0x55dd654c65d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc02cd60 tx=0x7fa0cc02cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dedf68a9bf26b43e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0bb7fe640 1 -- 172.21.15.76:0/536410091 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc044040 con 0x55dd654b7250 2024-03-12T05:56:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0bb7fe640 1 -- 172.21.15.76:0/536410091 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa0cc02c880 con 0x55dd654b7250 2024-03-12T05:56:12.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.831+0000 7fa0e4feb740 1 -- 172.21.15.76:0/536410091 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa0d401a040 con 0x55dd654b7250 2024-03-12T05:56:12.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.832+0000 7fa0bb7fe640 1 -- 172.21.15.76:0/536410091 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0cc004300 con 0x55dd654b7250 2024-03-12T05:56:12.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.832+0000 7fa0e4feb740 10 client.15708 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:12.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.832+0000 7fa0e4feb740 1 -- 172.21.15.76:0/536410091 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55dd653df8c0 con 0x55dd654b7250 2024-03-12T05:56:12.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.832+0000 7fa0bb7fe640 1 -- 172.21.15.76:0/536410091 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0d401a040 con 0x55dd654b7250 2024-03-12T05:56:12.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.832+0000 7fa0bb7fe640 1 -- 172.21.15.76:0/536410091 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa0cc03c040 con 0x55dd654b7250 2024-03-12T05:56:12.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.832+0000 7fa0e4feb740 10 client.15708 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:12.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.832+0000 7fa0e4feb740 10 client.15708 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:56:12.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.832+0000 7fa0e4feb740 20 client.15708 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:12.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.832+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/536410091 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55dd65528b40 0x55dd6552af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.832+0000 7fa0e4feb740 4 client.15708 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:56:12.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.832+0000 7fa0e4feb740 1 -- 172.21.15.76:0/536410091 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dd651aa890 con 0x55dd65528b40 2024-03-12T05:56:12.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.832+0000 7fa0db7fe640 1 --2- 172.21.15.76:0/536410091 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55dd65528b40 0x55dd6552af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0bb7fe640 10 client.15708 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:56:12.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0db7fe640 1 --2- 172.21.15.76:0/536410091 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55dd65528b40 0x55dd6552af60 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a41192d06b37959b server_cookie=ee9ff922d6beb406 in_seq=0 out_seq=0 2024-03-12T05:56:12.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0bb7fe640 1 -- 172.21.15.76:0/536410091 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dd651aa890 con 0x55dd65528b40 2024-03-12T05:56:12.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0bb7fe640 10 client.15708 handle_command_reply: tid=0 2024-03-12T05:56:12.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0e4feb740 1 client.15708 shutdown 2024-03-12T05:56:12.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0e4feb740 1 -- 172.21.15.76:0/536410091 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7250 msgr2=0x55dd654c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/536410091 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7250 0x55dd654c65d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc02cd60 tx=0x7fa0cc02cd90 comp rx=0 tx=0).stop 2024-03-12T05:56:12.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0e4feb740 1 -- 172.21.15.76:0/536410091 shutdown_connections 2024-03-12T05:56:12.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/536410091 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55dd65528b40 0x55dd6552af60 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/536410091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dd654c0750 0x55dd65527030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/536410091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dd654b7b70 0x55dd654c6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0bb7fe640 0 client.15708 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:56:12.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0e4feb740 1 --2- 172.21.15.76:0/536410091 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dd654b7250 0x55dd654c65d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0e4feb740 1 -- 172.21.15.76:0/536410091 >> 172.21.15.76:0/536410091 conn(0x55dd654b1980 msgr2=0x55dd654b8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0e4feb740 1 -- 172.21.15.76:0/536410091 shutdown_connections 2024-03-12T05:56:12.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0e4feb740 1 -- 172.21.15.76:0/536410091 wait complete. 2024-03-12T05:56:12.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.833+0000 7fa0e4feb740 20 client.15708 trim_cache size 0 max 16384 2024-03-12T05:56:12.845 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.834+0000 7fa0e2415640 1 -- 172.21.15.76:0/385168508 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa0c40a1cc0 msgr2=0x7fa0c40a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.834+0000 7fa0e2415640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa0c40a1cc0 0x7fa0c40a4180 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa0d400afd0 tx=0x7fa0d4016000 comp rx=0 tx=0).stop 2024-03-12T05:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.834+0000 7fa0e2415640 1 -- 172.21.15.76:0/385168508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0dc102cc0 msgr2=0x7fa0dc19d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.834+0000 7fa0e2415640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0dc102cc0 0x7fa0dc19d950 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000d950 tx=0x7fa0d000de10 comp rx=0 tx=0).stop 2024-03-12T05:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.834+0000 7fa0e2415640 1 -- 172.21.15.76:0/385168508 shutdown_connections 2024-03-12T05:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.834+0000 7fa0e2415640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa0c40a1cc0 0x7fa0c40a4180 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.834+0000 7fa0e2415640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0dc10b770 0x7fa0dc197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.834+0000 7fa0e2415640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0dc102cc0 0x7fa0dc19d950 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.834+0000 7fa0e2415640 1 --2- 172.21.15.76:0/385168508 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa0dc1023a0 0x7fa0dc19d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.834+0000 7fa0e2415640 1 -- 172.21.15.76:0/385168508 >> 172.21.15.76:0/385168508 conn(0x7fa0dc077230 msgr2=0x7fa0dc0fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.834+0000 7fa0e2415640 1 -- 172.21.15.76:0/385168508 shutdown_connections 2024-03-12T05:56:12.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.834+0000 7fa0e2415640 1 -- 172.21.15.76:0/385168508 wait complete. 2024-03-12T05:56:12.849 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:56:12.849 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:56:12.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.951+0000 7fb652177640 1 -- 172.21.15.76:0/3657121747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb64c1023a0 msgr2=0x7fb64c102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.951+0000 7fb652177640 1 --2- 172.21.15.76:0/3657121747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb64c1023a0 0x7fb64c102780 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009900 tx=0x7fb63c02bcd0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.951+0000 7fb652177640 1 -- 172.21.15.76:0/3657121747 shutdown_connections 2024-03-12T05:56:12.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.951+0000 7fb652177640 1 --2- 172.21.15.76:0/3657121747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb64c10b770 0x7fb64c10dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.951+0000 7fb652177640 1 --2- 172.21.15.76:0/3657121747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb64c102cc0 0x7fb64c10b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.951+0000 7fb652177640 1 --2- 172.21.15.76:0/3657121747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb64c1023a0 0x7fb64c102780 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.951+0000 7fb652177640 1 -- 172.21.15.76:0/3657121747 >> 172.21.15.76:0/3657121747 conn(0x7fb64c077230 msgr2=0x7fb64c077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.951+0000 7fb652177640 1 -- 172.21.15.76:0/3657121747 shutdown_connections 2024-03-12T05:56:12.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.951+0000 7fb652177640 1 -- 172.21.15.76:0/3657121747 wait complete. 2024-03-12T05:56:12.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.951+0000 7fb652177640 1 Processor -- start 2024-03-12T05:56:12.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb652177640 1 -- start start 2024-03-12T05:56:12.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb652177640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb64c1023a0 0x7fb64c19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb652177640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb64c102cc0 0x7fb64c19d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb652177640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb64c10b770 0x7fb64c197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb652177640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c197c60 con 0x7fb64c102cc0 2024-03-12T05:56:12.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb652177640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c197dd0 con 0x7fb64c10b770 2024-03-12T05:56:12.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb652177640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64c197f90 con 0x7fb64c1023a0 2024-03-12T05:56:12.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb64affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb64c102cc0 0x7fb64c19d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb64affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb64c102cc0 0x7fb64c19d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57458/0 (socket says 172.21.15.76:57458) 2024-03-12T05:56:12.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb64affd640 1 -- 172.21.15.76:0/3563020121 learned_addr learned my addr 172.21.15.76:0/3563020121 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb64b7fe640 1 --2- 172.21.15.76:0/3563020121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb64c1023a0 0x7fb64c19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb64bfff640 1 --2- 172.21.15.76:0/3563020121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb64c10b770 0x7fb64c197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb64affd640 1 -- 172.21.15.76:0/3563020121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb64c1023a0 msgr2=0x7fb64c19d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb64affd640 1 --2- 172.21.15.76:0/3563020121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb64c1023a0 0x7fb64c19d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb64affd640 1 -- 172.21.15.76:0/3563020121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb64c10b770 msgr2=0x7fb64c197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb64affd640 1 --2- 172.21.15.76:0/3563020121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb64c10b770 0x7fb64c197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.952+0000 7fb64affd640 1 -- 172.21.15.76:0/3563020121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63c009560 con 0x7fb64c102cc0 2024-03-12T05:56:12.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.953+0000 7fb64affd640 1 --2- 172.21.15.76:0/3563020121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb64c102cc0 0x7fb64c19d9a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb64000c920 tx=0x7fb64000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.953+0000 7fb648ff9640 1 -- 172.21.15.76:0/3563020121 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640007c70 con 0x7fb64c102cc0 2024-03-12T05:56:12.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.953+0000 7fb648ff9640 1 -- 172.21.15.76:0/3563020121 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb640007dd0 con 0x7fb64c102cc0 2024-03-12T05:56:12.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.953+0000 7fb652177640 1 -- 172.21.15.76:0/3563020121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64c198270 con 0x7fb64c102cc0 2024-03-12T05:56:12.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.953+0000 7fb648ff9640 1 -- 172.21.15.76:0/3563020121 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640018860 con 0x7fb64c102cc0 2024-03-12T05:56:12.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.953+0000 7fb652177640 1 -- 172.21.15.76:0/3563020121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64c1a4500 con 0x7fb64c102cc0 2024-03-12T05:56:12.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.955+0000 7fb648ff9640 1 -- 172.21.15.76:0/3563020121 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fb64000fac0 con 0x7fb64c102cc0 2024-03-12T05:56:12.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.955+0000 7fb654e21740 1 Processor -- start 2024-03-12T05:56:12.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.955+0000 7fb654e21740 1 -- start start 2024-03-12T05:56:12.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.955+0000 7fb654e21740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 0x5653ca1d3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.955+0000 7fb654e21740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 0x5653ca1dc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.955+0000 7fb654e21740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1deb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.955+0000 7fb654e21740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ca18e960 con 0x5653ca1dc750 2024-03-12T05:56:12.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.955+0000 7fb654e21740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ca142310 con 0x5653ca1d3250 2024-03-12T05:56:12.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.955+0000 7fb654e21740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ca13f840 con 0x5653ca1d3b70 2024-03-12T05:56:12.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.955+0000 7fb64affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1deb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb64affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1deb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57468/0 (socket says 172.21.15.76:57468) 2024-03-12T05:56:12.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb64affd640 1 -- 172.21.15.76:0/2283462604 learned_addr learned my addr 172.21.15.76:0/2283462604 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb648ff9640 1 --2- 172.21.15.76:0/3563020121 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb6340a1cc0 0x7fb6340a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb64bfff640 1 --2- 172.21.15.76:0/2283462604 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 0x5653ca1d3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb64b7fe640 1 --2- 172.21.15.76:0/2283462604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 0x5653ca1dc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb64bfff640 1 --2- 172.21.15.76:0/3563020121 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb6340a1cc0 0x7fb6340a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb64affd640 1 -- 172.21.15.76:0/2283462604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 msgr2=0x5653ca1dc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb64affd640 1 --2- 172.21.15.76:0/2283462604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 0x5653ca1dc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb64affd640 1 -- 172.21.15.76:0/2283462604 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 msgr2=0x5653ca1d3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb64affd640 1 --2- 172.21.15.76:0/2283462604 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 0x5653ca1d3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb648ff9640 1 -- 172.21.15.76:0/3563020121 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb6400cdb20 con 0x7fb64c102cc0 2024-03-12T05:56:12.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb64affd640 1 -- 172.21.15.76:0/2283462604 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653ca0fd060 con 0x5653ca1dc750 2024-03-12T05:56:12.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb64bfff640 1 --2- 172.21.15.76:0/2283462604 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 0x5653ca1d3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:12.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb64b7fe640 1 --2- 172.21.15.76:0/2283462604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 0x5653ca1dc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:12.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb64affd640 1 --2- 172.21.15.76:0/2283462604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1deb40 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb6400cb1b0 tx=0x7fb64000fec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5315c6d22cf88f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb64bfff640 1 --2- 172.21.15.76:0/3563020121 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb6340a1cc0 0x7fb6340a4180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb64400a6d0 tx=0x7fb64400a420 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb631ffb640 1 -- 172.21.15.76:0/2283462604 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64009a070 con 0x5653ca1dc750 2024-03-12T05:56:12.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb631ffb640 1 -- 172.21.15.76:0/2283462604 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6400d2040 con 0x5653ca1dc750 2024-03-12T05:56:12.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.956+0000 7fb631ffb640 1 -- 172.21.15.76:0/2283462604 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64009e680 con 0x5653ca1dc750 2024-03-12T05:56:12.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.957+0000 7fb654e21740 1 -- 172.21.15.76:0/2283462604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 msgr2=0x5653ca1deb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.957+0000 7fb654e21740 1 --2- 172.21.15.76:0/2283462604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1deb40 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb6400cb1b0 tx=0x7fb64000fec0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.957+0000 7fb654e21740 1 -- 172.21.15.76:0/2283462604 shutdown_connections 2024-03-12T05:56:12.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.957+0000 7fb654e21740 1 --2- 172.21.15.76:0/2283462604 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1deb40 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.957+0000 7fb654e21740 1 --2- 172.21.15.76:0/2283462604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 0x5653ca1dc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.957+0000 7fb654e21740 1 --2- 172.21.15.76:0/2283462604 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 0x5653ca1d3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.957+0000 7fb654e21740 1 -- 172.21.15.76:0/2283462604 >> 172.21.15.76:0/2283462604 conn(0x5653ca1cd980 msgr2=0x5653ca1cdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.957+0000 7fb654e21740 1 -- 172.21.15.76:0/2283462604 shutdown_connections 2024-03-12T05:56:12.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.957+0000 7fb654e21740 1 -- 172.21.15.76:0/2283462604 wait complete. 2024-03-12T05:56:12.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.957+0000 7fb654e21740 1 -- start start 2024-03-12T05:56:12.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb654e21740 1 Processor -- start 2024-03-12T05:56:12.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb654e21740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 0x5653ca1d9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb654e21740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 0x5653ca1d9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb654e21740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1da1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb654e21740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ca13b020 con 0x5653ca1dc750 2024-03-12T05:56:12.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb654e21740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653c9e0e430 con 0x5653ca1d3250 2024-03-12T05:56:12.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb654e21740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653c9dcae00 con 0x5653ca1d3b70 2024-03-12T05:56:12.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb654e21740 10 client.0 _ll_register_callbacks cb 0x5653ca0e2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:12.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb64bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1da1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb64bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1da1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57478/0 (socket says 172.21.15.76:57478) 2024-03-12T05:56:12.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb64bfff640 1 -- 172.21.15.76:0/2848620562 learned_addr learned my addr 172.21.15.76:0/2848620562 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb64b7fe640 1 --2- 172.21.15.76:0/2848620562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 0x5653ca1d9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb64affd640 1 --2- 172.21.15.76:0/2848620562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 0x5653ca1d9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb64bfff640 1 -- 172.21.15.76:0/2848620562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 msgr2=0x5653ca1d9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb64bfff640 1 --2- 172.21.15.76:0/2848620562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 0x5653ca1d9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb64bfff640 1 -- 172.21.15.76:0/2848620562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 msgr2=0x5653ca1d9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb64bfff640 1 --2- 172.21.15.76:0/2848620562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 0x5653ca1d9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.958+0000 7fb64bfff640 1 -- 172.21.15.76:0/2848620562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64c1a4500 con 0x5653ca1dc750 2024-03-12T05:56:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb64bfff640 1 --2- 172.21.15.76:0/2848620562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1da1b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb644011c60 tx=0x7fb644004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a47d1a6d90c727a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb62affd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb62affd640 1 -- 172.21.15.76:0/2848620562 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644011c90 con 0x5653ca1dc750 2024-03-12T05:56:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb62affd640 1 -- 172.21.15.76:0/2848620562 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb644011df0 con 0x5653ca1dc750 2024-03-12T05:56:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb654e21740 1 -- 172.21.15.76:0/2848620562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5653c9e3a7e0 con 0x5653ca1dc750 2024-03-12T05:56:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb62affd640 1 -- 172.21.15.76:0/2848620562 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb644004b20 con 0x5653ca1dc750 2024-03-12T05:56:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb654e21740 10 client.15720 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb654e21740 1 -- 172.21.15.76:0/2848620562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5653ca1396a0 con 0x5653ca1dc750 2024-03-12T05:56:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb62affd640 1 -- 172.21.15.76:0/2848620562 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5653c9e3a7e0 con 0x5653ca1dc750 2024-03-12T05:56:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb62affd640 1 -- 172.21.15.76:0/2848620562 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb64400f450 con 0x5653ca1dc750 2024-03-12T05:56:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb654e21740 10 client.15720 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb654e21740 10 client.15720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb654e21740 20 client.15720 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb654e21740 1 --2- 172.21.15.76:0/2848620562 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653ca1ecb60 0x5653ca1eef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb654e21740 4 client.15720 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:12.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.959+0000 7fb654e21740 1 -- 172.21.15.76:0/2848620562 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5653c9e77720 con 0x5653ca1ecb60 2024-03-12T05:56:12.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.960+0000 7fb64b7fe640 1 --2- 172.21.15.76:0/2848620562 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653ca1ecb60 0x5653ca1eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.960+0000 7fb62affd640 10 client.15720 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:12.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.960+0000 7fb64b7fe640 1 --2- 172.21.15.76:0/2848620562 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653ca1ecb60 0x5653ca1eef80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fd8405a1d72b50b server_cookie=bc0f3e8fa96e93b8 in_seq=0 out_seq=0 2024-03-12T05:56:12.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb62affd640 1 -- 172.21.15.76:0/2848620562 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5653c9e77720 con 0x5653ca1ecb60 2024-03-12T05:56:12.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb62affd640 10 client.15720 handle_command_reply: tid=0 2024-03-12T05:56:12.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb654e21740 1 client.15720 shutdown 2024-03-12T05:56:12.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb654e21740 1 -- 172.21.15.76:0/2848620562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 msgr2=0x5653ca1da1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb654e21740 1 --2- 172.21.15.76:0/2848620562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1da1b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb644011c60 tx=0x7fb644004a40 comp rx=0 tx=0).stop 2024-03-12T05:56:12.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb654e21740 1 -- 172.21.15.76:0/2848620562 shutdown_connections 2024-03-12T05:56:12.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb654e21740 1 --2- 172.21.15.76:0/2848620562 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653ca1ecb60 0x5653ca1eef80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb654e21740 1 --2- 172.21.15.76:0/2848620562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1da1b0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb62affd640 0 client.15720 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:12.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb654e21740 1 --2- 172.21.15.76:0/2848620562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 0x5653ca1d9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb654e21740 1 --2- 172.21.15.76:0/2848620562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 0x5653ca1d9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb654e21740 1 -- 172.21.15.76:0/2848620562 >> 172.21.15.76:0/2848620562 conn(0x5653ca1cd980 msgr2=0x5653ca1cf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb654e21740 1 -- 172.21.15.76:0/2848620562 shutdown_connections 2024-03-12T05:56:12.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb654e21740 1 -- 172.21.15.76:0/2848620562 wait complete. 2024-03-12T05:56:12.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.961+0000 7fb654e21740 20 client.15720 trim_cache size 0 max 16384 2024-03-12T05:56:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.983+0000 7fb654e21740 1 Processor -- start 2024-03-12T05:56:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.983+0000 7fb654e21740 1 -- start start 2024-03-12T05:56:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.983+0000 7fb654e21740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 0x5653ca1e39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.983+0000 7fb654e21740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 0x5653ca1d9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.983+0000 7fb654e21740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.983+0000 7fb654e21740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ca0c0130 con 0x5653ca1dc750 2024-03-12T05:56:12.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.983+0000 7fb654e21740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ca1ce980 con 0x5653ca1d3250 2024-03-12T05:56:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.983+0000 7fb654e21740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ca1e3f20 con 0x5653ca1d3b70 2024-03-12T05:56:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.983+0000 7fb64bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.983+0000 7fb64bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1cca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57488/0 (socket says 172.21.15.76:57488) 2024-03-12T05:56:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.983+0000 7fb64bfff640 1 -- 172.21.15.76:0/536329930 learned_addr learned my addr 172.21.15.76:0/536329930 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.983+0000 7fb64b7fe640 1 --2- 172.21.15.76:0/536329930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 0x5653ca1e39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.983+0000 7fb64bfff640 1 -- 172.21.15.76:0/536329930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 msgr2=0x5653ca1d9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.983+0000 7fb64bfff640 1 --2- 172.21.15.76:0/536329930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 0x5653ca1d9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.984+0000 7fb64bfff640 1 -- 172.21.15.76:0/536329930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 msgr2=0x5653ca1e39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.984+0000 7fb64bfff640 1 --2- 172.21.15.76:0/536329930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 0x5653ca1e39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.984+0000 7fb64bfff640 1 -- 172.21.15.76:0/536329930 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653ca0d5c20 con 0x5653ca1dc750 2024-03-12T05:56:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.984+0000 7fb64b7fe640 1 --2- 172.21.15.76:0/536329930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 0x5653ca1e39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:56:12.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.984+0000 7fb64bfff640 1 --2- 172.21.15.76:0/536329930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1cca00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb64400a940 tx=0x7fb644011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa60418d0160d5b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.984+0000 7fb6317fa640 1 -- 172.21.15.76:0/536329930 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64400f450 con 0x5653ca1dc750 2024-03-12T05:56:12.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.984+0000 7fb6317fa640 1 -- 172.21.15.76:0/536329930 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb64400f5b0 con 0x5653ca1dc750 2024-03-12T05:56:12.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.984+0000 7fb6317fa640 1 -- 172.21.15.76:0/536329930 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644013e60 con 0x5653ca1dc750 2024-03-12T05:56:12.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.984+0000 7fb654e21740 1 -- 172.21.15.76:0/536329930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 msgr2=0x5653ca1cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.984+0000 7fb654e21740 1 --2- 172.21.15.76:0/536329930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1cca00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb64400a940 tx=0x7fb644011890 comp rx=0 tx=0).stop 2024-03-12T05:56:12.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 -- 172.21.15.76:0/536329930 shutdown_connections 2024-03-12T05:56:12.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 --2- 172.21.15.76:0/536329930 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca1cca00 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 --2- 172.21.15.76:0/536329930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3b70 0x5653ca1d9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 --2- 172.21.15.76:0/536329930 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3250 0x5653ca1e39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 -- 172.21.15.76:0/536329930 >> 172.21.15.76:0/536329930 conn(0x5653ca1cd980 msgr2=0x5653ca1cc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 -- 172.21.15.76:0/536329930 shutdown_connections 2024-03-12T05:56:12.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 -- 172.21.15.76:0/536329930 wait complete. 2024-03-12T05:56:12.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 -- start start 2024-03-12T05:56:12.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 Processor -- start 2024-03-12T05:56:12.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3250 0x5653ca241330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3b70 0x5653ca241870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca23b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ca23bd00 con 0x5653ca1dc750 2024-03-12T05:56:12.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ca23be70 con 0x5653ca1d3b70 2024-03-12T05:56:12.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653ca23c030 con 0x5653ca1d3250 2024-03-12T05:56:12.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb654e21740 10 client.0 _ll_register_callbacks cb 0x7fb6400d14c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:12.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb64bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca23b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.985+0000 7fb64bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca23b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57490/0 (socket says 172.21.15.76:57490) 2024-03-12T05:56:12.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb64bfff640 1 -- 172.21.15.76:0/2157010743 learned_addr learned my addr 172.21.15.76:0/2157010743 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:12.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb64b7fe640 1 --2- 172.21.15.76:0/2157010743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3250 0x5653ca241330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb64affd640 1 --2- 172.21.15.76:0/2157010743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3b70 0x5653ca241870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb64bfff640 1 -- 172.21.15.76:0/2157010743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3250 msgr2=0x5653ca241330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb64bfff640 1 --2- 172.21.15.76:0/2157010743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3250 0x5653ca241330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb64bfff640 1 -- 172.21.15.76:0/2157010743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3b70 msgr2=0x5653ca241870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:12.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb64bfff640 1 --2- 172.21.15.76:0/2157010743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3b70 0x5653ca241870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb64bfff640 1 -- 172.21.15.76:0/2157010743 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5653ca1396a0 con 0x5653ca1dc750 2024-03-12T05:56:12.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb64bfff640 1 --2- 172.21.15.76:0/2157010743 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca23b740 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb64400f760 tx=0x7fb64400f4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2db068da7d899e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:12.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb62affd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:12.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb62affd640 1 -- 172.21.15.76:0/2157010743 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644013c00 con 0x5653ca1dc750 2024-03-12T05:56:12.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb62affd640 1 -- 172.21.15.76:0/2157010743 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb644013d60 con 0x5653ca1dc750 2024-03-12T05:56:12.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb654e21740 1 -- 172.21.15.76:0/2157010743 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb6400d2040 con 0x5653ca1dc750 2024-03-12T05:56:12.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb62affd640 1 -- 172.21.15.76:0/2157010743 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb644004a00 con 0x5653ca1dc750 2024-03-12T05:56:12.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb654e21740 10 client.15726 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:12.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.986+0000 7fb654e21740 1 -- 172.21.15.76:0/2157010743 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5653ca0fb8c0 con 0x5653ca1dc750 2024-03-12T05:56:12.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.987+0000 7fb62affd640 1 -- 172.21.15.76:0/2157010743 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6400d2040 con 0x5653ca1dc750 2024-03-12T05:56:12.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.987+0000 7fb62affd640 1 -- 172.21.15.76:0/2157010743 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb644005820 con 0x5653ca1dc750 2024-03-12T05:56:12.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.987+0000 7fb654e21740 10 client.15726 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:12.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.987+0000 7fb654e21740 10 client.15726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:12.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.987+0000 7fb654e21740 20 client.15726 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:12.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.987+0000 7fb654e21740 1 --2- 172.21.15.76:0/2157010743 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653ca248cf0 0x5653ca24b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:12.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.987+0000 7fb654e21740 4 client.15726 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:56:12.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.987+0000 7fb654e21740 1 -- 172.21.15.76:0/2157010743 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5653c9ec6890 con 0x5653ca248cf0 2024-03-12T05:56:12.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.987+0000 7fb64b7fe640 1 --2- 172.21.15.76:0/2157010743 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653ca248cf0 0x5653ca24b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:12.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.987+0000 7fb64b7fe640 1 --2- 172.21.15.76:0/2157010743 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653ca248cf0 0x5653ca24b0e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ca1aee9a33ebd13 server_cookie=f0dcc61b99351fa6 in_seq=0 out_seq=0 2024-03-12T05:56:12.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.987+0000 7fb62affd640 10 client.15726 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:12.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb62affd640 1 -- 172.21.15.76:0/2157010743 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5653c9ec6890 con 0x5653ca248cf0 2024-03-12T05:56:12.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb62affd640 10 client.15726 handle_command_reply: tid=0 2024-03-12T05:56:12.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb654e21740 1 client.15726 shutdown 2024-03-12T05:56:12.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb654e21740 1 -- 172.21.15.76:0/2157010743 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 msgr2=0x5653ca23b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:12.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb654e21740 1 --2- 172.21.15.76:0/2157010743 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca23b740 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fb64400f760 tx=0x7fb64400f4c0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb654e21740 1 -- 172.21.15.76:0/2157010743 shutdown_connections 2024-03-12T05:56:12.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb654e21740 1 --2- 172.21.15.76:0/2157010743 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653ca248cf0 0x5653ca24b0e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb654e21740 1 --2- 172.21.15.76:0/2157010743 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653ca1dc750 0x5653ca23b740 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb654e21740 1 --2- 172.21.15.76:0/2157010743 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653ca1d3b70 0x5653ca241870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb62affd640 0 client.15726 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:12.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb654e21740 1 --2- 172.21.15.76:0/2157010743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653ca1d3250 0x5653ca241330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:12.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb654e21740 1 -- 172.21.15.76:0/2157010743 >> 172.21.15.76:0/2157010743 conn(0x5653ca1cd980 msgr2=0x5653ca1cf640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:12.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb654e21740 1 -- 172.21.15.76:0/2157010743 shutdown_connections 2024-03-12T05:56:12.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb654e21740 1 -- 172.21.15.76:0/2157010743 wait complete. 2024-03-12T05:56:12.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb654e21740 20 client.15726 trim_cache size 0 max 16384 2024-03-12T05:56:13.000 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:56:13.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb652177640 1 -- 172.21.15.76:0/3563020121 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb6340a1cc0 msgr2=0x7fb6340a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:13.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb652177640 1 --2- 172.21.15.76:0/3563020121 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb6340a1cc0 0x7fb6340a4180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb64400a6d0 tx=0x7fb64400a420 comp rx=0 tx=0).stop 2024-03-12T05:56:13.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.988+0000 7fb652177640 1 -- 172.21.15.76:0/3563020121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb64c102cc0 msgr2=0x7fb64c19d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:13.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.989+0000 7fb652177640 1 --2- 172.21.15.76:0/3563020121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb64c102cc0 0x7fb64c19d9a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb64000c920 tx=0x7fb64000cde0 comp rx=0 tx=0).stop 2024-03-12T05:56:13.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.989+0000 7fb652177640 1 -- 172.21.15.76:0/3563020121 shutdown_connections 2024-03-12T05:56:13.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.989+0000 7fb652177640 1 --2- 172.21.15.76:0/3563020121 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb6340a1cc0 0x7fb6340a4180 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:13.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.989+0000 7fb652177640 1 --2- 172.21.15.76:0/3563020121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb64c10b770 0x7fb64c197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:13.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.989+0000 7fb652177640 1 --2- 172.21.15.76:0/3563020121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb64c102cc0 0x7fb64c19d9a0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:13.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.989+0000 7fb652177640 1 --2- 172.21.15.76:0/3563020121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb64c1023a0 0x7fb64c19d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:13.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.989+0000 7fb652177640 1 -- 172.21.15.76:0/3563020121 >> 172.21.15.76:0/3563020121 conn(0x7fb64c077230 msgr2=0x7fb64c0fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:13.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.989+0000 7fb652177640 1 -- 172.21.15.76:0/3563020121 shutdown_connections 2024-03-12T05:56:13.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:12.989+0000 7fb652177640 1 -- 172.21.15.76:0/3563020121 wait complete. 2024-03-12T05:56:13.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:13 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T05:56:13.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:13 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:13.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:13 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:56:13.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:13 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:13.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:13 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:56:13.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:13 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4272162475' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:56:13.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:13 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T05:56:13.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:13 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:13.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:13 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:56:13.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:13 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:13.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:13 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:56:13.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:13 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4272162475' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:56:13.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:13 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T05:56:13.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:13 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:13.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:13 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:56:13.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:13 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:13.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:13 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:56:13.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:13 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4272162475' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:56:14.005 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:56:14.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:56:14.005 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:56:14.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.108+0000 7f8625363640 1 -- 172.21.15.76:0/2308988485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8620100290 msgr2=0x7f8620100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.108+0000 7f8625363640 1 --2- 172.21.15.76:0/2308988485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8620100290 0x7f8620100670 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8610009930 tx=0x7f861002bc30 comp rx=0 tx=0).stop 2024-03-12T05:56:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.108+0000 7f8625363640 1 -- 172.21.15.76:0/2308988485 shutdown_connections 2024-03-12T05:56:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.108+0000 7f8625363640 1 --2- 172.21.15.76:0/2308988485 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8620105360 0x7f862010cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.108+0000 7f8625363640 1 --2- 172.21.15.76:0/2308988485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8620100c40 0x7f8620104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.108+0000 7f8625363640 1 --2- 172.21.15.76:0/2308988485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8620100290 0x7f8620100670 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.108+0000 7f8625363640 1 -- 172.21.15.76:0/2308988485 >> 172.21.15.76:0/2308988485 conn(0x7f8620077230 msgr2=0x7f8620077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.109+0000 7f8625363640 1 -- 172.21.15.76:0/2308988485 shutdown_connections 2024-03-12T05:56:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.109+0000 7f8625363640 1 -- 172.21.15.76:0/2308988485 wait complete. 2024-03-12T05:56:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.109+0000 7f8625363640 1 Processor -- start 2024-03-12T05:56:14.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.109+0000 7f8625363640 1 -- start start 2024-03-12T05:56:14.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.109+0000 7f8625363640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8620100290 0x7f8620198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.109+0000 7f8625363640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8620105360 0x7f8620198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.109+0000 7f8625363640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8620199150 0x7f86201a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.109+0000 7f8625363640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86201a3da0 con 0x7f8620199150 2024-03-12T05:56:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f8625363640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86201a3f10 con 0x7f8620100290 2024-03-12T05:56:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f8625363640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86201a40d0 con 0x7f8620105360 2024-03-12T05:56:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f861f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8620199150 0x7f86201a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f861f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8620199150 0x7f86201a3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57506/0 (socket says 172.21.15.76:57506) 2024-03-12T05:56:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f861f7fe640 1 -- 172.21.15.76:0/3429517277 learned_addr learned my addr 172.21.15.76:0/3429517277 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f861e7fc640 1 --2- 172.21.15.76:0/3429517277 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8620105360 0x7f8620198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f861effd640 1 --2- 172.21.15.76:0/3429517277 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8620100290 0x7f8620198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f861f7fe640 1 -- 172.21.15.76:0/3429517277 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8620105360 msgr2=0x7f8620198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f861f7fe640 1 --2- 172.21.15.76:0/3429517277 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8620105360 0x7f8620198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f861f7fe640 1 -- 172.21.15.76:0/3429517277 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8620100290 msgr2=0x7f8620198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f861f7fe640 1 --2- 172.21.15.76:0/3429517277 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8620100290 0x7f8620198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f861f7fe640 1 -- 172.21.15.76:0/3429517277 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8610009560 con 0x7f8620199150 2024-03-12T05:56:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f861f7fe640 1 --2- 172.21.15.76:0/3429517277 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8620199150 0x7f86201a3700 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f861800da40 tx=0x7f861800df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:14.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f8607fff640 1 -- 172.21.15.76:0/3429517277 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861800bdd0 con 0x7f8620199150 2024-03-12T05:56:14.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f8607fff640 1 -- 172.21.15.76:0/3429517277 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8618009890 con 0x7f8620199150 2024-03-12T05:56:14.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f8625363640 1 -- 172.21.15.76:0/3429517277 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86201a43b0 con 0x7f8620199150 2024-03-12T05:56:14.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f8607fff640 1 -- 172.21.15.76:0/3429517277 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86180106c0 con 0x7f8620199150 2024-03-12T05:56:14.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.110+0000 7f8625363640 1 -- 172.21.15.76:0/3429517277 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86201a4870 con 0x7f8620199150 2024-03-12T05:56:14.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f8607fff640 1 -- 172.21.15.76:0/3429517277 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f86180041b0 con 0x7f8620199150 2024-03-12T05:56:14.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f862800b740 1 Processor -- start 2024-03-12T05:56:14.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f862800b740 1 -- start start 2024-03-12T05:56:14.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f862800b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cb380 0x5560497cb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f862800b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497cbd30 0x5560497d42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f862800b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497d48b0 0x5560497d6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f862800b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556049739f90 con 0x5560497cbd30 2024-03-12T05:56:14.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f862800b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560497374c0 con 0x5560497d48b0 2024-03-12T05:56:14.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f862800b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556049732ca0 con 0x5560497cb380 2024-03-12T05:56:14.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f861effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497cbd30 0x5560497d42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f861effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497cbd30 0x5560497d42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57510/0 (socket says 172.21.15.76:57510) 2024-03-12T05:56:14.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f861effd640 1 -- 172.21.15.76:0/662729557 learned_addr learned my addr 172.21.15.76:0/662729557 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:14.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f8607fff640 1 --2- 172.21.15.76:0/3429517277 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f86080a1cc0 0x7f86080a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f861e7fc640 1 --2- 172.21.15.76:0/662729557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cb380 0x5560497cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f861f7fe640 1 --2- 172.21.15.76:0/3429517277 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f86080a1cc0 0x7f86080a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f861effd640 1 -- 172.21.15.76:0/662729557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cb380 msgr2=0x5560497cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.113+0000 7f861effd640 1 --2- 172.21.15.76:0/662729557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cb380 0x5560497cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.114+0000 7f861effd640 1 -- 172.21.15.76:0/662729557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497d48b0 msgr2=0x5560497d6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.114+0000 7f861e7fc640 1 --2- 172.21.15.76:0/662729557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497d48b0 0x5560497d6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.114+0000 7f861effd640 1 --2- 172.21.15.76:0/662729557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497d48b0 0x5560497d6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.114+0000 7f861effd640 1 -- 172.21.15.76:0/662729557 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560496f50b0 con 0x5560497cbd30 2024-03-12T05:56:14.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.114+0000 7f861e7fc640 1 --2- 172.21.15.76:0/662729557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497d48b0 0x5560497d6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:56:14.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.114+0000 7f8607fff640 1 -- 172.21.15.76:0/3429517277 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f86180cd610 con 0x7f8620199150 2024-03-12T05:56:14.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.114+0000 7f861effd640 1 --2- 172.21.15.76:0/662729557 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497cbd30 0x5560497d42e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8610036040 tx=0x7f8610004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc023399f90d868e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:14.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.114+0000 7f861f7fe640 1 --2- 172.21.15.76:0/3429517277 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f86080a1cc0 0x7f86080a4180 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f86180cac70 tx=0x7f861807b050 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:14.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.114+0000 7f86057fa640 1 -- 172.21.15.76:0/662729557 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86100049a0 con 0x5560497cbd30 2024-03-12T05:56:14.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.114+0000 7f86057fa640 1 -- 172.21.15.76:0/662729557 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8610004b00 con 0x5560497cbd30 2024-03-12T05:56:14.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.114+0000 7f86057fa640 1 -- 172.21.15.76:0/662729557 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8610005210 con 0x5560497cbd30 2024-03-12T05:56:14.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 -- 172.21.15.76:0/662729557 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497cbd30 msgr2=0x5560497d42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 --2- 172.21.15.76:0/662729557 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497cbd30 0x5560497d42e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8610036040 tx=0x7f8610004770 comp rx=0 tx=0).stop 2024-03-12T05:56:14.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 -- 172.21.15.76:0/662729557 shutdown_connections 2024-03-12T05:56:14.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 --2- 172.21.15.76:0/662729557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497d48b0 0x5560497d6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 --2- 172.21.15.76:0/662729557 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497cbd30 0x5560497d42e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 --2- 172.21.15.76:0/662729557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cb380 0x5560497cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 -- 172.21.15.76:0/662729557 >> 172.21.15.76:0/662729557 conn(0x5560497c5840 msgr2=0x5560497c5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:14.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 -- 172.21.15.76:0/662729557 shutdown_connections 2024-03-12T05:56:14.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 -- 172.21.15.76:0/662729557 wait complete. 2024-03-12T05:56:14.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 -- start start 2024-03-12T05:56:14.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 Processor -- start 2024-03-12T05:56:14.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cb380 0x5560497d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497cbd30 0x5560497d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497d48b0 0x5560497d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560496d7f40 con 0x5560497d48b0 2024-03-12T05:56:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560494064b0 con 0x5560497cbd30 2024-03-12T05:56:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560496b8670 con 0x5560497cb380 2024-03-12T05:56:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.115+0000 7f862800b740 10 client.0 _ll_register_callbacks cb 0x556049725f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f861e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497d48b0 0x5560497d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f861e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497d48b0 0x5560497d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57520/0 (socket says 172.21.15.76:57520) 2024-03-12T05:56:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f861e7fc640 1 -- 172.21.15.76:0/110555470 learned_addr learned my addr 172.21.15.76:0/110555470 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f861e7fc640 1 --2- 172.21.15.76:0/110555470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cb380 0x5560497d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f861effd640 1 --2- 172.21.15.76:0/110555470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497cbd30 0x5560497d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f861e7fc640 1 -- 172.21.15.76:0/110555470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cb380 msgr2=0x5560497d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f861e7fc640 1 --2- 172.21.15.76:0/110555470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cb380 0x5560497d1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f861e7fc640 1 -- 172.21.15.76:0/110555470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497cbd30 msgr2=0x5560497d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f861e7fc640 1 --2- 172.21.15.76:0/110555470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497cbd30 0x5560497d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f861e7fc640 1 -- 172.21.15.76:0/110555470 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f862010d740 con 0x5560497d48b0 2024-03-12T05:56:14.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f861e7fc640 1 --2- 172.21.15.76:0/110555470 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497d48b0 0x5560497d22b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f8614011f00 tx=0x7f86140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6dce8a066e92f40 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:14.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f8602ff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:14.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f8602ff5640 1 -- 172.21.15.76:0/110555470 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8614016040 con 0x5560497d48b0 2024-03-12T05:56:14.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f8602ff5640 1 -- 172.21.15.76:0/110555470 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8614004220 con 0x5560497d48b0 2024-03-12T05:56:14.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f862800b740 1 -- 172.21.15.76:0/110555470 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55604970cf40 con 0x5560497d48b0 2024-03-12T05:56:14.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f8602ff5640 1 -- 172.21.15.76:0/110555470 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86140043e0 con 0x5560497d48b0 2024-03-12T05:56:14.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f862800b740 10 client.15738 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:14.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.116+0000 7f862800b740 1 -- 172.21.15.76:0/110555470 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x556049731320 con 0x5560497d48b0 2024-03-12T05:56:14.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.117+0000 7f8602ff5640 1 -- 172.21.15.76:0/110555470 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55604970cf40 con 0x5560497d48b0 2024-03-12T05:56:14.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.117+0000 7f8602ff5640 1 -- 172.21.15.76:0/110555470 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f861401c040 con 0x5560497d48b0 2024-03-12T05:56:14.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.117+0000 7f862800b740 10 client.15738 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:14.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.117+0000 7f862800b740 10 client.15738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:14.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.117+0000 7f862800b740 20 client.15738 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:14.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.117+0000 7f862800b740 1 --2- 172.21.15.76:0/110555470 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5560497e4d80 0x5560497e71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.117+0000 7f862800b740 4 client.15738 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:14.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.117+0000 7f862800b740 1 -- 172.21.15.76:0/110555470 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556049729310 con 0x5560497e4d80 2024-03-12T05:56:14.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.117+0000 7f861effd640 1 --2- 172.21.15.76:0/110555470 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5560497e4d80 0x5560497e71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f8602ff5640 10 client.15738 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:14.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f861effd640 1 --2- 172.21.15.76:0/110555470 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5560497e4d80 0x5560497e71a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c8d58a8753c35ec server_cookie=77d85373d73e3eb5 in_seq=0 out_seq=0 2024-03-12T05:56:14.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f8602ff5640 1 -- 172.21.15.76:0/110555470 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556049729310 con 0x5560497e4d80 2024-03-12T05:56:14.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f8602ff5640 10 client.15738 handle_command_reply: tid=0 2024-03-12T05:56:14.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f862800b740 1 client.15738 shutdown 2024-03-12T05:56:14.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f862800b740 1 -- 172.21.15.76:0/110555470 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497d48b0 msgr2=0x5560497d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f862800b740 1 --2- 172.21.15.76:0/110555470 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497d48b0 0x5560497d22b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f8614011f00 tx=0x7f86140046c0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f862800b740 1 -- 172.21.15.76:0/110555470 shutdown_connections 2024-03-12T05:56:14.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f862800b740 1 --2- 172.21.15.76:0/110555470 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5560497e4d80 0x5560497e71a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f862800b740 1 --2- 172.21.15.76:0/110555470 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497d48b0 0x5560497d22b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f862800b740 1 --2- 172.21.15.76:0/110555470 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497cbd30 0x5560497d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f8602ff5640 0 client.15738 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:14.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f862800b740 1 --2- 172.21.15.76:0/110555470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cb380 0x5560497d1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f862800b740 1 -- 172.21.15.76:0/110555470 >> 172.21.15.76:0/110555470 conn(0x5560497c5840 msgr2=0x5560497c77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:14.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f862800b740 1 -- 172.21.15.76:0/110555470 shutdown_connections 2024-03-12T05:56:14.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f862800b740 1 -- 172.21.15.76:0/110555470 wait complete. 2024-03-12T05:56:14.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.118+0000 7f862800b740 20 client.15738 trim_cache size 0 max 16384 2024-03-12T05:56:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.135+0000 7f862800b740 1 Processor -- start 2024-03-12T05:56:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.135+0000 7f862800b740 1 -- start start 2024-03-12T05:56:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.135+0000 7f862800b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497cb380 0x5560497d3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.135+0000 7f862800b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cbd30 0x5560497c4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.135+0000 7f862800b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497d48b0 0x5560497db880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.135+0000 7f862800b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560497c56d0 con 0x5560497cb380 2024-03-12T05:56:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.135+0000 7f862800b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560497c6640 con 0x5560497d48b0 2024-03-12T05:56:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.135+0000 7f862800b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560497cacd0 con 0x5560497cbd30 2024-03-12T05:56:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.136+0000 7f861e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497cb380 0x5560497d3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.136+0000 7f861e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497cb380 0x5560497d3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57530/0 (socket says 172.21.15.76:57530) 2024-03-12T05:56:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.136+0000 7f861e7fc640 1 -- 172.21.15.76:0/1518065091 learned_addr learned my addr 172.21.15.76:0/1518065091 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.136+0000 7f861effd640 1 --2- 172.21.15.76:0/1518065091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cbd30 0x5560497c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.136+0000 7f861e7fc640 1 --2- 172.21.15.76:0/1518065091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497d48b0 0x5560497db880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.136+0000 7f861e7fc640 1 -- 172.21.15.76:0/1518065091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cbd30 msgr2=0x5560497c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.136+0000 7f861e7fc640 1 --2- 172.21.15.76:0/1518065091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cbd30 0x5560497c4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.136+0000 7f861e7fc640 1 -- 172.21.15.76:0/1518065091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497d48b0 msgr2=0x5560497db880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.136+0000 7f861e7fc640 1 --2- 172.21.15.76:0/1518065091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497d48b0 0x5560497db880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.136+0000 7f861e7fc640 1 -- 172.21.15.76:0/1518065091 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560496cdc00 con 0x5560497cb380 2024-03-12T05:56:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.136+0000 7f861e7fc640 1 --2- 172.21.15.76:0/1518065091 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497cb380 0x5560497d3f60 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f861400a610 tx=0x7f861400d550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca4a8daea0442b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.136+0000 7f8604ff9640 1 -- 172.21.15.76:0/1518065091 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8614016070 con 0x5560497cb380 2024-03-12T05:56:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.136+0000 7f8604ff9640 1 -- 172.21.15.76:0/1518065091 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f861400fea0 con 0x5560497cb380 2024-03-12T05:56:14.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.136+0000 7f8604ff9640 1 -- 172.21.15.76:0/1518065091 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8614032ba0 con 0x5560497cb380 2024-03-12T05:56:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.137+0000 7f862800b740 1 -- 172.21.15.76:0/1518065091 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497cb380 msgr2=0x5560497d3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.137+0000 7f862800b740 1 --2- 172.21.15.76:0/1518065091 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497cb380 0x5560497d3f60 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f861400a610 tx=0x7f861400d550 comp rx=0 tx=0).stop 2024-03-12T05:56:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.137+0000 7f862800b740 1 -- 172.21.15.76:0/1518065091 shutdown_connections 2024-03-12T05:56:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.137+0000 7f862800b740 1 --2- 172.21.15.76:0/1518065091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497d48b0 0x5560497db880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.137+0000 7f862800b740 1 --2- 172.21.15.76:0/1518065091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cbd30 0x5560497c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.137+0000 7f862800b740 1 --2- 172.21.15.76:0/1518065091 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497cb380 0x5560497d3f60 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.137+0000 7f862800b740 1 -- 172.21.15.76:0/1518065091 >> 172.21.15.76:0/1518065091 conn(0x5560497c5840 msgr2=0x5560497c4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.137+0000 7f862800b740 1 -- 172.21.15.76:0/1518065091 shutdown_connections 2024-03-12T05:56:14.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.137+0000 7f862800b740 1 -- 172.21.15.76:0/1518065091 wait complete. 2024-03-12T05:56:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.137+0000 7f862800b740 1 -- start start 2024-03-12T05:56:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.137+0000 7f862800b740 1 Processor -- start 2024-03-12T05:56:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f862800b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497cb380 0x556049835ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f862800b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cbd30 0x5560498361e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f862800b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497d48b0 0x5560497db190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f862800b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560497db750 con 0x5560497d48b0 2024-03-12T05:56:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f862800b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560497db8c0 con 0x5560497cb380 2024-03-12T05:56:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f862800b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560497dba80 con 0x5560497cbd30 2024-03-12T05:56:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f862800b740 10 client.0 _ll_register_callbacks cb 0x7f861002c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f861e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497d48b0 0x5560497db190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f861e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497d48b0 0x5560497db190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57532/0 (socket says 172.21.15.76:57532) 2024-03-12T05:56:14.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f861e7fc640 1 -- 172.21.15.76:0/1951740907 learned_addr learned my addr 172.21.15.76:0/1951740907 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f861effd640 1 --2- 172.21.15.76:0/1951740907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cbd30 0x5560498361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f861e7fc640 1 --2- 172.21.15.76:0/1951740907 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497cb380 0x556049835ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f861e7fc640 1 -- 172.21.15.76:0/1951740907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cbd30 msgr2=0x5560498361e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f861e7fc640 1 --2- 172.21.15.76:0/1951740907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cbd30 0x5560498361e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f861e7fc640 1 -- 172.21.15.76:0/1951740907 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497cb380 msgr2=0x556049835ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f861e7fc640 1 --2- 172.21.15.76:0/1951740907 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497cb380 0x556049835ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f861e7fc640 1 -- 172.21.15.76:0/1951740907 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556049731320 con 0x5560497d48b0 2024-03-12T05:56:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f861e7fc640 1 --2- 172.21.15.76:0/1951740907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497d48b0 0x5560497db190 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8614001eb0 tx=0x7f861402a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709c3e4401088769 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f8602ff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f8602ff5640 1 -- 172.21.15.76:0/1951740907 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8614016030 con 0x5560497d48b0 2024-03-12T05:56:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f8602ff5640 1 -- 172.21.15.76:0/1951740907 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8614000e40 con 0x5560497d48b0 2024-03-12T05:56:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f862800b740 1 -- 172.21.15.76:0/1951740907 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5560497d1660 con 0x5560497d48b0 2024-03-12T05:56:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f8602ff5640 1 -- 172.21.15.76:0/1951740907 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8614005270 con 0x5560497d48b0 2024-03-12T05:56:14.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f862800b740 10 client.15744 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.138+0000 7f862800b740 1 -- 172.21.15.76:0/1951740907 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5560496f3910 con 0x5560497d48b0 2024-03-12T05:56:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.139+0000 7f8602ff5640 1 -- 172.21.15.76:0/1951740907 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5560497d1660 con 0x5560497d48b0 2024-03-12T05:56:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.139+0000 7f8602ff5640 1 -- 172.21.15.76:0/1951740907 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f861401c040 con 0x5560497d48b0 2024-03-12T05:56:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.139+0000 7f862800b740 10 client.15744 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.139+0000 7f862800b740 10 client.15744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.139+0000 7f862800b740 20 client.15744 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.139+0000 7f862800b740 1 --2- 172.21.15.76:0/1951740907 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55604983d900 0x55604983fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.139+0000 7f862800b740 4 client.15744 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:56:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.139+0000 7f862800b740 1 -- 172.21.15.76:0/1951740907 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x556049758900 con 0x55604983d900 2024-03-12T05:56:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.139+0000 7f861effd640 1 --2- 172.21.15.76:0/1951740907 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55604983d900 0x55604983fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.140+0000 7f8602ff5640 10 client.15744 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.140+0000 7f861effd640 1 --2- 172.21.15.76:0/1951740907 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55604983d900 0x55604983fd20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aebe309cd5dd24d1 server_cookie=62b6152635763c7b in_seq=0 out_seq=0 2024-03-12T05:56:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f8602ff5640 1 -- 172.21.15.76:0/1951740907 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556049758900 con 0x55604983d900 2024-03-12T05:56:14.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f8602ff5640 10 client.15744 handle_command_reply: tid=0 2024-03-12T05:56:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f862800b740 1 client.15744 shutdown 2024-03-12T05:56:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f862800b740 1 -- 172.21.15.76:0/1951740907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497d48b0 msgr2=0x5560497db190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f862800b740 1 --2- 172.21.15.76:0/1951740907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497d48b0 0x5560497db190 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8614001eb0 tx=0x7f861402a040 comp rx=0 tx=0).stop 2024-03-12T05:56:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f862800b740 1 -- 172.21.15.76:0/1951740907 shutdown_connections 2024-03-12T05:56:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f862800b740 1 --2- 172.21.15.76:0/1951740907 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55604983d900 0x55604983fd20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f862800b740 1 --2- 172.21.15.76:0/1951740907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560497d48b0 0x5560497db190 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f862800b740 1 --2- 172.21.15.76:0/1951740907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560497cbd30 0x5560498361e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f862800b740 1 --2- 172.21.15.76:0/1951740907 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560497cb380 0x556049835ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f8602ff5640 0 client.15744 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f862800b740 1 -- 172.21.15.76:0/1951740907 >> 172.21.15.76:0/1951740907 conn(0x5560497c5840 msgr2=0x5560497c48e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f862800b740 1 -- 172.21.15.76:0/1951740907 shutdown_connections 2024-03-12T05:56:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f862800b740 1 -- 172.21.15.76:0/1951740907 wait complete. 2024-03-12T05:56:14.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f862800b740 20 client.15744 trim_cache size 0 max 16384 2024-03-12T05:56:14.152 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:56:14.152 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T05:56:14.152 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "7ce9fb14-f96a-4173-861b-ded3bde7b133", 2024-03-12T05:56:14.152 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T05:56:14.152 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:56:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f8625363640 1 -- 172.21.15.76:0/3429517277 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f86080a1cc0 msgr2=0x7f86080a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f8625363640 1 --2- 172.21.15.76:0/3429517277 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f86080a1cc0 0x7f86080a4180 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f86180cac70 tx=0x7f861807b050 comp rx=0 tx=0).stop 2024-03-12T05:56:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f8625363640 1 -- 172.21.15.76:0/3429517277 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8620199150 msgr2=0x7f86201a3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.141+0000 7f8625363640 1 --2- 172.21.15.76:0/3429517277 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8620199150 0x7f86201a3700 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f861800da40 tx=0x7f861800df00 comp rx=0 tx=0).stop 2024-03-12T05:56:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.142+0000 7f8625363640 1 -- 172.21.15.76:0/3429517277 shutdown_connections 2024-03-12T05:56:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.142+0000 7f8625363640 1 --2- 172.21.15.76:0/3429517277 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f86080a1cc0 0x7f86080a4180 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.142+0000 7f8625363640 1 --2- 172.21.15.76:0/3429517277 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8620199150 0x7f86201a3700 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.142+0000 7f8625363640 1 --2- 172.21.15.76:0/3429517277 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8620105360 0x7f8620198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.142+0000 7f8625363640 1 --2- 172.21.15.76:0/3429517277 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8620100290 0x7f8620198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.142+0000 7f8625363640 1 -- 172.21.15.76:0/3429517277 >> 172.21.15.76:0/3429517277 conn(0x7f8620077230 msgr2=0x7f86200fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.142+0000 7f8625363640 1 -- 172.21.15.76:0/3429517277 shutdown_connections 2024-03-12T05:56:14.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.142+0000 7f8625363640 1 -- 172.21.15.76:0/3429517277 wait complete. 2024-03-12T05:56:14.157 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:56:14.157 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:56:14.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.258+0000 7f78d120b640 1 -- 172.21.15.76:0/3012519369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78cc100c10 msgr2=0x7f78cc104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.258+0000 7f78d120b640 1 --2- 172.21.15.76:0/3012519369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78cc100c10 0x7f78cc104c50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009930 tx=0x7f78b402bc30 comp rx=0 tx=0).stop 2024-03-12T05:56:14.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.258+0000 7f78d120b640 1 -- 172.21.15.76:0/3012519369 shutdown_connections 2024-03-12T05:56:14.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.258+0000 7f78d120b640 1 --2- 172.21.15.76:0/3012519369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78cc105330 0x7f78cc10cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.258+0000 7f78d120b640 1 --2- 172.21.15.76:0/3012519369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78cc100c10 0x7f78cc104c50 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.258+0000 7f78d120b640 1 --2- 172.21.15.76:0/3012519369 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78cc100260 0x7f78cc100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.258+0000 7f78d120b640 1 -- 172.21.15.76:0/3012519369 >> 172.21.15.76:0/3012519369 conn(0x7f78cc077180 msgr2=0x7f78cc077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:14.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.258+0000 7f78d120b640 1 -- 172.21.15.76:0/3012519369 shutdown_connections 2024-03-12T05:56:14.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.258+0000 7f78d120b640 1 -- 172.21.15.76:0/3012519369 wait complete. 2024-03-12T05:56:14.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.259+0000 7f78d120b640 1 Processor -- start 2024-03-12T05:56:14.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.259+0000 7f78d120b640 1 -- start start 2024-03-12T05:56:14.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.259+0000 7f78d120b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78cc100260 0x7f78cc19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.259+0000 7f78d120b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78cc105330 0x7f78cc19d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.259+0000 7f78d120b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78cc1975e0 0x7f78cc197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.259+0000 7f78d120b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc197fd0 con 0x7f78cc105330 2024-03-12T05:56:14.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.259+0000 7f78d120b640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc198140 con 0x7f78cc1975e0 2024-03-12T05:56:14.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.259+0000 7f78d120b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78cc198300 con 0x7f78cc100260 2024-03-12T05:56:14.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.259+0000 7f78c3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78cc105330 0x7f78cc19d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.259+0000 7f78c3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78cc105330 0x7f78cc19d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57552/0 (socket says 172.21.15.76:57552) 2024-03-12T05:56:14.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.259+0000 7f78c3fff640 1 -- 172.21.15.76:0/258481115 learned_addr learned my addr 172.21.15.76:0/258481115 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:14.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.260+0000 7f78cad9d640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78cc100260 0x7f78cc19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.260+0000 7f78c3fff640 1 -- 172.21.15.76:0/258481115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78cc100260 msgr2=0x7f78cc19d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.260+0000 7f78c3fff640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78cc100260 0x7f78cc19d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.260+0000 7f78cb59e640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78cc1975e0 0x7f78cc197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.260+0000 7f78c3fff640 1 -- 172.21.15.76:0/258481115 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78cc1975e0 msgr2=0x7f78cc197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.260+0000 7f78c3fff640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78cc1975e0 0x7f78cc197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.260+0000 7f78c3fff640 1 -- 172.21.15.76:0/258481115 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b4009560 con 0x7f78cc105330 2024-03-12T05:56:14.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.260+0000 7f78cb59e640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78cc1975e0 0x7f78cc197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:56:14.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.260+0000 7f78c3fff640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78cc105330 0x7f78cc19d970 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009230 tx=0x7f78b4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:14.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.260+0000 7f78c8d99640 1 -- 172.21.15.76:0/258481115 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b402cbc0 con 0x7f78cc105330 2024-03-12T05:56:14.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.260+0000 7f78c8d99640 1 -- 172.21.15.76:0/258481115 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78b40052d0 con 0x7f78cc105330 2024-03-12T05:56:14.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.260+0000 7f78c8d99640 1 -- 172.21.15.76:0/258481115 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4003e30 con 0x7f78cc105330 2024-03-12T05:56:14.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.260+0000 7f78d120b640 1 -- 172.21.15.76:0/258481115 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78cc1a42f0 con 0x7f78cc105330 2024-03-12T05:56:14.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.260+0000 7f78d120b640 1 -- 172.21.15.76:0/258481115 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78cc1a47a0 con 0x7f78cc105330 2024-03-12T05:56:14.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.263+0000 7f78c8d99640 1 -- 172.21.15.76:0/258481115 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f78b4005440 con 0x7f78cc105330 2024-03-12T05:56:14.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.263+0000 7f78d3dd7740 1 Processor -- start 2024-03-12T05:56:14.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.263+0000 7f78d3dd7740 1 -- start start 2024-03-12T05:56:14.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.263+0000 7f78d3dd7740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 0x558ae3c35690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.263+0000 7f78d3dd7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c35bd0 0x558ae3c3e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.263+0000 7f78d3dd7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 0x558ae3c40ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.263+0000 7f78d3dd7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae3ba4280 con 0x558ae3c35bd0 2024-03-12T05:56:14.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.263+0000 7f78d3dd7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae3ba17b0 con 0x558ae3c3e7b0 2024-03-12T05:56:14.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.263+0000 7f78d3dd7740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae3b9cf90 con 0x558ae3c352b0 2024-03-12T05:56:14.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.263+0000 7f78cad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c35bd0 0x558ae3c3e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.263+0000 7f78c8d99640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f78b80a1cc0 0x7f78b80a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.263+0000 7f78cad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c35bd0 0x558ae3c3e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57568/0 (socket says 172.21.15.76:57568) 2024-03-12T05:56:14.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.263+0000 7f78cad9d640 1 -- 172.21.15.76:0/2883840941 learned_addr learned my addr 172.21.15.76:0/2883840941 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78c3fff640 1 --2- 172.21.15.76:0/2883840941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 0x558ae3c40ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78cb59e640 1 --2- 172.21.15.76:0/2883840941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 0x558ae3c35690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78cb59e640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f78b80a1cc0 0x7f78b80a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78cad9d640 1 -- 172.21.15.76:0/2883840941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 msgr2=0x558ae3c35690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78cad9d640 1 --2- 172.21.15.76:0/2883840941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 0x558ae3c35690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78cad9d640 1 -- 172.21.15.76:0/2883840941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 msgr2=0x558ae3c40ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78cad9d640 1 --2- 172.21.15.76:0/2883840941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 0x558ae3c40ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78c8d99640 1 -- 172.21.15.76:0/258481115 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f78b40efe50 con 0x7f78cc105330 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78cad9d640 1 -- 172.21.15.76:0/2883840941 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ae3b5b570 con 0x558ae3c35bd0 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78c3fff640 1 --2- 172.21.15.76:0/2883840941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 0x558ae3c40ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78cad9d640 1 --2- 172.21.15.76:0/2883840941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c35bd0 0x558ae3c3e270 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f78bc00bcb0 tx=0x7f78bc00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0465cef35e89036 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78c17fa640 1 -- 172.21.15.76:0/2883840941 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc01a460 con 0x558ae3c35bd0 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78cb59e640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f78b80a1cc0 0x7f78b80a4180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f78c4005c40 tx=0x7f78c4009200 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78c17fa640 1 -- 172.21.15.76:0/2883840941 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78bc01aaa0 con 0x558ae3c35bd0 2024-03-12T05:56:14.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.264+0000 7f78c17fa640 1 -- 172.21.15.76:0/2883840941 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc011b50 con 0x558ae3c35bd0 2024-03-12T05:56:14.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.265+0000 7f78d3dd7740 1 -- 172.21.15.76:0/2883840941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c35bd0 msgr2=0x558ae3c3e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.265+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/2883840941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c35bd0 0x558ae3c3e270 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f78bc00bcb0 tx=0x7f78bc00f730 comp rx=0 tx=0).stop 2024-03-12T05:56:14.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.265+0000 7f78d3dd7740 1 -- 172.21.15.76:0/2883840941 shutdown_connections 2024-03-12T05:56:14.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.265+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/2883840941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 0x558ae3c40ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.265+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/2883840941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c35bd0 0x558ae3c3e270 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.265+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/2883840941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 0x558ae3c35690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.265+0000 7f78d3dd7740 1 -- 172.21.15.76:0/2883840941 >> 172.21.15.76:0/2883840941 conn(0x558ae3c2f990 msgr2=0x558ae3c2fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:14.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.265+0000 7f78d3dd7740 1 -- 172.21.15.76:0/2883840941 shutdown_connections 2024-03-12T05:56:14.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.265+0000 7f78d3dd7740 1 -- 172.21.15.76:0/2883840941 wait complete. 2024-03-12T05:56:14.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.265+0000 7f78d3dd7740 1 -- start start 2024-03-12T05:56:14.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.265+0000 7f78d3dd7740 1 Processor -- start 2024-03-12T05:56:14.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78d3dd7740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 0x558ae3c3e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78d3dd7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c35bd0 0x558ae3c3b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78d3dd7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c3e7b0 0x558ae3c3bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78d3dd7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae3870430 con 0x558ae3c3e7b0 2024-03-12T05:56:14.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78d3dd7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae382ce00 con 0x558ae3c35bd0 2024-03-12T05:56:14.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78d3dd7740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae3b21fd0 con 0x558ae3c352b0 2024-03-12T05:56:14.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78d3dd7740 10 client.0 _ll_register_callbacks cb 0x558ae3b77fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:14.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78cb59e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c3e7b0 0x558ae3c3bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78cad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 0x558ae3c3e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78cb59e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c3e7b0 0x558ae3c3bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57570/0 (socket says 172.21.15.76:57570) 2024-03-12T05:56:14.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78cb59e640 1 -- 172.21.15.76:0/78012636 learned_addr learned my addr 172.21.15.76:0/78012636 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:14.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78c3fff640 1 --2- 172.21.15.76:0/78012636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c35bd0 0x558ae3c3b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78cb59e640 1 -- 172.21.15.76:0/78012636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 msgr2=0x558ae3c3e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78cb59e640 1 --2- 172.21.15.76:0/78012636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 0x558ae3c3e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78cb59e640 1 -- 172.21.15.76:0/78012636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c35bd0 msgr2=0x558ae3c3b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78cb59e640 1 --2- 172.21.15.76:0/78012636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c35bd0 0x558ae3c3b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78cb59e640 1 -- 172.21.15.76:0/78012636 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78bc00b8e0 con 0x558ae3c3e7b0 2024-03-12T05:56:14.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78cad9d640 1 --2- 172.21.15.76:0/78012636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 0x558ae3c3e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:14.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78c3fff640 1 --2- 172.21.15.76:0/78012636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c35bd0 0x558ae3c3b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78cb59e640 1 --2- 172.21.15.76:0/78012636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c3e7b0 0x558ae3c3bcd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f78c400f980 tx=0x7f78c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a911ac50eb1cbb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.266+0000 7f78b27fc640 1 -- 172.21.15.76:0/78012636 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c4004d20 con 0x558ae3c3e7b0 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.267+0000 7f78b27fc640 1 -- 172.21.15.76:0/78012636 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f78c4011970 con 0x558ae3c3e7b0 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.267+0000 7f78d3dd7740 1 -- 172.21.15.76:0/78012636 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558ae389c7e0 con 0x558ae3c3e7b0 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.267+0000 7f78b27fc640 1 -- 172.21.15.76:0/78012636 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78c4004e80 con 0x558ae3c3e7b0 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.267+0000 7f78d3dd7740 10 client.15756 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.267+0000 7f78d3dd7740 1 -- 172.21.15.76:0/78012636 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x558ae3b9b610 con 0x558ae3c3e7b0 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.267+0000 7f78b27fc640 1 -- 172.21.15.76:0/78012636 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558ae389c7e0 con 0x558ae3c3e7b0 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.267+0000 7f78b27fc640 1 -- 172.21.15.76:0/78012636 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f78c401f400 con 0x558ae3c3e7b0 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.267+0000 7f78d3dd7740 10 client.15756 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.267+0000 7f78d3dd7740 10 client.15756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.267+0000 7f78d3dd7740 20 client.15756 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.267+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/78012636 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558ae3c4e9b0 0x558ae3c50dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.267+0000 7f78d3dd7740 4 client.15756 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.267+0000 7f78d3dd7740 1 -- 172.21.15.76:0/78012636 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558ae38d9720 con 0x558ae3c4e9b0 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.267+0000 7f78cad9d640 1 --2- 172.21.15.76:0/78012636 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558ae3c4e9b0 0x558ae3c50dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.268+0000 7f78b27fc640 10 client.15756 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.268+0000 7f78cad9d640 1 --2- 172.21.15.76:0/78012636 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558ae3c4e9b0 0x558ae3c50dd0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad72f239f53071df server_cookie=dd18dacaa12892f6 in_seq=0 out_seq=0 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.268+0000 7f78b27fc640 1 -- 172.21.15.76:0/78012636 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558ae38d9720 con 0x558ae3c4e9b0 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.268+0000 7f78b27fc640 10 client.15756 handle_command_reply: tid=0 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.269+0000 7f78d3dd7740 1 client.15756 shutdown 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.269+0000 7f78d3dd7740 1 -- 172.21.15.76:0/78012636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c3e7b0 msgr2=0x558ae3c3bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.269+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/78012636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c3e7b0 0x558ae3c3bcd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f78c400f980 tx=0x7f78c4004a40 comp rx=0 tx=0).stop 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.269+0000 7f78d3dd7740 1 -- 172.21.15.76:0/78012636 shutdown_connections 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.269+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/78012636 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558ae3c4e9b0 0x558ae3c50dd0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.269+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/78012636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c3e7b0 0x558ae3c3bcd0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.269+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/78012636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c35bd0 0x558ae3c3b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.269+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/78012636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 0x558ae3c3e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.269+0000 7f78d3dd7740 1 -- 172.21.15.76:0/78012636 >> 172.21.15.76:0/78012636 conn(0x558ae3c2f990 msgr2=0x558ae3c31730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:14.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.269+0000 7f78b27fc640 0 client.15756 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:14.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.269+0000 7f78d3dd7740 1 -- 172.21.15.76:0/78012636 shutdown_connections 2024-03-12T05:56:14.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.269+0000 7f78d3dd7740 1 -- 172.21.15.76:0/78012636 wait complete. 2024-03-12T05:56:14.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.269+0000 7f78d3dd7740 20 client.15756 trim_cache size 0 max 16384 2024-03-12T05:56:14.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.284+0000 7f78d3dd7740 1 Processor -- start 2024-03-12T05:56:14.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.284+0000 7f78d3dd7740 1 -- start start 2024-03-12T05:56:14.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.284+0000 7f78d3dd7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c352b0 0x558ae3c55750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.284+0000 7f78d3dd7740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c35bd0 0x558ae3c45660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78d3dd7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 0x558ae3c45ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78d3dd7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae3c34f40 con 0x558ae3c352b0 2024-03-12T05:56:14.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78d3dd7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae3c30c10 con 0x558ae3c3e7b0 2024-03-12T05:56:14.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78d3dd7740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae3c309e0 con 0x558ae3c35bd0 2024-03-12T05:56:14.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78cad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c352b0 0x558ae3c55750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78cad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c352b0 0x558ae3c55750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57578/0 (socket says 172.21.15.76:57578) 2024-03-12T05:56:14.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78cad9d640 1 -- 172.21.15.76:0/867606215 learned_addr learned my addr 172.21.15.76:0/867606215 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:14.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78cb59e640 1 --2- 172.21.15.76:0/867606215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 0x558ae3c45ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78cad9d640 1 -- 172.21.15.76:0/867606215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c35bd0 msgr2=0x558ae3c45660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78c3fff640 1 --2- 172.21.15.76:0/867606215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c35bd0 0x558ae3c45660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78cad9d640 1 --2- 172.21.15.76:0/867606215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c35bd0 0x558ae3c45660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78cad9d640 1 -- 172.21.15.76:0/867606215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 msgr2=0x558ae3c45ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78cad9d640 1 --2- 172.21.15.76:0/867606215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 0x558ae3c45ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78cad9d640 1 -- 172.21.15.76:0/867606215 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ae3b37ac0 con 0x558ae3c352b0 2024-03-12T05:56:14.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78cb59e640 1 --2- 172.21.15.76:0/867606215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 0x558ae3c45ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:14.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78cad9d640 1 --2- 172.21.15.76:0/867606215 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c352b0 0x558ae3c55750 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f78bc011760 tx=0x7f78bc011790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff847b4027617a61 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:14.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78c0ff9640 1 -- 172.21.15.76:0/867606215 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc02c070 con 0x558ae3c352b0 2024-03-12T05:56:14.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78c0ff9640 1 -- 172.21.15.76:0/867606215 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78bc00bea0 con 0x558ae3c352b0 2024-03-12T05:56:14.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.285+0000 7f78c0ff9640 1 -- 172.21.15.76:0/867606215 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc01a850 con 0x558ae3c352b0 2024-03-12T05:56:14.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.286+0000 7f78d3dd7740 1 -- 172.21.15.76:0/867606215 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c352b0 msgr2=0x558ae3c55750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.286+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/867606215 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c352b0 0x558ae3c55750 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f78bc011760 tx=0x7f78bc011790 comp rx=0 tx=0).stop 2024-03-12T05:56:14.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.286+0000 7f78d3dd7740 1 -- 172.21.15.76:0/867606215 shutdown_connections 2024-03-12T05:56:14.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.286+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/867606215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 0x558ae3c45ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.286+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/867606215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c35bd0 0x558ae3c45660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.286+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/867606215 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c352b0 0x558ae3c55750 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.286+0000 7f78d3dd7740 1 -- 172.21.15.76:0/867606215 >> 172.21.15.76:0/867606215 conn(0x558ae3c2f990 msgr2=0x558ae3c36910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.286+0000 7f78d3dd7740 1 -- 172.21.15.76:0/867606215 shutdown_connections 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.286+0000 7f78d3dd7740 1 -- 172.21.15.76:0/867606215 wait complete. 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.286+0000 7f78d3dd7740 1 -- start start 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78d3dd7740 1 Processor -- start 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78d3dd7740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 0x558ae3c437e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78d3dd7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c35bd0 0x558ae3c43d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78d3dd7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 0x558ae3ca3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78d3dd7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae3c444c0 con 0x558ae3c35bd0 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78d3dd7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae3ca4440 con 0x558ae3c3e7b0 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78d3dd7740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae3ca4600 con 0x558ae3c352b0 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78d3dd7740 10 client.0 _ll_register_callbacks cb 0x7f78bc00b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78c3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c35bd0 0x558ae3c43d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78c3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c35bd0 0x558ae3c43d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57584/0 (socket says 172.21.15.76:57584) 2024-03-12T05:56:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78c3fff640 1 -- 172.21.15.76:0/2388598022 learned_addr learned my addr 172.21.15.76:0/2388598022 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78cad9d640 1 --2- 172.21.15.76:0/2388598022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 0x558ae3c437e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78cb59e640 1 --2- 172.21.15.76:0/2388598022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 0x558ae3ca3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78c3fff640 1 -- 172.21.15.76:0/2388598022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 msgr2=0x558ae3c437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78c3fff640 1 --2- 172.21.15.76:0/2388598022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 0x558ae3c437e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78c3fff640 1 -- 172.21.15.76:0/2388598022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 msgr2=0x558ae3ca3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78c3fff640 1 --2- 172.21.15.76:0/2388598022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 0x558ae3ca3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78c3fff640 1 -- 172.21.15.76:0/2388598022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558ae3b5b570 con 0x558ae3c35bd0 2024-03-12T05:56:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78cad9d640 1 --2- 172.21.15.76:0/2388598022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 0x558ae3c437e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78c3fff640 1 --2- 172.21.15.76:0/2388598022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c35bd0 0x558ae3c43d20 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f78b40edbd0 tx=0x7f78b408bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508d5f42209da84d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78b27fc640 1 -- 172.21.15.76:0/2388598022 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40c4e80 con 0x558ae3c35bd0 2024-03-12T05:56:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78b27fc640 1 -- 172.21.15.76:0/2388598022 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f78b40bd070 con 0x558ae3c35bd0 2024-03-12T05:56:14.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.287+0000 7f78d3dd7740 1 -- 172.21.15.76:0/2388598022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78bc01aaa0 con 0x558ae3c35bd0 2024-03-12T05:56:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.288+0000 7f78b27fc640 1 -- 172.21.15.76:0/2388598022 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78b4003800 con 0x558ae3c35bd0 2024-03-12T05:56:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.288+0000 7f78d3dd7740 10 client.15762 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.288+0000 7f78d3dd7740 1 -- 172.21.15.76:0/2388598022 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x558ae3b5d6a0 con 0x558ae3c35bd0 2024-03-12T05:56:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.288+0000 7f78b27fc640 1 -- 172.21.15.76:0/2388598022 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78bc01aaa0 con 0x558ae3c35bd0 2024-03-12T05:56:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.288+0000 7f78b27fc640 1 -- 172.21.15.76:0/2388598022 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f78b40edd20 con 0x558ae3c35bd0 2024-03-12T05:56:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.288+0000 7f78d3dd7740 10 client.15762 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.288+0000 7f78d3dd7740 10 client.15762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.288+0000 7f78d3dd7740 20 client.15762 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.288+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/2388598022 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558ae3ca5c50 0x558ae3ca8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.288+0000 7f78d3dd7740 4 client.15762 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:56:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.288+0000 7f78d3dd7740 1 -- 172.21.15.76:0/2388598022 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558ae3928890 con 0x558ae3ca5c50 2024-03-12T05:56:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.288+0000 7f78cad9d640 1 --2- 172.21.15.76:0/2388598022 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558ae3ca5c50 0x558ae3ca8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78b27fc640 10 client.15762 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78cad9d640 1 --2- 172.21.15.76:0/2388598022 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558ae3ca5c50 0x558ae3ca8070 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba3b8b39cd77b8fc server_cookie=6ed522831a46bccc in_seq=0 out_seq=0 2024-03-12T05:56:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78b27fc640 1 -- 172.21.15.76:0/2388598022 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558ae3928890 con 0x558ae3ca5c50 2024-03-12T05:56:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78b27fc640 10 client.15762 handle_command_reply: tid=0 2024-03-12T05:56:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78d3dd7740 1 client.15762 shutdown 2024-03-12T05:56:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78d3dd7740 1 -- 172.21.15.76:0/2388598022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c35bd0 msgr2=0x558ae3c43d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/2388598022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c35bd0 0x558ae3c43d20 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f78b40edbd0 tx=0x7f78b408bd50 comp rx=0 tx=0).stop 2024-03-12T05:56:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78d3dd7740 1 -- 172.21.15.76:0/2388598022 shutdown_connections 2024-03-12T05:56:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/2388598022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ae3c3e7b0 0x558ae3ca3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/2388598022 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558ae3ca5c50 0x558ae3ca8070 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/2388598022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ae3c35bd0 0x558ae3c43d20 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78b27fc640 0 client.15762 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78d3dd7740 1 --2- 172.21.15.76:0/2388598022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae3c352b0 0x558ae3c437e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78d3dd7740 1 -- 172.21.15.76:0/2388598022 >> 172.21.15.76:0/2388598022 conn(0x558ae3c2f990 msgr2=0x558ae3c55750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78d3dd7740 1 -- 172.21.15.76:0/2388598022 shutdown_connections 2024-03-12T05:56:14.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78d3dd7740 1 -- 172.21.15.76:0/2388598022 wait complete. 2024-03-12T05:56:14.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.289+0000 7f78d3dd7740 20 client.15762 trim_cache size 0 max 16384 2024-03-12T05:56:14.301 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:56:14.301 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:56:14.301 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T05:56:14.301 INFO:teuthology.orchestra.run.smithi076.stdout: "7ce9fb14-f96a-4173-861b-ded3bde7b133": { 2024-03-12T05:56:14.301 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T05:56:14.301 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "7ce9fb14-f96a-4173-861b-ded3bde7b133", 2024-03-12T05:56:14.301 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T05:56:14.301 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:56:14.301 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:56:14.302 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:56:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.290+0000 7f78d120b640 1 -- 172.21.15.76:0/258481115 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f78b80a1cc0 msgr2=0x7f78b80a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.290+0000 7f78d120b640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f78b80a1cc0 0x7f78b80a4180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f78c4005c40 tx=0x7f78c4009200 comp rx=0 tx=0).stop 2024-03-12T05:56:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.290+0000 7f78d120b640 1 -- 172.21.15.76:0/258481115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78cc105330 msgr2=0x7f78cc19d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.290+0000 7f78d120b640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78cc105330 0x7f78cc19d970 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009230 tx=0x7f78b4004270 comp rx=0 tx=0).stop 2024-03-12T05:56:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.290+0000 7f78d120b640 1 -- 172.21.15.76:0/258481115 shutdown_connections 2024-03-12T05:56:14.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.290+0000 7f78d120b640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f78b80a1cc0 0x7f78b80a4180 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.290+0000 7f78d120b640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f78cc1975e0 0x7f78cc197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.290+0000 7f78d120b640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f78cc105330 0x7f78cc19d970 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.290+0000 7f78d120b640 1 --2- 172.21.15.76:0/258481115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78cc100260 0x7f78cc19d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:14.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.290+0000 7f78d120b640 1 -- 172.21.15.76:0/258481115 >> 172.21.15.76:0/258481115 conn(0x7f78cc077180 msgr2=0x7f78cc068810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:14.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.290+0000 7f78d120b640 1 -- 172.21.15.76:0/258481115 shutdown_connections 2024-03-12T05:56:14.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:14.290+0000 7f78d120b640 1 -- 172.21.15.76:0/258481115 wait complete. 2024-03-12T05:56:14.304 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:14 smithi076 ceph-mon[32095]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-12T05:56:14.305 INFO:tasks.cephfs.filesystem:scrub status for tag:7ce9fb14-f96a-4173-861b-ded3bde7b133 - {'path': '/', 'tag': '7ce9fb14-f96a-4173-861b-ded3bde7b133', 'options': 'recursive,force'} 2024-03-12T05:56:14.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:14 smithi082 ceph-mon[36382]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-12T05:56:14.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:14 smithi088 ceph-mon[36585]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-12T05:56:15.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:15 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T05:56:15.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:15 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:15.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:15 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:56:15.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:15 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:15.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:15 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T05:56:15.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:15 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:15.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:15 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:56:15.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:15 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:15.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:15 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T05:56:15.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:15 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:15.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:15 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:56:15.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:15 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:16.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:16 smithi082 ceph-mon[36382]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-03-12T05:56:16.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:16 smithi088 ceph-mon[36585]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-03-12T05:56:16.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:16 smithi076 ceph-mon[32095]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-03-12T05:56:18.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:18 smithi082 ceph-mon[36382]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:56:18.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:18 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:56:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:18 smithi088 ceph-mon[36585]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:56:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:18 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:56:18.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:18 smithi076 ceph-mon[32095]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:56:18.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:18 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:56:20.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:20 smithi082 ceph-mon[36382]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-12T05:56:20.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:20 smithi088 ceph-mon[36585]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-12T05:56:20.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:20 smithi076 ceph-mon[32095]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-12T05:56:22.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:22 smithi082 ceph-mon[36382]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-12T05:56:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:22 smithi088 ceph-mon[36585]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-12T05:56:22.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:22 smithi076 ceph-mon[32095]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-12T05:56:24.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:24 smithi082 ceph-mon[36382]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 8 op/s 2024-03-12T05:56:24.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:24 smithi088 ceph-mon[36585]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 8 op/s 2024-03-12T05:56:24.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:24 smithi076 ceph-mon[32095]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 8 op/s 2024-03-12T05:56:26.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:26 smithi082 ceph-mon[36382]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-12T05:56:26.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:26 smithi088 ceph-mon[36585]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-12T05:56:26.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:26 smithi076 ceph-mon[32095]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-12T05:56:28.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:28 smithi082 ceph-mon[36382]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-12T05:56:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:28 smithi088 ceph-mon[36585]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-12T05:56:28.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:28 smithi076 ceph-mon[32095]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-12T05:56:30.434 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:30 smithi082 ceph-mon[36382]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-12T05:56:30.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:30 smithi088 ceph-mon[36585]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-12T05:56:30.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:30 smithi076 ceph-mon[32095]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-12T05:56:32.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:32 smithi082 ceph-mon[36382]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-12T05:56:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:32 smithi088 ceph-mon[36585]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-12T05:56:32.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:32 smithi076 ceph-mon[32095]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-12T05:56:34.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:34 smithi082 ceph-mon[36382]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-12T05:56:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:34 smithi088 ceph-mon[36585]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-12T05:56:34.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:34 smithi076 ceph-mon[32095]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-12T05:56:36.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:36 smithi082 ceph-mon[36382]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:36 smithi088 ceph-mon[36585]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:36.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:36 smithi076 ceph-mon[32095]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:38.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:38 smithi082 ceph-mon[36382]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:38 smithi088 ceph-mon[36585]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:38.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:38 smithi076 ceph-mon[32095]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:40.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:40 smithi082 ceph-mon[36382]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:56:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:40 smithi088 ceph-mon[36585]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:56:40.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:40 smithi076 ceph-mon[32095]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:56:41.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:41 smithi082 ceph-mon[36382]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:41.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:41 smithi088 ceph-mon[36585]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:41.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:41 smithi076 ceph-mon[32095]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:44.306 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:56:44.306 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:56:44.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:44 smithi076 ceph-mon[32095]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:56:44.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.428+0000 7fbef6c88640 1 -- 172.21.15.76:0/2751247254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbef0100780 msgr2=0x7fbef0100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:44.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.428+0000 7fbef6c88640 1 --2- 172.21.15.76:0/2751247254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbef0100780 0x7fbef0100be0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fbedc009900 tx=0x7fbedc02bc80 comp rx=0 tx=0).stop 2024-03-12T05:56:44.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.428+0000 7fbef6c88640 1 -- 172.21.15.76:0/2751247254 shutdown_connections 2024-03-12T05:56:44.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.428+0000 7fbef6c88640 1 --2- 172.21.15.76:0/2751247254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbef0101120 0x7fbef010ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.428+0000 7fbef6c88640 1 --2- 172.21.15.76:0/2751247254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbef0100780 0x7fbef0100be0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.428+0000 7fbef6c88640 1 --2- 172.21.15.76:0/2751247254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbef0104fe0 0x7fbef01053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.428+0000 7fbef6c88640 1 -- 172.21.15.76:0/2751247254 >> 172.21.15.76:0/2751247254 conn(0x7fbef00775d0 msgr2=0x7fbef00fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:44.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.428+0000 7fbef6c88640 1 -- 172.21.15.76:0/2751247254 shutdown_connections 2024-03-12T05:56:44.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.428+0000 7fbef6c88640 1 -- 172.21.15.76:0/2751247254 wait complete. 2024-03-12T05:56:44.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.429+0000 7fbef6c88640 1 Processor -- start 2024-03-12T05:56:44.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.429+0000 7fbef6c88640 1 -- start start 2024-03-12T05:56:44.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.429+0000 7fbef6c88640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbef0100780 0x7fbef01955e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.429+0000 7fbef6c88640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbef0101120 0x7fbef0195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.429+0000 7fbef6c88640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbef0104fe0 0x7fbef0199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.429+0000 7fbef6c88640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef019a4a0 con 0x7fbef0100780 2024-03-12T05:56:44.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.429+0000 7fbef6c88640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef019a610 con 0x7fbef0101120 2024-03-12T05:56:44.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.429+0000 7fbef6c88640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef019a7d0 con 0x7fbef0104fe0 2024-03-12T05:56:44.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.429+0000 7fbef4a24640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbef0100780 0x7fbef01955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.429+0000 7fbef4a24640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbef0100780 0x7fbef01955e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40972/0 (socket says 172.21.15.76:40972) 2024-03-12T05:56:44.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.429+0000 7fbef4a24640 1 -- 172.21.15.76:0/2760933356 learned_addr learned my addr 172.21.15.76:0/2760933356 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:44.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.429+0000 7fbeeffff640 1 --2- 172.21.15.76:0/2760933356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbef0101120 0x7fbef0195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.429+0000 7fbef5225640 1 --2- 172.21.15.76:0/2760933356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbef0104fe0 0x7fbef0199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.430+0000 7fbeeffff640 1 -- 172.21.15.76:0/2760933356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbef0104fe0 msgr2=0x7fbef0199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:44.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.430+0000 7fbeeffff640 1 --2- 172.21.15.76:0/2760933356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbef0104fe0 0x7fbef0199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.430+0000 7fbeeffff640 1 -- 172.21.15.76:0/2760933356 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbef0100780 msgr2=0x7fbef01955e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:44.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.430+0000 7fbeeffff640 1 --2- 172.21.15.76:0/2760933356 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbef0100780 0x7fbef01955e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.430+0000 7fbeeffff640 1 -- 172.21.15.76:0/2760933356 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbedc009560 con 0x7fbef0101120 2024-03-12T05:56:44.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.430+0000 7fbeeffff640 1 --2- 172.21.15.76:0/2760933356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbef0101120 0x7fbef0195b20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbedc036040 tx=0x7fbedc02c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:44.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.430+0000 7fbeedffb640 1 -- 172.21.15.76:0/2760933356 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0043d0 con 0x7fbef0101120 2024-03-12T05:56:44.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.430+0000 7fbeedffb640 1 -- 172.21.15.76:0/2760933356 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbedc0052d0 con 0x7fbef0101120 2024-03-12T05:56:44.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.430+0000 7fbef6c88640 1 -- 172.21.15.76:0/2760933356 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef019aa50 con 0x7fbef0101120 2024-03-12T05:56:44.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.430+0000 7fbeedffb640 1 -- 172.21.15.76:0/2760933356 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc003c90 con 0x7fbef0101120 2024-03-12T05:56:44.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.430+0000 7fbef6c88640 1 -- 172.21.15.76:0/2760933356 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef01a6740 con 0x7fbef0101120 2024-03-12T05:56:44.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.433+0000 7fbeedffb640 1 -- 172.21.15.76:0/2760933356 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fbedc005440 con 0x7fbef0101120 2024-03-12T05:56:44.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.434+0000 7fbeedffb640 1 --2- 172.21.15.76:0/2760933356 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbee00a1da0 0x7fbee00a4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.434+0000 7fbeedffb640 1 -- 172.21.15.76:0/2760933356 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbedc0ef970 con 0x7fbef0101120 2024-03-12T05:56:44.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.434+0000 7fbef4a24640 1 --2- 172.21.15.76:0/2760933356 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbee00a1da0 0x7fbee00a4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.434+0000 7fbef9867740 1 Processor -- start 2024-03-12T05:56:44.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.434+0000 7fbef9867740 1 -- start start 2024-03-12T05:56:44.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef9867740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x560523498ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef9867740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234993e0 0x5605234a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef9867740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a1fc0 0x5605234a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef9867740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605233ff280 con 0x560523498ac0 2024-03-12T05:56:44.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef9867740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605233fc7b0 con 0x5605234993e0 2024-03-12T05:56:44.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef9867740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605233f7f90 con 0x5605234a1fc0 2024-03-12T05:56:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef5225640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a1fc0 0x5605234a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef5225640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a1fc0 0x5605234a43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.76:46868/0 (socket says 172.21.15.76:46868) 2024-03-12T05:56:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef5225640 1 -- 172.21.15.76:0/3850245420 learned_addr learned my addr 172.21.15.76:0/3850245420 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbeeffff640 1 --2- 172.21.15.76:0/3850245420 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234993e0 0x5605234a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef4a24640 1 --2- 172.21.15.76:0/2760933356 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbee00a1da0 0x7fbee00a4260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009ce0 tx=0x7fbee4005f90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef5225640 1 -- 172.21.15.76:0/3850245420 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234993e0 msgr2=0x5605234a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:44.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef5225640 1 --2- 172.21.15.76:0/3850245420 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234993e0 0x5605234a1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef5225640 1 -- 172.21.15.76:0/3850245420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 msgr2=0x560523498ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef4a24640 1 --2- 172.21.15.76:0/3850245420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x560523498ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef5225640 1 --2- 172.21.15.76:0/3850245420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x560523498ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.435+0000 7fbef5225640 1 -- 172.21.15.76:0/3850245420 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605233b6570 con 0x5605234a1fc0 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.436+0000 7fbef4a24640 1 --2- 172.21.15.76:0/3850245420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x560523498ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.436+0000 7fbef5225640 1 --2- 172.21.15.76:0/3850245420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a1fc0 0x5605234a43b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009f20 tx=0x7fbee800c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=445a8cb7982fe041 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.436+0000 7fbedaffd640 1 -- 172.21.15.76:0/3850245420 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8011070 con 0x5605234a1fc0 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.436+0000 7fbedaffd640 1 -- 172.21.15.76:0/3850245420 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbee8009a50 con 0x5605234a1fc0 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.436+0000 7fbedaffd640 1 -- 172.21.15.76:0/3850245420 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee800a8c0 con 0x5605234a1fc0 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.437+0000 7fbef9867740 1 -- 172.21.15.76:0/3850245420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a1fc0 msgr2=0x5605234a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.437+0000 7fbef9867740 1 --2- 172.21.15.76:0/3850245420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a1fc0 0x5605234a43b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009f20 tx=0x7fbee800c8b0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.437+0000 7fbef9867740 1 -- 172.21.15.76:0/3850245420 shutdown_connections 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.437+0000 7fbef9867740 1 --2- 172.21.15.76:0/3850245420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a1fc0 0x5605234a43b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.437+0000 7fbef9867740 1 --2- 172.21.15.76:0/3850245420 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234993e0 0x5605234a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.437+0000 7fbef9867740 1 --2- 172.21.15.76:0/3850245420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x560523498ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.437+0000 7fbef9867740 1 -- 172.21.15.76:0/3850245420 >> 172.21.15.76:0/3850245420 conn(0x5605234931a0 msgr2=0x560523493580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:44.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.437+0000 7fbef9867740 1 -- 172.21.15.76:0/3850245420 shutdown_connections 2024-03-12T05:56:44.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.437+0000 7fbef9867740 1 -- 172.21.15.76:0/3850245420 wait complete. 2024-03-12T05:56:44.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.437+0000 7fbef9867740 1 -- start start 2024-03-12T05:56:44.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.437+0000 7fbef9867740 1 Processor -- start 2024-03-12T05:56:44.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbef9867740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x5605234a1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbef9867740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234993e0 0x56052349edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbef9867740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a1fc0 0x56052349f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbef9867740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605230cb430 con 0x560523498ac0 2024-03-12T05:56:44.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbef9867740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560523087e00 con 0x5605234993e0 2024-03-12T05:56:44.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbef9867740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56052337cfd0 con 0x5605234a1fc0 2024-03-12T05:56:44.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbef9867740 10 client.0 _ll_register_callbacks cb 0x5605233d2fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:44.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbef4a24640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x5605234a1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbef4a24640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x5605234a1a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40982/0 (socket says 172.21.15.76:40982) 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbef4a24640 1 -- 172.21.15.76:0/3908020325 learned_addr learned my addr 172.21.15.76:0/3908020325 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbeeffff640 1 --2- 172.21.15.76:0/3908020325 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234993e0 0x56052349edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbef5225640 1 --2- 172.21.15.76:0/3908020325 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a1fc0 0x56052349f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbeeffff640 1 -- 172.21.15.76:0/3908020325 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a1fc0 msgr2=0x56052349f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbeeffff640 1 --2- 172.21.15.76:0/3908020325 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a1fc0 0x56052349f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbeeffff640 1 -- 172.21.15.76:0/3908020325 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 msgr2=0x5605234a1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbeeffff640 1 --2- 172.21.15.76:0/3908020325 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x5605234a1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbeeffff640 1 -- 172.21.15.76:0/3908020325 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee8009bd0 con 0x5605234993e0 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.438+0000 7fbef4a24640 1 --2- 172.21.15.76:0/3908020325 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x5605234a1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.439+0000 7fbeeffff640 1 --2- 172.21.15.76:0/3908020325 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234993e0 0x56052349edc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0ed020 tx=0x7fbedc0bc2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1c25998d60afb59 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.439+0000 7fbecffff640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.439+0000 7fbecffff640 1 -- 172.21.15.76:0/3908020325 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0bcb30 con 0x5605234993e0 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.439+0000 7fbecffff640 1 -- 172.21.15.76:0/3908020325 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbedc0bc430 con 0x5605234993e0 2024-03-12T05:56:44.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.439+0000 7fbef9867740 1 -- 172.21.15.76:0/3908020325 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560523134720 con 0x5605234993e0 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.439+0000 7fbecffff640 1 -- 172.21.15.76:0/3908020325 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbedc0c0580 con 0x5605234993e0 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.439+0000 7fbef9867740 10 client.25235 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.439+0000 7fbef9867740 1 -- 172.21.15.76:0/3908020325 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5605233f6610 con 0x5605234993e0 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.439+0000 7fbecffff640 1 -- 172.21.15.76:0/3908020325 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560523134720 con 0x5605234993e0 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.439+0000 7fbecffff640 1 -- 172.21.15.76:0/3908020325 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbedc03c070 con 0x5605234993e0 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.440+0000 7fbef9867740 10 client.25235 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.440+0000 7fbef9867740 10 client.25235 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.440+0000 7fbef9867740 20 client.25235 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.440+0000 7fbef9867740 1 --2- 172.21.15.76:0/3908020325 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5605234b1d90 0x5605234b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.440+0000 7fbef9867740 4 client.25235 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.440+0000 7fbef9867740 1 -- 172.21.15.76:0/3908020325 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560523183890 con 0x5605234b1d90 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.440+0000 7fbef4a24640 1 --2- 172.21.15.76:0/3908020325 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5605234b1d90 0x5605234b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.440+0000 7fbecffff640 10 client.25235 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.440+0000 7fbef4a24640 1 --2- 172.21.15.76:0/3908020325 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5605234b1d90 0x5605234b41b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cce14fa404e5540d server_cookie=b709596a3178213a in_seq=0 out_seq=0 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbecffff640 1 -- 172.21.15.76:0/3908020325 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560523183890 con 0x5605234b1d90 2024-03-12T05:56:44.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbecffff640 10 client.25235 handle_command_reply: tid=0 2024-03-12T05:56:44.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbef9867740 1 client.25235 shutdown 2024-03-12T05:56:44.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbef9867740 1 -- 172.21.15.76:0/3908020325 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234993e0 msgr2=0x56052349edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:44.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbef9867740 1 --2- 172.21.15.76:0/3908020325 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234993e0 0x56052349edc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0ed020 tx=0x7fbedc0bc2c0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbef9867740 1 -- 172.21.15.76:0/3908020325 shutdown_connections 2024-03-12T05:56:44.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbef9867740 1 --2- 172.21.15.76:0/3908020325 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a1fc0 0x56052349f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbef9867740 1 --2- 172.21.15.76:0/3908020325 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234993e0 0x56052349edc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbef9867740 1 --2- 172.21.15.76:0/3908020325 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5605234b1d90 0x5605234b41b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbecffff640 0 client.25235 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:44.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbef9867740 1 --2- 172.21.15.76:0/3908020325 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x5605234a1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbef9867740 1 -- 172.21.15.76:0/3908020325 >> 172.21.15.76:0/3908020325 conn(0x5605234931a0 msgr2=0x560523494f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:44.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbef9867740 1 -- 172.21.15.76:0/3908020325 shutdown_connections 2024-03-12T05:56:44.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbef9867740 1 -- 172.21.15.76:0/3908020325 wait complete. 2024-03-12T05:56:44.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.441+0000 7fbef9867740 20 client.25235 trim_cache size 0 max 16384 2024-03-12T05:56:44.451 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:44 smithi082 ceph-mon[36382]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:56:44.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.456+0000 7fbef9867740 1 Processor -- start 2024-03-12T05:56:44.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.456+0000 7fbef9867740 1 -- start start 2024-03-12T05:56:44.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbef9867740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560523498ac0 0x5605234a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbef9867740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605234993e0 0x5605234a93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbef9867740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234a1fc0 0x5605234a14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbef9867740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605231838c0 con 0x5605234993e0 2024-03-12T05:56:44.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbef9867740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560523134750 con 0x5605234a1fc0 2024-03-12T05:56:44.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbef9867740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560523494420 con 0x560523498ac0 2024-03-12T05:56:44.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbeeffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605234993e0 0x5605234a93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbeeffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605234993e0 0x5605234a93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40988/0 (socket says 172.21.15.76:40988) 2024-03-12T05:56:44.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbeeffff640 1 -- 172.21.15.76:0/1704977844 learned_addr learned my addr 172.21.15.76:0/1704977844 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:44.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbef5225640 1 --2- 172.21.15.76:0/1704977844 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234a1fc0 0x5605234a14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbef4a24640 1 --2- 172.21.15.76:0/1704977844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560523498ac0 0x5605234a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbeeffff640 1 -- 172.21.15.76:0/1704977844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560523498ac0 msgr2=0x5605234a8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:44.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbeeffff640 1 --2- 172.21.15.76:0/1704977844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560523498ac0 0x5605234a8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbeeffff640 1 -- 172.21.15.76:0/1704977844 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234a1fc0 msgr2=0x5605234a14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:44.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbeeffff640 1 --2- 172.21.15.76:0/1704977844 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234a1fc0 0x5605234a14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbeeffff640 1 -- 172.21.15.76:0/1704977844 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560523392ac0 con 0x5605234993e0 2024-03-12T05:56:44.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.457+0000 7fbeeffff640 1 --2- 172.21.15.76:0/1704977844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605234993e0 0x5605234a93b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fbedc048000 tx=0x7fbedc02bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e32a8e4a713aa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:44.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.458+0000 7fbeda7fc640 1 -- 172.21.15.76:0/1704977844 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0bcb30 con 0x5605234993e0 2024-03-12T05:56:44.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.458+0000 7fbeda7fc640 1 -- 172.21.15.76:0/1704977844 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbedc0bc430 con 0x5605234993e0 2024-03-12T05:56:44.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.458+0000 7fbeda7fc640 1 -- 172.21.15.76:0/1704977844 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0c4c60 con 0x5605234993e0 2024-03-12T05:56:44.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.459+0000 7fbef9867740 1 -- 172.21.15.76:0/1704977844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605234993e0 msgr2=0x5605234a93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:44.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.459+0000 7fbef9867740 1 --2- 172.21.15.76:0/1704977844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605234993e0 0x5605234a93b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fbedc048000 tx=0x7fbedc02bc80 comp rx=0 tx=0).stop 2024-03-12T05:56:44.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.459+0000 7fbef9867740 1 -- 172.21.15.76:0/1704977844 shutdown_connections 2024-03-12T05:56:44.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.459+0000 7fbef9867740 1 --2- 172.21.15.76:0/1704977844 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234a1fc0 0x5605234a14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.459+0000 7fbef9867740 1 --2- 172.21.15.76:0/1704977844 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605234993e0 0x5605234a93b0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.459+0000 7fbef9867740 1 --2- 172.21.15.76:0/1704977844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560523498ac0 0x5605234a8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.459+0000 7fbef9867740 1 -- 172.21.15.76:0/1704977844 >> 172.21.15.76:0/1704977844 conn(0x5605234931a0 msgr2=0x5605234b8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:44.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.459+0000 7fbef9867740 1 -- 172.21.15.76:0/1704977844 shutdown_connections 2024-03-12T05:56:44.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.459+0000 7fbef9867740 1 -- 172.21.15.76:0/1704977844 wait complete. 2024-03-12T05:56:44.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.459+0000 7fbef9867740 1 -- start start 2024-03-12T05:56:44.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.459+0000 7fbef9867740 1 Processor -- start 2024-03-12T05:56:44.469 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbef9867740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x5605234a7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbef9867740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234a1fc0 0x5605234a80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbef9867740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a8610 0x5605235082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbef9867740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560523508820 con 0x560523498ac0 2024-03-12T05:56:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbef4a24640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x5605234a7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbef9867740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560523508990 con 0x5605234a1fc0 2024-03-12T05:56:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbef9867740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560523508b50 con 0x5605234a8610 2024-03-12T05:56:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbef9867740 10 client.0 _ll_register_callbacks cb 0x7fbee8009710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbef4a24640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x5605234a7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40994/0 (socket says 172.21.15.76:40994) 2024-03-12T05:56:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbef4a24640 1 -- 172.21.15.76:0/3717942620 learned_addr learned my addr 172.21.15.76:0/3717942620 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbef5225640 1 --2- 172.21.15.76:0/3717942620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a8610 0x5605235082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbeeffff640 1 --2- 172.21.15.76:0/3717942620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234a1fc0 0x5605234a80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbeeffff640 1 -- 172.21.15.76:0/3717942620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a8610 msgr2=0x5605235082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbeeffff640 1 --2- 172.21.15.76:0/3717942620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a8610 0x5605235082e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbeeffff640 1 -- 172.21.15.76:0/3717942620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 msgr2=0x5605234a7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:44.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbeeffff640 1 --2- 172.21.15.76:0/3717942620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x5605234a7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbeeffff640 1 -- 172.21.15.76:0/3717942620 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbedc0d6020 con 0x5605234a1fc0 2024-03-12T05:56:44.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.460+0000 7fbef4a24640 1 --2- 172.21.15.76:0/3717942620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x5605234a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:44.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.461+0000 7fbeeffff640 1 --2- 172.21.15.76:0/3717942620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234a1fc0 0x5605234a80d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x5605234993b0 tx=0x7fbedc0c4970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da3bd0a3b111288 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:44.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.461+0000 7fbecffff640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T05:56:44.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.461+0000 7fbecffff640 1 -- 172.21.15.76:0/3717942620 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0c49e0 con 0x5605234a1fc0 2024-03-12T05:56:44.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.461+0000 7fbecffff640 1 -- 172.21.15.76:0/3717942620 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbedc0bcb30 con 0x5605234a1fc0 2024-03-12T05:56:44.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.461+0000 7fbef9867740 1 -- 172.21.15.76:0/3717942620 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5605230f77e0 con 0x5605234a1fc0 2024-03-12T05:56:44.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.461+0000 7fbecffff640 1 -- 172.21.15.76:0/3717942620 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbedc0c4bd0 con 0x5605234a1fc0 2024-03-12T05:56:44.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.461+0000 7fbef9867740 10 client.25241 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:44.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.461+0000 7fbef9867740 1 -- 172.21.15.76:0/3717942620 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5605233b86a0 con 0x5605234a1fc0 2024-03-12T05:56:44.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.462+0000 7fbecffff640 1 -- 172.21.15.76:0/3717942620 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5605230f77e0 con 0x5605234a1fc0 2024-03-12T05:56:44.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.462+0000 7fbecffff640 1 -- 172.21.15.76:0/3717942620 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbedc03c070 con 0x5605234a1fc0 2024-03-12T05:56:44.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.462+0000 7fbef9867740 10 client.25241 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.462+0000 7fbef9867740 10 client.25241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.462+0000 7fbef9867740 20 client.25241 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.462+0000 7fbef9867740 1 --2- 172.21.15.76:0/3717942620 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560523509e50 0x56052350c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.462+0000 7fbef9867740 4 client.25241 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:56:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.462+0000 7fbef9867740 1 -- 172.21.15.76:0/3717942620 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5605230e64e0 con 0x560523509e50 2024-03-12T05:56:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.462+0000 7fbef4a24640 1 --2- 172.21.15.76:0/3717942620 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560523509e50 0x56052350c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbecffff640 10 client.25241 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbef4a24640 1 --2- 172.21.15.76:0/3717942620 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560523509e50 0x56052350c240 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b2f8123cad12e5b server_cookie=d74e364042621d7c in_seq=0 out_seq=0 2024-03-12T05:56:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbecffff640 1 -- 172.21.15.76:0/3717942620 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5605230e64e0 con 0x560523509e50 2024-03-12T05:56:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbecffff640 10 client.25241 handle_command_reply: tid=0 2024-03-12T05:56:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbef9867740 1 client.25241 shutdown 2024-03-12T05:56:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbef9867740 1 -- 172.21.15.76:0/3717942620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234a1fc0 msgr2=0x5605234a80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbef9867740 1 --2- 172.21.15.76:0/3717942620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234a1fc0 0x5605234a80d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x5605234993b0 tx=0x7fbedc0c4970 comp rx=0 tx=0).stop 2024-03-12T05:56:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbef9867740 1 -- 172.21.15.76:0/3717942620 shutdown_connections 2024-03-12T05:56:44.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbef9867740 1 --2- 172.21.15.76:0/3717942620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605234a8610 0x5605235082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbef9867740 1 --2- 172.21.15.76:0/3717942620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605234a1fc0 0x5605234a80d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbef9867740 1 --2- 172.21.15.76:0/3717942620 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560523509e50 0x56052350c240 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbef9867740 1 --2- 172.21.15.76:0/3717942620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560523498ac0 0x5605234a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbecffff640 0 client.25241 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:44.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbef9867740 1 -- 172.21.15.76:0/3717942620 >> 172.21.15.76:0/3717942620 conn(0x5605234931a0 msgr2=0x5605234891c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:44.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbef9867740 1 -- 172.21.15.76:0/3717942620 shutdown_connections 2024-03-12T05:56:44.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbef9867740 1 -- 172.21.15.76:0/3717942620 wait complete. 2024-03-12T05:56:44.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.463+0000 7fbef9867740 20 client.25241 trim_cache size 0 max 16384 2024-03-12T05:56:44.473 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:56:44.473 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T05:56:44.474 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T05:56:44.474 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:56:44.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.464+0000 7fbef6c88640 1 -- 172.21.15.76:0/2760933356 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbee00a1da0 msgr2=0x7fbee00a4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:44.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.464+0000 7fbef6c88640 1 --2- 172.21.15.76:0/2760933356 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbee00a1da0 0x7fbee00a4260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009ce0 tx=0x7fbee4005f90 comp rx=0 tx=0).stop 2024-03-12T05:56:44.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.464+0000 7fbef6c88640 1 -- 172.21.15.76:0/2760933356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbef0101120 msgr2=0x7fbef0195b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:44.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.464+0000 7fbef6c88640 1 --2- 172.21.15.76:0/2760933356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbef0101120 0x7fbef0195b20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbedc036040 tx=0x7fbedc02c740 comp rx=0 tx=0).stop 2024-03-12T05:56:44.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.464+0000 7fbef6c88640 1 -- 172.21.15.76:0/2760933356 shutdown_connections 2024-03-12T05:56:44.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.464+0000 7fbef6c88640 1 --2- 172.21.15.76:0/2760933356 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbee00a1da0 0x7fbee00a4260 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.464+0000 7fbef6c88640 1 --2- 172.21.15.76:0/2760933356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbef0104fe0 0x7fbef0199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.464+0000 7fbef6c88640 1 --2- 172.21.15.76:0/2760933356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbef0101120 0x7fbef0195b20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.464+0000 7fbef6c88640 1 --2- 172.21.15.76:0/2760933356 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbef0100780 0x7fbef01955e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.464+0000 7fbef6c88640 1 -- 172.21.15.76:0/2760933356 >> 172.21.15.76:0/2760933356 conn(0x7fbef00775d0 msgr2=0x7fbef0102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:44.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.464+0000 7fbef6c88640 1 -- 172.21.15.76:0/2760933356 shutdown_connections 2024-03-12T05:56:44.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.464+0000 7fbef6c88640 1 -- 172.21.15.76:0/2760933356 wait complete. 2024-03-12T05:56:44.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:44 smithi088 ceph-mon[36585]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:56:44.487 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:56:44.487 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:56:44.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.629+0000 7f4672084640 1 -- 172.21.15.76:0/547400552 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f466c106470 msgr2=0x7f466c10e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:44.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.629+0000 7f4672084640 1 --2- 172.21.15.76:0/547400552 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f466c106470 0x7f466c10e040 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4664009cd0 tx=0x7f466402c100 comp rx=0 tx=0).stop 2024-03-12T05:56:44.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.630+0000 7f4672084640 1 -- 172.21.15.76:0/547400552 shutdown_connections 2024-03-12T05:56:44.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.630+0000 7f4672084640 1 --2- 172.21.15.76:0/547400552 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f466c106470 0x7f466c10e040 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.630+0000 7f4672084640 1 --2- 172.21.15.76:0/547400552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f466c101d50 0x7f466c105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.630+0000 7f4672084640 1 --2- 172.21.15.76:0/547400552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f466c1013a0 0x7f466c101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.630+0000 7f4672084640 1 -- 172.21.15.76:0/547400552 >> 172.21.15.76:0/547400552 conn(0x7f466c0fcc40 msgr2=0x7f466c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:44.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.630+0000 7f4672084640 1 -- 172.21.15.76:0/547400552 shutdown_connections 2024-03-12T05:56:44.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.630+0000 7f4672084640 1 -- 172.21.15.76:0/547400552 wait complete. 2024-03-12T05:56:44.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.631+0000 7f4672084640 1 Processor -- start 2024-03-12T05:56:44.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.631+0000 7f4672084640 1 -- start start 2024-03-12T05:56:44.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.631+0000 7f4672084640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f466c1013a0 0x7f466c19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.631+0000 7f4672084640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f466c106470 0x7f466c19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.631+0000 7f4672084640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f466c1975a0 0x7f466c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.631+0000 7f4672084640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c197f90 con 0x7f466c106470 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.631+0000 7f4672084640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c198100 con 0x7f466c1013a0 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.631+0000 7f4672084640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466c1982c0 con 0x7f466c1975a0 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.631+0000 7f466affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f466c106470 0x7f466c19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.631+0000 7f466affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f466c106470 0x7f466c19d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41008/0 (socket says 172.21.15.76:41008) 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.631+0000 7f466affd640 1 -- 172.21.15.76:0/4048488863 learned_addr learned my addr 172.21.15.76:0/4048488863 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.631+0000 7f466b7fe640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f466c1013a0 0x7f466c19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.632+0000 7f466bfff640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f466c1975a0 0x7f466c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.632+0000 7f466affd640 1 -- 172.21.15.76:0/4048488863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f466c1975a0 msgr2=0x7f466c197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.632+0000 7f466affd640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f466c1975a0 0x7f466c197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.632+0000 7f466affd640 1 -- 172.21.15.76:0/4048488863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f466c1013a0 msgr2=0x7f466c19d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.632+0000 7f466affd640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f466c1013a0 0x7f466c19d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.632+0000 7f466affd640 1 -- 172.21.15.76:0/4048488863 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4664009900 con 0x7f466c106470 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.632+0000 7f466bfff640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f466c1975a0 0x7f466c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.632+0000 7f466affd640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f466c106470 0x7f466c19d930 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f466000e9f0 tx=0x7f466000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:44.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.632+0000 7f4668ff9640 1 -- 172.21.15.76:0/4048488863 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46600096c0 con 0x7f466c106470 2024-03-12T05:56:44.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.632+0000 7f4668ff9640 1 -- 172.21.15.76:0/4048488863 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4660005500 con 0x7f466c106470 2024-03-12T05:56:44.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.632+0000 7f4672084640 1 -- 172.21.15.76:0/4048488863 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466c0733b0 con 0x7f466c106470 2024-03-12T05:56:44.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.633+0000 7f4672084640 1 -- 172.21.15.76:0/4048488863 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466c073860 con 0x7f466c106470 2024-03-12T05:56:44.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.633+0000 7f4668ff9640 1 -- 172.21.15.76:0/4048488863 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4660010430 con 0x7f466c106470 2024-03-12T05:56:44.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.635+0000 7f4672084640 1 -- 172.21.15.76:0/4048488863 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4658005290 con 0x7f466c106470 2024-03-12T05:56:44.640 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.636+0000 7f4668ff9640 1 -- 172.21.15.76:0/4048488863 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f466000ccc0 con 0x7f466c106470 2024-03-12T05:56:44.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.636+0000 7f4668ff9640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46540a1e10 0x7f46540a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:44.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.637+0000 7f4668ff9640 1 -- 172.21.15.76:0/4048488863 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4660014070 con 0x7f466c106470 2024-03-12T05:56:44.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.637+0000 7f466b7fe640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46540a1e10 0x7f46540a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:44.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.637+0000 7f466b7fe640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46540a1e10 0x7f46540a42d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f465c009bd0 tx=0x7f465c005f90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:44.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.641+0000 7f4668ff9640 1 -- 172.21.15.76:0/4048488863 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4660096910 con 0x7f466c106470 2024-03-12T05:56:44.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.876+0000 7f4672084640 1 -- 172.21.15.76:0/4048488863 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4658005810 con 0x7f466c106470 2024-03-12T05:56:44.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.877+0000 7f4668ff9640 1 -- 172.21.15.76:0/4048488863 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11328 (secure 0 0 0) 0x7f4660099fd0 con 0x7f466c106470 2024-03-12T05:56:44.882 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:56:44.882 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:55:33.056004+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:56:44.883 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 22 2024-03-12T05:56:44.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.880+0000 7f4672084640 1 -- 172.21.15.76:0/4048488863 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46540a1e10 msgr2=0x7f46540a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:44.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.880+0000 7f4672084640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46540a1e10 0x7f46540a42d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f465c009bd0 tx=0x7f465c005f90 comp rx=0 tx=0).stop 2024-03-12T05:56:44.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.880+0000 7f4672084640 1 -- 172.21.15.76:0/4048488863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f466c106470 msgr2=0x7f466c19d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:44.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.880+0000 7f4672084640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f466c106470 0x7f466c19d930 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f466000e9f0 tx=0x7f466000eeb0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.880+0000 7f4672084640 1 -- 172.21.15.76:0/4048488863 shutdown_connections 2024-03-12T05:56:44.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.880+0000 7f4672084640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46540a1e10 0x7f46540a42d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.880+0000 7f4672084640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f466c1975a0 0x7f466c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.880+0000 7f4672084640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f466c106470 0x7f466c19d930 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.880+0000 7f4672084640 1 --2- 172.21.15.76:0/4048488863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f466c1013a0 0x7f466c19d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:44.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.880+0000 7f4672084640 1 -- 172.21.15.76:0/4048488863 >> 172.21.15.76:0/4048488863 conn(0x7f466c0fcc40 msgr2=0x7f466c0fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:44.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.880+0000 7f4672084640 1 -- 172.21.15.76:0/4048488863 shutdown_connections 2024-03-12T05:56:44.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:44.880+0000 7f4672084640 1 -- 172.21.15.76:0/4048488863 wait complete. 2024-03-12T05:56:44.901 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:56:44.902 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:56:45.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.040+0000 7ff583b19640 1 -- 172.21.15.76:0/1396560684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff57c100ea0 msgr2=0x7ff57c104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.040+0000 7ff583b19640 1 --2- 172.21.15.76:0/1396560684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff57c100ea0 0x7ff57c104c50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009860 tx=0x7ff56c02bb70 comp rx=0 tx=0).stop 2024-03-12T05:56:45.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.040+0000 7ff583b19640 1 -- 172.21.15.76:0/1396560684 shutdown_connections 2024-03-12T05:56:45.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.040+0000 7ff583b19640 1 --2- 172.21.15.76:0/1396560684 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff57c105330 0x7ff57c10cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.040+0000 7ff583b19640 1 --2- 172.21.15.76:0/1396560684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff57c100ea0 0x7ff57c104c50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.040+0000 7ff583b19640 1 --2- 172.21.15.76:0/1396560684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff57c1004f0 0x7ff57c1008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.040+0000 7ff583b19640 1 -- 172.21.15.76:0/1396560684 >> 172.21.15.76:0/1396560684 conn(0x7ff57c077ef0 msgr2=0x7ff57c078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.041+0000 7ff583b19640 1 -- 172.21.15.76:0/1396560684 shutdown_connections 2024-03-12T05:56:45.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.041+0000 7ff583b19640 1 -- 172.21.15.76:0/1396560684 wait complete. 2024-03-12T05:56:45.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.041+0000 7ff583b19640 1 Processor -- start 2024-03-12T05:56:45.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.041+0000 7ff583b19640 1 -- start start 2024-03-12T05:56:45.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff583b19640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff57c1004f0 0x7ff57c19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff583b19640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff57c105330 0x7ff57c19d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff583b19640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff57c1975e0 0x7ff57c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff583b19640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c197fd0 con 0x7ff57c1975e0 2024-03-12T05:56:45.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff583b19640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c198140 con 0x7ff57c105330 2024-03-12T05:56:45.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff583b19640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c198300 con 0x7ff57c1004f0 2024-03-12T05:56:45.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff5810b4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff57c105330 0x7ff57c19d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff5810b4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff57c105330 0x7ff57c19d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:47036/0 (socket says 172.21.15.76:47036) 2024-03-12T05:56:45.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff5810b4640 1 -- 172.21.15.76:0/3909943619 learned_addr learned my addr 172.21.15.76:0/3909943619 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff5818b5640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff57c1004f0 0x7ff57c19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff5820b6640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff57c1975e0 0x7ff57c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff5810b4640 1 -- 172.21.15.76:0/3909943619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff57c1004f0 msgr2=0x7ff57c19d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff5810b4640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff57c1004f0 0x7ff57c19d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff5810b4640 1 -- 172.21.15.76:0/3909943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff57c1975e0 msgr2=0x7ff57c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.042+0000 7ff5810b4640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff57c1975e0 0x7ff57c197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.043+0000 7ff5810b4640 1 -- 172.21.15.76:0/3909943619 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff578009560 con 0x7ff57c105330 2024-03-12T05:56:45.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.043+0000 7ff5820b6640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff57c1975e0 0x7ff57c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:45.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.043+0000 7ff5810b4640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff57c105330 0x7ff57c19d970 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff56c000c00 tx=0x7ff56c02ca00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.043+0000 7ff572ffd640 1 -- 172.21.15.76:0/3909943619 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c004460 con 0x7ff57c105330 2024-03-12T05:56:45.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.043+0000 7ff572ffd640 1 -- 172.21.15.76:0/3909943619 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff56c034a10 con 0x7ff57c105330 2024-03-12T05:56:45.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.043+0000 7ff583b19640 1 -- 172.21.15.76:0/3909943619 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56c009490 con 0x7ff57c105330 2024-03-12T05:56:45.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.044+0000 7ff572ffd640 1 -- 172.21.15.76:0/3909943619 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c004e00 con 0x7ff57c105330 2024-03-12T05:56:45.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.044+0000 7ff583b19640 1 -- 172.21.15.76:0/3909943619 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57c1a45b0 con 0x7ff57c105330 2024-03-12T05:56:45.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.046+0000 7ff572ffd640 1 -- 172.21.15.76:0/3909943619 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7ff56c03b040 con 0x7ff57c105330 2024-03-12T05:56:45.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.047+0000 7ff572ffd640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff5680a1d80 0x7ff5680a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.047+0000 7ff5818b5640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff5680a1d80 0x7ff5680a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.047+0000 7ff572ffd640 1 -- 172.21.15.76:0/3909943619 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff56c0ef380 con 0x7ff57c105330 2024-03-12T05:56:45.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.048+0000 7ff5818b5640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff5680a1d80 0x7ff5680a4240 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff578009530 tx=0x7ff578009260 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.048+0000 7ff5867c5740 1 Processor -- start 2024-03-12T05:56:45.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.049+0000 7ff5867c5740 1 -- start start 2024-03-12T05:56:45.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.049+0000 7ff5867c5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543250 0x5574bc543630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.049+0000 7ff5867c5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc543b70 0x5574bc54c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.049+0000 7ff5867c5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc54c750 0x5574bc54eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.049+0000 7ff5867c5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574bc4fe960 con 0x5574bc543250 2024-03-12T05:56:45.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.049+0000 7ff5867c5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574bc4b2310 con 0x5574bc54c750 2024-03-12T05:56:45.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.049+0000 7ff5867c5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574bc4af840 con 0x5574bc543b70 2024-03-12T05:56:45.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.049+0000 7ff5818b5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543250 0x5574bc543630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.049+0000 7ff5818b5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543250 0x5574bc543630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41040/0 (socket says 172.21.15.76:41040) 2024-03-12T05:56:45.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.049+0000 7ff5818b5640 1 -- 172.21.15.76:0/933607809 learned_addr learned my addr 172.21.15.76:0/933607809 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.049+0000 7ff5820b6640 1 --2- 172.21.15.76:0/933607809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc54c750 0x5574bc54eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.049+0000 7ff5810b4640 1 --2- 172.21.15.76:0/933607809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc543b70 0x5574bc54c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.050+0000 7ff5820b6640 1 -- 172.21.15.76:0/933607809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc543b70 msgr2=0x5574bc54c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.050+0000 7ff5820b6640 1 --2- 172.21.15.76:0/933607809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc543b70 0x5574bc54c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.050+0000 7ff5820b6640 1 -- 172.21.15.76:0/933607809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543250 msgr2=0x5574bc543630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.050+0000 7ff5820b6640 1 --2- 172.21.15.76:0/933607809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543250 0x5574bc543630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.050+0000 7ff5820b6640 1 -- 172.21.15.76:0/933607809 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574bc46d060 con 0x5574bc54c750 2024-03-12T05:56:45.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.050+0000 7ff5810b4640 1 --2- 172.21.15.76:0/933607809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc543b70 0x5574bc54c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:45.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.050+0000 7ff5818b5640 1 --2- 172.21.15.76:0/933607809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543250 0x5574bc543630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:45.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.050+0000 7ff5820b6640 1 --2- 172.21.15.76:0/933607809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc54c750 0x5574bc54eb40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff574009fc0 tx=0x7ff574011bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b19c20a03be98ef server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.050+0000 7ff567fff640 1 -- 172.21.15.76:0/933607809 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574004d20 con 0x5574bc54c750 2024-03-12T05:56:45.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.050+0000 7ff567fff640 1 -- 172.21.15.76:0/933607809 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff574004e80 con 0x5574bc54c750 2024-03-12T05:56:45.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.050+0000 7ff567fff640 1 -- 172.21.15.76:0/933607809 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5740042b0 con 0x5574bc54c750 2024-03-12T05:56:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.051+0000 7ff5867c5740 1 -- 172.21.15.76:0/933607809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc54c750 msgr2=0x5574bc54eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.051+0000 7ff5867c5740 1 --2- 172.21.15.76:0/933607809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc54c750 0x5574bc54eb40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff574009fc0 tx=0x7ff574011bf0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.051+0000 7ff5867c5740 1 -- 172.21.15.76:0/933607809 shutdown_connections 2024-03-12T05:56:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.051+0000 7ff5867c5740 1 --2- 172.21.15.76:0/933607809 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc54c750 0x5574bc54eb40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.051+0000 7ff5867c5740 1 --2- 172.21.15.76:0/933607809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc543b70 0x5574bc54c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.051+0000 7ff5867c5740 1 --2- 172.21.15.76:0/933607809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543250 0x5574bc543630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.051+0000 7ff5867c5740 1 -- 172.21.15.76:0/933607809 >> 172.21.15.76:0/933607809 conn(0x5574bc53d980 msgr2=0x5574bc53dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.052+0000 7ff5867c5740 1 -- 172.21.15.76:0/933607809 shutdown_connections 2024-03-12T05:56:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.052+0000 7ff5867c5740 1 -- 172.21.15.76:0/933607809 wait complete. 2024-03-12T05:56:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.052+0000 7ff5867c5740 1 -- start start 2024-03-12T05:56:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.052+0000 7ff5867c5740 1 Processor -- start 2024-03-12T05:56:45.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.053+0000 7ff5867c5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543250 0x5574bc549750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.053+0000 7ff5867c5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc543b70 0x5574bc549c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.053+0000 7ff5867c5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc54c750 0x5574bc54a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.053+0000 7ff5867c5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574bc4ab020 con 0x5574bc543250 2024-03-12T05:56:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.053+0000 7ff5818b5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543250 0x5574bc549750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.053+0000 7ff5810b4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc543b70 0x5574bc549c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.053+0000 7ff5818b5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543250 0x5574bc549750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41048/0 (socket says 172.21.15.76:41048) 2024-03-12T05:56:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.053+0000 7ff5818b5640 1 -- 172.21.15.76:0/1791064043 learned_addr learned my addr 172.21.15.76:0/1791064043 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.053+0000 7ff5867c5740 1 -- 172.21.15.76:0/1791064043 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574bc17e430 con 0x5574bc543b70 2024-03-12T05:56:45.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.053+0000 7ff5867c5740 1 -- 172.21.15.76:0/1791064043 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574bc13ae00 con 0x5574bc54c750 2024-03-12T05:56:45.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.053+0000 7ff5867c5740 10 client.0 _ll_register_callbacks cb 0x5574bc452cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:45.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.053+0000 7ff5820b6640 1 --2- 172.21.15.76:0/1791064043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc54c750 0x5574bc54a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.054+0000 7ff5818b5640 1 -- 172.21.15.76:0/1791064043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc54c750 msgr2=0x5574bc54a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.054+0000 7ff5818b5640 1 --2- 172.21.15.76:0/1791064043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc54c750 0x5574bc54a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.054+0000 7ff5818b5640 1 -- 172.21.15.76:0/1791064043 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc543b70 msgr2=0x5574bc549c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.054+0000 7ff5818b5640 1 --2- 172.21.15.76:0/1791064043 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc543b70 0x5574bc549c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.054+0000 7ff5818b5640 1 -- 172.21.15.76:0/1791064043 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff574011440 con 0x5574bc543250 2024-03-12T05:56:45.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.054+0000 7ff5818b5640 1 --2- 172.21.15.76:0/1791064043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543250 0x5574bc549750 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff57801dc10 tx=0x7ff578004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd6db14a8392d6f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.054+0000 7ff5657fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.054+0000 7ff5657fa640 1 -- 172.21.15.76:0/1791064043 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578022070 con 0x5574bc543250 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.054+0000 7ff5867c5740 1 -- 172.21.15.76:0/1791064043 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5574bc1e7720 con 0x5574bc543250 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.054+0000 7ff5657fa640 1 -- 172.21.15.76:0/1791064043 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff57800eb80 con 0x5574bc543250 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.055+0000 7ff5657fa640 1 -- 172.21.15.76:0/1791064043 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff57800ed40 con 0x5574bc543250 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.055+0000 7ff5867c5740 10 client.15795 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.055+0000 7ff5867c5740 1 -- 172.21.15.76:0/1791064043 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5574bc4a96a0 con 0x5574bc543250 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.055+0000 7ff5657fa640 1 -- 172.21.15.76:0/1791064043 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5574bc1e7720 con 0x5574bc543250 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.056+0000 7ff5657fa640 1 -- 172.21.15.76:0/1791064043 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff57802f460 con 0x5574bc543250 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.056+0000 7ff5867c5740 10 client.15795 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.056+0000 7ff5867c5740 10 client.15795 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.056+0000 7ff5867c5740 20 client.15795 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.056+0000 7ff5867c5740 1 --2- 172.21.15.76:0/1791064043 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5574bc55cbb0 0x5574bc55efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.056+0000 7ff5867c5740 4 client.15795 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:45.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.056+0000 7ff5867c5740 1 -- 172.21.15.76:0/1791064043 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5574bc236890 con 0x5574bc55cbb0 2024-03-12T05:56:45.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.056+0000 7ff5810b4640 1 --2- 172.21.15.76:0/1791064043 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5574bc55cbb0 0x5574bc55efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.056+0000 7ff5657fa640 10 client.15795 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:56:45.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.056+0000 7ff5810b4640 1 --2- 172.21.15.76:0/1791064043 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5574bc55cbb0 0x5574bc55efd0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c8f6e54c6ebd2ee server_cookie=9c55baf00f7386ed in_seq=0 out_seq=0 2024-03-12T05:56:45.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.057+0000 7ff5657fa640 1 -- 172.21.15.76:0/1791064043 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5574bc236890 con 0x5574bc55cbb0 2024-03-12T05:56:45.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.057+0000 7ff5657fa640 10 client.15795 handle_command_reply: tid=0 2024-03-12T05:56:45.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.057+0000 7ff5867c5740 1 client.15795 shutdown 2024-03-12T05:56:45.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.057+0000 7ff5867c5740 1 -- 172.21.15.76:0/1791064043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543250 msgr2=0x5574bc549750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.057+0000 7ff5867c5740 1 --2- 172.21.15.76:0/1791064043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543250 0x5574bc549750 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff57801dc10 tx=0x7ff578004730 comp rx=0 tx=0).stop 2024-03-12T05:56:45.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.057+0000 7ff5867c5740 1 -- 172.21.15.76:0/1791064043 shutdown_connections 2024-03-12T05:56:45.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.057+0000 7ff5867c5740 1 --2- 172.21.15.76:0/1791064043 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5574bc55cbb0 0x5574bc55efd0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.057+0000 7ff5867c5740 1 --2- 172.21.15.76:0/1791064043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc54c750 0x5574bc54a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.058+0000 7ff5867c5740 1 --2- 172.21.15.76:0/1791064043 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc543b70 0x5574bc549c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.058+0000 7ff5867c5740 1 --2- 172.21.15.76:0/1791064043 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543250 0x5574bc549750 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.058+0000 7ff5867c5740 1 -- 172.21.15.76:0/1791064043 >> 172.21.15.76:0/1791064043 conn(0x5574bc53d980 msgr2=0x5574bc53f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.058+0000 7ff5657fa640 0 client.15795 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:56:45.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.058+0000 7ff5867c5740 1 -- 172.21.15.76:0/1791064043 shutdown_connections 2024-03-12T05:56:45.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.058+0000 7ff5867c5740 1 -- 172.21.15.76:0/1791064043 wait complete. 2024-03-12T05:56:45.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.058+0000 7ff5867c5740 20 client.15795 trim_cache size 0 max 16384 2024-03-12T05:56:45.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5867c5740 1 Processor -- start 2024-03-12T05:56:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5867c5740 1 -- start start 2024-03-12T05:56:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5867c5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc543250 0x5574bc54be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5867c5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543b70 0x5574bc549ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5867c5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc54c750 0x5574bc5469d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5867c5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574bc430130 con 0x5574bc543b70 2024-03-12T05:56:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5867c5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574bc52fa40 con 0x5574bc54c750 2024-03-12T05:56:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5867c5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574bc53e9d0 con 0x5574bc543250 2024-03-12T05:56:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5810b4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543b70 0x5574bc549ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5810b4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543b70 0x5574bc549ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41062/0 (socket says 172.21.15.76:41062) 2024-03-12T05:56:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5810b4640 1 -- 172.21.15.76:0/2133990135 learned_addr learned my addr 172.21.15.76:0/2133990135 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5820b6640 1 --2- 172.21.15.76:0/2133990135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc54c750 0x5574bc5469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5818b5640 1 --2- 172.21.15.76:0/2133990135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc543250 0x5574bc54be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5810b4640 1 -- 172.21.15.76:0/2133990135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc543250 msgr2=0x5574bc54be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5810b4640 1 --2- 172.21.15.76:0/2133990135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc543250 0x5574bc54be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5810b4640 1 -- 172.21.15.76:0/2133990135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc54c750 msgr2=0x5574bc5469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.076+0000 7ff5810b4640 1 --2- 172.21.15.76:0/2133990135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc54c750 0x5574bc5469d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.077+0000 7ff5810b4640 1 -- 172.21.15.76:0/2133990135 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574bc445c20 con 0x5574bc543b70 2024-03-12T05:56:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.077+0000 7ff5820b6640 1 --2- 172.21.15.76:0/2133990135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc54c750 0x5574bc5469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.077+0000 7ff5810b4640 1 --2- 172.21.15.76:0/2133990135 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543b70 0x5574bc549ff0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff56c02c180 tx=0x7ff56c02bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc5c4ce6ee24809 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.077+0000 7ff5677fe640 1 -- 172.21.15.76:0/2133990135 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c0bb800 con 0x5574bc543b70 2024-03-12T05:56:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.077+0000 7ff5677fe640 1 -- 172.21.15.76:0/2133990135 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff56c0b9fe0 con 0x5574bc543b70 2024-03-12T05:56:45.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.077+0000 7ff5677fe640 1 -- 172.21.15.76:0/2133990135 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c0c0080 con 0x5574bc543b70 2024-03-12T05:56:45.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.078+0000 7ff5867c5740 1 -- 172.21.15.76:0/2133990135 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543b70 msgr2=0x5574bc549ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.078+0000 7ff5867c5740 1 --2- 172.21.15.76:0/2133990135 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543b70 0x5574bc549ff0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff56c02c180 tx=0x7ff56c02bb70 comp rx=0 tx=0).stop 2024-03-12T05:56:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.078+0000 7ff5867c5740 1 -- 172.21.15.76:0/2133990135 shutdown_connections 2024-03-12T05:56:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.078+0000 7ff5867c5740 1 --2- 172.21.15.76:0/2133990135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc54c750 0x5574bc5469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.078+0000 7ff5867c5740 1 --2- 172.21.15.76:0/2133990135 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543b70 0x5574bc549ff0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.078+0000 7ff5867c5740 1 --2- 172.21.15.76:0/2133990135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc543250 0x5574bc54be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.078+0000 7ff5867c5740 1 -- 172.21.15.76:0/2133990135 >> 172.21.15.76:0/2133990135 conn(0x5574bc53d980 msgr2=0x5574bc563930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.078+0000 7ff5867c5740 1 -- 172.21.15.76:0/2133990135 shutdown_connections 2024-03-12T05:56:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.078+0000 7ff5867c5740 1 -- 172.21.15.76:0/2133990135 wait complete. 2024-03-12T05:56:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.078+0000 7ff5867c5740 1 -- start start 2024-03-12T05:56:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.078+0000 7ff5867c5740 1 Processor -- start 2024-03-12T05:56:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5867c5740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc543250 0x5574bc553ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5867c5740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543b70 0x5574bc551e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5867c5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc54c750 0x5574bc5525c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5867c5740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574bc552c90 con 0x5574bc543b70 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5867c5740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574bc5b30c0 con 0x5574bc543250 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5867c5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574bc5b3200 con 0x5574bc54c750 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5867c5740 10 client.0 _ll_register_callbacks cb 0x7ff574009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5818b5640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc543250 0x5574bc553ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5818b5640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc543250 0x5574bc553ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:47068/0 (socket says 172.21.15.76:47068) 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5818b5640 1 -- 172.21.15.76:0/4260748262 learned_addr learned my addr 172.21.15.76:0/4260748262 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5820b6640 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc54c750 0x5574bc5525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5810b4640 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543b70 0x5574bc551e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5818b5640 1 -- 172.21.15.76:0/4260748262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc54c750 msgr2=0x5574bc5525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5818b5640 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc54c750 0x5574bc5525c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5818b5640 1 -- 172.21.15.76:0/4260748262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543b70 msgr2=0x5574bc551e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5818b5640 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543b70 0x5574bc551e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5818b5640 1 -- 172.21.15.76:0/4260748262 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56c09ae70 con 0x5574bc543250 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5820b6640 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc54c750 0x5574bc5525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:45.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5810b4640 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543b70 0x5574bc551e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5818b5640 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc543250 0x5574bc553ae0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff578006010 tx=0x7ff578009a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=593d5d37b271fc53 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.079+0000 7ff5657fa640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.080+0000 7ff5657fa640 1 -- 172.21.15.76:0/4260748262 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578022040 con 0x5574bc543250 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.080+0000 7ff5657fa640 1 -- 172.21.15.76:0/4260748262 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff57800eb80 con 0x5574bc543250 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.080+0000 7ff5867c5740 1 -- 172.21.15.76:0/4260748262 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5574bc1aa7e0 con 0x5574bc543250 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.080+0000 7ff5657fa640 1 -- 172.21.15.76:0/4260748262 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff57800ed40 con 0x5574bc543250 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.080+0000 7ff5867c5740 10 client.25265 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.080+0000 7ff5867c5740 1 -- 172.21.15.76:0/4260748262 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5574bc46b8c0 con 0x5574bc543250 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.080+0000 7ff5657fa640 1 -- 172.21.15.76:0/4260748262 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff578005040 con 0x5574bc543250 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.080+0000 7ff5657fa640 1 -- 172.21.15.76:0/4260748262 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff5780047a0 con 0x5574bc543250 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.080+0000 7ff5867c5740 10 client.25265 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.080+0000 7ff5867c5740 10 client.25265 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.080+0000 7ff5867c5740 20 client.25265 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.080+0000 7ff5867c5740 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5574bc5b4940 0x5574bc5b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.080+0000 7ff5867c5740 4 client.25265 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.080+0000 7ff5867c5740 1 -- 172.21.15.76:0/4260748262 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5574bc1aa7e0 con 0x5574bc5b4940 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.081+0000 7ff5810b4640 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5574bc5b4940 0x5574bc5b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.081+0000 7ff5657fa640 10 client.25265 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.081+0000 7ff5810b4640 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5574bc5b4940 0x5574bc5b6d60 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=113919c132af881b server_cookie=d82ba0264f936cdb in_seq=0 out_seq=0 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.081+0000 7ff5657fa640 1 -- 172.21.15.76:0/4260748262 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5574bc1aa7e0 con 0x5574bc5b4940 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.081+0000 7ff5657fa640 10 client.25265 handle_command_reply: tid=0 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.081+0000 7ff5867c5740 1 client.25265 shutdown 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.081+0000 7ff5867c5740 1 -- 172.21.15.76:0/4260748262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc543250 msgr2=0x5574bc553ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.081+0000 7ff5867c5740 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc543250 0x5574bc553ae0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff578006010 tx=0x7ff578009a90 comp rx=0 tx=0).stop 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.081+0000 7ff5867c5740 1 -- 172.21.15.76:0/4260748262 shutdown_connections 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.081+0000 7ff5867c5740 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5574bc5b4940 0x5574bc5b6d60 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff5867c5740 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574bc54c750 0x5574bc5525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff5657fa640 0 client.25265 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff5867c5740 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5574bc543b70 0x5574bc551e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff5867c5740 1 --2- 172.21.15.76:0/4260748262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5574bc543250 0x5574bc553ae0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff5867c5740 1 -- 172.21.15.76:0/4260748262 >> 172.21.15.76:0/4260748262 conn(0x5574bc53d980 msgr2=0x5574bc53c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff5867c5740 1 -- 172.21.15.76:0/4260748262 shutdown_connections 2024-03-12T05:56:45.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff5867c5740 1 -- 172.21.15.76:0/4260748262 wait complete. 2024-03-12T05:56:45.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff5867c5740 20 client.25265 trim_cache size 0 max 16384 2024-03-12T05:56:45.091 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:56:45.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff583b19640 1 -- 172.21.15.76:0/3909943619 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff5680a1d80 msgr2=0x7ff5680a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff583b19640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff5680a1d80 0x7ff5680a4240 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff578009530 tx=0x7ff578009260 comp rx=0 tx=0).stop 2024-03-12T05:56:45.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff583b19640 1 -- 172.21.15.76:0/3909943619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff57c105330 msgr2=0x7ff57c19d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff583b19640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff57c105330 0x7ff57c19d970 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff56c000c00 tx=0x7ff56c02ca00 comp rx=0 tx=0).stop 2024-03-12T05:56:45.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff583b19640 1 -- 172.21.15.76:0/3909943619 shutdown_connections 2024-03-12T05:56:45.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff583b19640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff5680a1d80 0x7ff5680a4240 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff583b19640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff57c1975e0 0x7ff57c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff583b19640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff57c105330 0x7ff57c19d970 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff583b19640 1 --2- 172.21.15.76:0/3909943619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff57c1004f0 0x7ff57c19d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff583b19640 1 -- 172.21.15.76:0/3909943619 >> 172.21.15.76:0/3909943619 conn(0x7ff57c077ef0 msgr2=0x7ff57c0fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff583b19640 1 -- 172.21.15.76:0/3909943619 shutdown_connections 2024-03-12T05:56:45.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.082+0000 7ff583b19640 1 -- 172.21.15.76:0/3909943619 wait complete. 2024-03-12T05:56:45.098 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:56:45.098 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:56:45.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.204+0000 7fa08acf4640 1 -- 172.21.15.76:0/859741098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa084100c40 msgr2=0x7fa084104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.204+0000 7fa08acf4640 1 --2- 172.21.15.76:0/859741098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa084100c40 0x7fa084104c80 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa070009900 tx=0x7fa07002bc30 comp rx=0 tx=0).stop 2024-03-12T05:56:45.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.204+0000 7fa08acf4640 1 -- 172.21.15.76:0/859741098 shutdown_connections 2024-03-12T05:56:45.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.204+0000 7fa08acf4640 1 --2- 172.21.15.76:0/859741098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa084105360 0x7fa08410cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.204+0000 7fa08acf4640 1 --2- 172.21.15.76:0/859741098 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa084100c40 0x7fa084104c80 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.204+0000 7fa08acf4640 1 --2- 172.21.15.76:0/859741098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa084100290 0x7fa084100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.204+0000 7fa08acf4640 1 -- 172.21.15.76:0/859741098 >> 172.21.15.76:0/859741098 conn(0x7fa084077230 msgr2=0x7fa084077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.204+0000 7fa08acf4640 1 -- 172.21.15.76:0/859741098 shutdown_connections 2024-03-12T05:56:45.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.204+0000 7fa08acf4640 1 -- 172.21.15.76:0/859741098 wait complete. 2024-03-12T05:56:45.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.205+0000 7fa08acf4640 1 Processor -- start 2024-03-12T05:56:45.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.205+0000 7fa08acf4640 1 -- start start 2024-03-12T05:56:45.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.205+0000 7fa08acf4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa084100290 0x7fa08419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.205+0000 7fa08acf4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa084105360 0x7fa08419d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.205+0000 7fa08acf4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa084197650 0x7fa084197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.205+0000 7fa08acf4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa084198040 con 0x7fa084105360 2024-03-12T05:56:45.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.205+0000 7fa08acf4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0841981b0 con 0x7fa084197650 2024-03-12T05:56:45.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.205+0000 7fa08acf4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa084198370 con 0x7fa084100290 2024-03-12T05:56:45.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.205+0000 7fa083fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa084105360 0x7fa08419d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.205+0000 7fa083fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa084105360 0x7fa08419d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41082/0 (socket says 172.21.15.76:41082) 2024-03-12T05:56:45.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.205+0000 7fa083fff640 1 -- 172.21.15.76:0/3346109678 learned_addr learned my addr 172.21.15.76:0/3346109678 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.205+0000 7fa088a90640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa084100290 0x7fa08419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.206+0000 7fa083fff640 1 -- 172.21.15.76:0/3346109678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa084100290 msgr2=0x7fa08419d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.206+0000 7fa083fff640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa084100290 0x7fa08419d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.206+0000 7fa083fff640 1 -- 172.21.15.76:0/3346109678 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa084197650 msgr2=0x7fa084197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.206+0000 7fa089291640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa084197650 0x7fa084197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.206+0000 7fa083fff640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa084197650 0x7fa084197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.206+0000 7fa083fff640 1 -- 172.21.15.76:0/3346109678 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa070009560 con 0x7fa084105360 2024-03-12T05:56:45.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.206+0000 7fa088a90640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa084100290 0x7fa08419d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:45.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.206+0000 7fa089291640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa084197650 0x7fa084197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:56:45.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.206+0000 7fa083fff640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa084105360 0x7fa08419d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa070000c00 tx=0x7fa07002c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.206+0000 7fa081ffb640 1 -- 172.21.15.76:0/3346109678 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0700042d0 con 0x7fa084105360 2024-03-12T05:56:45.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.206+0000 7fa081ffb640 1 -- 172.21.15.76:0/3346109678 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa070004430 con 0x7fa084105360 2024-03-12T05:56:45.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.206+0000 7fa08acf4640 1 -- 172.21.15.76:0/3346109678 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0841a4310 con 0x7fa084105360 2024-03-12T05:56:45.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.206+0000 7fa081ffb640 1 -- 172.21.15.76:0/3346109678 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa070003c30 con 0x7fa084105360 2024-03-12T05:56:45.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.207+0000 7fa08acf4640 1 -- 172.21.15.76:0/3346109678 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0841a4810 con 0x7fa084105360 2024-03-12T05:56:45.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.210+0000 7fa08d99b740 1 Processor -- start 2024-03-12T05:56:45.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.210+0000 7fa08d99b740 1 -- start start 2024-03-12T05:56:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.210+0000 7fa08d99b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931c8f0 0x55cd3931ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.210+0000 7fa08d99b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3931d2a0 0x55cd39309430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.210+0000 7fa08d99b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39309eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.210+0000 7fa08d99b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd392ca960 con 0x55cd39309a50 2024-03-12T05:56:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.210+0000 7fa08d99b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd3927e310 con 0x55cd3931c8f0 2024-03-12T05:56:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.210+0000 7fa08d99b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd3927b840 con 0x55cd3931d2a0 2024-03-12T05:56:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.210+0000 7fa081ffb640 1 -- 172.21.15.76:0/3346109678 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fa07003e450 con 0x7fa084105360 2024-03-12T05:56:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.210+0000 7fa089291640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931c8f0 0x55cd3931ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.211+0000 7fa089291640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931c8f0 0x55cd3931ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:47082/0 (socket says 172.21.15.76:47082) 2024-03-12T05:56:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.211+0000 7fa089291640 1 -- 172.21.15.76:0/3721578757 learned_addr learned my addr 172.21.15.76:0/3721578757 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.211+0000 7fa088a90640 1 --2- 172.21.15.76:0/3721578757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3931d2a0 0x55cd39309430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.211+0000 7fa083fff640 1 --2- 172.21.15.76:0/3721578757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39309eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.211+0000 7fa089291640 1 -- 172.21.15.76:0/3721578757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3931d2a0 msgr2=0x55cd39309430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.211+0000 7fa089291640 1 --2- 172.21.15.76:0/3721578757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3931d2a0 0x55cd39309430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.211+0000 7fa089291640 1 -- 172.21.15.76:0/3721578757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 msgr2=0x55cd39309eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.211+0000 7fa089291640 1 --2- 172.21.15.76:0/3721578757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39309eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.211+0000 7fa089291640 1 -- 172.21.15.76:0/3721578757 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd39239060 con 0x55cd3931c8f0 2024-03-12T05:56:45.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.211+0000 7fa083fff640 1 --2- 172.21.15.76:0/3721578757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39309eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:56:45.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.211+0000 7fa081ffb640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa0740a1d40 0x7fa0740a4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.211+0000 7fa089291640 1 --2- 172.21.15.76:0/3721578757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931c8f0 0x55cd3931ccd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009f00 tx=0x7fa07c00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce3c04fb53e9873c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.211+0000 7fa088a90640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa0740a1d40 0x7fa0740a4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.211+0000 7fa081ffb640 1 -- 172.21.15.76:0/3346109678 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa0700eff20 con 0x7fa084105360 2024-03-12T05:56:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.212+0000 7fa06effd640 1 -- 172.21.15.76:0/3721578757 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07c003b70 con 0x55cd3931c8f0 2024-03-12T05:56:45.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.212+0000 7fa06effd640 1 -- 172.21.15.76:0/3721578757 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa07c003cd0 con 0x55cd3931c8f0 2024-03-12T05:56:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.212+0000 7fa06effd640 1 -- 172.21.15.76:0/3721578757 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07c011900 con 0x55cd3931c8f0 2024-03-12T05:56:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.212+0000 7fa08d99b740 1 -- 172.21.15.76:0/3721578757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931c8f0 msgr2=0x55cd3931ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.212+0000 7fa08d99b740 1 --2- 172.21.15.76:0/3721578757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931c8f0 0x55cd3931ccd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009f00 tx=0x7fa07c00f730 comp rx=0 tx=0).stop 2024-03-12T05:56:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.212+0000 7fa088a90640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa0740a1d40 0x7fa0740a4200 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x55cd393180d0 tx=0x7fa078005f90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.212+0000 7fa08d99b740 1 -- 172.21.15.76:0/3721578757 shutdown_connections 2024-03-12T05:56:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.212+0000 7fa08d99b740 1 --2- 172.21.15.76:0/3721578757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39309eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.212+0000 7fa08d99b740 1 --2- 172.21.15.76:0/3721578757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3931d2a0 0x55cd39309430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.212+0000 7fa08d99b740 1 --2- 172.21.15.76:0/3721578757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931c8f0 0x55cd3931ccd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.212+0000 7fa08d99b740 1 -- 172.21.15.76:0/3721578757 >> 172.21.15.76:0/3721578757 conn(0x55cd3930bef0 msgr2=0x55cd3930c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.212+0000 7fa08d99b740 1 -- 172.21.15.76:0/3721578757 shutdown_connections 2024-03-12T05:56:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.212+0000 7fa08d99b740 1 -- 172.21.15.76:0/3721578757 wait complete. 2024-03-12T05:56:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.213+0000 7fa08d99b740 1 -- start start 2024-03-12T05:56:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.213+0000 7fa08d99b740 1 Processor -- start 2024-03-12T05:56:45.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.213+0000 7fa08d99b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd39309a50 0x55cd39320920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.213+0000 7fa08d99b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd3931c8f0 0x55cd39320e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.213+0000 7fa08d99b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931d2a0 0x55cd393213a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.213+0000 7fa08d99b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd39277020 con 0x55cd3931c8f0 2024-03-12T05:56:45.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.213+0000 7fa08d99b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd38f4a430 con 0x55cd3931d2a0 2024-03-12T05:56:45.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.213+0000 7fa08d99b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd38f06e00 con 0x55cd39309a50 2024-03-12T05:56:45.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.213+0000 7fa08d99b740 10 client.0 _ll_register_callbacks cb 0x55cd3921ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:45.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa083fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd3931c8f0 0x55cd39320e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa089291640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931d2a0 0x55cd393213a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa089291640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931d2a0 0x55cd393213a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:47088/0 (socket says 172.21.15.76:47088) 2024-03-12T05:56:45.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa089291640 1 -- 172.21.15.76:0/964937342 learned_addr learned my addr 172.21.15.76:0/964937342 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa088a90640 1 --2- 172.21.15.76:0/964937342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd39309a50 0x55cd39320920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa089291640 1 -- 172.21.15.76:0/964937342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd39309a50 msgr2=0x55cd39320920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa089291640 1 --2- 172.21.15.76:0/964937342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd39309a50 0x55cd39320920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa089291640 1 -- 172.21.15.76:0/964937342 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd3931c8f0 msgr2=0x55cd39320e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa089291640 1 --2- 172.21.15.76:0/964937342 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd3931c8f0 0x55cd39320e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa089291640 1 -- 172.21.15.76:0/964937342 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa07c009bb0 con 0x55cd3931d2a0 2024-03-12T05:56:45.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa089291640 1 --2- 172.21.15.76:0/964937342 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931d2a0 0x55cd393213a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009ed0 tx=0x7fa07c003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d45181c694fb8127 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa063fff640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T05:56:45.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa063fff640 1 -- 172.21.15.76:0/964937342 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07c021070 con 0x55cd3931d2a0 2024-03-12T05:56:45.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa08d99b740 1 -- 172.21.15.76:0/964937342 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd38f767e0 con 0x55cd3931d2a0 2024-03-12T05:56:45.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa063fff640 1 -- 172.21.15.76:0/964937342 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa07c004c50 con 0x55cd3931d2a0 2024-03-12T05:56:45.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa063fff640 1 -- 172.21.15.76:0/964937342 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa07c004e10 con 0x55cd3931d2a0 2024-03-12T05:56:45.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa08d99b740 10 client.25274 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:45.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.214+0000 7fa08d99b740 1 -- 172.21.15.76:0/964937342 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55cd392756a0 con 0x55cd3931d2a0 2024-03-12T05:56:45.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.215+0000 7fa063fff640 1 -- 172.21.15.76:0/964937342 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55cd38f767e0 con 0x55cd3931d2a0 2024-03-12T05:56:45.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.215+0000 7fa063fff640 1 -- 172.21.15.76:0/964937342 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa07c011a90 con 0x55cd3931d2a0 2024-03-12T05:56:45.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.215+0000 7fa08d99b740 10 client.25274 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:45.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.215+0000 7fa08d99b740 10 client.25274 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:56:45.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.215+0000 7fa08d99b740 20 client.25274 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:45.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.215+0000 7fa08d99b740 1 --2- 172.21.15.76:0/964937342 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cd3930ec50 0x55cd39311040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.215+0000 7fa08d99b740 4 client.25274 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:45.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.215+0000 7fa08d99b740 1 -- 172.21.15.76:0/964937342 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd38fb3720 con 0x55cd3930ec50 2024-03-12T05:56:45.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.215+0000 7fa088a90640 1 --2- 172.21.15.76:0/964937342 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cd3930ec50 0x55cd39311040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.216+0000 7fa088a90640 1 --2- 172.21.15.76:0/964937342 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cd3930ec50 0x55cd39311040 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fb1c8e46b313493d server_cookie=41434f23cef5544d in_seq=0 out_seq=0 2024-03-12T05:56:45.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.216+0000 7fa063fff640 10 client.25274 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:56:45.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.216+0000 7fa063fff640 1 -- 172.21.15.76:0/964937342 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cd38fb3720 con 0x55cd3930ec50 2024-03-12T05:56:45.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.216+0000 7fa063fff640 10 client.25274 handle_command_reply: tid=0 2024-03-12T05:56:45.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.216+0000 7fa08d99b740 1 client.25274 shutdown 2024-03-12T05:56:45.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.216+0000 7fa08d99b740 1 -- 172.21.15.76:0/964937342 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931d2a0 msgr2=0x55cd393213a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.216+0000 7fa08d99b740 1 --2- 172.21.15.76:0/964937342 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931d2a0 0x55cd393213a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009ed0 tx=0x7fa07c003ba0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.216+0000 7fa08d99b740 1 -- 172.21.15.76:0/964937342 shutdown_connections 2024-03-12T05:56:45.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.216+0000 7fa08d99b740 1 --2- 172.21.15.76:0/964937342 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cd3930ec50 0x55cd39311040 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.216+0000 7fa063fff640 0 client.25274 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:56:45.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.216+0000 7fa08d99b740 1 --2- 172.21.15.76:0/964937342 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931d2a0 0x55cd393213a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.216+0000 7fa08d99b740 1 --2- 172.21.15.76:0/964937342 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd3931c8f0 0x55cd39320e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.216+0000 7fa08d99b740 1 --2- 172.21.15.76:0/964937342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd39309a50 0x55cd39320920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.217+0000 7fa08d99b740 1 -- 172.21.15.76:0/964937342 >> 172.21.15.76:0/964937342 conn(0x55cd3930bef0 msgr2=0x55cd3930adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.217+0000 7fa08d99b740 1 -- 172.21.15.76:0/964937342 shutdown_connections 2024-03-12T05:56:45.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.217+0000 7fa08d99b740 1 -- 172.21.15.76:0/964937342 wait complete. 2024-03-12T05:56:45.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.217+0000 7fa08d99b740 20 client.25274 trim_cache size 0 max 16384 2024-03-12T05:56:45.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.240+0000 7fa08d99b740 1 Processor -- start 2024-03-12T05:56:45.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.240+0000 7fa08d99b740 1 -- start start 2024-03-12T05:56:45.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.240+0000 7fa08d99b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39323050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa08d99b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3930ec50 0x55cd3931c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa08d99b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3931c8f0 0x55cd39320e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa08d99b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd391fc130 con 0x55cd39309a50 2024-03-12T05:56:45.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa088a90640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39323050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa088a90640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39323050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41106/0 (socket says 172.21.15.76:41106) 2024-03-12T05:56:45.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa088a90640 1 -- 172.21.15.76:0/906028421 learned_addr learned my addr 172.21.15.76:0/906028421 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa08d99b740 1 -- 172.21.15.76:0/906028421 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd39323c10 con 0x55cd3930ec50 2024-03-12T05:56:45.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa08d99b740 1 -- 172.21.15.76:0/906028421 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd38f767e0 con 0x55cd3931c8f0 2024-03-12T05:56:45.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa083fff640 1 --2- 172.21.15.76:0/906028421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3930ec50 0x55cd3931c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa089291640 1 --2- 172.21.15.76:0/906028421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3931c8f0 0x55cd39320e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa083fff640 1 -- 172.21.15.76:0/906028421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3931c8f0 msgr2=0x55cd39320e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa083fff640 1 --2- 172.21.15.76:0/906028421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3931c8f0 0x55cd39320e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa083fff640 1 -- 172.21.15.76:0/906028421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 msgr2=0x55cd39323050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa083fff640 1 --2- 172.21.15.76:0/906028421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39323050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa083fff640 1 -- 172.21.15.76:0/906028421 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd39211c20 con 0x55cd3930ec50 2024-03-12T05:56:45.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.241+0000 7fa089291640 1 --2- 172.21.15.76:0/906028421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3931c8f0 0x55cd39320e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:45.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.242+0000 7fa083fff640 1 --2- 172.21.15.76:0/906028421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3930ec50 0x55cd3931c0c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa0700ed630 tx=0x7fa0700bc930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd766c6c9dba922a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.242+0000 7fa06e7fc640 1 -- 172.21.15.76:0/906028421 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0700d4070 con 0x55cd3930ec50 2024-03-12T05:56:45.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.242+0000 7fa06e7fc640 1 -- 172.21.15.76:0/906028421 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0700b8d00 con 0x55cd3930ec50 2024-03-12T05:56:45.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.242+0000 7fa06e7fc640 1 -- 172.21.15.76:0/906028421 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa070003e20 con 0x55cd3930ec50 2024-03-12T05:56:45.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.243+0000 7fa08d99b740 1 -- 172.21.15.76:0/906028421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3930ec50 msgr2=0x55cd3931c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.243+0000 7fa08d99b740 1 --2- 172.21.15.76:0/906028421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3930ec50 0x55cd3931c0c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa0700ed630 tx=0x7fa0700bc930 comp rx=0 tx=0).stop 2024-03-12T05:56:45.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.243+0000 7fa08d99b740 1 -- 172.21.15.76:0/906028421 shutdown_connections 2024-03-12T05:56:45.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.243+0000 7fa08d99b740 1 --2- 172.21.15.76:0/906028421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3931c8f0 0x55cd39320e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.243+0000 7fa08d99b740 1 --2- 172.21.15.76:0/906028421 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3930ec50 0x55cd3931c0c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.243+0000 7fa08d99b740 1 --2- 172.21.15.76:0/906028421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39323050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.243+0000 7fa08d99b740 1 -- 172.21.15.76:0/906028421 >> 172.21.15.76:0/906028421 conn(0x55cd3930bef0 msgr2=0x55cd3931b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.243+0000 7fa08d99b740 1 -- 172.21.15.76:0/906028421 shutdown_connections 2024-03-12T05:56:45.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.243+0000 7fa08d99b740 1 -- 172.21.15.76:0/906028421 wait complete. 2024-03-12T05:56:45.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.243+0000 7fa08d99b740 1 -- start start 2024-03-12T05:56:45.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.243+0000 7fa08d99b740 1 Processor -- start 2024-03-12T05:56:45.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.243+0000 7fa08d99b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39333a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa08d99b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3930ec50 0x55cd39333f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa088a90640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39333a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa088a90640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39333a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41122/0 (socket says 172.21.15.76:41122) 2024-03-12T05:56:45.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa088a90640 1 -- 172.21.15.76:0/167878827 learned_addr learned my addr 172.21.15.76:0/167878827 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa08d99b740 1 --2- 172.21.15.76:0/167878827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931c8f0 0x55cd39331fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa08d99b740 1 -- 172.21.15.76:0/167878827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd39332620 con 0x55cd39309a50 2024-03-12T05:56:45.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa08d99b740 1 -- 172.21.15.76:0/167878827 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd39332790 con 0x55cd3931c8f0 2024-03-12T05:56:45.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa08d99b740 1 -- 172.21.15.76:0/167878827 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd39332950 con 0x55cd3930ec50 2024-03-12T05:56:45.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa08d99b740 10 client.0 _ll_register_callbacks cb 0x7fa08406aa10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:45.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa083fff640 1 --2- 172.21.15.76:0/167878827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3930ec50 0x55cd39333f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa089291640 1 --2- 172.21.15.76:0/167878827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931c8f0 0x55cd39331fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa088a90640 1 -- 172.21.15.76:0/167878827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3930ec50 msgr2=0x55cd39333f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa088a90640 1 --2- 172.21.15.76:0/167878827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3930ec50 0x55cd39333f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa088a90640 1 -- 172.21.15.76:0/167878827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931c8f0 msgr2=0x55cd39331fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa088a90640 1 --2- 172.21.15.76:0/167878827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931c8f0 0x55cd39331fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.244+0000 7fa088a90640 1 -- 172.21.15.76:0/167878827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd392756a0 con 0x55cd39309a50 2024-03-12T05:56:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa088a90640 1 --2- 172.21.15.76:0/167878827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39333a10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa078004560 tx=0x7fa078004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd22c26415a90f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa063fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa063fff640 1 -- 172.21.15.76:0/167878827 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa078036040 con 0x55cd39309a50 2024-03-12T05:56:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa063fff640 1 -- 172.21.15.76:0/167878827 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa07803b3d0 con 0x55cd39309a50 2024-03-12T05:56:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa08d99b740 1 -- 172.21.15.76:0/167878827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd3930c870 con 0x55cd39309a50 2024-03-12T05:56:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa063fff640 1 -- 172.21.15.76:0/167878827 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa078017070 con 0x55cd39309a50 2024-03-12T05:56:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa08d99b740 10 client.15813 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:45.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa08d99b740 1 -- 172.21.15.76:0/167878827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7fa0841a4810 con 0x55cd39309a50 2024-03-12T05:56:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa063fff640 1 -- 172.21.15.76:0/167878827 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa07800f560 con 0x55cd39309a50 2024-03-12T05:56:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa063fff640 1 -- 172.21.15.76:0/167878827 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa07804c450 con 0x55cd39309a50 2024-03-12T05:56:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa08d99b740 10 client.15813 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa08d99b740 10 client.15813 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:56:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa08d99b740 20 client.15813 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa08d99b740 1 --2- 172.21.15.76:0/167878827 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cd3931d2a0 0x55cd39382710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa08d99b740 4 client.15813 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:56:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.245+0000 7fa08d99b740 1 -- 172.21.15.76:0/167878827 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cd3930c870 con 0x55cd3931d2a0 2024-03-12T05:56:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.246+0000 7fa083fff640 1 --2- 172.21.15.76:0/167878827 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cd3931d2a0 0x55cd39382710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.246+0000 7fa063fff640 10 client.15813 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:56:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.246+0000 7fa083fff640 1 --2- 172.21.15.76:0/167878827 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cd3931d2a0 0x55cd39382710 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=89a1731ce695761a server_cookie=c0b49ae59f7b8cff in_seq=0 out_seq=0 2024-03-12T05:56:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.246+0000 7fa063fff640 1 -- 172.21.15.76:0/167878827 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cd3930c870 con 0x55cd3931d2a0 2024-03-12T05:56:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.246+0000 7fa063fff640 10 client.15813 handle_command_reply: tid=0 2024-03-12T05:56:45.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.246+0000 7fa08d99b740 1 client.15813 shutdown 2024-03-12T05:56:45.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.246+0000 7fa08d99b740 1 -- 172.21.15.76:0/167878827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 msgr2=0x55cd39333a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.246+0000 7fa08d99b740 1 --2- 172.21.15.76:0/167878827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39333a10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa078004560 tx=0x7fa078004730 comp rx=0 tx=0).stop 2024-03-12T05:56:45.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.247+0000 7fa08d99b740 1 -- 172.21.15.76:0/167878827 shutdown_connections 2024-03-12T05:56:45.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.247+0000 7fa08d99b740 1 --2- 172.21.15.76:0/167878827 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cd3931d2a0 0x55cd39382710 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.247+0000 7fa063fff640 0 client.15813 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:56:45.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.247+0000 7fa08d99b740 1 --2- 172.21.15.76:0/167878827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cd3931c8f0 0x55cd39331fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.247+0000 7fa08d99b740 1 --2- 172.21.15.76:0/167878827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cd3930ec50 0x55cd39333f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.247+0000 7fa08d99b740 1 --2- 172.21.15.76:0/167878827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cd39309a50 0x55cd39333a10 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.247+0000 7fa08d99b740 1 -- 172.21.15.76:0/167878827 >> 172.21.15.76:0/167878827 conn(0x55cd3930bef0 msgr2=0x55cd393082f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.247+0000 7fa08d99b740 1 -- 172.21.15.76:0/167878827 shutdown_connections 2024-03-12T05:56:45.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.247+0000 7fa08d99b740 1 -- 172.21.15.76:0/167878827 wait complete. 2024-03-12T05:56:45.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.247+0000 7fa08d99b740 20 client.15813 trim_cache size 0 max 16384 2024-03-12T05:56:45.257 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:56:45.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.247+0000 7fa08acf4640 1 -- 172.21.15.76:0/3346109678 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa0740a1d40 msgr2=0x7fa0740a4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.247+0000 7fa08acf4640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa0740a1d40 0x7fa0740a4200 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x55cd393180d0 tx=0x7fa078005f90 comp rx=0 tx=0).stop 2024-03-12T05:56:45.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.247+0000 7fa08acf4640 1 -- 172.21.15.76:0/3346109678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa084105360 msgr2=0x7fa08419d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.247+0000 7fa08acf4640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa084105360 0x7fa08419d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa070000c00 tx=0x7fa07002c5f0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.248+0000 7fa08acf4640 1 -- 172.21.15.76:0/3346109678 shutdown_connections 2024-03-12T05:56:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.248+0000 7fa08acf4640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa0740a1d40 0x7fa0740a4200 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.248+0000 7fa08acf4640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa084197650 0x7fa084197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.248+0000 7fa08acf4640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa084105360 0x7fa08419d990 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.248+0000 7fa08acf4640 1 --2- 172.21.15.76:0/3346109678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa084100290 0x7fa08419d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.248+0000 7fa08acf4640 1 -- 172.21.15.76:0/3346109678 >> 172.21.15.76:0/3346109678 conn(0x7fa084077230 msgr2=0x7fa0840fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.248+0000 7fa08acf4640 1 -- 172.21.15.76:0/3346109678 shutdown_connections 2024-03-12T05:56:45.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.248+0000 7fa08acf4640 1 -- 172.21.15.76:0/3346109678 wait complete. 2024-03-12T05:56:45.263 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:56:45.264 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:56:45.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.368+0000 7f5a0fc40640 1 -- 172.21.15.76:0/2525047658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08105360 msgr2=0x7f5a0810cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.368+0000 7f5a0fc40640 1 --2- 172.21.15.76:0/2525047658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08105360 0x7f5a0810cf30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009c10 tx=0x7f5a0002bef0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.368+0000 7f5a0fc40640 1 -- 172.21.15.76:0/2525047658 shutdown_connections 2024-03-12T05:56:45.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.368+0000 7f5a0fc40640 1 --2- 172.21.15.76:0/2525047658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08105360 0x7f5a0810cf30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.368+0000 7f5a0fc40640 1 --2- 172.21.15.76:0/2525047658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a08100c40 0x7f5a08104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.368+0000 7f5a0fc40640 1 --2- 172.21.15.76:0/2525047658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a08100290 0x7f5a08100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.368+0000 7f5a0fc40640 1 -- 172.21.15.76:0/2525047658 >> 172.21.15.76:0/2525047658 conn(0x7f5a08077230 msgr2=0x7f5a08077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.368+0000 7f5a0fc40640 1 -- 172.21.15.76:0/2525047658 shutdown_connections 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.368+0000 7f5a0fc40640 1 -- 172.21.15.76:0/2525047658 wait complete. 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.369+0000 7f5a0fc40640 1 Processor -- start 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.369+0000 7f5a0fc40640 1 -- start start 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.369+0000 7f5a0fc40640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08100290 0x7f5a08197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.369+0000 7f5a0fc40640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a08105360 0x7f5a08197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.369+0000 7f5a0fc40640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a0819bde0 0x7f5a0819c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.369+0000 7f5a0fc40640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0819c7d0 con 0x7f5a0819bde0 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.369+0000 7f5a0fc40640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0819c940 con 0x7f5a08100290 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.369+0000 7f5a0fc40640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0819cab0 con 0x7f5a08105360 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.369+0000 7f5a0e1dd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a0819bde0 0x7f5a0819c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.369+0000 7f5a0d9dc640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08100290 0x7f5a08197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.370+0000 7f5a0e1dd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a0819bde0 0x7f5a0819c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41150/0 (socket says 172.21.15.76:41150) 2024-03-12T05:56:45.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.370+0000 7f5a0e1dd640 1 -- 172.21.15.76:0/4110472908 learned_addr learned my addr 172.21.15.76:0/4110472908 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.370+0000 7f5a0d1db640 1 --2- 172.21.15.76:0/4110472908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a08105360 0x7f5a08197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.370+0000 7f5a0e1dd640 1 -- 172.21.15.76:0/4110472908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a08105360 msgr2=0x7f5a08197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.370+0000 7f5a0e1dd640 1 --2- 172.21.15.76:0/4110472908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a08105360 0x7f5a08197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.370+0000 7f5a0e1dd640 1 -- 172.21.15.76:0/4110472908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08100290 msgr2=0x7f5a08197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.370+0000 7f5a0e1dd640 1 --2- 172.21.15.76:0/4110472908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08100290 0x7f5a08197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.370+0000 7f5a0e1dd640 1 -- 172.21.15.76:0/4110472908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f8009560 con 0x7f5a0819bde0 2024-03-12T05:56:45.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.370+0000 7f5a0d9dc640 1 --2- 172.21.15.76:0/4110472908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08100290 0x7f5a08197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:45.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.370+0000 7f5a0e1dd640 1 --2- 172.21.15.76:0/4110472908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a0819bde0 0x7f5a0819c290 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5a0002c430 tx=0x7f5a00033610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.370+0000 7f59feffd640 1 -- 172.21.15.76:0/4110472908 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a00003d80 con 0x7f5a0819bde0 2024-03-12T05:56:45.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.370+0000 7f5a0fc40640 1 -- 172.21.15.76:0/4110472908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a00009840 con 0x7f5a0819bde0 2024-03-12T05:56:45.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.370+0000 7f5a0fc40640 1 -- 172.21.15.76:0/4110472908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a080fe080 con 0x7f5a0819bde0 2024-03-12T05:56:45.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.371+0000 7f59feffd640 1 -- 172.21.15.76:0/4110472908 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a00004520 con 0x7f5a0819bde0 2024-03-12T05:56:45.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.371+0000 7f59feffd640 1 -- 172.21.15.76:0/4110472908 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0003da30 con 0x7f5a0819bde0 2024-03-12T05:56:45.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.373+0000 7f59feffd640 1 -- 172.21.15.76:0/4110472908 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f5a00004690 con 0x7f5a0819bde0 2024-03-12T05:56:45.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.374+0000 7f59feffd640 1 --2- 172.21.15.76:0/4110472908 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f59f40a1c20 0x7f59f40a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.374+0000 7f59feffd640 1 -- 172.21.15.76:0/4110472908 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5a000efff0 con 0x7f5a0819bde0 2024-03-12T05:56:45.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.374+0000 7f5a0d9dc640 1 --2- 172.21.15.76:0/4110472908 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f59f40a1c20 0x7f59f40a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.374+0000 7f5a1280d740 1 Processor -- start 2024-03-12T05:56:45.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.374+0000 7f5a1280d740 1 -- start start 2024-03-12T05:56:45.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.374+0000 7f5a0d9dc640 1 --2- 172.21.15.76:0/4110472908 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f59f40a1c20 0x7f59f40a40e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5a04005fd0 tx=0x7f5a0400a400 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a1280d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256250 0x5564e0256630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a1280d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256b70 0x5564e025f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a1280d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 0x5564e0261b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a1280d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e0211960 con 0x5564e0256250 2024-03-12T05:56:45.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a1280d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e01c5310 con 0x5564e025f750 2024-03-12T05:56:45.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a1280d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e01c2840 con 0x5564e0256b70 2024-03-12T05:56:45.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a0d9dc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256250 0x5564e0256630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a0d9dc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256250 0x5564e0256630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41162/0 (socket says 172.21.15.76:41162) 2024-03-12T05:56:45.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a0d9dc640 1 -- 172.21.15.76:0/3977527524 learned_addr learned my addr 172.21.15.76:0/3977527524 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a0e1dd640 1 --2- 172.21.15.76:0/3977527524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 0x5564e0261b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a0d1db640 1 --2- 172.21.15.76:0/3977527524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256b70 0x5564e025f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a0d9dc640 1 -- 172.21.15.76:0/3977527524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256b70 msgr2=0x5564e025f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a0d9dc640 1 --2- 172.21.15.76:0/3977527524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256b70 0x5564e025f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a0d9dc640 1 -- 172.21.15.76:0/3977527524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 msgr2=0x5564e0261b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a0d9dc640 1 --2- 172.21.15.76:0/3977527524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 0x5564e0261b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a0d9dc640 1 -- 172.21.15.76:0/3977527524 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564e0180060 con 0x5564e0256250 2024-03-12T05:56:45.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.375+0000 7f5a0d9dc640 1 --2- 172.21.15.76:0/3977527524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256250 0x5564e0256630 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f5a04010890 tx=0x7f5a04012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b90790aa5cd3e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.376+0000 7f59f3fff640 1 -- 172.21.15.76:0/3977527524 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0401bd40 con 0x5564e0256250 2024-03-12T05:56:45.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.376+0000 7f59f3fff640 1 -- 172.21.15.76:0/3977527524 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a04020070 con 0x5564e0256250 2024-03-12T05:56:45.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.376+0000 7f59f3fff640 1 -- 172.21.15.76:0/3977527524 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a04003dd0 con 0x5564e0256250 2024-03-12T05:56:45.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.377+0000 7f5a1280d740 1 -- 172.21.15.76:0/3977527524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256250 msgr2=0x5564e0256630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.377+0000 7f5a1280d740 1 --2- 172.21.15.76:0/3977527524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256250 0x5564e0256630 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f5a04010890 tx=0x7f5a04012000 comp rx=0 tx=0).stop 2024-03-12T05:56:45.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.377+0000 7f5a1280d740 1 -- 172.21.15.76:0/3977527524 shutdown_connections 2024-03-12T05:56:45.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.377+0000 7f5a1280d740 1 --2- 172.21.15.76:0/3977527524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 0x5564e0261b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.377+0000 7f5a1280d740 1 --2- 172.21.15.76:0/3977527524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256b70 0x5564e025f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.377+0000 7f5a1280d740 1 --2- 172.21.15.76:0/3977527524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256250 0x5564e0256630 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.377+0000 7f5a1280d740 1 -- 172.21.15.76:0/3977527524 >> 172.21.15.76:0/3977527524 conn(0x5564e0250980 msgr2=0x5564e0250d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.377+0000 7f5a1280d740 1 -- 172.21.15.76:0/3977527524 shutdown_connections 2024-03-12T05:56:45.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.377+0000 7f5a1280d740 1 -- 172.21.15.76:0/3977527524 wait complete. 2024-03-12T05:56:45.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.377+0000 7f5a1280d740 1 -- start start 2024-03-12T05:56:45.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.377+0000 7f5a1280d740 1 Processor -- start 2024-03-12T05:56:45.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.378+0000 7f5a1280d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256250 0x5564e025c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.378+0000 7f5a1280d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256b70 0x5564e025cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.378+0000 7f5a1280d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 0x5564e025d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.378+0000 7f5a1280d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e01be020 con 0x5564e0256b70 2024-03-12T05:56:45.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.378+0000 7f5a1280d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564dfe91430 con 0x5564e025f750 2024-03-12T05:56:45.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.378+0000 7f5a1280d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564dfe4de00 con 0x5564e0256250 2024-03-12T05:56:45.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.378+0000 7f5a1280d740 10 client.0 _ll_register_callbacks cb 0x5564e0165cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:45.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.378+0000 7f5a0d1db640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256b70 0x5564e025cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.378+0000 7f5a0d1db640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256b70 0x5564e025cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41168/0 (socket says 172.21.15.76:41168) 2024-03-12T05:56:45.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.378+0000 7f5a0d1db640 1 -- 172.21.15.76:0/2720411201 learned_addr learned my addr 172.21.15.76:0/2720411201 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.378+0000 7f5a0d9dc640 1 --2- 172.21.15.76:0/2720411201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256250 0x5564e025c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.378+0000 7f5a0e1dd640 1 --2- 172.21.15.76:0/2720411201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 0x5564e025d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f5a0d9dc640 1 -- 172.21.15.76:0/2720411201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 msgr2=0x5564e025d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f5a0d9dc640 1 --2- 172.21.15.76:0/2720411201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 0x5564e025d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f5a0d9dc640 1 -- 172.21.15.76:0/2720411201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256b70 msgr2=0x5564e025cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f5a0d9dc640 1 --2- 172.21.15.76:0/2720411201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256b70 0x5564e025cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f5a0d9dc640 1 -- 172.21.15.76:0/2720411201 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a04010570 con 0x5564e0256250 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f5a0d1db640 1 --2- 172.21.15.76:0/2720411201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256b70 0x5564e025cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f5a0d9dc640 1 --2- 172.21.15.76:0/2720411201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256250 0x5564e025c760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5a0400aea0 tx=0x7f5a04005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f129206e417fb4f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f59f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f59f17fa640 1 -- 172.21.15.76:0/2720411201 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0402a040 con 0x5564e0256250 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f5a1280d740 1 -- 172.21.15.76:0/2720411201 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5564dfebd7e0 con 0x5564e0256250 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f59f17fa640 1 -- 172.21.15.76:0/2720411201 <== mon.1 v2:172.21.15.88:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5a04022070 con 0x5564e0256250 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f59f17fa640 1 -- 172.21.15.76:0/2720411201 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a04003770 con 0x5564e0256250 2024-03-12T05:56:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f5a1280d740 10 client.25240 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:45.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f5a1280d740 1 -- 172.21.15.76:0/2720411201 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5564e01bc6a0 con 0x5564e0256250 2024-03-12T05:56:45.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.379+0000 7f59f17fa640 1 -- 172.21.15.76:0/2720411201 <== mon.1 v2:172.21.15.88:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5564dfebd7e0 con 0x5564e0256250 2024-03-12T05:56:45.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.380+0000 7f59f17fa640 1 -- 172.21.15.76:0/2720411201 <== mon.1 v2:172.21.15.88:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5a04018040 con 0x5564e0256250 2024-03-12T05:56:45.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.380+0000 7f5a1280d740 10 client.25240 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:45.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.380+0000 7f5a1280d740 10 client.25240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:45.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.380+0000 7f5a1280d740 20 client.25240 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:45.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.380+0000 7f5a1280d740 1 --2- 172.21.15.76:0/2720411201 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5564e026faf0 0x5564e0271f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.380+0000 7f5a1280d740 4 client.25240 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:45.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.380+0000 7f5a1280d740 1 -- 172.21.15.76:0/2720411201 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5564dfefa720 con 0x5564e026faf0 2024-03-12T05:56:45.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.380+0000 7f5a0d1db640 1 --2- 172.21.15.76:0/2720411201 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5564e026faf0 0x5564e0271f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.380+0000 7f59f17fa640 10 client.25240 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:45.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.381+0000 7f5a0d1db640 1 --2- 172.21.15.76:0/2720411201 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5564e026faf0 0x5564e0271f10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=344524867dae1b4b server_cookie=8694db7f135bc54d in_seq=0 out_seq=0 2024-03-12T05:56:45.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.381+0000 7f59f17fa640 1 -- 172.21.15.76:0/2720411201 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5564dfefa720 con 0x5564e026faf0 2024-03-12T05:56:45.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.381+0000 7f59f17fa640 10 client.25240 handle_command_reply: tid=0 2024-03-12T05:56:45.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.381+0000 7f5a1280d740 1 client.25240 shutdown 2024-03-12T05:56:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.381+0000 7f5a1280d740 1 -- 172.21.15.76:0/2720411201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256250 msgr2=0x5564e025c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.381+0000 7f5a1280d740 1 --2- 172.21.15.76:0/2720411201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256250 0x5564e025c760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5a0400aea0 tx=0x7f5a04005eb0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.381+0000 7f5a1280d740 1 -- 172.21.15.76:0/2720411201 shutdown_connections 2024-03-12T05:56:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.382+0000 7f5a1280d740 1 --2- 172.21.15.76:0/2720411201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 0x5564e025d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.382+0000 7f5a1280d740 1 --2- 172.21.15.76:0/2720411201 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5564e026faf0 0x5564e0271f10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.382+0000 7f5a1280d740 1 --2- 172.21.15.76:0/2720411201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256b70 0x5564e025cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.382+0000 7f5a1280d740 1 --2- 172.21.15.76:0/2720411201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256250 0x5564e025c760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.382+0000 7f5a1280d740 1 -- 172.21.15.76:0/2720411201 >> 172.21.15.76:0/2720411201 conn(0x5564e0250980 msgr2=0x5564e02526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.382+0000 7f59f17fa640 0 client.25240 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.382+0000 7f5a1280d740 1 -- 172.21.15.76:0/2720411201 shutdown_connections 2024-03-12T05:56:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.382+0000 7f5a1280d740 1 -- 172.21.15.76:0/2720411201 wait complete. 2024-03-12T05:56:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.382+0000 7f5a1280d740 20 client.25240 trim_cache size 0 max 16384 2024-03-12T05:56:45.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.405+0000 7f5a1280d740 1 Processor -- start 2024-03-12T05:56:45.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.405+0000 7f5a1280d740 1 -- start start 2024-03-12T05:56:45.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.405+0000 7f5a1280d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256250 0x5564e025ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.405+0000 7f5a0d9dc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256250 0x5564e025ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.405+0000 7f5a0d9dc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256250 0x5564e025ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41184/0 (socket says 172.21.15.76:41184) 2024-03-12T05:56:45.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.405+0000 7f5a0d9dc640 1 -- 172.21.15.76:0/1319342193 learned_addr learned my addr 172.21.15.76:0/1319342193 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.405+0000 7f5a1280d740 1 --2- 172.21.15.76:0/1319342193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e0256b70 0x5564e0257a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.405+0000 7f5a1280d740 1 --2- 172.21.15.76:0/1319342193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e025f750 0x5564e0257fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.405+0000 7f5a1280d740 1 -- 172.21.15.76:0/1319342193 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e0143130 con 0x5564e0256250 2024-03-12T05:56:45.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.405+0000 7f5a1280d740 1 -- 172.21.15.76:0/1319342193 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e0255ec0 con 0x5564e0256b70 2024-03-12T05:56:45.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.405+0000 7f5a1280d740 1 -- 172.21.15.76:0/1319342193 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e025c550 con 0x5564e025f750 2024-03-12T05:56:45.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.406+0000 7f5a0d1db640 1 --2- 172.21.15.76:0/1319342193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e0256b70 0x5564e0257a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.406+0000 7f5a0e1dd640 1 --2- 172.21.15.76:0/1319342193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e025f750 0x5564e0257fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.406+0000 7f5a0d1db640 1 -- 172.21.15.76:0/1319342193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e025f750 msgr2=0x5564e0257fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.406+0000 7f5a0d1db640 1 --2- 172.21.15.76:0/1319342193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e025f750 0x5564e0257fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.406+0000 7f5a0d1db640 1 -- 172.21.15.76:0/1319342193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256250 msgr2=0x5564e025ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.406+0000 7f5a0d1db640 1 --2- 172.21.15.76:0/1319342193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256250 0x5564e025ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.406+0000 7f5a0d1db640 1 -- 172.21.15.76:0/1319342193 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564e0158c20 con 0x5564e0256b70 2024-03-12T05:56:45.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.406+0000 7f5a0e1dd640 1 --2- 172.21.15.76:0/1319342193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e025f750 0x5564e0257fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:56:45.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.406+0000 7f5a0d9dc640 1 --2- 172.21.15.76:0/1319342193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256250 0x5564e025ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:45.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.406+0000 7f5a0d1db640 1 --2- 172.21.15.76:0/1319342193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e0256b70 0x5564e0257a90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x5564e0272810 tx=0x7f59f8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fbed96b766e9671 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.406+0000 7f59f37fe640 1 -- 172.21.15.76:0/1319342193 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f8015040 con 0x5564e0256b70 2024-03-12T05:56:45.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.406+0000 7f59f37fe640 1 -- 172.21.15.76:0/1319342193 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f8016040 con 0x5564e0256b70 2024-03-12T05:56:45.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.406+0000 7f59f37fe640 1 -- 172.21.15.76:0/1319342193 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f80047a0 con 0x5564e0256b70 2024-03-12T05:56:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.407+0000 7f5a1280d740 1 -- 172.21.15.76:0/1319342193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e0256b70 msgr2=0x5564e0257a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.407+0000 7f5a1280d740 1 --2- 172.21.15.76:0/1319342193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e0256b70 0x5564e0257a90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x5564e0272810 tx=0x7f59f8009260 comp rx=0 tx=0).stop 2024-03-12T05:56:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.407+0000 7f5a1280d740 1 -- 172.21.15.76:0/1319342193 shutdown_connections 2024-03-12T05:56:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.407+0000 7f5a1280d740 1 --2- 172.21.15.76:0/1319342193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e025f750 0x5564e0257fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.407+0000 7f5a1280d740 1 --2- 172.21.15.76:0/1319342193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e0256b70 0x5564e0257a90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.407+0000 7f5a1280d740 1 --2- 172.21.15.76:0/1319342193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256250 0x5564e025ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.407+0000 7f5a1280d740 1 -- 172.21.15.76:0/1319342193 >> 172.21.15.76:0/1319342193 conn(0x5564e0250980 msgr2=0x5564e0276890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.407+0000 7f5a1280d740 1 -- 172.21.15.76:0/1319342193 shutdown_connections 2024-03-12T05:56:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.407+0000 7f5a1280d740 1 -- 172.21.15.76:0/1319342193 wait complete. 2024-03-12T05:56:45.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.408+0000 7f5a1280d740 1 -- start start 2024-03-12T05:56:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.408+0000 7f5a1280d740 1 Processor -- start 2024-03-12T05:56:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.408+0000 7f5a1280d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256250 0x5564e02669a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.408+0000 7f5a1280d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256b70 0x5564e0264d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a1280d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 0x5564e0265280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a1280d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e0265950 con 0x5564e0256b70 2024-03-12T05:56:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a1280d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e0265ac0 con 0x5564e025f750 2024-03-12T05:56:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a1280d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564e02ce830 con 0x5564e0256250 2024-03-12T05:56:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a1280d740 10 client.0 _ll_register_callbacks cb 0x7f5a0400ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a0d1db640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256b70 0x5564e0264d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a0d1db640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256b70 0x5564e0264d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41186/0 (socket says 172.21.15.76:41186) 2024-03-12T05:56:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a0d1db640 1 -- 172.21.15.76:0/9443615 learned_addr learned my addr 172.21.15.76:0/9443615 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a0d9dc640 1 --2- 172.21.15.76:0/9443615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256250 0x5564e02669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a0e1dd640 1 --2- 172.21.15.76:0/9443615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 0x5564e0265280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a0d1db640 1 -- 172.21.15.76:0/9443615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256250 msgr2=0x5564e02669a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a0d1db640 1 --2- 172.21.15.76:0/9443615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256250 0x5564e02669a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a0d1db640 1 -- 172.21.15.76:0/9443615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 msgr2=0x5564e0265280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a0d1db640 1 --2- 172.21.15.76:0/9443615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 0x5564e0265280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a0d1db640 1 -- 172.21.15.76:0/9443615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5564e01bc6a0 con 0x5564e0256b70 2024-03-12T05:56:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.409+0000 7f5a0d1db640 1 --2- 172.21.15.76:0/9443615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256b70 0x5564e0264d40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f59f8010c70 tx=0x7f59f8004780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb7bc7b72937326 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.410+0000 7f59f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.410+0000 7f59f17fa640 1 -- 172.21.15.76:0/9443615 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f8015070 con 0x5564e0256b70 2024-03-12T05:56:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.410+0000 7f59f17fa640 1 -- 172.21.15.76:0/9443615 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f59f8010500 con 0x5564e0256b70 2024-03-12T05:56:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.410+0000 7f5a1280d740 1 -- 172.21.15.76:0/9443615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a04020070 con 0x5564e0256b70 2024-03-12T05:56:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.410+0000 7f59f17fa640 1 -- 172.21.15.76:0/9443615 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59f80106e0 con 0x5564e0256b70 2024-03-12T05:56:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.410+0000 7f5a1280d740 10 client.15831 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.410+0000 7f5a1280d740 1 -- 172.21.15.76:0/9443615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x7f59f8010980 con 0x5564e0256b70 2024-03-12T05:56:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.410+0000 7f59f17fa640 1 -- 172.21.15.76:0/9443615 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59f8004e60 con 0x5564e0256b70 2024-03-12T05:56:45.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.410+0000 7f59f17fa640 1 -- 172.21.15.76:0/9443615 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f59f80223e0 con 0x5564e0256b70 2024-03-12T05:56:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.411+0000 7f5a1280d740 10 client.15831 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.411+0000 7f5a1280d740 10 client.15831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.411+0000 7f5a1280d740 20 client.15831 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.411+0000 7f5a1280d740 1 --2- 172.21.15.76:0/9443615 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5564e02cfeb0 0x5564e02d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.411+0000 7f5a1280d740 4 client.15831 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:56:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.411+0000 7f5a1280d740 1 -- 172.21.15.76:0/9443615 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f5a04020070 con 0x5564e02cfeb0 2024-03-12T05:56:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.411+0000 7f5a0d9dc640 1 --2- 172.21.15.76:0/9443615 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5564e02cfeb0 0x5564e02d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.411+0000 7f59f17fa640 10 client.15831 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:45.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.411+0000 7f5a0d9dc640 1 --2- 172.21.15.76:0/9443615 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5564e02cfeb0 0x5564e02d22d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94a699453899429d server_cookie=97a402ca42d8d520 in_seq=0 out_seq=0 2024-03-12T05:56:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.412+0000 7f59f17fa640 1 -- 172.21.15.76:0/9443615 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f5a04020070 con 0x5564e02cfeb0 2024-03-12T05:56:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.412+0000 7f59f17fa640 10 client.15831 handle_command_reply: tid=0 2024-03-12T05:56:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.412+0000 7f5a1280d740 1 client.15831 shutdown 2024-03-12T05:56:45.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.412+0000 7f5a1280d740 1 -- 172.21.15.76:0/9443615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256b70 msgr2=0x5564e0264d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.412+0000 7f5a1280d740 1 --2- 172.21.15.76:0/9443615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256b70 0x5564e0264d40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f59f8010c70 tx=0x7f59f8004780 comp rx=0 tx=0).stop 2024-03-12T05:56:45.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.413+0000 7f5a1280d740 1 -- 172.21.15.76:0/9443615 shutdown_connections 2024-03-12T05:56:45.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.413+0000 7f5a1280d740 1 --2- 172.21.15.76:0/9443615 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5564e025f750 0x5564e0265280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.413+0000 7f5a1280d740 1 --2- 172.21.15.76:0/9443615 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5564e02cfeb0 0x5564e02d22d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.413+0000 7f5a1280d740 1 --2- 172.21.15.76:0/9443615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5564e0256b70 0x5564e0264d40 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.413+0000 7f59f17fa640 0 client.15831 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:45.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.413+0000 7f5a1280d740 1 --2- 172.21.15.76:0/9443615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5564e0256250 0x5564e02669a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.413+0000 7f5a1280d740 1 -- 172.21.15.76:0/9443615 >> 172.21.15.76:0/9443615 conn(0x5564e0250980 msgr2=0x5564e025ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.413+0000 7f5a1280d740 1 -- 172.21.15.76:0/9443615 shutdown_connections 2024-03-12T05:56:45.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.413+0000 7f5a1280d740 1 -- 172.21.15.76:0/9443615 wait complete. 2024-03-12T05:56:45.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.413+0000 7f5a1280d740 20 client.15831 trim_cache size 0 max 16384 2024-03-12T05:56:45.422 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:56:45.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.413+0000 7f5a0fc40640 1 -- 172.21.15.76:0/4110472908 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f59f40a1c20 msgr2=0x7f59f40a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.414+0000 7f5a0fc40640 1 --2- 172.21.15.76:0/4110472908 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f59f40a1c20 0x7f59f40a40e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5a04005fd0 tx=0x7f5a0400a400 comp rx=0 tx=0).stop 2024-03-12T05:56:45.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.414+0000 7f5a0fc40640 1 -- 172.21.15.76:0/4110472908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a0819bde0 msgr2=0x7f5a0819c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:45.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.414+0000 7f5a0fc40640 1 --2- 172.21.15.76:0/4110472908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a0819bde0 0x7f5a0819c290 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5a0002c430 tx=0x7f5a00033610 comp rx=0 tx=0).stop 2024-03-12T05:56:45.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.414+0000 7f5a0fc40640 1 -- 172.21.15.76:0/4110472908 shutdown_connections 2024-03-12T05:56:45.423 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.414+0000 7f5a0fc40640 1 --2- 172.21.15.76:0/4110472908 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f59f40a1c20 0x7f59f40a40e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.414+0000 7f5a0fc40640 1 --2- 172.21.15.76:0/4110472908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a0819bde0 0x7f5a0819c290 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.414+0000 7f5a0fc40640 1 --2- 172.21.15.76:0/4110472908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a08105360 0x7f5a08197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.414+0000 7f5a0fc40640 1 --2- 172.21.15.76:0/4110472908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a08100290 0x7f5a08197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:45.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.414+0000 7f5a0fc40640 1 -- 172.21.15.76:0/4110472908 >> 172.21.15.76:0/4110472908 conn(0x7f5a08077230 msgr2=0x7f5a080684f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:45.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.414+0000 7f5a0fc40640 1 -- 172.21.15.76:0/4110472908 shutdown_connections 2024-03-12T05:56:45.424 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:45.414+0000 7f5a0fc40640 1 -- 172.21.15.76:0/4110472908 wait complete. 2024-03-12T05:56:45.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:45 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4048488863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:56:45.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:45 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4048488863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:56:45.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:45 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4048488863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:56:46.436 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:56:46.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:56:46.436 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:56:46.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:46 smithi082 ceph-mon[36382]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:46.460 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:46 smithi076 ceph-mon[32095]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:46 smithi088 ceph-mon[36585]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:46.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.539+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3772523360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb384106460 msgr2=0x7fb38410e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.539+0000 7fb38b9d7640 1 --2- 172.21.15.76:0/3772523360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb384106460 0x7fb38410e030 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009cd0 tx=0x7fb37c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.539+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3772523360 shutdown_connections 2024-03-12T05:56:46.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.539+0000 7fb38b9d7640 1 --2- 172.21.15.76:0/3772523360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb384106460 0x7fb38410e030 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.539+0000 7fb38b9d7640 1 --2- 172.21.15.76:0/3772523360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb384101d40 0x7fb384105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.539+0000 7fb38b9d7640 1 --2- 172.21.15.76:0/3772523360 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb384101390 0x7fb384101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.539+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3772523360 >> 172.21.15.76:0/3772523360 conn(0x7fb3840fcc10 msgr2=0x7fb3840ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:46.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.539+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3772523360 shutdown_connections 2024-03-12T05:56:46.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.539+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3772523360 wait complete. 2024-03-12T05:56:46.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.540+0000 7fb38b9d7640 1 Processor -- start 2024-03-12T05:56:46.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.540+0000 7fb38b9d7640 1 -- start start 2024-03-12T05:56:46.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.540+0000 7fb38b9d7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb384101390 0x7fb38419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.540+0000 7fb38b9d7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb384106460 0x7fb38419d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.540+0000 7fb38b9d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb384197610 0x7fb384197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.540+0000 7fb388f72640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb384106460 0x7fb38419d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.540+0000 7fb38b9d7640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb384198000 con 0x7fb384106460 2024-03-12T05:56:46.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.540+0000 7fb388f72640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb384106460 0x7fb38419d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41200/0 (socket says 172.21.15.76:41200) 2024-03-12T05:56:46.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.540+0000 7fb388f72640 1 -- 172.21.15.76:0/3981059865 learned_addr learned my addr 172.21.15.76:0/3981059865 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:46.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.540+0000 7fb389773640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb384101390 0x7fb38419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.540+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3981059865 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb384198170 con 0x7fb384101390 2024-03-12T05:56:46.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.540+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3981059865 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb384198330 con 0x7fb384197610 2024-03-12T05:56:46.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.541+0000 7fb389f74640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb384197610 0x7fb384197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.541+0000 7fb388f72640 1 -- 172.21.15.76:0/3981059865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb384197610 msgr2=0x7fb384197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.541+0000 7fb388f72640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb384197610 0x7fb384197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.541+0000 7fb388f72640 1 -- 172.21.15.76:0/3981059865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb384101390 msgr2=0x7fb38419d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.541+0000 7fb388f72640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb384101390 0x7fb38419d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.541+0000 7fb388f72640 1 -- 172.21.15.76:0/3981059865 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37c009900 con 0x7fb384106460 2024-03-12T05:56:46.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.541+0000 7fb389f74640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb384197610 0x7fb384197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:46.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.541+0000 7fb388f72640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb384106460 0x7fb38419d9a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb37400d740 tx=0x7fb37400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:46.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.541+0000 7fb37a7fc640 1 -- 172.21.15.76:0/3981059865 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb374009c30 con 0x7fb384106460 2024-03-12T05:56:46.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.541+0000 7fb37a7fc640 1 -- 172.21.15.76:0/3981059865 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb37400de50 con 0x7fb384106460 2024-03-12T05:56:46.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.541+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3981059865 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3841a4320 con 0x7fb384106460 2024-03-12T05:56:46.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.541+0000 7fb37a7fc640 1 -- 172.21.15.76:0/3981059865 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb374010460 con 0x7fb384106460 2024-03-12T05:56:46.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.542+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3981059865 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3841a47e0 con 0x7fb384106460 2024-03-12T05:56:46.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.543+0000 7fb37a7fc640 1 -- 172.21.15.76:0/3981059865 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fb374004f10 con 0x7fb384106460 2024-03-12T05:56:46.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.544+0000 7fb37a7fc640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb3700a1cc0 0x7fb3700a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.544+0000 7fb37a7fc640 1 -- 172.21.15.76:0/3981059865 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb3740cd3a0 con 0x7fb384106460 2024-03-12T05:56:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.544+0000 7fb389773640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb3700a1cc0 0x7fb3700a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.544+0000 7fb38e5ab740 1 Processor -- start 2024-03-12T05:56:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.544+0000 7fb38e5ab740 1 -- start start 2024-03-12T05:56:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.544+0000 7fb38e5ab740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67fff790 0x562f67fffb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.544+0000 7fb38e5ab740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f68000140 0x562f680086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.544+0000 7fb38e5ab740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68008cc0 0x562f6800b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.544+0000 7fb38e5ab740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f67f69f90 con 0x562f68008cc0 2024-03-12T05:56:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.544+0000 7fb38e5ab740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f67f674c0 con 0x562f68000140 2024-03-12T05:56:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.544+0000 7fb38e5ab740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f67f62ca0 con 0x562f67fff790 2024-03-12T05:56:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.545+0000 7fb389773640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb3700a1cc0 0x7fb3700a4180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb38000bbe0 tx=0x7fb38000b390 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.545+0000 7fb389f74640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68008cc0 0x562f6800b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.545+0000 7fb388f72640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f68000140 0x562f680086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.545+0000 7fb389f74640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68008cc0 0x562f6800b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41202/0 (socket says 172.21.15.76:41202) 2024-03-12T05:56:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.545+0000 7fb389f74640 1 -- 172.21.15.76:0/2402369038 learned_addr learned my addr 172.21.15.76:0/2402369038 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.545+0000 7fb389773640 1 --2- 172.21.15.76:0/2402369038 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67fff790 0x562f67fffb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.545+0000 7fb389f74640 1 -- 172.21.15.76:0/2402369038 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67fff790 msgr2=0x562f67fffb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.545+0000 7fb389f74640 1 --2- 172.21.15.76:0/2402369038 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67fff790 0x562f67fffb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.545+0000 7fb389f74640 1 -- 172.21.15.76:0/2402369038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f68000140 msgr2=0x562f680086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.545+0000 7fb389f74640 1 --2- 172.21.15.76:0/2402369038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f68000140 0x562f680086f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.545+0000 7fb389f74640 1 -- 172.21.15.76:0/2402369038 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f67f250b0 con 0x562f68008cc0 2024-03-12T05:56:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.545+0000 7fb388f72640 1 --2- 172.21.15.76:0/2402369038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f68000140 0x562f680086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.545+0000 7fb389f74640 1 --2- 172.21.15.76:0/2402369038 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68008cc0 0x562f6800b110 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb37c02c700 tx=0x7fb37c032d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc57b6f35423878 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.546+0000 7fb36f7fe640 1 -- 172.21.15.76:0/2402369038 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c034440 con 0x562f68008cc0 2024-03-12T05:56:46.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.546+0000 7fb36f7fe640 1 -- 172.21.15.76:0/2402369038 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb37c0345a0 con 0x562f68008cc0 2024-03-12T05:56:46.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.546+0000 7fb36f7fe640 1 -- 172.21.15.76:0/2402369038 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c003bd0 con 0x562f68008cc0 2024-03-12T05:56:46.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.546+0000 7fb38e5ab740 1 -- 172.21.15.76:0/2402369038 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68008cc0 msgr2=0x562f6800b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.546+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/2402369038 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68008cc0 0x562f6800b110 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb37c02c700 tx=0x7fb37c032d70 comp rx=0 tx=0).stop 2024-03-12T05:56:46.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.546+0000 7fb38e5ab740 1 -- 172.21.15.76:0/2402369038 shutdown_connections 2024-03-12T05:56:46.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.546+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/2402369038 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68008cc0 0x562f6800b110 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.546+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/2402369038 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f68000140 0x562f680086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.546+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/2402369038 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67fff790 0x562f67fffb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.546+0000 7fb38e5ab740 1 -- 172.21.15.76:0/2402369038 >> 172.21.15.76:0/2402369038 conn(0x562f67ff9c50 msgr2=0x562f67ffa030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.547+0000 7fb38e5ab740 1 -- 172.21.15.76:0/2402369038 shutdown_connections 2024-03-12T05:56:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.547+0000 7fb38e5ab740 1 -- 172.21.15.76:0/2402369038 wait complete. 2024-03-12T05:56:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.547+0000 7fb38e5ab740 1 -- start start 2024-03-12T05:56:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.547+0000 7fb38e5ab740 1 Processor -- start 2024-03-12T05:56:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.547+0000 7fb38e5ab740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f67fff790 0x562f68005cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.547+0000 7fb38e5ab740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f68000140 0x562f680061f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.547+0000 7fb38e5ab740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68008cc0 0x562f68006730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.547+0000 7fb38e5ab740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f67f07f40 con 0x562f68008cc0 2024-03-12T05:56:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.547+0000 7fb38e5ab740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f67c364b0 con 0x562f67fff790 2024-03-12T05:56:46.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.547+0000 7fb38e5ab740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f67ee8670 con 0x562f68000140 2024-03-12T05:56:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb38e5ab740 10 client.0 _ll_register_callbacks cb 0x562f67f55f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb388f72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f68000140 0x562f680061f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb388f72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f68000140 0x562f680061f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.76:47138/0 (socket says 172.21.15.76:47138) 2024-03-12T05:56:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb388f72640 1 -- 172.21.15.76:0/3554691437 learned_addr learned my addr 172.21.15.76:0/3554691437 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb389f74640 1 --2- 172.21.15.76:0/3554691437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68008cc0 0x562f68006730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb389773640 1 --2- 172.21.15.76:0/3554691437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f67fff790 0x562f68005cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb388f72640 1 -- 172.21.15.76:0/3554691437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f67fff790 msgr2=0x562f68005cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb388f72640 1 --2- 172.21.15.76:0/3554691437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f67fff790 0x562f68005cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb388f72640 1 -- 172.21.15.76:0/3554691437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68008cc0 msgr2=0x562f68006730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb388f72640 1 --2- 172.21.15.76:0/3554691437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68008cc0 0x562f68006730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb388f72640 1 -- 172.21.15.76:0/3554691437 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38410e840 con 0x562f68000140 2024-03-12T05:56:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb389f74640 1 --2- 172.21.15.76:0/3554691437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68008cc0 0x562f68006730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb388f72640 1 --2- 172.21.15.76:0/3554691437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f68000140 0x562f680061f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb3740caab0 tx=0x7fb374099e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=241c0acd49a2ae98 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:46.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb36cff9640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.548+0000 7fb36cff9640 1 -- 172.21.15.76:0/3554691437 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3740980b0 con 0x562f68000140 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.549+0000 7fb38e5ab740 1 -- 172.21.15.76:0/3554691437 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562f67f3cf40 con 0x562f68000140 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.549+0000 7fb36cff9640 1 -- 172.21.15.76:0/3554691437 <== mon.1 v2:172.21.15.88:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb374096420 con 0x562f68000140 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.549+0000 7fb36cff9640 1 -- 172.21.15.76:0/3554691437 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3740965e0 con 0x562f68000140 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.549+0000 7fb38e5ab740 10 client.25255 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.549+0000 7fb38e5ab740 1 -- 172.21.15.76:0/3554691437 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x562f67f61320 con 0x562f68000140 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.549+0000 7fb36cff9640 1 -- 172.21.15.76:0/3554691437 <== mon.1 v2:172.21.15.88:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb37407c070 con 0x562f68000140 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.549+0000 7fb36cff9640 1 -- 172.21.15.76:0/3554691437 <== mon.1 v2:172.21.15.88:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb3740cad10 con 0x562f68000140 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.549+0000 7fb38e5ab740 10 client.25255 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.549+0000 7fb38e5ab740 10 client.25255 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.549+0000 7fb38e5ab740 20 client.25255 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.549+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/3554691437 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562f680191d0 0x562f6801b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.549+0000 7fb38e5ab740 4 client.25255 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.549+0000 7fb38e5ab740 1 -- 172.21.15.76:0/3554691437 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562f67f59310 con 0x562f680191d0 2024-03-12T05:56:46.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.549+0000 7fb389773640 1 --2- 172.21.15.76:0/3554691437 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562f680191d0 0x562f6801b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.550+0000 7fb36cff9640 10 client.25255 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.550+0000 7fb389773640 1 --2- 172.21.15.76:0/3554691437 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562f680191d0 0x562f6801b5c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c4b989d9cf3b8d2 server_cookie=9518e78ed7b380d1 in_seq=0 out_seq=0 2024-03-12T05:56:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.550+0000 7fb36cff9640 1 -- 172.21.15.76:0/3554691437 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562f67f59310 con 0x562f680191d0 2024-03-12T05:56:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.550+0000 7fb36cff9640 10 client.25255 handle_command_reply: tid=0 2024-03-12T05:56:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.550+0000 7fb38e5ab740 1 client.25255 shutdown 2024-03-12T05:56:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.551+0000 7fb38e5ab740 1 -- 172.21.15.76:0/3554691437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f68000140 msgr2=0x562f680061f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.551+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/3554691437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f68000140 0x562f680061f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb3740caab0 tx=0x7fb374099e70 comp rx=0 tx=0).stop 2024-03-12T05:56:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.551+0000 7fb38e5ab740 1 -- 172.21.15.76:0/3554691437 shutdown_connections 2024-03-12T05:56:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.551+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/3554691437 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562f680191d0 0x562f6801b5c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.551+0000 7fb36cff9640 0 client.25255 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.551+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/3554691437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68008cc0 0x562f68006730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.551+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/3554691437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f68000140 0x562f680061f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.551+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/3554691437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f67fff790 0x562f68005cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.551+0000 7fb38e5ab740 1 -- 172.21.15.76:0/3554691437 >> 172.21.15.76:0/3554691437 conn(0x562f67ff9c50 msgr2=0x562f67ffbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:46.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.551+0000 7fb38e5ab740 1 -- 172.21.15.76:0/3554691437 shutdown_connections 2024-03-12T05:56:46.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.551+0000 7fb38e5ab740 1 -- 172.21.15.76:0/3554691437 wait complete. 2024-03-12T05:56:46.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.551+0000 7fb38e5ab740 20 client.25255 trim_cache size 0 max 16384 2024-03-12T05:56:46.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.566+0000 7fb38e5ab740 1 Processor -- start 2024-03-12T05:56:46.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.566+0000 7fb38e5ab740 1 -- start start 2024-03-12T05:56:46.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.566+0000 7fb38e5ab740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67ff9c50 0x562f680083e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.566+0000 7fb38e5ab740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f67fff790 0x562f68006550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.566+0000 7fb38e5ab740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f68000140 0x562f68002f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.566+0000 7fb38e5ab740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f67ff1180 con 0x562f67fff790 2024-03-12T05:56:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.566+0000 7fb38e5ab740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f68014ba0 con 0x562f68000140 2024-03-12T05:56:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.566+0000 7fb38e5ab740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f68004200 con 0x562f67ff9c50 2024-03-12T05:56:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.566+0000 7fb388f72640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f67fff790 0x562f68006550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.567+0000 7fb388f72640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f67fff790 0x562f68006550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41212/0 (socket says 172.21.15.76:41212) 2024-03-12T05:56:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.567+0000 7fb388f72640 1 -- 172.21.15.76:0/3461990643 learned_addr learned my addr 172.21.15.76:0/3461990643 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.567+0000 7fb389773640 1 --2- 172.21.15.76:0/3461990643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67ff9c50 0x562f680083e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.567+0000 7fb389f74640 1 --2- 172.21.15.76:0/3461990643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f68000140 0x562f68002f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.567+0000 7fb388f72640 1 -- 172.21.15.76:0/3461990643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67ff9c50 msgr2=0x562f680083e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.567+0000 7fb388f72640 1 --2- 172.21.15.76:0/3461990643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67ff9c50 0x562f680083e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.567+0000 7fb388f72640 1 -- 172.21.15.76:0/3461990643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f68000140 msgr2=0x562f68002f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.567+0000 7fb388f72640 1 --2- 172.21.15.76:0/3461990643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f68000140 0x562f68002f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.567+0000 7fb388f72640 1 -- 172.21.15.76:0/3461990643 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f67efdc00 con 0x562f67fff790 2024-03-12T05:56:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.567+0000 7fb388f72640 1 --2- 172.21.15.76:0/3461990643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f67fff790 0x562f68006550 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fb374096780 tx=0x7fb3740048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46fd6aa132d284f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:46.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.567+0000 7fb36effd640 1 -- 172.21.15.76:0/3461990643 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3740993a0 con 0x562f67fff790 2024-03-12T05:56:46.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.567+0000 7fb36effd640 1 -- 172.21.15.76:0/3461990643 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb37407c050 con 0x562f67fff790 2024-03-12T05:56:46.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.567+0000 7fb36effd640 1 -- 172.21.15.76:0/3461990643 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3740980b0 con 0x562f67fff790 2024-03-12T05:56:46.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.568+0000 7fb38e5ab740 1 -- 172.21.15.76:0/3461990643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f67fff790 msgr2=0x562f68006550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.568+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/3461990643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f67fff790 0x562f68006550 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fb374096780 tx=0x7fb3740048b0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.568+0000 7fb38e5ab740 1 -- 172.21.15.76:0/3461990643 shutdown_connections 2024-03-12T05:56:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.568+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/3461990643 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f68000140 0x562f68002f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.568+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/3461990643 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f67fff790 0x562f68006550 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.568+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/3461990643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67ff9c50 0x562f680083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.568+0000 7fb38e5ab740 1 -- 172.21.15.76:0/3461990643 >> 172.21.15.76:0/3461990643 conn(0x562f68000890 msgr2=0x562f68004580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:46.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.568+0000 7fb38e5ab740 1 -- 172.21.15.76:0/3461990643 shutdown_connections 2024-03-12T05:56:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.568+0000 7fb38e5ab740 1 -- 172.21.15.76:0/3461990643 wait complete. 2024-03-12T05:56:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.568+0000 7fb38e5ab740 1 -- start start 2024-03-12T05:56:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.568+0000 7fb38e5ab740 1 Processor -- start 2024-03-12T05:56:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb38e5ab740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f67ff9c50 0x562f6800e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb38e5ab740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67fff790 0x562f6800edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb38e5ab740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68000140 0x562f6800f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb38e5ab740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f6806f760 con 0x562f68000140 2024-03-12T05:56:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb38e5ab740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f6806f8a0 con 0x562f67ff9c50 2024-03-12T05:56:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb389f74640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68000140 0x562f6800f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb38e5ab740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f6806fa60 con 0x562f67fff790 2024-03-12T05:56:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb389f74640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68000140 0x562f6800f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41228/0 (socket says 172.21.15.76:41228) 2024-03-12T05:56:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb389f74640 1 -- 172.21.15.76:0/2039891679 learned_addr learned my addr 172.21.15.76:0/2039891679 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb38e5ab740 10 client.0 _ll_register_callbacks cb 0x7fb37c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:46.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb388f72640 1 --2- 172.21.15.76:0/2039891679 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67fff790 0x562f6800edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb389773640 1 --2- 172.21.15.76:0/2039891679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f67ff9c50 0x562f6800e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb389f74640 1 -- 172.21.15.76:0/2039891679 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67fff790 msgr2=0x562f6800edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb389f74640 1 --2- 172.21.15.76:0/2039891679 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67fff790 0x562f6800edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb389f74640 1 -- 172.21.15.76:0/2039891679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f67ff9c50 msgr2=0x562f6800e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb389f74640 1 --2- 172.21.15.76:0/2039891679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f67ff9c50 0x562f6800e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.569+0000 7fb389f74640 1 -- 172.21.15.76:0/2039891679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f67f61320 con 0x562f68000140 2024-03-12T05:56:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb389f74640 1 --2- 172.21.15.76:0/2039891679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68000140 0x562f6800f310 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb37c005fe0 tx=0x7fb37c0093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb31735a6a5574dd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb36cff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:46.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb36cff9640 1 -- 172.21.15.76:0/2039891679 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c0326b0 con 0x562f68000140 2024-03-12T05:56:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb36cff9640 1 -- 172.21.15.76:0/2039891679 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb37c034a90 con 0x562f68000140 2024-03-12T05:56:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb38e5ab740 1 -- 172.21.15.76:0/2039891679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562f67f88900 con 0x562f68000140 2024-03-12T05:56:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb36cff9640 1 -- 172.21.15.76:0/2039891679 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb37c032810 con 0x562f68000140 2024-03-12T05:56:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb38e5ab740 10 client.15849 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb38e5ab740 1 -- 172.21.15.76:0/2039891679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x562f67f23910 con 0x562f68000140 2024-03-12T05:56:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb36cff9640 1 -- 172.21.15.76:0/2039891679 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb37c003e50 con 0x562f68000140 2024-03-12T05:56:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb36cff9640 1 -- 172.21.15.76:0/2039891679 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb37c004500 con 0x562f68000140 2024-03-12T05:56:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb38e5ab740 10 client.15849 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:46.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb38e5ab740 10 client.15849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb38e5ab740 20 client.15849 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/2039891679 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562f68008cc0 0x562f68072f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb38e5ab740 4 client.15849 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:56:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.570+0000 7fb38e5ab740 1 -- 172.21.15.76:0/2039891679 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562f67f88900 con 0x562f68008cc0 2024-03-12T05:56:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.571+0000 7fb389773640 1 --2- 172.21.15.76:0/2039891679 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562f68008cc0 0x562f68072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.571+0000 7fb36cff9640 10 client.15849 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.571+0000 7fb389773640 1 --2- 172.21.15.76:0/2039891679 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562f68008cc0 0x562f68072f70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1488b340633e28dc server_cookie=4814d9cc017ba981 in_seq=0 out_seq=0 2024-03-12T05:56:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.572+0000 7fb36cff9640 1 -- 172.21.15.76:0/2039891679 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562f67f88900 con 0x562f68008cc0 2024-03-12T05:56:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.572+0000 7fb36cff9640 10 client.15849 handle_command_reply: tid=0 2024-03-12T05:56:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.572+0000 7fb38e5ab740 1 client.15849 shutdown 2024-03-12T05:56:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.572+0000 7fb38e5ab740 1 -- 172.21.15.76:0/2039891679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68000140 msgr2=0x562f6800f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.572+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/2039891679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68000140 0x562f6800f310 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb37c005fe0 tx=0x7fb37c0093f0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38e5ab740 1 -- 172.21.15.76:0/2039891679 shutdown_connections 2024-03-12T05:56:46.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/2039891679 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562f68008cc0 0x562f68072f70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/2039891679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562f68000140 0x562f6800f310 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/2039891679 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f67fff790 0x562f6800edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb36cff9640 0 client.15849 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38e5ab740 1 --2- 172.21.15.76:0/2039891679 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562f67ff9c50 0x562f6800e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38e5ab740 1 -- 172.21.15.76:0/2039891679 >> 172.21.15.76:0/2039891679 conn(0x562f68000890 msgr2=0x562f67ff4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38e5ab740 1 -- 172.21.15.76:0/2039891679 shutdown_connections 2024-03-12T05:56:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38e5ab740 1 -- 172.21.15.76:0/2039891679 wait complete. 2024-03-12T05:56:46.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38e5ab740 20 client.15849 trim_cache size 0 max 16384 2024-03-12T05:56:46.585 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:56:46.585 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T05:56:46.586 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "ef5e7405-f201-46e6-9645-62bee7d6b192", 2024-03-12T05:56:46.586 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T05:56:46.586 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:56:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3981059865 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb3700a1cc0 msgr2=0x7fb3700a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38b9d7640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb3700a1cc0 0x7fb3700a4180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb38000bbe0 tx=0x7fb38000b390 comp rx=0 tx=0).stop 2024-03-12T05:56:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3981059865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb384106460 msgr2=0x7fb38419d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38b9d7640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb384106460 0x7fb38419d9a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb37400d740 tx=0x7fb37400dc00 comp rx=0 tx=0).stop 2024-03-12T05:56:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3981059865 shutdown_connections 2024-03-12T05:56:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38b9d7640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb3700a1cc0 0x7fb3700a4180 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38b9d7640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb384197610 0x7fb384197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38b9d7640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb384106460 0x7fb38419d9a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38b9d7640 1 --2- 172.21.15.76:0/3981059865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb384101390 0x7fb38419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3981059865 >> 172.21.15.76:0/3981059865 conn(0x7fb3840fcc10 msgr2=0x7fb3840fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.573+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3981059865 shutdown_connections 2024-03-12T05:56:46.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.574+0000 7fb38b9d7640 1 -- 172.21.15.76:0/3981059865 wait complete. 2024-03-12T05:56:46.589 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:56:46.589 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:56:46.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.779+0000 7fc0f4288640 1 -- 172.21.15.76:0/3595656465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0ec100c40 msgr2=0x7fc0ec104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.779+0000 7fc0f4288640 1 --2- 172.21.15.76:0/3595656465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0ec100c40 0x7fc0ec104c80 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc009930 tx=0x7fc0dc02bc30 comp rx=0 tx=0).stop 2024-03-12T05:56:46.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.779+0000 7fc0f4288640 1 -- 172.21.15.76:0/3595656465 shutdown_connections 2024-03-12T05:56:46.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.779+0000 7fc0f4288640 1 --2- 172.21.15.76:0/3595656465 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0ec105360 0x7fc0ec10cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.779+0000 7fc0f4288640 1 --2- 172.21.15.76:0/3595656465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0ec100c40 0x7fc0ec104c80 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.779+0000 7fc0f4288640 1 --2- 172.21.15.76:0/3595656465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0ec100290 0x7fc0ec100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.779+0000 7fc0f4288640 1 -- 172.21.15.76:0/3595656465 >> 172.21.15.76:0/3595656465 conn(0x7fc0ec077140 msgr2=0x7fc0ec077550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:46.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.779+0000 7fc0f4288640 1 -- 172.21.15.76:0/3595656465 shutdown_connections 2024-03-12T05:56:46.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.779+0000 7fc0f4288640 1 -- 172.21.15.76:0/3595656465 wait complete. 2024-03-12T05:56:46.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.780+0000 7fc0f4288640 1 Processor -- start 2024-03-12T05:56:46.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.780+0000 7fc0f4288640 1 -- start start 2024-03-12T05:56:46.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.780+0000 7fc0f4288640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0ec100290 0x7fc0ec19d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.780+0000 7fc0f4288640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0ec105360 0x7fc0ec19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.780+0000 7fc0f4288640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0ec1976d0 0x7fc0ec197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.780+0000 7fc0f4288640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec1981a0 con 0x7fc0ec100290 2024-03-12T05:56:46.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.780+0000 7fc0f4288640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec198310 con 0x7fc0ec105360 2024-03-12T05:56:46.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.780+0000 7fc0f4288640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec1984d0 con 0x7fc0ec1976d0 2024-03-12T05:56:46.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.780+0000 7fc0f2024640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0ec100290 0x7fc0ec19d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.780+0000 7fc0f2024640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0ec100290 0x7fc0ec19d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41238/0 (socket says 172.21.15.76:41238) 2024-03-12T05:56:46.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.780+0000 7fc0f2024640 1 -- 172.21.15.76:0/1013962859 learned_addr learned my addr 172.21.15.76:0/1013962859 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:46.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.781+0000 7fc0f1823640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0ec105360 0x7fc0ec19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.781+0000 7fc0f2825640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0ec1976d0 0x7fc0ec197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.781+0000 7fc0f1823640 1 -- 172.21.15.76:0/1013962859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0ec1976d0 msgr2=0x7fc0ec197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.781+0000 7fc0f1823640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0ec1976d0 0x7fc0ec197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.781+0000 7fc0f1823640 1 -- 172.21.15.76:0/1013962859 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0ec100290 msgr2=0x7fc0ec19d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.781+0000 7fc0f1823640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0ec100290 0x7fc0ec19d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.781+0000 7fc0f1823640 1 -- 172.21.15.76:0/1013962859 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e4009840 con 0x7fc0ec105360 2024-03-12T05:56:46.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.781+0000 7fc0f2024640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0ec100290 0x7fc0ec19d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:46.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.781+0000 7fc0f1823640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0ec105360 0x7fc0ec19da60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc009230 tx=0x7fc0dc02c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:46.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.781+0000 7fc0e37fe640 1 -- 172.21.15.76:0/1013962859 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc0042d0 con 0x7fc0ec105360 2024-03-12T05:56:46.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.781+0000 7fc0e37fe640 1 -- 172.21.15.76:0/1013962859 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0dc004430 con 0x7fc0ec105360 2024-03-12T05:56:46.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.781+0000 7fc0f4288640 1 -- 172.21.15.76:0/1013962859 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0dc009560 con 0x7fc0ec105360 2024-03-12T05:56:46.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.782+0000 7fc0e37fe640 1 -- 172.21.15.76:0/1013962859 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc003a10 con 0x7fc0ec105360 2024-03-12T05:56:46.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.782+0000 7fc0f4288640 1 -- 172.21.15.76:0/1013962859 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ec1a45b0 con 0x7fc0ec105360 2024-03-12T05:56:46.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.785+0000 7fc0f6e69740 1 Processor -- start 2024-03-12T05:56:46.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.785+0000 7fc0e37fe640 1 -- 172.21.15.76:0/1013962859 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7fc0dc005360 con 0x7fc0ec105360 2024-03-12T05:56:46.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.785+0000 7fc0f6e69740 1 -- start start 2024-03-12T05:56:46.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.785+0000 7fc0f6e69740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee12b0 0x5619c4ee1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.785+0000 7fc0f6e69740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee1bd0 0x5619c4eea270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.785+0000 7fc0f6e69740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4eea7b0 0x5619c4eecba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.785+0000 7fc0f6e69740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c4e50280 con 0x5619c4ee1bd0 2024-03-12T05:56:46.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.785+0000 7fc0f6e69740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c4e4d7b0 con 0x5619c4eea7b0 2024-03-12T05:56:46.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.785+0000 7fc0f6e69740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c4e48f90 con 0x5619c4ee12b0 2024-03-12T05:56:46.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.786+0000 7fc0f2825640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee12b0 0x5619c4ee1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.786+0000 7fc0f2024640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee1bd0 0x5619c4eea270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.786+0000 7fc0f2825640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee12b0 0x5619c4ee1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.76:47176/0 (socket says 172.21.15.76:47176) 2024-03-12T05:56:46.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.786+0000 7fc0f2825640 1 -- 172.21.15.76:0/1803946025 learned_addr learned my addr 172.21.15.76:0/1803946025 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:46.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.786+0000 7fc0f1823640 1 --2- 172.21.15.76:0/1803946025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4eea7b0 0x5619c4eecba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.786+0000 7fc0f2825640 1 -- 172.21.15.76:0/1803946025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4eea7b0 msgr2=0x5619c4eecba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.786+0000 7fc0f2825640 1 --2- 172.21.15.76:0/1803946025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4eea7b0 0x5619c4eecba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.786+0000 7fc0f2825640 1 -- 172.21.15.76:0/1803946025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee1bd0 msgr2=0x5619c4eea270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.786+0000 7fc0f2825640 1 --2- 172.21.15.76:0/1803946025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee1bd0 0x5619c4eea270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.786+0000 7fc0e37fe640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc0d80a1cc0 0x7fc0d80a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.786+0000 7fc0f2825640 1 -- 172.21.15.76:0/1803946025 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619c4e07570 con 0x5619c4ee12b0 2024-03-12T05:56:46.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.787+0000 7fc0f1823640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc0d80a1cc0 0x7fc0d80a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.787+0000 7fc0e37fe640 1 -- 172.21.15.76:0/1013962859 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc0dc0eff10 con 0x7fc0ec105360 2024-03-12T05:56:46.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.787+0000 7fc0f2825640 1 --2- 172.21.15.76:0/1803946025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee12b0 0x5619c4ee1690 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4011570 tx=0x7fc0e4011a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19072632649d3d6c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:46.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.787+0000 7fc0f2024640 1 --2- 172.21.15.76:0/1803946025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee1bd0 0x5619c4eea270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:46.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.787+0000 7fc0e0bfe640 1 -- 172.21.15.76:0/1803946025 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4004d20 con 0x5619c4ee12b0 2024-03-12T05:56:46.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.787+0000 7fc0e0bfe640 1 -- 172.21.15.76:0/1803946025 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0e4004e80 con 0x5619c4ee12b0 2024-03-12T05:56:46.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.788+0000 7fc0e0bfe640 1 -- 172.21.15.76:0/1803946025 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e400f8a0 con 0x5619c4ee12b0 2024-03-12T05:56:46.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.788+0000 7fc0f6e69740 1 -- 172.21.15.76:0/1803946025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee12b0 msgr2=0x5619c4ee1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.788+0000 7fc0f1823640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc0d80a1cc0 0x7fc0d80a4180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc0ed680 tx=0x7fc0dc009260 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:46.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.788+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/1803946025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee12b0 0x5619c4ee1690 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4011570 tx=0x7fc0e4011a30 comp rx=0 tx=0).stop 2024-03-12T05:56:46.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.788+0000 7fc0f6e69740 1 -- 172.21.15.76:0/1803946025 shutdown_connections 2024-03-12T05:56:46.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.788+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/1803946025 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4eea7b0 0x5619c4eecba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.788+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/1803946025 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee1bd0 0x5619c4eea270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.788+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/1803946025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee12b0 0x5619c4ee1690 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.788+0000 7fc0f6e69740 1 -- 172.21.15.76:0/1803946025 >> 172.21.15.76:0/1803946025 conn(0x5619c4edb990 msgr2=0x5619c4edbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:46.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.788+0000 7fc0f6e69740 1 -- 172.21.15.76:0/1803946025 shutdown_connections 2024-03-12T05:56:46.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.788+0000 7fc0f6e69740 1 -- 172.21.15.76:0/1803946025 wait complete. 2024-03-12T05:56:46.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.789+0000 7fc0f6e69740 1 -- start start 2024-03-12T05:56:46.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.789+0000 7fc0f6e69740 1 Processor -- start 2024-03-12T05:56:46.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.789+0000 7fc0f6e69740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee12b0 0x5619c4eea160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.789+0000 7fc0f6e69740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee1bd0 0x5619c4ee74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.789+0000 7fc0f6e69740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4eea7b0 0x5619c4ee7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.789+0000 7fc0f6e69740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c4b1c430 con 0x5619c4ee1bd0 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.789+0000 7fc0f6e69740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c4ad8e00 con 0x5619c4eea7b0 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.789+0000 7fc0f6e69740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c4dcdfd0 con 0x5619c4ee12b0 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.789+0000 7fc0f2024640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee1bd0 0x5619c4ee74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.789+0000 7fc0f2024640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee1bd0 0x5619c4ee74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41264/0 (socket says 172.21.15.76:41264) 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.789+0000 7fc0f2024640 1 -- 172.21.15.76:0/2438478638 learned_addr learned my addr 172.21.15.76:0/2438478638 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.789+0000 7fc0f2825640 1 --2- 172.21.15.76:0/2438478638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee12b0 0x5619c4eea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0f6e69740 10 client.0 _ll_register_callbacks cb 0x5619c4e23fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0f2825640 1 --2- 172.21.15.76:0/2438478638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4eea7b0 0x5619c4ee7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0f2024640 1 -- 172.21.15.76:0/2438478638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee12b0 msgr2=0x5619c4eea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0f2024640 1 --2- 172.21.15.76:0/2438478638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee12b0 0x5619c4eea160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0f2024640 1 -- 172.21.15.76:0/2438478638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4eea7b0 msgr2=0x5619c4ee7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0f2024640 1 --2- 172.21.15.76:0/2438478638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4eea7b0 0x5619c4ee7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0f2024640 1 -- 172.21.15.76:0/2438478638 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e4009da0 con 0x5619c4ee1bd0 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0f2825640 1 --2- 172.21.15.76:0/2438478638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee12b0 0x5619c4eea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0f6e69740 1 -- 172.21.15.76:0/2438478638 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5619c4b85720 con 0x5619c4ee1bd0 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0f2024640 1 --2- 172.21.15.76:0/2438478638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee1bd0 0x5619c4ee74f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8012d20 tx=0x7fc0e800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74d7a2841eb247c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0d5ffb640 10 client.15861 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0d5ffb640 1 -- 172.21.15.76:0/2438478638 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8017070 con 0x5619c4ee1bd0 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0d5ffb640 1 -- 172.21.15.76:0/2438478638 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc0e8004020 con 0x5619c4ee1bd0 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0d5ffb640 1 -- 172.21.15.76:0/2438478638 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0e8004240 con 0x5619c4ee1bd0 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0f6e69740 10 client.15861 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.790+0000 7fc0f6e69740 1 -- 172.21.15.76:0/2438478638 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({fsmap=22}) v3 -- 0x5619c4e47610 con 0x5619c4ee1bd0 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.791+0000 7fc0d5ffb640 1 -- 172.21.15.76:0/2438478638 <== mon.0 v2:172.21.15.76:3300/0 4 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc0e80035f0 con 0x5619c4ee1bd0 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.791+0000 7fc0f6e69740 10 client.15861 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.791+0000 7fc0f6e69740 10 client.15861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.791+0000 7fc0f6e69740 20 client.15861 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.791+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/2438478638 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5619c4efa940 0x5619c4efcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.791+0000 7fc0f6e69740 4 client.15861 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:56:46.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.791+0000 7fc0f6e69740 1 -- 172.21.15.76:0/2438478638 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5619c4bd4890 con 0x5619c4efa940 2024-03-12T05:56:46.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.791+0000 7fc0f2825640 1 --2- 172.21.15.76:0/2438478638 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5619c4efa940 0x5619c4efcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.792+0000 7fc0f2825640 1 --2- 172.21.15.76:0/2438478638 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5619c4efa940 0x5619c4efcd60 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4617ebe380a0e8c2 server_cookie=44bc1a1c1fbd2d32 in_seq=0 out_seq=0 2024-03-12T05:56:46.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.792+0000 7fc0d5ffb640 10 client.15861 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:46.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.792+0000 7fc0d5ffb640 1 -- 172.21.15.76:0/2438478638 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5619c4bd4890 con 0x5619c4efa940 2024-03-12T05:56:46.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.792+0000 7fc0d5ffb640 10 client.15861 handle_command_reply: tid=0 2024-03-12T05:56:46.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.792+0000 7fc0f6e69740 1 client.15861 shutdown 2024-03-12T05:56:46.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.793+0000 7fc0f6e69740 1 -- 172.21.15.76:0/2438478638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee1bd0 msgr2=0x5619c4ee74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.793+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/2438478638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee1bd0 0x5619c4ee74f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8012d20 tx=0x7fc0e800f790 comp rx=0 tx=0).stop 2024-03-12T05:56:46.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.793+0000 7fc0f6e69740 1 -- 172.21.15.76:0/2438478638 shutdown_connections 2024-03-12T05:56:46.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.793+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/2438478638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4eea7b0 0x5619c4ee7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.793+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/2438478638 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5619c4efa940 0x5619c4efcd60 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.793+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/2438478638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee1bd0 0x5619c4ee74f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.793+0000 7fc0d5ffb640 0 client.15861 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:46.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.793+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/2438478638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee12b0 0x5619c4eea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.793+0000 7fc0f6e69740 1 -- 172.21.15.76:0/2438478638 >> 172.21.15.76:0/2438478638 conn(0x5619c4edb990 msgr2=0x5619c4edd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:46.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.793+0000 7fc0f6e69740 1 -- 172.21.15.76:0/2438478638 shutdown_connections 2024-03-12T05:56:46.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.793+0000 7fc0f6e69740 1 -- 172.21.15.76:0/2438478638 wait complete. 2024-03-12T05:56:46.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.793+0000 7fc0f6e69740 20 client.15861 trim_cache size 0 max 16384 2024-03-12T05:56:46.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.822+0000 7fc0f6e69740 1 Processor -- start 2024-03-12T05:56:46.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f6e69740 1 -- start start 2024-03-12T05:56:46.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f6e69740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee12b0 0x5619c4ee9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f6e69740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee1bd0 0x5619c4eea160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f6e69740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4eea7b0 0x5619c4ee5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f6e69740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c4ed5e50 con 0x5619c4ee12b0 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f6e69740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c4edc9e0 con 0x5619c4eea7b0 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f6e69740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c4ee22e0 con 0x5619c4ee1bd0 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f2825640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee12b0 0x5619c4ee9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f2825640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee12b0 0x5619c4ee9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41278/0 (socket says 172.21.15.76:41278) 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f2825640 1 -- 172.21.15.76:0/1331519857 learned_addr learned my addr 172.21.15.76:0/1331519857 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f2024640 1 --2- 172.21.15.76:0/1331519857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee1bd0 0x5619c4eea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f2825640 1 -- 172.21.15.76:0/1331519857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee1bd0 msgr2=0x5619c4eea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f2825640 1 --2- 172.21.15.76:0/1331519857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee1bd0 0x5619c4eea160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f2825640 1 -- 172.21.15.76:0/1331519857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4eea7b0 msgr2=0x5619c4ee5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f2825640 1 --2- 172.21.15.76:0/1331519857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4eea7b0 0x5619c4ee5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.823+0000 7fc0f2825640 1 -- 172.21.15.76:0/1331519857 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619c4de3ac0 con 0x5619c4ee12b0 2024-03-12T05:56:46.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.824+0000 7fc0f2825640 1 --2- 172.21.15.76:0/1331519857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee12b0 0x5619c4ee9c20 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc0e401b040 tx=0x7fc0e4011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe81df1206ec2e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:46.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.824+0000 7fc0d7fff640 1 -- 172.21.15.76:0/1331519857 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4003b70 con 0x5619c4ee12b0 2024-03-12T05:56:46.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.824+0000 7fc0d7fff640 1 -- 172.21.15.76:0/1331519857 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0e40042b0 con 0x5619c4ee12b0 2024-03-12T05:56:46.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.824+0000 7fc0d7fff640 1 -- 172.21.15.76:0/1331519857 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e400fc50 con 0x5619c4ee12b0 2024-03-12T05:56:46.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.824+0000 7fc0f6e69740 1 -- 172.21.15.76:0/1331519857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee12b0 msgr2=0x5619c4ee9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.824+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/1331519857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee12b0 0x5619c4ee9c20 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc0e401b040 tx=0x7fc0e4011a30 comp rx=0 tx=0).stop 2024-03-12T05:56:46.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.825+0000 7fc0f6e69740 1 -- 172.21.15.76:0/1331519857 shutdown_connections 2024-03-12T05:56:46.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.825+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/1331519857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4eea7b0 0x5619c4ee5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.825+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/1331519857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee1bd0 0x5619c4eea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.825+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/1331519857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4ee12b0 0x5619c4ee9c20 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.825+0000 7fc0f6e69740 1 -- 172.21.15.76:0/1331519857 >> 172.21.15.76:0/1331519857 conn(0x5619c4edb990 msgr2=0x5619c4ee27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:46.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.825+0000 7fc0f6e69740 1 -- 172.21.15.76:0/1331519857 shutdown_connections 2024-03-12T05:56:46.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.825+0000 7fc0f6e69740 1 -- 172.21.15.76:0/1331519857 wait complete. 2024-03-12T05:56:46.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.825+0000 7fc0f6e69740 1 -- start start 2024-03-12T05:56:46.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.825+0000 7fc0f6e69740 1 Processor -- start 2024-03-12T05:56:46.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.825+0000 7fc0f6e69740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4ee12b0 0x5619c4ee4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.825+0000 7fc0f6e69740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee1bd0 0x5619c4ee4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.825+0000 7fc0f6e69740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4eea7b0 0x5619c4f59650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f6e69740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c4ee55f0 con 0x5619c4eea7b0 2024-03-12T05:56:46.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f6e69740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c4f59bc0 con 0x5619c4ee12b0 2024-03-12T05:56:46.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f6e69740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c4f59d80 con 0x5619c4ee1bd0 2024-03-12T05:56:46.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f2825640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4eea7b0 0x5619c4f59650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f2825640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4eea7b0 0x5619c4f59650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41284/0 (socket says 172.21.15.76:41284) 2024-03-12T05:56:46.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f2825640 1 -- 172.21.15.76:0/99555046 learned_addr learned my addr 172.21.15.76:0/99555046 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:56:46.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f6e69740 10 client.0 _ll_register_callbacks cb 0x7fc0e4009cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:56:46.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f2024640 1 --2- 172.21.15.76:0/99555046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee1bd0 0x5619c4ee4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f2825640 1 --2- 172.21.15.76:0/99555046 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4ee12b0 0x5619c4ee4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f2825640 1 -- 172.21.15.76:0/99555046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee1bd0 msgr2=0x5619c4ee4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f2825640 1 --2- 172.21.15.76:0/99555046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee1bd0 0x5619c4ee4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f2825640 1 -- 172.21.15.76:0/99555046 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4ee12b0 msgr2=0x5619c4ee4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:56:46.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f2825640 1 --2- 172.21.15.76:0/99555046 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4ee12b0 0x5619c4ee4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f2825640 1 -- 172.21.15.76:0/99555046 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619c4e07570 con 0x5619c4eea7b0 2024-03-12T05:56:46.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f2024640 1 --2- 172.21.15.76:0/99555046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee1bd0 0x5619c4ee4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:56:46.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f2825640 1 --2- 172.21.15.76:0/99555046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4eea7b0 0x5619c4f59650 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4005c40 tx=0x7fc0e400f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14c5bb3b303159f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:56:46.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:56:46.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0d5ffb640 1 -- 172.21.15.76:0/99555046 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e400fc40 con 0x5619c4eea7b0 2024-03-12T05:56:46.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0d5ffb640 1 -- 172.21.15.76:0/99555046 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc0e4004720 con 0x5619c4eea7b0 2024-03-12T05:56:46.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.826+0000 7fc0f6e69740 1 -- 172.21.15.76:0/99555046 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5619c4b487e0 con 0x5619c4eea7b0 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.827+0000 7fc0d5ffb640 1 -- 172.21.15.76:0/99555046 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0e4003d80 con 0x5619c4eea7b0 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.827+0000 7fc0f6e69740 10 client.15867 fetch_fsmap learned FSMap version 22 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.827+0000 7fc0f6e69740 1 -- 172.21.15.76:0/99555046 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5619c4e096a0 con 0x5619c4eea7b0 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.827+0000 7fc0d5ffb640 1 -- 172.21.15.76:0/99555046 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0e4005000 con 0x5619c4eea7b0 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.827+0000 7fc0d5ffb640 1 -- 172.21.15.76:0/99555046 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc0e4048450 con 0x5619c4eea7b0 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.827+0000 7fc0f6e69740 10 client.15867 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.827+0000 7fc0f6e69740 10 client.15867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.827+0000 7fc0f6e69740 20 client.15867 populate_metadata read hostname 'smithi076' 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.827+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/99555046 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5619c4efa940 0x5619c4f5d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.827+0000 7fc0f6e69740 4 client.15867 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.827+0000 7fc0f6e69740 1 -- 172.21.15.76:0/99555046 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5619c4b487e0 con 0x5619c4efa940 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.828+0000 7fc0f2024640 1 --2- 172.21.15.76:0/99555046 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5619c4efa940 0x5619c4f5d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.828+0000 7fc0d5ffb640 10 client.15867 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.828+0000 7fc0f2024640 1 --2- 172.21.15.76:0/99555046 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5619c4efa940 0x5619c4f5d2d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7b87be362bbf37e server_cookie=446000c164244785 in_seq=0 out_seq=0 2024-03-12T05:56:46.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.828+0000 7fc0d5ffb640 1 -- 172.21.15.76:0/99555046 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5619c4b487e0 con 0x5619c4efa940 2024-03-12T05:56:46.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.828+0000 7fc0d5ffb640 10 client.15867 handle_command_reply: tid=0 2024-03-12T05:56:46.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.828+0000 7fc0f6e69740 1 client.15867 shutdown 2024-03-12T05:56:46.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.828+0000 7fc0f6e69740 1 -- 172.21.15.76:0/99555046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4eea7b0 msgr2=0x5619c4f59650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.828+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/99555046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4eea7b0 0x5619c4f59650 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4005c40 tx=0x7fc0e400f7f0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.828+0000 7fc0f6e69740 1 -- 172.21.15.76:0/99555046 shutdown_connections 2024-03-12T05:56:46.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.828+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/99555046 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5619c4efa940 0x5619c4f5d2d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.828+0000 7fc0d5ffb640 0 client.15867 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:56:46.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.828+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/99555046 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5619c4eea7b0 0x5619c4f59650 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/99555046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619c4ee1bd0 0x5619c4ee4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f6e69740 1 --2- 172.21.15.76:0/99555046 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5619c4ee12b0 0x5619c4ee4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f6e69740 1 -- 172.21.15.76:0/99555046 >> 172.21.15.76:0/99555046 conn(0x5619c4edb990 msgr2=0x5619c4ee59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:46.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f6e69740 1 -- 172.21.15.76:0/99555046 shutdown_connections 2024-03-12T05:56:46.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f6e69740 1 -- 172.21.15.76:0/99555046 wait complete. 2024-03-12T05:56:46.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f6e69740 20 client.15867 trim_cache size 0 max 16384 2024-03-12T05:56:46.839 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:56:46.839 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:56:46.839 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T05:56:46.839 INFO:teuthology.orchestra.run.smithi076.stdout: "ef5e7405-f201-46e6-9645-62bee7d6b192": { 2024-03-12T05:56:46.839 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T05:56:46.839 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "ef5e7405-f201-46e6-9645-62bee7d6b192", 2024-03-12T05:56:46.839 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T05:56:46.839 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:56:46.839 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:56:46.839 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:56:46.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f4288640 1 -- 172.21.15.76:0/1013962859 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc0d80a1cc0 msgr2=0x7fc0d80a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f4288640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc0d80a1cc0 0x7fc0d80a4180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc0ed680 tx=0x7fc0dc009260 comp rx=0 tx=0).stop 2024-03-12T05:56:46.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f4288640 1 -- 172.21.15.76:0/1013962859 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0ec105360 msgr2=0x7fc0ec19da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:56:46.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f4288640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0ec105360 0x7fc0ec19da60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc009230 tx=0x7fc0dc02c5d0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f4288640 1 -- 172.21.15.76:0/1013962859 shutdown_connections 2024-03-12T05:56:46.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f4288640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc0d80a1cc0 0x7fc0d80a4180 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f4288640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0ec1976d0 0x7fc0ec197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f4288640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc0ec105360 0x7fc0ec19da60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f4288640 1 --2- 172.21.15.76:0/1013962859 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc0ec100290 0x7fc0ec19d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:56:46.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f4288640 1 -- 172.21.15.76:0/1013962859 >> 172.21.15.76:0/1013962859 conn(0x7fc0ec077140 msgr2=0x7fc0ec103fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:56:46.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f4288640 1 -- 172.21.15.76:0/1013962859 shutdown_connections 2024-03-12T05:56:46.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:56:46.829+0000 7fc0f4288640 1 -- 172.21.15.76:0/1013962859 wait complete. 2024-03-12T05:56:46.845 INFO:tasks.cephfs.filesystem:scrub status for tag:ef5e7405-f201-46e6-9645-62bee7d6b192 - {'path': '/', 'tag': 'ef5e7405-f201-46e6-9645-62bee7d6b192', 'options': 'recursive,force'} 2024-03-12T05:56:48.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:48 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T05:56:48.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:48 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:48.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:48 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:56:48.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:48 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:48.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:48 smithi082 ceph-mon[36382]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:48.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:48 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:56:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:48 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T05:56:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:48 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:48 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:56:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:48 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:48 smithi088 ceph-mon[36585]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:48 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:56:48.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:48 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T05:56:48.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:48 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:48.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:48 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:56:48.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:48 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:56:48.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:48 smithi076 ceph-mon[32095]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:56:48.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:48 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:56:50.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:50 smithi082 ceph-mon[36382]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:56:50.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:50 smithi088 ceph-mon[36585]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:56:50.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:50 smithi076 ceph-mon[32095]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:56:52.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:52 smithi082 ceph-mon[36382]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:56:52.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:52 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:56:52.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:52 smithi088 ceph-mon[36585]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:56:52.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:52 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:56:52.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:52 smithi076 ceph-mon[32095]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:56:52.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:52 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:56:53.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:53 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:56:53.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:53 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:56:53.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:53 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:56:53.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:53 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:56:53.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:53 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:56:53.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:53 smithi082 ceph-mon[36382]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:56:53.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:53 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:56:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:53 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:56:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:53 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:56:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:53 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:56:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:53 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:56:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:53 smithi088 ceph-mon[36585]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:56:53.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:53 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:56:53.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:53 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:56:53.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:53 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:56:53.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:53 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:56:53.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:53 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:56:53.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:53 smithi076 ceph-mon[32095]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:56:54.137 INFO:tasks.workunit.client.0.smithi076.stderr:Note: switching to 'b6a4997be2b8bb74411268cc42753f29ae9a78ad'. 2024-03-12T05:56:54.137 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-12T05:56:54.137 INFO:tasks.workunit.client.0.smithi076.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-12T05:56:54.137 INFO:tasks.workunit.client.0.smithi076.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-12T05:56:54.137 INFO:tasks.workunit.client.0.smithi076.stderr:state without impacting any branches by switching back to a branch. 2024-03-12T05:56:54.137 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-12T05:56:54.137 INFO:tasks.workunit.client.0.smithi076.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-12T05:56:54.137 INFO:tasks.workunit.client.0.smithi076.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-12T05:56:54.137 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-12T05:56:54.137 INFO:tasks.workunit.client.0.smithi076.stderr: git switch -c 2024-03-12T05:56:54.137 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-12T05:56:54.137 INFO:tasks.workunit.client.0.smithi076.stderr:Or undo this operation with: 2024-03-12T05:56:54.138 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-12T05:56:54.138 INFO:tasks.workunit.client.0.smithi076.stderr: git switch - 2024-03-12T05:56:54.138 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-12T05:56:54.138 INFO:tasks.workunit.client.0.smithi076.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-12T05:56:54.138 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-12T05:56:54.138 INFO:tasks.workunit.client.0.smithi076.stderr:HEAD is now at b6a4997be2b Merge pull request #56060 from rhcs-dashboard/wip-64808-squid 2024-03-12T05:56:54.140 DEBUG:teuthology.orchestra.run.smithi076:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-12T05:56:54.168 INFO:tasks.workunit.client.0.smithi076.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-12T05:56:54.170 INFO:tasks.workunit.client.0.smithi076.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-12T05:56:54.170 INFO:tasks.workunit.client.0.smithi076.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-12T05:56:54.969 INFO:tasks.workunit.client.0.smithi076.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-12T05:56:55.037 INFO:tasks.workunit.client.0.smithi076.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-12T05:56:55.077 INFO:tasks.workunit.client.0.smithi076.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-12T05:56:55.078 INFO:tasks.workunit.client.0.smithi076.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-12T05:56:55.078 INFO:tasks.workunit.client.0.smithi076.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-12T05:56:55.118 INFO:tasks.workunit.client.0.smithi076.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-12T05:56:55.121 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T05:56:55.122 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-12T05:56:55.177 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-03-12T05:56:55.179 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-03-12T05:56:55.179 DEBUG:teuthology.orchestra.run.smithi076:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6a4997be2b8bb74411268cc42753f29ae9a78ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-03-12T05:56:55.238 INFO:tasks.workunit.client.0.smithi076.stderr:+ set -e 2024-03-12T05:56:55.238 INFO:tasks.workunit.client.0.smithi076.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-03-12T05:56:55.240 INFO:tasks.workunit.client.0.smithi076.stderr:Cloning into 'xfstests-dev'... 2024-03-12T05:56:56.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:56 smithi082 ceph-mon[36382]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:56:56.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:56 smithi088 ceph-mon[36585]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:56:56.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:56 smithi076 ceph-mon[32095]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:56:58.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:56:58 smithi082 ceph-mon[36382]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:56:58.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:56:58 smithi088 ceph-mon[36585]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:56:58.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:56:58 smithi076 ceph-mon[32095]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:57:00.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:00 smithi082 ceph-mon[36382]: pgmap v39: 97 pgs: 97 active+clean; 14 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-03-12T05:57:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:00 smithi088 ceph-mon[36585]: pgmap v39: 97 pgs: 97 active+clean; 14 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-03-12T05:57:00.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:00 smithi076 ceph-mon[32095]: pgmap v39: 97 pgs: 97 active+clean; 14 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-03-12T05:57:02.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:02 smithi082 ceph-mon[36382]: pgmap v40: 97 pgs: 97 active+clean; 14 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-03-12T05:57:02.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:02 smithi088 ceph-mon[36585]: pgmap v40: 97 pgs: 97 active+clean; 14 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-03-12T05:57:02.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:02 smithi076 ceph-mon[32095]: pgmap v40: 97 pgs: 97 active+clean; 14 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 1.1 MiB/s wr, 8 op/s 2024-03-12T05:57:04.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:04 smithi082 ceph-mon[36382]: pgmap v41: 97 pgs: 97 active+clean; 15 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 30 op/s 2024-03-12T05:57:04.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:04 smithi088 ceph-mon[36585]: pgmap v41: 97 pgs: 97 active+clean; 15 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 30 op/s 2024-03-12T05:57:04.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:04 smithi076 ceph-mon[32095]: pgmap v41: 97 pgs: 97 active+clean; 15 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 30 op/s 2024-03-12T05:57:06.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:06 smithi082 ceph-mon[36382]: pgmap v42: 97 pgs: 97 active+clean; 15 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 28 op/s 2024-03-12T05:57:06.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:06 smithi088 ceph-mon[36585]: pgmap v42: 97 pgs: 97 active+clean; 15 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 28 op/s 2024-03-12T05:57:06.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:06 smithi076 ceph-mon[32095]: pgmap v42: 97 pgs: 97 active+clean; 15 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 28 op/s 2024-03-12T05:57:08.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:08 smithi082 ceph-mon[36382]: pgmap v43: 97 pgs: 97 active+clean; 15 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 28 op/s 2024-03-12T05:57:08.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:08 smithi088 ceph-mon[36585]: pgmap v43: 97 pgs: 97 active+clean; 15 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 28 op/s 2024-03-12T05:57:08.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:08 smithi076 ceph-mon[32095]: pgmap v43: 97 pgs: 97 active+clean; 15 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 28 op/s 2024-03-12T05:57:10.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:10 smithi082 ceph-mon[36382]: pgmap v44: 97 pgs: 97 active+clean; 18 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 36 op/s 2024-03-12T05:57:10.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:10 smithi088 ceph-mon[36585]: pgmap v44: 97 pgs: 97 active+clean; 18 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 36 op/s 2024-03-12T05:57:10.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:10 smithi076 ceph-mon[32095]: pgmap v44: 97 pgs: 97 active+clean; 18 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 36 op/s 2024-03-12T05:57:12.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:12 smithi082 ceph-mon[36382]: pgmap v45: 97 pgs: 97 active+clean; 18 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 384 KiB/s wr, 30 op/s 2024-03-12T05:57:12.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:12 smithi088 ceph-mon[36585]: pgmap v45: 97 pgs: 97 active+clean; 18 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 384 KiB/s wr, 30 op/s 2024-03-12T05:57:12.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:12 smithi076 ceph-mon[32095]: pgmap v45: 97 pgs: 97 active+clean; 18 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 384 KiB/s wr, 30 op/s 2024-03-12T05:57:14.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:14 smithi082 ceph-mon[36382]: pgmap v46: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 651 KiB/s wr, 49 op/s 2024-03-12T05:57:14.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:14 smithi088 ceph-mon[36585]: pgmap v46: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 651 KiB/s wr, 49 op/s 2024-03-12T05:57:14.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:14 smithi076 ceph-mon[32095]: pgmap v46: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 651 KiB/s wr, 49 op/s 2024-03-12T05:57:16.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:16 smithi082 ceph-mon[36382]: pgmap v47: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 513 KiB/s wr, 26 op/s 2024-03-12T05:57:16.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:16 smithi088 ceph-mon[36585]: pgmap v47: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 513 KiB/s wr, 26 op/s 2024-03-12T05:57:16.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:16 smithi076 ceph-mon[32095]: pgmap v47: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 513 KiB/s wr, 26 op/s 2024-03-12T05:57:16.847 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:57:16.847 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:57:16.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.958+0000 7eff276fb640 1 -- 172.21.15.76:0/535459728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff20100280 msgr2=0x7eff20100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:16.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.958+0000 7eff276fb640 1 --2- 172.21.15.76:0/535459728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff20100280 0x7eff20100660 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7eff1c009930 tx=0x7eff1c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:57:16.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.958+0000 7eff276fb640 1 -- 172.21.15.76:0/535459728 shutdown_connections 2024-03-12T05:57:16.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.958+0000 7eff276fb640 1 --2- 172.21.15.76:0/535459728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff20105350 0x7eff2010cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.958+0000 7eff276fb640 1 --2- 172.21.15.76:0/535459728 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7eff20100c30 0x7eff20104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.958+0000 7eff276fb640 1 --2- 172.21.15.76:0/535459728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff20100280 0x7eff20100660 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.958+0000 7eff276fb640 1 -- 172.21.15.76:0/535459728 >> 172.21.15.76:0/535459728 conn(0x7eff200fbac0 msgr2=0x7eff200fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:16.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.958+0000 7eff276fb640 1 -- 172.21.15.76:0/535459728 shutdown_connections 2024-03-12T05:57:16.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.958+0000 7eff276fb640 1 -- 172.21.15.76:0/535459728 wait complete. 2024-03-12T05:57:16.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff276fb640 1 Processor -- start 2024-03-12T05:57:16.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff276fb640 1 -- start start 2024-03-12T05:57:16.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff276fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff20100280 0x7eff2019d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff276fb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff20105350 0x7eff2019d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff276fb640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7eff20197630 0x7eff20197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff276fb640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff20198020 con 0x7eff20105350 2024-03-12T05:57:16.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff276fb640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff20198190 con 0x7eff20197630 2024-03-12T05:57:16.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff276fb640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff20198350 con 0x7eff20100280 2024-03-12T05:57:16.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff24c96640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff20105350 0x7eff2019d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff24c96640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff20105350 0x7eff2019d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47704/0 (socket says 172.21.15.76:47704) 2024-03-12T05:57:16.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff24c96640 1 -- 172.21.15.76:0/509225110 learned_addr learned my addr 172.21.15.76:0/509225110 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:16.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff25c98640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7eff20197630 0x7eff20197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff25497640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff20100280 0x7eff2019d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff24c96640 1 -- 172.21.15.76:0/509225110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff20100280 msgr2=0x7eff2019d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:16.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff24c96640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff20100280 0x7eff2019d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff24c96640 1 -- 172.21.15.76:0/509225110 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7eff20197630 msgr2=0x7eff20197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:16.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff24c96640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7eff20197630 0x7eff20197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff24c96640 1 -- 172.21.15.76:0/509225110 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1c009560 con 0x7eff20105350 2024-03-12T05:57:16.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.959+0000 7eff25c98640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7eff20197630 0x7eff20197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:16.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.960+0000 7eff24c96640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff20105350 0x7eff2019d970 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7eff1000e9c0 tx=0x7eff1000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:16.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.960+0000 7eff167fc640 1 -- 172.21.15.76:0/509225110 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10009700 con 0x7eff20105350 2024-03-12T05:57:16.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.960+0000 7eff167fc640 1 -- 172.21.15.76:0/509225110 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff1000cd80 con 0x7eff20105350 2024-03-12T05:57:16.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.960+0000 7eff167fc640 1 -- 172.21.15.76:0/509225110 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10010430 con 0x7eff20105350 2024-03-12T05:57:16.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.960+0000 7eff276fb640 1 -- 172.21.15.76:0/509225110 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff201a42f0 con 0x7eff20105350 2024-03-12T05:57:16.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.960+0000 7eff276fb640 1 -- 172.21.15.76:0/509225110 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff201a4800 con 0x7eff20105350 2024-03-12T05:57:16.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.962+0000 7eff167fc640 1 -- 172.21.15.76:0/509225110 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7eff10005000 con 0x7eff20105350 2024-03-12T05:57:16.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.963+0000 7eff2a3ae740 1 Processor -- start 2024-03-12T05:57:16.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.963+0000 7eff2a3ae740 1 -- start start 2024-03-12T05:57:16.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.963+0000 7eff2a3ae740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fd3a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.963+0000 7eff2a3ae740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 0x56517fd47680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.963+0000 7eff2a3ae740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 0x56517fd49fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.963+0000 7eff2a3ae740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56517fca9280 con 0x56517fd39fe0 2024-03-12T05:57:16.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.963+0000 7eff2a3ae740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56517fca67b0 con 0x56517fd3a900 2024-03-12T05:57:16.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.963+0000 7eff2a3ae740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56517fca1f90 con 0x56517fd47bc0 2024-03-12T05:57:16.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.963+0000 7eff167fc640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7eff0c0a1cc0 0x7eff0c0a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.963+0000 7eff25c98640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fd3a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.963+0000 7eff25c98640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fd3a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47710/0 (socket says 172.21.15.76:47710) 2024-03-12T05:57:16.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.963+0000 7eff25c98640 1 -- 172.21.15.76:0/2166409931 learned_addr learned my addr 172.21.15.76:0/2166409931 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:16.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.963+0000 7eff25c98640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7eff0c0a1cc0 0x7eff0c0a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.963+0000 7eff167fc640 1 -- 172.21.15.76:0/509225110 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7eff10014070 con 0x7eff20105350 2024-03-12T05:57:16.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.964+0000 7eff24c96640 1 --2- 172.21.15.76:0/2166409931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 0x56517fd49fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.964+0000 7eff25497640 1 --2- 172.21.15.76:0/2166409931 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 0x56517fd47680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.964+0000 7eff25c98640 1 -- 172.21.15.76:0/2166409931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 msgr2=0x56517fd49fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:16.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.964+0000 7eff25c98640 1 --2- 172.21.15.76:0/2166409931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 0x56517fd49fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.964+0000 7eff25c98640 1 -- 172.21.15.76:0/2166409931 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 msgr2=0x56517fd47680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:16.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.964+0000 7eff25c98640 1 --2- 172.21.15.76:0/2166409931 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 0x56517fd47680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.964+0000 7eff25c98640 1 -- 172.21.15.76:0/2166409931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56517fc60570 con 0x56517fd39fe0 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.964+0000 7eff25c98640 1 --2- 172.21.15.76:0/2166409931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fd3a3c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7eff18012d90 tx=0x7eff180107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d559ca331e6ff713 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.964+0000 7eff25c98640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7eff0c0a1cc0 0x7eff0c0a4180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7eff18010ca0 tx=0x7eff1801d000 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.964+0000 7eff0b7fe640 1 -- 172.21.15.76:0/2166409931 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff18012870 con 0x56517fd39fe0 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.964+0000 7eff0b7fe640 1 -- 172.21.15.76:0/2166409931 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff18009d00 con 0x56517fd39fe0 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.964+0000 7eff0b7fe640 1 -- 172.21.15.76:0/2166409931 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff180054c0 con 0x56517fd39fe0 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.965+0000 7eff2a3ae740 1 -- 172.21.15.76:0/2166409931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 msgr2=0x56517fd3a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.965+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/2166409931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fd3a3c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7eff18012d90 tx=0x7eff180107a0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.965+0000 7eff2a3ae740 1 -- 172.21.15.76:0/2166409931 shutdown_connections 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.965+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/2166409931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 0x56517fd49fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.965+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/2166409931 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 0x56517fd47680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.965+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/2166409931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fd3a3c0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.965+0000 7eff2a3ae740 1 -- 172.21.15.76:0/2166409931 >> 172.21.15.76:0/2166409931 conn(0x56517fd34990 msgr2=0x56517fd34d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.965+0000 7eff2a3ae740 1 -- 172.21.15.76:0/2166409931 shutdown_connections 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.965+0000 7eff2a3ae740 1 -- 172.21.15.76:0/2166409931 wait complete. 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.965+0000 7eff2a3ae740 1 -- start start 2024-03-12T05:57:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.965+0000 7eff2a3ae740 1 Processor -- start 2024-03-12T05:57:16.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff2a3ae740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fd44a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff2a3ae740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 0x56517fd44f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff2a3ae740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 0x56517fd454c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff2a3ae740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56517f975430 con 0x56517fd39fe0 2024-03-12T05:57:16.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff2a3ae740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56517f931e00 con 0x56517fd3a900 2024-03-12T05:57:16.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff2a3ae740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56517fc26fd0 con 0x56517fd47bc0 2024-03-12T05:57:16.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff2a3ae740 10 client.0 _ll_register_callbacks cb 0x56517fc7cfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:16.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff25497640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fd44a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff25497640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fd44a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47714/0 (socket says 172.21.15.76:47714) 2024-03-12T05:57:16.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff25497640 1 -- 172.21.15.76:0/3453403159 learned_addr learned my addr 172.21.15.76:0/3453403159 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:16.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff25c98640 1 --2- 172.21.15.76:0/3453403159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 0x56517fd454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff24c96640 1 --2- 172.21.15.76:0/3453403159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 0x56517fd44f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff25497640 1 -- 172.21.15.76:0/3453403159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 msgr2=0x56517fd454c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:16.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff25497640 1 --2- 172.21.15.76:0/3453403159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 0x56517fd454c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff25497640 1 -- 172.21.15.76:0/3453403159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 msgr2=0x56517fd44f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff25497640 1 --2- 172.21.15.76:0/3453403159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 0x56517fd44f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff25497640 1 -- 172.21.15.76:0/3453403159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff1801a040 con 0x56517fd39fe0 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff25497640 1 --2- 172.21.15.76:0/3453403159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fd44a40 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7eff1c006010 tx=0x7eff1c004c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b2bddb28d34ec1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff08ff9640 1 -- 172.21.15.76:0/3453403159 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c005d50 con 0x56517fd39fe0 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.966+0000 7eff08ff9640 1 -- 172.21.15.76:0/3453403159 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7eff1c02c5d0 con 0x56517fd39fe0 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.967+0000 7eff2a3ae740 1 -- 172.21.15.76:0/3453403159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56517f9a17e0 con 0x56517fd39fe0 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.967+0000 7eff08ff9640 1 -- 172.21.15.76:0/3453403159 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff1c004040 con 0x56517fd39fe0 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.967+0000 7eff2a3ae740 10 client.15879 fetch_fsmap learned FSMap version 22 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.967+0000 7eff2a3ae740 1 -- 172.21.15.76:0/3453403159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x56517fca0610 con 0x56517fd39fe0 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.967+0000 7eff08ff9640 1 -- 172.21.15.76:0/3453403159 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56517f9a17e0 con 0x56517fd39fe0 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.967+0000 7eff08ff9640 1 -- 172.21.15.76:0/3453403159 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7eff1c03c040 con 0x56517fd39fe0 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.967+0000 7eff2a3ae740 10 client.15879 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.967+0000 7eff2a3ae740 10 client.15879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.967+0000 7eff2a3ae740 20 client.15879 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.967+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/3453403159 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56517fd57cc0 0x56517fd5a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.967+0000 7eff2a3ae740 4 client.15879 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.967+0000 7eff2a3ae740 1 -- 172.21.15.76:0/3453403159 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56517f9de720 con 0x56517fd57cc0 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.967+0000 7eff24c96640 1 --2- 172.21.15.76:0/3453403159 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56517fd57cc0 0x56517fd5a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.968+0000 7eff08ff9640 10 client.15879 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.968+0000 7eff24c96640 1 --2- 172.21.15.76:0/3453403159 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56517fd57cc0 0x56517fd5a0b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15ef018e24e27532 server_cookie=6f0c57316a3142f6 in_seq=0 out_seq=0 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.968+0000 7eff08ff9640 1 -- 172.21.15.76:0/3453403159 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56517f9de720 con 0x56517fd57cc0 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.968+0000 7eff08ff9640 10 client.15879 handle_command_reply: tid=0 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.968+0000 7eff2a3ae740 1 client.15879 shutdown 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.969+0000 7eff2a3ae740 1 -- 172.21.15.76:0/3453403159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 msgr2=0x56517fd44a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.969+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/3453403159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fd44a40 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7eff1c006010 tx=0x7eff1c004c10 comp rx=0 tx=0).stop 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.969+0000 7eff2a3ae740 1 -- 172.21.15.76:0/3453403159 shutdown_connections 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.969+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/3453403159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 0x56517fd454c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.969+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/3453403159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 0x56517fd44f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.969+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/3453403159 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56517fd57cc0 0x56517fd5a0b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.969+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/3453403159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fd44a40 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.969+0000 7eff08ff9640 0 client.15879 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:16.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.969+0000 7eff2a3ae740 1 -- 172.21.15.76:0/3453403159 >> 172.21.15.76:0/3453403159 conn(0x56517fd34990 msgr2=0x56517fd36730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:16.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.969+0000 7eff2a3ae740 1 -- 172.21.15.76:0/3453403159 shutdown_connections 2024-03-12T05:57:16.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.969+0000 7eff2a3ae740 1 -- 172.21.15.76:0/3453403159 wait complete. 2024-03-12T05:57:16.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.969+0000 7eff2a3ae740 20 client.15879 trim_cache size 0 max 16384 2024-03-12T05:57:16.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff2a3ae740 1 Processor -- start 2024-03-12T05:57:16.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff2a3ae740 1 -- start start 2024-03-12T05:57:16.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff2a3ae740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd39fe0 0x56517fd43ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff2a3ae740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd3a900 0x56517fd452e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff2a3ae740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd47bc0 0x56517fd364d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff2a3ae740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56517fd35790 con 0x56517fd3a900 2024-03-12T05:57:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff2a3ae740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56517fd440e0 con 0x56517fd47bc0 2024-03-12T05:57:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff2a3ae740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56517fd3f850 con 0x56517fd39fe0 2024-03-12T05:57:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff24c96640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd3a900 0x56517fd452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff24c96640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd3a900 0x56517fd452e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47716/0 (socket says 172.21.15.76:47716) 2024-03-12T05:57:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff24c96640 1 -- 172.21.15.76:0/722205858 learned_addr learned my addr 172.21.15.76:0/722205858 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff25c98640 1 --2- 172.21.15.76:0/722205858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd47bc0 0x56517fd364d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff25497640 1 --2- 172.21.15.76:0/722205858 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd39fe0 0x56517fd43ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff24c96640 1 -- 172.21.15.76:0/722205858 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd39fe0 msgr2=0x56517fd43ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff24c96640 1 --2- 172.21.15.76:0/722205858 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd39fe0 0x56517fd43ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff24c96640 1 -- 172.21.15.76:0/722205858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd47bc0 msgr2=0x56517fd364d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff24c96640 1 --2- 172.21.15.76:0/722205858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd47bc0 0x56517fd364d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff24c96640 1 -- 172.21.15.76:0/722205858 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56517fc3cac0 con 0x56517fd3a900 2024-03-12T05:57:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff25497640 1 --2- 172.21.15.76:0/722205858 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd39fe0 0x56517fd43ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:57:16.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.984+0000 7eff24c96640 1 --2- 172.21.15.76:0/722205858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd3a900 0x56517fd452e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7eff10096e50 tx=0x7eff10005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3105562fe9f0d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:16.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.985+0000 7eff0affd640 1 -- 172.21.15.76:0/722205858 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1009a9f0 con 0x56517fd3a900 2024-03-12T05:57:16.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.985+0000 7eff0affd640 1 -- 172.21.15.76:0/722205858 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff1009eb40 con 0x56517fd3a900 2024-03-12T05:57:16.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.985+0000 7eff0affd640 1 -- 172.21.15.76:0/722205858 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff100aa390 con 0x56517fd3a900 2024-03-12T05:57:16.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.985+0000 7eff2a3ae740 1 -- 172.21.15.76:0/722205858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd3a900 msgr2=0x56517fd452e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:16.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.985+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/722205858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd3a900 0x56517fd452e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7eff10096e50 tx=0x7eff10005690 comp rx=0 tx=0).stop 2024-03-12T05:57:16.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.985+0000 7eff2a3ae740 1 -- 172.21.15.76:0/722205858 shutdown_connections 2024-03-12T05:57:16.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.985+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/722205858 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd47bc0 0x56517fd364d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.985+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/722205858 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd3a900 0x56517fd452e0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.985+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/722205858 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd39fe0 0x56517fd43ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.985+0000 7eff2a3ae740 1 -- 172.21.15.76:0/722205858 >> 172.21.15.76:0/722205858 conn(0x56517fd34990 msgr2=0x56517fd35dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:16.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.985+0000 7eff2a3ae740 1 -- 172.21.15.76:0/722205858 shutdown_connections 2024-03-12T05:57:16.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.985+0000 7eff2a3ae740 1 -- 172.21.15.76:0/722205858 wait complete. 2024-03-12T05:57:16.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.985+0000 7eff2a3ae740 1 -- start start 2024-03-12T05:57:16.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff2a3ae740 1 Processor -- start 2024-03-12T05:57:16.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff2a3ae740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fda8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff2a3ae740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 0x56517fda8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff2a3ae740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 0x56517fd4de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff2a3ae740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56517fd4e440 con 0x56517fd39fe0 2024-03-12T05:57:16.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff2a3ae740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56517fd4e5b0 con 0x56517fd3a900 2024-03-12T05:57:16.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff2a3ae740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56517fd4e770 con 0x56517fd47bc0 2024-03-12T05:57:16.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff2a3ae740 10 client.0 _ll_register_callbacks cb 0x7eff18012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:16.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff25497640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fda8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff25497640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fda8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47730/0 (socket says 172.21.15.76:47730) 2024-03-12T05:57:16.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff25497640 1 -- 172.21.15.76:0/2178923314 learned_addr learned my addr 172.21.15.76:0/2178923314 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:16.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff24c96640 1 --2- 172.21.15.76:0/2178923314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 0x56517fda8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff25c98640 1 --2- 172.21.15.76:0/2178923314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 0x56517fd4de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff25497640 1 -- 172.21.15.76:0/2178923314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 msgr2=0x56517fd4de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:16.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff25497640 1 --2- 172.21.15.76:0/2178923314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 0x56517fd4de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff25497640 1 -- 172.21.15.76:0/2178923314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 msgr2=0x56517fda8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:16.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff25497640 1 --2- 172.21.15.76:0/2178923314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 0x56517fda8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:16.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff25497640 1 -- 172.21.15.76:0/2178923314 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff100ba040 con 0x56517fd39fe0 2024-03-12T05:57:16.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.986+0000 7eff25c98640 1 --2- 172.21.15.76:0/2178923314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 0x56517fd4de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:16.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff25497640 1 --2- 172.21.15.76:0/2178923314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fda8720 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7eff1c005940 tx=0x7eff1c004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e18802204a6ab6b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:16.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:16.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff08ff9640 1 -- 172.21.15.76:0/2178923314 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0039c0 con 0x56517fd39fe0 2024-03-12T05:57:16.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff08ff9640 1 -- 172.21.15.76:0/2178923314 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7eff1c000e40 con 0x56517fd39fe0 2024-03-12T05:57:16.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff2a3ae740 1 -- 172.21.15.76:0/2178923314 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff18009d00 con 0x56517fd39fe0 2024-03-12T05:57:16.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff08ff9640 1 -- 172.21.15.76:0/2178923314 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff1c02c5d0 con 0x56517fd39fe0 2024-03-12T05:57:16.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff2a3ae740 10 client.15885 fetch_fsmap learned FSMap version 22 2024-03-12T05:57:16.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff2a3ae740 1 -- 172.21.15.76:0/2178923314 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x56517fc626a0 con 0x56517fd39fe0 2024-03-12T05:57:16.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff08ff9640 1 -- 172.21.15.76:0/2178923314 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff1c05e020 con 0x56517fd39fe0 2024-03-12T05:57:16.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff08ff9640 1 -- 172.21.15.76:0/2178923314 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7eff1c004020 con 0x56517fd39fe0 2024-03-12T05:57:16.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff2a3ae740 10 client.15885 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:57:16.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff2a3ae740 10 client.15885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:16.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff2a3ae740 20 client.15885 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:16.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/2178923314 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56517fdb0440 0x56517fdb2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:16.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff2a3ae740 4 client.15885 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:57:16.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff2a3ae740 1 -- 172.21.15.76:0/2178923314 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56517fa2d890 con 0x56517fdb0440 2024-03-12T05:57:16.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.987+0000 7eff24c96640 1 --2- 172.21.15.76:0/2178923314 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56517fdb0440 0x56517fdb2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:16.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff24c96640 1 --2- 172.21.15.76:0/2178923314 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56517fdb0440 0x56517fdb2860 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f63466a8fbb1f688 server_cookie=162176148518f7c7 in_seq=0 out_seq=0 2024-03-12T05:57:16.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff08ff9640 10 client.15885 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:16.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff08ff9640 1 -- 172.21.15.76:0/2178923314 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56517fa2d890 con 0x56517fdb0440 2024-03-12T05:57:16.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff08ff9640 10 client.15885 handle_command_reply: tid=0 2024-03-12T05:57:16.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff2a3ae740 1 client.15885 shutdown 2024-03-12T05:57:17.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff2a3ae740 1 -- 172.21.15.76:0/2178923314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 msgr2=0x56517fda8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/2178923314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fda8720 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7eff1c005940 tx=0x7eff1c004b50 comp rx=0 tx=0).stop 2024-03-12T05:57:17.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff2a3ae740 1 -- 172.21.15.76:0/2178923314 shutdown_connections 2024-03-12T05:57:17.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/2178923314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56517fd47bc0 0x56517fd4de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/2178923314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56517fd3a900 0x56517fda8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/2178923314 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56517fdb0440 0x56517fdb2860 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff2a3ae740 1 --2- 172.21.15.76:0/2178923314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56517fd39fe0 0x56517fda8720 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff08ff9640 0 client.15885 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:17.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff2a3ae740 1 -- 172.21.15.76:0/2178923314 >> 172.21.15.76:0/2178923314 conn(0x56517fd34990 msgr2=0x56517fd33a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff2a3ae740 1 -- 172.21.15.76:0/2178923314 shutdown_connections 2024-03-12T05:57:17.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff2a3ae740 1 -- 172.21.15.76:0/2178923314 wait complete. 2024-03-12T05:57:17.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.988+0000 7eff2a3ae740 20 client.15885 trim_cache size 0 max 16384 2024-03-12T05:57:17.001 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:57:17.001 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T05:57:17.001 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T05:57:17.001 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:57:17.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.989+0000 7eff276fb640 1 -- 172.21.15.76:0/509225110 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7eff0c0a1cc0 msgr2=0x7eff0c0a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.989+0000 7eff276fb640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7eff0c0a1cc0 0x7eff0c0a4180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7eff18010ca0 tx=0x7eff1801d000 comp rx=0 tx=0).stop 2024-03-12T05:57:17.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.989+0000 7eff276fb640 1 -- 172.21.15.76:0/509225110 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff20105350 msgr2=0x7eff2019d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.989+0000 7eff276fb640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff20105350 0x7eff2019d970 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7eff1000e9c0 tx=0x7eff1000ee80 comp rx=0 tx=0).stop 2024-03-12T05:57:17.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.989+0000 7eff276fb640 1 -- 172.21.15.76:0/509225110 shutdown_connections 2024-03-12T05:57:17.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.989+0000 7eff276fb640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7eff0c0a1cc0 0x7eff0c0a4180 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.989+0000 7eff276fb640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7eff20197630 0x7eff20197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.989+0000 7eff276fb640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7eff20105350 0x7eff2019d970 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.989+0000 7eff276fb640 1 --2- 172.21.15.76:0/509225110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff20100280 0x7eff2019d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.989+0000 7eff276fb640 1 -- 172.21.15.76:0/509225110 >> 172.21.15.76:0/509225110 conn(0x7eff200fbac0 msgr2=0x7eff20069cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.989+0000 7eff276fb640 1 -- 172.21.15.76:0/509225110 shutdown_connections 2024-03-12T05:57:17.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:16.989+0000 7eff276fb640 1 -- 172.21.15.76:0/509225110 wait complete. 2024-03-12T05:57:17.005 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:57:17.005 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:57:17.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.106+0000 7ff686d3a640 1 -- 172.21.15.76:0/208249970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6800ff9f0 msgr2=0x7ff6800ffe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.106+0000 7ff686d3a640 1 --2- 172.21.15.76:0/208249970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6800ff9f0 0x7ff6800ffe70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff674009930 tx=0x7ff67402bc30 comp rx=0 tx=0).stop 2024-03-12T05:57:17.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.106+0000 7ff686d3a640 1 -- 172.21.15.76:0/208249970 shutdown_connections 2024-03-12T05:57:17.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.106+0000 7ff686d3a640 1 --2- 172.21.15.76:0/208249970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6801003b0 0x7ff68010e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.106+0000 7ff686d3a640 1 --2- 172.21.15.76:0/208249970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6800ff9f0 0x7ff6800ffe70 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.106+0000 7ff686d3a640 1 --2- 172.21.15.76:0/208249970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6801062c0 0x7ff6801066a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.106+0000 7ff686d3a640 1 -- 172.21.15.76:0/208249970 >> 172.21.15.76:0/208249970 conn(0x7ff680076540 msgr2=0x7ff680076950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.106+0000 7ff686d3a640 1 -- 172.21.15.76:0/208249970 shutdown_connections 2024-03-12T05:57:17.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.106+0000 7ff686d3a640 1 -- 172.21.15.76:0/208249970 wait complete. 2024-03-12T05:57:17.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff686d3a640 1 Processor -- start 2024-03-12T05:57:17.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff686d3a640 1 -- start start 2024-03-12T05:57:17.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff686d3a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6800ff9f0 0x7ff68019d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff686d3a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6801003b0 0x7ff68019db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff686d3a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6801062c0 0x7ff6801976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff686d3a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff680197e70 con 0x7ff6801062c0 2024-03-12T05:57:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff686d3a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff680197fe0 con 0x7ff6801003b0 2024-03-12T05:57:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff686d3a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6801981a0 con 0x7ff6800ff9f0 2024-03-12T05:57:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff6852d7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6801062c0 0x7ff6801976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff6852d7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6801062c0 0x7ff6801976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47742/0 (socket says 172.21.15.76:47742) 2024-03-12T05:57:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff6852d7640 1 -- 172.21.15.76:0/360230422 learned_addr learned my addr 172.21.15.76:0/360230422 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff684ad6640 1 --2- 172.21.15.76:0/360230422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6800ff9f0 0x7ff68019d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff67ffff640 1 --2- 172.21.15.76:0/360230422 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6801003b0 0x7ff68019db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff6852d7640 1 -- 172.21.15.76:0/360230422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6800ff9f0 msgr2=0x7ff68019d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff6852d7640 1 --2- 172.21.15.76:0/360230422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6800ff9f0 0x7ff68019d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff6852d7640 1 -- 172.21.15.76:0/360230422 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6801003b0 msgr2=0x7ff68019db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff6852d7640 1 --2- 172.21.15.76:0/360230422 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6801003b0 0x7ff68019db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.107+0000 7ff6852d7640 1 -- 172.21.15.76:0/360230422 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff674009560 con 0x7ff6801062c0 2024-03-12T05:57:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.108+0000 7ff6852d7640 1 --2- 172.21.15.76:0/360230422 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6801062c0 0x7ff6801976e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff67800eaa0 tx=0x7ff67800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.108+0000 7ff67dffb640 1 -- 172.21.15.76:0/360230422 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67800cdb0 con 0x7ff6801062c0 2024-03-12T05:57:17.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.108+0000 7ff67dffb640 1 -- 172.21.15.76:0/360230422 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff678005020 con 0x7ff6801062c0 2024-03-12T05:57:17.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.108+0000 7ff67dffb640 1 -- 172.21.15.76:0/360230422 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6780116b0 con 0x7ff6801062c0 2024-03-12T05:57:17.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.108+0000 7ff686d3a640 1 -- 172.21.15.76:0/360230422 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff680198480 con 0x7ff6801062c0 2024-03-12T05:57:17.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.108+0000 7ff686d3a640 1 -- 172.21.15.76:0/360230422 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6801a45a0 con 0x7ff6801062c0 2024-03-12T05:57:17.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.109+0000 7ff686d3a640 1 -- 172.21.15.76:0/360230422 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff680069440 con 0x7ff6801062c0 2024-03-12T05:57:17.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.110+0000 7ff67dffb640 1 -- 172.21.15.76:0/360230422 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7ff678005520 con 0x7ff6801062c0 2024-03-12T05:57:17.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.111+0000 7ff67dffb640 1 --2- 172.21.15.76:0/360230422 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff6680a1d40 0x7ff6680a4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.111+0000 7ff684ad6640 1 --2- 172.21.15.76:0/360230422 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff6680a1d40 0x7ff6680a4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.111+0000 7ff67dffb640 1 -- 172.21.15.76:0/360230422 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff6780cdbf0 con 0x7ff6801062c0 2024-03-12T05:57:17.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.111+0000 7ff684ad6640 1 --2- 172.21.15.76:0/360230422 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff6680a1d40 0x7ff6680a4200 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff67000bbe0 tx=0x7ff67000b390 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.115+0000 7ff67dffb640 1 -- 172.21.15.76:0/360230422 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff6780968f0 con 0x7ff6801062c0 2024-03-12T05:57:17.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.342+0000 7ff686d3a640 1 -- 172.21.15.76:0/360230422 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff680068450 con 0x7ff6801062c0 2024-03-12T05:57:17.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.343+0000 7ff67dffb640 1 -- 172.21.15.76:0/360230422 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11328 (secure 0 0 0) 0x7ff678099fb0 con 0x7ff6801062c0 2024-03-12T05:57:17.348 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:57:17.348 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:55:33.056004+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:57:17.348 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 22 2024-03-12T05:57:17.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.345+0000 7ff686d3a640 1 -- 172.21.15.76:0/360230422 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff6680a1d40 msgr2=0x7ff6680a4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.345+0000 7ff686d3a640 1 --2- 172.21.15.76:0/360230422 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff6680a1d40 0x7ff6680a4200 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff67000bbe0 tx=0x7ff67000b390 comp rx=0 tx=0).stop 2024-03-12T05:57:17.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.345+0000 7ff686d3a640 1 -- 172.21.15.76:0/360230422 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6801062c0 msgr2=0x7ff6801976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.345+0000 7ff686d3a640 1 --2- 172.21.15.76:0/360230422 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6801062c0 0x7ff6801976e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff67800eaa0 tx=0x7ff67800ef60 comp rx=0 tx=0).stop 2024-03-12T05:57:17.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.345+0000 7ff686d3a640 1 -- 172.21.15.76:0/360230422 shutdown_connections 2024-03-12T05:57:17.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.345+0000 7ff686d3a640 1 --2- 172.21.15.76:0/360230422 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff6680a1d40 0x7ff6680a4200 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.345+0000 7ff686d3a640 1 --2- 172.21.15.76:0/360230422 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6801062c0 0x7ff6801976e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.345+0000 7ff686d3a640 1 --2- 172.21.15.76:0/360230422 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6801003b0 0x7ff68019db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.345+0000 7ff686d3a640 1 --2- 172.21.15.76:0/360230422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6800ff9f0 0x7ff68019d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.345+0000 7ff686d3a640 1 -- 172.21.15.76:0/360230422 >> 172.21.15.76:0/360230422 conn(0x7ff680076540 msgr2=0x7ff6800fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.345+0000 7ff686d3a640 1 -- 172.21.15.76:0/360230422 shutdown_connections 2024-03-12T05:57:17.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.345+0000 7ff686d3a640 1 -- 172.21.15.76:0/360230422 wait complete. 2024-03-12T05:57:17.361 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:57:17.361 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:57:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.466+0000 7f7c2ba4d640 1 -- 172.21.15.76:0/2940248482 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c24100290 msgr2=0x7f7c24100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.466+0000 7f7c2ba4d640 1 --2- 172.21.15.76:0/2940248482 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c24100290 0x7f7c24100670 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7c20009930 tx=0x7f7c2002bc30 comp rx=0 tx=0).stop 2024-03-12T05:57:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.466+0000 7f7c2ba4d640 1 -- 172.21.15.76:0/2940248482 shutdown_connections 2024-03-12T05:57:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.466+0000 7f7c2ba4d640 1 --2- 172.21.15.76:0/2940248482 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c24105360 0x7f7c2410cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.466+0000 7f7c2ba4d640 1 --2- 172.21.15.76:0/2940248482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c24100c40 0x7f7c24104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.466+0000 7f7c2ba4d640 1 --2- 172.21.15.76:0/2940248482 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c24100290 0x7f7c24100670 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.466+0000 7f7c2ba4d640 1 -- 172.21.15.76:0/2940248482 >> 172.21.15.76:0/2940248482 conn(0x7f7c24077230 msgr2=0x7f7c24077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.466+0000 7f7c2ba4d640 1 -- 172.21.15.76:0/2940248482 shutdown_connections 2024-03-12T05:57:17.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.466+0000 7f7c2ba4d640 1 -- 172.21.15.76:0/2940248482 wait complete. 2024-03-12T05:57:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.467+0000 7f7c2ba4d640 1 Processor -- start 2024-03-12T05:57:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.467+0000 7f7c2ba4d640 1 -- start start 2024-03-12T05:57:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.467+0000 7f7c2ba4d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c24100290 0x7f7c2419d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.467+0000 7f7c2ba4d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c24105360 0x7f7c2419dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.467+0000 7f7c2ba4d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c241976b0 0x7f7c24197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.467+0000 7f7c2ba4d640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c24198180 con 0x7f7c241976b0 2024-03-12T05:57:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.467+0000 7f7c2ba4d640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c241982f0 con 0x7f7c24105360 2024-03-12T05:57:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.467+0000 7f7c2ba4d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c241984b0 con 0x7f7c24100290 2024-03-12T05:57:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.467+0000 7f7c29fea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c241976b0 0x7f7c24197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.467+0000 7f7c28fe8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c24105360 0x7f7c2419dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.467+0000 7f7c29fea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c241976b0 0x7f7c24197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47762/0 (socket says 172.21.15.76:47762) 2024-03-12T05:57:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.467+0000 7f7c29fea640 1 -- 172.21.15.76:0/4287650496 learned_addr learned my addr 172.21.15.76:0/4287650496 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.467+0000 7f7c297e9640 1 --2- 172.21.15.76:0/4287650496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c24100290 0x7f7c2419d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.468+0000 7f7c28fe8640 1 -- 172.21.15.76:0/4287650496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c24100290 msgr2=0x7f7c2419d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.468+0000 7f7c28fe8640 1 --2- 172.21.15.76:0/4287650496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c24100290 0x7f7c2419d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.468+0000 7f7c28fe8640 1 -- 172.21.15.76:0/4287650496 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c241976b0 msgr2=0x7f7c24197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.468+0000 7f7c28fe8640 1 --2- 172.21.15.76:0/4287650496 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c241976b0 0x7f7c24197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.468+0000 7f7c28fe8640 1 -- 172.21.15.76:0/4287650496 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c009840 con 0x7f7c24105360 2024-03-12T05:57:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.468+0000 7f7c29fea640 1 --2- 172.21.15.76:0/4287650496 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c241976b0 0x7f7c24197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:57:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.468+0000 7f7c28fe8640 1 --2- 172.21.15.76:0/4287650496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c24105360 0x7f7c2419dae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7c2406a870 tx=0x7f7c1400dbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.468+0000 7f7c1a7fc640 1 -- 172.21.15.76:0/4287650496 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c14004020 con 0x7f7c24105360 2024-03-12T05:57:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.468+0000 7f7c2ba4d640 1 -- 172.21.15.76:0/4287650496 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c20009560 con 0x7f7c24105360 2024-03-12T05:57:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.468+0000 7f7c1a7fc640 1 -- 172.21.15.76:0/4287650496 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c14004180 con 0x7f7c24105360 2024-03-12T05:57:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.468+0000 7f7c1a7fc640 1 -- 172.21.15.76:0/4287650496 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c14004a00 con 0x7f7c24105360 2024-03-12T05:57:17.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.468+0000 7f7c2ba4d640 1 -- 172.21.15.76:0/4287650496 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c241a4700 con 0x7f7c24105360 2024-03-12T05:57:17.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.470+0000 7f7c2e61c740 1 Processor -- start 2024-03-12T05:57:17.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c2e61c740 1 -- start start 2024-03-12T05:57:17.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c1a7fc640 1 -- 172.21.15.76:0/4287650496 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f7c140042f0 con 0x7f7c24105360 2024-03-12T05:57:17.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c2e61c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747250 0x556669747630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c2e61c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747b70 0x556669750210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c2e61c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x556669752b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c2e61c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556669702960 con 0x556669747b70 2024-03-12T05:57:17.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c2e61c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566696b6310 con 0x556669747250 2024-03-12T05:57:17.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c2e61c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566696b3840 con 0x556669750750 2024-03-12T05:57:17.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c297e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747b70 0x556669750210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c297e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747b70 0x556669750210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47764/0 (socket says 172.21.15.76:47764) 2024-03-12T05:57:17.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c29fea640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747250 0x556669747630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c297e9640 1 -- 172.21.15.76:0/1391645403 learned_addr learned my addr 172.21.15.76:0/1391645403 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c28fe8640 1 --2- 172.21.15.76:0/1391645403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x556669752b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c297e9640 1 -- 172.21.15.76:0/1391645403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 msgr2=0x556669752b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c297e9640 1 --2- 172.21.15.76:0/1391645403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x556669752b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c297e9640 1 -- 172.21.15.76:0/1391645403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747250 msgr2=0x556669747630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c297e9640 1 --2- 172.21.15.76:0/1391645403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747250 0x556669747630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c297e9640 1 -- 172.21.15.76:0/1391645403 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556669671060 con 0x556669747b70 2024-03-12T05:57:17.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c28fe8640 1 --2- 172.21.15.76:0/1391645403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x556669752b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:17.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c1a7fc640 1 --2- 172.21.15.76:0/4287650496 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7c100a1cc0 0x7f7c100a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c29fea640 1 --2- 172.21.15.76:0/4287650496 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7c100a1cc0 0x7f7c100a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c297e9640 1 --2- 172.21.15.76:0/1391645403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747b70 0x556669750210 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7c20000c00 tx=0x7f7c2002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd193885513a1a5f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.471+0000 7f7c1a7fc640 1 -- 172.21.15.76:0/4287650496 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7c140cd6e0 con 0x7f7c24105360 2024-03-12T05:57:17.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.472+0000 7f7c0f7fe640 1 -- 172.21.15.76:0/1391645403 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c20004940 con 0x556669747b70 2024-03-12T05:57:17.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.472+0000 7f7c0f7fe640 1 -- 172.21.15.76:0/1391645403 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c20004aa0 con 0x556669747b70 2024-03-12T05:57:17.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.472+0000 7f7c0f7fe640 1 -- 172.21.15.76:0/1391645403 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c200052b0 con 0x556669747b70 2024-03-12T05:57:17.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.472+0000 7f7c2e61c740 1 -- 172.21.15.76:0/1391645403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747b70 msgr2=0x556669750210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.472+0000 7f7c29fea640 1 --2- 172.21.15.76:0/4287650496 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7c100a1cc0 0x7f7c100a4180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c005c40 tx=0x7f7c1c012040 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.472+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/1391645403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747b70 0x556669750210 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7c20000c00 tx=0x7f7c2002c640 comp rx=0 tx=0).stop 2024-03-12T05:57:17.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.472+0000 7f7c2e61c740 1 -- 172.21.15.76:0/1391645403 shutdown_connections 2024-03-12T05:57:17.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.472+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/1391645403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x556669752b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.472+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/1391645403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747b70 0x556669750210 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.472+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/1391645403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747250 0x556669747630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.472+0000 7f7c2e61c740 1 -- 172.21.15.76:0/1391645403 >> 172.21.15.76:0/1391645403 conn(0x556669741980 msgr2=0x556669741d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.472+0000 7f7c2e61c740 1 -- 172.21.15.76:0/1391645403 shutdown_connections 2024-03-12T05:57:17.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.473+0000 7f7c2e61c740 1 -- 172.21.15.76:0/1391645403 wait complete. 2024-03-12T05:57:17.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.473+0000 7f7c2e61c740 1 -- start start 2024-03-12T05:57:17.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.473+0000 7f7c2e61c740 1 Processor -- start 2024-03-12T05:57:17.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.473+0000 7f7c2e61c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747250 0x55666974d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.473+0000 7f7c2e61c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747b70 0x55666974dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.473+0000 7f7c2e61c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x55666974e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.473+0000 7f7c2e61c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566696af020 con 0x556669747250 2024-03-12T05:57:17.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.473+0000 7f7c2e61c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x556669382430 con 0x556669747b70 2024-03-12T05:57:17.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.473+0000 7f7c2e61c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55666933ee00 con 0x556669750750 2024-03-12T05:57:17.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.473+0000 7f7c2e61c740 10 client.0 _ll_register_callbacks cb 0x556669656cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:17.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.473+0000 7f7c297e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747250 0x55666974d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.473+0000 7f7c28fe8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747b70 0x55666974dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.473+0000 7f7c297e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747250 0x55666974d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47780/0 (socket says 172.21.15.76:47780) 2024-03-12T05:57:17.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.473+0000 7f7c297e9640 1 -- 172.21.15.76:0/2823090518 learned_addr learned my addr 172.21.15.76:0/2823090518 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c29fea640 1 --2- 172.21.15.76:0/2823090518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x55666974e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c297e9640 1 -- 172.21.15.76:0/2823090518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 msgr2=0x55666974e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c297e9640 1 --2- 172.21.15.76:0/2823090518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x55666974e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c297e9640 1 -- 172.21.15.76:0/2823090518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747b70 msgr2=0x55666974dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c297e9640 1 --2- 172.21.15.76:0/2823090518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747b70 0x55666974dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c297e9640 1 -- 172.21.15.76:0/2823090518 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2410d740 con 0x556669747250 2024-03-12T05:57:17.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c297e9640 1 --2- 172.21.15.76:0/2823090518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747250 0x55666974d730 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7c20004a70 tx=0x7f7c20004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8278a4df75bddb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c0cff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:17.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c0cff9640 1 -- 172.21.15.76:0/2823090518 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c20005440 con 0x556669747250 2024-03-12T05:57:17.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c0cff9640 1 -- 172.21.15.76:0/2823090518 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7c200055a0 con 0x556669747250 2024-03-12T05:57:17.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c2e61c740 1 -- 172.21.15.76:0/2823090518 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5566693ae7e0 con 0x556669747250 2024-03-12T05:57:17.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c0cff9640 1 -- 172.21.15.76:0/2823090518 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c20049450 con 0x556669747250 2024-03-12T05:57:17.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c2e61c740 10 client.15903 fetch_fsmap learned FSMap version 22 2024-03-12T05:57:17.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c2e61c740 1 -- 172.21.15.76:0/2823090518 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5566696ad6a0 con 0x556669747250 2024-03-12T05:57:17.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c0cff9640 1 -- 172.21.15.76:0/2823090518 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5566693ae7e0 con 0x556669747250 2024-03-12T05:57:17.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.474+0000 7f7c0cff9640 1 -- 172.21.15.76:0/2823090518 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7c2003c040 con 0x556669747250 2024-03-12T05:57:17.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.475+0000 7f7c2e61c740 10 client.15903 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:57:17.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.475+0000 7f7c2e61c740 10 client.15903 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:57:17.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.475+0000 7f7c2e61c740 20 client.15903 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:17.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.475+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/2823090518 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5566697607b0 0x556669762ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.475+0000 7f7c2e61c740 4 client.15903 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:57:17.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.475+0000 7f7c2e61c740 1 -- 172.21.15.76:0/2823090518 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566693eb720 con 0x5566697607b0 2024-03-12T05:57:17.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.475+0000 7f7c28fe8640 1 --2- 172.21.15.76:0/2823090518 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5566697607b0 0x556669762ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.475+0000 7f7c0cff9640 10 client.15903 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:57:17.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.475+0000 7f7c28fe8640 1 --2- 172.21.15.76:0/2823090518 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5566697607b0 0x556669762ba0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=93214f8d9358c9dd server_cookie=46fa57899f29f9a6 in_seq=0 out_seq=0 2024-03-12T05:57:17.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c0cff9640 1 -- 172.21.15.76:0/2823090518 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5566693eb720 con 0x5566697607b0 2024-03-12T05:57:17.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c0cff9640 10 client.15903 handle_command_reply: tid=0 2024-03-12T05:57:17.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c2e61c740 1 client.15903 shutdown 2024-03-12T05:57:17.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c2e61c740 1 -- 172.21.15.76:0/2823090518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747250 msgr2=0x55666974d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/2823090518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747250 0x55666974d730 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7c20004a70 tx=0x7f7c20004df0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c2e61c740 1 -- 172.21.15.76:0/2823090518 shutdown_connections 2024-03-12T05:57:17.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/2823090518 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5566697607b0 0x556669762ba0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/2823090518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x55666974e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/2823090518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747b70 0x55666974dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c0cff9640 0 client.15903 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:57:17.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/2823090518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747250 0x55666974d730 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c2e61c740 1 -- 172.21.15.76:0/2823090518 >> 172.21.15.76:0/2823090518 conn(0x556669741980 msgr2=0x5566697436d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c2e61c740 1 -- 172.21.15.76:0/2823090518 shutdown_connections 2024-03-12T05:57:17.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c2e61c740 1 -- 172.21.15.76:0/2823090518 wait complete. 2024-03-12T05:57:17.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.476+0000 7f7c2e61c740 20 client.15903 trim_cache size 0 max 16384 2024-03-12T05:57:17.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.488+0000 7f7c2e61c740 1 Processor -- start 2024-03-12T05:57:17.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.488+0000 7f7c2e61c740 1 -- start start 2024-03-12T05:57:17.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.488+0000 7f7c2e61c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747250 0x55666974fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.488+0000 7f7c2e61c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747b70 0x55666974dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.488+0000 7f7c2e61c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x55666974e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.488+0000 7f7c2e61c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556669634130 con 0x556669747b70 2024-03-12T05:57:17.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.488+0000 7f7c2e61c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55666974e730 con 0x556669747250 2024-03-12T05:57:17.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.488+0000 7f7c2e61c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556669742980 con 0x556669750750 2024-03-12T05:57:17.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.488+0000 7f7c28fe8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747b70 0x55666974dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.489+0000 7f7c28fe8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747b70 0x55666974dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47782/0 (socket says 172.21.15.76:47782) 2024-03-12T05:57:17.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.489+0000 7f7c297e9640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747250 0x55666974fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.489+0000 7f7c28fe8640 1 -- 172.21.15.76:0/273749857 learned_addr learned my addr 172.21.15.76:0/273749857 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.489+0000 7f7c29fea640 1 --2- 172.21.15.76:0/273749857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x55666974e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.489+0000 7f7c28fe8640 1 -- 172.21.15.76:0/273749857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 msgr2=0x55666974e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.489+0000 7f7c28fe8640 1 --2- 172.21.15.76:0/273749857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x55666974e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.489+0000 7f7c28fe8640 1 -- 172.21.15.76:0/273749857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747250 msgr2=0x55666974fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.489+0000 7f7c28fe8640 1 --2- 172.21.15.76:0/273749857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747250 0x55666974fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.489+0000 7f7c28fe8640 1 -- 172.21.15.76:0/273749857 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556669649c20 con 0x556669747b70 2024-03-12T05:57:17.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.489+0000 7f7c29fea640 1 --2- 172.21.15.76:0/273749857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x55666974e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:57:17.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.489+0000 7f7c28fe8640 1 --2- 172.21.15.76:0/273749857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747b70 0x55666974dc80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7c14096550 tx=0x7f7c140052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6fb165ca11f056 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.489+0000 7f7c0effd640 1 -- 172.21.15.76:0/273749857 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1409a7f0 con 0x556669747b70 2024-03-12T05:57:17.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.489+0000 7f7c0effd640 1 -- 172.21.15.76:0/273749857 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c140af030 con 0x556669747b70 2024-03-12T05:57:17.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.489+0000 7f7c0effd640 1 -- 172.21.15.76:0/273749857 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c14098480 con 0x556669747b70 2024-03-12T05:57:17.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 -- 172.21.15.76:0/273749857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747b70 msgr2=0x55666974dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/273749857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747b70 0x55666974dc80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7c14096550 tx=0x7f7c140052d0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 -- 172.21.15.76:0/273749857 shutdown_connections 2024-03-12T05:57:17.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/273749857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x55666974e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/273749857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747b70 0x55666974dc80 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/273749857 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747250 0x55666974fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 -- 172.21.15.76:0/273749857 >> 172.21.15.76:0/273749857 conn(0x556669741980 msgr2=0x5566697489f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 -- 172.21.15.76:0/273749857 shutdown_connections 2024-03-12T05:57:17.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 -- 172.21.15.76:0/273749857 wait complete. 2024-03-12T05:57:17.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 -- start start 2024-03-12T05:57:17.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 Processor -- start 2024-03-12T05:57:17.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747250 0x556669756220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747b70 0x556669756760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x556669756ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566697b6e10 con 0x556669747250 2024-03-12T05:57:17.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566697b6f80 con 0x556669747b70 2024-03-12T05:57:17.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.490+0000 7f7c2e61c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566697b7140 con 0x556669750750 2024-03-12T05:57:17.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c2e61c740 10 client.0 _ll_register_callbacks cb 0x7f7c1c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:17.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c297e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747250 0x556669756220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c297e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747250 0x556669756220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47798/0 (socket says 172.21.15.76:47798) 2024-03-12T05:57:17.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c297e9640 1 -- 172.21.15.76:0/737179706 learned_addr learned my addr 172.21.15.76:0/737179706 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c28fe8640 1 --2- 172.21.15.76:0/737179706 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747b70 0x556669756760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c29fea640 1 --2- 172.21.15.76:0/737179706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x556669756ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c297e9640 1 -- 172.21.15.76:0/737179706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 msgr2=0x556669756ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c297e9640 1 --2- 172.21.15.76:0/737179706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x556669756ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c297e9640 1 -- 172.21.15.76:0/737179706 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747b70 msgr2=0x556669756760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c297e9640 1 --2- 172.21.15.76:0/737179706 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747b70 0x556669756760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c297e9640 1 -- 172.21.15.76:0/737179706 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c241a4700 con 0x556669747250 2024-03-12T05:57:17.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c297e9640 1 --2- 172.21.15.76:0/737179706 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747250 0x556669756220 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7c20034740 tx=0x7f7c2002c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6985cfadf9f5853 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c0cff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:17.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c0cff9640 1 -- 172.21.15.76:0/737179706 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c20005440 con 0x556669747250 2024-03-12T05:57:17.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c0cff9640 1 -- 172.21.15.76:0/737179706 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7c20000e40 con 0x556669747250 2024-03-12T05:57:17.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c2e61c740 1 -- 172.21.15.76:0/737179706 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c20004aa0 con 0x556669747250 2024-03-12T05:57:17.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c0cff9640 1 -- 172.21.15.76:0/737179706 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c200055a0 con 0x556669747250 2024-03-12T05:57:17.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c2e61c740 10 client.15909 fetch_fsmap learned FSMap version 22 2024-03-12T05:57:17.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.491+0000 7f7c2e61c740 1 -- 172.21.15.76:0/737179706 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55666966f8c0 con 0x556669747250 2024-03-12T05:57:17.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.492+0000 7f7c0cff9640 1 -- 172.21.15.76:0/737179706 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c20004aa0 con 0x556669747250 2024-03-12T05:57:17.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.492+0000 7f7c0cff9640 1 -- 172.21.15.76:0/737179706 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7c200035f0 con 0x556669747250 2024-03-12T05:57:17.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.492+0000 7f7c2e61c740 10 client.15909 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:57:17.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.492+0000 7f7c2e61c740 10 client.15909 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:57:17.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.492+0000 7f7c2e61c740 20 client.15909 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:17.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.492+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/737179706 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5566697b84a0 0x5566697ba8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.492+0000 7f7c2e61c740 4 client.15909 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:57:17.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.492+0000 7f7c2e61c740 1 -- 172.21.15.76:0/737179706 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55666943a890 con 0x5566697b84a0 2024-03-12T05:57:17.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.492+0000 7f7c28fe8640 1 --2- 172.21.15.76:0/737179706 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5566697b84a0 0x5566697ba8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.492+0000 7f7c28fe8640 1 --2- 172.21.15.76:0/737179706 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5566697b84a0 0x5566697ba8c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e480dff7ffe1aa8 server_cookie=35393d6f2cfc8aa0 in_seq=0 out_seq=0 2024-03-12T05:57:17.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.492+0000 7f7c0cff9640 10 client.15909 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:57:17.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.492+0000 7f7c0cff9640 1 -- 172.21.15.76:0/737179706 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55666943a890 con 0x5566697b84a0 2024-03-12T05:57:17.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.492+0000 7f7c0cff9640 10 client.15909 handle_command_reply: tid=0 2024-03-12T05:57:17.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.492+0000 7f7c2e61c740 1 client.15909 shutdown 2024-03-12T05:57:17.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2e61c740 1 -- 172.21.15.76:0/737179706 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747250 msgr2=0x556669756220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/737179706 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747250 0x556669756220 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7c20034740 tx=0x7f7c2002c340 comp rx=0 tx=0).stop 2024-03-12T05:57:17.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2e61c740 1 -- 172.21.15.76:0/737179706 shutdown_connections 2024-03-12T05:57:17.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/737179706 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5566697b84a0 0x5566697ba8c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/737179706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556669750750 0x556669756ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c0cff9640 0 client.15909 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:57:17.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/737179706 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556669747b70 0x556669756760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2e61c740 1 --2- 172.21.15.76:0/737179706 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556669747250 0x556669756220 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2e61c740 1 -- 172.21.15.76:0/737179706 >> 172.21.15.76:0/737179706 conn(0x556669741980 msgr2=0x5566697402f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2e61c740 1 -- 172.21.15.76:0/737179706 shutdown_connections 2024-03-12T05:57:17.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2e61c740 1 -- 172.21.15.76:0/737179706 wait complete. 2024-03-12T05:57:17.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2e61c740 20 client.15909 trim_cache size 0 max 16384 2024-03-12T05:57:17.506 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:57:17.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2ba4d640 1 -- 172.21.15.76:0/4287650496 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7c100a1cc0 msgr2=0x7f7c100a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2ba4d640 1 --2- 172.21.15.76:0/4287650496 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7c100a1cc0 0x7f7c100a4180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c005c40 tx=0x7f7c1c012040 comp rx=0 tx=0).stop 2024-03-12T05:57:17.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2ba4d640 1 -- 172.21.15.76:0/4287650496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c24105360 msgr2=0x7f7c2419dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2ba4d640 1 --2- 172.21.15.76:0/4287650496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c24105360 0x7f7c2419dae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7c2406a870 tx=0x7f7c1400dbb0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2ba4d640 1 -- 172.21.15.76:0/4287650496 shutdown_connections 2024-03-12T05:57:17.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2ba4d640 1 --2- 172.21.15.76:0/4287650496 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7c100a1cc0 0x7f7c100a4180 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2ba4d640 1 --2- 172.21.15.76:0/4287650496 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c241976b0 0x7f7c24197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2ba4d640 1 --2- 172.21.15.76:0/4287650496 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c24105360 0x7f7c2419dae0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2ba4d640 1 --2- 172.21.15.76:0/4287650496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c24100290 0x7f7c2419d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2ba4d640 1 -- 172.21.15.76:0/4287650496 >> 172.21.15.76:0/4287650496 conn(0x7f7c24077230 msgr2=0x7f7c240688b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2ba4d640 1 -- 172.21.15.76:0/4287650496 shutdown_connections 2024-03-12T05:57:17.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.493+0000 7f7c2ba4d640 1 -- 172.21.15.76:0/4287650496 wait complete. 2024-03-12T05:57:17.509 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:57:17.509 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:57:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.630+0000 7f3678488640 1 -- 172.21.15.76:0/4085066304 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3670102cc0 msgr2=0x7f367010b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.630+0000 7f3678488640 1 --2- 172.21.15.76:0/4085066304 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3670102cc0 0x7f367010b1a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3660009860 tx=0x7f366002bc10 comp rx=0 tx=0).stop 2024-03-12T05:57:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.630+0000 7f3678488640 1 -- 172.21.15.76:0/4085066304 shutdown_connections 2024-03-12T05:57:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.630+0000 7f3678488640 1 --2- 172.21.15.76:0/4085066304 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f367010b770 0x7f367010dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.630+0000 7f3678488640 1 --2- 172.21.15.76:0/4085066304 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3670102cc0 0x7f367010b1a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.630+0000 7f3678488640 1 --2- 172.21.15.76:0/4085066304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36701023a0 0x7f3670102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.630+0000 7f3678488640 1 -- 172.21.15.76:0/4085066304 >> 172.21.15.76:0/4085066304 conn(0x7f3670077230 msgr2=0x7f3670077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.630+0000 7f3678488640 1 -- 172.21.15.76:0/4085066304 shutdown_connections 2024-03-12T05:57:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.630+0000 7f3678488640 1 -- 172.21.15.76:0/4085066304 wait complete. 2024-03-12T05:57:17.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3678488640 1 Processor -- start 2024-03-12T05:57:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3678488640 1 -- start start 2024-03-12T05:57:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3678488640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36701023a0 0x7f367019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3678488640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3670102cc0 0x7f367019d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3678488640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f367010b770 0x7f3670197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3678488640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3670197ca0 con 0x7f36701023a0 2024-03-12T05:57:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3678488640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3670197e10 con 0x7f3670102cc0 2024-03-12T05:57:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3678488640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3670197fd0 con 0x7f367010b770 2024-03-12T05:57:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3676224640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36701023a0 0x7f367019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3675a23640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3670102cc0 0x7f367019d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3675a23640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3670102cc0 0x7f367019d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:50006/0 (socket says 172.21.15.76:50006) 2024-03-12T05:57:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3675a23640 1 -- 172.21.15.76:0/1095936095 learned_addr learned my addr 172.21.15.76:0/1095936095 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3676a25640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f367010b770 0x7f3670197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3675a23640 1 -- 172.21.15.76:0/1095936095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f367010b770 msgr2=0x7f3670197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3675a23640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f367010b770 0x7f3670197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3675a23640 1 -- 172.21.15.76:0/1095936095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36701023a0 msgr2=0x7f367019d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3675a23640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36701023a0 0x7f367019d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3675a23640 1 -- 172.21.15.76:0/1095936095 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3660009490 con 0x7f3670102cc0 2024-03-12T05:57:17.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3676224640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36701023a0 0x7f367019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:57:17.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3676a25640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f367010b770 0x7f3670197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:17.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.631+0000 7f3675a23640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3670102cc0 0x7f367019d990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f366002c220 tx=0x7f36600346f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.632+0000 7f36677fe640 1 -- 172.21.15.76:0/1095936095 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366002c7b0 con 0x7f3670102cc0 2024-03-12T05:57:17.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.632+0000 7f3678488640 1 -- 172.21.15.76:0/1095936095 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3670198250 con 0x7f3670102cc0 2024-03-12T05:57:17.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.632+0000 7f36677fe640 1 -- 172.21.15.76:0/1095936095 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36600044d0 con 0x7f3670102cc0 2024-03-12T05:57:17.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.632+0000 7f36677fe640 1 -- 172.21.15.76:0/1095936095 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36600406b0 con 0x7f3670102cc0 2024-03-12T05:57:17.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.632+0000 7f3678488640 1 -- 172.21.15.76:0/1095936095 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36701a44f0 con 0x7f3670102cc0 2024-03-12T05:57:17.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.634+0000 7f36677fe640 1 -- 172.21.15.76:0/1095936095 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f3660003810 con 0x7f3670102cc0 2024-03-12T05:57:17.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.634+0000 7f367b067740 1 Processor -- start 2024-03-12T05:57:17.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.634+0000 7f367b067740 1 -- start start 2024-03-12T05:57:17.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.634+0000 7f367b067740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 0x55e73e492360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.634+0000 7f367b067740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e4928a0 0x55e73e49f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.634+0000 7f367b067740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e49fb60 0x55e73e4a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.634+0000 7f367b067740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73e44d960 con 0x55e73e49fb60 2024-03-12T05:57:17.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.634+0000 7f367b067740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73e401310 con 0x55e73e4928a0 2024-03-12T05:57:17.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.634+0000 7f367b067740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73e3fe840 con 0x55e73e491f80 2024-03-12T05:57:17.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3675a23640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e49fb60 0x55e73e4a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3676224640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e4928a0 0x55e73e49f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3676224640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e4928a0 0x55e73e49f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:50014/0 (socket says 172.21.15.76:50014) 2024-03-12T05:57:17.641 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3676224640 1 -- 172.21.15.76:0/2932675022 learned_addr learned my addr 172.21.15.76:0/2932675022 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.642 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3676a25640 1 --2- 172.21.15.76:0/2932675022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 0x55e73e492360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f36677fe640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f365c0a1cc0 0x7f365c0a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3676224640 1 -- 172.21.15.76:0/2932675022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 msgr2=0x55e73e492360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3676224640 1 --2- 172.21.15.76:0/2932675022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 0x55e73e492360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3676224640 1 -- 172.21.15.76:0/2932675022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e49fb60 msgr2=0x55e73e4a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3676224640 1 --2- 172.21.15.76:0/2932675022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e49fb60 0x55e73e4a1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3676224640 1 -- 172.21.15.76:0/2932675022 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e73e3bc060 con 0x55e73e4928a0 2024-03-12T05:57:17.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3676a25640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f365c0a1cc0 0x7f365c0a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f36677fe640 1 -- 172.21.15.76:0/1095936095 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f36600ef9f0 con 0x7f3670102cc0 2024-03-12T05:57:17.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3676224640 1 --2- 172.21.15.76:0/2932675022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e4928a0 0x55e73e49f620 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f366c0104d0 tx=0x7f366c010990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da53462db8ff43bc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3664bfe640 1 -- 172.21.15.76:0/2932675022 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c004220 con 0x55e73e4928a0 2024-03-12T05:57:17.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3676a25640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f365c0a1cc0 0x7f365c0a4180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3668005fd0 tx=0x7f3668011040 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3664bfe640 1 -- 172.21.15.76:0/2932675022 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f366c015070 con 0x55e73e4928a0 2024-03-12T05:57:17.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.635+0000 7f3664bfe640 1 -- 172.21.15.76:0/2932675022 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c011960 con 0x55e73e4928a0 2024-03-12T05:57:17.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.636+0000 7f367b067740 1 -- 172.21.15.76:0/2932675022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e4928a0 msgr2=0x55e73e49f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.636+0000 7f367b067740 1 --2- 172.21.15.76:0/2932675022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e4928a0 0x55e73e49f620 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f366c0104d0 tx=0x7f366c010990 comp rx=0 tx=0).stop 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.636+0000 7f367b067740 1 -- 172.21.15.76:0/2932675022 shutdown_connections 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.636+0000 7f367b067740 1 --2- 172.21.15.76:0/2932675022 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e49fb60 0x55e73e4a1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.636+0000 7f367b067740 1 --2- 172.21.15.76:0/2932675022 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e4928a0 0x55e73e49f620 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.636+0000 7f367b067740 1 --2- 172.21.15.76:0/2932675022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 0x55e73e492360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.636+0000 7f367b067740 1 -- 172.21.15.76:0/2932675022 >> 172.21.15.76:0/2932675022 conn(0x55e73e48c980 msgr2=0x55e73e48cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.636+0000 7f367b067740 1 -- 172.21.15.76:0/2932675022 shutdown_connections 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.636+0000 7f367b067740 1 -- 172.21.15.76:0/2932675022 wait complete. 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.636+0000 7f367b067740 1 -- start start 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f367b067740 1 Processor -- start 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f367b067740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 0x55e73e49cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f367b067740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e49d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f367b067740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e49fb60 0x55e73e49d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f367b067740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73e3fa020 con 0x55e73e4928a0 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f367b067740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73e0cd430 con 0x55e73e49fb60 2024-03-12T05:57:17.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f367b067740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73e089e00 con 0x55e73e491f80 2024-03-12T05:57:17.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f367b067740 10 client.0 _ll_register_callbacks cb 0x55e73e3a1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:17.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f3675a23640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e49d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f3676a25640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e49fb60 0x55e73e49d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f3675a23640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e49d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47822/0 (socket says 172.21.15.76:47822) 2024-03-12T05:57:17.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f3675a23640 1 -- 172.21.15.76:0/2916291879 learned_addr learned my addr 172.21.15.76:0/2916291879 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f3676224640 1 --2- 172.21.15.76:0/2916291879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 0x55e73e49cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f3675a23640 1 -- 172.21.15.76:0/2916291879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 msgr2=0x55e73e49cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f3675a23640 1 --2- 172.21.15.76:0/2916291879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 0x55e73e49cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f3675a23640 1 -- 172.21.15.76:0/2916291879 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e49fb60 msgr2=0x55e73e49d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f3675a23640 1 --2- 172.21.15.76:0/2916291879 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e49fb60 0x55e73e49d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f3675a23640 1 -- 172.21.15.76:0/2916291879 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c00bd30 con 0x55e73e4928a0 2024-03-12T05:57:17.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f3675a23640 1 --2- 172.21.15.76:0/2916291879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e49d080 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f36600b8e90 tx=0x7f366004e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e584a4f338df353 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f3659ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:17.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f3659ffb640 1 -- 172.21.15.76:0/2916291879 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36600b87b0 con 0x55e73e4928a0 2024-03-12T05:57:17.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f367b067740 1 -- 172.21.15.76:0/2916291879 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e73e0f97e0 con 0x55e73e4928a0 2024-03-12T05:57:17.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.637+0000 7f3659ffb640 1 -- 172.21.15.76:0/2916291879 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f36600bcc10 con 0x55e73e4928a0 2024-03-12T05:57:17.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.638+0000 7f3659ffb640 1 -- 172.21.15.76:0/2916291879 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36600bc2e0 con 0x55e73e4928a0 2024-03-12T05:57:17.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.638+0000 7f367b067740 10 client.15921 fetch_fsmap learned FSMap version 22 2024-03-12T05:57:17.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.638+0000 7f367b067740 1 -- 172.21.15.76:0/2916291879 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55e73e3f86a0 con 0x55e73e4928a0 2024-03-12T05:57:17.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.638+0000 7f3659ffb640 1 -- 172.21.15.76:0/2916291879 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e73e0f97e0 con 0x55e73e4928a0 2024-03-12T05:57:17.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.638+0000 7f3659ffb640 1 -- 172.21.15.76:0/2916291879 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f36600c4820 con 0x55e73e4928a0 2024-03-12T05:57:17.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.638+0000 7f367b067740 10 client.15921 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:57:17.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.638+0000 7f367b067740 10 client.15921 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:57:17.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.638+0000 7f367b067740 20 client.15921 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:17.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.638+0000 7f367b067740 1 --2- 172.21.15.76:0/2916291879 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e73e4afdc0 0x55e73e4b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.638+0000 7f367b067740 4 client.15921 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:57:17.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.638+0000 7f367b067740 1 -- 172.21.15.76:0/2916291879 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e73e136720 con 0x55e73e4afdc0 2024-03-12T05:57:17.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.638+0000 7f3676224640 1 --2- 172.21.15.76:0/2916291879 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e73e4afdc0 0x55e73e4b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.639+0000 7f3659ffb640 10 client.15921 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:57:17.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.639+0000 7f3676224640 1 --2- 172.21.15.76:0/2916291879 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e73e4afdc0 0x55e73e4b21b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b72d83bd3d94d65d server_cookie=ad8147de4f2860fb in_seq=0 out_seq=0 2024-03-12T05:57:17.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.639+0000 7f3659ffb640 1 -- 172.21.15.76:0/2916291879 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e73e136720 con 0x55e73e4afdc0 2024-03-12T05:57:17.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.639+0000 7f3659ffb640 10 client.15921 handle_command_reply: tid=0 2024-03-12T05:57:17.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.639+0000 7f367b067740 1 client.15921 shutdown 2024-03-12T05:57:17.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.639+0000 7f367b067740 1 -- 172.21.15.76:0/2916291879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 msgr2=0x55e73e49d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.639+0000 7f367b067740 1 --2- 172.21.15.76:0/2916291879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e49d080 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f36600b8e90 tx=0x7f366004e000 comp rx=0 tx=0).stop 2024-03-12T05:57:17.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.640+0000 7f367b067740 1 -- 172.21.15.76:0/2916291879 shutdown_connections 2024-03-12T05:57:17.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.640+0000 7f367b067740 1 --2- 172.21.15.76:0/2916291879 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e73e4afdc0 0x55e73e4b21b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.640+0000 7f367b067740 1 --2- 172.21.15.76:0/2916291879 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e49fb60 0x55e73e49d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.640+0000 7f367b067740 1 --2- 172.21.15.76:0/2916291879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e49d080 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.640+0000 7f3659ffb640 0 client.15921 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:57:17.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.640+0000 7f367b067740 1 --2- 172.21.15.76:0/2916291879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 0x55e73e49cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.640+0000 7f367b067740 1 -- 172.21.15.76:0/2916291879 >> 172.21.15.76:0/2916291879 conn(0x55e73e48c980 msgr2=0x55e73e48e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.640+0000 7f367b067740 1 -- 172.21.15.76:0/2916291879 shutdown_connections 2024-03-12T05:57:17.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.640+0000 7f367b067740 1 -- 172.21.15.76:0/2916291879 wait complete. 2024-03-12T05:57:17.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.640+0000 7f367b067740 20 client.15921 trim_cache size 0 max 16384 2024-03-12T05:57:17.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f367b067740 1 Processor -- start 2024-03-12T05:57:17.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f367b067740 1 -- start start 2024-03-12T05:57:17.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f367b067740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 0x55e73e49f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f367b067740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e499dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f367b067740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e49fb60 0x55e73e49a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f367b067740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73e37f130 con 0x55e73e4928a0 2024-03-12T05:57:17.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f367b067740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73e0f97e0 con 0x55e73e49fb60 2024-03-12T05:57:17.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f367b067740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73e47ea40 con 0x55e73e491f80 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f3675a23640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e499dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f3675a23640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e499dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47824/0 (socket says 172.21.15.76:47824) 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f3675a23640 1 -- 172.21.15.76:0/866794745 learned_addr learned my addr 172.21.15.76:0/866794745 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f3676a25640 1 --2- 172.21.15.76:0/866794745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e49fb60 0x55e73e49a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f3676224640 1 --2- 172.21.15.76:0/866794745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 0x55e73e49f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f3675a23640 1 -- 172.21.15.76:0/866794745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 msgr2=0x55e73e49f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f3675a23640 1 --2- 172.21.15.76:0/866794745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 0x55e73e49f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f3675a23640 1 -- 172.21.15.76:0/866794745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e49fb60 msgr2=0x55e73e49a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f3675a23640 1 --2- 172.21.15.76:0/866794745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e49fb60 0x55e73e49a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f3675a23640 1 -- 172.21.15.76:0/866794745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e73e394c20 con 0x55e73e4928a0 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.652+0000 7f3676224640 1 --2- 172.21.15.76:0/866794745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 0x55e73e49f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.653+0000 7f3675a23640 1 --2- 172.21.15.76:0/866794745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e499dc0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f36600b8ee0 tx=0x7f3660003bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d9f8681b1fc099 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.653+0000 7f365bfff640 1 -- 172.21.15.76:0/866794745 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36600ed040 con 0x55e73e4928a0 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.653+0000 7f365bfff640 1 -- 172.21.15.76:0/866794745 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36600b87b0 con 0x55e73e4928a0 2024-03-12T05:57:17.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.653+0000 7f365bfff640 1 -- 172.21.15.76:0/866794745 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36600c4cb0 con 0x55e73e4928a0 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.653+0000 7f367b067740 1 -- 172.21.15.76:0/866794745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 msgr2=0x55e73e499dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.653+0000 7f367b067740 1 --2- 172.21.15.76:0/866794745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e499dc0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f36600b8ee0 tx=0x7f3660003bb0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.653+0000 7f367b067740 1 -- 172.21.15.76:0/866794745 shutdown_connections 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.653+0000 7f367b067740 1 --2- 172.21.15.76:0/866794745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e49fb60 0x55e73e49a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.653+0000 7f367b067740 1 --2- 172.21.15.76:0/866794745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e499dc0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.653+0000 7f367b067740 1 --2- 172.21.15.76:0/866794745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e491f80 0x55e73e49f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.653+0000 7f367b067740 1 -- 172.21.15.76:0/866794745 >> 172.21.15.76:0/866794745 conn(0x55e73e48c980 msgr2=0x55e73e4b6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.653+0000 7f367b067740 1 -- 172.21.15.76:0/866794745 shutdown_connections 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.653+0000 7f367b067740 1 -- 172.21.15.76:0/866794745 wait complete. 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f367b067740 1 -- start start 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f367b067740 1 Processor -- start 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f367b067740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e491f80 0x55e73e4a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f367b067740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e4a6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f367b067740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e49fb60 0x55e73e5062a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f367b067740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73e5067e0 con 0x55e73e4928a0 2024-03-12T05:57:17.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f367b067740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73e506920 con 0x55e73e491f80 2024-03-12T05:57:17.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f367b067740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73e506a60 con 0x55e73e49fb60 2024-03-12T05:57:17.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f367b067740 10 client.0 _ll_register_callbacks cb 0x7f366c00bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:17.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f3675a23640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e4a6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f3675a23640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e4a6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47828/0 (socket says 172.21.15.76:47828) 2024-03-12T05:57:17.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f3675a23640 1 -- 172.21.15.76:0/512404280 learned_addr learned my addr 172.21.15.76:0/512404280 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f3676224640 1 --2- 172.21.15.76:0/512404280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e491f80 0x55e73e4a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f3676a25640 1 --2- 172.21.15.76:0/512404280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e49fb60 0x55e73e5062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f3675a23640 1 -- 172.21.15.76:0/512404280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e49fb60 msgr2=0x55e73e5062a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f3675a23640 1 --2- 172.21.15.76:0/512404280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e49fb60 0x55e73e5062a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f3675a23640 1 -- 172.21.15.76:0/512404280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e491f80 msgr2=0x55e73e4a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f3675a23640 1 --2- 172.21.15.76:0/512404280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e491f80 0x55e73e4a5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f3675a23640 1 -- 172.21.15.76:0/512404280 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e73e3f86a0 con 0x55e73e4928a0 2024-03-12T05:57:17.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f3676a25640 1 --2- 172.21.15.76:0/512404280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e49fb60 0x55e73e5062a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f3675a23640 1 --2- 172.21.15.76:0/512404280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e4a6000 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3660009830 tx=0x7f36600efd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87bf837c3de3935 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f3659ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.654+0000 7f3659ffb640 1 -- 172.21.15.76:0/512404280 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36600bc2e0 con 0x55e73e4928a0 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.655+0000 7f3659ffb640 1 -- 172.21.15.76:0/512404280 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f36600bc510 con 0x55e73e4928a0 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.655+0000 7f367b067740 1 -- 172.21.15.76:0/512404280 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f366c015070 con 0x55e73e4928a0 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.655+0000 7f3659ffb640 1 -- 172.21.15.76:0/512404280 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36600ba810 con 0x55e73e4928a0 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.655+0000 7f367b067740 10 client.15927 fetch_fsmap learned FSMap version 22 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.655+0000 7f367b067740 1 -- 172.21.15.76:0/512404280 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55e73e3ba8c0 con 0x55e73e4928a0 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.655+0000 7f3659ffb640 1 -- 172.21.15.76:0/512404280 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36600d4070 con 0x55e73e4928a0 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.655+0000 7f3659ffb640 1 -- 172.21.15.76:0/512404280 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f36600c0120 con 0x55e73e4928a0 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.655+0000 7f367b067740 10 client.15927 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.655+0000 7f367b067740 10 client.15927 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.655+0000 7f367b067740 20 client.15927 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.655+0000 7f367b067740 1 --2- 172.21.15.76:0/512404280 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e73e507c10 0x55e73e50a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.655+0000 7f367b067740 4 client.15927 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.655+0000 7f367b067740 1 -- 172.21.15.76:0/512404280 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e73e185890 con 0x55e73e507c10 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.655+0000 7f3676224640 1 --2- 172.21.15.76:0/512404280 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e73e507c10 0x55e73e50a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f3676224640 1 --2- 172.21.15.76:0/512404280 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e73e507c10 0x55e73e50a000 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=46667a364a2a446b server_cookie=2de4659223ffc493 in_seq=0 out_seq=0 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f3659ffb640 10 client.15927 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f3659ffb640 1 -- 172.21.15.76:0/512404280 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e73e185890 con 0x55e73e507c10 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f3659ffb640 10 client.15927 handle_command_reply: tid=0 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f367b067740 1 client.15927 shutdown 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f367b067740 1 -- 172.21.15.76:0/512404280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 msgr2=0x55e73e4a6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f367b067740 1 --2- 172.21.15.76:0/512404280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e4a6000 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3660009830 tx=0x7f36600efd60 comp rx=0 tx=0).stop 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f367b067740 1 -- 172.21.15.76:0/512404280 shutdown_connections 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f367b067740 1 --2- 172.21.15.76:0/512404280 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e73e507c10 0x55e73e50a000 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f367b067740 1 --2- 172.21.15.76:0/512404280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e73e49fb60 0x55e73e5062a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f367b067740 1 --2- 172.21.15.76:0/512404280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e73e4928a0 0x55e73e4a6000 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f3659ffb640 0 client.15927 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f367b067740 1 --2- 172.21.15.76:0/512404280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e73e491f80 0x55e73e4a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f367b067740 1 -- 172.21.15.76:0/512404280 >> 172.21.15.76:0/512404280 conn(0x55e73e48c980 msgr2=0x55e73e49cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f367b067740 1 -- 172.21.15.76:0/512404280 shutdown_connections 2024-03-12T05:57:17.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f367b067740 1 -- 172.21.15.76:0/512404280 wait complete. 2024-03-12T05:57:17.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.656+0000 7f367b067740 20 client.15927 trim_cache size 0 max 16384 2024-03-12T05:57:17.668 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:57:17.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.657+0000 7f3678488640 1 -- 172.21.15.76:0/1095936095 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f365c0a1cc0 msgr2=0x7f365c0a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.657+0000 7f3678488640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f365c0a1cc0 0x7f365c0a4180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3668005fd0 tx=0x7f3668011040 comp rx=0 tx=0).stop 2024-03-12T05:57:17.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.657+0000 7f3678488640 1 -- 172.21.15.76:0/1095936095 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3670102cc0 msgr2=0x7f367019d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.657+0000 7f3678488640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3670102cc0 0x7f367019d990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f366002c220 tx=0x7f36600346f0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.657+0000 7f3678488640 1 -- 172.21.15.76:0/1095936095 shutdown_connections 2024-03-12T05:57:17.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.657+0000 7f3678488640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f365c0a1cc0 0x7f365c0a4180 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.657+0000 7f3678488640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f367010b770 0x7f3670197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.657+0000 7f3678488640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3670102cc0 0x7f367019d990 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.657+0000 7f3678488640 1 --2- 172.21.15.76:0/1095936095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36701023a0 0x7f367019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.657+0000 7f3678488640 1 -- 172.21.15.76:0/1095936095 >> 172.21.15.76:0/1095936095 conn(0x7f3670077230 msgr2=0x7f36700fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.657+0000 7f3678488640 1 -- 172.21.15.76:0/1095936095 shutdown_connections 2024-03-12T05:57:17.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.657+0000 7f3678488640 1 -- 172.21.15.76:0/1095936095 wait complete. 2024-03-12T05:57:17.672 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:57:17.672 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:57:17.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.809+0000 7f00c4815640 1 -- 172.21.15.76:0/1852525028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00c0100c40 msgr2=0x7f00c0104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.809+0000 7f00c4815640 1 --2- 172.21.15.76:0/1852525028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00c0100c40 0x7f00c0104c80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009930 tx=0x7f00a802bc30 comp rx=0 tx=0).stop 2024-03-12T05:57:17.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.809+0000 7f00c4815640 1 -- 172.21.15.76:0/1852525028 shutdown_connections 2024-03-12T05:57:17.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.809+0000 7f00c4815640 1 --2- 172.21.15.76:0/1852525028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f00c0105360 0x7f00c010cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.809+0000 7f00c4815640 1 --2- 172.21.15.76:0/1852525028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00c0100c40 0x7f00c0104c80 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.809+0000 7f00c4815640 1 --2- 172.21.15.76:0/1852525028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c0100290 0x7f00c0100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.809+0000 7f00c4815640 1 -- 172.21.15.76:0/1852525028 >> 172.21.15.76:0/1852525028 conn(0x7f00c0077230 msgr2=0x7f00c0077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.809+0000 7f00c4815640 1 -- 172.21.15.76:0/1852525028 shutdown_connections 2024-03-12T05:57:17.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.809+0000 7f00c4815640 1 -- 172.21.15.76:0/1852525028 wait complete. 2024-03-12T05:57:17.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.810+0000 7f00c4815640 1 Processor -- start 2024-03-12T05:57:17.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.810+0000 7f00c4815640 1 -- start start 2024-03-12T05:57:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.810+0000 7f00c4815640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00c0100290 0x7f00c01985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.810+0000 7f00c4815640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f00c0105360 0x7f00c0198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.810+0000 7f00c4815640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c0199080 0x7f00c01a3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.810+0000 7f00c4815640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c01a3d00 con 0x7f00c0100290 2024-03-12T05:57:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.810+0000 7f00c4815640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c01a3e70 con 0x7f00c0105360 2024-03-12T05:57:17.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.810+0000 7f00c4815640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c01a4030 con 0x7f00c0199080 2024-03-12T05:57:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.810+0000 7f00be59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00c0100290 0x7f00c01985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.810+0000 7f00be59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00c0100290 0x7f00c01985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47846/0 (socket says 172.21.15.76:47846) 2024-03-12T05:57:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.810+0000 7f00be59c640 1 -- 172.21.15.76:0/2410917657 learned_addr learned my addr 172.21.15.76:0/2410917657 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.811+0000 7f00b7fff640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f00c0105360 0x7f00c0198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.811+0000 7f00bed9d640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c0199080 0x7f00c01a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.811+0000 7f00be59c640 1 -- 172.21.15.76:0/2410917657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c0199080 msgr2=0x7f00c01a3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.811+0000 7f00be59c640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c0199080 0x7f00c01a3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.811+0000 7f00be59c640 1 -- 172.21.15.76:0/2410917657 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f00c0105360 msgr2=0x7f00c0198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.811+0000 7f00be59c640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f00c0105360 0x7f00c0198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.811+0000 7f00be59c640 1 -- 172.21.15.76:0/2410917657 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a8009560 con 0x7f00c0100290 2024-03-12T05:57:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.811+0000 7f00bed9d640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c0199080 0x7f00c01a3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:57:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.811+0000 7f00be59c640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00c0100290 0x7f00c01985b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f00b000c920 tx=0x7f00b000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.811+0000 7f00b77fe640 1 -- 172.21.15.76:0/2410917657 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b0007c70 con 0x7f00c0100290 2024-03-12T05:57:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.811+0000 7f00b77fe640 1 -- 172.21.15.76:0/2410917657 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00b0007dd0 con 0x7f00c0100290 2024-03-12T05:57:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.811+0000 7f00c4815640 1 -- 172.21.15.76:0/2410917657 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c01a4310 con 0x7f00c0100290 2024-03-12T05:57:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.811+0000 7f00b77fe640 1 -- 172.21.15.76:0/2410917657 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b0018860 con 0x7f00c0100290 2024-03-12T05:57:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.811+0000 7f00c4815640 1 -- 172.21.15.76:0/2410917657 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c01a47f0 con 0x7f00c0100290 2024-03-12T05:57:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.813+0000 7f00b77fe640 1 -- 172.21.15.76:0/2410917657 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 25) v1 ==== 136512+0+0 (secure 0 0 0) 0x7f00b000fac0 con 0x7f00c0100290 2024-03-12T05:57:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.813+0000 7f00c73ed740 1 Processor -- start 2024-03-12T05:57:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00c73ed740 1 -- start start 2024-03-12T05:57:17.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00c73ed740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a717630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00c73ed740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a717b70 0x55c68a720210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00c73ed740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a720750 0x55c68a722b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00c73ed740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68a6d2960 con 0x55c68a720750 2024-03-12T05:57:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00c73ed740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68a686310 con 0x55c68a717250 2024-03-12T05:57:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00c73ed740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68a683840 con 0x55c68a717b70 2024-03-12T05:57:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00b77fe640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f00ac0a1cc0 0x7f00ac0a4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00b7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a720750 0x55c68a722b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00bed9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a717630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00b7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a720750 0x55c68a722b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47860/0 (socket says 172.21.15.76:47860) 2024-03-12T05:57:17.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00b7fff640 1 -- 172.21.15.76:0/548674176 learned_addr learned my addr 172.21.15.76:0/548674176 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00bed9d640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f00ac0a1cc0 0x7f00ac0a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00be59c640 1 --2- 172.21.15.76:0/548674176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a717b70 0x55c68a720210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00b77fe640 1 -- 172.21.15.76:0/2410917657 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f00b00cdbe0 con 0x7f00c0100290 2024-03-12T05:57:17.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00b7fff640 1 -- 172.21.15.76:0/548674176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a717b70 msgr2=0x55c68a720210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00b7fff640 1 --2- 172.21.15.76:0/548674176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a717b70 0x55c68a720210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00b7fff640 1 -- 172.21.15.76:0/548674176 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 msgr2=0x55c68a717630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00b7fff640 1 --2- 172.21.15.76:0/548674176 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a717630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00b7fff640 1 -- 172.21.15.76:0/548674176 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c68a641060 con 0x55c68a720750 2024-03-12T05:57:17.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00bed9d640 1 --2- 172.21.15.76:0/548674176 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a717630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:57:17.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.814+0000 7f00be59c640 1 --2- 172.21.15.76:0/548674176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a717b70 0x55c68a720210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:17.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.815+0000 7f00b7fff640 1 --2- 172.21.15.76:0/548674176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a720750 0x55c68a722b40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f00c006a870 tx=0x7f00a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162265307fb969e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.815+0000 7f00bed9d640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f00ac0a1cc0 0x7f00ac0a4180 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f00b80094d0 tx=0x7f00b8009500 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.815+0000 7f00b4ff9640 1 -- 172.21.15.76:0/548674176 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8004970 con 0x55c68a720750 2024-03-12T05:57:17.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.815+0000 7f00b4ff9640 1 -- 172.21.15.76:0/548674176 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00a8004ad0 con 0x55c68a720750 2024-03-12T05:57:17.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.815+0000 7f00b4ff9640 1 -- 172.21.15.76:0/548674176 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8005220 con 0x55c68a720750 2024-03-12T05:57:17.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 -- 172.21.15.76:0/548674176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a720750 msgr2=0x55c68a722b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 --2- 172.21.15.76:0/548674176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a720750 0x55c68a722b40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f00c006a870 tx=0x7f00a80046c0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 -- 172.21.15.76:0/548674176 shutdown_connections 2024-03-12T05:57:17.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 --2- 172.21.15.76:0/548674176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a720750 0x55c68a722b40 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 --2- 172.21.15.76:0/548674176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a717b70 0x55c68a720210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 --2- 172.21.15.76:0/548674176 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a717630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 -- 172.21.15.76:0/548674176 >> 172.21.15.76:0/548674176 conn(0x55c68a711980 msgr2=0x55c68a711d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 -- 172.21.15.76:0/548674176 shutdown_connections 2024-03-12T05:57:17.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 -- 172.21.15.76:0/548674176 wait complete. 2024-03-12T05:57:17.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 -- start start 2024-03-12T05:57:17.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 Processor -- start 2024-03-12T05:57:17.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a71d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a71dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a720750 0x55c68a71e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68a67f020 con 0x55c68a717b70 2024-03-12T05:57:17.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68a352430 con 0x55c68a717250 2024-03-12T05:57:17.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68a30ee00 con 0x55c68a720750 2024-03-12T05:57:17.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.816+0000 7f00c73ed740 10 client.0 _ll_register_callbacks cb 0x55c68a626cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:17.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00b7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a71dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00be59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a71d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00b7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a71dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47862/0 (socket says 172.21.15.76:47862) 2024-03-12T05:57:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00be59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a71d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:50080/0 (socket says 172.21.15.76:50080) 2024-03-12T05:57:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00b7fff640 1 -- 172.21.15.76:0/2153802487 learned_addr learned my addr 172.21.15.76:0/2153802487 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00bed9d640 1 --2- 172.21.15.76:0/2153802487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a720750 0x55c68a71e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00b7fff640 1 -- 172.21.15.76:0/2153802487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a720750 msgr2=0x55c68a71e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00b7fff640 1 --2- 172.21.15.76:0/2153802487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a720750 0x55c68a71e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00b7fff640 1 -- 172.21.15.76:0/2153802487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 msgr2=0x55c68a71d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00b7fff640 1 --2- 172.21.15.76:0/2153802487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a71d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00b7fff640 1 -- 172.21.15.76:0/2153802487 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c010d740 con 0x55c68a717b70 2024-03-12T05:57:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00be59c640 1 --2- 172.21.15.76:0/2153802487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a71d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:57:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00b7fff640 1 --2- 172.21.15.76:0/2153802487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a71dc80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f00a802c170 tx=0x7f00a8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15097ddbf0264cd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00a5ffb640 1 -- 172.21.15.76:0/2153802487 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8004970 con 0x55c68a717b70 2024-03-12T05:57:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00a5ffb640 1 -- 172.21.15.76:0/2153802487 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f00a8005620 con 0x55c68a717b70 2024-03-12T05:57:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00c73ed740 1 -- 172.21.15.76:0/2153802487 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c68a37e7e0 con 0x55c68a717b70 2024-03-12T05:57:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00a5ffb640 1 -- 172.21.15.76:0/2153802487 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00a804a450 con 0x55c68a717b70 2024-03-12T05:57:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00c73ed740 10 client.15939 fetch_fsmap learned FSMap version 22 2024-03-12T05:57:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00c73ed740 1 -- 172.21.15.76:0/2153802487 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55c68a67d6a0 con 0x55c68a717b70 2024-03-12T05:57:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.817+0000 7f00a5ffb640 1 -- 172.21.15.76:0/2153802487 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c68a37e7e0 con 0x55c68a717b70 2024-03-12T05:57:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.818+0000 7f00a5ffb640 1 -- 172.21.15.76:0/2153802487 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f00a803c050 con 0x55c68a717b70 2024-03-12T05:57:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.818+0000 7f00c73ed740 10 client.15939 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:57:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.818+0000 7f00c73ed740 10 client.15939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.818+0000 7f00c73ed740 20 client.15939 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.818+0000 7f00c73ed740 1 --2- 172.21.15.76:0/2153802487 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c68a7306b0 0x55c68a732af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.818+0000 7f00c73ed740 4 client.15939 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:57:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.818+0000 7f00c73ed740 1 -- 172.21.15.76:0/2153802487 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c68a3bb720 con 0x55c68a7306b0 2024-03-12T05:57:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.818+0000 7f00be59c640 1 --2- 172.21.15.76:0/2153802487 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c68a7306b0 0x55c68a732af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.818+0000 7f00be59c640 1 --2- 172.21.15.76:0/2153802487 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c68a7306b0 0x55c68a732af0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1de771da142a18fd server_cookie=d76f0e0f9a4f6c73 in_seq=0 out_seq=0 2024-03-12T05:57:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.818+0000 7f00a5ffb640 10 client.15939 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00a5ffb640 1 -- 172.21.15.76:0/2153802487 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c68a3bb720 con 0x55c68a7306b0 2024-03-12T05:57:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00a5ffb640 10 client.15939 handle_command_reply: tid=0 2024-03-12T05:57:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00c73ed740 1 client.15939 shutdown 2024-03-12T05:57:17.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00c73ed740 1 -- 172.21.15.76:0/2153802487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 msgr2=0x55c68a71dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00c73ed740 1 --2- 172.21.15.76:0/2153802487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a71dc80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f00a802c170 tx=0x7f00a8005fd0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00c73ed740 1 -- 172.21.15.76:0/2153802487 shutdown_connections 2024-03-12T05:57:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00c73ed740 1 --2- 172.21.15.76:0/2153802487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a720750 0x55c68a71e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00c73ed740 1 --2- 172.21.15.76:0/2153802487 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c68a7306b0 0x55c68a732af0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00c73ed740 1 --2- 172.21.15.76:0/2153802487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a71dc80 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00a5ffb640 0 client.15939 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00c73ed740 1 --2- 172.21.15.76:0/2153802487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a71d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00c73ed740 1 -- 172.21.15.76:0/2153802487 >> 172.21.15.76:0/2153802487 conn(0x55c68a711980 msgr2=0x55c68a7136d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00c73ed740 1 -- 172.21.15.76:0/2153802487 shutdown_connections 2024-03-12T05:57:17.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00c73ed740 1 -- 172.21.15.76:0/2153802487 wait complete. 2024-03-12T05:57:17.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.820+0000 7f00c73ed740 20 client.15939 trim_cache size 0 max 16384 2024-03-12T05:57:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.832+0000 7f00c73ed740 1 Processor -- start 2024-03-12T05:57:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.832+0000 7f00c73ed740 1 -- start start 2024-03-12T05:57:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00c73ed740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a717250 0x55c68a71dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00c73ed740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a710a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00c73ed740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a720750 0x55c68a7275a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00c73ed740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68a604130 con 0x55c68a717b70 2024-03-12T05:57:17.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00c73ed740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68a716eb0 con 0x55c68a720750 2024-03-12T05:57:17.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00c73ed740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68a72d5b0 con 0x55c68a717250 2024-03-12T05:57:17.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00b7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a710a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00b7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a710a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47866/0 (socket says 172.21.15.76:47866) 2024-03-12T05:57:17.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00b7fff640 1 -- 172.21.15.76:0/3913660799 learned_addr learned my addr 172.21.15.76:0/3913660799 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00bed9d640 1 --2- 172.21.15.76:0/3913660799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a720750 0x55c68a7275a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00be59c640 1 --2- 172.21.15.76:0/3913660799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a717250 0x55c68a71dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00b7fff640 1 -- 172.21.15.76:0/3913660799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a717250 msgr2=0x55c68a71dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00b7fff640 1 --2- 172.21.15.76:0/3913660799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a717250 0x55c68a71dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00b7fff640 1 -- 172.21.15.76:0/3913660799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a720750 msgr2=0x55c68a7275a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00b7fff640 1 --2- 172.21.15.76:0/3913660799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a720750 0x55c68a7275a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00b7fff640 1 -- 172.21.15.76:0/3913660799 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c68a619c20 con 0x55c68a717b70 2024-03-12T05:57:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00b7fff640 1 --2- 172.21.15.76:0/3913660799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a710a00 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f00a802c240 tx=0x7f00a802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac5e267180e5ef4e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00a7fff640 1 -- 172.21.15.76:0/3913660799 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8004e70 con 0x55c68a717b70 2024-03-12T05:57:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00a7fff640 1 -- 172.21.15.76:0/3913660799 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00a8005870 con 0x55c68a717b70 2024-03-12T05:57:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.833+0000 7f00a7fff640 1 -- 172.21.15.76:0/3913660799 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a804a6b0 con 0x55c68a717b70 2024-03-12T05:57:17.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.834+0000 7f00c73ed740 1 -- 172.21.15.76:0/3913660799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 msgr2=0x55c68a710a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.834+0000 7f00c73ed740 1 --2- 172.21.15.76:0/3913660799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a710a00 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f00a802c240 tx=0x7f00a802bc30 comp rx=0 tx=0).stop 2024-03-12T05:57:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.834+0000 7f00c73ed740 1 -- 172.21.15.76:0/3913660799 shutdown_connections 2024-03-12T05:57:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.834+0000 7f00c73ed740 1 --2- 172.21.15.76:0/3913660799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a720750 0x55c68a7275a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.834+0000 7f00c73ed740 1 --2- 172.21.15.76:0/3913660799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a710a00 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.834+0000 7f00c73ed740 1 --2- 172.21.15.76:0/3913660799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a717250 0x55c68a71dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.834+0000 7f00c73ed740 1 -- 172.21.15.76:0/3913660799 >> 172.21.15.76:0/3913660799 conn(0x55c68a711980 msgr2=0x55c68a7102f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.834+0000 7f00c73ed740 1 -- 172.21.15.76:0/3913660799 shutdown_connections 2024-03-12T05:57:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.834+0000 7f00c73ed740 1 -- 172.21.15.76:0/3913660799 wait complete. 2024-03-12T05:57:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.834+0000 7f00c73ed740 1 -- start start 2024-03-12T05:57:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.834+0000 7f00c73ed740 1 Processor -- start 2024-03-12T05:57:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00c73ed740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a7260f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00c73ed740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a726630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00c73ed740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a720750 0x55c68a726b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00c73ed740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68a78b030 con 0x55c68a717b70 2024-03-12T05:57:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00c73ed740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68a78b1a0 con 0x55c68a717250 2024-03-12T05:57:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00c73ed740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68a78b360 con 0x55c68a720750 2024-03-12T05:57:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00c73ed740 10 client.0 _ll_register_callbacks cb 0x7f00a8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00b7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a726630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00b7fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a726630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47882/0 (socket says 172.21.15.76:47882) 2024-03-12T05:57:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00b7fff640 1 -- 172.21.15.76:0/2276384910 learned_addr learned my addr 172.21.15.76:0/2276384910 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00be59c640 1 --2- 172.21.15.76:0/2276384910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a7260f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00bed9d640 1 --2- 172.21.15.76:0/2276384910 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a720750 0x55c68a726b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00b7fff640 1 -- 172.21.15.76:0/2276384910 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a720750 msgr2=0x55c68a726b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00b7fff640 1 --2- 172.21.15.76:0/2276384910 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a720750 0x55c68a726b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00b7fff640 1 -- 172.21.15.76:0/2276384910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 msgr2=0x55c68a7260f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00b7fff640 1 --2- 172.21.15.76:0/2276384910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a7260f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00b7fff640 1 -- 172.21.15.76:0/2276384910 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c68a67d6a0 con 0x55c68a717b70 2024-03-12T05:57:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00b7fff640 1 --2- 172.21.15.76:0/2276384910 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a726630 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f00a802c170 tx=0x7f00a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cccee3bf1265302 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:17.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00a5ffb640 1 -- 172.21.15.76:0/2276384910 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a802cc00 con 0x55c68a717b70 2024-03-12T05:57:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00a5ffb640 1 -- 172.21.15.76:0/2276384910 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f00a8000e40 con 0x55c68a717b70 2024-03-12T05:57:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.835+0000 7f00c73ed740 1 -- 172.21.15.76:0/2276384910 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f00a8004ad0 con 0x55c68a717b70 2024-03-12T05:57:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.836+0000 7f00a5ffb640 1 -- 172.21.15.76:0/2276384910 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00a8004040 con 0x55c68a717b70 2024-03-12T05:57:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.836+0000 7f00c73ed740 10 client.15945 fetch_fsmap learned FSMap version 22 2024-03-12T05:57:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.836+0000 7f00c73ed740 1 -- 172.21.15.76:0/2276384910 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55c68a63f8c0 con 0x55c68a717b70 2024-03-12T05:57:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.836+0000 7f00a5ffb640 1 -- 172.21.15.76:0/2276384910 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00a8072080 con 0x55c68a717b70 2024-03-12T05:57:17.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.836+0000 7f00a5ffb640 1 -- 172.21.15.76:0/2276384910 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f00a803c070 con 0x55c68a717b70 2024-03-12T05:57:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.836+0000 7f00c73ed740 10 client.15945 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:57:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.836+0000 7f00c73ed740 10 client.15945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.836+0000 7f00c73ed740 20 client.15945 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.836+0000 7f00c73ed740 1 --2- 172.21.15.76:0/2276384910 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c68a78c7e0 0x55c68a78ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.836+0000 7f00c73ed740 4 client.15945 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:57:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.836+0000 7f00c73ed740 1 -- 172.21.15.76:0/2276384910 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c68a40a890 con 0x55c68a78c7e0 2024-03-12T05:57:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.836+0000 7f00be59c640 1 --2- 172.21.15.76:0/2276384910 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c68a78c7e0 0x55c68a78ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.836+0000 7f00a5ffb640 10 client.15945 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:17.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.836+0000 7f00be59c640 1 --2- 172.21.15.76:0/2276384910 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c68a78c7e0 0x55c68a78ec00 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35a42842820e650b server_cookie=9e08a39c9a3b1008 in_seq=0 out_seq=0 2024-03-12T05:57:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.838+0000 7f00a5ffb640 1 -- 172.21.15.76:0/2276384910 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c68a40a890 con 0x55c68a78c7e0 2024-03-12T05:57:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.838+0000 7f00a5ffb640 10 client.15945 handle_command_reply: tid=0 2024-03-12T05:57:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.838+0000 7f00c73ed740 1 client.15945 shutdown 2024-03-12T05:57:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.838+0000 7f00c73ed740 1 -- 172.21.15.76:0/2276384910 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 msgr2=0x55c68a726630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.838+0000 7f00c73ed740 1 --2- 172.21.15.76:0/2276384910 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a726630 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f00a802c170 tx=0x7f00a80046c0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.838+0000 7f00c73ed740 1 -- 172.21.15.76:0/2276384910 shutdown_connections 2024-03-12T05:57:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.838+0000 7f00c73ed740 1 --2- 172.21.15.76:0/2276384910 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68a720750 0x55c68a726b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.838+0000 7f00c73ed740 1 --2- 172.21.15.76:0/2276384910 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c68a78c7e0 0x55c68a78ec00 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.838+0000 7f00c73ed740 1 --2- 172.21.15.76:0/2276384910 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c68a717b70 0x55c68a726630 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.838+0000 7f00c73ed740 1 --2- 172.21.15.76:0/2276384910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c68a717250 0x55c68a7260f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.838+0000 7f00a5ffb640 0 client.15945 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.838+0000 7f00c73ed740 1 -- 172.21.15.76:0/2276384910 >> 172.21.15.76:0/2276384910 conn(0x55c68a711980 msgr2=0x55c68a710b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c73ed740 1 -- 172.21.15.76:0/2276384910 shutdown_connections 2024-03-12T05:57:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c73ed740 1 -- 172.21.15.76:0/2276384910 wait complete. 2024-03-12T05:57:17.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c73ed740 20 client.15945 trim_cache size 0 max 16384 2024-03-12T05:57:17.864 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:57:17.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c4815640 1 -- 172.21.15.76:0/2410917657 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f00ac0a1cc0 msgr2=0x7f00ac0a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c4815640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f00ac0a1cc0 0x7f00ac0a4180 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f00b80094d0 tx=0x7f00b8009500 comp rx=0 tx=0).stop 2024-03-12T05:57:17.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c4815640 1 -- 172.21.15.76:0/2410917657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00c0100290 msgr2=0x7f00c01985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:17.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c4815640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00c0100290 0x7f00c01985b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f00b000c920 tx=0x7f00b000cde0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c4815640 1 -- 172.21.15.76:0/2410917657 shutdown_connections 2024-03-12T05:57:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c4815640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f00ac0a1cc0 0x7f00ac0a4180 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c4815640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c0199080 0x7f00c01a3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c4815640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f00c0105360 0x7f00c0198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c4815640 1 --2- 172.21.15.76:0/2410917657 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f00c0100290 0x7f00c01985b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c4815640 1 -- 172.21.15.76:0/2410917657 >> 172.21.15.76:0/2410917657 conn(0x7f00c0077230 msgr2=0x7f00c00fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c4815640 1 -- 172.21.15.76:0/2410917657 shutdown_connections 2024-03-12T05:57:17.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:17.839+0000 7f00c4815640 1 -- 172.21.15.76:0/2410917657 wait complete. 2024-03-12T05:57:18.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:18 smithi082 ceph-mon[36382]: pgmap v48: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 513 KiB/s wr, 26 op/s 2024-03-12T05:57:18.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:18 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/360230422' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:57:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:18 smithi088 ceph-mon[36585]: pgmap v48: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 513 KiB/s wr, 26 op/s 2024-03-12T05:57:18.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:18 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/360230422' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:57:18.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:18 smithi076 ceph-mon[32095]: pgmap v48: 97 pgs: 97 active+clean; 20 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 513 KiB/s wr, 26 op/s 2024-03-12T05:57:18.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:18 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/360230422' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:57:18.529 INFO:tasks.workunit.client.0.smithi076.stderr:Updating files: 7% (321/4261) Updating files: 8% (341/4261) Updating files: 9% (384/4261) Updating files: 10% (427/4261) Updating files: 11% (469/4261) Updating files: 11% (489/4261) Updating files: 12% (512/4261) Updating files: 12% (548/4261) Updating files: 13% (554/4261) Updating files: 14% (597/4261) Updating files: 15% (640/4261) Updating files: 16% (682/4261) Updating files: 17% (725/4261) Updating files: 18% (767/4261) Updating files: 19% (810/4261) Updating files: 20% (853/4261) Updating files: 21% (895/4261) Updating files: 22% (938/4261) Updating files: 23% (981/4261) Updating files: 24% (1023/4261) Updating files: 25% (1066/4261) Updating files: 26% (1108/4261) Updating files: 27% (1151/4261) Updating files: 28% (1194/4261) Updating files: 29% (1236/4261) Updating files: 29% (1267/4261) Updating files: 30% (1279/4261) Updating files: 31% (1321/4261) Updating files: 32% (1364/4261) Updating files: 33% (1407/4261) Updating files: 34% (1449/4261) Updating files: 35% (1492/4261) Updating files: 36% (1534/4261) Updating files: 37% (1577/4261) Updating files: 38% (1620/4261) Updating files: 39% (1662/4261) Updating files: 40% (1705/4261) Updating files: 41% (1748/4261) Updating files: 42% (1790/4261) Updating files: 43% (1833/4261) Updating files: 44% (1875/4261) Updating files: 45% (1918/4261) Updating files: 46% (1961/4261) Updating files: 47% (2003/4261) Updating files: 48% (2046/4261) Updating files: 49% (2088/4261) Updating files: 50% (2131/4261) Updating files: 51% (2174/4261) Updating files: 51% (2176/4261) Updating files: 52% (2216/4261) Updating files: 53% (2259/4261) Updating files: 54% (2301/4261) Updating files: 55% (2344/4261) Updating files: 56% (2387/4261) Updating files: 57% (2429/4261) Updating files: 58% (2472/4261) Updating files: 59% (2514/4261) Updating files: 60% (2557/4261) Updating files: 61% (2600/4261) Updating files: 62% (2642/4261) Updating files: 63% (2685/4261) Updating files: 64% (2728/4261) Updating files: 64% (2757/4261) Updating files: 65% (2770/4261) Updating files: 65% (2788/4261) Updating files: 66% (2813/4261) Updating files: 67% (2855/4261) Updating files: 68% (2898/4261) Updating files: 69% (2941/4261) Updating files: 70% (2983/4261) Updating files: 70% (2995/4261) Updating files: 71% (3026/4261) Updating files: 71% (3051/4261) Updating files: 72% (3068/4261) Updating files: 73% (3111/4261) Updating files: 74% (3154/4261) Updating files: 75% (3196/4261) Updating files: 76% (3239/4261) Updating files: 77% (3281/4261) Updating files: 78% (3324/4261) Updating files: 79% (3367/4261) Updating files: 80% (3409/4261) Updating files: 81% (3452/4261) Updating files: 82% (3495/4261) Updating files: 83% (3537/4261) Updating files: 84% (3580/4261) Updating files: 85% (3622/4261) Updating files: 86% (3665/4261) Updating files: 87% (3708/4261) Updating files: 88% (3750/4261) Updating files: 89% (3793/4261) Updating files: 90% (3835/4261) Updating files: 91% (3878/4261) Updating files: 92% (3921/4261) Updating files: 93% (3963/4261) Updating files: 94% (4006/4261) Updating files: 95% (4048/4261) Updating files: 95% (4050/4261) Updating files: 96% (4091/4261) Updating files: 97% (4134/4261) Updating files: 98% (4176/4261) Updating files: 99% (4219/4261) Updating files: 100% (4261/4261) Updating files: 100% (4261/4261), done. 2024-03-12T05:57:18.536 INFO:tasks.workunit.client.0.smithi076.stderr:+ cd xfstests-dev 2024-03-12T05:57:18.536 INFO:tasks.workunit.client.0.smithi076.stderr:+ git checkout 12973fc04fd10d4af086901e10ffa8e48866b735 2024-03-12T05:57:18.869 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:57:18.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:57:18.870 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:57:18.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.973+0000 7f73599e9640 1 -- 172.21.15.76:0/1605485717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7354105350 msgr2=0x7f735410cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:18.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.973+0000 7f73599e9640 1 --2- 172.21.15.76:0/1605485717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7354105350 0x7f735410cf20 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f734c009cd0 tx=0x7f734c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.973+0000 7f73599e9640 1 -- 172.21.15.76:0/1605485717 shutdown_connections 2024-03-12T05:57:18.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.973+0000 7f73599e9640 1 --2- 172.21.15.76:0/1605485717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7354105350 0x7f735410cf20 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.973+0000 7f73599e9640 1 --2- 172.21.15.76:0/1605485717 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7354100c30 0x7f7354104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.973+0000 7f73599e9640 1 --2- 172.21.15.76:0/1605485717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7354100280 0x7f7354100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.973+0000 7f73599e9640 1 -- 172.21.15.76:0/1605485717 >> 172.21.15.76:0/1605485717 conn(0x7f73540771a0 msgr2=0x7f73540775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:18.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.973+0000 7f73599e9640 1 -- 172.21.15.76:0/1605485717 shutdown_connections 2024-03-12T05:57:18.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.973+0000 7f73599e9640 1 -- 172.21.15.76:0/1605485717 wait complete. 2024-03-12T05:57:18.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.973+0000 7f73599e9640 1 Processor -- start 2024-03-12T05:57:18.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f73599e9640 1 -- start start 2024-03-12T05:57:18.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f73599e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7354100280 0x7f73541985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:18.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f73599e9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7354105350 0x7f7354198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:18.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f73599e9640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7354199090 0x7f73541a3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:18.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f73599e9640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73541a3d10 con 0x7f7354100280 2024-03-12T05:57:18.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f73599e9640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73541a3e80 con 0x7f7354199090 2024-03-12T05:57:18.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f73599e9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73541a4040 con 0x7f7354105350 2024-03-12T05:57:18.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f7352ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7354100280 0x7f73541985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:18.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f7352ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7354100280 0x7f73541985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47912/0 (socket says 172.21.15.76:47912) 2024-03-12T05:57:18.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f7352ffd640 1 -- 172.21.15.76:0/347141575 learned_addr learned my addr 172.21.15.76:0/347141575 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:18.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f73537fe640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7354199090 0x7f73541a3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:18.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f73527fc640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7354105350 0x7f7354198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f7352ffd640 1 -- 172.21.15.76:0/347141575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7354105350 msgr2=0x7f7354198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f7352ffd640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7354105350 0x7f7354198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f7352ffd640 1 -- 172.21.15.76:0/347141575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7354199090 msgr2=0x7f73541a3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f7352ffd640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7354199090 0x7f73541a3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f7352ffd640 1 -- 172.21.15.76:0/347141575 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c009900 con 0x7f7354100280 2024-03-12T05:57:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.974+0000 7f73537fe640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7354199090 0x7f73541a3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:57:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.975+0000 7f7352ffd640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7354100280 0x7f73541985c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f734400c890 tx=0x7f734400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.975+0000 7f73589e7640 1 -- 172.21.15.76:0/347141575 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7344004220 con 0x7f7354100280 2024-03-12T05:57:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.975+0000 7f73589e7640 1 -- 172.21.15.76:0/347141575 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7344007e90 con 0x7f7354100280 2024-03-12T05:57:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.975+0000 7f73599e9640 1 -- 172.21.15.76:0/347141575 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73541a4320 con 0x7f7354100280 2024-03-12T05:57:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.975+0000 7f73589e7640 1 -- 172.21.15.76:0/347141575 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7344004a00 con 0x7f7354100280 2024-03-12T05:57:18.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.975+0000 7f73599e9640 1 -- 172.21.15.76:0/347141575 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73541a4830 con 0x7f7354100280 2024-03-12T05:57:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.977+0000 7f73589e7640 1 -- 172.21.15.76:0/347141575 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f73440079b0 con 0x7f7354100280 2024-03-12T05:57:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.977+0000 7f735c5c2740 1 Processor -- start 2024-03-12T05:57:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.977+0000 7f73589e7640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f733c0a1ba0 0x7f733c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.977+0000 7f735c5c2740 1 -- start start 2024-03-12T05:57:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f73537fe640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f733c0a1ba0 0x7f733c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f735c5c2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e5951c380 0x557e5951c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f73589e7640 1 -- 172.21.15.76:0/347141575 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f73440cd490 con 0x7f7354100280 2024-03-12T05:57:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f735c5c2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951cd30 0x557e595252e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f735c5c2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e595258b0 0x557e59527d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f735c5c2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e5948af90 con 0x557e5951c380 2024-03-12T05:57:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f735c5c2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e594884c0 con 0x557e5951cd30 2024-03-12T05:57:18.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f735c5c2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e59483ca0 con 0x557e595258b0 2024-03-12T05:57:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f7352ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e5951c380 0x557e5951c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f7352ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e5951c380 0x557e5951c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47918/0 (socket says 172.21.15.76:47918) 2024-03-12T05:57:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f7352ffd640 1 -- 172.21.15.76:0/453669510 learned_addr learned my addr 172.21.15.76:0/453669510 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f73527fc640 1 --2- 172.21.15.76:0/453669510 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951cd30 0x557e595252e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f73537fe640 1 --2- 172.21.15.76:0/453669510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e595258b0 0x557e59527d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f7352ffd640 1 -- 172.21.15.76:0/453669510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e595258b0 msgr2=0x557e59527d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f7352ffd640 1 --2- 172.21.15.76:0/453669510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e595258b0 0x557e59527d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f7352ffd640 1 -- 172.21.15.76:0/453669510 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951cd30 msgr2=0x557e595252e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f7352ffd640 1 --2- 172.21.15.76:0/453669510 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951cd30 0x557e595252e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f7352ffd640 1 -- 172.21.15.76:0/453669510 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e594460b0 con 0x557e5951c380 2024-03-12T05:57:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f73537fe640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f733c0a1ba0 0x7f733c0a4060 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f734c005cc0 tx=0x7f734c032f30 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f7352ffd640 1 --2- 172.21.15.76:0/453669510 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e5951c380 0x557e5951c760 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f7344096b10 tx=0x7f734409a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4a7d70f5ba6b8b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f7339ffb640 1 -- 172.21.15.76:0/453669510 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73440a2830 con 0x557e5951c380 2024-03-12T05:57:18.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f7339ffb640 1 -- 172.21.15.76:0/453669510 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73440981e0 con 0x557e5951c380 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.978+0000 7f7339ffb640 1 -- 172.21.15.76:0/453669510 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7344099590 con 0x557e5951c380 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.979+0000 7f735c5c2740 1 -- 172.21.15.76:0/453669510 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e5951c380 msgr2=0x557e5951c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.979+0000 7f735c5c2740 1 --2- 172.21.15.76:0/453669510 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e5951c380 0x557e5951c760 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f7344096b10 tx=0x7f734409a140 comp rx=0 tx=0).stop 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.979+0000 7f735c5c2740 1 -- 172.21.15.76:0/453669510 shutdown_connections 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.979+0000 7f735c5c2740 1 --2- 172.21.15.76:0/453669510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e595258b0 0x557e59527d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.979+0000 7f735c5c2740 1 --2- 172.21.15.76:0/453669510 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951cd30 0x557e595252e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.979+0000 7f735c5c2740 1 --2- 172.21.15.76:0/453669510 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e5951c380 0x557e5951c760 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.979+0000 7f735c5c2740 1 -- 172.21.15.76:0/453669510 >> 172.21.15.76:0/453669510 conn(0x557e59516840 msgr2=0x557e59516c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.979+0000 7f735c5c2740 1 -- 172.21.15.76:0/453669510 shutdown_connections 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.979+0000 7f735c5c2740 1 -- 172.21.15.76:0/453669510 wait complete. 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.979+0000 7f735c5c2740 1 -- start start 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f735c5c2740 1 Processor -- start 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f735c5c2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e5951c380 0x557e59522860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f735c5c2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 0x557e59522da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:18.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f735c5c2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e595258b0 0x557e595232e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f735c5c2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e59428f40 con 0x557e5951c380 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f735c5c2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e591574b0 con 0x557e595258b0 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f735c5c2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e59409670 con 0x557e5951cd30 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f735c5c2740 10 client.0 _ll_register_callbacks cb 0x557e59476f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f7352ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e5951c380 0x557e59522860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f73537fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e595258b0 0x557e595232e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f7352ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e5951c380 0x557e59522860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47932/0 (socket says 172.21.15.76:47932) 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f7352ffd640 1 -- 172.21.15.76:0/3640281872 learned_addr learned my addr 172.21.15.76:0/3640281872 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f73527fc640 1 --2- 172.21.15.76:0/3640281872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 0x557e59522da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f73537fe640 1 -- 172.21.15.76:0/3640281872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 msgr2=0x557e59522da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f73537fe640 1 --2- 172.21.15.76:0/3640281872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 0x557e59522da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f73537fe640 1 -- 172.21.15.76:0/3640281872 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e5951c380 msgr2=0x557e59522860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f73537fe640 1 --2- 172.21.15.76:0/3640281872 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e5951c380 0x557e59522860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f73537fe640 1 -- 172.21.15.76:0/3640281872 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73541a4830 con 0x557e595258b0 2024-03-12T05:57:18.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f7352ffd640 1 --2- 172.21.15.76:0/3640281872 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e5951c380 0x557e59522860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f73537fe640 1 --2- 172.21.15.76:0/3640281872 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e595258b0 0x557e595232e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f734c02c630 tx=0x7f734c02c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a99b6bcf8f238a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.980+0000 7f7332ffd640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f7332ffd640 1 -- 172.21.15.76:0/3640281872 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c003b70 con 0x557e595258b0 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f735c5c2740 1 -- 172.21.15.76:0/3640281872 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557e5945df40 con 0x557e595258b0 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f7332ffd640 1 -- 172.21.15.76:0/3640281872 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f734c003cd0 con 0x557e595258b0 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f7332ffd640 1 -- 172.21.15.76:0/3640281872 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f734c032510 con 0x557e595258b0 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f735c5c2740 10 client.25421 fetch_fsmap learned FSMap version 22 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f735c5c2740 1 -- 172.21.15.76:0/3640281872 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x557e59482320 con 0x557e595258b0 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f7332ffd640 1 -- 172.21.15.76:0/3640281872 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557e5945df40 con 0x557e595258b0 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f7332ffd640 1 -- 172.21.15.76:0/3640281872 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f734c041760 con 0x557e595258b0 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f735c5c2740 10 client.25421 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f735c5c2740 10 client.25421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f735c5c2740 20 client.25421 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f735c5c2740 1 --2- 172.21.15.76:0/3640281872 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557e59535ee0 0x557e59538300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f735c5c2740 4 client.25421 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:57:18.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f735c5c2740 1 -- 172.21.15.76:0/3640281872 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557e5947a310 con 0x557e59535ee0 2024-03-12T05:57:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.981+0000 7f7352ffd640 1 --2- 172.21.15.76:0/3640281872 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557e59535ee0 0x557e59538300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.982+0000 7f7332ffd640 10 client.25421 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.982+0000 7f7352ffd640 1 --2- 172.21.15.76:0/3640281872 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557e59535ee0 0x557e59538300 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dca8dca9b1be39f9 server_cookie=a26749824c7b795a in_seq=0 out_seq=0 2024-03-12T05:57:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f7332ffd640 1 -- 172.21.15.76:0/3640281872 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557e5947a310 con 0x557e59535ee0 2024-03-12T05:57:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f7332ffd640 10 client.25421 handle_command_reply: tid=0 2024-03-12T05:57:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f735c5c2740 1 client.25421 shutdown 2024-03-12T05:57:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f735c5c2740 1 -- 172.21.15.76:0/3640281872 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e595258b0 msgr2=0x557e595232e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f735c5c2740 1 --2- 172.21.15.76:0/3640281872 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e595258b0 0x557e595232e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f734c02c630 tx=0x7f734c02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f735c5c2740 1 -- 172.21.15.76:0/3640281872 shutdown_connections 2024-03-12T05:57:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f735c5c2740 1 --2- 172.21.15.76:0/3640281872 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e595258b0 0x557e595232e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f735c5c2740 1 --2- 172.21.15.76:0/3640281872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 0x557e59522da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f735c5c2740 1 --2- 172.21.15.76:0/3640281872 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557e59535ee0 0x557e59538300 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f735c5c2740 1 --2- 172.21.15.76:0/3640281872 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e5951c380 0x557e59522860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f7332ffd640 0 client.25421 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:18.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f735c5c2740 1 -- 172.21.15.76:0/3640281872 >> 172.21.15.76:0/3640281872 conn(0x557e59516840 msgr2=0x557e595187a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:18.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f735c5c2740 1 -- 172.21.15.76:0/3640281872 shutdown_connections 2024-03-12T05:57:18.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f735c5c2740 1 -- 172.21.15.76:0/3640281872 wait complete. 2024-03-12T05:57:18.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.983+0000 7f735c5c2740 20 client.25421 trim_cache size 0 max 16384 2024-03-12T05:57:19.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.998+0000 7f735c5c2740 1 Processor -- start 2024-03-12T05:57:19.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.998+0000 7f735c5c2740 1 -- start start 2024-03-12T05:57:19.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.998+0000 7f735c5c2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951c380 0x557e59524f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.998+0000 7f735c5c2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 0x557e5952ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.998+0000 7f735c5c2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e595258b0 0x557e59523100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.998+0000 7f735c5c2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e5951bcd0 con 0x557e595258b0 2024-03-12T05:57:19.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.998+0000 7f735c5c2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e5951e4b0 con 0x557e5951c380 2024-03-12T05:57:19.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.998+0000 7f735c5c2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e59512110 con 0x557e5951cd30 2024-03-12T05:57:19.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f73537fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e595258b0 0x557e59523100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f73537fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e595258b0 0x557e59523100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47942/0 (socket says 172.21.15.76:47942) 2024-03-12T05:57:19.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f73537fe640 1 -- 172.21.15.76:0/1516198280 learned_addr learned my addr 172.21.15.76:0/1516198280 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:19.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f7352ffd640 1 --2- 172.21.15.76:0/1516198280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951c380 0x557e59524f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f73527fc640 1 --2- 172.21.15.76:0/1516198280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 0x557e5952ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f73537fe640 1 -- 172.21.15.76:0/1516198280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 msgr2=0x557e5952ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:19.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f73537fe640 1 --2- 172.21.15.76:0/1516198280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 0x557e5952ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f73537fe640 1 -- 172.21.15.76:0/1516198280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951c380 msgr2=0x557e59524f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f73537fe640 1 --2- 172.21.15.76:0/1516198280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951c380 0x557e59524f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f73537fe640 1 -- 172.21.15.76:0/1516198280 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e5941ec00 con 0x557e595258b0 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f7352ffd640 1 --2- 172.21.15.76:0/1516198280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951c380 0x557e59524f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f73537fe640 1 --2- 172.21.15.76:0/1516198280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e595258b0 0x557e59523100 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f734c006010 tx=0x7f734c034e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c786e5b9b6320 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f73397fa640 1 -- 172.21.15.76:0/1516198280 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c041640 con 0x557e595258b0 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f73397fa640 1 -- 172.21.15.76:0/1516198280 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f734c032510 con 0x557e595258b0 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:18.999+0000 7f73397fa640 1 -- 172.21.15.76:0/1516198280 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c004500 con 0x557e595258b0 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.000+0000 7f735c5c2740 1 -- 172.21.15.76:0/1516198280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e595258b0 msgr2=0x557e59523100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.000+0000 7f735c5c2740 1 --2- 172.21.15.76:0/1516198280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e595258b0 0x557e59523100 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f734c006010 tx=0x7f734c034e30 comp rx=0 tx=0).stop 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.000+0000 7f735c5c2740 1 -- 172.21.15.76:0/1516198280 shutdown_connections 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.000+0000 7f735c5c2740 1 --2- 172.21.15.76:0/1516198280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e595258b0 0x557e59523100 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.000+0000 7f735c5c2740 1 --2- 172.21.15.76:0/1516198280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 0x557e5952ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.000+0000 7f735c5c2740 1 --2- 172.21.15.76:0/1516198280 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951c380 0x557e59524f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.000+0000 7f735c5c2740 1 -- 172.21.15.76:0/1516198280 >> 172.21.15.76:0/1516198280 conn(0x557e59516840 msgr2=0x557e5953cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.000+0000 7f735c5c2740 1 -- 172.21.15.76:0/1516198280 shutdown_connections 2024-03-12T05:57:19.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.000+0000 7f735c5c2740 1 -- 172.21.15.76:0/1516198280 wait complete. 2024-03-12T05:57:19.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.000+0000 7f735c5c2740 1 -- start start 2024-03-12T05:57:19.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.000+0000 7f735c5c2740 1 Processor -- start 2024-03-12T05:57:19.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f735c5c2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951c380 0x557e5958a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f735c5c2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 0x557e5958ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f735c5c2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e595258b0 0x557e595849f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f735c5c2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e59584fb0 con 0x557e595258b0 2024-03-12T05:57:19.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f735c5c2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e59585120 con 0x557e5951c380 2024-03-12T05:57:19.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f735c5c2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e595852e0 con 0x557e5951cd30 2024-03-12T05:57:19.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f735c5c2740 10 client.0 _ll_register_callbacks cb 0x7f734400a520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:19.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f7352ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951c380 0x557e5958a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f73537fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e595258b0 0x557e595849f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f73537fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e595258b0 0x557e595849f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47952/0 (socket says 172.21.15.76:47952) 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f7352ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951c380 0x557e5958a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:50144/0 (socket says 172.21.15.76:50144) 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f73537fe640 1 -- 172.21.15.76:0/2298896931 learned_addr learned my addr 172.21.15.76:0/2298896931 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f73527fc640 1 --2- 172.21.15.76:0/2298896931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 0x557e5958ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f73537fe640 1 -- 172.21.15.76:0/2298896931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 msgr2=0x557e5958ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f73537fe640 1 --2- 172.21.15.76:0/2298896931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 0x557e5958ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f73537fe640 1 -- 172.21.15.76:0/2298896931 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951c380 msgr2=0x557e5958a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f73537fe640 1 --2- 172.21.15.76:0/2298896931 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951c380 0x557e5958a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f73537fe640 1 -- 172.21.15.76:0/2298896931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557e59482320 con 0x557e595258b0 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f7352ffd640 1 --2- 172.21.15.76:0/2298896931 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951c380 0x557e5958a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f73537fe640 1 --2- 172.21.15.76:0/2298896931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e595258b0 0x557e595849f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f734c033000 tx=0x7f734c0416f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c206d047128fe109 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f7332ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f7332ffd640 1 -- 172.21.15.76:0/2298896931 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c003b80 con 0x557e595258b0 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f7332ffd640 1 -- 172.21.15.76:0/2298896931 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f734c003ce0 con 0x557e595258b0 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.001+0000 7f735c5c2740 1 -- 172.21.15.76:0/2298896931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557e594a9900 con 0x557e595258b0 2024-03-12T05:57:19.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.002+0000 7f7332ffd640 1 -- 172.21.15.76:0/2298896931 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f734c004280 con 0x557e595258b0 2024-03-12T05:57:19.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.002+0000 7f735c5c2740 10 client.15963 fetch_fsmap learned FSMap version 22 2024-03-12T05:57:19.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.002+0000 7f735c5c2740 1 -- 172.21.15.76:0/2298896931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x557e59444910 con 0x557e595258b0 2024-03-12T05:57:19.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.002+0000 7f7332ffd640 1 -- 172.21.15.76:0/2298896931 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f734c076020 con 0x557e595258b0 2024-03-12T05:57:19.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.002+0000 7f7332ffd640 1 -- 172.21.15.76:0/2298896931 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f734c0727f0 con 0x557e595258b0 2024-03-12T05:57:19.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.002+0000 7f735c5c2740 10 client.15963 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:57:19.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.002+0000 7f735c5c2740 10 client.15963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:19.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.002+0000 7f735c5c2740 20 client.15963 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:19.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.002+0000 7f735c5c2740 1 --2- 172.21.15.76:0/2298896931 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557e595920e0 0x557e595944d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.002+0000 7f735c5c2740 4 client.15963 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:57:19.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.002+0000 7f735c5c2740 1 -- 172.21.15.76:0/2298896931 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x557e59183860 con 0x557e595920e0 2024-03-12T05:57:19.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.002+0000 7f7352ffd640 1 --2- 172.21.15.76:0/2298896931 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557e595920e0 0x557e595944d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.002+0000 7f7332ffd640 10 client.15963 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:19.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.002+0000 7f7352ffd640 1 --2- 172.21.15.76:0/2298896931 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557e595920e0 0x557e595944d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0443009d84ec97b server_cookie=8f2e1375f7496085 in_seq=0 out_seq=0 2024-03-12T05:57:19.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f7332ffd640 1 -- 172.21.15.76:0/2298896931 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x557e59183860 con 0x557e595920e0 2024-03-12T05:57:19.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f7332ffd640 10 client.15963 handle_command_reply: tid=0 2024-03-12T05:57:19.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f735c5c2740 1 client.15963 shutdown 2024-03-12T05:57:19.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f735c5c2740 1 -- 172.21.15.76:0/2298896931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e595258b0 msgr2=0x557e595849f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:19.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f735c5c2740 1 --2- 172.21.15.76:0/2298896931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e595258b0 0x557e595849f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f734c033000 tx=0x7f734c0416f0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f735c5c2740 1 -- 172.21.15.76:0/2298896931 shutdown_connections 2024-03-12T05:57:19.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f735c5c2740 1 --2- 172.21.15.76:0/2298896931 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557e595920e0 0x557e595944d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f735c5c2740 1 --2- 172.21.15.76:0/2298896931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557e595258b0 0x557e595849f0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f735c5c2740 1 --2- 172.21.15.76:0/2298896931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e5951cd30 0x557e5958ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f735c5c2740 1 --2- 172.21.15.76:0/2298896931 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557e5951c380 0x557e5958a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f7332ffd640 0 client.15963 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:19.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f735c5c2740 1 -- 172.21.15.76:0/2298896931 >> 172.21.15.76:0/2298896931 conn(0x557e59516840 msgr2=0x557e59517ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:19.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f735c5c2740 1 -- 172.21.15.76:0/2298896931 shutdown_connections 2024-03-12T05:57:19.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f735c5c2740 1 -- 172.21.15.76:0/2298896931 wait complete. 2024-03-12T05:57:19.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.004+0000 7f735c5c2740 20 client.15963 trim_cache size 0 max 16384 2024-03-12T05:57:19.015 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:57:19.015 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T05:57:19.015 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "aacf2548-be29-4634-9c90-6f8c4bc73d94", 2024-03-12T05:57:19.015 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T05:57:19.015 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:57:19.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.005+0000 7f73599e9640 1 -- 172.21.15.76:0/347141575 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f733c0a1ba0 msgr2=0x7f733c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:19.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.005+0000 7f73599e9640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f733c0a1ba0 0x7f733c0a4060 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f734c005cc0 tx=0x7f734c032f30 comp rx=0 tx=0).stop 2024-03-12T05:57:19.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.005+0000 7f73599e9640 1 -- 172.21.15.76:0/347141575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7354100280 msgr2=0x7f73541985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:19.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.005+0000 7f73599e9640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7354100280 0x7f73541985c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f734400c890 tx=0x7f734400cd50 comp rx=0 tx=0).stop 2024-03-12T05:57:19.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.005+0000 7f73599e9640 1 -- 172.21.15.76:0/347141575 shutdown_connections 2024-03-12T05:57:19.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.005+0000 7f73599e9640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f733c0a1ba0 0x7f733c0a4060 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.005+0000 7f73599e9640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7354199090 0x7f73541a3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.005+0000 7f73599e9640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7354105350 0x7f7354198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.005+0000 7f73599e9640 1 --2- 172.21.15.76:0/347141575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7354100280 0x7f73541985c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.005+0000 7f73599e9640 1 -- 172.21.15.76:0/347141575 >> 172.21.15.76:0/347141575 conn(0x7f73540771a0 msgr2=0x7f73540fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:19.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.005+0000 7f73599e9640 1 -- 172.21.15.76:0/347141575 shutdown_connections 2024-03-12T05:57:19.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.005+0000 7f73599e9640 1 -- 172.21.15.76:0/347141575 wait complete. 2024-03-12T05:57:19.021 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:57:19.021 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:57:19.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.127+0000 7f486a00b640 1 -- 172.21.15.76:0/3671898335 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4864100260 msgr2=0x7f4864100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:19.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.127+0000 7f486a00b640 1 --2- 172.21.15.76:0/3671898335 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4864100260 0x7f4864100640 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4854009930 tx=0x7f485402bc30 comp rx=0 tx=0).stop 2024-03-12T05:57:19.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.127+0000 7f486a00b640 1 -- 172.21.15.76:0/3671898335 shutdown_connections 2024-03-12T05:57:19.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.127+0000 7f486a00b640 1 --2- 172.21.15.76:0/3671898335 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4864105330 0x7f486410cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.127+0000 7f486a00b640 1 --2- 172.21.15.76:0/3671898335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4864100c10 0x7f4864104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.127+0000 7f486a00b640 1 --2- 172.21.15.76:0/3671898335 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4864100260 0x7f4864100640 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.127+0000 7f486a00b640 1 -- 172.21.15.76:0/3671898335 >> 172.21.15.76:0/3671898335 conn(0x7f4864077180 msgr2=0x7f4864077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:19.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.127+0000 7f486a00b640 1 -- 172.21.15.76:0/3671898335 shutdown_connections 2024-03-12T05:57:19.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.127+0000 7f486a00b640 1 -- 172.21.15.76:0/3671898335 wait complete. 2024-03-12T05:57:19.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.127+0000 7f486a00b640 1 Processor -- start 2024-03-12T05:57:19.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.127+0000 7f486a00b640 1 -- start start 2024-03-12T05:57:19.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f486a00b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4864100260 0x7f486419d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f486a00b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4864105330 0x7f486419d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f486a00b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48641975d0 0x7f4864197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f486a00b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4864197fc0 con 0x7f4864100260 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f486a00b640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4864198130 con 0x7f48641975d0 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f486a00b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48641982f0 con 0x7f4864105330 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f48637fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4864100260 0x7f486419d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f48637fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4864100260 0x7f486419d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47964/0 (socket says 172.21.15.76:47964) 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f48637fe640 1 -- 172.21.15.76:0/4238854434 learned_addr learned my addr 172.21.15.76:0/4238854434 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f485bfff640 1 --2- 172.21.15.76:0/4238854434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4864105330 0x7f486419d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f4863fff640 1 --2- 172.21.15.76:0/4238854434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48641975d0 0x7f4864197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f48637fe640 1 -- 172.21.15.76:0/4238854434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4864105330 msgr2=0x7f486419d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f48637fe640 1 --2- 172.21.15.76:0/4238854434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4864105330 0x7f486419d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f48637fe640 1 -- 172.21.15.76:0/4238854434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48641975d0 msgr2=0x7f4864197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f48637fe640 1 --2- 172.21.15.76:0/4238854434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48641975d0 0x7f4864197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f48637fe640 1 -- 172.21.15.76:0/4238854434 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4854009560 con 0x7f4864100260 2024-03-12T05:57:19.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f48637fe640 1 --2- 172.21.15.76:0/4238854434 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4864100260 0x7f486419d420 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4854009230 tx=0x7f485402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:19.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f48617fa640 1 -- 172.21.15.76:0/4238854434 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4854004350 con 0x7f4864100260 2024-03-12T05:57:19.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f48617fa640 1 -- 172.21.15.76:0/4238854434 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48540044b0 con 0x7f4864100260 2024-03-12T05:57:19.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f486a00b640 1 -- 172.21.15.76:0/4238854434 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48641a42e0 con 0x7f4864100260 2024-03-12T05:57:19.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f48617fa640 1 -- 172.21.15.76:0/4238854434 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4854003d00 con 0x7f4864100260 2024-03-12T05:57:19.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.128+0000 7f486a00b640 1 -- 172.21.15.76:0/4238854434 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48641a47e0 con 0x7f4864100260 2024-03-12T05:57:19.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f48617fa640 1 -- 172.21.15.76:0/4238854434 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4854004df0 con 0x7f4864100260 2024-03-12T05:57:19.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f486cbd7740 1 Processor -- start 2024-03-12T05:57:19.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f486cbd7740 1 -- start start 2024-03-12T05:57:19.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f486cbd7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d4cfe0 0x55c409d4d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f486cbd7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 0x55c409d5a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f48617fa640 1 --2- 172.21.15.76:0/4238854434 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f48500a1c20 0x7f48500a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f486cbd7740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d5abc0 0x55c409d5cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f486cbd7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c409cbc280 con 0x55c409d4cfe0 2024-03-12T05:57:19.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f486cbd7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c409cb97b0 con 0x55c409d4d900 2024-03-12T05:57:19.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f486cbd7740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c409cb4f90 con 0x55c409d5abc0 2024-03-12T05:57:19.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f4863fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d4cfe0 0x55c409d4d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f4863fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d4cfe0 0x55c409d4d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47968/0 (socket says 172.21.15.76:47968) 2024-03-12T05:57:19.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f4863fff640 1 -- 172.21.15.76:0/977411926 learned_addr learned my addr 172.21.15.76:0/977411926 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:19.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f48637fe640 1 --2- 172.21.15.76:0/977411926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 0x55c409d5a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.131+0000 7f485bfff640 1 --2- 172.21.15.76:0/4238854434 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f48500a1c20 0x7f48500a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.132+0000 7f48617fa640 1 -- 172.21.15.76:0/4238854434 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f48540ef5d0 con 0x7f4864100260 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.132+0000 7f4863fff640 1 -- 172.21.15.76:0/977411926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d5abc0 msgr2=0x55c409d5cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.132+0000 7f4863fff640 1 --2- 172.21.15.76:0/977411926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d5abc0 0x55c409d5cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.132+0000 7f4863fff640 1 -- 172.21.15.76:0/977411926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 msgr2=0x55c409d5a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.132+0000 7f4863fff640 1 --2- 172.21.15.76:0/977411926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 0x55c409d5a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.132+0000 7f4863fff640 1 -- 172.21.15.76:0/977411926 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c409c73570 con 0x55c409d4cfe0 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.132+0000 7f4863fff640 1 --2- 172.21.15.76:0/977411926 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d4cfe0 0x55c409d4d3c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f485c005c40 tx=0x7f485c0139f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab44cf65b807a0b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.132+0000 7f485bfff640 1 --2- 172.21.15.76:0/4238854434 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f48500a1c20 0x7f48500a40e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f48640feb30 tx=0x7f484c005f90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.132+0000 7f4859ffb640 1 -- 172.21.15.76:0/977411926 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c003970 con 0x55c409d4cfe0 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.132+0000 7f4859ffb640 1 -- 172.21.15.76:0/977411926 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f485c004020 con 0x55c409d4cfe0 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.132+0000 7f4859ffb640 1 -- 172.21.15.76:0/977411926 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c010870 con 0x55c409d4cfe0 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.133+0000 7f486cbd7740 1 -- 172.21.15.76:0/977411926 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d4cfe0 msgr2=0x55c409d4d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.133+0000 7f486cbd7740 1 --2- 172.21.15.76:0/977411926 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d4cfe0 0x55c409d4d3c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f485c005c40 tx=0x7f485c0139f0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.133+0000 7f486cbd7740 1 -- 172.21.15.76:0/977411926 shutdown_connections 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.133+0000 7f486cbd7740 1 --2- 172.21.15.76:0/977411926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d5abc0 0x55c409d5cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.133+0000 7f486cbd7740 1 --2- 172.21.15.76:0/977411926 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 0x55c409d5a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.133+0000 7f486cbd7740 1 --2- 172.21.15.76:0/977411926 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d4cfe0 0x55c409d4d3c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.133+0000 7f486cbd7740 1 -- 172.21.15.76:0/977411926 >> 172.21.15.76:0/977411926 conn(0x55c409d47990 msgr2=0x55c409d47d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.133+0000 7f486cbd7740 1 -- 172.21.15.76:0/977411926 shutdown_connections 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.133+0000 7f486cbd7740 1 -- 172.21.15.76:0/977411926 wait complete. 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.133+0000 7f486cbd7740 1 -- start start 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.133+0000 7f486cbd7740 1 Processor -- start 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f486cbd7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4cfe0 0x55c409d5a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f486cbd7740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4d900 0x55c409d579b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f486cbd7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d580d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f486cbd7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c409988430 con 0x55c409d5abc0 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f486cbd7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c409944e00 con 0x55c409d4cfe0 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f486cbd7740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c409c39fd0 con 0x55c409d4d900 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f486cbd7740 10 client.0 _ll_register_callbacks cb 0x55c409c8ffd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f4863fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f48637fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4cfe0 0x55c409d5a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f4863fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d580d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47978/0 (socket says 172.21.15.76:47978) 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f4863fff640 1 -- 172.21.15.76:0/944095634 learned_addr learned my addr 172.21.15.76:0/944095634 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f48637fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4cfe0 0x55c409d5a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:50162/0 (socket says 172.21.15.76:50162) 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f485bfff640 1 --2- 172.21.15.76:0/944095634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4d900 0x55c409d579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f4863fff640 1 -- 172.21.15.76:0/944095634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4d900 msgr2=0x55c409d579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f4863fff640 1 --2- 172.21.15.76:0/944095634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4d900 0x55c409d579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f4863fff640 1 -- 172.21.15.76:0/944095634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4cfe0 msgr2=0x55c409d5a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f4863fff640 1 --2- 172.21.15.76:0/944095634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4cfe0 0x55c409d5a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f4863fff640 1 -- 172.21.15.76:0/944095634 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485c009d60 con 0x55c409d5abc0 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f4863fff640 1 --2- 172.21.15.76:0/944095634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d580d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f485c009270 tx=0x7f485c005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41854e5b27a2811c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f4842ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f4842ffd640 1 -- 172.21.15.76:0/944095634 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c0045a0 con 0x55c409d5abc0 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f4842ffd640 1 -- 172.21.15.76:0/944095634 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f485c010610 con 0x55c409d5abc0 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f486cbd7740 1 -- 172.21.15.76:0/944095634 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c4099b47e0 con 0x55c409d5abc0 2024-03-12T05:57:19.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.134+0000 7f4842ffd640 1 -- 172.21.15.76:0/944095634 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f485c010810 con 0x55c409d5abc0 2024-03-12T05:57:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.135+0000 7f486cbd7740 10 client.15975 fetch_fsmap learned FSMap version 22 2024-03-12T05:57:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.135+0000 7f486cbd7740 1 -- 172.21.15.76:0/944095634 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55c409cb3610 con 0x55c409d5abc0 2024-03-12T05:57:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.135+0000 7f4842ffd640 1 -- 172.21.15.76:0/944095634 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f485c031020 con 0x55c409d5abc0 2024-03-12T05:57:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.135+0000 7f4842ffd640 1 -- 172.21.15.76:0/944095634 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f485c02da00 con 0x55c409d5abc0 2024-03-12T05:57:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.135+0000 7f486cbd7740 10 client.15975 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:57:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.135+0000 7f486cbd7740 10 client.15975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.135+0000 7f486cbd7740 20 client.15975 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.135+0000 7f486cbd7740 1 --2- 172.21.15.76:0/944095634 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c409d6ae30 0x55c409d6d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.135+0000 7f486cbd7740 4 client.15975 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:57:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.135+0000 7f486cbd7740 1 -- 172.21.15.76:0/944095634 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c4099f1720 con 0x55c409d6ae30 2024-03-12T05:57:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.135+0000 7f48637fe640 1 --2- 172.21.15.76:0/944095634 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c409d6ae30 0x55c409d6d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.135+0000 7f48637fe640 1 --2- 172.21.15.76:0/944095634 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c409d6ae30 0x55c409d6d250 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db5eeb547bdfb56f server_cookie=fb2ea7ca7b5cf3d2 in_seq=0 out_seq=0 2024-03-12T05:57:19.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.135+0000 7f4842ffd640 10 client.15975 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:19.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f4842ffd640 1 -- 172.21.15.76:0/944095634 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c4099f1720 con 0x55c409d6ae30 2024-03-12T05:57:19.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f4842ffd640 10 client.15975 handle_command_reply: tid=0 2024-03-12T05:57:19.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f486cbd7740 1 client.15975 shutdown 2024-03-12T05:57:19.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f486cbd7740 1 -- 172.21.15.76:0/944095634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 msgr2=0x55c409d580d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:19.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f486cbd7740 1 --2- 172.21.15.76:0/944095634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d580d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f485c009270 tx=0x7f485c005a50 comp rx=0 tx=0).stop 2024-03-12T05:57:19.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f486cbd7740 1 -- 172.21.15.76:0/944095634 shutdown_connections 2024-03-12T05:57:19.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f486cbd7740 1 --2- 172.21.15.76:0/944095634 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c409d6ae30 0x55c409d6d250 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f486cbd7740 1 --2- 172.21.15.76:0/944095634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d580d0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f4842ffd640 0 client.15975 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:19.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f486cbd7740 1 --2- 172.21.15.76:0/944095634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4d900 0x55c409d579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f486cbd7740 1 --2- 172.21.15.76:0/944095634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4cfe0 0x55c409d5a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f486cbd7740 1 -- 172.21.15.76:0/944095634 >> 172.21.15.76:0/944095634 conn(0x55c409d47990 msgr2=0x55c409d49730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:19.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f486cbd7740 1 -- 172.21.15.76:0/944095634 shutdown_connections 2024-03-12T05:57:19.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f486cbd7740 1 -- 172.21.15.76:0/944095634 wait complete. 2024-03-12T05:57:19.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.165+0000 7f486cbd7740 20 client.15975 trim_cache size 0 max 16384 2024-03-12T05:57:19.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.179+0000 7f486cbd7740 1 Processor -- start 2024-03-12T05:57:19.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.179+0000 7f486cbd7740 1 -- start start 2024-03-12T05:57:19.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f486cbd7740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4cfe0 0x55c409d57950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f486cbd7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 0x55c409d5a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f486cbd7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d5a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f486cbd7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c409d671f0 con 0x55c409d5abc0 2024-03-12T05:57:19.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f486cbd7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c409d4cc10 con 0x55c409d4d900 2024-03-12T05:57:19.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f486cbd7740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c409d432b0 con 0x55c409d4cfe0 2024-03-12T05:57:19.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f4863fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d5a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f4863fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d5a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47980/0 (socket says 172.21.15.76:47980) 2024-03-12T05:57:19.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f4863fff640 1 -- 172.21.15.76:0/2529439898 learned_addr learned my addr 172.21.15.76:0/2529439898 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:19.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f485bfff640 1 --2- 172.21.15.76:0/2529439898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 0x55c409d5a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f48637fe640 1 --2- 172.21.15.76:0/2529439898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4cfe0 0x55c409d57950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f4863fff640 1 -- 172.21.15.76:0/2529439898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4cfe0 msgr2=0x55c409d57950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:19.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f4863fff640 1 --2- 172.21.15.76:0/2529439898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4cfe0 0x55c409d57950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f4863fff640 1 -- 172.21.15.76:0/2529439898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 msgr2=0x55c409d5a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:19.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f4863fff640 1 --2- 172.21.15.76:0/2529439898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 0x55c409d5a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f4863fff640 1 -- 172.21.15.76:0/2529439898 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c409c4fac0 con 0x55c409d5abc0 2024-03-12T05:57:19.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f4863fff640 1 --2- 172.21.15.76:0/2529439898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d5a620 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f485c005d70 tx=0x7f485c005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b0d381015bf471 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:19.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.180+0000 7f48597fa640 1 -- 172.21.15.76:0/2529439898 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c009be0 con 0x55c409d5abc0 2024-03-12T05:57:19.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.181+0000 7f48597fa640 1 -- 172.21.15.76:0/2529439898 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f485c031070 con 0x55c409d5abc0 2024-03-12T05:57:19.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.181+0000 7f48597fa640 1 -- 172.21.15.76:0/2529439898 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c02daf0 con 0x55c409d5abc0 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.181+0000 7f486cbd7740 1 -- 172.21.15.76:0/2529439898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 msgr2=0x55c409d5a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.181+0000 7f486cbd7740 1 --2- 172.21.15.76:0/2529439898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d5a620 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f485c005d70 tx=0x7f485c005f70 comp rx=0 tx=0).stop 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.181+0000 7f486cbd7740 1 -- 172.21.15.76:0/2529439898 shutdown_connections 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.181+0000 7f486cbd7740 1 --2- 172.21.15.76:0/2529439898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d5a620 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.181+0000 7f486cbd7740 1 --2- 172.21.15.76:0/2529439898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 0x55c409d5a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.181+0000 7f486cbd7740 1 --2- 172.21.15.76:0/2529439898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4cfe0 0x55c409d57950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.181+0000 7f486cbd7740 1 -- 172.21.15.76:0/2529439898 >> 172.21.15.76:0/2529439898 conn(0x55c409d47990 msgr2=0x55c409d57240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.181+0000 7f486cbd7740 1 -- 172.21.15.76:0/2529439898 shutdown_connections 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.181+0000 7f486cbd7740 1 -- 172.21.15.76:0/2529439898 wait complete. 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.181+0000 7f486cbd7740 1 -- start start 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f486cbd7740 1 Processor -- start 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f486cbd7740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4cfe0 0x55c409d5e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f486cbd7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 0x55c409d5e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f486cbd7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d5eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f486cbd7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c409dc1330 con 0x55c409d5abc0 2024-03-12T05:57:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f486cbd7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c409dc1470 con 0x55c409d4d900 2024-03-12T05:57:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f486cbd7740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c409dc15b0 con 0x55c409d4cfe0 2024-03-12T05:57:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f486cbd7740 10 client.0 _ll_register_callbacks cb 0x7f485c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f4863fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d5eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f4863fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d5eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47996/0 (socket says 172.21.15.76:47996) 2024-03-12T05:57:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f4863fff640 1 -- 172.21.15.76:0/2574484529 learned_addr learned my addr 172.21.15.76:0/2574484529 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f485bfff640 1 --2- 172.21.15.76:0/2574484529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 0x55c409d5e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f48637fe640 1 --2- 172.21.15.76:0/2574484529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4cfe0 0x55c409d5e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f4863fff640 1 -- 172.21.15.76:0/2574484529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4cfe0 msgr2=0x55c409d5e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f4863fff640 1 --2- 172.21.15.76:0/2574484529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4cfe0 0x55c409d5e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f4863fff640 1 -- 172.21.15.76:0/2574484529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 msgr2=0x55c409d5e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f4863fff640 1 --2- 172.21.15.76:0/2574484529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 0x55c409d5e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f4863fff640 1 -- 172.21.15.76:0/2574484529 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c409cb3610 con 0x55c409d5abc0 2024-03-12T05:57:19.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f4863fff640 1 --2- 172.21.15.76:0/2574484529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d5eed0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f485c013a30 tx=0x7f485c02d5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ccf0e8f6874494 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.182+0000 7f4842ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f4842ffd640 1 -- 172.21.15.76:0/2574484529 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f485c009be0 con 0x55c409d5abc0 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f4842ffd640 1 -- 172.21.15.76:0/2574484529 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f485c032080 con 0x55c409d5abc0 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f486cbd7740 1 -- 172.21.15.76:0/2574484529 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f485c004020 con 0x55c409d5abc0 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f4842ffd640 1 -- 172.21.15.76:0/2574484529 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f485c005410 con 0x55c409d5abc0 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f486cbd7740 10 client.15981 fetch_fsmap learned FSMap version 22 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f486cbd7740 1 -- 172.21.15.76:0/2574484529 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55c409c756a0 con 0x55c409d5abc0 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f4842ffd640 1 -- 172.21.15.76:0/2574484529 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f485c004020 con 0x55c409d5abc0 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f4842ffd640 1 -- 172.21.15.76:0/2574484529 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f485c024b80 con 0x55c409d5abc0 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f486cbd7740 10 client.15981 fetch_fsmap finished waiting for FSMap version 22 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f486cbd7740 10 client.15981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f486cbd7740 20 client.15981 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f486cbd7740 1 --2- 172.21.15.76:0/2574484529 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c409dc2990 0x55c409dc4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f486cbd7740 4 client.15981 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:57:19.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f486cbd7740 1 -- 172.21.15.76:0/2574484529 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c409a40890 con 0x55c409dc2990 2024-03-12T05:57:19.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.183+0000 7f48637fe640 1 --2- 172.21.15.76:0/2574484529 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c409dc2990 0x55c409dc4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:19.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.184+0000 7f4842ffd640 10 client.15981 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:19.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.184+0000 7f48637fe640 1 --2- 172.21.15.76:0/2574484529 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c409dc2990 0x55c409dc4db0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a41b87b18667c355 server_cookie=f6c76d0a6275b3ce in_seq=0 out_seq=0 2024-03-12T05:57:19.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.185+0000 7f4842ffd640 1 -- 172.21.15.76:0/2574484529 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55c409a40890 con 0x55c409dc2990 2024-03-12T05:57:19.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.185+0000 7f4842ffd640 10 client.15981 handle_command_reply: tid=0 2024-03-12T05:57:19.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.185+0000 7f486cbd7740 1 client.15981 shutdown 2024-03-12T05:57:19.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.185+0000 7f486cbd7740 1 -- 172.21.15.76:0/2574484529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 msgr2=0x55c409d5eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:19.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.185+0000 7f486cbd7740 1 --2- 172.21.15.76:0/2574484529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d5eed0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f485c013a30 tx=0x7f485c02d5b0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.185+0000 7f486cbd7740 1 -- 172.21.15.76:0/2574484529 shutdown_connections 2024-03-12T05:57:19.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.185+0000 7f486cbd7740 1 --2- 172.21.15.76:0/2574484529 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c409dc2990 0x55c409dc4db0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.185+0000 7f486cbd7740 1 --2- 172.21.15.76:0/2574484529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c409d5abc0 0x55c409d5eed0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.185+0000 7f486cbd7740 1 --2- 172.21.15.76:0/2574484529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c409d4d900 0x55c409d5e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.185+0000 7f4842ffd640 0 client.15981 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:19.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.185+0000 7f486cbd7740 1 --2- 172.21.15.76:0/2574484529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c409d4cfe0 0x55c409d5e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.185+0000 7f486cbd7740 1 -- 172.21.15.76:0/2574484529 >> 172.21.15.76:0/2574484529 conn(0x55c409d47990 msgr2=0x55c409d56280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:19.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486cbd7740 1 -- 172.21.15.76:0/2574484529 shutdown_connections 2024-03-12T05:57:19.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486cbd7740 1 -- 172.21.15.76:0/2574484529 wait complete. 2024-03-12T05:57:19.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486cbd7740 20 client.15981 trim_cache size 0 max 16384 2024-03-12T05:57:19.196 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:57:19.196 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "scrub active (4065 inodes in the stack)", 2024-03-12T05:57:19.196 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T05:57:19.196 INFO:teuthology.orchestra.run.smithi076.stdout: "aacf2548-be29-4634-9c90-6f8c4bc73d94": { 2024-03-12T05:57:19.197 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T05:57:19.197 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "aacf2548-be29-4634-9c90-6f8c4bc73d94", 2024-03-12T05:57:19.197 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T05:57:19.197 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:57:19.197 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:57:19.197 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:57:19.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486a00b640 1 -- 172.21.15.76:0/4238854434 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f48500a1c20 msgr2=0x7f48500a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:19.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486a00b640 1 --2- 172.21.15.76:0/4238854434 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f48500a1c20 0x7f48500a40e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f48640feb30 tx=0x7f484c005f90 comp rx=0 tx=0).stop 2024-03-12T05:57:19.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486a00b640 1 -- 172.21.15.76:0/4238854434 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4864100260 msgr2=0x7f486419d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:19.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486a00b640 1 --2- 172.21.15.76:0/4238854434 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4864100260 0x7f486419d420 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4854009230 tx=0x7f485402c650 comp rx=0 tx=0).stop 2024-03-12T05:57:19.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486a00b640 1 -- 172.21.15.76:0/4238854434 shutdown_connections 2024-03-12T05:57:19.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486a00b640 1 --2- 172.21.15.76:0/4238854434 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f48500a1c20 0x7f48500a40e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486a00b640 1 --2- 172.21.15.76:0/4238854434 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f48641975d0 0x7f4864197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486a00b640 1 --2- 172.21.15.76:0/4238854434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4864105330 0x7f486419d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486a00b640 1 --2- 172.21.15.76:0/4238854434 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4864100260 0x7f486419d420 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:19.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486a00b640 1 -- 172.21.15.76:0/4238854434 >> 172.21.15.76:0/4238854434 conn(0x7f4864077180 msgr2=0x7f4864106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:19.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486a00b640 1 -- 172.21.15.76:0/4238854434 shutdown_connections 2024-03-12T05:57:19.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:19.186+0000 7f486a00b640 1 -- 172.21.15.76:0/4238854434 wait complete. 2024-03-12T05:57:19.201 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:19 smithi076 ceph-mon[32095]: mgrmap e26: x(active, since 91s), standbys: z, y 2024-03-12T05:57:19.202 INFO:tasks.cephfs.filesystem:scrub status for tag:aacf2548-be29-4634-9c90-6f8c4bc73d94 - {'path': '/', 'tag': 'aacf2548-be29-4634-9c90-6f8c4bc73d94', 'options': 'recursive,force'} 2024-03-12T05:57:19.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:19 smithi082 ceph-mon[36382]: mgrmap e26: x(active, since 91s), standbys: z, y 2024-03-12T05:57:19.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:19 smithi088 ceph-mon[36585]: mgrmap e26: x(active, since 91s), standbys: z, y 2024-03-12T05:57:20.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:20 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T05:57:20.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:20 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:20.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:20 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:20.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:20 smithi082 ceph-mon[36382]: pgmap v49: 97 pgs: 97 active+clean; 22 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 725 KiB/s wr, 32 op/s 2024-03-12T05:57:20.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:20 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T05:57:20.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:20 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:20.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:20 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:20.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:20 smithi088 ceph-mon[36585]: pgmap v49: 97 pgs: 97 active+clean; 22 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 725 KiB/s wr, 32 op/s 2024-03-12T05:57:20.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:20 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T05:57:20.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:20 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:20.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:20 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:20.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:20 smithi076 ceph-mon[32095]: pgmap v49: 97 pgs: 97 active+clean; 22 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 725 KiB/s wr, 32 op/s 2024-03-12T05:57:21.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:21 smithi082 ceph-mon[36382]: pgmap v50: 97 pgs: 97 active+clean; 22 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 479 KiB/s wr, 24 op/s 2024-03-12T05:57:21.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:21 smithi088 ceph-mon[36585]: pgmap v50: 97 pgs: 97 active+clean; 22 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 479 KiB/s wr, 24 op/s 2024-03-12T05:57:21.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:21 smithi076 ceph-mon[32095]: pgmap v50: 97 pgs: 97 active+clean; 22 MiB data, 434 MiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 479 KiB/s wr, 24 op/s 2024-03-12T05:57:23.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:23 smithi082 ceph-mon[36382]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:57:23.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:23 smithi082 ceph-mon[36382]: pgmap v51: 97 pgs: 97 active+clean; 43 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 404 op/s 2024-03-12T05:57:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:23 smithi088 ceph-mon[36585]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:57:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:23 smithi088 ceph-mon[36585]: pgmap v51: 97 pgs: 97 active+clean; 43 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 404 op/s 2024-03-12T05:57:23.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:23 smithi076 ceph-mon[32095]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:57:23.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:23 smithi076 ceph-mon[32095]: pgmap v51: 97 pgs: 97 active+clean; 43 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 404 op/s 2024-03-12T05:57:26.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: pgmap v52: 97 pgs: 97 active+clean; 43 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 385 op/s 2024-03-12T05:57:26.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:26.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:26 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: pgmap v52: 97 pgs: 97 active+clean; 43 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 385 op/s 2024-03-12T05:57:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:26.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:26 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: pgmap v52: 97 pgs: 97 active+clean; 43 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 385 op/s 2024-03-12T05:57:26.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:26.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:26.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:26 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:27.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:27 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:27.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:27 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:27.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:27 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:28.346 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:28 smithi082 ceph-mon[36382]: pgmap v53: 97 pgs: 97 active+clean; 43 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 385 op/s 2024-03-12T05:57:28.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:28 smithi088 ceph-mon[36585]: pgmap v53: 97 pgs: 97 active+clean; 43 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 385 op/s 2024-03-12T05:57:28.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:28 smithi076 ceph-mon[32095]: pgmap v53: 97 pgs: 97 active+clean; 43 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 385 op/s 2024-03-12T05:57:29.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:29 smithi082 ceph-mon[36382]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:57:29.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:29 smithi088 ceph-mon[36585]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:57:29.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:29 smithi076 ceph-mon[32095]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:57:30.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:30 smithi082 ceph-mon[36382]: pgmap v54: 97 pgs: 97 active+clean; 52 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 513 op/s 2024-03-12T05:57:30.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:30 smithi088 ceph-mon[36585]: pgmap v54: 97 pgs: 97 active+clean; 52 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 513 op/s 2024-03-12T05:57:30.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:30 smithi076 ceph-mon[32095]: pgmap v54: 97 pgs: 97 active+clean; 52 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.2 MiB/s wr, 513 op/s 2024-03-12T05:57:32.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:32 smithi082 ceph-mon[36382]: pgmap v55: 97 pgs: 97 active+clean; 52 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 507 op/s 2024-03-12T05:57:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:32 smithi088 ceph-mon[36585]: pgmap v55: 97 pgs: 97 active+clean; 52 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 507 op/s 2024-03-12T05:57:32.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:32 smithi076 ceph-mon[32095]: pgmap v55: 97 pgs: 97 active+clean; 52 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 507 op/s 2024-03-12T05:57:33.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:33 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:57:33.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:33 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:57:33.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:33 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:57:33.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:33 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:57:33.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:33 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:57:33.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:33 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:57:34.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:34 smithi082 ceph-mon[36382]: pgmap v56: 97 pgs: 97 active+clean; 55 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 575 op/s 2024-03-12T05:57:34.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:34 smithi088 ceph-mon[36585]: pgmap v56: 97 pgs: 97 active+clean; 55 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 575 op/s 2024-03-12T05:57:34.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:34 smithi076 ceph-mon[32095]: pgmap v56: 97 pgs: 97 active+clean; 55 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 575 op/s 2024-03-12T05:57:35.734 INFO:tasks.workunit.client.0.smithi076.stderr:Note: switching to '12973fc04fd10d4af086901e10ffa8e48866b735'. 2024-03-12T05:57:35.734 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-12T05:57:35.734 INFO:tasks.workunit.client.0.smithi076.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-12T05:57:35.734 INFO:tasks.workunit.client.0.smithi076.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-12T05:57:35.734 INFO:tasks.workunit.client.0.smithi076.stderr:state without impacting any branches by switching back to a branch. 2024-03-12T05:57:35.734 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-12T05:57:35.734 INFO:tasks.workunit.client.0.smithi076.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-12T05:57:35.735 INFO:tasks.workunit.client.0.smithi076.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-12T05:57:35.735 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-12T05:57:35.735 INFO:tasks.workunit.client.0.smithi076.stderr: git switch -c 2024-03-12T05:57:35.735 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-12T05:57:35.735 INFO:tasks.workunit.client.0.smithi076.stderr:Or undo this operation with: 2024-03-12T05:57:35.735 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-12T05:57:35.735 INFO:tasks.workunit.client.0.smithi076.stderr: git switch - 2024-03-12T05:57:35.735 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-12T05:57:35.735 INFO:tasks.workunit.client.0.smithi076.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-12T05:57:35.735 INFO:tasks.workunit.client.0.smithi076.stderr: 2024-03-12T05:57:35.735 INFO:tasks.workunit.client.0.smithi076.stderr:HEAD is now at 12973fc0 btrfs/022: Correct seq number in golden output 2024-03-12T05:57:35.736 INFO:tasks.workunit.client.0.smithi076.stderr:+ make -j4 2024-03-12T05:57:35.744 INFO:tasks.workunit.client.0.smithi076.stdout:libtoolize -cfi 2024-03-12T05:57:36.158 INFO:tasks.workunit.client.0.smithi076.stdout:libtoolize: putting auxiliary files in '.'. 2024-03-12T05:57:36.158 INFO:tasks.workunit.client.0.smithi076.stdout:libtoolize: copying file './config.guess' 2024-03-12T05:57:36.162 INFO:tasks.workunit.client.0.smithi076.stdout:libtoolize: copying file './config.sub' 2024-03-12T05:57:36.167 INFO:tasks.workunit.client.0.smithi076.stdout:libtoolize: copying file './install-sh' 2024-03-12T05:57:36.172 INFO:tasks.workunit.client.0.smithi076.stdout:libtoolize: copying file './ltmain.sh' 2024-03-12T05:57:36.195 INFO:tasks.workunit.client.0.smithi076.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-03-12T05:57:36.195 INFO:tasks.workunit.client.0.smithi076.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-03-12T05:57:36.201 INFO:tasks.workunit.client.0.smithi076.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-03-12T05:57:36.207 INFO:tasks.workunit.client.0.smithi076.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-03-12T05:57:36.213 INFO:tasks.workunit.client.0.smithi076.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-03-12T05:57:36.219 INFO:tasks.workunit.client.0.smithi076.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-03-12T05:57:36.225 INFO:tasks.workunit.client.0.smithi076.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-03-12T05:57:36.227 INFO:tasks.workunit.client.0.smithi076.stdout:cp include/install-sh . 2024-03-12T05:57:36.238 INFO:tasks.workunit.client.0.smithi076.stdout:aclocal -I m4 2024-03-12T05:57:36.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:36 smithi082 ceph-mon[36382]: pgmap v57: 97 pgs: 97 active+clean; 55 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 196 op/s 2024-03-12T05:57:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:36 smithi088 ceph-mon[36585]: pgmap v57: 97 pgs: 97 active+clean; 55 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 196 op/s 2024-03-12T05:57:36.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:36 smithi076 ceph-mon[32095]: pgmap v57: 97 pgs: 97 active+clean; 55 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 196 op/s 2024-03-12T05:57:37.319 INFO:tasks.workunit.client.0.smithi076.stdout:autoheader 2024-03-12T05:57:37.620 INFO:tasks.workunit.client.0.smithi076.stdout:autoconf 2024-03-12T05:57:37.794 INFO:tasks.workunit.client.0.smithi076.stdout:./configure \ 2024-03-12T05:57:37.794 INFO:tasks.workunit.client.0.smithi076.stdout: --libexecdir=/usr/lib \ 2024-03-12T05:57:37.795 INFO:tasks.workunit.client.0.smithi076.stdout: --exec_prefix=/var/lib 2024-03-12T05:57:37.927 INFO:tasks.workunit.client.0.smithi076.stdout:checking build system type... x86_64-pc-linux-gnu 2024-03-12T05:57:37.927 INFO:tasks.workunit.client.0.smithi076.stdout:checking host system type... x86_64-pc-linux-gnu 2024-03-12T05:57:37.928 INFO:tasks.workunit.client.0.smithi076.stdout:checking how to print strings... printf 2024-03-12T05:57:37.928 INFO:tasks.workunit.client.0.smithi076.stdout:checking for gcc... gcc 2024-03-12T05:57:38.009 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether the C compiler works... yes 2024-03-12T05:57:38.009 INFO:tasks.workunit.client.0.smithi076.stdout:checking for C compiler default output file name... a.out 2024-03-12T05:57:38.048 INFO:tasks.workunit.client.0.smithi076.stdout:checking for suffix of executables... 2024-03-12T05:57:38.085 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether we are cross compiling... no 2024-03-12T05:57:38.121 INFO:tasks.workunit.client.0.smithi076.stdout:checking for suffix of object files... o 2024-03-12T05:57:38.144 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether we are using the GNU C compiler... yes 2024-03-12T05:57:38.169 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether gcc accepts -g... yes 2024-03-12T05:57:38.241 INFO:tasks.workunit.client.0.smithi076.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-12T05:57:38.245 INFO:tasks.workunit.client.0.smithi076.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-03-12T05:57:38.248 INFO:tasks.workunit.client.0.smithi076.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-12T05:57:38.250 INFO:tasks.workunit.client.0.smithi076.stdout:checking for egrep... /usr/bin/grep -E 2024-03-12T05:57:38.251 INFO:tasks.workunit.client.0.smithi076.stdout:checking for fgrep... /usr/bin/grep -F 2024-03-12T05:57:38.254 INFO:tasks.workunit.client.0.smithi076.stdout:checking for ld used by gcc... /usr/bin/ld 2024-03-12T05:57:38.255 INFO:tasks.workunit.client.0.smithi076.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-03-12T05:57:38.270 INFO:tasks.workunit.client.0.smithi076.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-03-12T05:57:38.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:38 smithi082 ceph-mon[36382]: pgmap v58: 97 pgs: 97 active+clean; 55 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 196 op/s 2024-03-12T05:57:38.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:38 smithi088 ceph-mon[36585]: pgmap v58: 97 pgs: 97 active+clean; 55 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 196 op/s 2024-03-12T05:57:38.503 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:38 smithi076 ceph-mon[32095]: pgmap v58: 97 pgs: 97 active+clean; 55 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 196 op/s 2024-03-12T05:57:39.195 INFO:tasks.workunit.client.0.smithi076.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-03-12T05:57:39.195 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether ln -s works... yes 2024-03-12T05:57:39.206 INFO:tasks.workunit.client.0.smithi076.stdout:checking the maximum length of command line arguments... 1572864 2024-03-12T05:57:39.209 INFO:tasks.workunit.client.0.smithi076.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-03-12T05:57:39.209 INFO:tasks.workunit.client.0.smithi076.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-03-12T05:57:39.209 INFO:tasks.workunit.client.0.smithi076.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-03-12T05:57:39.209 INFO:tasks.workunit.client.0.smithi076.stdout:checking for objdump... objdump 2024-03-12T05:57:39.210 INFO:tasks.workunit.client.0.smithi076.stdout:checking how to recognize dependent libraries... pass_all 2024-03-12T05:57:39.211 INFO:tasks.workunit.client.0.smithi076.stdout:checking for dlltool... no 2024-03-12T05:57:39.211 INFO:tasks.workunit.client.0.smithi076.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-03-12T05:57:39.211 INFO:tasks.workunit.client.0.smithi076.stdout:checking for ar... ar 2024-03-12T05:57:39.292 INFO:tasks.workunit.client.0.smithi076.stdout:checking for archiver @FILE support... @ 2024-03-12T05:57:39.292 INFO:tasks.workunit.client.0.smithi076.stdout:checking for strip... strip 2024-03-12T05:57:39.292 INFO:tasks.workunit.client.0.smithi076.stdout:checking for ranlib... ranlib 2024-03-12T05:57:39.292 INFO:tasks.workunit.client.0.smithi076.stdout:checking for gawk... gawk 2024-03-12T05:57:39.398 INFO:tasks.workunit.client.0.smithi076.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-03-12T05:57:39.399 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sysroot... no 2024-03-12T05:57:39.411 INFO:tasks.workunit.client.0.smithi076.stdout:checking for a working dd... /usr/bin/dd 2024-03-12T05:57:39.425 INFO:tasks.workunit.client.0.smithi076.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-03-12T05:57:39.459 INFO:tasks.workunit.client.0.smithi076.stdout:checking for mt... no 2024-03-12T05:57:39.472 INFO:tasks.workunit.client.0.smithi076.stdout:checking if : is a manifest tool... no 2024-03-12T05:57:39.554 INFO:tasks.workunit.client.0.smithi076.stdout:checking how to run the C preprocessor... gcc -E 2024-03-12T05:57:39.753 INFO:tasks.workunit.client.0.smithi076.stdout:checking for ANSI C header files... yes 2024-03-12T05:57:39.788 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/types.h... yes 2024-03-12T05:57:39.830 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/stat.h... yes 2024-03-12T05:57:39.871 INFO:tasks.workunit.client.0.smithi076.stdout:checking for stdlib.h... yes 2024-03-12T05:57:39.910 INFO:tasks.workunit.client.0.smithi076.stdout:checking for string.h... yes 2024-03-12T05:57:39.958 INFO:tasks.workunit.client.0.smithi076.stdout:checking for memory.h... yes 2024-03-12T05:57:39.996 INFO:tasks.workunit.client.0.smithi076.stdout:checking for strings.h... yes 2024-03-12T05:57:40.035 INFO:tasks.workunit.client.0.smithi076.stdout:checking for inttypes.h... yes 2024-03-12T05:57:40.075 INFO:tasks.workunit.client.0.smithi076.stdout:checking for stdint.h... yes 2024-03-12T05:57:40.116 INFO:tasks.workunit.client.0.smithi076.stdout:checking for unistd.h... yes 2024-03-12T05:57:40.161 INFO:tasks.workunit.client.0.smithi076.stdout:checking for dlfcn.h... yes 2024-03-12T05:57:40.167 INFO:tasks.workunit.client.0.smithi076.stdout:checking for objdir... .libs 2024-03-12T05:57:40.283 INFO:tasks.workunit.client.0.smithi076.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-03-12T05:57:40.283 INFO:tasks.workunit.client.0.smithi076.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-03-12T05:57:40.317 INFO:tasks.workunit.client.0.smithi076.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-03-12T05:57:40.361 INFO:tasks.workunit.client.0.smithi076.stdout:checking if gcc static flag -static works... no 2024-03-12T05:57:40.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:40 smithi082 ceph-mon[36382]: pgmap v59: 97 pgs: 97 active+clean; 59 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 265 op/s 2024-03-12T05:57:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:40 smithi088 ceph-mon[36585]: pgmap v59: 97 pgs: 97 active+clean; 59 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 265 op/s 2024-03-12T05:57:40.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:40 smithi076 ceph-mon[32095]: pgmap v59: 97 pgs: 97 active+clean; 59 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 265 op/s 2024-03-12T05:57:40.615 INFO:tasks.workunit.client.0.smithi076.stdout:checking if gcc supports -c -o file.o... yes 2024-03-12T05:57:40.615 INFO:tasks.workunit.client.0.smithi076.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-03-12T05:57:40.625 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-03-12T05:57:41.247 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether -lc should be explicitly linked in... no 2024-03-12T05:57:41.311 INFO:tasks.workunit.client.0.smithi076.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-03-12T05:57:41.311 INFO:tasks.workunit.client.0.smithi076.stdout:checking how to hardcode library paths into programs... immediate 2024-03-12T05:57:41.313 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether stripping libraries is possible... yes 2024-03-12T05:57:41.313 INFO:tasks.workunit.client.0.smithi076.stdout:checking if libtool supports shared libraries... yes 2024-03-12T05:57:41.313 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether to build shared libraries... yes 2024-03-12T05:57:41.313 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether to build static libraries... yes 2024-03-12T05:57:41.319 INFO:tasks.workunit.client.0.smithi076.stdout:checking for gcc... (cached) gcc 2024-03-12T05:57:41.355 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-12T05:57:41.356 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-12T05:57:41.356 INFO:tasks.workunit.client.0.smithi076.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-12T05:57:41.356 INFO:tasks.workunit.client.0.smithi076.stdout:checking for gmake... /usr/bin/gmake 2024-03-12T05:57:41.356 INFO:tasks.workunit.client.0.smithi076.stderr:./configure: line 11968: test: $(SHELL): binary operator expected 2024-03-12T05:57:41.357 INFO:tasks.workunit.client.0.smithi076.stdout:checking for tar... /bin/tar 2024-03-12T05:57:41.357 INFO:tasks.workunit.client.0.smithi076.stdout:checking for gzip... /bin/gzip 2024-03-12T05:57:41.380 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether gcc -MM is supported... yes 2024-03-12T05:57:41.380 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sort... /bin/sort 2024-03-12T05:57:41.380 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether ln -s works... yes 2024-03-12T05:57:41.380 INFO:tasks.workunit.client.0.smithi076.stdout:checking for rpm... /bin/rpm 2024-03-12T05:57:41.384 INFO:tasks.workunit.client.0.smithi076.stdout:checking for rpmbuild... no 2024-03-12T05:57:41.384 INFO:tasks.workunit.client.0.smithi076.stdout:checking for ANSI C header files... (cached) yes 2024-03-12T05:57:41.435 INFO:tasks.workunit.client.0.smithi076.stdout:checking assert.h usability... yes 2024-03-12T05:57:41.450 INFO:tasks.workunit.client.0.smithi076.stdout:checking assert.h presence... yes 2024-03-12T05:57:41.450 INFO:tasks.workunit.client.0.smithi076.stdout:checking for assert.h... yes 2024-03-12T05:57:41.498 INFO:tasks.workunit.client.0.smithi076.stdout:checking bstring.h usability... no 2024-03-12T05:57:41.524 INFO:tasks.workunit.client.0.smithi076.stdout:checking bstring.h presence... no 2024-03-12T05:57:41.524 INFO:tasks.workunit.client.0.smithi076.stdout:checking for bstring.h... no 2024-03-12T05:57:41.561 INFO:tasks.workunit.client.0.smithi076.stdout:checking libgen.h usability... yes 2024-03-12T05:57:41.577 INFO:tasks.workunit.client.0.smithi076.stdout:checking libgen.h presence... yes 2024-03-12T05:57:41.577 INFO:tasks.workunit.client.0.smithi076.stdout:checking for libgen.h... yes 2024-03-12T05:57:41.626 INFO:tasks.workunit.client.0.smithi076.stdout:checking dirent.h usability... yes 2024-03-12T05:57:41.642 INFO:tasks.workunit.client.0.smithi076.stdout:checking dirent.h presence... yes 2024-03-12T05:57:41.642 INFO:tasks.workunit.client.0.smithi076.stdout:checking for dirent.h... yes 2024-03-12T05:57:41.686 INFO:tasks.workunit.client.0.smithi076.stdout:checking errno.h usability... yes 2024-03-12T05:57:41.702 INFO:tasks.workunit.client.0.smithi076.stdout:checking errno.h presence... yes 2024-03-12T05:57:41.702 INFO:tasks.workunit.client.0.smithi076.stdout:checking for errno.h... yes 2024-03-12T05:57:41.755 INFO:tasks.workunit.client.0.smithi076.stdout:checking malloc.h usability... yes 2024-03-12T05:57:41.772 INFO:tasks.workunit.client.0.smithi076.stdout:checking malloc.h presence... yes 2024-03-12T05:57:41.772 INFO:tasks.workunit.client.0.smithi076.stdout:checking for malloc.h... yes 2024-03-12T05:57:41.823 INFO:tasks.workunit.client.0.smithi076.stdout:checking uuid.h usability... no 2024-03-12T05:57:41.850 INFO:tasks.workunit.client.0.smithi076.stdout:checking uuid.h presence... no 2024-03-12T05:57:41.851 INFO:tasks.workunit.client.0.smithi076.stdout:checking for uuid.h... no 2024-03-12T05:57:41.895 INFO:tasks.workunit.client.0.smithi076.stdout:checking uuid/uuid.h usability... yes 2024-03-12T05:57:41.912 INFO:tasks.workunit.client.0.smithi076.stdout:checking uuid/uuid.h presence... yes 2024-03-12T05:57:41.912 INFO:tasks.workunit.client.0.smithi076.stdout:checking for uuid/uuid.h... yes 2024-03-12T05:57:41.961 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/uuid.h usability... no 2024-03-12T05:57:41.987 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/uuid.h presence... no 2024-03-12T05:57:41.988 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/uuid.h... no 2024-03-12T05:57:42.026 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/file.h usability... yes 2024-03-12T05:57:42.042 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/file.h presence... yes 2024-03-12T05:57:42.042 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/file.h... yes 2024-03-12T05:57:42.091 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/fcntl.h usability... yes 2024-03-12T05:57:42.108 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/fcntl.h presence... yes 2024-03-12T05:57:42.108 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/fcntl.h... yes 2024-03-12T05:57:42.159 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/syssgi.h usability... no 2024-03-12T05:57:42.189 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/syssgi.h presence... no 2024-03-12T05:57:42.190 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/syssgi.h... no 2024-03-12T05:57:42.301 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/param.h usability... yes 2024-03-12T05:57:42.321 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/param.h presence... yes 2024-03-12T05:57:42.321 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/param.h... yes 2024-03-12T05:57:42.338 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/stat.h... (cached) yes 2024-03-12T05:57:42.377 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/statvfs.h usability... yes 2024-03-12T05:57:42.393 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/statvfs.h presence... yes 2024-03-12T05:57:42.394 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/statvfs.h... yes 2024-03-12T05:57:42.438 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/time.h usability... yes 2024-03-12T05:57:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:42 smithi082 ceph-mon[36382]: pgmap v60: 97 pgs: 97 active+clean; 59 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 840 KiB/s wr, 137 op/s 2024-03-12T05:57:42.455 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/time.h presence... yes 2024-03-12T05:57:42.455 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/time.h... yes 2024-03-12T05:57:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:42 smithi088 ceph-mon[36585]: pgmap v60: 97 pgs: 97 active+clean; 59 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 840 KiB/s wr, 137 op/s 2024-03-12T05:57:42.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:42 smithi076 ceph-mon[32095]: pgmap v60: 97 pgs: 97 active+clean; 59 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 840 KiB/s wr, 137 op/s 2024-03-12T05:57:42.499 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/ioctl.h usability... yes 2024-03-12T05:57:42.515 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/ioctl.h presence... yes 2024-03-12T05:57:42.515 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/ioctl.h... yes 2024-03-12T05:57:42.558 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/wait.h usability... yes 2024-03-12T05:57:42.575 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/wait.h presence... yes 2024-03-12T05:57:42.575 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/wait.h... yes 2024-03-12T05:57:42.593 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/types.h... (cached) yes 2024-03-12T05:57:42.597 INFO:tasks.workunit.client.0.smithi076.stdout:checking for strings.h... (cached) yes 2024-03-12T05:57:42.632 INFO:tasks.workunit.client.0.smithi076.stdout:checking err.h usability... yes 2024-03-12T05:57:42.648 INFO:tasks.workunit.client.0.smithi076.stdout:checking err.h presence... yes 2024-03-12T05:57:42.648 INFO:tasks.workunit.client.0.smithi076.stdout:checking for err.h... yes 2024-03-12T05:57:42.696 INFO:tasks.workunit.client.0.smithi076.stdout:checking linux/falloc.h usability... yes 2024-03-12T05:57:42.710 INFO:tasks.workunit.client.0.smithi076.stdout:checking linux/falloc.h presence... yes 2024-03-12T05:57:42.710 INFO:tasks.workunit.client.0.smithi076.stdout:checking for linux/falloc.h... yes 2024-03-12T05:57:42.764 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-03-12T05:57:42.792 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-03-12T05:57:42.793 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/fs/xfs_fsops.h... no 2024-03-12T05:57:42.838 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/fs/xfs_itable.h usability... no 2024-03-12T05:57:42.864 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/fs/xfs_itable.h presence... no 2024-03-12T05:57:42.865 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/fs/xfs_itable.h... no 2024-03-12T05:57:42.907 INFO:tasks.workunit.client.0.smithi076.stdout:checking xfs/platform_defs.h usability... no 2024-03-12T05:57:42.934 INFO:tasks.workunit.client.0.smithi076.stdout:checking xfs/platform_defs.h presence... no 2024-03-12T05:57:42.934 INFO:tasks.workunit.client.0.smithi076.stdout:checking for xfs/platform_defs.h... no 2024-03-12T05:57:42.976 INFO:tasks.workunit.client.0.smithi076.stdout:checking btrfs/ioctl.h usability... no 2024-03-12T05:57:43.002 INFO:tasks.workunit.client.0.smithi076.stdout:checking btrfs/ioctl.h presence... no 2024-03-12T05:57:43.002 INFO:tasks.workunit.client.0.smithi076.stdout:checking for btrfs/ioctl.h... no 2024-03-12T05:57:43.043 INFO:tasks.workunit.client.0.smithi076.stdout:checking cifs/ioctl.h usability... no 2024-03-12T05:57:43.068 INFO:tasks.workunit.client.0.smithi076.stdout:checking cifs/ioctl.h presence... no 2024-03-12T05:57:43.068 INFO:tasks.workunit.client.0.smithi076.stdout:checking for cifs/ioctl.h... no 2024-03-12T05:57:43.108 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/mman.h usability... yes 2024-03-12T05:57:43.124 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/mman.h presence... yes 2024-03-12T05:57:43.124 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/mman.h... yes 2024-03-12T05:57:43.159 INFO:tasks.workunit.client.0.smithi076.stdout:checking for xfs/xfs_log_format.h... no 2024-03-12T05:57:43.211 INFO:tasks.workunit.client.0.smithi076.stdout:checking for uuid_compare... no 2024-03-12T05:57:43.249 INFO:tasks.workunit.client.0.smithi076.stdout:checking for uuid_compare in -luuid... yes 2024-03-12T05:57:43.317 INFO:tasks.workunit.client.0.smithi076.stdout:checking for xfs/xfs.h... yes 2024-03-12T05:57:43.349 INFO:tasks.workunit.client.0.smithi076.stdout:checking for xfs/libxfs.h... no 2024-03-12T05:57:43.377 INFO:tasks.workunit.client.0.smithi076.stdout:checking whether xlog_assign_lsn is declared... no 2024-03-12T05:57:43.415 INFO:tasks.workunit.client.0.smithi076.stdout:checking for xfs/xqm.h... yes 2024-03-12T05:57:43.466 INFO:tasks.workunit.client.0.smithi076.stdout:checking xfsctl from xfs/xfs.h... ok 2024-03-12T05:57:43.500 INFO:tasks.workunit.client.0.smithi076.stdout:checking xfs/handle.h usability... yes 2024-03-12T05:57:43.515 INFO:tasks.workunit.client.0.smithi076.stdout:checking xfs/handle.h presence... yes 2024-03-12T05:57:43.515 INFO:tasks.workunit.client.0.smithi076.stdout:checking for xfs/handle.h... yes 2024-03-12T05:57:43.554 INFO:tasks.workunit.client.0.smithi076.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-03-12T05:57:43.627 INFO:tasks.workunit.client.0.smithi076.stdout:checking attr/attributes.h usability... yes 2024-03-12T05:57:43.643 INFO:tasks.workunit.client.0.smithi076.stdout:checking attr/attributes.h presence... yes 2024-03-12T05:57:43.643 INFO:tasks.workunit.client.0.smithi076.stdout:checking for attr/attributes.h... yes 2024-03-12T05:57:43.700 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/attributes.h usability... no 2024-03-12T05:57:43.726 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/attributes.h presence... no 2024-03-12T05:57:43.726 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/attributes.h... no 2024-03-12T05:57:43.765 INFO:tasks.workunit.client.0.smithi076.stdout:checking for attr_list in -lattr... yes 2024-03-12T05:57:43.803 INFO:tasks.workunit.client.0.smithi076.stdout:checking for attr_set in -lattr... yes 2024-03-12T05:57:43.833 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/acl.h usability... yes 2024-03-12T05:57:43.849 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/acl.h presence... yes 2024-03-12T05:57:43.849 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/acl.h... yes 2024-03-12T05:57:43.879 INFO:tasks.workunit.client.0.smithi076.stdout:checking acl/libacl.h usability... yes 2024-03-12T05:57:43.896 INFO:tasks.workunit.client.0.smithi076.stdout:checking acl/libacl.h presence... yes 2024-03-12T05:57:43.896 INFO:tasks.workunit.client.0.smithi076.stdout:checking for acl/libacl.h... yes 2024-03-12T05:57:43.935 INFO:tasks.workunit.client.0.smithi076.stdout:checking for acl_init in -lacl... yes 2024-03-12T05:57:43.992 INFO:tasks.workunit.client.0.smithi076.stdout:checking gdbm-ndbm.h usability... no 2024-03-12T05:57:44.019 INFO:tasks.workunit.client.0.smithi076.stdout:checking gdbm-ndbm.h presence... no 2024-03-12T05:57:44.019 INFO:tasks.workunit.client.0.smithi076.stdout:checking for gdbm-ndbm.h... no 2024-03-12T05:57:44.055 INFO:tasks.workunit.client.0.smithi076.stdout:checking gdbm/ndbm.h usability... no 2024-03-12T05:57:44.081 INFO:tasks.workunit.client.0.smithi076.stdout:checking gdbm/ndbm.h presence... no 2024-03-12T05:57:44.082 INFO:tasks.workunit.client.0.smithi076.stdout:checking for gdbm/ndbm.h... no 2024-03-12T05:57:44.117 INFO:tasks.workunit.client.0.smithi076.stdout:checking ndbm.h usability... no 2024-03-12T05:57:44.146 INFO:tasks.workunit.client.0.smithi076.stdout:checking ndbm.h presence... no 2024-03-12T05:57:44.146 INFO:tasks.workunit.client.0.smithi076.stdout:checking for ndbm.h... no 2024-03-12T05:57:44.176 INFO:tasks.workunit.client.0.smithi076.stdout:checking libaio.h usability... yes 2024-03-12T05:57:44.193 INFO:tasks.workunit.client.0.smithi076.stdout:checking libaio.h presence... yes 2024-03-12T05:57:44.193 INFO:tasks.workunit.client.0.smithi076.stdout:checking for libaio.h... yes 2024-03-12T05:57:44.242 INFO:tasks.workunit.client.0.smithi076.stdout:checking for dm_make_handle in -ldm... no 2024-03-12T05:57:44.242 INFO:tasks.workunit.client.0.smithi076.stdout: 2024-03-12T05:57:44.242 INFO:tasks.workunit.client.0.smithi076.stdout:WARNING: could not find a valid DMAPI base library. 2024-03-12T05:57:44.242 INFO:tasks.workunit.client.0.smithi076.stdout:If you want DMAPI support please install the data migration 2024-03-12T05:57:44.242 INFO:tasks.workunit.client.0.smithi076.stdout:API (dmapi) library package. Alternatively, run "make install" 2024-03-12T05:57:44.242 INFO:tasks.workunit.client.0.smithi076.stdout:from the dmapi source. 2024-03-12T05:57:44.242 INFO:tasks.workunit.client.0.smithi076.stdout: 2024-03-12T05:57:44.273 INFO:tasks.workunit.client.0.smithi076.stdout:checking linux/fiemap.h usability... yes 2024-03-12T05:57:44.288 INFO:tasks.workunit.client.0.smithi076.stdout:checking linux/fiemap.h presence... yes 2024-03-12T05:57:44.288 INFO:tasks.workunit.client.0.smithi076.stdout:checking for linux/fiemap.h... yes 2024-03-12T05:57:44.325 INFO:tasks.workunit.client.0.smithi076.stdout:checking for fallocate... yes 2024-03-12T05:57:44.373 INFO:tasks.workunit.client.0.smithi076.stdout:checking for open_by_handle_at... yes 2024-03-12T05:57:44.420 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/prctl.h usability... yes 2024-03-12T05:57:44.437 INFO:tasks.workunit.client.0.smithi076.stdout:checking sys/prctl.h presence... yes 2024-03-12T05:57:44.437 INFO:tasks.workunit.client.0.smithi076.stdout:checking for sys/prctl.h... yes 2024-03-12T05:57:44.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:44 smithi082 ceph-mon[36382]: pgmap v61: 97 pgs: 97 active+clean; 73 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 325 op/s 2024-03-12T05:57:44.468 INFO:tasks.workunit.client.0.smithi076.stdout:checking linux/fs.h usability... yes 2024-03-12T05:57:44.484 INFO:tasks.workunit.client.0.smithi076.stdout:checking linux/fs.h presence... yes 2024-03-12T05:57:44.484 INFO:tasks.workunit.client.0.smithi076.stdout:checking for linux/fs.h... yes 2024-03-12T05:57:44.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:44 smithi088 ceph-mon[36585]: pgmap v61: 97 pgs: 97 active+clean; 73 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 325 op/s 2024-03-12T05:57:44.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:44 smithi076 ceph-mon[32095]: pgmap v61: 97 pgs: 97 active+clean; 73 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 325 op/s 2024-03-12T05:57:44.542 INFO:tasks.workunit.client.0.smithi076.stdout:checking btrfsutil.h usability... no 2024-03-12T05:57:44.574 INFO:tasks.workunit.client.0.smithi076.stdout:checking btrfsutil.h presence... no 2024-03-12T05:57:44.574 INFO:tasks.workunit.client.0.smithi076.stdout:checking for btrfsutil.h... no 2024-03-12T05:57:44.628 INFO:tasks.workunit.client.0.smithi076.stdout:checking for copy_file_range... yes 2024-03-12T05:57:44.671 INFO:tasks.workunit.client.0.smithi076.stdout:checking for renameat2... yes 2024-03-12T05:57:44.690 INFO:tasks.workunit.client.0.smithi076.stdout:configure: creating ./config.status 2024-03-12T05:57:45.037 INFO:tasks.workunit.client.0.smithi076.stdout:config.status: creating include/builddefs 2024-03-12T05:57:45.050 INFO:tasks.workunit.client.0.smithi076.stdout:config.status: creating include/config.h 2024-03-12T05:57:45.060 INFO:tasks.workunit.client.0.smithi076.stdout:config.status: executing libtool commands 2024-03-12T05:57:45.132 INFO:tasks.workunit.client.0.smithi076.stdout:Building include 2024-03-12T05:57:45.132 INFO:tasks.workunit.client.0.smithi076.stdout:Building lib 2024-03-12T05:57:45.132 INFO:tasks.workunit.client.0.smithi076.stdout:Building tests 2024-03-12T05:57:45.137 INFO:tasks.workunit.client.0.smithi076.stdout: [LTDEP] 2024-03-12T05:57:45.295 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] dataascii.lo 2024-03-12T05:57:45.295 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] databin.lo 2024-03-12T05:57:45.295 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] datapid.lo 2024-03-12T05:57:45.295 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] file_lock.lo 2024-03-12T05:57:45.402 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] forker.lo 2024-03-12T05:57:45.416 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] pattern.lo 2024-03-12T05:57:45.448 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] open_flags.lo 2024-03-12T05:57:45.466 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] random_range.lo 2024-03-12T05:57:45.548 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] string_to_tokens.lo 2024-03-12T05:57:45.549 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] str_to_bytes.lo 2024-03-12T05:57:45.599 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] tlibio.lo 2024-03-12T05:57:45.660 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] write_log.lo 2024-03-12T05:57:45.708 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] random.lo 2024-03-12T05:57:45.896 INFO:tasks.workunit.client.0.smithi076.stdout: [LD] 2024-03-12T05:57:45.994 INFO:tasks.workunit.client.0.smithi076.stdout:Building ltp 2024-03-12T05:57:45.994 INFO:tasks.workunit.client.0.smithi076.stdout:Building src 2024-03-12T05:57:45.994 INFO:tasks.workunit.client.0.smithi076.stdout:Building m4 2024-03-12T05:57:45.994 INFO:tasks.workunit.client.0.smithi076.stdout:Building common 2024-03-12T05:57:46.002 INFO:tasks.workunit.client.0.smithi076.stdout: [DEP] 2024-03-12T05:57:46.014 INFO:tasks.workunit.client.0.smithi076.stdout: [DEP] 2024-03-12T05:57:46.213 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] doio 2024-03-12T05:57:46.213 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] fsstress 2024-03-12T05:57:46.213 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] fsx 2024-03-12T05:57:46.356 INFO:tasks.workunit.client.0.smithi076.stderr:fsstress.c: In function 'attr_list_path': 2024-03-12T05:57:46.357 INFO:tasks.workunit.client.0.smithi076.stderr:fsstress.c:859:9: warning: 'attr_list' is deprecated: Use listxattr or llistxattr instead [-Wdeprecated-declarations] 2024-03-12T05:57:46.357 INFO:tasks.workunit.client.0.smithi076.stderr: 859 | rval = attr_list(name->path, buffer, buffersize, flags, cursor); 2024-03-12T05:57:46.357 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~ 2024-03-12T05:57:46.357 INFO:tasks.workunit.client.0.smithi076.stderr:In file included from ../src/global.h:25, 2024-03-12T05:57:46.357 INFO:tasks.workunit.client.0.smithi076.stderr: from fsstress.c:12: 2024-03-12T05:57:46.357 INFO:tasks.workunit.client.0.smithi076.stderr:/usr/include/attr/attributes.h:165:12: note: declared here 2024-03-12T05:57:46.357 INFO:tasks.workunit.client.0.smithi076.stderr: 165 | extern int attr_list(const char *__path, char *__buffer, const int __buffersize, 2024-03-12T05:57:46.357 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~~~~~ 2024-03-12T05:57:46.357 INFO:tasks.workunit.client.0.smithi076.stderr:fsstress.c: In function 'attr_remove_path': 2024-03-12T05:57:46.357 INFO:tasks.workunit.client.0.smithi076.stderr:fsstress.c:878:9: warning: 'attr_remove' is deprecated: Use removexattr or lremovexattr instead [-Wdeprecated-declarations] 2024-03-12T05:57:46.357 INFO:tasks.workunit.client.0.smithi076.stderr: 878 | rval = attr_remove(name->path, attrname, flags); 2024-03-12T05:57:46.357 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~ 2024-03-12T05:57:46.357 INFO:tasks.workunit.client.0.smithi076.stderr:In file included from ../src/global.h:25, 2024-03-12T05:57:46.357 INFO:tasks.workunit.client.0.smithi076.stderr: from fsstress.c:12: 2024-03-12T05:57:46.358 INFO:tasks.workunit.client.0.smithi076.stderr:/usr/include/attr/attributes.h:152:12: note: declared here 2024-03-12T05:57:46.358 INFO:tasks.workunit.client.0.smithi076.stderr: 152 | extern int attr_remove (const char *__path, const char *__attrname, 2024-03-12T05:57:46.358 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~~~~~~~ 2024-03-12T05:57:46.358 INFO:tasks.workunit.client.0.smithi076.stderr:fsstress.c: In function 'attr_set_path': 2024-03-12T05:57:46.358 INFO:tasks.workunit.client.0.smithi076.stderr:fsstress.c:898:9: warning: 'attr_set' is deprecated: Use setxattr or lsetxattr instead [-Wdeprecated-declarations] 2024-03-12T05:57:46.358 INFO:tasks.workunit.client.0.smithi076.stderr: 898 | rval = attr_set(name->path, attrname, attrvalue, valuelength, flags); 2024-03-12T05:57:46.358 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~ 2024-03-12T05:57:46.358 INFO:tasks.workunit.client.0.smithi076.stderr:In file included from ../src/global.h:25, 2024-03-12T05:57:46.358 INFO:tasks.workunit.client.0.smithi076.stderr: from fsstress.c:12: 2024-03-12T05:57:46.358 INFO:tasks.workunit.client.0.smithi076.stderr:/usr/include/attr/attributes.h:139:12: note: declared here 2024-03-12T05:57:46.358 INFO:tasks.workunit.client.0.smithi076.stderr: 139 | extern int attr_set (const char *__path, const char *__attrname, 2024-03-12T05:57:46.358 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~~~~ 2024-03-12T05:57:46.367 INFO:tasks.workunit.client.0.smithi076.stderr:fsstress.c: In function 'allocsp_f': 2024-03-12T05:57:46.367 INFO:tasks.workunit.client.0.smithi076.stderr:fsstress.c:2047:32: error: 'XFS_IOC_ALLOCSP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-03-12T05:57:46.367 INFO:tasks.workunit.client.0.smithi076.stderr: 2047 | e = xfsctl(f.path, fd, XFS_IOC_ALLOCSP64, &fl) < 0 ? errno : 0; 2024-03-12T05:57:46.367 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~~~~~~~~~~~~~ 2024-03-12T05:57:46.367 INFO:tasks.workunit.client.0.smithi076.stderr: | XFS_IOC_RESVSP64 2024-03-12T05:57:46.368 INFO:tasks.workunit.client.0.smithi076.stderr:fsstress.c:2047:32: note: each undeclared identifier is reported only once for each function it appears in 2024-03-12T05:57:46.384 INFO:tasks.workunit.client.0.smithi076.stderr:fsstress.c: In function 'freesp_f': 2024-03-12T05:57:46.385 INFO:tasks.workunit.client.0.smithi076.stderr:fsstress.c:3640:32: error: 'XFS_IOC_FREESP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-03-12T05:57:46.385 INFO:tasks.workunit.client.0.smithi076.stderr: 3640 | e = xfsctl(f.path, fd, XFS_IOC_FREESP64, &fl) < 0 ? errno : 0; 2024-03-12T05:57:46.385 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~~~~~~~~~~~~ 2024-03-12T05:57:46.385 INFO:tasks.workunit.client.0.smithi076.stderr: | XFS_IOC_RESVSP64 2024-03-12T05:57:46.417 INFO:tasks.workunit.client.0.smithi076.stderr:gmake[3]: *** [Makefile:47: fsstress] Error 1 2024-03-12T05:57:46.417 INFO:tasks.workunit.client.0.smithi076.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-03-12T05:57:46.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:46 smithi082 ceph-mon[36382]: pgmap v62: 97 pgs: 97 active+clean; 73 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.8 MiB/s wr, 257 op/s 2024-03-12T05:57:46.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:46 smithi088 ceph-mon[36585]: pgmap v62: 97 pgs: 97 active+clean; 73 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.8 MiB/s wr, 257 op/s 2024-03-12T05:57:46.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:46 smithi076 ceph-mon[32095]: pgmap v62: 97 pgs: 97 active+clean; 73 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.8 MiB/s wr, 257 op/s 2024-03-12T05:57:46.777 INFO:tasks.workunit.client.0.smithi076.stderr:doio.c: In function 'parse_cmdline': 2024-03-12T05:57:46.777 INFO:tasks.workunit.client.0.smithi076.stderr:doio.c:3113:33: warning: '%s' directive argument is null [-Wformat-overflow=] 2024-03-12T05:57:46.777 INFO:tasks.workunit.client.0.smithi076.stderr: 3113 | fprintf(stderr, 2024-03-12T05:57:46.778 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~~~~~~~~~~~ 2024-03-12T05:57:46.778 INFO:tasks.workunit.client.0.smithi076.stderr: 3114 | "%s%s: Illegal -C arg (%s). Must be one of: ", 2024-03-12T05:57:46.778 INFO:tasks.workunit.client.0.smithi076.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-12T05:57:46.778 INFO:tasks.workunit.client.0.smithi076.stderr: 3115 | Prog, TagName, tok); 2024-03-12T05:57:46.778 INFO:tasks.workunit.client.0.smithi076.stderr: | ~~~~~~~~~~~~~~~~~~~ 2024-03-12T05:57:46.816 INFO:tasks.workunit.client.0.smithi076.stderr:fsx.c: In function 'main': 2024-03-12T05:57:46.816 INFO:tasks.workunit.client.0.smithi076.stderr:fsx.c:2662:33: warning: 'strncpy' specified bound 4096 equals destination size [-Wstringop-truncation] 2024-03-12T05:57:46.816 INFO:tasks.workunit.client.0.smithi076.stderr: 2662 | strncpy(opsfile, optarg, sizeof(opsfile)); 2024-03-12T05:57:46.816 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-12T05:57:46.816 INFO:tasks.workunit.client.0.smithi076.stderr:fsx.c:2632:25: warning: 'strncpy' specified bound 1024 equals destination size [-Wstringop-truncation] 2024-03-12T05:57:46.816 INFO:tasks.workunit.client.0.smithi076.stderr: 2632 | strncpy(dname, optarg, sizeof(dname)); 2024-03-12T05:57:46.816 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-12T05:57:46.912 INFO:tasks.workunit.client.0.smithi076.stderr:gmake[2]: *** [include/buildrules:30: ltp] Error 2 2024-03-12T05:57:46.912 INFO:tasks.workunit.client.0.smithi076.stderr:gmake[2]: *** Waiting for unfinished jobs.... 2024-03-12T05:57:47.827 INFO:tasks.workunit.client.0.smithi076.stdout: [DEP] 2024-03-12T05:57:47.828 INFO:tasks.workunit.client.0.smithi076.stdout: [DEP] 2024-03-12T05:57:47.829 INFO:tasks.workunit.client.0.smithi076.stdout: [DEP] 2024-03-12T05:57:48.051 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] dirstress 2024-03-12T05:57:48.051 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] fill 2024-03-12T05:57:48.052 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] fill2 2024-03-12T05:57:48.052 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] getpagesize 2024-03-12T05:57:48.211 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] holes 2024-03-12T05:57:48.227 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] lstat64 2024-03-12T05:57:48.312 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] nametest 2024-03-12T05:57:48.320 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] permname 2024-03-12T05:57:48.427 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] randholes 2024-03-12T05:57:48.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:48 smithi082 ceph-mon[36382]: pgmap v63: 97 pgs: 97 active+clean; 73 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.8 MiB/s wr, 257 op/s 2024-03-12T05:57:48.447 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] runas 2024-03-12T05:57:48.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:48 smithi088 ceph-mon[36585]: pgmap v63: 97 pgs: 97 active+clean; 73 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.8 MiB/s wr, 257 op/s 2024-03-12T05:57:48.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:48 smithi076 ceph-mon[32095]: pgmap v63: 97 pgs: 97 active+clean; 73 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.8 MiB/s wr, 257 op/s 2024-03-12T05:57:48.540 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] truncfile 2024-03-12T05:57:48.576 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] usemem 2024-03-12T05:57:48.644 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] mmapcat 2024-03-12T05:57:48.713 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] append_reader 2024-03-12T05:57:48.730 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] append_writer 2024-03-12T05:57:48.752 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] dirperf 2024-03-12T05:57:48.810 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] metaperf 2024-03-12T05:57:48.877 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] devzero 2024-03-12T05:57:48.908 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] feature 2024-03-12T05:57:48.997 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] alloc 2024-03-12T05:57:49.082 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] fault 2024-03-12T05:57:49.134 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] fstest 2024-03-12T05:57:49.134 INFO:tasks.workunit.client.0.smithi076.stdout: [CC] t_access_root 2024-03-12T05:57:49.137 INFO:tasks.workunit.client.0.smithi076.stderr:alloc.c: In function 'main': 2024-03-12T05:57:49.138 INFO:tasks.workunit.client.0.smithi076.stderr:alloc.c:154:1: error: 'bozo' undeclared (first use in this function) 2024-03-12T05:57:49.138 INFO:tasks.workunit.client.0.smithi076.stderr: 154 | bozo! 2024-03-12T05:57:49.138 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~ 2024-03-12T05:57:49.138 INFO:tasks.workunit.client.0.smithi076.stderr:alloc.c:154:1: note: each undeclared identifier is reported only once for each function it appears in 2024-03-12T05:57:49.138 INFO:tasks.workunit.client.0.smithi076.stderr:alloc.c:154:5: error: expected ';' before '!' token 2024-03-12T05:57:49.138 INFO:tasks.workunit.client.0.smithi076.stderr: 154 | bozo! 2024-03-12T05:57:49.138 INFO:tasks.workunit.client.0.smithi076.stderr: | ^ 2024-03-12T05:57:49.138 INFO:tasks.workunit.client.0.smithi076.stderr: | ; 2024-03-12T05:57:49.139 INFO:tasks.workunit.client.0.smithi076.stderr:alloc.c:184:25: error: 'rflag' undeclared (first use in this function); did you mean 'nflag'? 2024-03-12T05:57:49.139 INFO:tasks.workunit.client.0.smithi076.stderr: 184 | rflag = 1; 2024-03-12T05:57:49.139 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~ 2024-03-12T05:57:49.139 INFO:tasks.workunit.client.0.smithi076.stderr: | nflag 2024-03-12T05:57:49.139 INFO:tasks.workunit.client.0.smithi076.stderr:alloc.c:320:5: error: expected ';' before '!' token 2024-03-12T05:57:49.139 INFO:tasks.workunit.client.0.smithi076.stderr: 320 | bozo! 2024-03-12T05:57:49.139 INFO:tasks.workunit.client.0.smithi076.stderr: | ^ 2024-03-12T05:57:49.139 INFO:tasks.workunit.client.0.smithi076.stderr: | ; 2024-03-12T05:57:49.139 INFO:tasks.workunit.client.0.smithi076.stderr:alloc.c:128:25: warning: variable 'f' set but not used [-Wunused-but-set-variable] 2024-03-12T05:57:49.139 INFO:tasks.workunit.client.0.smithi076.stderr: 128 | struct flock64 f; 2024-03-12T05:57:49.139 INFO:tasks.workunit.client.0.smithi076.stderr: | ^ 2024-03-12T05:57:49.142 INFO:tasks.workunit.client.0.smithi076.stderr:gmake[3]: *** [Makefile:83: alloc] Error 1 2024-03-12T05:57:49.142 INFO:tasks.workunit.client.0.smithi076.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-03-12T05:57:49.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:57:49.203 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:57:49.307 INFO:tasks.workunit.client.0.smithi076.stderr:fstest.c: In function 'run_child': 2024-03-12T05:57:49.308 INFO:tasks.workunit.client.0.smithi076.stderr:fstest.c:95:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-03-12T05:57:49.308 INFO:tasks.workunit.client.0.smithi076.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-12T05:57:49.308 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~ 2024-03-12T05:57:49.308 INFO:tasks.workunit.client.0.smithi076.stderr:fstest.c:95:24: note: directive argument in the range [0, 2147483647] 2024-03-12T05:57:49.308 INFO:tasks.workunit.client.0.smithi076.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-12T05:57:49.308 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~~~~~~~ 2024-03-12T05:57:49.308 INFO:tasks.workunit.client.0.smithi076.stderr:fstest.c:95:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-03-12T05:57:49.308 INFO:tasks.workunit.client.0.smithi076.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-12T05:57:49.308 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-12T05:57:49.308 INFO:tasks.workunit.client.0.smithi076.stderr:fstest.c:166:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-03-12T05:57:49.309 INFO:tasks.workunit.client.0.smithi076.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-12T05:57:49.309 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~ 2024-03-12T05:57:49.309 INFO:tasks.workunit.client.0.smithi076.stderr:fstest.c:166:24: note: directive argument in the range [0, 2147483647] 2024-03-12T05:57:49.309 INFO:tasks.workunit.client.0.smithi076.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-12T05:57:49.309 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~~~~~~~ 2024-03-12T05:57:49.309 INFO:tasks.workunit.client.0.smithi076.stderr:fstest.c:166:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-03-12T05:57:49.309 INFO:tasks.workunit.client.0.smithi076.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-12T05:57:49.309 INFO:tasks.workunit.client.0.smithi076.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-12T05:57:49.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.310+0000 7fa3f738a640 1 -- 172.21.15.76:0/2129193646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3f0100260 msgr2=0x7fa3f0100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.310+0000 7fa3f738a640 1 --2- 172.21.15.76:0/2129193646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3f0100260 0x7fa3f0100640 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009930 tx=0x7fa3e002bc30 comp rx=0 tx=0).stop 2024-03-12T05:57:49.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.310+0000 7fa3f738a640 1 -- 172.21.15.76:0/2129193646 shutdown_connections 2024-03-12T05:57:49.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.310+0000 7fa3f738a640 1 --2- 172.21.15.76:0/2129193646 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa3f0105330 0x7fa3f010cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.310+0000 7fa3f738a640 1 --2- 172.21.15.76:0/2129193646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f0100c10 0x7fa3f0104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.310+0000 7fa3f738a640 1 --2- 172.21.15.76:0/2129193646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3f0100260 0x7fa3f0100640 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.310+0000 7fa3f738a640 1 -- 172.21.15.76:0/2129193646 >> 172.21.15.76:0/2129193646 conn(0x7fa3f0077180 msgr2=0x7fa3f0077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:49.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.310+0000 7fa3f738a640 1 -- 172.21.15.76:0/2129193646 shutdown_connections 2024-03-12T05:57:49.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.310+0000 7fa3f738a640 1 -- 172.21.15.76:0/2129193646 wait complete. 2024-03-12T05:57:49.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f738a640 1 Processor -- start 2024-03-12T05:57:49.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f738a640 1 -- start start 2024-03-12T05:57:49.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f738a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3f0100260 0x7fa3f019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f738a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa3f0105330 0x7fa3f019d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f738a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f0197640 0x7fa3f0197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f738a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f0198030 con 0x7fa3f0100260 2024-03-12T05:57:49.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f738a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f01981a0 con 0x7fa3f0105330 2024-03-12T05:57:49.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f738a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f0198360 con 0x7fa3f0197640 2024-03-12T05:57:49.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f5126640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3f0100260 0x7fa3f019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f5126640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3f0100260 0x7fa3f019d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37826/0 (socket says 172.21.15.76:37826) 2024-03-12T05:57:49.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f5126640 1 -- 172.21.15.76:0/2889212257 learned_addr learned my addr 172.21.15.76:0/2889212257 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:49.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f4925640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa3f0105330 0x7fa3f019d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f5927640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f0197640 0x7fa3f0197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f5126640 1 -- 172.21.15.76:0/2889212257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f0197640 msgr2=0x7fa3f0197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f5126640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f0197640 0x7fa3f0197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f5126640 1 -- 172.21.15.76:0/2889212257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa3f0105330 msgr2=0x7fa3f019d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f5126640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa3f0105330 0x7fa3f019d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f5126640 1 -- 172.21.15.76:0/2889212257 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e0009560 con 0x7fa3f0100260 2024-03-12T05:57:49.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f4925640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa3f0105330 0x7fa3f019d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:49.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f5927640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f0197640 0x7fa3f0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:49.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.311+0000 7fa3f5126640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3f0100260 0x7fa3f019d440 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009230 tx=0x7fa3e002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:49.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.312+0000 7fa3e67fc640 1 -- 172.21.15.76:0/2889212257 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e0004350 con 0x7fa3f0100260 2024-03-12T05:57:49.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.312+0000 7fa3e67fc640 1 -- 172.21.15.76:0/2889212257 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3e00044b0 con 0x7fa3f0100260 2024-03-12T05:57:49.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.312+0000 7fa3f738a640 1 -- 172.21.15.76:0/2889212257 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f01a4300 con 0x7fa3f0100260 2024-03-12T05:57:49.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.312+0000 7fa3e67fc640 1 -- 172.21.15.76:0/2889212257 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e0003d00 con 0x7fa3f0100260 2024-03-12T05:57:49.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.312+0000 7fa3f738a640 1 -- 172.21.15.76:0/2889212257 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f01a47b0 con 0x7fa3f0100260 2024-03-12T05:57:49.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.314+0000 7fa3e67fc640 1 -- 172.21.15.76:0/2889212257 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fa3e0004df0 con 0x7fa3f0100260 2024-03-12T05:57:49.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.314+0000 7fa3fa035740 1 Processor -- start 2024-03-12T05:57:49.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.314+0000 7fa3fa035740 1 -- start start 2024-03-12T05:57:49.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3fa035740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df5ffe0 0x55e85df603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3e67fc640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa3dc0a1ba0 0x7fa3dc0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3fa035740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df60900 0x55e85df6d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3fa035740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df6dbc0 0x55e85df6ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3fa035740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e85decf280 con 0x55e85df60900 2024-03-12T05:57:49.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3fa035740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e85decc7b0 con 0x55e85df5ffe0 2024-03-12T05:57:49.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3fa035740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e85dec7f90 con 0x55e85df6dbc0 2024-03-12T05:57:49.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3f4925640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df60900 0x55e85df6d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3f5126640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa3dc0a1ba0 0x7fa3dc0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3f5927640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df6dbc0 0x55e85df6ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3f4925640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df60900 0x55e85df6d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37830/0 (socket says 172.21.15.76:37830) 2024-03-12T05:57:49.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3f4925640 1 -- 172.21.15.76:0/1636678898 learned_addr learned my addr 172.21.15.76:0/1636678898 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3f5927640 1 --2- 172.21.15.76:0/1636678898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df5ffe0 0x55e85df603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3e67fc640 1 -- 172.21.15.76:0/2889212257 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa3e00ef770 con 0x7fa3f0100260 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3f4925640 1 -- 172.21.15.76:0/1636678898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df6dbc0 msgr2=0x55e85df6ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3f4925640 1 --2- 172.21.15.76:0/1636678898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df6dbc0 0x55e85df6ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3f4925640 1 -- 172.21.15.76:0/1636678898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df5ffe0 msgr2=0x55e85df603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3f4925640 1 --2- 172.21.15.76:0/1636678898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df5ffe0 0x55e85df603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3f4925640 1 -- 172.21.15.76:0/1636678898 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e85de86570 con 0x55e85df60900 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3f5927640 1 --2- 172.21.15.76:0/1636678898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df5ffe0 0x55e85df603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3f5126640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa3dc0a1ba0 0x7fa3dc0a4060 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3e00b8eb0 tx=0x7fa3e009a040 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3f4925640 1 --2- 172.21.15.76:0/1636678898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df60900 0x55e85df6d680 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec009c70 tx=0x7fa3ec0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739f2e63e0c253c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.315+0000 7fa3db7fe640 1 -- 172.21.15.76:0/1636678898 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec012450 con 0x55e85df60900 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.316+0000 7fa3db7fe640 1 -- 172.21.15.76:0/1636678898 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3ec012a90 con 0x55e85df60900 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.316+0000 7fa3db7fe640 1 -- 172.21.15.76:0/1636678898 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec00fbd0 con 0x55e85df60900 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.316+0000 7fa3fa035740 1 -- 172.21.15.76:0/1636678898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df60900 msgr2=0x55e85df6d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.316+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1636678898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df60900 0x55e85df6d680 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec009c70 tx=0x7fa3ec0117b0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.316+0000 7fa3fa035740 1 -- 172.21.15.76:0/1636678898 shutdown_connections 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.316+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1636678898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df6dbc0 0x55e85df6ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.316+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1636678898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df60900 0x55e85df6d680 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.316+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1636678898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df5ffe0 0x55e85df603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.316+0000 7fa3fa035740 1 -- 172.21.15.76:0/1636678898 >> 172.21.15.76:0/1636678898 conn(0x55e85df5a990 msgr2=0x55e85df5ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.316+0000 7fa3fa035740 1 -- 172.21.15.76:0/1636678898 shutdown_connections 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.316+0000 7fa3fa035740 1 -- 172.21.15.76:0/1636678898 wait complete. 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.317+0000 7fa3fa035740 1 -- start start 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.317+0000 7fa3fa035740 1 Processor -- start 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.317+0000 7fa3fa035740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 0x55e85df6d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.317+0000 7fa3fa035740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 0x55e85df6a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.317+0000 7fa3fa035740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df6af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.317+0000 7fa3fa035740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e85db9b430 con 0x55e85df6dbc0 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.317+0000 7fa3fa035740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e85db57e00 con 0x55e85df60900 2024-03-12T05:57:49.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.317+0000 7fa3fa035740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e85de4cfd0 con 0x55e85df5ffe0 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.317+0000 7fa3fa035740 10 client.0 _ll_register_callbacks cb 0x55e85dea2fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.317+0000 7fa3f5927640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df6af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.317+0000 7fa3f5927640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df6af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37842/0 (socket says 172.21.15.76:37842) 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.317+0000 7fa3f5927640 1 -- 172.21.15.76:0/1688536249 learned_addr learned my addr 172.21.15.76:0/1688536249 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.317+0000 7fa3f5927640 1 --2- 172.21.15.76:0/1688536249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 0x55e85df6d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3f4925640 1 --2- 172.21.15.76:0/1688536249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 0x55e85df6a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3f5927640 1 -- 172.21.15.76:0/1688536249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 msgr2=0x55e85df6d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3f5927640 1 --2- 172.21.15.76:0/1688536249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 0x55e85df6d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3f5927640 1 -- 172.21.15.76:0/1688536249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 msgr2=0x55e85df6a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3f5927640 1 --2- 172.21.15.76:0/1688536249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 0x55e85df6a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3f5927640 1 -- 172.21.15.76:0/1688536249 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ec0098a0 con 0x55e85df6dbc0 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3f5927640 1 --2- 172.21.15.76:0/1688536249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df6af80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8011f80 tx=0x7fa3e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9600a964b2e20b9b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3d8ff9640 1 -- 172.21.15.76:0/1688536249 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8004d20 con 0x55e85df6dbc0 2024-03-12T05:57:49.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3d8ff9640 1 -- 172.21.15.76:0/1688536249 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa3e800d550 con 0x55e85df6dbc0 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3fa035740 1 -- 172.21.15.76:0/1688536249 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e85dbc77e0 con 0x55e85df6dbc0 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3d8ff9640 1 -- 172.21.15.76:0/1688536249 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3e8004e80 con 0x55e85df6dbc0 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3fa035740 10 client.15993 fetch_fsmap learned FSMap version 24 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3fa035740 1 -- 172.21.15.76:0/1688536249 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55e85dec6610 con 0x55e85df6dbc0 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3d8ff9640 1 -- 172.21.15.76:0/1688536249 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e85dbc77e0 con 0x55e85df6dbc0 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3d8ff9640 1 -- 172.21.15.76:0/1688536249 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa3e80053d0 con 0x55e85df6dbc0 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3fa035740 10 client.15993 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3fa035740 10 client.15993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3fa035740 20 client.15993 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.318+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1688536249 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e85df7ddb0 0x55e85df801d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.319+0000 7fa3fa035740 4 client.15993 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.319+0000 7fa3fa035740 1 -- 172.21.15.76:0/1688536249 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e85dc04720 con 0x55e85df7ddb0 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.319+0000 7fa3f4925640 1 --2- 172.21.15.76:0/1688536249 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e85df7ddb0 0x55e85df801d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.319+0000 7fa3d8ff9640 10 client.15993 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:49.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.319+0000 7fa3f4925640 1 --2- 172.21.15.76:0/1688536249 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e85df7ddb0 0x55e85df801d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7bdff5f369d2e4e server_cookie=1d39e2b891c0941c in_seq=0 out_seq=0 2024-03-12T05:57:49.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3d8ff9640 1 -- 172.21.15.76:0/1688536249 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e85dc04720 con 0x55e85df7ddb0 2024-03-12T05:57:49.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3d8ff9640 10 client.15993 handle_command_reply: tid=0 2024-03-12T05:57:49.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3fa035740 1 client.15993 shutdown 2024-03-12T05:57:49.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3fa035740 1 -- 172.21.15.76:0/1688536249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 msgr2=0x55e85df6af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1688536249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df6af80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8011f80 tx=0x7fa3e8004a40 comp rx=0 tx=0).stop 2024-03-12T05:57:49.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3fa035740 1 -- 172.21.15.76:0/1688536249 shutdown_connections 2024-03-12T05:57:49.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1688536249 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e85df7ddb0 0x55e85df801d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1688536249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df6af80 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3d8ff9640 0 client.15993 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:49.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1688536249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 0x55e85df6a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1688536249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 0x55e85df6d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3fa035740 1 -- 172.21.15.76:0/1688536249 >> 172.21.15.76:0/1688536249 conn(0x55e85df5a990 msgr2=0x55e85df5c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:49.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3fa035740 1 -- 172.21.15.76:0/1688536249 shutdown_connections 2024-03-12T05:57:49.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3fa035740 1 -- 172.21.15.76:0/1688536249 wait complete. 2024-03-12T05:57:49.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.320+0000 7fa3fa035740 20 client.15993 trim_cache size 0 max 16384 2024-03-12T05:57:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.335+0000 7fa3fa035740 1 Processor -- start 2024-03-12T05:57:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.335+0000 7fa3fa035740 1 -- start start 2024-03-12T05:57:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.335+0000 7fa3fa035740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 0x55e85df6d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.335+0000 7fa3fa035740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 0x55e85df6d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.335+0000 7fa3fa035740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df6a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.335+0000 7fa3fa035740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e85df560d0 con 0x55e85df6dbc0 2024-03-12T05:57:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.335+0000 7fa3fa035740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e85df7c5b0 con 0x55e85df60900 2024-03-12T05:57:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.335+0000 7fa3fa035740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e85df5a7f0 con 0x55e85df5ffe0 2024-03-12T05:57:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3f5927640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df6a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3f5927640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df6a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37850/0 (socket says 172.21.15.76:37850) 2024-03-12T05:57:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3f5927640 1 -- 172.21.15.76:0/278223737 learned_addr learned my addr 172.21.15.76:0/278223737 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:49.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3f5927640 1 --2- 172.21.15.76:0/278223737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 0x55e85df6d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3f4925640 1 --2- 172.21.15.76:0/278223737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 0x55e85df6d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3f5927640 1 -- 172.21.15.76:0/278223737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 msgr2=0x55e85df6d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3f5927640 1 --2- 172.21.15.76:0/278223737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 0x55e85df6d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3f5927640 1 -- 172.21.15.76:0/278223737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 msgr2=0x55e85df6d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3f5927640 1 --2- 172.21.15.76:0/278223737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 0x55e85df6d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3f5927640 1 -- 172.21.15.76:0/278223737 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e85de62ac0 con 0x55e85df6dbc0 2024-03-12T05:57:49.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3f4925640 1 --2- 172.21.15.76:0/278223737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 0x55e85df6d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:57:49.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3f5927640 1 --2- 172.21.15.76:0/278223737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df6a930 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8009fd0 tx=0x7fa3e8010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05d744b4d89026c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:49.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3daffd640 1 -- 172.21.15.76:0/278223737 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8004ab0 con 0x55e85df6dbc0 2024-03-12T05:57:49.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3daffd640 1 -- 172.21.15.76:0/278223737 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3e8005500 con 0x55e85df6dbc0 2024-03-12T05:57:49.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.336+0000 7fa3daffd640 1 -- 172.21.15.76:0/278223737 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e800d970 con 0x55e85df6dbc0 2024-03-12T05:57:49.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.337+0000 7fa3fa035740 1 -- 172.21.15.76:0/278223737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 msgr2=0x55e85df6a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.337+0000 7fa3fa035740 1 --2- 172.21.15.76:0/278223737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df6a930 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8009fd0 tx=0x7fa3e8010f90 comp rx=0 tx=0).stop 2024-03-12T05:57:49.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.337+0000 7fa3fa035740 1 -- 172.21.15.76:0/278223737 shutdown_connections 2024-03-12T05:57:49.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.337+0000 7fa3fa035740 1 --2- 172.21.15.76:0/278223737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df6a930 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.337+0000 7fa3fa035740 1 --2- 172.21.15.76:0/278223737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 0x55e85df6d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.337+0000 7fa3fa035740 1 --2- 172.21.15.76:0/278223737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 0x55e85df6d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.337+0000 7fa3fa035740 1 -- 172.21.15.76:0/278223737 >> 172.21.15.76:0/278223737 conn(0x55e85df5a990 msgr2=0x55e85df84b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:49.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.337+0000 7fa3fa035740 1 -- 172.21.15.76:0/278223737 shutdown_connections 2024-03-12T05:57:49.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.337+0000 7fa3fa035740 1 -- 172.21.15.76:0/278223737 wait complete. 2024-03-12T05:57:49.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.337+0000 7fa3fa035740 1 -- start start 2024-03-12T05:57:49.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.337+0000 7fa3fa035740 1 Processor -- start 2024-03-12T05:57:49.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3fa035740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 0x55e85df71380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3fa035740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 0x55e85df718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3fa035740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df71e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3fa035740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e85df75340 con 0x55e85df6dbc0 2024-03-12T05:57:49.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3fa035740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e85dfd4290 con 0x55e85df60900 2024-03-12T05:57:49.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3fa035740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e85dfd43d0 con 0x55e85df5ffe0 2024-03-12T05:57:49.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3fa035740 10 client.0 _ll_register_callbacks cb 0x7fa3ec009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:49.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3f5927640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df71e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3f5927640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df71e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37858/0 (socket says 172.21.15.76:37858) 2024-03-12T05:57:49.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3f5927640 1 -- 172.21.15.76:0/1894376866 learned_addr learned my addr 172.21.15.76:0/1894376866 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:49.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3f5927640 1 --2- 172.21.15.76:0/1894376866 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 0x55e85df71380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3f4925640 1 --2- 172.21.15.76:0/1894376866 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 0x55e85df718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3f5927640 1 -- 172.21.15.76:0/1894376866 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 msgr2=0x55e85df71380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3f5927640 1 --2- 172.21.15.76:0/1894376866 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 0x55e85df71380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3f5927640 1 -- 172.21.15.76:0/1894376866 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 msgr2=0x55e85df718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3f5927640 1 --2- 172.21.15.76:0/1894376866 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 0x55e85df718c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3f5927640 1 -- 172.21.15.76:0/1894376866 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e85dec6610 con 0x55e85df6dbc0 2024-03-12T05:57:49.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3f5927640 1 --2- 172.21.15.76:0/1894376866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df71e00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8005fd0 tx=0x7fa3e800d550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae842c0614be8bef server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:49.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.338+0000 7fa3d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:49.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3d8ff9640 1 -- 172.21.15.76:0/1894376866 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8005170 con 0x55e85df6dbc0 2024-03-12T05:57:49.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3d8ff9640 1 -- 172.21.15.76:0/1894376866 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa3e8032040 con 0x55e85df6dbc0 2024-03-12T05:57:49.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3fa035740 1 -- 172.21.15.76:0/1894376866 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e85df6a7a0 con 0x55e85df6dbc0 2024-03-12T05:57:49.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3d8ff9640 1 -- 172.21.15.76:0/1894376866 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3e80052d0 con 0x55e85df6dbc0 2024-03-12T05:57:49.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3fa035740 10 client.15999 fetch_fsmap learned FSMap version 24 2024-03-12T05:57:49.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3fa035740 1 -- 172.21.15.76:0/1894376866 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55e85de886a0 con 0x55e85df6dbc0 2024-03-12T05:57:49.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3d8ff9640 1 -- 172.21.15.76:0/1894376866 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e85df6a7a0 con 0x55e85df6dbc0 2024-03-12T05:57:49.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3d8ff9640 1 -- 172.21.15.76:0/1894376866 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa3e800dcf0 con 0x55e85df6dbc0 2024-03-12T05:57:49.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3fa035740 10 client.15999 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:57:49.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3fa035740 10 client.15999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:49.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3fa035740 20 client.15999 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:49.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1894376866 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e85dfd56d0 0x55e85dfd7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3fa035740 4 client.15999 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:57:49.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3fa035740 1 -- 172.21.15.76:0/1894376866 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fa3ec012a90 con 0x55e85dfd56d0 2024-03-12T05:57:49.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.339+0000 7fa3f4925640 1 --2- 172.21.15.76:0/1894376866 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e85dfd56d0 0x55e85dfd7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3d8ff9640 10 client.15999 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:49.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3f4925640 1 --2- 172.21.15.76:0/1894376866 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e85dfd56d0 0x55e85dfd7ac0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45041461899d1613 server_cookie=2315ab813b073d93 in_seq=0 out_seq=0 2024-03-12T05:57:49.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3d8ff9640 1 -- 172.21.15.76:0/1894376866 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7fa3ec012a90 con 0x55e85dfd56d0 2024-03-12T05:57:49.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3d8ff9640 10 client.15999 handle_command_reply: tid=0 2024-03-12T05:57:49.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3fa035740 1 client.15999 shutdown 2024-03-12T05:57:49.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3fa035740 1 -- 172.21.15.76:0/1894376866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 msgr2=0x55e85df71e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1894376866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df71e00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8005fd0 tx=0x7fa3e800d550 comp rx=0 tx=0).stop 2024-03-12T05:57:49.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3fa035740 1 -- 172.21.15.76:0/1894376866 shutdown_connections 2024-03-12T05:57:49.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1894376866 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e85dfd56d0 0x55e85dfd7ac0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1894376866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e85df6dbc0 0x55e85df71e00 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1894376866 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e85df60900 0x55e85df718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3d8ff9640 0 client.15999 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:49.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3fa035740 1 --2- 172.21.15.76:0/1894376866 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e85df5ffe0 0x55e85df71380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3fa035740 1 -- 172.21.15.76:0/1894376866 >> 172.21.15.76:0/1894376866 conn(0x55e85df5a990 msgr2=0x55e85df6a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:49.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3fa035740 1 -- 172.21.15.76:0/1894376866 shutdown_connections 2024-03-12T05:57:49.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3fa035740 1 -- 172.21.15.76:0/1894376866 wait complete. 2024-03-12T05:57:49.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.340+0000 7fa3fa035740 20 client.15999 trim_cache size 0 max 16384 2024-03-12T05:57:49.354 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:57:49.354 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T05:57:49.354 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T05:57:49.354 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:57:49.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.341+0000 7fa3f738a640 1 -- 172.21.15.76:0/2889212257 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa3dc0a1ba0 msgr2=0x7fa3dc0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.341+0000 7fa3f738a640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa3dc0a1ba0 0x7fa3dc0a4060 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3e00b8eb0 tx=0x7fa3e009a040 comp rx=0 tx=0).stop 2024-03-12T05:57:49.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.341+0000 7fa3f738a640 1 -- 172.21.15.76:0/2889212257 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3f0100260 msgr2=0x7fa3f019d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.341+0000 7fa3f738a640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3f0100260 0x7fa3f019d440 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009230 tx=0x7fa3e002c650 comp rx=0 tx=0).stop 2024-03-12T05:57:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.341+0000 7fa3f738a640 1 -- 172.21.15.76:0/2889212257 shutdown_connections 2024-03-12T05:57:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.341+0000 7fa3f738a640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fa3dc0a1ba0 0x7fa3dc0a4060 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.341+0000 7fa3f738a640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f0197640 0x7fa3f0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.341+0000 7fa3f738a640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa3f0105330 0x7fa3f019d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.341+0000 7fa3f738a640 1 --2- 172.21.15.76:0/2889212257 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa3f0100260 0x7fa3f019d440 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.341+0000 7fa3f738a640 1 -- 172.21.15.76:0/2889212257 >> 172.21.15.76:0/2889212257 conn(0x7fa3f0077180 msgr2=0x7fa3f00688b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.341+0000 7fa3f738a640 1 -- 172.21.15.76:0/2889212257 shutdown_connections 2024-03-12T05:57:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.341+0000 7fa3f738a640 1 -- 172.21.15.76:0/2889212257 wait complete. 2024-03-12T05:57:49.357 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:57:49.357 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:57:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-12T05:57:49.383 INFO:tasks.workunit.client.0.smithi076.stderr:gmake[2]: *** [include/buildrules:30: src] Error 2 2024-03-12T05:57:49.383 INFO:tasks.workunit.client.0.smithi076.stderr:make[1]: *** [Makefile:53: default] Error 2 2024-03-12T05:57:49.384 INFO:tasks.workunit.client.0.smithi076.stderr:make: *** [Makefile:51: default] Error 2 2024-03-12T05:57:49.384 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-03-12T05:57:49.384 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-12T05:57:49.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.457+0000 7f94f4883640 1 -- 172.21.15.76:0/2358766047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94f010b770 msgr2=0x7f94f010dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.457+0000 7f94f4883640 1 --2- 172.21.15.76:0/2358766047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94f010b770 0x7f94f010dbc0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f94e8009ca0 tx=0x7f94e802c180 comp rx=0 tx=0).stop 2024-03-12T05:57:49.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.457+0000 7f94f4883640 1 -- 172.21.15.76:0/2358766047 shutdown_connections 2024-03-12T05:57:49.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.457+0000 7f94f4883640 1 --2- 172.21.15.76:0/2358766047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94f010b770 0x7f94f010dbc0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.457+0000 7f94f4883640 1 --2- 172.21.15.76:0/2358766047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94f0102cc0 0x7f94f010b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.457+0000 7f94f4883640 1 --2- 172.21.15.76:0/2358766047 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94f01023a0 0x7f94f0102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.457+0000 7f94f4883640 1 -- 172.21.15.76:0/2358766047 >> 172.21.15.76:0/2358766047 conn(0x7f94f0077200 msgr2=0x7f94f0077610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:49.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.457+0000 7f94f4883640 1 -- 172.21.15.76:0/2358766047 shutdown_connections 2024-03-12T05:57:49.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.457+0000 7f94f4883640 1 -- 172.21.15.76:0/2358766047 wait complete. 2024-03-12T05:57:49.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.457+0000 7f94f4883640 1 Processor -- start 2024-03-12T05:57:49.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94f4883640 1 -- start start 2024-03-12T05:57:49.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94f4883640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94f01023a0 0x7f94f019d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94f4883640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94f0102cc0 0x7f94f019d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94f4883640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94f010b770 0x7f94f01975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94f4883640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f0197c20 con 0x7f94f0102cc0 2024-03-12T05:57:49.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94f4883640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f0197d90 con 0x7f94f01023a0 2024-03-12T05:57:49.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94f4883640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f0197f50 con 0x7f94f010b770 2024-03-12T05:57:49.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94edd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94f0102cc0 0x7f94f019d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94ee59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94f01023a0 0x7f94f019d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94edd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94f0102cc0 0x7f94f019d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37880/0 (socket says 172.21.15.76:37880) 2024-03-12T05:57:49.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94edd9b640 1 -- 172.21.15.76:0/2894089366 learned_addr learned my addr 172.21.15.76:0/2894089366 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:49.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94eed9d640 1 --2- 172.21.15.76:0/2894089366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94f010b770 0x7f94f01975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94ee59c640 1 -- 172.21.15.76:0/2894089366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94f010b770 msgr2=0x7f94f01975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94ee59c640 1 --2- 172.21.15.76:0/2894089366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94f010b770 0x7f94f01975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94ee59c640 1 -- 172.21.15.76:0/2894089366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94f0102cc0 msgr2=0x7f94f019d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94ee59c640 1 --2- 172.21.15.76:0/2894089366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94f0102cc0 0x7f94f019d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94ee59c640 1 -- 172.21.15.76:0/2894089366 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e8009900 con 0x7f94f01023a0 2024-03-12T05:57:49.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.458+0000 7f94ee59c640 1 --2- 172.21.15.76:0/2894089366 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94f01023a0 0x7f94f019d420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f94e400d6d0 tx=0x7f94e400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:49.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.459+0000 7f94df7fe640 1 -- 172.21.15.76:0/2894089366 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e4005220 con 0x7f94f01023a0 2024-03-12T05:57:49.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.459+0000 7f94f4883640 1 -- 172.21.15.76:0/2894089366 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f0198230 con 0x7f94f01023a0 2024-03-12T05:57:49.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.459+0000 7f94f4883640 1 -- 172.21.15.76:0/2894089366 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f01a44c0 con 0x7f94f01023a0 2024-03-12T05:57:49.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.459+0000 7f94df7fe640 1 -- 172.21.15.76:0/2894089366 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94e4005590 con 0x7f94f01023a0 2024-03-12T05:57:49.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.459+0000 7f94df7fe640 1 -- 172.21.15.76:0/2894089366 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e4010460 con 0x7f94f01023a0 2024-03-12T05:57:49.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.460+0000 7f94f4883640 1 -- 172.21.15.76:0/2894089366 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d0005290 con 0x7f94f01023a0 2024-03-12T05:57:49.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.461+0000 7f94df7fe640 1 -- 172.21.15.76:0/2894089366 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f94e40046c0 con 0x7f94f01023a0 2024-03-12T05:57:49.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.462+0000 7f94df7fe640 1 --2- 172.21.15.76:0/2894089366 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f94d80a1c20 0x7f94d80a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.462+0000 7f94edd9b640 1 --2- 172.21.15.76:0/2894089366 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f94d80a1c20 0x7f94d80a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.462+0000 7f94df7fe640 1 -- 172.21.15.76:0/2894089366 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f94e40cd190 con 0x7f94f01023a0 2024-03-12T05:57:49.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.462+0000 7f94edd9b640 1 --2- 172.21.15.76:0/2894089366 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f94d80a1c20 0x7f94d80a40e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f94f010d3c0 tx=0x7f94e0005f90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:49.470 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.465+0000 7f94df7fe640 1 -- 172.21.15.76:0/2894089366 <== mon.2 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f94e40961c0 con 0x7f94f01023a0 2024-03-12T05:57:49.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.680+0000 7f94f4883640 1 -- 172.21.15.76:0/2894089366 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94d00058f0 con 0x7f94f01023a0 2024-03-12T05:57:49.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.681+0000 7f94df7fe640 1 -- 172.21.15.76:0/2894089366 <== mon.2 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11331 (secure 0 0 0) 0x7f94d00058f0 con 0x7f94f01023a0 2024-03-12T05:57:49.690 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:57:49.690 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:57:27.157038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:57:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 24 2024-03-12T05:57:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.683+0000 7f94f4883640 1 -- 172.21.15.76:0/2894089366 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f94d80a1c20 msgr2=0x7f94d80a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.683+0000 7f94f4883640 1 --2- 172.21.15.76:0/2894089366 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f94d80a1c20 0x7f94d80a40e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f94f010d3c0 tx=0x7f94e0005f90 comp rx=0 tx=0).stop 2024-03-12T05:57:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.683+0000 7f94f4883640 1 -- 172.21.15.76:0/2894089366 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94f01023a0 msgr2=0x7f94f019d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.683+0000 7f94f4883640 1 --2- 172.21.15.76:0/2894089366 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94f01023a0 0x7f94f019d420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f94e400d6d0 tx=0x7f94e400db90 comp rx=0 tx=0).stop 2024-03-12T05:57:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.684+0000 7f94f4883640 1 -- 172.21.15.76:0/2894089366 shutdown_connections 2024-03-12T05:57:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.684+0000 7f94f4883640 1 --2- 172.21.15.76:0/2894089366 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f94d80a1c20 0x7f94d80a40e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.684+0000 7f94f4883640 1 --2- 172.21.15.76:0/2894089366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94f010b770 0x7f94f01975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.684+0000 7f94f4883640 1 --2- 172.21.15.76:0/2894089366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94f0102cc0 0x7f94f019d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.684+0000 7f94f4883640 1 --2- 172.21.15.76:0/2894089366 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f94f01023a0 0x7f94f019d420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.684+0000 7f94f4883640 1 -- 172.21.15.76:0/2894089366 >> 172.21.15.76:0/2894089366 conn(0x7f94f0077200 msgr2=0x7f94f00fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:49.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.684+0000 7f94f4883640 1 -- 172.21.15.76:0/2894089366 shutdown_connections 2024-03-12T05:57:49.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.684+0000 7f94f4883640 1 -- 172.21.15.76:0/2894089366 wait complete. 2024-03-12T05:57:49.700 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:57:49.700 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:57:49.720 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi076 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6a4997be2b8bb74411268cc42753f29ae9a78ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-12T05:57:49.938 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=33bc5d8b5898496ea0bbd3125e311c19 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi076 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6a4997be2b8bb74411268cc42753f29ae9a78ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-12T05:57:49.940 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-12T05:57:49.950 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-12T05:57:49.950 INFO:teuthology.task.exec:Executing custom commands... 2024-03-12T05:57:49.951 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2024-03-12T05:57:49.951 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-12T05:57:49.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.858+0000 7fbe53898640 1 -- 172.21.15.76:0/583101487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe4c101380 msgr2=0x7fbe4c101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.858+0000 7fbe53898640 1 --2- 172.21.15.76:0/583101487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe4c101380 0x7fbe4c101760 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c009930 tx=0x7fbe3c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:57:49.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.858+0000 7fbe53898640 1 -- 172.21.15.76:0/583101487 shutdown_connections 2024-03-12T05:57:49.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.858+0000 7fbe53898640 1 --2- 172.21.15.76:0/583101487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe4c106450 0x7fbe4c10e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.858+0000 7fbe53898640 1 --2- 172.21.15.76:0/583101487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe4c101d30 0x7fbe4c105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.858+0000 7fbe53898640 1 --2- 172.21.15.76:0/583101487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe4c101380 0x7fbe4c101760 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.858+0000 7fbe53898640 1 -- 172.21.15.76:0/583101487 >> 172.21.15.76:0/583101487 conn(0x7fbe4c0fcc20 msgr2=0x7fbe4c0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:49.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.858+0000 7fbe53898640 1 -- 172.21.15.76:0/583101487 shutdown_connections 2024-03-12T05:57:49.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.858+0000 7fbe53898640 1 -- 172.21.15.76:0/583101487 wait complete. 2024-03-12T05:57:49.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe53898640 1 Processor -- start 2024-03-12T05:57:49.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe53898640 1 -- start start 2024-03-12T05:57:49.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe53898640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe4c101380 0x7fbe4c19d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe53898640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe4c106450 0x7fbe4c19db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe53898640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe4c197710 0x7fbe4c197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe53898640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c1981e0 con 0x7fbe4c106450 2024-03-12T05:57:49.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe53898640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c198350 con 0x7fbe4c101380 2024-03-12T05:57:49.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe53898640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c198510 con 0x7fbe4c197710 2024-03-12T05:57:49.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe50e33640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe4c106450 0x7fbe4c19db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe50e33640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe4c106450 0x7fbe4c19db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37896/0 (socket says 172.21.15.76:37896) 2024-03-12T05:57:49.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe50e33640 1 -- 172.21.15.76:0/3056741208 learned_addr learned my addr 172.21.15.76:0/3056741208 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:49.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe51634640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe4c101380 0x7fbe4c19d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe51e35640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe4c197710 0x7fbe4c197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe50e33640 1 -- 172.21.15.76:0/3056741208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe4c197710 msgr2=0x7fbe4c197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe50e33640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe4c197710 0x7fbe4c197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe50e33640 1 -- 172.21.15.76:0/3056741208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe4c101380 msgr2=0x7fbe4c19d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe50e33640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe4c101380 0x7fbe4c19d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe50e33640 1 -- 172.21.15.76:0/3056741208 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3c009560 con 0x7fbe4c106450 2024-03-12T05:57:49.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe51634640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe4c101380 0x7fbe4c19d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:57:49.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe51e35640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe4c197710 0x7fbe4c197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:57:49.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.859+0000 7fbe50e33640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe4c106450 0x7fbe4c19db40 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fbe4800b450 tx=0x7fbe4800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:49.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.860+0000 7fbe427fc640 1 -- 172.21.15.76:0/3056741208 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe48011890 con 0x7fbe4c106450 2024-03-12T05:57:49.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.860+0000 7fbe53898640 1 -- 172.21.15.76:0/3056741208 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4c1a4460 con 0x7fbe4c106450 2024-03-12T05:57:49.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.860+0000 7fbe427fc640 1 -- 172.21.15.76:0/3056741208 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe480103e0 con 0x7fbe4c106450 2024-03-12T05:57:49.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.860+0000 7fbe427fc640 1 -- 172.21.15.76:0/3056741208 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4800f580 con 0x7fbe4c106450 2024-03-12T05:57:49.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.860+0000 7fbe53898640 1 -- 172.21.15.76:0/3056741208 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4c1a4990 con 0x7fbe4c106450 2024-03-12T05:57:49.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.862+0000 7fbe427fc640 1 -- 172.21.15.76:0/3056741208 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fbe480108e0 con 0x7fbe4c106450 2024-03-12T05:57:49.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.862+0000 7fbe56460740 1 Processor -- start 2024-03-12T05:57:49.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.862+0000 7fbe56460740 1 -- start start 2024-03-12T05:57:49.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe56460740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe024c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe56460740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe0255210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe56460740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 0x564fe0257b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe56460740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe0207960 con 0x564fe024cb70 2024-03-12T05:57:49.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe56460740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe01bb310 con 0x564fe0255750 2024-03-12T05:57:49.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe56460740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe01b8840 con 0x564fe024c250 2024-03-12T05:57:49.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe427fc640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbe380a1ba0 0x7fbe380a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe51634640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe0255210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe51634640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe0255210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37908/0 (socket says 172.21.15.76:37908) 2024-03-12T05:57:49.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe51634640 1 -- 172.21.15.76:0/539809511 learned_addr learned my addr 172.21.15.76:0/539809511 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:49.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe51e35640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbe380a1ba0 0x7fbe380a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe51e35640 1 --2- 172.21.15.76:0/539809511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe024c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe427fc640 1 -- 172.21.15.76:0/3056741208 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbe480cd5a0 con 0x7fbe4c106450 2024-03-12T05:57:49.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe50e33640 1 --2- 172.21.15.76:0/539809511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 0x564fe0257b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe51634640 1 -- 172.21.15.76:0/539809511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 msgr2=0x564fe024c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe51634640 1 --2- 172.21.15.76:0/539809511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe024c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe51634640 1 -- 172.21.15.76:0/539809511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 msgr2=0x564fe0257b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe51634640 1 --2- 172.21.15.76:0/539809511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 0x564fe0257b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe51634640 1 -- 172.21.15.76:0/539809511 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564fe0176060 con 0x564fe024cb70 2024-03-12T05:57:49.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe51e35640 1 --2- 172.21.15.76:0/539809511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe024c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:49.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe50e33640 1 --2- 172.21.15.76:0/539809511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 0x564fe0257b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:49.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe51634640 1 --2- 172.21.15.76:0/539809511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe0255210 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c006010 tx=0x7fbe3c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=755b81453f9baf37 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:49.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.863+0000 7fbe51e35640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbe380a1ba0 0x7fbe380a4060 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009fd0 tx=0x7fbe44009530 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:49.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.864+0000 7fbe377fe640 1 -- 172.21.15.76:0/539809511 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c004b00 con 0x564fe024cb70 2024-03-12T05:57:49.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.864+0000 7fbe377fe640 1 -- 172.21.15.76:0/539809511 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe3c004030 con 0x564fe024cb70 2024-03-12T05:57:49.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.864+0000 7fbe377fe640 1 -- 172.21.15.76:0/539809511 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c005300 con 0x564fe024cb70 2024-03-12T05:57:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.864+0000 7fbe56460740 1 -- 172.21.15.76:0/539809511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 msgr2=0x564fe0255210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.864+0000 7fbe56460740 1 --2- 172.21.15.76:0/539809511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe0255210 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c006010 tx=0x7fbe3c005e80 comp rx=0 tx=0).stop 2024-03-12T05:57:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.864+0000 7fbe56460740 1 -- 172.21.15.76:0/539809511 shutdown_connections 2024-03-12T05:57:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.864+0000 7fbe56460740 1 --2- 172.21.15.76:0/539809511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 0x564fe0257b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.864+0000 7fbe56460740 1 --2- 172.21.15.76:0/539809511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe0255210 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.864+0000 7fbe56460740 1 --2- 172.21.15.76:0/539809511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe024c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.864+0000 7fbe56460740 1 -- 172.21.15.76:0/539809511 >> 172.21.15.76:0/539809511 conn(0x564fe0246980 msgr2=0x564fe0246d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:49.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.864+0000 7fbe56460740 1 -- 172.21.15.76:0/539809511 shutdown_connections 2024-03-12T05:57:49.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.864+0000 7fbe56460740 1 -- 172.21.15.76:0/539809511 wait complete. 2024-03-12T05:57:49.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe56460740 1 -- start start 2024-03-12T05:57:49.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe56460740 1 Processor -- start 2024-03-12T05:57:49.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe56460740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe0252770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe56460740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe0252cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe56460740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 0x564fe02531f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe56460740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe01b4020 con 0x564fe024cb70 2024-03-12T05:57:49.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe56460740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fdfe87430 con 0x564fe0255750 2024-03-12T05:57:49.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe56460740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fdfe43e00 con 0x564fe024c250 2024-03-12T05:57:49.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe56460740 10 client.0 _ll_register_callbacks cb 0x564fe015bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:49.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe50e33640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe0252cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe50e33640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe0252cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37914/0 (socket says 172.21.15.76:37914) 2024-03-12T05:57:49.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe50e33640 1 -- 172.21.15.76:0/3661613651 learned_addr learned my addr 172.21.15.76:0/3661613651 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:49.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe51634640 1 --2- 172.21.15.76:0/3661613651 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe0252770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe50e33640 1 -- 172.21.15.76:0/3661613651 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 msgr2=0x564fe0252770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe50e33640 1 --2- 172.21.15.76:0/3661613651 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe0252770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe51e35640 1 --2- 172.21.15.76:0/3661613651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 0x564fe02531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe50e33640 1 -- 172.21.15.76:0/3661613651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 msgr2=0x564fe02531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.865+0000 7fbe50e33640 1 --2- 172.21.15.76:0/3661613651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 0x564fe02531f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe50e33640 1 -- 172.21.15.76:0/3661613651 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4c10e830 con 0x564fe024cb70 2024-03-12T05:57:49.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe51634640 1 --2- 172.21.15.76:0/3661613651 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe0252770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:49.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe50e33640 1 --2- 172.21.15.76:0/3661613651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe0252cb0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fbe4800be10 tx=0x7fbe4809a9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77afda1d1573a665 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:49.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe34ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:49.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe34ff9640 1 -- 172.21.15.76:0/3661613651 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe48010760 con 0x564fe024cb70 2024-03-12T05:57:49.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe34ff9640 1 -- 172.21.15.76:0/3661613651 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbe4809e400 con 0x564fe024cb70 2024-03-12T05:57:49.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe56460740 1 -- 172.21.15.76:0/3661613651 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564fdfeb37e0 con 0x564fe024cb70 2024-03-12T05:57:49.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe34ff9640 1 -- 172.21.15.76:0/3661613651 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe480cb140 con 0x564fe024cb70 2024-03-12T05:57:49.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe56460740 10 client.16017 fetch_fsmap learned FSMap version 24 2024-03-12T05:57:49.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe56460740 1 -- 172.21.15.76:0/3661613651 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564fe01b26a0 con 0x564fe024cb70 2024-03-12T05:57:49.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe34ff9640 1 -- 172.21.15.76:0/3661613651 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564fdfeb37e0 con 0x564fe024cb70 2024-03-12T05:57:49.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe34ff9640 1 -- 172.21.15.76:0/3661613651 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbe4806c020 con 0x564fe024cb70 2024-03-12T05:57:49.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe56460740 10 client.16017 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:57:49.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe56460740 10 client.16017 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:57:49.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe56460740 20 client.16017 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:49.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe56460740 1 --2- 172.21.15.76:0/3661613651 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564fe0265b30 0x564fe0267f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe56460740 4 client.16017 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:57:49.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.866+0000 7fbe56460740 1 -- 172.21.15.76:0/3661613651 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564fdfef0720 con 0x564fe0265b30 2024-03-12T05:57:49.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.867+0000 7fbe51634640 1 --2- 172.21.15.76:0/3661613651 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564fe0265b30 0x564fe0267f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.867+0000 7fbe51634640 1 --2- 172.21.15.76:0/3661613651 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564fe0265b30 0x564fe0267f40 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7891fe480f541cbb server_cookie=bf07694403050acb in_seq=0 out_seq=0 2024-03-12T05:57:49.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.867+0000 7fbe34ff9640 10 client.16017 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:57:49.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe34ff9640 1 -- 172.21.15.76:0/3661613651 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564fdfef0720 con 0x564fe0265b30 2024-03-12T05:57:49.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe34ff9640 10 client.16017 handle_command_reply: tid=0 2024-03-12T05:57:49.992 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe56460740 1 client.16017 shutdown 2024-03-12T05:57:49.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe56460740 1 -- 172.21.15.76:0/3661613651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 msgr2=0x564fe0252cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe56460740 1 --2- 172.21.15.76:0/3661613651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe0252cb0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fbe4800be10 tx=0x7fbe4809a9b0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe56460740 1 -- 172.21.15.76:0/3661613651 shutdown_connections 2024-03-12T05:57:49.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe56460740 1 --2- 172.21.15.76:0/3661613651 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564fe0265b30 0x564fe0267f40 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe56460740 1 --2- 172.21.15.76:0/3661613651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 0x564fe02531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe56460740 1 --2- 172.21.15.76:0/3661613651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe0252cb0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe56460740 1 --2- 172.21.15.76:0/3661613651 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe0252770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.993 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe34ff9640 0 client.16017 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:57:49.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe56460740 1 -- 172.21.15.76:0/3661613651 >> 172.21.15.76:0/3661613651 conn(0x564fe0246980 msgr2=0x564fe02486d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:49.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe56460740 1 -- 172.21.15.76:0/3661613651 shutdown_connections 2024-03-12T05:57:49.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe56460740 1 -- 172.21.15.76:0/3661613651 wait complete. 2024-03-12T05:57:49.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.868+0000 7fbe56460740 20 client.16017 trim_cache size 0 max 16384 2024-03-12T05:57:49.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.879+0000 7fbe56460740 1 Processor -- start 2024-03-12T05:57:49.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.879+0000 7fbe56460740 1 -- start start 2024-03-12T05:57:49.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe56460740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe0253010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe56460740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe024cb70 0x564fe0245a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe56460740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe0255750 0x564fe025c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe56460740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe0139130 con 0x564fe0255750 2024-03-12T05:57:49.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe56460740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe0247980 con 0x564fe024cb70 2024-03-12T05:57:49.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe56460740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe026d190 con 0x564fe024c250 2024-03-12T05:57:49.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe51e35640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe0255750 0x564fe025c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe51e35640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe0255750 0x564fe025c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37918/0 (socket says 172.21.15.76:37918) 2024-03-12T05:57:49.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe51e35640 1 -- 172.21.15.76:0/3121404253 learned_addr learned my addr 172.21.15.76:0/3121404253 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:49.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe50e33640 1 --2- 172.21.15.76:0/3121404253 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe024cb70 0x564fe0245a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe51634640 1 --2- 172.21.15.76:0/3121404253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe0253010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe51e35640 1 -- 172.21.15.76:0/3121404253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 msgr2=0x564fe0253010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe51e35640 1 --2- 172.21.15.76:0/3121404253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe0253010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe51e35640 1 -- 172.21.15.76:0/3121404253 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe024cb70 msgr2=0x564fe0245a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe51e35640 1 --2- 172.21.15.76:0/3121404253 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe024cb70 0x564fe0245a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe51e35640 1 -- 172.21.15.76:0/3121404253 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564fe014ec20 con 0x564fe0255750 2024-03-12T05:57:49.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe51e35640 1 --2- 172.21.15.76:0/3121404253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe0255750 0x564fe025c5a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009810 tx=0x7fbe44005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385c12c568393ab3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:49.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe36ffd640 1 -- 172.21.15.76:0/3121404253 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe44014ea0 con 0x564fe0255750 2024-03-12T05:57:49.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe36ffd640 1 -- 172.21.15.76:0/3121404253 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe440142f0 con 0x564fe0255750 2024-03-12T05:57:49.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.880+0000 7fbe36ffd640 1 -- 172.21.15.76:0/3121404253 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe44029a70 con 0x564fe0255750 2024-03-12T05:57:49.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.881+0000 7fbe56460740 1 -- 172.21.15.76:0/3121404253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe0255750 msgr2=0x564fe025c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:49.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.881+0000 7fbe56460740 1 --2- 172.21.15.76:0/3121404253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe0255750 0x564fe025c5a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009810 tx=0x7fbe44005bd0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.881+0000 7fbe56460740 1 -- 172.21.15.76:0/3121404253 shutdown_connections 2024-03-12T05:57:49.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.881+0000 7fbe56460740 1 --2- 172.21.15.76:0/3121404253 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe0255750 0x564fe025c5a0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.881+0000 7fbe56460740 1 --2- 172.21.15.76:0/3121404253 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe024cb70 0x564fe0245a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.881+0000 7fbe56460740 1 --2- 172.21.15.76:0/3121404253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe0253010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.881+0000 7fbe56460740 1 -- 172.21.15.76:0/3121404253 >> 172.21.15.76:0/3121404253 conn(0x564fe0246980 msgr2=0x564fe02452f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:49.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.881+0000 7fbe56460740 1 -- 172.21.15.76:0/3121404253 shutdown_connections 2024-03-12T05:57:49.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.881+0000 7fbe56460740 1 -- 172.21.15.76:0/3121404253 wait complete. 2024-03-12T05:57:49.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.881+0000 7fbe56460740 1 -- start start 2024-03-12T05:57:49.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.881+0000 7fbe56460740 1 Processor -- start 2024-03-12T05:57:49.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe56460740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe025ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe56460740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe025bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe56460740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 0x564fe02b5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:49.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe56460740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe02b5570 con 0x564fe024cb70 2024-03-12T05:57:49.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe56460740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe02b56e0 con 0x564fe0255750 2024-03-12T05:57:49.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe56460740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fe02b58a0 con 0x564fe024c250 2024-03-12T05:57:49.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe56460740 10 client.0 _ll_register_callbacks cb 0x7fbe3c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:49.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe50e33640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe025bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe50e33640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe025bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37924/0 (socket says 172.21.15.76:37924) 2024-03-12T05:57:49.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe50e33640 1 -- 172.21.15.76:0/4294483589 learned_addr learned my addr 172.21.15.76:0/4294483589 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:49.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe51634640 1 --2- 172.21.15.76:0/4294483589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe025ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe51e35640 1 --2- 172.21.15.76:0/4294483589 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 0x564fe02b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:49.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe50e33640 1 -- 172.21.15.76:0/4294483589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 msgr2=0x564fe025ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:49.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe50e33640 1 --2- 172.21.15.76:0/4294483589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe025ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:49.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe50e33640 1 -- 172.21.15.76:0/4294483589 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 msgr2=0x564fe02b5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe50e33640 1 --2- 172.21.15.76:0/4294483589 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 0x564fe02b5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe50e33640 1 -- 172.21.15.76:0/4294483589 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe44019070 con 0x564fe024cb70 2024-03-12T05:57:50.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe50e33640 1 --2- 172.21.15.76:0/4294483589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe025bf50 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x564fe0251fd0 tx=0x7fbe48011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e07a919b9c0a6fa server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:50.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe34ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:50.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe34ff9640 1 -- 172.21.15.76:0/4294483589 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe48099900 con 0x564fe024cb70 2024-03-12T05:57:50.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe34ff9640 1 -- 172.21.15.76:0/4294483589 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbe4809e400 con 0x564fe024cb70 2024-03-12T05:57:50.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.882+0000 7fbe56460740 1 -- 172.21.15.76:0/4294483589 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564fdff3f890 con 0x564fe024cb70 2024-03-12T05:57:50.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.883+0000 7fbe34ff9640 1 -- 172.21.15.76:0/4294483589 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe48099a60 con 0x564fe024cb70 2024-03-12T05:57:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.883+0000 7fbe56460740 10 client.16023 fetch_fsmap learned FSMap version 24 2024-03-12T05:57:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.883+0000 7fbe56460740 1 -- 172.21.15.76:0/4294483589 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564fe01748c0 con 0x564fe024cb70 2024-03-12T05:57:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.883+0000 7fbe34ff9640 1 -- 172.21.15.76:0/4294483589 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564fdff3f890 con 0x564fe024cb70 2024-03-12T05:57:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.883+0000 7fbe34ff9640 1 -- 172.21.15.76:0/4294483589 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbe4809f320 con 0x564fe024cb70 2024-03-12T05:57:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.883+0000 7fbe56460740 10 client.16023 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:57:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.883+0000 7fbe56460740 10 client.16023 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:57:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.883+0000 7fbe56460740 20 client.16023 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.883+0000 7fbe56460740 1 --2- 172.21.15.76:0/4294483589 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564fe02be020 0x564fe02c0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.883+0000 7fbe56460740 4 client.16023 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:57:50.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.883+0000 7fbe56460740 1 -- 172.21.15.76:0/4294483589 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564fdfe5f700 con 0x564fe02be020 2024-03-12T05:57:50.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.883+0000 7fbe51634640 1 --2- 172.21.15.76:0/4294483589 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564fe02be020 0x564fe02c0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.883+0000 7fbe34ff9640 10 client.16023 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:57:50.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.883+0000 7fbe51634640 1 --2- 172.21.15.76:0/4294483589 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564fe02be020 0x564fe02c0440 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7abf7fbdf9baed13 server_cookie=bad44b87a4073a25 in_seq=0 out_seq=0 2024-03-12T05:57:50.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe34ff9640 1 -- 172.21.15.76:0/4294483589 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564fdfe5f700 con 0x564fe02be020 2024-03-12T05:57:50.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe34ff9640 10 client.16023 handle_command_reply: tid=0 2024-03-12T05:57:50.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe56460740 1 client.16023 shutdown 2024-03-12T05:57:50.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe56460740 1 -- 172.21.15.76:0/4294483589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 msgr2=0x564fe025bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe56460740 1 --2- 172.21.15.76:0/4294483589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe025bf50 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x564fe0251fd0 tx=0x7fbe48011000 comp rx=0 tx=0).stop 2024-03-12T05:57:50.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe56460740 1 -- 172.21.15.76:0/4294483589 shutdown_connections 2024-03-12T05:57:50.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe56460740 1 --2- 172.21.15.76:0/4294483589 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564fe02be020 0x564fe02c0440 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe56460740 1 --2- 172.21.15.76:0/4294483589 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564fe0255750 0x564fe02b5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe34ff9640 0 client.16023 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:57:50.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe56460740 1 --2- 172.21.15.76:0/4294483589 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564fe024cb70 0x564fe025bf50 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe56460740 1 --2- 172.21.15.76:0/4294483589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fe024c250 0x564fe025ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe56460740 1 -- 172.21.15.76:0/4294483589 >> 172.21.15.76:0/4294483589 conn(0x564fe0246980 msgr2=0x564fe0245b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:50.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe56460740 1 -- 172.21.15.76:0/4294483589 shutdown_connections 2024-03-12T05:57:50.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe56460740 1 -- 172.21.15.76:0/4294483589 wait complete. 2024-03-12T05:57:50.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe56460740 20 client.16023 trim_cache size 0 max 16384 2024-03-12T05:57:50.004 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:57:50.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe53898640 1 -- 172.21.15.76:0/3056741208 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbe380a1ba0 msgr2=0x7fbe380a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe53898640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbe380a1ba0 0x7fbe380a4060 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009fd0 tx=0x7fbe44009530 comp rx=0 tx=0).stop 2024-03-12T05:57:50.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe53898640 1 -- 172.21.15.76:0/3056741208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe4c106450 msgr2=0x7fbe4c19db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe53898640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe4c106450 0x7fbe4c19db40 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fbe4800b450 tx=0x7fbe4800b910 comp rx=0 tx=0).stop 2024-03-12T05:57:50.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe53898640 1 -- 172.21.15.76:0/3056741208 shutdown_connections 2024-03-12T05:57:50.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe53898640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbe380a1ba0 0x7fbe380a4060 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe53898640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe4c197710 0x7fbe4c197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe53898640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe4c106450 0x7fbe4c19db40 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe53898640 1 --2- 172.21.15.76:0/3056741208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe4c101380 0x7fbe4c19d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.884+0000 7fbe53898640 1 -- 172.21.15.76:0/3056741208 >> 172.21.15.76:0/3056741208 conn(0x7fbe4c0fcc20 msgr2=0x7fbe4c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:50.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.885+0000 7fbe53898640 1 -- 172.21.15.76:0/3056741208 shutdown_connections 2024-03-12T05:57:50.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:49.885+0000 7fbe53898640 1 -- 172.21.15.76:0/3056741208 wait complete. 2024-03-12T05:57:50.008 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:57:50.008 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:57:50.035 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-12T05:57:50.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.109+0000 7f81f6062640 1 -- 172.21.15.76:0/2786243172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81f0110e60 msgr2=0x7f81f0118a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.109+0000 7f81f6062640 1 --2- 172.21.15.76:0/2786243172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81f0110e60 0x7f81f0118a30 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009930 tx=0x7f81e802bc30 comp rx=0 tx=0).stop 2024-03-12T05:57:50.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.109+0000 7f81f6062640 1 -- 172.21.15.76:0/2786243172 shutdown_connections 2024-03-12T05:57:50.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.109+0000 7f81f6062640 1 --2- 172.21.15.76:0/2786243172 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81f0110e60 0x7f81f0118a30 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.109+0000 7f81f6062640 1 --2- 172.21.15.76:0/2786243172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81f010c750 0x7f81f0110780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.109+0000 7f81f6062640 1 --2- 172.21.15.76:0/2786243172 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81f010bda0 0x7f81f010c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.109+0000 7f81f6062640 1 -- 172.21.15.76:0/2786243172 >> 172.21.15.76:0/2786243172 conn(0x7f81f006c2c0 msgr2=0x7f81f010a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:50.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.109+0000 7f81f6062640 1 -- 172.21.15.76:0/2786243172 shutdown_connections 2024-03-12T05:57:50.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.109+0000 7f81f6062640 1 -- 172.21.15.76:0/2786243172 wait complete. 2024-03-12T05:57:50.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.110+0000 7f81f6062640 1 Processor -- start 2024-03-12T05:57:50.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.110+0000 7f81f6062640 1 -- start start 2024-03-12T05:57:50.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.110+0000 7f81f6062640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81f010bda0 0x7f81f01a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.110+0000 7f81f6062640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81f0110e60 0x7f81f01a2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.110+0000 7f81f6062640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81f01a6a50 0x7f81f01a6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.110+0000 7f81f6062640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f01a7440 con 0x7f81f01a6a50 2024-03-12T05:57:50.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.110+0000 7f81f6062640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f01a75b0 con 0x7f81f010bda0 2024-03-12T05:57:50.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.110+0000 7f81f6062640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f01a7720 con 0x7f81f0110e60 2024-03-12T05:57:50.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81effff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81f01a6a50 0x7f81f01a6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81ef7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81f010bda0 0x7f81f01a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81effff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81f01a6a50 0x7f81f01a6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37940/0 (socket says 172.21.15.76:37940) 2024-03-12T05:57:50.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81ef7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81f010bda0 0x7f81f01a21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:33054/0 (socket says 172.21.15.76:33054) 2024-03-12T05:57:50.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81effff640 1 -- 172.21.15.76:0/3538900168 learned_addr learned my addr 172.21.15.76:0/3538900168 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:50.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81eeffd640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81f0110e60 0x7f81f01a2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81effff640 1 -- 172.21.15.76:0/3538900168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81f0110e60 msgr2=0x7f81f01a2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81effff640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81f0110e60 0x7f81f01a2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81effff640 1 -- 172.21.15.76:0/3538900168 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81f010bda0 msgr2=0x7f81f01a21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81effff640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81f010bda0 0x7f81f01a21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81effff640 1 -- 172.21.15.76:0/3538900168 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e8009560 con 0x7f81f01a6a50 2024-03-12T05:57:50.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81eeffd640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81f0110e60 0x7f81f01a2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:50.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81ef7fe640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81f010bda0 0x7f81f01a21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:57:50.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81effff640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81f01a6a50 0x7f81f01a6f00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f81e8005940 tx=0x7f81e802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:50.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81ecff9640 1 -- 172.21.15.76:0/3538900168 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e80042d0 con 0x7f81f01a6a50 2024-03-12T05:57:50.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81ecff9640 1 -- 172.21.15.76:0/3538900168 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81e8004430 con 0x7f81f01a6a50 2024-03-12T05:57:50.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81f6062640 1 -- 172.21.15.76:0/3538900168 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f01aed00 con 0x7f81f01a6a50 2024-03-12T05:57:50.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81ecff9640 1 -- 172.21.15.76:0/3538900168 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e8003b50 con 0x7f81f01a6a50 2024-03-12T05:57:50.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.111+0000 7f81f6062640 1 -- 172.21.15.76:0/3538900168 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f01af180 con 0x7f81f01a6a50 2024-03-12T05:57:50.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.113+0000 7f81ecff9640 1 -- 172.21.15.76:0/3538900168 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f81e8005460 con 0x7f81f01a6a50 2024-03-12T05:57:50.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81f8c31740 1 Processor -- start 2024-03-12T05:57:50.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81f8c31740 1 -- start start 2024-03-12T05:57:50.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81f8c31740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8250 0x5575d8fa8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81f8c31740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8b70 0x5575d8fb1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81f8c31740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fb1750 0x5575d8fb3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81f8c31740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575d8f63960 con 0x5575d8fa8b70 2024-03-12T05:57:50.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81f8c31740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575d8f17310 con 0x5575d8fb1750 2024-03-12T05:57:50.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81f8c31740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575d8f14840 con 0x5575d8fa8250 2024-03-12T05:57:50.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81ecff9640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f81dc0a1ba0 0x7f81dc0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81ef7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8b70 0x5575d8fb1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81ef7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8b70 0x5575d8fb1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37942/0 (socket says 172.21.15.76:37942) 2024-03-12T05:57:50.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81ef7fe640 1 -- 172.21.15.76:0/2412724426 learned_addr learned my addr 172.21.15.76:0/2412724426 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:50.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81effff640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f81dc0a1ba0 0x7f81dc0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81eeffd640 1 --2- 172.21.15.76:0/2412724426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8250 0x5575d8fa8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81eeffd640 1 --2- 172.21.15.76:0/2412724426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fb1750 0x5575d8fb3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81ecff9640 1 -- 172.21.15.76:0/3538900168 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f81e80ef870 con 0x7f81f01a6a50 2024-03-12T05:57:50.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81ef7fe640 1 -- 172.21.15.76:0/2412724426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8250 msgr2=0x5575d8fa8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81ef7fe640 1 --2- 172.21.15.76:0/2412724426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8250 0x5575d8fa8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81ef7fe640 1 -- 172.21.15.76:0/2412724426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fb1750 msgr2=0x5575d8fb3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81ef7fe640 1 --2- 172.21.15.76:0/2412724426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fb1750 0x5575d8fb3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.114+0000 7f81ef7fe640 1 -- 172.21.15.76:0/2412724426 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575d8ed2060 con 0x5575d8fa8b70 2024-03-12T05:57:50.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.115+0000 7f81effff640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f81dc0a1ba0 0x7f81dc0a4060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f81e80ed5f0 tx=0x7f81e80a7350 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:50.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.115+0000 7f81ef7fe640 1 --2- 172.21.15.76:0/2412724426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8b70 0x5575d8fb1210 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f81e0005f40 tx=0x7f81e0011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b3e203b0a7bc26 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:50.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.115+0000 7f81e5ffb640 1 -- 172.21.15.76:0/2412724426 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e0011330 con 0x5575d8fa8b70 2024-03-12T05:57:50.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.115+0000 7f81e5ffb640 1 -- 172.21.15.76:0/2412724426 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81e000fe00 con 0x5575d8fa8b70 2024-03-12T05:57:50.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.115+0000 7f81e5ffb640 1 -- 172.21.15.76:0/2412724426 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e00193e0 con 0x5575d8fa8b70 2024-03-12T05:57:50.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.115+0000 7f81f8c31740 1 -- 172.21.15.76:0/2412724426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8b70 msgr2=0x5575d8fb1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.115+0000 7f81f8c31740 1 --2- 172.21.15.76:0/2412724426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8b70 0x5575d8fb1210 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f81e0005f40 tx=0x7f81e0011d30 comp rx=0 tx=0).stop 2024-03-12T05:57:50.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.115+0000 7f81f8c31740 1 -- 172.21.15.76:0/2412724426 shutdown_connections 2024-03-12T05:57:50.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.115+0000 7f81f8c31740 1 --2- 172.21.15.76:0/2412724426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fb1750 0x5575d8fb3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.115+0000 7f81f8c31740 1 --2- 172.21.15.76:0/2412724426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8b70 0x5575d8fb1210 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.115+0000 7f81f8c31740 1 --2- 172.21.15.76:0/2412724426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8250 0x5575d8fa8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.115+0000 7f81f8c31740 1 -- 172.21.15.76:0/2412724426 >> 172.21.15.76:0/2412724426 conn(0x5575d8fa2980 msgr2=0x5575d8fa2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:50.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.115+0000 7f81f8c31740 1 -- 172.21.15.76:0/2412724426 shutdown_connections 2024-03-12T05:57:50.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.115+0000 7f81f8c31740 1 -- 172.21.15.76:0/2412724426 wait complete. 2024-03-12T05:57:50.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.116+0000 7f81f8c31740 1 -- start start 2024-03-12T05:57:50.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.116+0000 7f81f8c31740 1 Processor -- start 2024-03-12T05:57:50.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.116+0000 7f81f8c31740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8250 0x5575d8fae760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.116+0000 7f81f8c31740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8b70 0x5575d8faeca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.116+0000 7f81f8c31740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fb1750 0x5575d8faf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.116+0000 7f81f8c31740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575d8f10020 con 0x5575d8fa8250 2024-03-12T05:57:50.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.116+0000 7f81f8c31740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575d8be3430 con 0x5575d8fb1750 2024-03-12T05:57:50.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.116+0000 7f81f8c31740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575d8b9fe00 con 0x5575d8fa8b70 2024-03-12T05:57:50.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.116+0000 7f81f8c31740 10 client.0 _ll_register_callbacks cb 0x5575d8eb7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:50.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.116+0000 7f81eeffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8250 0x5575d8fae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.116+0000 7f81eeffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8250 0x5575d8fae760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37950/0 (socket says 172.21.15.76:37950) 2024-03-12T05:57:50.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.116+0000 7f81eeffd640 1 -- 172.21.15.76:0/3756398442 learned_addr learned my addr 172.21.15.76:0/3756398442 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:50.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.116+0000 7f81ef7fe640 1 --2- 172.21.15.76:0/3756398442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8b70 0x5575d8faeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81eeffd640 1 -- 172.21.15.76:0/3756398442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8b70 msgr2=0x5575d8faeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81eeffd640 1 --2- 172.21.15.76:0/3756398442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8b70 0x5575d8faeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81eeffd640 1 -- 172.21.15.76:0/3756398442 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fb1750 msgr2=0x5575d8faf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81eeffd640 1 --2- 172.21.15.76:0/3756398442 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fb1750 0x5575d8faf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81eeffd640 1 -- 172.21.15.76:0/3756398442 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e0011500 con 0x5575d8fa8250 2024-03-12T05:57:50.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81eeffd640 1 --2- 172.21.15.76:0/3756398442 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8250 0x5575d8fae760 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f81d800bae0 tx=0x7f81d8003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f55c17851076cc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:50.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:50.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81ceffd640 1 -- 172.21.15.76:0/3756398442 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8014310 con 0x5575d8fa8250 2024-03-12T05:57:50.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81ceffd640 1 -- 172.21.15.76:0/3756398442 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f81d8014b60 con 0x5575d8fa8250 2024-03-12T05:57:50.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81f8c31740 1 -- 172.21.15.76:0/3756398442 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5575d8c0f7e0 con 0x5575d8fa8250 2024-03-12T05:57:50.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81ceffd640 1 -- 172.21.15.76:0/3756398442 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81d8014d20 con 0x5575d8fa8250 2024-03-12T05:57:50.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81f8c31740 10 client.16035 fetch_fsmap learned FSMap version 24 2024-03-12T05:57:50.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81f8c31740 1 -- 172.21.15.76:0/3756398442 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5575d8f0e6a0 con 0x5575d8fa8250 2024-03-12T05:57:50.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81ceffd640 1 -- 172.21.15.76:0/3756398442 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5575d8c0f7e0 con 0x5575d8fa8250 2024-03-12T05:57:50.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81ceffd640 1 -- 172.21.15.76:0/3756398442 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f81d8011500 con 0x5575d8fa8250 2024-03-12T05:57:50.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81f8c31740 10 client.16035 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:57:50.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81f8c31740 10 client.16035 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:57:50.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81f8c31740 20 client.16035 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:50.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81f8c31740 1 --2- 172.21.15.76:0/3756398442 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5575d8fc1b90 0x5575d8fc3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81f8c31740 4 client.16035 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:57:50.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.117+0000 7f81f8c31740 1 -- 172.21.15.76:0/3756398442 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5575d8c4c720 con 0x5575d8fc1b90 2024-03-12T05:57:50.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.118+0000 7f81ef7fe640 1 --2- 172.21.15.76:0/3756398442 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5575d8fc1b90 0x5575d8fc3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.118+0000 7f81ceffd640 10 client.16035 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:57:50.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.118+0000 7f81ef7fe640 1 --2- 172.21.15.76:0/3756398442 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5575d8fc1b90 0x5575d8fc3fb0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a3efa61d054990bb server_cookie=799b0447fa01d9cb in_seq=0 out_seq=0 2024-03-12T05:57:50.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81ceffd640 1 -- 172.21.15.76:0/3756398442 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5575d8c4c720 con 0x5575d8fc1b90 2024-03-12T05:57:50.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81ceffd640 10 client.16035 handle_command_reply: tid=0 2024-03-12T05:57:50.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81f8c31740 1 client.16035 shutdown 2024-03-12T05:57:50.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81f8c31740 1 -- 172.21.15.76:0/3756398442 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8250 msgr2=0x5575d8fae760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81f8c31740 1 --2- 172.21.15.76:0/3756398442 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8250 0x5575d8fae760 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f81d800bae0 tx=0x7f81d8003d00 comp rx=0 tx=0).stop 2024-03-12T05:57:50.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81f8c31740 1 -- 172.21.15.76:0/3756398442 shutdown_connections 2024-03-12T05:57:50.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81f8c31740 1 --2- 172.21.15.76:0/3756398442 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5575d8fc1b90 0x5575d8fc3fb0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81f8c31740 1 --2- 172.21.15.76:0/3756398442 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fb1750 0x5575d8faf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81f8c31740 1 --2- 172.21.15.76:0/3756398442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8b70 0x5575d8faeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81ceffd640 0 client.16035 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:57:50.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81f8c31740 1 --2- 172.21.15.76:0/3756398442 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8250 0x5575d8fae760 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81f8c31740 1 -- 172.21.15.76:0/3756398442 >> 172.21.15.76:0/3756398442 conn(0x5575d8fa2980 msgr2=0x5575d8fa46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:50.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81f8c31740 1 -- 172.21.15.76:0/3756398442 shutdown_connections 2024-03-12T05:57:50.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81f8c31740 1 -- 172.21.15.76:0/3756398442 wait complete. 2024-03-12T05:57:50.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.119+0000 7f81f8c31740 20 client.16035 trim_cache size 0 max 16384 2024-03-12T05:57:50.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.131+0000 7f81f8c31740 1 Processor -- start 2024-03-12T05:57:50.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.131+0000 7f81f8c31740 1 -- start start 2024-03-12T05:57:50.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.131+0000 7f81f8c31740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8250 0x5575d8fb0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.131+0000 7f81f8c31740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8b70 0x5575d8fa9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.131+0000 7f81f8c31740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fb1750 0x5575d8fa9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.131+0000 7f81f8c31740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575d8e95130 con 0x5575d8fa8250 2024-03-12T05:57:50.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.131+0000 7f81f8c31740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575d8c0f7e0 con 0x5575d8fb1750 2024-03-12T05:57:50.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.131+0000 7f81f8c31740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575d8fbe690 con 0x5575d8fa8b70 2024-03-12T05:57:50.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.131+0000 7f81eeffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8250 0x5575d8fb0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.131+0000 7f81eeffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8250 0x5575d8fb0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37964/0 (socket says 172.21.15.76:37964) 2024-03-12T05:57:50.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.131+0000 7f81eeffd640 1 -- 172.21.15.76:0/3326710287 learned_addr learned my addr 172.21.15.76:0/3326710287 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:50.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.131+0000 7f81ef7fe640 1 --2- 172.21.15.76:0/3326710287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8b70 0x5575d8fa9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.132+0000 7f81eeffd640 1 --2- 172.21.15.76:0/3326710287 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fb1750 0x5575d8fa9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.132+0000 7f81eeffd640 1 -- 172.21.15.76:0/3326710287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8b70 msgr2=0x5575d8fa9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.132+0000 7f81eeffd640 1 --2- 172.21.15.76:0/3326710287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8b70 0x5575d8fa9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.132+0000 7f81eeffd640 1 -- 172.21.15.76:0/3326710287 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fb1750 msgr2=0x5575d8fa9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.132+0000 7f81eeffd640 1 --2- 172.21.15.76:0/3326710287 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fb1750 0x5575d8fa9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.132+0000 7f81eeffd640 1 -- 172.21.15.76:0/3326710287 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575d8eaac20 con 0x5575d8fa8250 2024-03-12T05:57:50.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.132+0000 7f81eeffd640 1 --2- 172.21.15.76:0/3326710287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8250 0x5575d8fb0e90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f81d8011960 tx=0x7f81d8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb51a9ce6f83951e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:50.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.132+0000 7f81e57fa640 1 -- 172.21.15.76:0/3326710287 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8014110 con 0x5575d8fa8250 2024-03-12T05:57:50.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.132+0000 7f81e57fa640 1 -- 172.21.15.76:0/3326710287 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81d801a070 con 0x5575d8fa8250 2024-03-12T05:57:50.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.132+0000 7f81e57fa640 1 -- 172.21.15.76:0/3326710287 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8014dc0 con 0x5575d8fa8250 2024-03-12T05:57:50.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 -- 172.21.15.76:0/3326710287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8250 msgr2=0x5575d8fb0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 --2- 172.21.15.76:0/3326710287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8250 0x5575d8fb0e90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f81d8011960 tx=0x7f81d8005740 comp rx=0 tx=0).stop 2024-03-12T05:57:50.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 -- 172.21.15.76:0/3326710287 shutdown_connections 2024-03-12T05:57:50.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 --2- 172.21.15.76:0/3326710287 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fb1750 0x5575d8fa9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 --2- 172.21.15.76:0/3326710287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fa8b70 0x5575d8fa9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 --2- 172.21.15.76:0/3326710287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8250 0x5575d8fb0e90 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 -- 172.21.15.76:0/3326710287 >> 172.21.15.76:0/3326710287 conn(0x5575d8fa2980 msgr2=0x5575d8f9da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:50.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 -- 172.21.15.76:0/3326710287 shutdown_connections 2024-03-12T05:57:50.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 -- 172.21.15.76:0/3326710287 wait complete. 2024-03-12T05:57:50.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 -- start start 2024-03-12T05:57:50.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 Processor -- start 2024-03-12T05:57:50.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fa8250 0x5575d8fb7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8b70 0x5575d8fb79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fb1750 0x5575d90180e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575d8fb8250 con 0x5575d8fa8b70 2024-03-12T05:57:50.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.133+0000 7f81f8c31740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575d9018620 con 0x5575d8fa8250 2024-03-12T05:57:50.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81f8c31740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575d9018760 con 0x5575d8fb1750 2024-03-12T05:57:50.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81f8c31740 10 client.0 _ll_register_callbacks cb 0x7f81e0009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:50.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81ef7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8b70 0x5575d8fb79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81ef7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8b70 0x5575d8fb79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37980/0 (socket says 172.21.15.76:37980) 2024-03-12T05:57:50.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81ef7fe640 1 -- 172.21.15.76:0/2032610271 learned_addr learned my addr 172.21.15.76:0/2032610271 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:50.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81eeffd640 1 --2- 172.21.15.76:0/2032610271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fb1750 0x5575d90180e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81ef7fe640 1 -- 172.21.15.76:0/2032610271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fb1750 msgr2=0x5575d90180e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81ef7fe640 1 --2- 172.21.15.76:0/2032610271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fb1750 0x5575d90180e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81ef7fe640 1 -- 172.21.15.76:0/2032610271 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fa8250 msgr2=0x5575d8fb7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81ef7fe640 1 --2- 172.21.15.76:0/2032610271 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fa8250 0x5575d8fb7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81ef7fe640 1 -- 172.21.15.76:0/2032610271 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5575d8f0e6a0 con 0x5575d8fa8b70 2024-03-12T05:57:50.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81ef7fe640 1 --2- 172.21.15.76:0/2032610271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8b70 0x5575d8fb79a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f81e00117f0 tx=0x7f81e00113a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a2e9c972bd80e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:50.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:50.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81ceffd640 1 -- 172.21.15.76:0/2032610271 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e000f920 con 0x5575d8fa8b70 2024-03-12T05:57:50.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81f8c31740 1 -- 172.21.15.76:0/2032610271 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5575d8fae590 con 0x5575d8fa8b70 2024-03-12T05:57:50.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81ceffd640 1 -- 172.21.15.76:0/2032610271 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f81e000fa80 con 0x5575d8fa8b70 2024-03-12T05:57:50.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81ceffd640 1 -- 172.21.15.76:0/2032610271 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81e000fc40 con 0x5575d8fa8b70 2024-03-12T05:57:50.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81f8c31740 10 client.16041 fetch_fsmap learned FSMap version 24 2024-03-12T05:57:50.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81f8c31740 1 -- 172.21.15.76:0/2032610271 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5575d8ed08c0 con 0x5575d8fa8b70 2024-03-12T05:57:50.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.134+0000 7f81ceffd640 1 -- 172.21.15.76:0/2032610271 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5575d8fae590 con 0x5575d8fa8b70 2024-03-12T05:57:50.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.135+0000 7f81ceffd640 1 -- 172.21.15.76:0/2032610271 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f81e00290b0 con 0x5575d8fa8b70 2024-03-12T05:57:50.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.135+0000 7f81f8c31740 10 client.16041 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:57:50.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.135+0000 7f81f8c31740 10 client.16041 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:57:50.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.135+0000 7f81f8c31740 20 client.16041 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:50.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.135+0000 7f81f8c31740 1 --2- 172.21.15.76:0/2032610271 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5575d9019f90 0x5575d901c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.135+0000 7f81f8c31740 4 client.16041 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:57:50.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.135+0000 7f81f8c31740 1 -- 172.21.15.76:0/2032610271 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f81e000fe00 con 0x5575d9019f90 2024-03-12T05:57:50.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.135+0000 7f81eeffd640 1 --2- 172.21.15.76:0/2032610271 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5575d9019f90 0x5575d901c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.135+0000 7f81eeffd640 1 --2- 172.21.15.76:0/2032610271 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5575d9019f90 0x5575d901c3b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cac5462e8ff3756c server_cookie=d86cf42ed0761113 in_seq=0 out_seq=0 2024-03-12T05:57:50.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.135+0000 7f81ceffd640 10 client.16041 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:57:50.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.135+0000 7f81ceffd640 1 -- 172.21.15.76:0/2032610271 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f81e000fe00 con 0x5575d9019f90 2024-03-12T05:57:50.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.135+0000 7f81ceffd640 10 client.16041 handle_command_reply: tid=0 2024-03-12T05:57:50.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.135+0000 7f81f8c31740 1 client.16041 shutdown 2024-03-12T05:57:50.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f8c31740 1 -- 172.21.15.76:0/2032610271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8b70 msgr2=0x5575d8fb79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f8c31740 1 --2- 172.21.15.76:0/2032610271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8b70 0x5575d8fb79a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f81e00117f0 tx=0x7f81e00113a0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f8c31740 1 -- 172.21.15.76:0/2032610271 shutdown_connections 2024-03-12T05:57:50.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f8c31740 1 --2- 172.21.15.76:0/2032610271 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5575d9019f90 0x5575d901c3b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f8c31740 1 --2- 172.21.15.76:0/2032610271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575d8fb1750 0x5575d90180e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f8c31740 1 --2- 172.21.15.76:0/2032610271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5575d8fa8b70 0x5575d8fb79a0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81ceffd640 0 client.16041 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:57:50.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f8c31740 1 --2- 172.21.15.76:0/2032610271 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5575d8fa8250 0x5575d8fb7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f8c31740 1 -- 172.21.15.76:0/2032610271 >> 172.21.15.76:0/2032610271 conn(0x5575d8fa2980 msgr2=0x5575d8fa9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:50.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f8c31740 1 -- 172.21.15.76:0/2032610271 shutdown_connections 2024-03-12T05:57:50.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f8c31740 1 -- 172.21.15.76:0/2032610271 wait complete. 2024-03-12T05:57:50.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f8c31740 20 client.16041 trim_cache size 0 max 16384 2024-03-12T05:57:50.158 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:57:50.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f6062640 1 -- 172.21.15.76:0/3538900168 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f81dc0a1ba0 msgr2=0x7f81dc0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f6062640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f81dc0a1ba0 0x7f81dc0a4060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f81e80ed5f0 tx=0x7f81e80a7350 comp rx=0 tx=0).stop 2024-03-12T05:57:50.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f6062640 1 -- 172.21.15.76:0/3538900168 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81f01a6a50 msgr2=0x7f81f01a6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f6062640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81f01a6a50 0x7f81f01a6f00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f81e8005940 tx=0x7f81e802c5d0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f6062640 1 -- 172.21.15.76:0/3538900168 shutdown_connections 2024-03-12T05:57:50.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f6062640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f81dc0a1ba0 0x7f81dc0a4060 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f6062640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81f01a6a50 0x7f81f01a6f00 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f6062640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81f0110e60 0x7f81f01a2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f6062640 1 --2- 172.21.15.76:0/3538900168 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f81f010bda0 0x7f81f01a21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f6062640 1 -- 172.21.15.76:0/3538900168 >> 172.21.15.76:0/3538900168 conn(0x7f81f006c2c0 msgr2=0x7f81f010f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f6062640 1 -- 172.21.15.76:0/3538900168 shutdown_connections 2024-03-12T05:57:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.136+0000 7f81f6062640 1 -- 172.21.15.76:0/3538900168 wait complete. 2024-03-12T05:57:50.161 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:57:50.162 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:57:50.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.299+0000 7f5870eb5640 1 -- 172.21.15.76:0/2473759611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f586c102390 msgr2=0x7f586c102770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.299+0000 7f5870eb5640 1 --2- 172.21.15.76:0/2473759611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f586c102390 0x7f586c102770 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f585c009900 tx=0x7f585c02bcd0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 -- 172.21.15.76:0/2473759611 shutdown_connections 2024-03-12T05:57:50.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 --2- 172.21.15.76:0/2473759611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f586c10b760 0x7f586c10dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 --2- 172.21.15.76:0/2473759611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f586c102cb0 0x7f586c10b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 --2- 172.21.15.76:0/2473759611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f586c102390 0x7f586c102770 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 -- 172.21.15.76:0/2473759611 >> 172.21.15.76:0/2473759611 conn(0x7f586c077250 msgr2=0x7f586c077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:50.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 -- 172.21.15.76:0/2473759611 shutdown_connections 2024-03-12T05:57:50.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 -- 172.21.15.76:0/2473759611 wait complete. 2024-03-12T05:57:50.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 Processor -- start 2024-03-12T05:57:50.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 -- start start 2024-03-12T05:57:50.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f586c102390 0x7f586c19d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f586c102cb0 0x7f586c19dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f586c10b760 0x7f586c197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586c197df0 con 0x7f586c10b760 2024-03-12T05:57:50.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586c197f60 con 0x7f586c102390 2024-03-12T05:57:50.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.300+0000 7f5870eb5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f586c198120 con 0x7f586c102cb0 2024-03-12T05:57:50.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f586ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f586c10b760 0x7f586c197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f586ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f586c10b760 0x7f586c197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38006/0 (socket says 172.21.15.76:38006) 2024-03-12T05:57:50.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f586ad9d640 1 -- 172.21.15.76:0/3676780802 learned_addr learned my addr 172.21.15.76:0/3676780802 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:50.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f586a59c640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f586c102390 0x7f586c19d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f5869d9b640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f586c102cb0 0x7f586c19dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f586ad9d640 1 -- 172.21.15.76:0/3676780802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f586c102cb0 msgr2=0x7f586c19dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f586ad9d640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f586c102cb0 0x7f586c19dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f586ad9d640 1 -- 172.21.15.76:0/3676780802 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f586c102390 msgr2=0x7f586c19d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f586ad9d640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f586c102390 0x7f586c19d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f586ad9d640 1 -- 172.21.15.76:0/3676780802 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585c009560 con 0x7f586c10b760 2024-03-12T05:57:50.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f586a59c640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f586c102390 0x7f586c19d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:57:50.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f586ad9d640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f586c10b760 0x7f586c197660 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f586400e960 tx=0x7f586400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:50.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f585b7fe640 1 -- 172.21.15.76:0/3676780802 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5864004d20 con 0x7f586c10b760 2024-03-12T05:57:50.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f585b7fe640 1 -- 172.21.15.76:0/3676780802 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5864004e80 con 0x7f586c10b760 2024-03-12T05:57:50.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f5870eb5640 1 -- 172.21.15.76:0/3676780802 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586c198400 con 0x7f586c10b760 2024-03-12T05:57:50.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f585b7fe640 1 -- 172.21.15.76:0/3676780802 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58640116d0 con 0x7f586c10b760 2024-03-12T05:57:50.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.301+0000 7f5870eb5640 1 -- 172.21.15.76:0/3676780802 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f586c1a4520 con 0x7f586c10b760 2024-03-12T05:57:50.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.303+0000 7f585b7fe640 1 -- 172.21.15.76:0/3676780802 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5864005340 con 0x7f586c10b760 2024-03-12T05:57:50.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.304+0000 7f5873b63740 1 Processor -- start 2024-03-12T05:57:50.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.304+0000 7f585b7fe640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f58540a1ba0 0x7f58540a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.304+0000 7f5873b63740 1 -- start start 2024-03-12T05:57:50.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.304+0000 7f585b7fe640 1 -- 172.21.15.76:0/3676780802 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f58640cdc40 con 0x7f586c10b760 2024-03-12T05:57:50.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.304+0000 7f5869d9b640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f58540a1ba0 0x7f58540a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.304+0000 7f5873b63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816250 0x5616ca816630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.304+0000 7f5873b63740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816b70 0x5616ca81f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.304+0000 7f5873b63740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 0x5616ca821b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.304+0000 7f5873b63740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ca7d1960 con 0x5616ca816b70 2024-03-12T05:57:50.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.304+0000 7f5873b63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ca785310 con 0x5616ca816250 2024-03-12T05:57:50.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.304+0000 7f5873b63740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ca782840 con 0x5616ca81f750 2024-03-12T05:57:50.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.304+0000 7f5869d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816b70 0x5616ca81f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.304+0000 7f5869d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816b70 0x5616ca81f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38016/0 (socket says 172.21.15.76:38016) 2024-03-12T05:57:50.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.304+0000 7f5869d9b640 1 -- 172.21.15.76:0/283566722 learned_addr learned my addr 172.21.15.76:0/283566722 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:50.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.305+0000 7f586a59c640 1 --2- 172.21.15.76:0/283566722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816250 0x5616ca816630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.305+0000 7f586ad9d640 1 --2- 172.21.15.76:0/283566722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 0x5616ca821b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.305+0000 7f5869d9b640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f58540a1ba0 0x7f58540a4060 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5860012500 tx=0x7f586000bf90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:50.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.305+0000 7f5869d9b640 1 -- 172.21.15.76:0/283566722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 msgr2=0x5616ca821b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.305+0000 7f5869d9b640 1 --2- 172.21.15.76:0/283566722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 0x5616ca821b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.305+0000 7f5869d9b640 1 -- 172.21.15.76:0/283566722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816250 msgr2=0x5616ca816630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.305+0000 7f5869d9b640 1 --2- 172.21.15.76:0/283566722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816250 0x5616ca816630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.305+0000 7f5869d9b640 1 -- 172.21.15.76:0/283566722 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616ca740060 con 0x5616ca816b70 2024-03-12T05:57:50.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.305+0000 7f586ad9d640 1 --2- 172.21.15.76:0/283566722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 0x5616ca821b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:57:50.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.305+0000 7f5869d9b640 1 --2- 172.21.15.76:0/283566722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816b70 0x5616ca81f210 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f586000e810 tx=0x7f586001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4cb12c02d60e8e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:50.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.305+0000 7f5858ff9640 1 -- 172.21.15.76:0/283566722 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586000ed90 con 0x5616ca816b70 2024-03-12T05:57:50.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.305+0000 7f5858ff9640 1 -- 172.21.15.76:0/283566722 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f586001ec90 con 0x5616ca816b70 2024-03-12T05:57:50.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.305+0000 7f5858ff9640 1 -- 172.21.15.76:0/283566722 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586001fbf0 con 0x5616ca816b70 2024-03-12T05:57:50.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.306+0000 7f5873b63740 1 -- 172.21.15.76:0/283566722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816b70 msgr2=0x5616ca81f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.306+0000 7f5873b63740 1 --2- 172.21.15.76:0/283566722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816b70 0x5616ca81f210 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f586000e810 tx=0x7f586001e000 comp rx=0 tx=0).stop 2024-03-12T05:57:50.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.306+0000 7f5873b63740 1 -- 172.21.15.76:0/283566722 shutdown_connections 2024-03-12T05:57:50.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.306+0000 7f5873b63740 1 --2- 172.21.15.76:0/283566722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 0x5616ca821b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.306+0000 7f5873b63740 1 --2- 172.21.15.76:0/283566722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816b70 0x5616ca81f210 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.306+0000 7f5873b63740 1 --2- 172.21.15.76:0/283566722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816250 0x5616ca816630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.306+0000 7f5873b63740 1 -- 172.21.15.76:0/283566722 >> 172.21.15.76:0/283566722 conn(0x5616ca810980 msgr2=0x5616ca810d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:50.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.306+0000 7f5873b63740 1 -- 172.21.15.76:0/283566722 shutdown_connections 2024-03-12T05:57:50.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.306+0000 7f5873b63740 1 -- 172.21.15.76:0/283566722 wait complete. 2024-03-12T05:57:50.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.306+0000 7f5873b63740 1 -- start start 2024-03-12T05:57:50.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.306+0000 7f5873b63740 1 Processor -- start 2024-03-12T05:57:50.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5873b63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816250 0x5616ca81c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5873b63740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816b70 0x5616ca81cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5873b63740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 0x5616ca81d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5873b63740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ca77e020 con 0x5616ca816b70 2024-03-12T05:57:50.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5873b63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ca451430 con 0x5616ca816250 2024-03-12T05:57:50.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5873b63740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ca40de00 con 0x5616ca81f750 2024-03-12T05:57:50.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5873b63740 10 client.0 _ll_register_callbacks cb 0x5616ca725cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:50.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5869d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816b70 0x5616ca81cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5869d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816b70 0x5616ca81cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38030/0 (socket says 172.21.15.76:38030) 2024-03-12T05:57:50.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5869d9b640 1 -- 172.21.15.76:0/1111817214 learned_addr learned my addr 172.21.15.76:0/1111817214 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:50.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f586ad9d640 1 --2- 172.21.15.76:0/1111817214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 0x5616ca81d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f586a59c640 1 --2- 172.21.15.76:0/1111817214 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816250 0x5616ca81c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5869d9b640 1 -- 172.21.15.76:0/1111817214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 msgr2=0x5616ca81d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5869d9b640 1 --2- 172.21.15.76:0/1111817214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 0x5616ca81d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5869d9b640 1 -- 172.21.15.76:0/1111817214 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816250 msgr2=0x5616ca81c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5869d9b640 1 --2- 172.21.15.76:0/1111817214 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816250 0x5616ca81c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5869d9b640 1 -- 172.21.15.76:0/1111817214 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f586000e590 con 0x5616ca816b70 2024-03-12T05:57:50.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f586a59c640 1 --2- 172.21.15.76:0/1111817214 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816250 0x5616ca81c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:50.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5869d9b640 1 --2- 172.21.15.76:0/1111817214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816b70 0x5616ca81cc40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f58600056d0 tx=0x7f586001f750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625c66fbb76a2456 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:50.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5851ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:50.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5851ffb640 1 -- 172.21.15.76:0/1111817214 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586001e990 con 0x5616ca816b70 2024-03-12T05:57:50.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5873b63740 1 -- 172.21.15.76:0/1111817214 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616ca47d7e0 con 0x5616ca816b70 2024-03-12T05:57:50.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5851ffb640 1 -- 172.21.15.76:0/1111817214 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f586001f970 con 0x5616ca816b70 2024-03-12T05:57:50.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5851ffb640 1 -- 172.21.15.76:0/1111817214 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f586001eaf0 con 0x5616ca816b70 2024-03-12T05:57:50.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5873b63740 10 client.16053 fetch_fsmap learned FSMap version 24 2024-03-12T05:57:50.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.307+0000 7f5873b63740 1 -- 172.21.15.76:0/1111817214 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5616ca77c6a0 con 0x5616ca816b70 2024-03-12T05:57:50.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.308+0000 7f5851ffb640 1 -- 172.21.15.76:0/1111817214 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5616ca47d7e0 con 0x5616ca816b70 2024-03-12T05:57:50.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.308+0000 7f5851ffb640 1 -- 172.21.15.76:0/1111817214 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f58600273c0 con 0x5616ca816b70 2024-03-12T05:57:50.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.308+0000 7f5873b63740 10 client.16053 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:57:50.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.308+0000 7f5873b63740 10 client.16053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.308+0000 7f5873b63740 20 client.16053 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.308+0000 7f5873b63740 1 --2- 172.21.15.76:0/1111817214 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5616ca82fa10 0x5616ca831e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.308+0000 7f5873b63740 4 client.16053 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:57:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.308+0000 7f5873b63740 1 -- 172.21.15.76:0/1111817214 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616ca4ba720 con 0x5616ca82fa10 2024-03-12T05:57:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.308+0000 7f586a59c640 1 --2- 172.21.15.76:0/1111817214 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5616ca82fa10 0x5616ca831e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.309+0000 7f5851ffb640 10 client.16053 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.309+0000 7f586a59c640 1 --2- 172.21.15.76:0/1111817214 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5616ca82fa10 0x5616ca831e30 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed7fcbe30c49de5e server_cookie=3d2e29b7ceb1f93e in_seq=0 out_seq=0 2024-03-12T05:57:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.309+0000 7f5851ffb640 1 -- 172.21.15.76:0/1111817214 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5616ca4ba720 con 0x5616ca82fa10 2024-03-12T05:57:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.309+0000 7f5851ffb640 10 client.16053 handle_command_reply: tid=0 2024-03-12T05:57:50.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.309+0000 7f5873b63740 1 client.16053 shutdown 2024-03-12T05:57:50.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.309+0000 7f5873b63740 1 -- 172.21.15.76:0/1111817214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816b70 msgr2=0x5616ca81cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.309+0000 7f5873b63740 1 --2- 172.21.15.76:0/1111817214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816b70 0x5616ca81cc40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f58600056d0 tx=0x7f586001f750 comp rx=0 tx=0).stop 2024-03-12T05:57:50.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.309+0000 7f5873b63740 1 -- 172.21.15.76:0/1111817214 shutdown_connections 2024-03-12T05:57:50.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.309+0000 7f5873b63740 1 --2- 172.21.15.76:0/1111817214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 0x5616ca81d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.309+0000 7f5873b63740 1 --2- 172.21.15.76:0/1111817214 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5616ca82fa10 0x5616ca831e30 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.309+0000 7f5873b63740 1 --2- 172.21.15.76:0/1111817214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816b70 0x5616ca81cc40 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.309+0000 7f5851ffb640 0 client.16053 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:50.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.309+0000 7f5873b63740 1 --2- 172.21.15.76:0/1111817214 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816250 0x5616ca81c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.309+0000 7f5873b63740 1 -- 172.21.15.76:0/1111817214 >> 172.21.15.76:0/1111817214 conn(0x5616ca810980 msgr2=0x5616ca8126d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:50.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.310+0000 7f5873b63740 1 -- 172.21.15.76:0/1111817214 shutdown_connections 2024-03-12T05:57:50.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.310+0000 7f5873b63740 1 -- 172.21.15.76:0/1111817214 wait complete. 2024-03-12T05:57:50.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.310+0000 7f5873b63740 20 client.16053 trim_cache size 0 max 16384 2024-03-12T05:57:50.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.321+0000 7f5873b63740 1 Processor -- start 2024-03-12T05:57:50.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.321+0000 7f5873b63740 1 -- start start 2024-03-12T05:57:50.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.321+0000 7f5873b63740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca816250 0x5616ca826980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.321+0000 7f5873b63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816b70 0x5616ca81cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.321+0000 7f5873b63740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca81f750 0x5616ca80fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.321+0000 7f5873b63740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ca703130 con 0x5616ca81f750 2024-03-12T05:57:50.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.321+0000 7f5873b63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ca8107f0 con 0x5616ca816b70 2024-03-12T05:57:50.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.321+0000 7f5873b63740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ca821b40 con 0x5616ca816250 2024-03-12T05:57:50.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.321+0000 7f586ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca81f750 0x5616ca80fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.321+0000 7f586ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca81f750 0x5616ca80fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38040/0 (socket says 172.21.15.76:38040) 2024-03-12T05:57:50.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.321+0000 7f586ad9d640 1 -- 172.21.15.76:0/1873409705 learned_addr learned my addr 172.21.15.76:0/1873409705 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:50.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f586a59c640 1 --2- 172.21.15.76:0/1873409705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca816250 0x5616ca826980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f586ad9d640 1 -- 172.21.15.76:0/1873409705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca816250 msgr2=0x5616ca826980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f5869d9b640 1 --2- 172.21.15.76:0/1873409705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816b70 0x5616ca81cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f586ad9d640 1 --2- 172.21.15.76:0/1873409705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca816250 0x5616ca826980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f586ad9d640 1 -- 172.21.15.76:0/1873409705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816b70 msgr2=0x5616ca81cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f586ad9d640 1 --2- 172.21.15.76:0/1873409705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816b70 0x5616ca81cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f586ad9d640 1 -- 172.21.15.76:0/1873409705 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616ca718c20 con 0x5616ca81f750 2024-03-12T05:57:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f586a59c640 1 --2- 172.21.15.76:0/1873409705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca816250 0x5616ca826980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:57:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f586ad9d640 1 --2- 172.21.15.76:0/1873409705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca81f750 0x5616ca80fa00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f586409a880 tx=0x7f5864096cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ece640fa5190cc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f5853fff640 1 -- 172.21.15.76:0/1873409705 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586409e990 con 0x5616ca81f750 2024-03-12T05:57:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f5853fff640 1 -- 172.21.15.76:0/1873409705 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58640a2fe0 con 0x5616ca81f750 2024-03-12T05:57:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f5853fff640 1 -- 172.21.15.76:0/1873409705 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58640970c0 con 0x5616ca81f750 2024-03-12T05:57:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f5873b63740 1 -- 172.21.15.76:0/1873409705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca81f750 msgr2=0x5616ca80fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f5873b63740 1 --2- 172.21.15.76:0/1873409705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca81f750 0x5616ca80fa00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f586409a880 tx=0x7f5864096cc0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f5873b63740 1 -- 172.21.15.76:0/1873409705 shutdown_connections 2024-03-12T05:57:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f5873b63740 1 --2- 172.21.15.76:0/1873409705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca81f750 0x5616ca80fa00 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f5873b63740 1 --2- 172.21.15.76:0/1873409705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816b70 0x5616ca81cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f5873b63740 1 --2- 172.21.15.76:0/1873409705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca816250 0x5616ca826980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f5873b63740 1 -- 172.21.15.76:0/1873409705 >> 172.21.15.76:0/1873409705 conn(0x5616ca810980 msgr2=0x5616ca80f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f5873b63740 1 -- 172.21.15.76:0/1873409705 shutdown_connections 2024-03-12T05:57:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.322+0000 7f5873b63740 1 -- 172.21.15.76:0/1873409705 wait complete. 2024-03-12T05:57:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f5873b63740 1 -- start start 2024-03-12T05:57:50.338 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f5873b63740 1 Processor -- start 2024-03-12T05:57:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f5873b63740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816250 0x5616ca8844f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f5873b63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816b70 0x5616ca884a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f5873b63740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 0x5616ca87e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f5873b63740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ca87eec0 con 0x5616ca816250 2024-03-12T05:57:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f5873b63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ca87f030 con 0x5616ca816b70 2024-03-12T05:57:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f5873b63740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ca87f1f0 con 0x5616ca81f750 2024-03-12T05:57:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f5873b63740 10 client.0 _ll_register_callbacks cb 0x7f586000b9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:50.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f586a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816250 0x5616ca8844f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f586a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816250 0x5616ca8844f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38056/0 (socket says 172.21.15.76:38056) 2024-03-12T05:57:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f586a59c640 1 -- 172.21.15.76:0/2659908698 learned_addr learned my addr 172.21.15.76:0/2659908698 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f586ad9d640 1 --2- 172.21.15.76:0/2659908698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 0x5616ca87e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f586a59c640 1 -- 172.21.15.76:0/2659908698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 msgr2=0x5616ca87e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f586a59c640 1 --2- 172.21.15.76:0/2659908698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 0x5616ca87e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f586a59c640 1 -- 172.21.15.76:0/2659908698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816b70 msgr2=0x5616ca884a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f5869d9b640 1 --2- 172.21.15.76:0/2659908698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816b70 0x5616ca884a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f586a59c640 1 --2- 172.21.15.76:0/2659908698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816b70 0x5616ca884a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f586a59c640 1 -- 172.21.15.76:0/2659908698 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f586c1a4520 con 0x5616ca816250 2024-03-12T05:57:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.323+0000 7f5869d9b640 1 --2- 172.21.15.76:0/2659908698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816b70 0x5616ca884a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:57:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f586a59c640 1 --2- 172.21.15.76:0/2659908698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816250 0x5616ca8844f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f585c02c210 tx=0x7f585c02cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62cd35d391df7de server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5851ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5851ffb640 1 -- 172.21.15.76:0/2659908698 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585c044040 con 0x5616ca816250 2024-03-12T05:57:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5851ffb640 1 -- 172.21.15.76:0/2659908698 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f585c004030 con 0x5616ca816250 2024-03-12T05:57:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5873b63740 1 -- 172.21.15.76:0/2659908698 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f586001ec90 con 0x5616ca816250 2024-03-12T05:57:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5851ffb640 1 -- 172.21.15.76:0/2659908698 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f585c0041f0 con 0x5616ca816250 2024-03-12T05:57:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5873b63740 10 client.16059 fetch_fsmap learned FSMap version 24 2024-03-12T05:57:50.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5873b63740 1 -- 172.21.15.76:0/2659908698 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5616ca73e8c0 con 0x5616ca816250 2024-03-12T05:57:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5851ffb640 1 -- 172.21.15.76:0/2659908698 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f586001ec90 con 0x5616ca816250 2024-03-12T05:57:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5851ffb640 1 -- 172.21.15.76:0/2659908698 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f585c03c040 con 0x5616ca816250 2024-03-12T05:57:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5873b63740 10 client.16059 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:57:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5873b63740 10 client.16059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5873b63740 20 client.16059 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5873b63740 1 --2- 172.21.15.76:0/2659908698 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5616ca88c400 0x5616ca88e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5873b63740 4 client.16059 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:57:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5873b63740 1 -- 172.21.15.76:0/2659908698 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5616ca509890 con 0x5616ca88c400 2024-03-12T05:57:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.324+0000 7f5869d9b640 1 --2- 172.21.15.76:0/2659908698 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5616ca88c400 0x5616ca88e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5869d9b640 1 --2- 172.21.15.76:0/2659908698 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5616ca88c400 0x5616ca88e820 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f6a677807b5face server_cookie=1a313b96c3ba1ce in_seq=0 out_seq=0 2024-03-12T05:57:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5851ffb640 10 client.16059 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5851ffb640 1 -- 172.21.15.76:0/2659908698 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5616ca509890 con 0x5616ca88c400 2024-03-12T05:57:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5851ffb640 10 client.16059 handle_command_reply: tid=0 2024-03-12T05:57:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5873b63740 1 client.16059 shutdown 2024-03-12T05:57:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5873b63740 1 -- 172.21.15.76:0/2659908698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816250 msgr2=0x5616ca8844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5873b63740 1 --2- 172.21.15.76:0/2659908698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816250 0x5616ca8844f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f585c02c210 tx=0x7f585c02cb70 comp rx=0 tx=0).stop 2024-03-12T05:57:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5873b63740 1 -- 172.21.15.76:0/2659908698 shutdown_connections 2024-03-12T05:57:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5873b63740 1 --2- 172.21.15.76:0/2659908698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5616ca81f750 0x5616ca87e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5873b63740 1 --2- 172.21.15.76:0/2659908698 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5616ca816b70 0x5616ca884a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5873b63740 1 --2- 172.21.15.76:0/2659908698 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5616ca88c400 0x5616ca88e820 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5873b63740 1 --2- 172.21.15.76:0/2659908698 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5616ca816250 0x5616ca8844f0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5851ffb640 0 client.16059 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5873b63740 1 -- 172.21.15.76:0/2659908698 >> 172.21.15.76:0/2659908698 conn(0x5616ca810980 msgr2=0x5616ca80fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5873b63740 1 -- 172.21.15.76:0/2659908698 shutdown_connections 2024-03-12T05:57:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5873b63740 1 -- 172.21.15.76:0/2659908698 wait complete. 2024-03-12T05:57:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.325+0000 7f5873b63740 20 client.16059 trim_cache size 0 max 16384 2024-03-12T05:57:50.345 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:57:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.326+0000 7f5870eb5640 1 -- 172.21.15.76:0/3676780802 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f58540a1ba0 msgr2=0x7f58540a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.326+0000 7f5870eb5640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f58540a1ba0 0x7f58540a4060 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5860012500 tx=0x7f586000bf90 comp rx=0 tx=0).stop 2024-03-12T05:57:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.326+0000 7f5870eb5640 1 -- 172.21.15.76:0/3676780802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f586c10b760 msgr2=0x7f586c197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.326+0000 7f5870eb5640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f586c10b760 0x7f586c197660 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f586400e960 tx=0x7f586400ee20 comp rx=0 tx=0).stop 2024-03-12T05:57:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.326+0000 7f5870eb5640 1 -- 172.21.15.76:0/3676780802 shutdown_connections 2024-03-12T05:57:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.326+0000 7f5870eb5640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f58540a1ba0 0x7f58540a4060 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.326+0000 7f5870eb5640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f586c10b760 0x7f586c197660 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.326+0000 7f5870eb5640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f586c102cb0 0x7f586c19dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.326+0000 7f5870eb5640 1 --2- 172.21.15.76:0/3676780802 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f586c102390 0x7f586c19d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.326+0000 7f5870eb5640 1 -- 172.21.15.76:0/3676780802 >> 172.21.15.76:0/3676780802 conn(0x7f586c077250 msgr2=0x7f586c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.326+0000 7f5870eb5640 1 -- 172.21.15.76:0/3676780802 shutdown_connections 2024-03-12T05:57:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:50.326+0000 7f5870eb5640 1 -- 172.21.15.76:0/3676780802 wait complete. 2024-03-12T05:57:50.348 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:50 smithi076 ceph-mon[32095]: pgmap v64: 97 pgs: 97 active+clean; 76 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 279 op/s 2024-03-12T05:57:50.348 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:50 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2894089366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:57:50.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:50 smithi082 ceph-mon[36382]: pgmap v64: 97 pgs: 97 active+clean; 76 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 279 op/s 2024-03-12T05:57:50.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:50 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2894089366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:57:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:50 smithi088 ceph-mon[36585]: pgmap v64: 97 pgs: 97 active+clean; 76 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 279 op/s 2024-03-12T05:57:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:50 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2894089366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:57:51.350 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:57:51.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:57:51.351 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:57:51.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:51 smithi082 ceph-mon[36382]: pgmap v65: 97 pgs: 97 active+clean; 76 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 1.6 MiB/s wr, 209 op/s 2024-03-12T05:57:51.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.463+0000 7fdf1314f640 1 -- 172.21.15.76:0/1267855940 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf0c100c40 msgr2=0x7fdf0c104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.463+0000 7fdf1314f640 1 --2- 172.21.15.76:0/1267855940 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf0c100c40 0x7fdf0c104c80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009930 tx=0x7fdf0002bc30 comp rx=0 tx=0).stop 2024-03-12T05:57:51.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.463+0000 7fdf1314f640 1 -- 172.21.15.76:0/1267855940 shutdown_connections 2024-03-12T05:57:51.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.463+0000 7fdf1314f640 1 --2- 172.21.15.76:0/1267855940 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf0c105360 0x7fdf0c10cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.463+0000 7fdf1314f640 1 --2- 172.21.15.76:0/1267855940 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf0c100c40 0x7fdf0c104c80 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.463+0000 7fdf1314f640 1 --2- 172.21.15.76:0/1267855940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf0c100290 0x7fdf0c100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.463+0000 7fdf1314f640 1 -- 172.21.15.76:0/1267855940 >> 172.21.15.76:0/1267855940 conn(0x7fdf0c077230 msgr2=0x7fdf0c077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:51.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.463+0000 7fdf1314f640 1 -- 172.21.15.76:0/1267855940 shutdown_connections 2024-03-12T05:57:51.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.463+0000 7fdf1314f640 1 -- 172.21.15.76:0/1267855940 wait complete. 2024-03-12T05:57:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.464+0000 7fdf1314f640 1 Processor -- start 2024-03-12T05:57:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.464+0000 7fdf1314f640 1 -- start start 2024-03-12T05:57:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.464+0000 7fdf1314f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf0c100290 0x7fdf0c19d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.464+0000 7fdf1314f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf0c105360 0x7fdf0c19d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.464+0000 7fdf1314f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf0c1975c0 0x7fdf0c197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.464+0000 7fdf1314f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c197fb0 con 0x7fdf0c100290 2024-03-12T05:57:51.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.464+0000 7fdf1314f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c198120 con 0x7fdf0c1975c0 2024-03-12T05:57:51.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.464+0000 7fdf1314f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c1982e0 con 0x7fdf0c105360 2024-03-12T05:57:51.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.464+0000 7fdf10eeb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf0c100290 0x7fdf0c19d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.464+0000 7fdf10eeb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf0c100290 0x7fdf0c19d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41936/0 (socket says 172.21.15.76:41936) 2024-03-12T05:57:51.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.464+0000 7fdf10eeb640 1 -- 172.21.15.76:0/1565361334 learned_addr learned my addr 172.21.15.76:0/1565361334 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:51.490 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.465+0000 7fdf116ec640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf0c1975c0 0x7fdf0c197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.465+0000 7fdf0bfff640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf0c105360 0x7fdf0c19d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.465+0000 7fdf10eeb640 1 -- 172.21.15.76:0/1565361334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf0c105360 msgr2=0x7fdf0c19d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.465+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf0c105360 0x7fdf0c19d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.491 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.465+0000 7fdf10eeb640 1 -- 172.21.15.76:0/1565361334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf0c1975c0 msgr2=0x7fdf0c197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.465+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf0c1975c0 0x7fdf0c197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.465+0000 7fdf10eeb640 1 -- 172.21.15.76:0/1565361334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf00009560 con 0x7fdf0c100290 2024-03-12T05:57:51.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.465+0000 7fdf116ec640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf0c1975c0 0x7fdf0c197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:57:51.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.465+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf0c100290 0x7fdf0c19d410 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fdefc00c970 tx=0x7fdefc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:51.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.465+0000 7fdf09ffb640 1 -- 172.21.15.76:0/1565361334 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc007bb0 con 0x7fdf0c100290 2024-03-12T05:57:51.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.465+0000 7fdf1314f640 1 -- 172.21.15.76:0/1565361334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0c1a42d0 con 0x7fdf0c100290 2024-03-12T05:57:51.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.465+0000 7fdf09ffb640 1 -- 172.21.15.76:0/1565361334 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdefc007d10 con 0x7fdf0c100290 2024-03-12T05:57:51.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.465+0000 7fdf09ffb640 1 -- 172.21.15.76:0/1565361334 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc00f450 con 0x7fdf0c100290 2024-03-12T05:57:51.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.465+0000 7fdf1314f640 1 -- 172.21.15.76:0/1565361334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0c1a4790 con 0x7fdf0c100290 2024-03-12T05:57:51.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.467+0000 7fdf09ffb640 1 -- 172.21.15.76:0/1565361334 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fdefc004f70 con 0x7fdf0c100290 2024-03-12T05:57:51.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.467+0000 7fdf15dfe740 1 Processor -- start 2024-03-12T05:57:51.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf15dfe740 1 -- start start 2024-03-12T05:57:51.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf09ffb640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdef40a1ba0 0x7fdef40a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf15dfe740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ec3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf15dfe740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3c10 0x556787ecc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf15dfe740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 0x556787ecd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf15dfe740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556787e31f90 con 0x556787ec3260 2024-03-12T05:57:51.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf15dfe740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x556787e2f4c0 con 0x556787ec3c10 2024-03-12T05:57:51.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf15dfe740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556787e2aca0 con 0x556787eccd10 2024-03-12T05:57:51.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf116ec640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdef40a1ba0 0x7fdef40a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf10eeb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ec3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf09ffb640 1 -- 172.21.15.76:0/1565361334 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fdefc0cd5d0 con 0x7fdf0c100290 2024-03-12T05:57:51.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf10eeb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ec3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41942/0 (socket says 172.21.15.76:41942) 2024-03-12T05:57:51.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf10eeb640 1 -- 172.21.15.76:0/356199519 learned_addr learned my addr 172.21.15.76:0/356199519 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:51.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf0bfff640 1 --2- 172.21.15.76:0/356199519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3c10 0x556787ecc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf116ec640 1 --2- 172.21.15.76:0/356199519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 0x556787ecd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf10eeb640 1 -- 172.21.15.76:0/356199519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 msgr2=0x556787ecd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/356199519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 0x556787ecd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf10eeb640 1 -- 172.21.15.76:0/356199519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3c10 msgr2=0x556787ecc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/356199519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3c10 0x556787ecc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf10eeb640 1 -- 172.21.15.76:0/356199519 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556787ded0b0 con 0x556787ec3260 2024-03-12T05:57:51.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/356199519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ec3640 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdefc096690 tx=0x7fdefc099e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da9b03a483bf506 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:51.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.468+0000 7fdf116ec640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdef40a1ba0 0x7fdef40a4060 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdf040146e0 tx=0x7fdf04014ae0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:51.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.469+0000 7fdef2ffd640 1 -- 172.21.15.76:0/356199519 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc09e160 con 0x556787ec3260 2024-03-12T05:57:51.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.469+0000 7fdef2ffd640 1 -- 172.21.15.76:0/356199519 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdefc06aa90 con 0x556787ec3260 2024-03-12T05:57:51.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.469+0000 7fdef2ffd640 1 -- 172.21.15.76:0/356199519 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc0a29d0 con 0x556787ec3260 2024-03-12T05:57:51.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.469+0000 7fdf15dfe740 1 -- 172.21.15.76:0/356199519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 msgr2=0x556787ec3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.469+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/356199519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ec3640 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdefc096690 tx=0x7fdefc099e30 comp rx=0 tx=0).stop 2024-03-12T05:57:51.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.469+0000 7fdf15dfe740 1 -- 172.21.15.76:0/356199519 shutdown_connections 2024-03-12T05:57:51.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.469+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/356199519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 0x556787ecd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.469+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/356199519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3c10 0x556787ecc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.469+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/356199519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ec3640 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.469+0000 7fdf15dfe740 1 -- 172.21.15.76:0/356199519 >> 172.21.15.76:0/356199519 conn(0x556787ebd840 msgr2=0x556787ebdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:51.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.469+0000 7fdf15dfe740 1 -- 172.21.15.76:0/356199519 shutdown_connections 2024-03-12T05:57:51.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.469+0000 7fdf15dfe740 1 -- 172.21.15.76:0/356199519 wait complete. 2024-03-12T05:57:51.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.470+0000 7fdf15dfe740 1 -- start start 2024-03-12T05:57:51.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.470+0000 7fdf15dfe740 1 Processor -- start 2024-03-12T05:57:51.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.470+0000 7fdf15dfe740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ed07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.470+0000 7fdf15dfe740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787ec3c10 0x556787ed0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.470+0000 7fdf15dfe740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787eccd10 0x556787ed1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.470+0000 7fdf15dfe740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556787dcff40 con 0x556787ec3260 2024-03-12T05:57:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.470+0000 7fdf15dfe740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x556787afe4b0 con 0x556787eccd10 2024-03-12T05:57:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.470+0000 7fdf15dfe740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556787db0670 con 0x556787ec3c10 2024-03-12T05:57:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.470+0000 7fdf15dfe740 10 client.0 _ll_register_callbacks cb 0x556787e1df70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.470+0000 7fdf10eeb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ed07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.470+0000 7fdf10eeb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ed07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41952/0 (socket says 172.21.15.76:41952) 2024-03-12T05:57:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.470+0000 7fdf10eeb640 1 -- 172.21.15.76:0/3214443014 learned_addr learned my addr 172.21.15.76:0/3214443014 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.470+0000 7fdf0bfff640 1 --2- 172.21.15.76:0/3214443014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787ec3c10 0x556787ed0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.470+0000 7fdf116ec640 1 --2- 172.21.15.76:0/3214443014 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787eccd10 0x556787ed1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf10eeb640 1 -- 172.21.15.76:0/3214443014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787ec3c10 msgr2=0x556787ed0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/3214443014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787ec3c10 0x556787ed0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf10eeb640 1 -- 172.21.15.76:0/3214443014 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787eccd10 msgr2=0x556787ed1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/3214443014 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787eccd10 0x556787ed1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf10eeb640 1 -- 172.21.15.76:0/3214443014 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0c1a4790 con 0x556787ec3260 2024-03-12T05:57:51.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf116ec640 1 --2- 172.21.15.76:0/3214443014 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787eccd10 0x556787ed1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:51.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/3214443014 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ed07c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0058c0 tx=0x7fdefc099ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0e0e1c4b4a63e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:51.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdeebfff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:51.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdeebfff640 1 -- 172.21.15.76:0/3214443014 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc09e160 con 0x556787ec3260 2024-03-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdeebfff640 1 -- 172.21.15.76:0/3214443014 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fdefc09a230 con 0x556787ec3260 2024-03-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf15dfe740 1 -- 172.21.15.76:0/3214443014 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556787e04f40 con 0x556787ec3260 2024-03-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdeebfff640 1 -- 172.21.15.76:0/3214443014 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdefc0a29d0 con 0x556787ec3260 2024-03-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf15dfe740 10 client.16071 fetch_fsmap learned FSMap version 24 2024-03-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf15dfe740 1 -- 172.21.15.76:0/3214443014 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556787e29320 con 0x556787ec3260 2024-03-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdeebfff640 1 -- 172.21.15.76:0/3214443014 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdefc064020 con 0x556787ec3260 2024-03-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdeebfff640 1 -- 172.21.15.76:0/3214443014 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fdefc06ad20 con 0x556787ec3260 2024-03-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf15dfe740 10 client.16071 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:57:51.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf15dfe740 10 client.16071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf15dfe740 20 client.16071 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/3214443014 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556787ee0e10 0x556787ee3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf15dfe740 4 client.16071 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.471+0000 7fdf15dfe740 1 -- 172.21.15.76:0/3214443014 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556787e21310 con 0x556787ee0e10 2024-03-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.472+0000 7fdf0bfff640 1 --2- 172.21.15.76:0/3214443014 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556787ee0e10 0x556787ee3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.472+0000 7fdeebfff640 10 client.16071 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.472+0000 7fdf0bfff640 1 --2- 172.21.15.76:0/3214443014 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556787ee0e10 0x556787ee3230 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=789c6911e6fc592a server_cookie=5f538261c9dd3eb5 in_seq=0 out_seq=0 2024-03-12T05:57:51.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdeebfff640 1 -- 172.21.15.76:0/3214443014 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556787e21310 con 0x556787ee0e10 2024-03-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdeebfff640 10 client.16071 handle_command_reply: tid=0 2024-03-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdf15dfe740 1 client.16071 shutdown 2024-03-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdf15dfe740 1 -- 172.21.15.76:0/3214443014 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 msgr2=0x556787ed07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/3214443014 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ed07c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0058c0 tx=0x7fdefc099ea0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdf15dfe740 1 -- 172.21.15.76:0/3214443014 shutdown_connections 2024-03-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/3214443014 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787eccd10 0x556787ed1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/3214443014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787ec3c10 0x556787ed0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/3214443014 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556787ee0e10 0x556787ee3230 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/3214443014 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ed07c0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdeebfff640 0 client.16071 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:51.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdf15dfe740 1 -- 172.21.15.76:0/3214443014 >> 172.21.15.76:0/3214443014 conn(0x556787ebd840 msgr2=0x556787ebf7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:51.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdf15dfe740 1 -- 172.21.15.76:0/3214443014 shutdown_connections 2024-03-12T05:57:51.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdf15dfe740 1 -- 172.21.15.76:0/3214443014 wait complete. 2024-03-12T05:57:51.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.473+0000 7fdf15dfe740 20 client.16071 trim_cache size 0 max 16384 2024-03-12T05:57:51.508 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:51 smithi076 ceph-mon[32095]: pgmap v65: 97 pgs: 97 active+clean; 76 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 1.6 MiB/s wr, 209 op/s 2024-03-12T05:57:51.508 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:51 smithi088 ceph-mon[36585]: pgmap v65: 97 pgs: 97 active+clean; 76 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 1.6 MiB/s wr, 209 op/s 2024-03-12T05:57:51.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.488+0000 7fdf15dfe740 1 Processor -- start 2024-03-12T05:57:51.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.488+0000 7fdf15dfe740 1 -- start start 2024-03-12T05:57:51.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.488+0000 7fdf15dfe740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ecad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.488+0000 7fdf15dfe740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3c10 0x556787ece100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.488+0000 7fdf15dfe740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 0x556787ed1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.488+0000 7fdf15dfe740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556787ec3ff0 con 0x556787ec3260 2024-03-12T05:57:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.488+0000 7fdf15dfe740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x556787eddd50 con 0x556787ec3c10 2024-03-12T05:57:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.488+0000 7fdf15dfe740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556787ec30f0 con 0x556787eccd10 2024-03-12T05:57:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.488+0000 7fdf10eeb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ecad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.488+0000 7fdf10eeb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ecad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41954/0 (socket says 172.21.15.76:41954) 2024-03-12T05:57:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.488+0000 7fdf10eeb640 1 -- 172.21.15.76:0/9078996 learned_addr learned my addr 172.21.15.76:0/9078996 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.488+0000 7fdf0bfff640 1 --2- 172.21.15.76:0/9078996 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3c10 0x556787ece100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.489+0000 7fdf116ec640 1 --2- 172.21.15.76:0/9078996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 0x556787ed1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.489+0000 7fdf10eeb640 1 -- 172.21.15.76:0/9078996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 msgr2=0x556787ed1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.489+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/9078996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 0x556787ed1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.489+0000 7fdf10eeb640 1 -- 172.21.15.76:0/9078996 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3c10 msgr2=0x556787ece100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.489+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/9078996 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3c10 0x556787ece100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.489+0000 7fdf10eeb640 1 -- 172.21.15.76:0/9078996 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556787dc5c00 con 0x556787ec3260 2024-03-12T05:57:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.489+0000 7fdf116ec640 1 --2- 172.21.15.76:0/9078996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 0x556787ed1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:57:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.489+0000 7fdf0bfff640 1 --2- 172.21.15.76:0/9078996 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3c10 0x556787ece100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.489+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/9078996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ecad10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fdefc007590 tx=0x7fdefc099e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0c73c45ad95f80 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.489+0000 7fdef27fc640 1 -- 172.21.15.76:0/9078996 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc09e380 con 0x556787ec3260 2024-03-12T05:57:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.489+0000 7fdef27fc640 1 -- 172.21.15.76:0/9078996 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdefc064030 con 0x556787ec3260 2024-03-12T05:57:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.489+0000 7fdef27fc640 1 -- 172.21.15.76:0/9078996 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc06a930 con 0x556787ec3260 2024-03-12T05:57:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 -- 172.21.15.76:0/9078996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 msgr2=0x556787ecad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/9078996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ecad10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fdefc007590 tx=0x7fdefc099e30 comp rx=0 tx=0).stop 2024-03-12T05:57:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 -- 172.21.15.76:0/9078996 shutdown_connections 2024-03-12T05:57:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/9078996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 0x556787ed1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/9078996 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3c10 0x556787ece100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/9078996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3260 0x556787ecad10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 -- 172.21.15.76:0/9078996 >> 172.21.15.76:0/9078996 conn(0x556787ebd840 msgr2=0x556787ebc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 -- 172.21.15.76:0/9078996 shutdown_connections 2024-03-12T05:57:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 -- 172.21.15.76:0/9078996 wait complete. 2024-03-12T05:57:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 -- start start 2024-03-12T05:57:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 Processor -- start 2024-03-12T05:57:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3260 0x556787ed7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3c10 0x556787ed7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 0x556787f31400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556787f31940 con 0x556787ec3c10 2024-03-12T05:57:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x556787f31a80 con 0x556787ec3260 2024-03-12T05:57:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556787f31c40 con 0x556787eccd10 2024-03-12T05:57:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf15dfe740 10 client.0 _ll_register_callbacks cb 0x7fdefc00a4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.490+0000 7fdf0bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3c10 0x556787ed7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdf0bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3c10 0x556787ed7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41968/0 (socket says 172.21.15.76:41968) 2024-03-12T05:57:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdf0bfff640 1 -- 172.21.15.76:0/400670571 learned_addr learned my addr 172.21.15.76:0/400670571 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/400670571 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3260 0x556787ed7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdf116ec640 1 --2- 172.21.15.76:0/400670571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 0x556787f31400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdf0bfff640 1 -- 172.21.15.76:0/400670571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 msgr2=0x556787f31400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdf0bfff640 1 --2- 172.21.15.76:0/400670571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 0x556787f31400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdf0bfff640 1 -- 172.21.15.76:0/400670571 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3260 msgr2=0x556787ed7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdf0bfff640 1 --2- 172.21.15.76:0/400670571 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3260 0x556787ed7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdf0bfff640 1 -- 172.21.15.76:0/400670571 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556787e29320 con 0x556787ec3c10 2024-03-12T05:57:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdf0bfff640 1 --2- 172.21.15.76:0/400670571 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3c10 0x556787ed7640 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fdf0002e010 tx=0x7fdf0002c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba913976be654df6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdeebfff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:57:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdeebfff640 1 -- 172.21.15.76:0/400670571 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00046040 con 0x556787ec3c10 2024-03-12T05:57:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdeebfff640 1 -- 172.21.15.76:0/400670571 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fdf00004040 con 0x556787ec3c10 2024-03-12T05:57:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdf15dfe740 1 -- 172.21.15.76:0/400670571 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556787e50900 con 0x556787ec3c10 2024-03-12T05:57:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdeebfff640 1 -- 172.21.15.76:0/400670571 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf00004b20 con 0x556787ec3c10 2024-03-12T05:57:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdf15dfe740 10 client.16077 fetch_fsmap learned FSMap version 24 2024-03-12T05:57:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdf15dfe740 1 -- 172.21.15.76:0/400670571 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556787deb910 con 0x556787ec3c10 2024-03-12T05:57:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.491+0000 7fdeebfff640 1 -- 172.21.15.76:0/400670571 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x556787e50900 con 0x556787ec3c10 2024-03-12T05:57:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.492+0000 7fdeebfff640 1 -- 172.21.15.76:0/400670571 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fdf0003c040 con 0x556787ec3c10 2024-03-12T05:57:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.492+0000 7fdf15dfe740 10 client.16077 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:57:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.492+0000 7fdf15dfe740 10 client.16077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.492+0000 7fdf15dfe740 20 client.16077 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.492+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/400670571 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556787f39560 0x556787f3b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.492+0000 7fdf15dfe740 4 client.16077 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:57:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.492+0000 7fdf15dfe740 1 -- 172.21.15.76:0/400670571 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x556787b2a860 con 0x556787f39560 2024-03-12T05:57:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.492+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/400670571 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556787f39560 0x556787f3b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.492+0000 7fdeebfff640 10 client.16077 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.492+0000 7fdf10eeb640 1 --2- 172.21.15.76:0/400670571 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556787f39560 0x556787f3b980 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=222d24a6faceb3b server_cookie=528b16ad08d07046 in_seq=0 out_seq=0 2024-03-12T05:57:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.493+0000 7fdeebfff640 1 -- 172.21.15.76:0/400670571 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556787b2a860 con 0x556787f39560 2024-03-12T05:57:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.493+0000 7fdeebfff640 10 client.16077 handle_command_reply: tid=0 2024-03-12T05:57:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.493+0000 7fdf15dfe740 1 client.16077 shutdown 2024-03-12T05:57:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.493+0000 7fdf15dfe740 1 -- 172.21.15.76:0/400670571 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3c10 msgr2=0x556787ed7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.493+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/400670571 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3c10 0x556787ed7640 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fdf0002e010 tx=0x7fdf0002c3a0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.493+0000 7fdf15dfe740 1 -- 172.21.15.76:0/400670571 shutdown_connections 2024-03-12T05:57:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.493+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/400670571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556787eccd10 0x556787f31400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.493+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/400670571 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556787f39560 0x556787f3b980 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.493+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/400670571 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556787ec3c10 0x556787ed7640 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.493+0000 7fdf15dfe740 1 --2- 172.21.15.76:0/400670571 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556787ec3260 0x556787ed7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.493+0000 7fdeebfff640 0 client.16077 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:51.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.493+0000 7fdf15dfe740 1 -- 172.21.15.76:0/400670571 >> 172.21.15.76:0/400670571 conn(0x556787ebd840 msgr2=0x556787ebc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:51.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf15dfe740 1 -- 172.21.15.76:0/400670571 shutdown_connections 2024-03-12T05:57:51.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf15dfe740 1 -- 172.21.15.76:0/400670571 wait complete. 2024-03-12T05:57:51.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf15dfe740 20 client.16077 trim_cache size 0 max 16384 2024-03-12T05:57:51.525 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:57:51.525 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T05:57:51.525 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "e379915f-5267-4751-be3e-2de4b98b5a9d", 2024-03-12T05:57:51.525 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T05:57:51.525 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:57:51.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf1314f640 1 -- 172.21.15.76:0/1565361334 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdef40a1ba0 msgr2=0x7fdef40a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf1314f640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdef40a1ba0 0x7fdef40a4060 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdf040146e0 tx=0x7fdf04014ae0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf1314f640 1 -- 172.21.15.76:0/1565361334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf0c100290 msgr2=0x7fdf0c19d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf1314f640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf0c100290 0x7fdf0c19d410 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fdefc00c970 tx=0x7fdefc00ce30 comp rx=0 tx=0).stop 2024-03-12T05:57:51.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf1314f640 1 -- 172.21.15.76:0/1565361334 shutdown_connections 2024-03-12T05:57:51.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf1314f640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdef40a1ba0 0x7fdef40a4060 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf1314f640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdf0c1975c0 0x7fdf0c197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf1314f640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf0c105360 0x7fdf0c19d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf1314f640 1 --2- 172.21.15.76:0/1565361334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf0c100290 0x7fdf0c19d410 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf1314f640 1 -- 172.21.15.76:0/1565361334 >> 172.21.15.76:0/1565361334 conn(0x7fdf0c077230 msgr2=0x7fdf0c0fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:51.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf1314f640 1 -- 172.21.15.76:0/1565361334 shutdown_connections 2024-03-12T05:57:51.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.494+0000 7fdf1314f640 1 -- 172.21.15.76:0/1565361334 wait complete. 2024-03-12T05:57:51.528 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:57:51.529 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:57:51.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.648+0000 7f43fd60f640 1 -- 172.21.15.76:0/3792527403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43f8100260 msgr2=0x7f43f8100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.648+0000 7f43fd60f640 1 --2- 172.21.15.76:0/3792527403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43f8100260 0x7f43f8100640 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f43e8009930 tx=0x7f43e802bc30 comp rx=0 tx=0).stop 2024-03-12T05:57:51.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.648+0000 7f43fd60f640 1 -- 172.21.15.76:0/3792527403 shutdown_connections 2024-03-12T05:57:51.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.648+0000 7f43fd60f640 1 --2- 172.21.15.76:0/3792527403 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43f8105330 0x7f43f810cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.648+0000 7f43fd60f640 1 --2- 172.21.15.76:0/3792527403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43f8100c10 0x7f43f8104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.648+0000 7f43fd60f640 1 --2- 172.21.15.76:0/3792527403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43f8100260 0x7f43f8100640 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.648+0000 7f43fd60f640 1 -- 172.21.15.76:0/3792527403 >> 172.21.15.76:0/3792527403 conn(0x7f43f8077180 msgr2=0x7f43f8077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:51.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.649+0000 7f43fd60f640 1 -- 172.21.15.76:0/3792527403 shutdown_connections 2024-03-12T05:57:51.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.649+0000 7f43fd60f640 1 -- 172.21.15.76:0/3792527403 wait complete. 2024-03-12T05:57:51.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.649+0000 7f43fd60f640 1 Processor -- start 2024-03-12T05:57:51.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.649+0000 7f43fd60f640 1 -- start start 2024-03-12T05:57:51.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43fd60f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43f8100260 0x7f43f819d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43fd60f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43f8105330 0x7f43f819d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43fd60f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43f8197620 0x7f43f8197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43fd60f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f8198010 con 0x7f43f8100260 2024-03-12T05:57:51.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43fd60f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f8198180 con 0x7f43f8105330 2024-03-12T05:57:51.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43fd60f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f8198340 con 0x7f43f8197620 2024-03-12T05:57:51.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43f6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43f8100260 0x7f43f819d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43f6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43f8100260 0x7f43f819d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:41986/0 (socket says 172.21.15.76:41986) 2024-03-12T05:57:51.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43f77fe640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43f8197620 0x7f43f8197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43f6ffd640 1 -- 172.21.15.76:0/3054796249 learned_addr learned my addr 172.21.15.76:0/3054796249 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:51.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43effff640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43f8105330 0x7f43f819d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43f6ffd640 1 -- 172.21.15.76:0/3054796249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43f8197620 msgr2=0x7f43f8197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43f6ffd640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43f8197620 0x7f43f8197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43f6ffd640 1 -- 172.21.15.76:0/3054796249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43f8105330 msgr2=0x7f43f819d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43f6ffd640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43f8105330 0x7f43f819d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43f6ffd640 1 -- 172.21.15.76:0/3054796249 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e8009560 con 0x7f43f8100260 2024-03-12T05:57:51.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43effff640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43f8105330 0x7f43f819d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:57:51.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.650+0000 7f43f6ffd640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43f8100260 0x7f43f819d420 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f43e8009230 tx=0x7f43e802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:51.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.651+0000 7f43f4ff9640 1 -- 172.21.15.76:0/3054796249 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e8004350 con 0x7f43f8100260 2024-03-12T05:57:51.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.651+0000 7f43f4ff9640 1 -- 172.21.15.76:0/3054796249 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43e80044b0 con 0x7f43f8100260 2024-03-12T05:57:51.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.651+0000 7f43fd60f640 1 -- 172.21.15.76:0/3054796249 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f81a42e0 con 0x7f43f8100260 2024-03-12T05:57:51.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.651+0000 7f43f4ff9640 1 -- 172.21.15.76:0/3054796249 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e8003d00 con 0x7f43f8100260 2024-03-12T05:57:51.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.651+0000 7f43fd60f640 1 -- 172.21.15.76:0/3054796249 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f81a47e0 con 0x7f43f8100260 2024-03-12T05:57:51.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.653+0000 7f43f4ff9640 1 -- 172.21.15.76:0/3054796249 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f43e8004df0 con 0x7f43f8100260 2024-03-12T05:57:51.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.654+0000 7f44001da740 1 Processor -- start 2024-03-12T05:57:51.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.654+0000 7f44001da740 1 -- start start 2024-03-12T05:57:51.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.654+0000 7f44001da740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e072b0 0x5603a1e07690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.654+0000 7f44001da740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e07bd0 0x5603a1e10270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.654+0000 7f44001da740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e107b0 0x5603a1e12ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.654+0000 7f44001da740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603a1d76280 con 0x5603a1e107b0 2024-03-12T05:57:51.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.654+0000 7f44001da740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603a1d737b0 con 0x5603a1e072b0 2024-03-12T05:57:51.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.654+0000 7f44001da740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603a1d6ef90 con 0x5603a1e07bd0 2024-03-12T05:57:51.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.654+0000 7f43f77fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e072b0 0x5603a1e07690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.654+0000 7f43f77fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e072b0 0x5603a1e07690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:37474/0 (socket says 172.21.15.76:37474) 2024-03-12T05:57:51.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.654+0000 7f43f77fe640 1 -- 172.21.15.76:0/2398224806 learned_addr learned my addr 172.21.15.76:0/2398224806 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:51.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.654+0000 7f43effff640 1 --2- 172.21.15.76:0/2398224806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e107b0 0x5603a1e12ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.655+0000 7f43f6ffd640 1 --2- 172.21.15.76:0/2398224806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e07bd0 0x5603a1e10270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.655+0000 7f43f4ff9640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f43e40a1c20 0x7f43e40a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.655+0000 7f43f77fe640 1 -- 172.21.15.76:0/2398224806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e07bd0 msgr2=0x5603a1e10270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.655+0000 7f43f77fe640 1 --2- 172.21.15.76:0/2398224806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e07bd0 0x5603a1e10270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.655+0000 7f43f77fe640 1 -- 172.21.15.76:0/2398224806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e107b0 msgr2=0x5603a1e12ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.655+0000 7f43f77fe640 1 --2- 172.21.15.76:0/2398224806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e107b0 0x5603a1e12ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.655+0000 7f43f77fe640 1 -- 172.21.15.76:0/2398224806 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603a1d2d570 con 0x5603a1e072b0 2024-03-12T05:57:51.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.655+0000 7f43f6ffd640 1 --2- 172.21.15.76:0/2398224806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e07bd0 0x5603a1e10270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:57:51.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.655+0000 7f43f4ff9640 1 -- 172.21.15.76:0/3054796249 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f43e80ef5b0 con 0x7f43f8100260 2024-03-12T05:57:51.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.655+0000 7f43f77fe640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f43e40a1c20 0x7f43e40a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.655+0000 7f43f77fe640 1 --2- 172.21.15.76:0/2398224806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e072b0 0x5603a1e07690 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009f50 tx=0x7f43f00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dad2ca351e6b44ea server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:51.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.655+0000 7f43ed7fa640 1 -- 172.21.15.76:0/2398224806 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0026070 con 0x5603a1e072b0 2024-03-12T05:57:51.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.655+0000 7f43ed7fa640 1 -- 172.21.15.76:0/2398224806 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43f00169f0 con 0x5603a1e072b0 2024-03-12T05:57:51.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.655+0000 7f43ed7fa640 1 -- 172.21.15.76:0/2398224806 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f0021870 con 0x5603a1e072b0 2024-03-12T05:57:51.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.656+0000 7f44001da740 1 -- 172.21.15.76:0/2398224806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e072b0 msgr2=0x5603a1e07690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.656+0000 7f43f77fe640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f43e40a1c20 0x7f43e40a40e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f43f0025000 tx=0x7f43f0026000 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:51.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.656+0000 7f44001da740 1 --2- 172.21.15.76:0/2398224806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e072b0 0x5603a1e07690 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009f50 tx=0x7f43f00118d0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.656+0000 7f44001da740 1 -- 172.21.15.76:0/2398224806 shutdown_connections 2024-03-12T05:57:51.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.656+0000 7f44001da740 1 --2- 172.21.15.76:0/2398224806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e107b0 0x5603a1e12ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.656+0000 7f44001da740 1 --2- 172.21.15.76:0/2398224806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e07bd0 0x5603a1e10270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.656+0000 7f44001da740 1 --2- 172.21.15.76:0/2398224806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e072b0 0x5603a1e07690 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.656+0000 7f44001da740 1 -- 172.21.15.76:0/2398224806 >> 172.21.15.76:0/2398224806 conn(0x5603a1e01990 msgr2=0x5603a1e01d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:51.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.656+0000 7f44001da740 1 -- 172.21.15.76:0/2398224806 shutdown_connections 2024-03-12T05:57:51.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.656+0000 7f44001da740 1 -- 172.21.15.76:0/2398224806 wait complete. 2024-03-12T05:57:51.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f44001da740 1 -- start start 2024-03-12T05:57:51.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f44001da740 1 Processor -- start 2024-03-12T05:57:51.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f44001da740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e072b0 0x5603a1e10190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f44001da740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e07bd0 0x5603a1e0d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f44001da740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e107b0 0x5603a1e0db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f44001da740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603a1a42430 con 0x5603a1e072b0 2024-03-12T05:57:51.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f44001da740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603a19fee00 con 0x5603a1e07bd0 2024-03-12T05:57:51.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f44001da740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603a1cf3fd0 con 0x5603a1e107b0 2024-03-12T05:57:51.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f44001da740 10 client.0 _ll_register_callbacks cb 0x5603a1d49fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:51.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f43f6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e072b0 0x5603a1e10190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f43effff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e07bd0 0x5603a1e0d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f43f6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e072b0 0x5603a1e10190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:42008/0 (socket says 172.21.15.76:42008) 2024-03-12T05:57:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f43effff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e07bd0 0x5603a1e0d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:37486/0 (socket says 172.21.15.76:37486) 2024-03-12T05:57:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f43f6ffd640 1 -- 172.21.15.76:0/2601310877 learned_addr learned my addr 172.21.15.76:0/2601310877 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.657+0000 7f43f77fe640 1 --2- 172.21.15.76:0/2601310877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e107b0 0x5603a1e0db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.658+0000 7f43effff640 1 -- 172.21.15.76:0/2601310877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e107b0 msgr2=0x5603a1e0db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.658+0000 7f43effff640 1 --2- 172.21.15.76:0/2601310877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e107b0 0x5603a1e0db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.658+0000 7f43effff640 1 -- 172.21.15.76:0/2601310877 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e072b0 msgr2=0x5603a1e10190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.658+0000 7f43effff640 1 --2- 172.21.15.76:0/2601310877 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e072b0 0x5603a1e10190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.658+0000 7f43effff640 1 -- 172.21.15.76:0/2601310877 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f0009c00 con 0x5603a1e07bd0 2024-03-12T05:57:51.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.658+0000 7f43effff640 1 --2- 172.21.15.76:0/2601310877 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e07bd0 0x5603a1e0d520 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f43f8106a10 tx=0x7f43e000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=133d7fcd082d2da5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:51.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.658+0000 7f43de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T05:57:51.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.658+0000 7f43de7fc640 1 -- 172.21.15.76:0/2601310877 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0017070 con 0x5603a1e07bd0 2024-03-12T05:57:51.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.658+0000 7f43de7fc640 1 -- 172.21.15.76:0/2601310877 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f43e0012110 con 0x5603a1e07bd0 2024-03-12T05:57:51.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.658+0000 7f44001da740 1 -- 172.21.15.76:0/2601310877 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5603a1a6e7e0 con 0x5603a1e07bd0 2024-03-12T05:57:51.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.658+0000 7f43de7fc640 1 -- 172.21.15.76:0/2601310877 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43e00122d0 con 0x5603a1e07bd0 2024-03-12T05:57:51.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.658+0000 7f44001da740 10 client.25523 fetch_fsmap learned FSMap version 24 2024-03-12T05:57:51.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.658+0000 7f44001da740 1 -- 172.21.15.76:0/2601310877 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5603a1d6d610 con 0x5603a1e07bd0 2024-03-12T05:57:51.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.659+0000 7f43de7fc640 1 -- 172.21.15.76:0/2601310877 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5603a1a6e7e0 con 0x5603a1e07bd0 2024-03-12T05:57:51.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.659+0000 7f43de7fc640 1 -- 172.21.15.76:0/2601310877 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f43e0010430 con 0x5603a1e07bd0 2024-03-12T05:57:51.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.659+0000 7f44001da740 10 client.25523 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:57:51.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.659+0000 7f44001da740 10 client.25523 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:51.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.659+0000 7f44001da740 20 client.25523 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:51.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.659+0000 7f44001da740 1 --2- 172.21.15.76:0/2601310877 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5603a1e20710 0x5603a1e22b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.659+0000 7f44001da740 4 client.25523 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:57:51.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.659+0000 7f44001da740 1 -- 172.21.15.76:0/2601310877 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5603a1aab720 con 0x5603a1e20710 2024-03-12T05:57:51.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.659+0000 7f43f6ffd640 1 --2- 172.21.15.76:0/2601310877 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5603a1e20710 0x5603a1e22b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.660+0000 7f43f6ffd640 1 --2- 172.21.15.76:0/2601310877 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5603a1e20710 0x5603a1e22b30 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=244d45f589238200 server_cookie=c09b91a5c8832065 in_seq=0 out_seq=0 2024-03-12T05:57:51.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.660+0000 7f43de7fc640 10 client.25523 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:51.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.660+0000 7f43de7fc640 1 -- 172.21.15.76:0/2601310877 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5603a1aab720 con 0x5603a1e20710 2024-03-12T05:57:51.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.660+0000 7f43de7fc640 10 client.25523 handle_command_reply: tid=0 2024-03-12T05:57:51.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.660+0000 7f44001da740 1 client.25523 shutdown 2024-03-12T05:57:51.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.661+0000 7f44001da740 1 -- 172.21.15.76:0/2601310877 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e07bd0 msgr2=0x5603a1e0d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.661+0000 7f44001da740 1 --2- 172.21.15.76:0/2601310877 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e07bd0 0x5603a1e0d520 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f43f8106a10 tx=0x7f43e000f790 comp rx=0 tx=0).stop 2024-03-12T05:57:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.661+0000 7f44001da740 1 -- 172.21.15.76:0/2601310877 shutdown_connections 2024-03-12T05:57:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.661+0000 7f44001da740 1 --2- 172.21.15.76:0/2601310877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e107b0 0x5603a1e0db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.661+0000 7f44001da740 1 --2- 172.21.15.76:0/2601310877 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e07bd0 0x5603a1e0d520 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.661+0000 7f44001da740 1 --2- 172.21.15.76:0/2601310877 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5603a1e20710 0x5603a1e22b30 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.661+0000 7f43de7fc640 0 client.25523 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:51.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.661+0000 7f44001da740 1 --2- 172.21.15.76:0/2601310877 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e072b0 0x5603a1e10190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.661+0000 7f44001da740 1 -- 172.21.15.76:0/2601310877 >> 172.21.15.76:0/2601310877 conn(0x5603a1e01990 msgr2=0x5603a1e03730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:51.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.661+0000 7f44001da740 1 -- 172.21.15.76:0/2601310877 shutdown_connections 2024-03-12T05:57:51.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.661+0000 7f44001da740 1 -- 172.21.15.76:0/2601310877 wait complete. 2024-03-12T05:57:51.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.661+0000 7f44001da740 20 client.25523 trim_cache size 0 max 16384 2024-03-12T05:57:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.680+0000 7f44001da740 1 Processor -- start 2024-03-12T05:57:51.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.680+0000 7f44001da740 1 -- start start 2024-03-12T05:57:51.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f44001da740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e072b0 0x5603a1e0c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f44001da740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e07bd0 0x5603a1e0fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f44001da740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e107b0 0x5603a1e10190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f44001da740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603a1e12fa0 con 0x5603a1e107b0 2024-03-12T05:57:51.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f44001da740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603a1a6e7e0 con 0x5603a1e072b0 2024-03-12T05:57:51.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f44001da740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603a1e1eeb0 con 0x5603a1e07bd0 2024-03-12T05:57:51.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f43f6ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e072b0 0x5603a1e0c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f43f77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e107b0 0x5603a1e10190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f43f6ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e072b0 0x5603a1e0c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:37490/0 (socket says 172.21.15.76:37490) 2024-03-12T05:57:51.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f43f6ffd640 1 -- 172.21.15.76:0/596497049 learned_addr learned my addr 172.21.15.76:0/596497049 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:51.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f43effff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e07bd0 0x5603a1e0fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f43f6ffd640 1 -- 172.21.15.76:0/596497049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e07bd0 msgr2=0x5603a1e0fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f43f6ffd640 1 --2- 172.21.15.76:0/596497049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e07bd0 0x5603a1e0fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f43f6ffd640 1 -- 172.21.15.76:0/596497049 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e107b0 msgr2=0x5603a1e10190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f43f6ffd640 1 --2- 172.21.15.76:0/596497049 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e107b0 0x5603a1e10190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f43f6ffd640 1 -- 172.21.15.76:0/596497049 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603a1d09ac0 con 0x5603a1e072b0 2024-03-12T05:57:51.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.681+0000 7f43f77fe640 1 --2- 172.21.15.76:0/596497049 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e107b0 0x5603a1e10190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:57:51.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.682+0000 7f43f6ffd640 1 --2- 172.21.15.76:0/596497049 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e072b0 0x5603a1e0c500 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f43e80ed330 tx=0x7f43e80dc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bac6a891a5d416a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:51.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.682+0000 7f43ecff9640 1 -- 172.21.15.76:0/596497049 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e80bca20 con 0x5603a1e072b0 2024-03-12T05:57:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.682+0000 7f43ecff9640 1 -- 172.21.15.76:0/596497049 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43e80b6e90 con 0x5603a1e072b0 2024-03-12T05:57:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.682+0000 7f43ecff9640 1 -- 172.21.15.76:0/596497049 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e80ba620 con 0x5603a1e072b0 2024-03-12T05:57:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.683+0000 7f44001da740 1 -- 172.21.15.76:0/596497049 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e072b0 msgr2=0x5603a1e0c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.683+0000 7f44001da740 1 --2- 172.21.15.76:0/596497049 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e072b0 0x5603a1e0c500 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f43e80ed330 tx=0x7f43e80dc040 comp rx=0 tx=0).stop 2024-03-12T05:57:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.683+0000 7f44001da740 1 -- 172.21.15.76:0/596497049 shutdown_connections 2024-03-12T05:57:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.683+0000 7f44001da740 1 --2- 172.21.15.76:0/596497049 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e107b0 0x5603a1e10190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.683+0000 7f44001da740 1 --2- 172.21.15.76:0/596497049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e07bd0 0x5603a1e0fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.683+0000 7f44001da740 1 --2- 172.21.15.76:0/596497049 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e072b0 0x5603a1e0c500 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.683+0000 7f44001da740 1 -- 172.21.15.76:0/596497049 >> 172.21.15.76:0/596497049 conn(0x5603a1e01990 msgr2=0x5603a1e0bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.683+0000 7f44001da740 1 -- 172.21.15.76:0/596497049 shutdown_connections 2024-03-12T05:57:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.683+0000 7f44001da740 1 -- 172.21.15.76:0/596497049 wait complete. 2024-03-12T05:57:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.683+0000 7f44001da740 1 -- start start 2024-03-12T05:57:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.683+0000 7f44001da740 1 Processor -- start 2024-03-12T05:57:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f44001da740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e072b0 0x5603a1e167e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f44001da740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e07bd0 0x5603a1e16d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f43f6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e072b0 0x5603a1e167e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f43f6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e072b0 0x5603a1e167e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:42028/0 (socket says 172.21.15.76:42028) 2024-03-12T05:57:51.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f44001da740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e107b0 0x5603a1e76c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f43f6ffd640 1 -- 172.21.15.76:0/830691511 learned_addr learned my addr 172.21.15.76:0/830691511 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:57:51.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f44001da740 1 -- 172.21.15.76:0/830691511 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603a1e77180 con 0x5603a1e072b0 2024-03-12T05:57:51.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f44001da740 1 -- 172.21.15.76:0/830691511 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603a1e772c0 con 0x5603a1e07bd0 2024-03-12T05:57:51.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f44001da740 1 -- 172.21.15.76:0/830691511 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603a1e77400 con 0x5603a1e107b0 2024-03-12T05:57:51.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f44001da740 10 client.0 _ll_register_callbacks cb 0x7f43f0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:57:51.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f43effff640 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e07bd0 0x5603a1e16d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f43f77fe640 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e107b0 0x5603a1e76c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f43effff640 1 -- 172.21.15.76:0/830691511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e107b0 msgr2=0x5603a1e76c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f43effff640 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e107b0 0x5603a1e76c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f43effff640 1 -- 172.21.15.76:0/830691511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e072b0 msgr2=0x5603a1e167e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:57:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f43effff640 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e072b0 0x5603a1e167e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f43effff640 1 -- 172.21.15.76:0/830691511 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5603a1d2d570 con 0x5603a1e07bd0 2024-03-12T05:57:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f43f6ffd640 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e072b0 0x5603a1e167e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:57:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.684+0000 7f43f77fe640 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e107b0 0x5603a1e76c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:57:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.685+0000 7f43effff640 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e07bd0 0x5603a1e16d20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x5603a1e0eaf0 tx=0x7f43e0012110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddfe6df15dd1d082 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:57:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.685+0000 7f43de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T05:57:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.685+0000 7f43de7fc640 1 -- 172.21.15.76:0/830691511 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0017040 con 0x5603a1e07bd0 2024-03-12T05:57:51.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.685+0000 7f43de7fc640 1 -- 172.21.15.76:0/830691511 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f43e0012280 con 0x5603a1e07bd0 2024-03-12T05:57:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.685+0000 7f44001da740 1 -- 172.21.15.76:0/830691511 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43f00169f0 con 0x5603a1e07bd0 2024-03-12T05:57:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.685+0000 7f43de7fc640 1 -- 172.21.15.76:0/830691511 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43e00109d0 con 0x5603a1e07bd0 2024-03-12T05:57:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.685+0000 7f44001da740 10 client.25529 fetch_fsmap learned FSMap version 24 2024-03-12T05:57:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.685+0000 7f44001da740 1 -- 172.21.15.76:0/830691511 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7f43f81a47e0 con 0x5603a1e07bd0 2024-03-12T05:57:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.685+0000 7f43de7fc640 1 -- 172.21.15.76:0/830691511 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43f00169f0 con 0x5603a1e07bd0 2024-03-12T05:57:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.686+0000 7f43de7fc640 1 -- 172.21.15.76:0/830691511 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f43e00037f0 con 0x5603a1e07bd0 2024-03-12T05:57:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.686+0000 7f44001da740 10 client.25529 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:57:51.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.686+0000 7f44001da740 10 client.25529 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:57:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.686+0000 7f44001da740 20 client.25529 populate_metadata read hostname 'smithi076' 2024-03-12T05:57:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.686+0000 7f44001da740 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5603a1e20710 0x5603a1e7ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:57:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.686+0000 7f44001da740 4 client.25529 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:57:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.686+0000 7f44001da740 1 -- 172.21.15.76:0/830691511 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5603a1afa890 con 0x5603a1e20710 2024-03-12T05:57:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.686+0000 7f43f6ffd640 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5603a1e20710 0x5603a1e7ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:57:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.686+0000 7f43f6ffd640 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5603a1e20710 0x5603a1e7ac30 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8650ecd7bc99121c server_cookie=6c27974605dc436b in_seq=0 out_seq=0 2024-03-12T05:57:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.686+0000 7f43de7fc640 10 client.25529 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f43de7fc640 1 -- 172.21.15.76:0/830691511 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5603a1afa890 con 0x5603a1e20710 2024-03-12T05:57:51.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f43de7fc640 10 client.25529 handle_command_reply: tid=0 2024-03-12T05:57:51.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f44001da740 1 client.25529 shutdown 2024-03-12T05:57:51.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f44001da740 1 -- 172.21.15.76:0/830691511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e07bd0 msgr2=0x5603a1e16d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f44001da740 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e07bd0 0x5603a1e16d20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x5603a1e0eaf0 tx=0x7f43e0012110 comp rx=0 tx=0).stop 2024-03-12T05:57:51.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f44001da740 1 -- 172.21.15.76:0/830691511 shutdown_connections 2024-03-12T05:57:51.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f44001da740 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5603a1e107b0 0x5603a1e76c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f44001da740 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5603a1e07bd0 0x5603a1e16d20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f44001da740 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5603a1e20710 0x5603a1e7ac30 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f44001da740 1 --2- 172.21.15.76:0/830691511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5603a1e072b0 0x5603a1e167e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f43de7fc640 0 client.25529 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:57:51.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f44001da740 1 -- 172.21.15.76:0/830691511 >> 172.21.15.76:0/830691511 conn(0x5603a1e01990 msgr2=0x5603a1e02e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:51.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f44001da740 1 -- 172.21.15.76:0/830691511 shutdown_connections 2024-03-12T05:57:51.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f44001da740 1 -- 172.21.15.76:0/830691511 wait complete. 2024-03-12T05:57:51.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.687+0000 7f44001da740 20 client.25529 trim_cache size 0 max 16384 2024-03-12T05:57:51.729 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:57:51.730 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:57:51.730 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T05:57:51.730 INFO:teuthology.orchestra.run.smithi076.stdout: "e379915f-5267-4751-be3e-2de4b98b5a9d": { 2024-03-12T05:57:51.730 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T05:57:51.730 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "e379915f-5267-4751-be3e-2de4b98b5a9d", 2024-03-12T05:57:51.730 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T05:57:51.730 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:57:51.730 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:57:51.731 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:57:51.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.688+0000 7f43fd60f640 1 -- 172.21.15.76:0/3054796249 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f43e40a1c20 msgr2=0x7f43e40a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.688+0000 7f43fd60f640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f43e40a1c20 0x7f43e40a40e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f43f0025000 tx=0x7f43f0026000 comp rx=0 tx=0).stop 2024-03-12T05:57:51.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.688+0000 7f43fd60f640 1 -- 172.21.15.76:0/3054796249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43f8100260 msgr2=0x7f43f819d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:57:51.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.688+0000 7f43fd60f640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43f8100260 0x7f43f819d420 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f43e8009230 tx=0x7f43e802c650 comp rx=0 tx=0).stop 2024-03-12T05:57:51.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.688+0000 7f43fd60f640 1 -- 172.21.15.76:0/3054796249 shutdown_connections 2024-03-12T05:57:51.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.688+0000 7f43fd60f640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f43e40a1c20 0x7f43e40a40e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.688+0000 7f43fd60f640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43f8197620 0x7f43f8197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.688+0000 7f43fd60f640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f43f8105330 0x7f43f819d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.688+0000 7f43fd60f640 1 --2- 172.21.15.76:0/3054796249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43f8100260 0x7f43f819d420 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:57:51.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.688+0000 7f43fd60f640 1 -- 172.21.15.76:0/3054796249 >> 172.21.15.76:0/3054796249 conn(0x7f43f8077180 msgr2=0x7f43f8104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:57:51.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.688+0000 7f43fd60f640 1 -- 172.21.15.76:0/3054796249 shutdown_connections 2024-03-12T05:57:51.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:57:51.688+0000 7f43fd60f640 1 -- 172.21.15.76:0/3054796249 wait complete. 2024-03-12T05:57:51.734 INFO:tasks.cephfs.filesystem:scrub status for tag:e379915f-5267-4751-be3e-2de4b98b5a9d - {'path': '/', 'tag': 'e379915f-5267-4751-be3e-2de4b98b5a9d', 'options': 'recursive,force'} 2024-03-12T05:57:52.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:52 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T05:57:52.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:52 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:52.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:52 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:52.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:52 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:52.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:52 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:57:52.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:52 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:52.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:52 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:52.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:52 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:52 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T05:57:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:52 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:52 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:52 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:52 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:57:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:52 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:52.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:52 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:52.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:52 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:52.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:52 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T05:57:52.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:52 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:52.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:52 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:52.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:52 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:52.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:52 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:57:52.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:52 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:52.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:52 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:52.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:52 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:57:53.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:53 smithi082 ceph-mon[36382]: pgmap v66: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 1.7 MiB/s wr, 219 op/s 2024-03-12T05:57:53.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:53 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:57:53.734 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:53 smithi088 ceph-mon[36585]: pgmap v66: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 1.7 MiB/s wr, 219 op/s 2024-03-12T05:57:53.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:53 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:57:53.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:53 smithi076 ceph-mon[32095]: pgmap v66: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 1.7 MiB/s wr, 219 op/s 2024-03-12T05:57:53.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:53 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:57:54.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:54 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:54.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:54 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:54.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:54 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:54.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:54 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:54.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:54 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:57:54.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:54 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:57:54.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:54 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:54.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:54 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:54.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:54 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:54.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:54 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:54.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:54 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:54.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:54 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:57:54.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:54 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:57:54.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:54 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:54.988 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:54 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:54.988 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:54 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:54.988 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:54 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:54.988 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:54 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:54.988 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:54 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:57:54.989 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:54 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:57:54.989 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:54 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:57:55.945 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:55 smithi082 ceph-mon[36382]: pgmap v67: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 341 KiB/s wr, 31 op/s 2024-03-12T05:57:55.984 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:55 smithi088 ceph-mon[36585]: pgmap v67: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 341 KiB/s wr, 31 op/s 2024-03-12T05:57:55.988 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:55 smithi076 ceph-mon[32095]: pgmap v67: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 341 KiB/s wr, 31 op/s 2024-03-12T05:57:58.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:57:58 smithi082 ceph-mon[36382]: pgmap v68: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 341 KiB/s wr, 31 op/s 2024-03-12T05:57:58.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:57:58 smithi088 ceph-mon[36585]: pgmap v68: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 341 KiB/s wr, 31 op/s 2024-03-12T05:57:58.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:57:58 smithi076 ceph-mon[32095]: pgmap v68: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 341 KiB/s wr, 31 op/s 2024-03-12T05:58:00.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:00 smithi082 ceph-mon[36382]: pgmap v69: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 342 KiB/s wr, 35 op/s 2024-03-12T05:58:00.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:00 smithi088 ceph-mon[36585]: pgmap v69: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 342 KiB/s wr, 35 op/s 2024-03-12T05:58:00.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:00 smithi076 ceph-mon[32095]: pgmap v69: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 342 KiB/s wr, 35 op/s 2024-03-12T05:58:02.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:02 smithi082 ceph-mon[36382]: pgmap v70: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-03-12T05:58:02.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:02 smithi088 ceph-mon[36585]: pgmap v70: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-03-12T05:58:02.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:02 smithi076 ceph-mon[32095]: pgmap v70: 97 pgs: 97 active+clean; 76 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 25 KiB/s wr, 12 op/s 2024-03-12T05:58:03.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:03 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:58:03.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:03 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:58:03.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:03 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:58:03.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:03 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:58:03.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:03 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:58:03.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:03 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:58:03.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:03 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:58:03.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:03 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:58:03.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:03 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:58:04.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:04 smithi082 ceph-mon[36382]: pgmap v71: 97 pgs: 97 active+clean; 79 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 429 KiB/s wr, 81 op/s 2024-03-12T05:58:04.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:04 smithi088 ceph-mon[36585]: pgmap v71: 97 pgs: 97 active+clean; 79 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 429 KiB/s wr, 81 op/s 2024-03-12T05:58:04.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:04 smithi076 ceph-mon[32095]: pgmap v71: 97 pgs: 97 active+clean; 79 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 429 KiB/s wr, 81 op/s 2024-03-12T05:58:06.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:06 smithi082 ceph-mon[36382]: pgmap v72: 97 pgs: 97 active+clean; 79 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 405 KiB/s wr, 71 op/s 2024-03-12T05:58:06.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:06 smithi088 ceph-mon[36585]: pgmap v72: 97 pgs: 97 active+clean; 79 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 405 KiB/s wr, 71 op/s 2024-03-12T05:58:06.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:06 smithi076 ceph-mon[32095]: pgmap v72: 97 pgs: 97 active+clean; 79 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 405 KiB/s wr, 71 op/s 2024-03-12T05:58:08.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:08 smithi082 ceph-mon[36382]: pgmap v73: 97 pgs: 97 active+clean; 79 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 405 KiB/s wr, 71 op/s 2024-03-12T05:58:08.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:08 smithi088 ceph-mon[36585]: pgmap v73: 97 pgs: 97 active+clean; 79 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 405 KiB/s wr, 71 op/s 2024-03-12T05:58:08.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:08 smithi076 ceph-mon[32095]: pgmap v73: 97 pgs: 97 active+clean; 79 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 405 KiB/s wr, 71 op/s 2024-03-12T05:58:10.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:10 smithi082 ceph-mon[36382]: pgmap v74: 97 pgs: 97 active+clean; 79 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 407 KiB/s wr, 75 op/s 2024-03-12T05:58:10.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:10 smithi088 ceph-mon[36585]: pgmap v74: 97 pgs: 97 active+clean; 79 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 407 KiB/s wr, 75 op/s 2024-03-12T05:58:10.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:10 smithi076 ceph-mon[32095]: pgmap v74: 97 pgs: 97 active+clean; 79 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 407 KiB/s wr, 75 op/s 2024-03-12T05:58:12.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:12 smithi082 ceph-mon[36382]: pgmap v75: 97 pgs: 97 active+clean; 79 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 406 KiB/s wr, 71 op/s 2024-03-12T05:58:12.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:12 smithi088 ceph-mon[36585]: pgmap v75: 97 pgs: 97 active+clean; 79 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 406 KiB/s wr, 71 op/s 2024-03-12T05:58:12.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:12 smithi076 ceph-mon[32095]: pgmap v75: 97 pgs: 97 active+clean; 79 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 406 KiB/s wr, 71 op/s 2024-03-12T05:58:14.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:14 smithi082 ceph-mon[36382]: pgmap v76: 97 pgs: 97 active+clean; 80 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 411 KiB/s wr, 75 op/s 2024-03-12T05:58:14.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:14 smithi088 ceph-mon[36585]: pgmap v76: 97 pgs: 97 active+clean; 80 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 411 KiB/s wr, 75 op/s 2024-03-12T05:58:14.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:14 smithi076 ceph-mon[32095]: pgmap v76: 97 pgs: 97 active+clean; 80 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 411 KiB/s wr, 75 op/s 2024-03-12T05:58:16.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:16 smithi082 ceph-mon[36382]: pgmap v77: 97 pgs: 97 active+clean; 80 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-03-12T05:58:16.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:16 smithi088 ceph-mon[36585]: pgmap v77: 97 pgs: 97 active+clean; 80 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-03-12T05:58:16.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:16 smithi076 ceph-mon[32095]: pgmap v77: 97 pgs: 97 active+clean; 80 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-03-12T05:58:18.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:18 smithi082 ceph-mon[36382]: pgmap v78: 97 pgs: 97 active+clean; 80 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-03-12T05:58:18.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:18 smithi088 ceph-mon[36585]: pgmap v78: 97 pgs: 97 active+clean; 80 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-03-12T05:58:18.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:18 smithi076 ceph-mon[32095]: pgmap v78: 97 pgs: 97 active+clean; 80 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-03-12T05:58:20.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:20 smithi082 ceph-mon[36382]: pgmap v79: 97 pgs: 97 active+clean; 80 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-03-12T05:58:20.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:20 smithi088 ceph-mon[36585]: pgmap v79: 97 pgs: 97 active+clean; 80 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-03-12T05:58:20.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:20 smithi076 ceph-mon[32095]: pgmap v79: 97 pgs: 97 active+clean; 80 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-03-12T05:58:21.734 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:58:21.735 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:58:21.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.847+0000 7fe62004e640 1 -- 172.21.15.76:0/3128215178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe618073520 msgr2=0x7fe618073900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:21.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.847+0000 7fe62004e640 1 --2- 172.21.15.76:0/3128215178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe618073520 0x7fe618073900 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe614009900 tx=0x7fe61402bcd0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.847+0000 7fe62004e640 1 -- 172.21.15.76:0/3128215178 shutdown_connections 2024-03-12T05:58:21.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.847+0000 7fe62004e640 1 --2- 172.21.15.76:0/3128215178 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe618112130 0x7fe618114520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.847+0000 7fe62004e640 1 --2- 172.21.15.76:0/3128215178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe618073e40 0x7fe618111bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.847+0000 7fe62004e640 1 --2- 172.21.15.76:0/3128215178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe618073520 0x7fe618073900 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.847+0000 7fe62004e640 1 -- 172.21.15.76:0/3128215178 >> 172.21.15.76:0/3128215178 conn(0x7fe6180fcc10 msgr2=0x7fe6180ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:21.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.848+0000 7fe62004e640 1 -- 172.21.15.76:0/3128215178 shutdown_connections 2024-03-12T05:58:21.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.848+0000 7fe62004e640 1 -- 172.21.15.76:0/3128215178 wait complete. 2024-03-12T05:58:21.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.848+0000 7fe62004e640 1 Processor -- start 2024-03-12T05:58:21.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.848+0000 7fe62004e640 1 -- start start 2024-03-12T05:58:21.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.848+0000 7fe62004e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe618073520 0x7fe61819ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.848+0000 7fe62004e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe618073e40 0x7fe61819e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.848+0000 7fe62004e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe618112130 0x7fe6181a2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.848+0000 7fe62004e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6181a2c70 con 0x7fe618073520 2024-03-12T05:58:21.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.848+0000 7fe62004e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6181a2de0 con 0x7fe618112130 2024-03-12T05:58:21.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.848+0000 7fe62004e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6181a2fa0 con 0x7fe618073e40 2024-03-12T05:58:21.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe61ddea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe618073520 0x7fe61819ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe61ddea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe618073520 0x7fe61819ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45396/0 (socket says 172.21.15.76:45396) 2024-03-12T05:58:21.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe61ddea640 1 -- 172.21.15.76:0/2531552432 learned_addr learned my addr 172.21.15.76:0/2531552432 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:21.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe61d5e9640 1 --2- 172.21.15.76:0/2531552432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe618073e40 0x7fe61819e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe61e5eb640 1 --2- 172.21.15.76:0/2531552432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe618112130 0x7fe6181a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe61ddea640 1 -- 172.21.15.76:0/2531552432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe618073e40 msgr2=0x7fe61819e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:21.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe61ddea640 1 --2- 172.21.15.76:0/2531552432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe618073e40 0x7fe61819e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe61ddea640 1 -- 172.21.15.76:0/2531552432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe618112130 msgr2=0x7fe6181a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:21.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe61ddea640 1 --2- 172.21.15.76:0/2531552432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe618112130 0x7fe6181a2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe61ddea640 1 -- 172.21.15.76:0/2531552432 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe614009560 con 0x7fe618073520 2024-03-12T05:58:21.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe61ddea640 1 --2- 172.21.15.76:0/2531552432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe618073520 0x7fe61819ddb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe614005940 tx=0x7fe61402c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:21.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe60effd640 1 -- 172.21.15.76:0/2531552432 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe614004350 con 0x7fe618073520 2024-03-12T05:58:21.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe60effd640 1 -- 172.21.15.76:0/2531552432 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6140044b0 con 0x7fe618073520 2024-03-12T05:58:21.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe62004e640 1 -- 172.21.15.76:0/2531552432 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6181a3220 con 0x7fe618073520 2024-03-12T05:58:21.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe60effd640 1 -- 172.21.15.76:0/2531552432 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe614003d30 con 0x7fe618073520 2024-03-12T05:58:21.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.849+0000 7fe62004e640 1 -- 172.21.15.76:0/2531552432 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6181aab00 con 0x7fe618073520 2024-03-12T05:58:21.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.851+0000 7fe60effd640 1 -- 172.21.15.76:0/2531552432 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe614004df0 con 0x7fe618073520 2024-03-12T05:58:21.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe622c25740 1 Processor -- start 2024-03-12T05:58:21.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe622c25740 1 -- start start 2024-03-12T05:58:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe60effd640 1 --2- 172.21.15.76:0/2531552432 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe6040a1ba0 0x7fe6040a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe622c25740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf882b0 0x560ebcf88690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe622c25740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf91270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe622c25740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf917b0 0x560ebcf93ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe622c25740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebcef7280 con 0x560ebcf88bd0 2024-03-12T05:58:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe622c25740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebcef47b0 con 0x560ebcf882b0 2024-03-12T05:58:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe622c25740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebceeff90 con 0x560ebcf917b0 2024-03-12T05:58:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe61e5eb640 1 --2- 172.21.15.76:0/2531552432 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe6040a1ba0 0x7fe6040a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe61d5e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf91270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe61d5e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf91270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45398/0 (socket says 172.21.15.76:45398) 2024-03-12T05:58:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe61d5e9640 1 -- 172.21.15.76:0/1337750561 learned_addr learned my addr 172.21.15.76:0/1337750561 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe60effd640 1 -- 172.21.15.76:0/2531552432 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe6140efec0 con 0x7fe618073520 2024-03-12T05:58:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.852+0000 7fe61ddea640 1 --2- 172.21.15.76:0/1337750561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf882b0 0x560ebcf88690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.853+0000 7fe61d5e9640 1 -- 172.21.15.76:0/1337750561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf917b0 msgr2=0x560ebcf93ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.853+0000 7fe61d5e9640 1 --2- 172.21.15.76:0/1337750561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf917b0 0x560ebcf93ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.853+0000 7fe61d5e9640 1 -- 172.21.15.76:0/1337750561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf882b0 msgr2=0x560ebcf88690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.853+0000 7fe61d5e9640 1 --2- 172.21.15.76:0/1337750561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf882b0 0x560ebcf88690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.853+0000 7fe61d5e9640 1 -- 172.21.15.76:0/1337750561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ebceae570 con 0x560ebcf88bd0 2024-03-12T05:58:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.853+0000 7fe61ddea640 1 --2- 172.21.15.76:0/1337750561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf882b0 0x560ebcf88690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.853+0000 7fe61e5eb640 1 --2- 172.21.15.76:0/2531552432 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe6040a1ba0 0x7fe6040a4060 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe610005c40 tx=0x7fe610009a80 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.853+0000 7fe61d5e9640 1 --2- 172.21.15.76:0/1337750561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf91270 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe608009c30 tx=0x7fe6080117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee86bb8e64efe97f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.853+0000 7fe603fff640 1 -- 172.21.15.76:0/1337750561 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe608009700 con 0x560ebcf88bd0 2024-03-12T05:58:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.853+0000 7fe603fff640 1 -- 172.21.15.76:0/1337750561 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe608012930 con 0x560ebcf88bd0 2024-03-12T05:58:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.853+0000 7fe603fff640 1 -- 172.21.15.76:0/1337750561 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60800fad0 con 0x560ebcf88bd0 2024-03-12T05:58:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.854+0000 7fe622c25740 1 -- 172.21.15.76:0/1337750561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 msgr2=0x560ebcf91270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.854+0000 7fe622c25740 1 --2- 172.21.15.76:0/1337750561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf91270 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe608009c30 tx=0x7fe6080117b0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.854+0000 7fe622c25740 1 -- 172.21.15.76:0/1337750561 shutdown_connections 2024-03-12T05:58:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.854+0000 7fe622c25740 1 --2- 172.21.15.76:0/1337750561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf917b0 0x560ebcf93ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.854+0000 7fe622c25740 1 --2- 172.21.15.76:0/1337750561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf91270 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.854+0000 7fe622c25740 1 --2- 172.21.15.76:0/1337750561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf882b0 0x560ebcf88690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.854+0000 7fe622c25740 1 -- 172.21.15.76:0/1337750561 >> 172.21.15.76:0/1337750561 conn(0x560ebcf82990 msgr2=0x560ebcf82d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.854+0000 7fe622c25740 1 -- 172.21.15.76:0/1337750561 shutdown_connections 2024-03-12T05:58:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.854+0000 7fe622c25740 1 -- 172.21.15.76:0/1337750561 wait complete. 2024-03-12T05:58:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.854+0000 7fe622c25740 1 -- start start 2024-03-12T05:58:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.854+0000 7fe622c25740 1 Processor -- start 2024-03-12T05:58:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe622c25740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf882b0 0x560ebcf91160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe622c25740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf88bd0 0x560ebcf8e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe622c25740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf917b0 0x560ebcf8eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe622c25740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebcbc3430 con 0x560ebcf882b0 2024-03-12T05:58:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe622c25740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebcb7fe00 con 0x560ebcf88bd0 2024-03-12T05:58:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe622c25740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebce74fd0 con 0x560ebcf917b0 2024-03-12T05:58:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe61ddea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf882b0 0x560ebcf91160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe622c25740 10 client.0 _ll_register_callbacks cb 0x560ebcecafd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe61ddea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf882b0 0x560ebcf91160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45400/0 (socket says 172.21.15.76:45400) 2024-03-12T05:58:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe61ddea640 1 -- 172.21.15.76:0/2849521016 learned_addr learned my addr 172.21.15.76:0/2849521016 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:21.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe61d5e9640 1 --2- 172.21.15.76:0/2849521016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf88bd0 0x560ebcf8e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe61e5eb640 1 --2- 172.21.15.76:0/2849521016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf917b0 0x560ebcf8eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe61ddea640 1 -- 172.21.15.76:0/2849521016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf917b0 msgr2=0x560ebcf8eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:21.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe61ddea640 1 --2- 172.21.15.76:0/2849521016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf917b0 0x560ebcf8eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe61ddea640 1 -- 172.21.15.76:0/2849521016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf88bd0 msgr2=0x560ebcf8e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:21.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe61ddea640 1 --2- 172.21.15.76:0/2849521016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf88bd0 0x560ebcf8e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe61ddea640 1 -- 172.21.15.76:0/2849521016 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe608009860 con 0x560ebcf882b0 2024-03-12T05:58:21.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe61ddea640 1 --2- 172.21.15.76:0/2849521016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf882b0 0x560ebcf91160 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe6140ed800 tx=0x7fe6140bc650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5625443878b000cb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:21.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe6017fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:21.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe6017fa640 1 -- 172.21.15.76:0/2849521016 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6140bcc30 con 0x560ebcf882b0 2024-03-12T05:58:21.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe6017fa640 1 -- 172.21.15.76:0/2849521016 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe61402ce40 con 0x560ebcf882b0 2024-03-12T05:58:21.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.855+0000 7fe622c25740 1 -- 172.21.15.76:0/2849521016 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560ebcbef7e0 con 0x560ebcf882b0 2024-03-12T05:58:21.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.856+0000 7fe6017fa640 1 -- 172.21.15.76:0/2849521016 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6140bd5c0 con 0x560ebcf882b0 2024-03-12T05:58:21.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.856+0000 7fe622c25740 10 client.16107 fetch_fsmap learned FSMap version 24 2024-03-12T05:58:21.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.856+0000 7fe622c25740 1 -- 172.21.15.76:0/2849521016 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560ebceee610 con 0x560ebcf882b0 2024-03-12T05:58:21.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.856+0000 7fe6017fa640 1 -- 172.21.15.76:0/2849521016 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560ebcbef7e0 con 0x560ebcf882b0 2024-03-12T05:58:21.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.856+0000 7fe6017fa640 1 -- 172.21.15.76:0/2849521016 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe61403c070 con 0x560ebcf882b0 2024-03-12T05:58:21.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.856+0000 7fe622c25740 10 client.16107 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:58:21.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.856+0000 7fe622c25740 10 client.16107 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:21.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.856+0000 7fe622c25740 20 client.16107 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:21.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.856+0000 7fe622c25740 1 --2- 172.21.15.76:0/2849521016 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560ebcfa1580 0x560ebcfa3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.856+0000 7fe622c25740 4 client.16107 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:58:21.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.856+0000 7fe622c25740 1 -- 172.21.15.76:0/2849521016 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ebcc2c720 con 0x560ebcfa1580 2024-03-12T05:58:21.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.856+0000 7fe61d5e9640 1 --2- 172.21.15.76:0/2849521016 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560ebcfa1580 0x560ebcfa3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.857+0000 7fe6017fa640 10 client.16107 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:21.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.857+0000 7fe61d5e9640 1 --2- 172.21.15.76:0/2849521016 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560ebcfa1580 0x560ebcfa3990 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=deb0d2a4daeac79c server_cookie=6dc0568f25e5eb28 in_seq=0 out_seq=0 2024-03-12T05:58:21.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.857+0000 7fe6017fa640 1 -- 172.21.15.76:0/2849521016 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560ebcc2c720 con 0x560ebcfa1580 2024-03-12T05:58:21.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.857+0000 7fe6017fa640 10 client.16107 handle_command_reply: tid=0 2024-03-12T05:58:21.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.857+0000 7fe622c25740 1 client.16107 shutdown 2024-03-12T05:58:21.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.857+0000 7fe622c25740 1 -- 172.21.15.76:0/2849521016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf882b0 msgr2=0x560ebcf91160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:21.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.857+0000 7fe622c25740 1 --2- 172.21.15.76:0/2849521016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf882b0 0x560ebcf91160 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe6140ed800 tx=0x7fe6140bc650 comp rx=0 tx=0).stop 2024-03-12T05:58:21.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.858+0000 7fe622c25740 1 -- 172.21.15.76:0/2849521016 shutdown_connections 2024-03-12T05:58:21.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.858+0000 7fe622c25740 1 --2- 172.21.15.76:0/2849521016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf917b0 0x560ebcf8eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.858+0000 7fe622c25740 1 --2- 172.21.15.76:0/2849521016 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf88bd0 0x560ebcf8e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.858+0000 7fe622c25740 1 --2- 172.21.15.76:0/2849521016 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560ebcfa1580 0x560ebcfa3990 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.858+0000 7fe622c25740 1 --2- 172.21.15.76:0/2849521016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf882b0 0x560ebcf91160 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.858+0000 7fe6017fa640 0 client.16107 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:21.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.858+0000 7fe622c25740 1 -- 172.21.15.76:0/2849521016 >> 172.21.15.76:0/2849521016 conn(0x560ebcf82990 msgr2=0x560ebcf84730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:21.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.858+0000 7fe622c25740 1 -- 172.21.15.76:0/2849521016 shutdown_connections 2024-03-12T05:58:21.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.858+0000 7fe622c25740 1 -- 172.21.15.76:0/2849521016 wait complete. 2024-03-12T05:58:21.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.858+0000 7fe622c25740 20 client.16107 trim_cache size 0 max 16384 2024-03-12T05:58:21.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.872+0000 7fe622c25740 1 Processor -- start 2024-03-12T05:58:21.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe622c25740 1 -- start start 2024-03-12T05:58:21.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe622c25740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf882b0 0x560ebcf90c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe622c25740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf98650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe622c25740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf917b0 0x560ebcf98b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe622c25740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebcf837b0 con 0x560ebcf88bd0 2024-03-12T05:58:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe622c25740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebcf7ce20 con 0x560ebcf917b0 2024-03-12T05:58:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe622c25740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebcf82210 con 0x560ebcf882b0 2024-03-12T05:58:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe61d5e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf98650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe61d5e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf98650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45416/0 (socket says 172.21.15.76:45416) 2024-03-12T05:58:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe61d5e9640 1 -- 172.21.15.76:0/2280882336 learned_addr learned my addr 172.21.15.76:0/2280882336 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe61e5eb640 1 --2- 172.21.15.76:0/2280882336 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf917b0 0x560ebcf98b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe61d5e9640 1 -- 172.21.15.76:0/2280882336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf882b0 msgr2=0x560ebcf90c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe61d5e9640 1 --2- 172.21.15.76:0/2280882336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf882b0 0x560ebcf90c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe61d5e9640 1 -- 172.21.15.76:0/2280882336 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf917b0 msgr2=0x560ebcf98b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe61d5e9640 1 --2- 172.21.15.76:0/2280882336 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf917b0 0x560ebcf98b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe61d5e9640 1 -- 172.21.15.76:0/2280882336 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ebce8aac0 con 0x560ebcf88bd0 2024-03-12T05:58:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe61d5e9640 1 --2- 172.21.15.76:0/2280882336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf98650 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe608009f00 tx=0x7fe6080093b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c647f3ca22903244 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe6037fe640 1 -- 172.21.15.76:0/2280882336 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe608012b20 con 0x560ebcf88bd0 2024-03-12T05:58:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.873+0000 7fe6037fe640 1 -- 172.21.15.76:0/2280882336 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe608012c80 con 0x560ebcf88bd0 2024-03-12T05:58:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.874+0000 7fe6037fe640 1 -- 172.21.15.76:0/2280882336 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60800f740 con 0x560ebcf88bd0 2024-03-12T05:58:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.874+0000 7fe622c25740 1 -- 172.21.15.76:0/2280882336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 msgr2=0x560ebcf98650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.874+0000 7fe622c25740 1 --2- 172.21.15.76:0/2280882336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf98650 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe608009f00 tx=0x7fe6080093b0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.874+0000 7fe622c25740 1 -- 172.21.15.76:0/2280882336 shutdown_connections 2024-03-12T05:58:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.874+0000 7fe622c25740 1 --2- 172.21.15.76:0/2280882336 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf917b0 0x560ebcf98b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.874+0000 7fe622c25740 1 --2- 172.21.15.76:0/2280882336 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf98650 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.874+0000 7fe622c25740 1 --2- 172.21.15.76:0/2280882336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf882b0 0x560ebcf90c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.874+0000 7fe622c25740 1 -- 172.21.15.76:0/2280882336 >> 172.21.15.76:0/2280882336 conn(0x560ebcf82990 msgr2=0x560ebcf8ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:21.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.874+0000 7fe622c25740 1 -- 172.21.15.76:0/2280882336 shutdown_connections 2024-03-12T05:58:21.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.874+0000 7fe622c25740 1 -- 172.21.15.76:0/2280882336 wait complete. 2024-03-12T05:58:21.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.874+0000 7fe622c25740 1 -- start start 2024-03-12T05:58:21.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.874+0000 7fe622c25740 1 Processor -- start 2024-03-12T05:58:21.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe622c25740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf882b0 0x560ebcf98230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe622c25740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf98770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe622c25740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf917b0 0x560ebcf95730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe622c25740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebcf95d40 con 0x560ebcf88bd0 2024-03-12T05:58:21.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe622c25740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebcf95eb0 con 0x560ebcf882b0 2024-03-12T05:58:21.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe622c25740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebcf96070 con 0x560ebcf917b0 2024-03-12T05:58:21.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe622c25740 10 client.0 _ll_register_callbacks cb 0x7fe608009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:21.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe61d5e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf98770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe61d5e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf98770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45430/0 (socket says 172.21.15.76:45430) 2024-03-12T05:58:21.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe61d5e9640 1 -- 172.21.15.76:0/1190122135 learned_addr learned my addr 172.21.15.76:0/1190122135 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:21.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe61ddea640 1 --2- 172.21.15.76:0/1190122135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf882b0 0x560ebcf98230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe61d5e9640 1 -- 172.21.15.76:0/1190122135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf917b0 msgr2=0x560ebcf95730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:21.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe61d5e9640 1 --2- 172.21.15.76:0/1190122135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf917b0 0x560ebcf95730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe61d5e9640 1 -- 172.21.15.76:0/1190122135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf882b0 msgr2=0x560ebcf98230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:21.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe61d5e9640 1 --2- 172.21.15.76:0/1190122135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf882b0 0x560ebcf98230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe61d5e9640 1 -- 172.21.15.76:0/1190122135 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60801e040 con 0x560ebcf88bd0 2024-03-12T05:58:21.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe61d5e9640 1 --2- 172.21.15.76:0/1190122135 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf98770 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x560ebcbef920 tx=0x7fe608012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4191b4de83e9b11 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:21.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.875+0000 7fe6017fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:21.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe6017fa640 1 -- 172.21.15.76:0/1190122135 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe608005570 con 0x560ebcf88bd0 2024-03-12T05:58:21.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe6017fa640 1 -- 172.21.15.76:0/1190122135 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe6080056d0 con 0x560ebcf88bd0 2024-03-12T05:58:21.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe622c25740 1 -- 172.21.15.76:0/1190122135 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe608012930 con 0x560ebcf88bd0 2024-03-12T05:58:21.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe6017fa640 1 -- 172.21.15.76:0/1190122135 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe608012450 con 0x560ebcf88bd0 2024-03-12T05:58:21.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe622c25740 10 client.16113 fetch_fsmap learned FSMap version 24 2024-03-12T05:58:21.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe622c25740 1 -- 172.21.15.76:0/1190122135 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560ebceb06a0 con 0x560ebcf88bd0 2024-03-12T05:58:21.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe6017fa640 1 -- 172.21.15.76:0/1190122135 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe608012930 con 0x560ebcf88bd0 2024-03-12T05:58:21.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe6017fa640 1 -- 172.21.15.76:0/1190122135 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe60802c020 con 0x560ebcf88bd0 2024-03-12T05:58:21.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe622c25740 10 client.16113 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:58:21.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe622c25740 10 client.16113 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:21.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe622c25740 20 client.16113 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:21.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe622c25740 1 --2- 172.21.15.76:0/1190122135 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560ebcff97a0 0x560ebcffbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:21.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe622c25740 4 client.16113 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:58:21.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe622c25740 1 -- 172.21.15.76:0/1190122135 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560ebcc7b890 con 0x560ebcff97a0 2024-03-12T05:58:21.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.876+0000 7fe61ddea640 1 --2- 172.21.15.76:0/1190122135 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560ebcff97a0 0x560ebcffbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:21.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe6017fa640 10 client.16113 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:21.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe61ddea640 1 --2- 172.21.15.76:0/1190122135 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560ebcff97a0 0x560ebcffbbc0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=657345df5018554 server_cookie=c723616d2b184d43 in_seq=0 out_seq=0 2024-03-12T05:58:21.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe6017fa640 1 -- 172.21.15.76:0/1190122135 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560ebcc7b890 con 0x560ebcff97a0 2024-03-12T05:58:21.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe6017fa640 10 client.16113 handle_command_reply: tid=0 2024-03-12T05:58:21.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe622c25740 1 client.16113 shutdown 2024-03-12T05:58:21.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe622c25740 1 -- 172.21.15.76:0/1190122135 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 msgr2=0x560ebcf98770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:21.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe622c25740 1 --2- 172.21.15.76:0/1190122135 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf98770 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x560ebcbef920 tx=0x7fe608012ba0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe622c25740 1 -- 172.21.15.76:0/1190122135 shutdown_connections 2024-03-12T05:58:21.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe622c25740 1 --2- 172.21.15.76:0/1190122135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560ebcf917b0 0x560ebcf95730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe622c25740 1 --2- 172.21.15.76:0/1190122135 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560ebcff97a0 0x560ebcffbbc0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe622c25740 1 --2- 172.21.15.76:0/1190122135 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560ebcf88bd0 0x560ebcf98770 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe622c25740 1 --2- 172.21.15.76:0/1190122135 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560ebcf882b0 0x560ebcf98230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe6017fa640 0 client.16113 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:21.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe622c25740 1 -- 172.21.15.76:0/1190122135 >> 172.21.15.76:0/1190122135 conn(0x560ebcf82990 msgr2=0x560ebcf811c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:21.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe622c25740 1 -- 172.21.15.76:0/1190122135 shutdown_connections 2024-03-12T05:58:21.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe622c25740 1 -- 172.21.15.76:0/1190122135 wait complete. 2024-03-12T05:58:21.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.877+0000 7fe622c25740 20 client.16113 trim_cache size 0 max 16384 2024-03-12T05:58:21.895 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:58:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T05:58:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T05:58:21.896 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:58:21.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.878+0000 7fe62004e640 1 -- 172.21.15.76:0/2531552432 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe6040a1ba0 msgr2=0x7fe6040a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:21.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.878+0000 7fe62004e640 1 --2- 172.21.15.76:0/2531552432 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe6040a1ba0 0x7fe6040a4060 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe610005c40 tx=0x7fe610009a80 comp rx=0 tx=0).stop 2024-03-12T05:58:21.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.878+0000 7fe62004e640 1 -- 172.21.15.76:0/2531552432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe618073520 msgr2=0x7fe61819ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:21.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.878+0000 7fe62004e640 1 --2- 172.21.15.76:0/2531552432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe618073520 0x7fe61819ddb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe614005940 tx=0x7fe61402c710 comp rx=0 tx=0).stop 2024-03-12T05:58:21.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.878+0000 7fe62004e640 1 -- 172.21.15.76:0/2531552432 shutdown_connections 2024-03-12T05:58:21.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.878+0000 7fe62004e640 1 --2- 172.21.15.76:0/2531552432 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe6040a1ba0 0x7fe6040a4060 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.878+0000 7fe62004e640 1 --2- 172.21.15.76:0/2531552432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe618112130 0x7fe6181a2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.878+0000 7fe62004e640 1 --2- 172.21.15.76:0/2531552432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe618073e40 0x7fe61819e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.878+0000 7fe62004e640 1 --2- 172.21.15.76:0/2531552432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe618073520 0x7fe61819ddb0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:21.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.878+0000 7fe62004e640 1 -- 172.21.15.76:0/2531552432 >> 172.21.15.76:0/2531552432 conn(0x7fe6180fcc10 msgr2=0x7fe6180fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:21.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.878+0000 7fe62004e640 1 -- 172.21.15.76:0/2531552432 shutdown_connections 2024-03-12T05:58:21.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.878+0000 7fe62004e640 1 -- 172.21.15.76:0/2531552432 wait complete. 2024-03-12T05:58:21.899 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:58:21.899 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:58:22.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.997+0000 7fbcaeef4640 1 -- 172.21.15.76:0/2953058070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbca8101390 msgr2=0x7fbca8101770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.997+0000 7fbcaeef4640 1 --2- 172.21.15.76:0/2953058070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbca8101390 0x7fbca8101770 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbca4009930 tx=0x7fbca402bc30 comp rx=0 tx=0).stop 2024-03-12T05:58:22.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.997+0000 7fbcaeef4640 1 -- 172.21.15.76:0/2953058070 shutdown_connections 2024-03-12T05:58:22.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.997+0000 7fbcaeef4640 1 --2- 172.21.15.76:0/2953058070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbca8106460 0x7fbca810e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.997+0000 7fbcaeef4640 1 --2- 172.21.15.76:0/2953058070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbca8101d40 0x7fbca8105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.997+0000 7fbcaeef4640 1 --2- 172.21.15.76:0/2953058070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbca8101390 0x7fbca8101770 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.997+0000 7fbcaeef4640 1 -- 172.21.15.76:0/2953058070 >> 172.21.15.76:0/2953058070 conn(0x7fbca80fcc30 msgr2=0x7fbca80ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.997+0000 7fbcaeef4640 1 -- 172.21.15.76:0/2953058070 shutdown_connections 2024-03-12T05:58:22.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.997+0000 7fbcaeef4640 1 -- 172.21.15.76:0/2953058070 wait complete. 2024-03-12T05:58:22.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbcaeef4640 1 Processor -- start 2024-03-12T05:58:22.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbcaeef4640 1 -- start start 2024-03-12T05:58:22.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbcaeef4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbca8101390 0x7fbca819d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbcaeef4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbca8106460 0x7fbca819db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbcaeef4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbca8197680 0x7fbca8197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbcaeef4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca8198150 con 0x7fbca8106460 2024-03-12T05:58:22.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbcaeef4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca81982c0 con 0x7fbca8101390 2024-03-12T05:58:22.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbcaeef4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca8198480 con 0x7fbca8197680 2024-03-12T05:58:22.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbc9ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbca8106460 0x7fbca819db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbc9ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbca8106460 0x7fbca819db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45446/0 (socket says 172.21.15.76:45446) 2024-03-12T05:58:22.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbc9ffff640 1 -- 172.21.15.76:0/4052172301 learned_addr learned my addr 172.21.15.76:0/4052172301 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbcacc90640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbca8101390 0x7fbca819d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbcad491640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbca8197680 0x7fbca8197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbc9ffff640 1 -- 172.21.15.76:0/4052172301 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbca8197680 msgr2=0x7fbca8197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbc9ffff640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbca8197680 0x7fbca8197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbc9ffff640 1 -- 172.21.15.76:0/4052172301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbca8101390 msgr2=0x7fbca819d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbc9ffff640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbca8101390 0x7fbca819d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbc9ffff640 1 -- 172.21.15.76:0/4052172301 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca4009560 con 0x7fbca8106460 2024-03-12T05:58:22.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.998+0000 7fbcad491640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbca8197680 0x7fbca8197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:58:22.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.999+0000 7fbc9ffff640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbca8106460 0x7fbca819db00 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fbc9800d770 tx=0x7fbc9800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.999+0000 7fbc9dffb640 1 -- 172.21.15.76:0/4052172301 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc98009c30 con 0x7fbca8106460 2024-03-12T05:58:22.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.999+0000 7fbc9dffb640 1 -- 172.21.15.76:0/4052172301 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc9800de80 con 0x7fbca8106460 2024-03-12T05:58:22.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.999+0000 7fbc9dffb640 1 -- 172.21.15.76:0/4052172301 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9800f6b0 con 0x7fbca8106460 2024-03-12T05:58:22.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.999+0000 7fbcaeef4640 1 -- 172.21.15.76:0/4052172301 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca81a4420 con 0x7fbca8106460 2024-03-12T05:58:22.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:21.999+0000 7fbcaeef4640 1 -- 172.21.15.76:0/4052172301 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca81a4950 con 0x7fbca8106460 2024-03-12T05:58:22.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.000+0000 7fbcaeef4640 1 -- 172.21.15.76:0/4052172301 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca8102860 con 0x7fbca8106460 2024-03-12T05:58:22.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.001+0000 7fbc9dffb640 1 -- 172.21.15.76:0/4052172301 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fbc98010460 con 0x7fbca8106460 2024-03-12T05:58:22.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.002+0000 7fbc9dffb640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbc900a1c20 0x7fbc900a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.002+0000 7fbcacc90640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbc900a1c20 0x7fbc900a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.002+0000 7fbc9dffb640 1 -- 172.21.15.76:0/4052172301 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbc980cd4c0 con 0x7fbca8106460 2024-03-12T05:58:22.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.002+0000 7fbcacc90640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbc900a1c20 0x7fbc900a40e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbca4005ab0 tx=0x7fbca4005fd0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.007+0000 7fbc9dffb640 1 -- 172.21.15.76:0/4052172301 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbc980963c0 con 0x7fbca8106460 2024-03-12T05:58:22.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.222+0000 7fbcaeef4640 1 -- 172.21.15.76:0/4052172301 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbca8109690 con 0x7fbca8106460 2024-03-12T05:58:22.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.223+0000 7fbc9dffb640 1 -- 172.21.15.76:0/4052172301 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11331 (secure 0 0 0) 0x7fbc980b1c60 con 0x7fbca8106460 2024-03-12T05:58:22.228 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:58:22.228 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:57:27.157038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:58:22.228 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 24 2024-03-12T05:58:22.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.225+0000 7fbcaeef4640 1 -- 172.21.15.76:0/4052172301 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbc900a1c20 msgr2=0x7fbc900a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.225+0000 7fbcaeef4640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbc900a1c20 0x7fbc900a40e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbca4005ab0 tx=0x7fbca4005fd0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.225+0000 7fbcaeef4640 1 -- 172.21.15.76:0/4052172301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbca8106460 msgr2=0x7fbca819db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.225+0000 7fbcaeef4640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbca8106460 0x7fbca819db00 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fbc9800d770 tx=0x7fbc9800dc30 comp rx=0 tx=0).stop 2024-03-12T05:58:22.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.225+0000 7fbcaeef4640 1 -- 172.21.15.76:0/4052172301 shutdown_connections 2024-03-12T05:58:22.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.225+0000 7fbcaeef4640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbc900a1c20 0x7fbc900a40e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.225+0000 7fbcaeef4640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbca8197680 0x7fbca8197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.225+0000 7fbcaeef4640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbca8106460 0x7fbca819db00 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.225+0000 7fbcaeef4640 1 --2- 172.21.15.76:0/4052172301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbca8101390 0x7fbca819d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.225+0000 7fbcaeef4640 1 -- 172.21.15.76:0/4052172301 >> 172.21.15.76:0/4052172301 conn(0x7fbca80fcc30 msgr2=0x7fbca80fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.225+0000 7fbcaeef4640 1 -- 172.21.15.76:0/4052172301 shutdown_connections 2024-03-12T05:58:22.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.225+0000 7fbcaeef4640 1 -- 172.21.15.76:0/4052172301 wait complete. 2024-03-12T05:58:22.241 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:58:22.242 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:58:22.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.342+0000 7ff5a30eb640 1 -- 172.21.15.76:0/2605066391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff59c100280 msgr2=0x7ff59c100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.342+0000 7ff5a30eb640 1 --2- 172.21.15.76:0/2605066391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff59c100280 0x7ff59c100660 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff590009930 tx=0x7ff59002bc30 comp rx=0 tx=0).stop 2024-03-12T05:58:22.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.342+0000 7ff5a30eb640 1 -- 172.21.15.76:0/2605066391 shutdown_connections 2024-03-12T05:58:22.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.342+0000 7ff5a30eb640 1 --2- 172.21.15.76:0/2605066391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff59c105350 0x7ff59c10cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.342+0000 7ff5a30eb640 1 --2- 172.21.15.76:0/2605066391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff59c100c30 0x7ff59c104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.342+0000 7ff5a30eb640 1 --2- 172.21.15.76:0/2605066391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff59c100280 0x7ff59c100660 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.342+0000 7ff5a30eb640 1 -- 172.21.15.76:0/2605066391 >> 172.21.15.76:0/2605066391 conn(0x7ff59c077170 msgr2=0x7ff59c077580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.342+0000 7ff5a30eb640 1 -- 172.21.15.76:0/2605066391 shutdown_connections 2024-03-12T05:58:22.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.342+0000 7ff5a30eb640 1 -- 172.21.15.76:0/2605066391 wait complete. 2024-03-12T05:58:22.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a30eb640 1 Processor -- start 2024-03-12T05:58:22.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a30eb640 1 -- start start 2024-03-12T05:58:22.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a30eb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff59c100280 0x7ff59c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a30eb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff59c105350 0x7ff59c198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a30eb640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff59c199120 0x7ff59c1a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a30eb640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c1a3d70 con 0x7ff59c100280 2024-03-12T05:58:22.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a30eb640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c1a3ee0 con 0x7ff59c199120 2024-03-12T05:58:22.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a30eb640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c1a40a0 con 0x7ff59c105350 2024-03-12T05:58:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a0e87640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff59c100280 0x7ff59c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a0e87640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff59c100280 0x7ff59c198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45466/0 (socket says 172.21.15.76:45466) 2024-03-12T05:58:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a0e87640 1 -- 172.21.15.76:0/857595159 learned_addr learned my addr 172.21.15.76:0/857595159 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff59bfff640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff59c105350 0x7ff59c198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a1688640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff59c199120 0x7ff59c1a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a0e87640 1 -- 172.21.15.76:0/857595159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff59c105350 msgr2=0x7ff59c198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a0e87640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff59c105350 0x7ff59c198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a0e87640 1 -- 172.21.15.76:0/857595159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff59c199120 msgr2=0x7ff59c1a36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a0e87640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff59c199120 0x7ff59c1a36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a0e87640 1 -- 172.21.15.76:0/857595159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff590009560 con 0x7ff59c100280 2024-03-12T05:58:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff59bfff640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff59c105350 0x7ff59c198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.343+0000 7ff5a0e87640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff59c100280 0x7ff59c198650 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff590009230 tx=0x7ff590004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.344+0000 7ff599ffb640 1 -- 172.21.15.76:0/857595159 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590004440 con 0x7ff59c100280 2024-03-12T05:58:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.344+0000 7ff599ffb640 1 -- 172.21.15.76:0/857595159 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5900052d0 con 0x7ff59c100280 2024-03-12T05:58:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.344+0000 7ff5a30eb640 1 -- 172.21.15.76:0/857595159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59c1a4320 con 0x7ff59c100280 2024-03-12T05:58:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.344+0000 7ff599ffb640 1 -- 172.21.15.76:0/857595159 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590003d70 con 0x7ff59c100280 2024-03-12T05:58:22.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.344+0000 7ff5a30eb640 1 -- 172.21.15.76:0/857595159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59c1a4830 con 0x7ff59c100280 2024-03-12T05:58:22.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.346+0000 7ff599ffb640 1 -- 172.21.15.76:0/857595159 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff590005440 con 0x7ff59c100280 2024-03-12T05:58:22.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.346+0000 7ff5a5d98740 1 Processor -- start 2024-03-12T05:58:22.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.346+0000 7ff5a5d98740 1 -- start start 2024-03-12T05:58:22.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.346+0000 7ff5a5d98740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344250 0x55e579344630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.346+0000 7ff5a5d98740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e579344b70 0x55e57934d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a5d98740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e57934d750 0x55e57934fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a5d98740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5792ff960 con 0x55e579344b70 2024-03-12T05:58:22.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a5d98740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5792b3310 con 0x55e579344250 2024-03-12T05:58:22.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a5d98740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5792b0840 con 0x55e57934d750 2024-03-12T05:58:22.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff599ffb640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff58c0a1ba0 0x7ff58c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a0e87640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e579344b70 0x55e57934d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a0e87640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e579344b70 0x55e57934d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45472/0 (socket says 172.21.15.76:45472) 2024-03-12T05:58:22.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a0e87640 1 -- 172.21.15.76:0/3858192704 learned_addr learned my addr 172.21.15.76:0/3858192704 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a1688640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff58c0a1ba0 0x7ff58c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff59bfff640 1 --2- 172.21.15.76:0/3858192704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e57934d750 0x55e57934fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a1688640 1 --2- 172.21.15.76:0/3858192704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344250 0x55e579344630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff599ffb640 1 -- 172.21.15.76:0/857595159 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff5900ef9f0 con 0x7ff59c100280 2024-03-12T05:58:22.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a0e87640 1 -- 172.21.15.76:0/3858192704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e57934d750 msgr2=0x55e57934fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a0e87640 1 --2- 172.21.15.76:0/3858192704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e57934d750 0x55e57934fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a0e87640 1 -- 172.21.15.76:0/3858192704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344250 msgr2=0x55e579344630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a0e87640 1 --2- 172.21.15.76:0/3858192704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344250 0x55e579344630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a0e87640 1 -- 172.21.15.76:0/3858192704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e57926e060 con 0x55e579344b70 2024-03-12T05:58:22.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a0e87640 1 --2- 172.21.15.76:0/3858192704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e579344b70 0x55e57934d210 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff590000c00 tx=0x7ff590009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f429fb5d8fc74e1d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff5a1688640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff58c0a1ba0 0x7ff58c0a4060 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff5940092f0 tx=0x7ff594009c30 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff586ffd640 1 -- 172.21.15.76:0/3858192704 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5900bc760 con 0x55e579344b70 2024-03-12T05:58:22.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff586ffd640 1 -- 172.21.15.76:0/3858192704 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5900c08b0 con 0x55e579344b70 2024-03-12T05:58:22.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.347+0000 7ff586ffd640 1 -- 172.21.15.76:0/3858192704 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5900bab50 con 0x55e579344b70 2024-03-12T05:58:22.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.348+0000 7ff5a5d98740 1 -- 172.21.15.76:0/3858192704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e579344b70 msgr2=0x55e57934d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.348+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/3858192704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e579344b70 0x55e57934d210 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff590000c00 tx=0x7ff590009260 comp rx=0 tx=0).stop 2024-03-12T05:58:22.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.348+0000 7ff5a5d98740 1 -- 172.21.15.76:0/3858192704 shutdown_connections 2024-03-12T05:58:22.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.348+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/3858192704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e57934d750 0x55e57934fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.348+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/3858192704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e579344b70 0x55e57934d210 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.348+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/3858192704 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344250 0x55e579344630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.348+0000 7ff5a5d98740 1 -- 172.21.15.76:0/3858192704 >> 172.21.15.76:0/3858192704 conn(0x55e57933e980 msgr2=0x55e57933ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.348+0000 7ff5a5d98740 1 -- 172.21.15.76:0/3858192704 shutdown_connections 2024-03-12T05:58:22.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.348+0000 7ff5a5d98740 1 -- 172.21.15.76:0/3858192704 wait complete. 2024-03-12T05:58:22.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.348+0000 7ff5a5d98740 1 -- start start 2024-03-12T05:58:22.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.348+0000 7ff5a5d98740 1 Processor -- start 2024-03-12T05:58:22.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a5d98740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e579344250 0x55e57934a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a5d98740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344b70 0x55e57934ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a5d98740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e57934d750 0x55e57934b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a5d98740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5792ac020 con 0x55e57934d750 2024-03-12T05:58:22.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a5d98740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e578f7f430 con 0x55e579344b70 2024-03-12T05:58:22.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a5d98740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e578f3be00 con 0x55e579344250 2024-03-12T05:58:22.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a5d98740 10 client.0 _ll_register_callbacks cb 0x55e579253cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:22.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a1688640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e57934d750 0x55e57934b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a1688640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e57934d750 0x55e57934b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45484/0 (socket says 172.21.15.76:45484) 2024-03-12T05:58:22.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a1688640 1 -- 172.21.15.76:0/1900067591 learned_addr learned my addr 172.21.15.76:0/1900067591 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a0e87640 1 --2- 172.21.15.76:0/1900067591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e579344250 0x55e57934a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff59bfff640 1 --2- 172.21.15.76:0/1900067591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344b70 0x55e57934ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a1688640 1 -- 172.21.15.76:0/1900067591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e579344250 msgr2=0x55e57934a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a1688640 1 --2- 172.21.15.76:0/1900067591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e579344250 0x55e57934a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a1688640 1 -- 172.21.15.76:0/1900067591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344b70 msgr2=0x55e57934ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a1688640 1 --2- 172.21.15.76:0/1900067591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344b70 0x55e57934ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a1688640 1 -- 172.21.15.76:0/1900067591 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59c1a4830 con 0x55e57934d750 2024-03-12T05:58:22.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a0e87640 1 --2- 172.21.15.76:0/1900067591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e579344250 0x55e57934a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:22.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff5a1688640 1 --2- 172.21.15.76:0/1900067591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e57934d750 0x55e57934b240 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff5940144e0 tx=0x7ff59401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb34b876b7d8e21 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.349+0000 7ff57bfff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:22.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.350+0000 7ff57bfff640 1 -- 172.21.15.76:0/1900067591 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff594012940 con 0x55e57934d750 2024-03-12T05:58:22.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.350+0000 7ff57bfff640 1 -- 172.21.15.76:0/1900067591 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff594012aa0 con 0x55e57934d750 2024-03-12T05:58:22.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.350+0000 7ff5a5d98740 1 -- 172.21.15.76:0/1900067591 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e578fab7e0 con 0x55e57934d750 2024-03-12T05:58:22.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.350+0000 7ff57bfff640 1 -- 172.21.15.76:0/1900067591 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff594012c60 con 0x55e57934d750 2024-03-12T05:58:22.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.350+0000 7ff5a5d98740 10 client.16131 fetch_fsmap learned FSMap version 24 2024-03-12T05:58:22.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.350+0000 7ff5a5d98740 1 -- 172.21.15.76:0/1900067591 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55e5792aa6a0 con 0x55e57934d750 2024-03-12T05:58:22.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.350+0000 7ff57bfff640 1 -- 172.21.15.76:0/1900067591 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e578fab7e0 con 0x55e57934d750 2024-03-12T05:58:22.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.350+0000 7ff57bfff640 1 -- 172.21.15.76:0/1900067591 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff594033450 con 0x55e57934d750 2024-03-12T05:58:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.350+0000 7ff5a5d98740 10 client.16131 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:58:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.350+0000 7ff5a5d98740 10 client.16131 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:58:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.350+0000 7ff5a5d98740 20 client.16131 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.350+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/1900067591 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e57935d920 0x55e57935fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.350+0000 7ff5a5d98740 4 client.16131 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:58:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.350+0000 7ff5a5d98740 1 -- 172.21.15.76:0/1900067591 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e578fe8720 con 0x55e57935d920 2024-03-12T05:58:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.351+0000 7ff5a0e87640 1 --2- 172.21.15.76:0/1900067591 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e57935d920 0x55e57935fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.351+0000 7ff5a0e87640 1 --2- 172.21.15.76:0/1900067591 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e57935d920 0x55e57935fd40 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=78bf752ee050b0b0 server_cookie=6b24ca564c2e3501 in_seq=0 out_seq=0 2024-03-12T05:58:22.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.351+0000 7ff57bfff640 10 client.16131 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:58:22.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff57bfff640 1 -- 172.21.15.76:0/1900067591 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e578fe8720 con 0x55e57935d920 2024-03-12T05:58:22.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff57bfff640 10 client.16131 handle_command_reply: tid=0 2024-03-12T05:58:22.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff5a5d98740 1 client.16131 shutdown 2024-03-12T05:58:22.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff5a5d98740 1 -- 172.21.15.76:0/1900067591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e57934d750 msgr2=0x55e57934b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/1900067591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e57934d750 0x55e57934b240 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff5940144e0 tx=0x7ff59401b000 comp rx=0 tx=0).stop 2024-03-12T05:58:22.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff5a5d98740 1 -- 172.21.15.76:0/1900067591 shutdown_connections 2024-03-12T05:58:22.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/1900067591 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e57935d920 0x55e57935fd40 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/1900067591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e57934d750 0x55e57934b240 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/1900067591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344b70 0x55e57934ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/1900067591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e579344250 0x55e57934a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff57bfff640 0 client.16131 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:58:22.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff5a5d98740 1 -- 172.21.15.76:0/1900067591 >> 172.21.15.76:0/1900067591 conn(0x55e57933e980 msgr2=0x55e5793406d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff5a5d98740 1 -- 172.21.15.76:0/1900067591 shutdown_connections 2024-03-12T05:58:22.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff5a5d98740 1 -- 172.21.15.76:0/1900067591 wait complete. 2024-03-12T05:58:22.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.352+0000 7ff5a5d98740 20 client.16131 trim_cache size 0 max 16384 2024-03-12T05:58:22.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.363+0000 7ff5a5d98740 1 Processor -- start 2024-03-12T05:58:22.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.363+0000 7ff5a5d98740 1 -- start start 2024-03-12T05:58:22.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a5d98740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344250 0x55e57934cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a5d98740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e579344b70 0x55e57934b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a5d98740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e57934d750 0x55e579347a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a5d98740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e579231130 con 0x55e57934d750 2024-03-12T05:58:22.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a5d98740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e57935a3f0 con 0x55e579344250 2024-03-12T05:58:22.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a5d98740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e57933f980 con 0x55e579344b70 2024-03-12T05:58:22.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a1688640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e57934d750 0x55e579347a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a1688640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e57934d750 0x55e579347a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45488/0 (socket says 172.21.15.76:45488) 2024-03-12T05:58:22.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a1688640 1 -- 172.21.15.76:0/3771547546 learned_addr learned my addr 172.21.15.76:0/3771547546 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff59bfff640 1 --2- 172.21.15.76:0/3771547546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e579344b70 0x55e57934b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a0e87640 1 --2- 172.21.15.76:0/3771547546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344250 0x55e57934cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a1688640 1 -- 172.21.15.76:0/3771547546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e579344b70 msgr2=0x55e57934b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a1688640 1 --2- 172.21.15.76:0/3771547546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e579344b70 0x55e57934b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a1688640 1 -- 172.21.15.76:0/3771547546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344250 msgr2=0x55e57934cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a1688640 1 --2- 172.21.15.76:0/3771547546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344250 0x55e57934cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a1688640 1 -- 172.21.15.76:0/3771547546 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e579246c20 con 0x55e57934d750 2024-03-12T05:58:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5a1688640 1 --2- 172.21.15.76:0/3771547546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e57934d750 0x55e579347a80 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff594009af0 tx=0x7ff594009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a85d990c9078ac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5867fc640 1 -- 172.21.15.76:0/3771547546 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff594033450 con 0x55e57934d750 2024-03-12T05:58:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5867fc640 1 -- 172.21.15.76:0/3771547546 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff59401a040 con 0x55e57934d750 2024-03-12T05:58:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.364+0000 7ff5867fc640 1 -- 172.21.15.76:0/3771547546 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff594012a20 con 0x55e57934d750 2024-03-12T05:58:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.365+0000 7ff5a5d98740 1 -- 172.21.15.76:0/3771547546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e57934d750 msgr2=0x55e579347a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.365+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/3771547546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e57934d750 0x55e579347a80 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff594009af0 tx=0x7ff594009e40 comp rx=0 tx=0).stop 2024-03-12T05:58:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.365+0000 7ff5a5d98740 1 -- 172.21.15.76:0/3771547546 shutdown_connections 2024-03-12T05:58:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.365+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/3771547546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e57934d750 0x55e579347a80 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.365+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/3771547546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e579344b70 0x55e57934b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.365+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/3771547546 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344250 0x55e57934cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.365+0000 7ff5a5d98740 1 -- 172.21.15.76:0/3771547546 >> 172.21.15.76:0/3771547546 conn(0x55e57933e980 msgr2=0x55e579345b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.365+0000 7ff5a5d98740 1 -- 172.21.15.76:0/3771547546 shutdown_connections 2024-03-12T05:58:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.365+0000 7ff5a5d98740 1 -- 172.21.15.76:0/3771547546 wait complete. 2024-03-12T05:58:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.365+0000 7ff5a5d98740 1 -- start start 2024-03-12T05:58:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.365+0000 7ff5a5d98740 1 Processor -- start 2024-03-12T05:58:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a5d98740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e579344250 0x55e579352d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a5d98740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344b70 0x55e579353260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a5d98740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e57934d750 0x55e5793537a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a5d98740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e579354cc0 con 0x55e579344250 2024-03-12T05:58:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a5d98740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5793b3e10 con 0x55e579344b70 2024-03-12T05:58:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a5d98740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5793b3f50 con 0x55e57934d750 2024-03-12T05:58:22.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a5d98740 10 client.0 _ll_register_callbacks cb 0x7ff5900bc530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:22.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a0e87640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e579344250 0x55e579352d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a0e87640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e579344250 0x55e579352d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45502/0 (socket says 172.21.15.76:45502) 2024-03-12T05:58:22.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a0e87640 1 -- 172.21.15.76:0/2963483738 learned_addr learned my addr 172.21.15.76:0/2963483738 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff59bfff640 1 --2- 172.21.15.76:0/2963483738 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344b70 0x55e579353260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a1688640 1 --2- 172.21.15.76:0/2963483738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e57934d750 0x55e5793537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a0e87640 1 -- 172.21.15.76:0/2963483738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e57934d750 msgr2=0x55e5793537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a0e87640 1 --2- 172.21.15.76:0/2963483738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e57934d750 0x55e5793537a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a0e87640 1 -- 172.21.15.76:0/2963483738 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344b70 msgr2=0x55e579353260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a0e87640 1 --2- 172.21.15.76:0/2963483738 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344b70 0x55e579353260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a0e87640 1 -- 172.21.15.76:0/2963483738 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5792aa6a0 con 0x55e579344250 2024-03-12T05:58:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a0e87640 1 --2- 172.21.15.76:0/2963483738 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e579344250 0x55e579352d20 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff5900ed770 tx=0x7ff59002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f90eaa0e7fe250 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff57bfff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff57bfff640 1 -- 172.21.15.76:0/2963483738 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5900c4f00 con 0x55e579344250 2024-03-12T05:58:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff57bfff640 1 -- 172.21.15.76:0/2963483738 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff5900bc760 con 0x55e579344250 2024-03-12T05:58:22.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a5d98740 1 -- 172.21.15.76:0/2963483738 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff5900c08b0 con 0x55e579344250 2024-03-12T05:58:22.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff57bfff640 1 -- 172.21.15.76:0/2963483738 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5900baa60 con 0x55e579344250 2024-03-12T05:58:22.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a5d98740 10 client.16137 fetch_fsmap learned FSMap version 24 2024-03-12T05:58:22.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.366+0000 7ff5a5d98740 1 -- 172.21.15.76:0/2963483738 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55e57926c8c0 con 0x55e579344250 2024-03-12T05:58:22.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.367+0000 7ff57bfff640 1 -- 172.21.15.76:0/2963483738 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5900c08b0 con 0x55e579344250 2024-03-12T05:58:22.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.367+0000 7ff57bfff640 1 -- 172.21.15.76:0/2963483738 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff59003c070 con 0x55e579344250 2024-03-12T05:58:22.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.367+0000 7ff5a5d98740 10 client.16137 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:58:22.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.367+0000 7ff5a5d98740 10 client.16137 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:58:22.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.367+0000 7ff5a5d98740 20 client.16137 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:22.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.367+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/2963483738 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e5793b5270 0x55e5793b7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.367+0000 7ff5a5d98740 4 client.16137 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:58:22.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.367+0000 7ff5a5d98740 1 -- 172.21.15.76:0/2963483738 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e579037890 con 0x55e5793b5270 2024-03-12T05:58:22.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.367+0000 7ff59bfff640 1 --2- 172.21.15.76:0/2963483738 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e5793b5270 0x55e5793b7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.367+0000 7ff57bfff640 10 client.16137 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:58:22.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.367+0000 7ff59bfff640 1 --2- 172.21.15.76:0/2963483738 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e5793b5270 0x55e5793b7660 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=56721e89e29ed129 server_cookie=f05bd2087708d8a4 in_seq=0 out_seq=0 2024-03-12T05:58:22.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff57bfff640 1 -- 172.21.15.76:0/2963483738 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e579037890 con 0x55e5793b5270 2024-03-12T05:58:22.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff57bfff640 10 client.16137 handle_command_reply: tid=0 2024-03-12T05:58:22.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a5d98740 1 client.16137 shutdown 2024-03-12T05:58:22.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a5d98740 1 -- 172.21.15.76:0/2963483738 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e579344250 msgr2=0x55e579352d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/2963483738 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e579344250 0x55e579352d20 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff5900ed770 tx=0x7ff59002bc30 comp rx=0 tx=0).stop 2024-03-12T05:58:22.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a5d98740 1 -- 172.21.15.76:0/2963483738 shutdown_connections 2024-03-12T05:58:22.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/2963483738 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55e5793b5270 0x55e5793b7660 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/2963483738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e57934d750 0x55e5793537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/2963483738 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e579344b70 0x55e579353260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff57bfff640 0 client.16137 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:58:22.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a5d98740 1 --2- 172.21.15.76:0/2963483738 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e579344250 0x55e579352d20 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a5d98740 1 -- 172.21.15.76:0/2963483738 >> 172.21.15.76:0/2963483738 conn(0x55e57933e980 msgr2=0x55e57933d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a5d98740 1 -- 172.21.15.76:0/2963483738 shutdown_connections 2024-03-12T05:58:22.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a5d98740 1 -- 172.21.15.76:0/2963483738 wait complete. 2024-03-12T05:58:22.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a5d98740 20 client.16137 trim_cache size 0 max 16384 2024-03-12T05:58:22.388 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:58:22.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a30eb640 1 -- 172.21.15.76:0/857595159 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff58c0a1ba0 msgr2=0x7ff58c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a30eb640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff58c0a1ba0 0x7ff58c0a4060 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff5940092f0 tx=0x7ff594009c30 comp rx=0 tx=0).stop 2024-03-12T05:58:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a30eb640 1 -- 172.21.15.76:0/857595159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff59c100280 msgr2=0x7ff59c198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a30eb640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff59c100280 0x7ff59c198650 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff590009230 tx=0x7ff590004040 comp rx=0 tx=0).stop 2024-03-12T05:58:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a30eb640 1 -- 172.21.15.76:0/857595159 shutdown_connections 2024-03-12T05:58:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a30eb640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff58c0a1ba0 0x7ff58c0a4060 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a30eb640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff59c199120 0x7ff59c1a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a30eb640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff59c105350 0x7ff59c198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.368+0000 7ff5a30eb640 1 --2- 172.21.15.76:0/857595159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff59c100280 0x7ff59c198650 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.369+0000 7ff5a30eb640 1 -- 172.21.15.76:0/857595159 >> 172.21.15.76:0/857595159 conn(0x7ff59c077170 msgr2=0x7ff59c0fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.369+0000 7ff5a30eb640 1 -- 172.21.15.76:0/857595159 shutdown_connections 2024-03-12T05:58:22.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.369+0000 7ff5a30eb640 1 -- 172.21.15.76:0/857595159 wait complete. 2024-03-12T05:58:22.391 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:22 smithi076 ceph-mon[32095]: pgmap v80: 97 pgs: 97 active+clean; 80 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-03-12T05:58:22.392 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:58:22.392 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:58:22.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:22 smithi082 ceph-mon[36382]: pgmap v80: 97 pgs: 97 active+clean; 80 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-03-12T05:58:22.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:22 smithi088 ceph-mon[36585]: pgmap v80: 97 pgs: 97 active+clean; 80 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-03-12T05:58:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.492+0000 7fd800b3a640 1 -- 172.21.15.76:0/3585598420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc100290 msgr2=0x7fd7fc100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.492+0000 7fd800b3a640 1 --2- 172.21.15.76:0/3585598420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc100290 0x7fd7fc100670 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec009930 tx=0x7fd7ec02bc30 comp rx=0 tx=0).stop 2024-03-12T05:58:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.492+0000 7fd800b3a640 1 -- 172.21.15.76:0/3585598420 shutdown_connections 2024-03-12T05:58:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.492+0000 7fd800b3a640 1 --2- 172.21.15.76:0/3585598420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7fc105360 0x7fd7fc10cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.492+0000 7fd800b3a640 1 --2- 172.21.15.76:0/3585598420 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc100c40 0x7fd7fc104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.492+0000 7fd800b3a640 1 --2- 172.21.15.76:0/3585598420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc100290 0x7fd7fc100670 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.492+0000 7fd800b3a640 1 -- 172.21.15.76:0/3585598420 >> 172.21.15.76:0/3585598420 conn(0x7fd7fc077230 msgr2=0x7fd7fc077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.492+0000 7fd800b3a640 1 -- 172.21.15.76:0/3585598420 shutdown_connections 2024-03-12T05:58:22.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.492+0000 7fd800b3a640 1 -- 172.21.15.76:0/3585598420 wait complete. 2024-03-12T05:58:22.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.492+0000 7fd800b3a640 1 Processor -- start 2024-03-12T05:58:22.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.492+0000 7fd800b3a640 1 -- start start 2024-03-12T05:58:22.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd800b3a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc100290 0x7fd7fc19d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd800b3a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7fc105360 0x7fd7fc19d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd800b3a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc197620 0x7fd7fc197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd800b3a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc198010 con 0x7fd7fc100290 2024-03-12T05:58:22.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd800b3a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc198180 con 0x7fd7fc197620 2024-03-12T05:58:22.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd800b3a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc198340 con 0x7fd7fc105360 2024-03-12T05:58:22.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7fa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc100290 0x7fd7fc19d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7fa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc100290 0x7fd7fc19d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45522/0 (socket says 172.21.15.76:45522) 2024-03-12T05:58:22.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7fa59c640 1 -- 172.21.15.76:0/202815931 learned_addr learned my addr 172.21.15.76:0/202815931 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7fad9d640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc197620 0x7fd7fc197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7f3fff640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7fc105360 0x7fd7fc19d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7fa59c640 1 -- 172.21.15.76:0/202815931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7fc105360 msgr2=0x7fd7fc19d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7fa59c640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7fc105360 0x7fd7fc19d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7fa59c640 1 -- 172.21.15.76:0/202815931 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc197620 msgr2=0x7fd7fc197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7fa59c640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc197620 0x7fd7fc197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7fa59c640 1 -- 172.21.15.76:0/202815931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec009560 con 0x7fd7fc100290 2024-03-12T05:58:22.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7f3fff640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7fc105360 0x7fd7fc19d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:22.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7fad9d640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc197620 0x7fd7fc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:22.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7fa59c640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc100290 0x7fd7fc19d470 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec009230 tx=0x7fd7ec02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7f37fe640 1 -- 172.21.15.76:0/202815931 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec004350 con 0x7fd7fc100290 2024-03-12T05:58:22.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd800b3a640 1 -- 172.21.15.76:0/202815931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7fc1a4330 con 0x7fd7fc100290 2024-03-12T05:58:22.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7f37fe640 1 -- 172.21.15.76:0/202815931 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7ec0044b0 con 0x7fd7fc100290 2024-03-12T05:58:22.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd7f37fe640 1 -- 172.21.15.76:0/202815931 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec003d70 con 0x7fd7fc100290 2024-03-12T05:58:22.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.493+0000 7fd800b3a640 1 -- 172.21.15.76:0/202815931 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7fc1a47e0 con 0x7fd7fc100290 2024-03-12T05:58:22.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.496+0000 7fd7f37fe640 1 -- 172.21.15.76:0/202815931 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd7ec004df0 con 0x7fd7fc100290 2024-03-12T05:58:22.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.496+0000 7fd8037e2740 1 Processor -- start 2024-03-12T05:58:22.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.496+0000 7fd8037e2740 1 -- start start 2024-03-12T05:58:22.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd8037e2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81ea250 0x55d1d81ea630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd8037e2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd8037e2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81f3750 0x55d1d81f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd8037e2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d81a5960 con 0x55d1d81eab70 2024-03-12T05:58:22.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd8037e2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d8159310 con 0x55d1d81ea250 2024-03-12T05:58:22.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd8037e2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d8156840 con 0x55d1d81f3750 2024-03-12T05:58:22.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7f37fe640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd7e80a1ba0 0x7fd7e80a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7fa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7f3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81f3750 0x55d1d81f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7fa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45528/0 (socket says 172.21.15.76:45528) 2024-03-12T05:58:22.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7fa59c640 1 -- 172.21.15.76:0/32159958 learned_addr learned my addr 172.21.15.76:0/32159958 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7fad9d640 1 --2- 172.21.15.76:0/32159958 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81ea250 0x55d1d81ea630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7fad9d640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd7e80a1ba0 0x7fd7e80a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7f37fe640 1 -- 172.21.15.76:0/202815931 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd7ec0ef860 con 0x7fd7fc100290 2024-03-12T05:58:22.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7fa59c640 1 -- 172.21.15.76:0/32159958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81f3750 msgr2=0x55d1d81f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7fa59c640 1 --2- 172.21.15.76:0/32159958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81f3750 0x55d1d81f5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7fa59c640 1 -- 172.21.15.76:0/32159958 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81ea250 msgr2=0x55d1d81ea630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7fa59c640 1 --2- 172.21.15.76:0/32159958 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81ea250 0x55d1d81ea630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7fa59c640 1 -- 172.21.15.76:0/32159958 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1d8114060 con 0x55d1d81eab70 2024-03-12T05:58:22.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7fad9d640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd7e80a1ba0 0x7fd7e80a4060 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4005fd0 tx=0x7fd7f4009b70 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.497+0000 7fd7fa59c640 1 --2- 172.21.15.76:0/32159958 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f3210 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0b8fa0 tx=0x7fd7ec009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b8b16f2832c786 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.498+0000 7fd7f0ff9640 1 -- 172.21.15.76:0/32159958 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec0bccd0 con 0x55d1d81eab70 2024-03-12T05:58:22.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.498+0000 7fd7f0ff9640 1 -- 172.21.15.76:0/32159958 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7ec0b8040 con 0x55d1d81eab70 2024-03-12T05:58:22.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.498+0000 7fd7f0ff9640 1 -- 172.21.15.76:0/32159958 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec0a70d0 con 0x55d1d81eab70 2024-03-12T05:58:22.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.498+0000 7fd8037e2740 1 -- 172.21.15.76:0/32159958 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 msgr2=0x55d1d81f3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.498+0000 7fd8037e2740 1 --2- 172.21.15.76:0/32159958 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f3210 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0b8fa0 tx=0x7fd7ec009260 comp rx=0 tx=0).stop 2024-03-12T05:58:22.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.498+0000 7fd8037e2740 1 -- 172.21.15.76:0/32159958 shutdown_connections 2024-03-12T05:58:22.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.498+0000 7fd8037e2740 1 --2- 172.21.15.76:0/32159958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81f3750 0x55d1d81f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.498+0000 7fd8037e2740 1 --2- 172.21.15.76:0/32159958 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f3210 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.498+0000 7fd8037e2740 1 --2- 172.21.15.76:0/32159958 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81ea250 0x55d1d81ea630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.498+0000 7fd8037e2740 1 -- 172.21.15.76:0/32159958 >> 172.21.15.76:0/32159958 conn(0x55d1d81e4980 msgr2=0x55d1d81e4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.498+0000 7fd8037e2740 1 -- 172.21.15.76:0/32159958 shutdown_connections 2024-03-12T05:58:22.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.498+0000 7fd8037e2740 1 -- 172.21.15.76:0/32159958 wait complete. 2024-03-12T05:58:22.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd8037e2740 1 -- start start 2024-03-12T05:58:22.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd8037e2740 1 Processor -- start 2024-03-12T05:58:22.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd8037e2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81ea250 0x55d1d81f0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd8037e2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81eab70 0x55d1d81f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd8037e2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81f3750 0x55d1d81f11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd8037e2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d8152020 con 0x55d1d81f3750 2024-03-12T05:58:22.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd8037e2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d7e25430 con 0x55d1d81eab70 2024-03-12T05:58:22.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd8037e2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d7de1e00 con 0x55d1d81ea250 2024-03-12T05:58:22.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd8037e2740 10 client.0 _ll_register_callbacks cb 0x55d1d80f9cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:22.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd7fad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81f3750 0x55d1d81f11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd7fad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81f3750 0x55d1d81f11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45530/0 (socket says 172.21.15.76:45530) 2024-03-12T05:58:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd7fad9d640 1 -- 172.21.15.76:0/3264626263 learned_addr learned my addr 172.21.15.76:0/3264626263 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd7fa59c640 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81ea250 0x55d1d81f0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd7f3fff640 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81eab70 0x55d1d81f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd7fad9d640 1 -- 172.21.15.76:0/3264626263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81ea250 msgr2=0x55d1d81f0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd7fad9d640 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81ea250 0x55d1d81f0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd7fad9d640 1 -- 172.21.15.76:0/3264626263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81eab70 msgr2=0x55d1d81f0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd7fad9d640 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81eab70 0x55d1d81f0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd7fad9d640 1 -- 172.21.15.76:0/3264626263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7fc1a47e0 con 0x55d1d81f3750 2024-03-12T05:58:22.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd7fa59c640 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81ea250 0x55d1d81f0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:58:22.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd7f3fff640 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81eab70 0x55d1d81f0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:58:22.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.499+0000 7fd7fad9d640 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81f3750 0x55d1d81f11f0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4010e90 tx=0x7fd7f4009a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94811ab6fe3afe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd7e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:22.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd7e1ffb640 1 -- 172.21.15.76:0/3264626263 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f4021e20 con 0x55d1d81f3750 2024-03-12T05:58:22.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd7e1ffb640 1 -- 172.21.15.76:0/3264626263 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd7f4004b20 con 0x55d1d81f3750 2024-03-12T05:58:22.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd8037e2740 1 -- 172.21.15.76:0/3264626263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d1d7e517e0 con 0x55d1d81f3750 2024-03-12T05:58:22.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd7e1ffb640 1 -- 172.21.15.76:0/3264626263 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7f4004ce0 con 0x55d1d81f3750 2024-03-12T05:58:22.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd8037e2740 10 client.16149 fetch_fsmap learned FSMap version 24 2024-03-12T05:58:22.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd8037e2740 1 -- 172.21.15.76:0/3264626263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55d1d81506a0 con 0x55d1d81f3750 2024-03-12T05:58:22.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd7e1ffb640 1 -- 172.21.15.76:0/3264626263 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d1d7e517e0 con 0x55d1d81f3750 2024-03-12T05:58:22.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd7e1ffb640 1 -- 172.21.15.76:0/3264626263 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd7f402b670 con 0x55d1d81f3750 2024-03-12T05:58:22.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd8037e2740 10 client.16149 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:58:22.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd8037e2740 10 client.16149 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:58:22.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd8037e2740 20 client.16149 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:22.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd8037e2740 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d1d82039e0 0x55d1d8205e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd8037e2740 4 client.16149 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:58:22.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.500+0000 7fd8037e2740 1 -- 172.21.15.76:0/3264626263 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d1d7e8e720 con 0x55d1d82039e0 2024-03-12T05:58:22.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.501+0000 7fd7fa59c640 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d1d82039e0 0x55d1d8205e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.501+0000 7fd7e1ffb640 10 client.16149 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:58:22.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.501+0000 7fd7fa59c640 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d1d82039e0 0x55d1d8205e00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b9d1b389057fcc65 server_cookie=dd22373bee2b1ea9 in_seq=0 out_seq=0 2024-03-12T05:58:22.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd7e1ffb640 1 -- 172.21.15.76:0/3264626263 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d1d7e8e720 con 0x55d1d82039e0 2024-03-12T05:58:22.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd7e1ffb640 10 client.16149 handle_command_reply: tid=0 2024-03-12T05:58:22.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd8037e2740 1 client.16149 shutdown 2024-03-12T05:58:22.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd8037e2740 1 -- 172.21.15.76:0/3264626263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81f3750 msgr2=0x55d1d81f11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd8037e2740 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81f3750 0x55d1d81f11f0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4010e90 tx=0x7fd7f4009a60 comp rx=0 tx=0).stop 2024-03-12T05:58:22.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd8037e2740 1 -- 172.21.15.76:0/3264626263 shutdown_connections 2024-03-12T05:58:22.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd8037e2740 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d1d82039e0 0x55d1d8205e00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd8037e2740 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81f3750 0x55d1d81f11f0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd7e1ffb640 0 client.16149 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:58:22.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd8037e2740 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81eab70 0x55d1d81f0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd8037e2740 1 --2- 172.21.15.76:0/3264626263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81ea250 0x55d1d81f0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd8037e2740 1 -- 172.21.15.76:0/3264626263 >> 172.21.15.76:0/3264626263 conn(0x55d1d81e4980 msgr2=0x55d1d81e66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd8037e2740 1 -- 172.21.15.76:0/3264626263 shutdown_connections 2024-03-12T05:58:22.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd8037e2740 1 -- 172.21.15.76:0/3264626263 wait complete. 2024-03-12T05:58:22.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.502+0000 7fd8037e2740 20 client.16149 trim_cache size 0 max 16384 2024-03-12T05:58:22.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.513+0000 7fd8037e2740 1 Processor -- start 2024-03-12T05:58:22.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.513+0000 7fd8037e2740 1 -- start start 2024-03-12T05:58:22.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.513+0000 7fd8037e2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81ea250 0x55d1d81f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.513+0000 7fd8037e2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.513+0000 7fd8037e2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81f3750 0x55d1d81ed9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.513+0000 7fd8037e2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d80d7130 con 0x55d1d81eab70 2024-03-12T05:58:22.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.513+0000 7fd8037e2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d81e05e0 con 0x55d1d81f3750 2024-03-12T05:58:22.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.513+0000 7fd8037e2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d8200430 con 0x55d1d81ea250 2024-03-12T05:58:22.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.514+0000 7fd7f3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.514+0000 7fd7f3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45534/0 (socket says 172.21.15.76:45534) 2024-03-12T05:58:22.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.514+0000 7fd7f3fff640 1 -- 172.21.15.76:0/2744615097 learned_addr learned my addr 172.21.15.76:0/2744615097 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.514+0000 7fd7fa59c640 1 --2- 172.21.15.76:0/2744615097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81ea250 0x55d1d81f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.514+0000 7fd7f3fff640 1 -- 172.21.15.76:0/2744615097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81ea250 msgr2=0x55d1d81f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.514+0000 7fd7f3fff640 1 --2- 172.21.15.76:0/2744615097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81ea250 0x55d1d81f2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.514+0000 7fd7f3fff640 1 -- 172.21.15.76:0/2744615097 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81f3750 msgr2=0x55d1d81ed9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.514+0000 7fd7fad9d640 1 --2- 172.21.15.76:0/2744615097 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81f3750 0x55d1d81ed9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.514+0000 7fd7f3fff640 1 --2- 172.21.15.76:0/2744615097 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81f3750 0x55d1d81ed9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.514+0000 7fd7f3fff640 1 -- 172.21.15.76:0/2744615097 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1d80ecc20 con 0x55d1d81eab70 2024-03-12T05:58:22.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.514+0000 7fd7f3fff640 1 --2- 172.21.15.76:0/2744615097 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f1010 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc198760 tx=0x7fd7e4012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a504a65394778518 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.514+0000 7fd7e3fff640 1 -- 172.21.15.76:0/2744615097 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4004250 con 0x55d1d81eab70 2024-03-12T05:58:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.514+0000 7fd7e3fff640 1 -- 172.21.15.76:0/2744615097 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7e40043b0 con 0x55d1d81eab70 2024-03-12T05:58:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.514+0000 7fd7e3fff640 1 -- 172.21.15.76:0/2744615097 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40035f0 con 0x55d1d81eab70 2024-03-12T05:58:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 -- 172.21.15.76:0/2744615097 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 msgr2=0x55d1d81f1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 --2- 172.21.15.76:0/2744615097 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f1010 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc198760 tx=0x7fd7e4012820 comp rx=0 tx=0).stop 2024-03-12T05:58:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 -- 172.21.15.76:0/2744615097 shutdown_connections 2024-03-12T05:58:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 --2- 172.21.15.76:0/2744615097 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81f3750 0x55d1d81ed9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 --2- 172.21.15.76:0/2744615097 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f1010 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 --2- 172.21.15.76:0/2744615097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81ea250 0x55d1d81f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 -- 172.21.15.76:0/2744615097 >> 172.21.15.76:0/2744615097 conn(0x55d1d81e4980 msgr2=0x55d1d81ebad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 -- 172.21.15.76:0/2744615097 shutdown_connections 2024-03-12T05:58:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 -- 172.21.15.76:0/2744615097 wait complete. 2024-03-12T05:58:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 -- start start 2024-03-12T05:58:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 Processor -- start 2024-03-12T05:58:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81ea250 0x55d1d81f9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81f3750 0x55d1d8259f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d825a450 con 0x55d1d81eab70 2024-03-12T05:58:22.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d825a590 con 0x55d1d81ea250 2024-03-12T05:58:22.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d825a720 con 0x55d1d81f3750 2024-03-12T05:58:22.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd8037e2740 10 client.0 _ll_register_callbacks cb 0x7fd7ec0b88e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:22.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd7f3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd7f3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45538/0 (socket says 172.21.15.76:45538) 2024-03-12T05:58:22.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd7f3fff640 1 -- 172.21.15.76:0/2868946979 learned_addr learned my addr 172.21.15.76:0/2868946979 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd7fad9d640 1 --2- 172.21.15.76:0/2868946979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81f3750 0x55d1d8259f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.515+0000 7fd7f3fff640 1 -- 172.21.15.76:0/2868946979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81f3750 msgr2=0x55d1d8259f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd7f3fff640 1 --2- 172.21.15.76:0/2868946979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81f3750 0x55d1d8259f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd7f3fff640 1 -- 172.21.15.76:0/2868946979 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81ea250 msgr2=0x55d1d81f9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd7f3fff640 1 --2- 172.21.15.76:0/2868946979 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81ea250 0x55d1d81f9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd7f3fff640 1 -- 172.21.15.76:0/2868946979 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e4008920 con 0x55d1d81eab70 2024-03-12T05:58:22.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd7fad9d640 1 --2- 172.21.15.76:0/2868946979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81f3750 0x55d1d8259f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:58:22.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd7f3fff640 1 --2- 172.21.15.76:0/2868946979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f9aa0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4013fd0 tx=0x7fd7e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d775203537641dc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd7e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:22.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd7e1ffb640 1 -- 172.21.15.76:0/2868946979 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4005200 con 0x55d1d81eab70 2024-03-12T05:58:22.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd7e1ffb640 1 -- 172.21.15.76:0/2868946979 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd7e4005360 con 0x55d1d81eab70 2024-03-12T05:58:22.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd8037e2740 1 -- 172.21.15.76:0/2868946979 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7ec0b8040 con 0x55d1d81eab70 2024-03-12T05:58:22.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd7e1ffb640 1 -- 172.21.15.76:0/2868946979 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7e4005520 con 0x55d1d81eab70 2024-03-12T05:58:22.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd8037e2740 10 client.16155 fetch_fsmap learned FSMap version 24 2024-03-12T05:58:22.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd8037e2740 1 -- 172.21.15.76:0/2868946979 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55d1d81128c0 con 0x55d1d81eab70 2024-03-12T05:58:22.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd7e1ffb640 1 -- 172.21.15.76:0/2868946979 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7ec0b8040 con 0x55d1d81eab70 2024-03-12T05:58:22.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd7e1ffb640 1 -- 172.21.15.76:0/2868946979 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd7e4023070 con 0x55d1d81eab70 2024-03-12T05:58:22.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd8037e2740 10 client.16155 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:58:22.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd8037e2740 10 client.16155 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:58:22.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd8037e2740 20 client.16155 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:22.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd8037e2740 1 --2- 172.21.15.76:0/2868946979 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d1d825be90 0x55d1d825e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd8037e2740 4 client.16155 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:58:22.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.516+0000 7fd8037e2740 1 -- 172.21.15.76:0/2868946979 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d1d7edd890 con 0x55d1d825be90 2024-03-12T05:58:22.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd7fa59c640 1 --2- 172.21.15.76:0/2868946979 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d1d825be90 0x55d1d825e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd7e1ffb640 10 client.16155 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:58:22.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd7fa59c640 1 --2- 172.21.15.76:0/2868946979 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d1d825be90 0x55d1d825e2b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ed96d9e3df1e6dc0 server_cookie=35ae7d114e70d5ec in_seq=0 out_seq=0 2024-03-12T05:58:22.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd7e1ffb640 1 -- 172.21.15.76:0/2868946979 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d1d7edd890 con 0x55d1d825be90 2024-03-12T05:58:22.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd7e1ffb640 10 client.16155 handle_command_reply: tid=0 2024-03-12T05:58:22.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd8037e2740 1 client.16155 shutdown 2024-03-12T05:58:22.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd8037e2740 1 -- 172.21.15.76:0/2868946979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 msgr2=0x55d1d81f9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd8037e2740 1 --2- 172.21.15.76:0/2868946979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f9aa0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4013fd0 tx=0x7fd7e4004750 comp rx=0 tx=0).stop 2024-03-12T05:58:22.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd8037e2740 1 -- 172.21.15.76:0/2868946979 shutdown_connections 2024-03-12T05:58:22.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd8037e2740 1 --2- 172.21.15.76:0/2868946979 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d1d825be90 0x55d1d825e2b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd8037e2740 1 --2- 172.21.15.76:0/2868946979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d1d81f3750 0x55d1d8259f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd8037e2740 1 --2- 172.21.15.76:0/2868946979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d1d81eab70 0x55d1d81f9aa0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd8037e2740 1 --2- 172.21.15.76:0/2868946979 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d1d81ea250 0x55d1d81f9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd7e1ffb640 0 client.16155 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:58:22.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd8037e2740 1 -- 172.21.15.76:0/2868946979 >> 172.21.15.76:0/2868946979 conn(0x55d1d81e4980 msgr2=0x55d1d81e32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd8037e2740 1 -- 172.21.15.76:0/2868946979 shutdown_connections 2024-03-12T05:58:22.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd8037e2740 1 -- 172.21.15.76:0/2868946979 wait complete. 2024-03-12T05:58:22.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.517+0000 7fd8037e2740 20 client.16155 trim_cache size 0 max 16384 2024-03-12T05:58:22.537 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:58:22.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.518+0000 7fd800b3a640 1 -- 172.21.15.76:0/202815931 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd7e80a1ba0 msgr2=0x7fd7e80a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.518+0000 7fd800b3a640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd7e80a1ba0 0x7fd7e80a4060 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4005fd0 tx=0x7fd7f4009b70 comp rx=0 tx=0).stop 2024-03-12T05:58:22.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.518+0000 7fd800b3a640 1 -- 172.21.15.76:0/202815931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc100290 msgr2=0x7fd7fc19d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.518+0000 7fd800b3a640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc100290 0x7fd7fc19d470 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec009230 tx=0x7fd7ec02c6c0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.518+0000 7fd800b3a640 1 -- 172.21.15.76:0/202815931 shutdown_connections 2024-03-12T05:58:22.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.518+0000 7fd800b3a640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd7e80a1ba0 0x7fd7e80a4060 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.518+0000 7fd800b3a640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd7fc197620 0x7fd7fc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.518+0000 7fd800b3a640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7fc105360 0x7fd7fc19d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.518+0000 7fd800b3a640 1 --2- 172.21.15.76:0/202815931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd7fc100290 0x7fd7fc19d470 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.518+0000 7fd800b3a640 1 -- 172.21.15.76:0/202815931 >> 172.21.15.76:0/202815931 conn(0x7fd7fc077230 msgr2=0x7fd7fc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.518+0000 7fd800b3a640 1 -- 172.21.15.76:0/202815931 shutdown_connections 2024-03-12T05:58:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.518+0000 7fd800b3a640 1 -- 172.21.15.76:0/202815931 wait complete. 2024-03-12T05:58:22.540 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:58:22.540 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:58:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.641+0000 7f3a09dac640 1 -- 172.21.15.76:0/532723459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a041023a0 msgr2=0x7f3a04102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.641+0000 7f3a09dac640 1 --2- 172.21.15.76:0/532723459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a041023a0 0x7f3a04102780 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f39f4009900 tx=0x7f39f402bcd0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.641+0000 7f3a09dac640 1 -- 172.21.15.76:0/532723459 shutdown_connections 2024-03-12T05:58:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.641+0000 7f3a09dac640 1 --2- 172.21.15.76:0/532723459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a0410b770 0x7f3a0410dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.641+0000 7f3a09dac640 1 --2- 172.21.15.76:0/532723459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a04102cc0 0x7f3a0410b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.641+0000 7f3a09dac640 1 --2- 172.21.15.76:0/532723459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a041023a0 0x7f3a04102780 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.641+0000 7f3a09dac640 1 -- 172.21.15.76:0/532723459 >> 172.21.15.76:0/532723459 conn(0x7f3a040771a0 msgr2=0x7f3a040775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.641+0000 7f3a09dac640 1 -- 172.21.15.76:0/532723459 shutdown_connections 2024-03-12T05:58:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.641+0000 7f3a09dac640 1 -- 172.21.15.76:0/532723459 wait complete. 2024-03-12T05:58:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.641+0000 7f3a09dac640 1 Processor -- start 2024-03-12T05:58:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.641+0000 7f3a09dac640 1 -- start start 2024-03-12T05:58:22.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a09dac640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a041023a0 0x7f3a040ffff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a09dac640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a04102cc0 0x7f3a04100530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a09dac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a0410b770 0x7f3a04100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a09dac640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a041a5ae0 con 0x7f3a041023a0 2024-03-12T05:58:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a09dac640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a041a5c50 con 0x7f3a04102cc0 2024-03-12T05:58:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a09dac640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a041a5e10 con 0x7f3a0410b770 2024-03-12T05:58:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a037fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a041023a0 0x7f3a040ffff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a037fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a041023a0 0x7f3a040ffff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45552/0 (socket says 172.21.15.76:45552) 2024-03-12T05:58:22.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a037fe640 1 -- 172.21.15.76:0/4063573377 learned_addr learned my addr 172.21.15.76:0/4063573377 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a03fff640 1 --2- 172.21.15.76:0/4063573377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a0410b770 0x7f3a04100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f39faffd640 1 --2- 172.21.15.76:0/4063573377 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a04102cc0 0x7f3a04100530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a037fe640 1 -- 172.21.15.76:0/4063573377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a0410b770 msgr2=0x7f3a04100a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a037fe640 1 --2- 172.21.15.76:0/4063573377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a0410b770 0x7f3a04100a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a037fe640 1 -- 172.21.15.76:0/4063573377 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a04102cc0 msgr2=0x7f3a04100530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a037fe640 1 --2- 172.21.15.76:0/4063573377 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a04102cc0 0x7f3a04100530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a037fe640 1 -- 172.21.15.76:0/4063573377 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f4009560 con 0x7f3a041023a0 2024-03-12T05:58:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.642+0000 7f3a037fe640 1 --2- 172.21.15.76:0/4063573377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a041023a0 0x7f3a040ffff0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f39f4005940 tx=0x7f39f4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.643+0000 7f3a017fa640 1 -- 172.21.15.76:0/4063573377 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f4004440 con 0x7f3a041023a0 2024-03-12T05:58:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.643+0000 7f3a017fa640 1 -- 172.21.15.76:0/4063573377 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39f40052d0 con 0x7f3a041023a0 2024-03-12T05:58:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.643+0000 7f3a017fa640 1 -- 172.21.15.76:0/4063573377 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f4003c90 con 0x7f3a041023a0 2024-03-12T05:58:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.643+0000 7f3a09dac640 1 -- 172.21.15.76:0/4063573377 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a041a6090 con 0x7f3a041023a0 2024-03-12T05:58:22.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.643+0000 7f3a09dac640 1 -- 172.21.15.76:0/4063573377 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a041a65a0 con 0x7f3a041023a0 2024-03-12T05:58:22.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.645+0000 7f3a017fa640 1 -- 172.21.15.76:0/4063573377 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f39f4005440 con 0x7f3a041023a0 2024-03-12T05:58:22.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f3a0c980740 1 Processor -- start 2024-03-12T05:58:22.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f3a0c980740 1 -- start start 2024-03-12T05:58:22.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f3a0c980740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674c4250 0x5576674c4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f3a0c980740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674c4b70 0x5576674cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f3a0c980740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674cd750 0x5576674cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f3a0c980740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55766747f960 con 0x5576674cd750 2024-03-12T05:58:22.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f3a0c980740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x557667433310 con 0x5576674c4b70 2024-03-12T05:58:22.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f3a0c980740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x557667430840 con 0x5576674c4250 2024-03-12T05:58:22.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f39faffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674cd750 0x5576674cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f3a017fa640 1 --2- 172.21.15.76:0/4063573377 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f39f00a1ba0 0x7f39f00a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f39faffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674cd750 0x5576674cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45560/0 (socket says 172.21.15.76:45560) 2024-03-12T05:58:22.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f39faffd640 1 -- 172.21.15.76:0/3082993367 learned_addr learned my addr 172.21.15.76:0/3082993367 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f3a03fff640 1 --2- 172.21.15.76:0/3082993367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674c4250 0x5576674c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f3a03fff640 1 --2- 172.21.15.76:0/4063573377 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f39f00a1ba0 0x7f39f00a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f3a017fa640 1 -- 172.21.15.76:0/4063573377 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f39f40efed0 con 0x7f3a041023a0 2024-03-12T05:58:22.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f3a037fe640 1 --2- 172.21.15.76:0/3082993367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674c4b70 0x5576674cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f39faffd640 1 -- 172.21.15.76:0/3082993367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674c4250 msgr2=0x5576674c4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f39faffd640 1 --2- 172.21.15.76:0/3082993367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674c4250 0x5576674c4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f39faffd640 1 -- 172.21.15.76:0/3082993367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674c4b70 msgr2=0x5576674cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f39faffd640 1 --2- 172.21.15.76:0/3082993367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674c4b70 0x5576674cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.646+0000 7f39faffd640 1 -- 172.21.15.76:0/3082993367 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576673ee060 con 0x5576674cd750 2024-03-12T05:58:22.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.647+0000 7f39faffd640 1 --2- 172.21.15.76:0/3082993367 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674cd750 0x5576674cfb40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f39ec00aca0 tx=0x7f39ec00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fafc136f8af5dd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.647+0000 7f3a03fff640 1 --2- 172.21.15.76:0/4063573377 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f39f00a1ba0 0x7f39f00a4060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009fd0 tx=0x7f39fc009600 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.647+0000 7f39f9ffb640 1 -- 172.21.15.76:0/3082993367 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec018640 con 0x5576674cd750 2024-03-12T05:58:22.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.647+0000 7f39f9ffb640 1 -- 172.21.15.76:0/3082993367 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39ec018c80 con 0x5576674cd750 2024-03-12T05:58:22.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.647+0000 7f39f9ffb640 1 -- 172.21.15.76:0/3082993367 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec010c60 con 0x5576674cd750 2024-03-12T05:58:22.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.647+0000 7f3a0c980740 1 -- 172.21.15.76:0/3082993367 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674cd750 msgr2=0x5576674cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.647+0000 7f3a0c980740 1 --2- 172.21.15.76:0/3082993367 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674cd750 0x5576674cfb40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f39ec00aca0 tx=0x7f39ec00f790 comp rx=0 tx=0).stop 2024-03-12T05:58:22.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.647+0000 7f3a0c980740 1 -- 172.21.15.76:0/3082993367 shutdown_connections 2024-03-12T05:58:22.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.647+0000 7f3a0c980740 1 --2- 172.21.15.76:0/3082993367 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674cd750 0x5576674cfb40 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.647+0000 7f3a0c980740 1 --2- 172.21.15.76:0/3082993367 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674c4b70 0x5576674cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.647+0000 7f3a0c980740 1 --2- 172.21.15.76:0/3082993367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674c4250 0x5576674c4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.647+0000 7f3a0c980740 1 -- 172.21.15.76:0/3082993367 >> 172.21.15.76:0/3082993367 conn(0x5576674be980 msgr2=0x5576674bed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.647+0000 7f3a0c980740 1 -- 172.21.15.76:0/3082993367 shutdown_connections 2024-03-12T05:58:22.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.647+0000 7f3a0c980740 1 -- 172.21.15.76:0/3082993367 wait complete. 2024-03-12T05:58:22.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.648+0000 7f3a0c980740 1 -- start start 2024-03-12T05:58:22.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.648+0000 7f3a0c980740 1 Processor -- start 2024-03-12T05:58:22.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.648+0000 7f3a0c980740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674c4250 0x5576674ca700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.648+0000 7f3a0c980740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674cac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.648+0000 7f3a0c980740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674cd750 0x5576674cb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.648+0000 7f3a0c980740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55766742c020 con 0x5576674c4b70 2024-03-12T05:58:22.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.648+0000 7f3a0c980740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576670ff430 con 0x5576674cd750 2024-03-12T05:58:22.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.648+0000 7f3a0c980740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576670bbe00 con 0x5576674c4250 2024-03-12T05:58:22.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.648+0000 7f3a0c980740 10 client.0 _ll_register_callbacks cb 0x5576673d3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:22.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.648+0000 7f39faffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.648+0000 7f39faffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674cac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45564/0 (socket says 172.21.15.76:45564) 2024-03-12T05:58:22.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.648+0000 7f39faffd640 1 -- 172.21.15.76:0/4080280204 learned_addr learned my addr 172.21.15.76:0/4080280204 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f3a037fe640 1 --2- 172.21.15.76:0/4080280204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674c4250 0x5576674ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f3a03fff640 1 --2- 172.21.15.76:0/4080280204 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674cd750 0x5576674cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f39faffd640 1 -- 172.21.15.76:0/4080280204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674c4250 msgr2=0x5576674ca700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f39faffd640 1 --2- 172.21.15.76:0/4080280204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674c4250 0x5576674ca700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f39faffd640 1 -- 172.21.15.76:0/4080280204 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674cd750 msgr2=0x5576674cb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f39faffd640 1 --2- 172.21.15.76:0/4080280204 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674cd750 0x5576674cb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f39faffd640 1 -- 172.21.15.76:0/4080280204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ec00a900 con 0x5576674c4b70 2024-03-12T05:58:22.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f39faffd640 1 --2- 172.21.15.76:0/4080280204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674cac40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f39ec01b040 tx=0x7f39ec005ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8afeac66801421b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f39e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:22.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f39e2ffd640 1 -- 172.21.15.76:0/4080280204 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec005040 con 0x5576674c4b70 2024-03-12T05:58:22.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f39e2ffd640 1 -- 172.21.15.76:0/4080280204 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f39ec0051a0 con 0x5576674c4b70 2024-03-12T05:58:22.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f3a0c980740 1 -- 172.21.15.76:0/4080280204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55766712b7e0 con 0x5576674c4b70 2024-03-12T05:58:22.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f39e2ffd640 1 -- 172.21.15.76:0/4080280204 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39ec005360 con 0x5576674c4b70 2024-03-12T05:58:22.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f3a0c980740 10 client.16167 fetch_fsmap learned FSMap version 24 2024-03-12T05:58:22.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f3a0c980740 1 -- 172.21.15.76:0/4080280204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55766742a6a0 con 0x5576674c4b70 2024-03-12T05:58:22.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f39e2ffd640 1 -- 172.21.15.76:0/4080280204 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55766712b7e0 con 0x5576674c4b70 2024-03-12T05:58:22.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.649+0000 7f39e2ffd640 1 -- 172.21.15.76:0/4080280204 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f39ec004930 con 0x5576674c4b70 2024-03-12T05:58:22.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.650+0000 7f3a0c980740 10 client.16167 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:58:22.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.650+0000 7f3a0c980740 10 client.16167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:22.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.650+0000 7f3a0c980740 20 client.16167 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:22.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.650+0000 7f3a0c980740 1 --2- 172.21.15.76:0/4080280204 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5576674dda90 0x5576674dfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.650+0000 7f3a0c980740 4 client.16167 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:58:22.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.650+0000 7f3a0c980740 1 -- 172.21.15.76:0/4080280204 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557667168720 con 0x5576674dda90 2024-03-12T05:58:22.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.650+0000 7f3a037fe640 1 --2- 172.21.15.76:0/4080280204 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5576674dda90 0x5576674dfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.650+0000 7f39e2ffd640 10 client.16167 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:22.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.650+0000 7f3a037fe640 1 --2- 172.21.15.76:0/4080280204 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5576674dda90 0x5576674dfeb0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fa2e1bc5e9a0e46 server_cookie=d16c7c308cec1ed3 in_seq=0 out_seq=0 2024-03-12T05:58:22.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f39e2ffd640 1 -- 172.21.15.76:0/4080280204 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557667168720 con 0x5576674dda90 2024-03-12T05:58:22.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f39e2ffd640 10 client.16167 handle_command_reply: tid=0 2024-03-12T05:58:22.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f3a0c980740 1 client.16167 shutdown 2024-03-12T05:58:22.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f3a0c980740 1 -- 172.21.15.76:0/4080280204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 msgr2=0x5576674cac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f3a0c980740 1 --2- 172.21.15.76:0/4080280204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674cac40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f39ec01b040 tx=0x7f39ec005ec0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f3a0c980740 1 -- 172.21.15.76:0/4080280204 shutdown_connections 2024-03-12T05:58:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f3a0c980740 1 --2- 172.21.15.76:0/4080280204 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674cd750 0x5576674cb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f3a0c980740 1 --2- 172.21.15.76:0/4080280204 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5576674dda90 0x5576674dfeb0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f3a0c980740 1 --2- 172.21.15.76:0/4080280204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674cac40 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f3a0c980740 1 --2- 172.21.15.76:0/4080280204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674c4250 0x5576674ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f3a0c980740 1 -- 172.21.15.76:0/4080280204 >> 172.21.15.76:0/4080280204 conn(0x5576674be980 msgr2=0x5576674c06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f39e2ffd640 0 client.16167 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f3a0c980740 1 -- 172.21.15.76:0/4080280204 shutdown_connections 2024-03-12T05:58:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f3a0c980740 1 -- 172.21.15.76:0/4080280204 wait complete. 2024-03-12T05:58:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.651+0000 7f3a0c980740 20 client.16167 trim_cache size 0 max 16384 2024-03-12T05:58:22.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f3a0c980740 1 Processor -- start 2024-03-12T05:58:22.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f3a0c980740 1 -- start start 2024-03-12T05:58:22.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f3a0c980740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674c4250 0x5576674cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f3a0c980740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f3a0c980740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674cd750 0x5576674c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f3a0c980740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576673b1130 con 0x5576674c4b70 2024-03-12T05:58:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f3a0c980740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576674c36a0 con 0x5576674cd750 2024-03-12T05:58:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f3a0c980740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576674cfb40 con 0x5576674c4250 2024-03-12T05:58:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f39faffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f39faffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674cafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45574/0 (socket says 172.21.15.76:45574) 2024-03-12T05:58:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f39faffd640 1 -- 172.21.15.76:0/262362467 learned_addr learned my addr 172.21.15.76:0/262362467 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f39faffd640 1 -- 172.21.15.76:0/262362467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674c4250 msgr2=0x5576674cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f39faffd640 1 --2- 172.21.15.76:0/262362467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674c4250 0x5576674cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f39faffd640 1 -- 172.21.15.76:0/262362467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674cd750 msgr2=0x5576674c59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f39faffd640 1 --2- 172.21.15.76:0/262362467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674cd750 0x5576674c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.663+0000 7f39faffd640 1 -- 172.21.15.76:0/262362467 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576673c6c20 con 0x5576674c4b70 2024-03-12T05:58:22.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.664+0000 7f39faffd640 1 --2- 172.21.15.76:0/262362467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674cafa0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f39ec004f10 tx=0x7f39ec00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9cf4052a40ceed server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.664+0000 7f39f97fa640 1 -- 172.21.15.76:0/262362467 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec00a780 con 0x5576674c4b70 2024-03-12T05:58:22.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.664+0000 7f39f97fa640 1 -- 172.21.15.76:0/262362467 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39ec010640 con 0x5576674c4b70 2024-03-12T05:58:22.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.664+0000 7f39f97fa640 1 -- 172.21.15.76:0/262362467 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec005230 con 0x5576674c4b70 2024-03-12T05:58:22.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.664+0000 7f3a0c980740 1 -- 172.21.15.76:0/262362467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 msgr2=0x5576674cafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.664+0000 7f3a0c980740 1 --2- 172.21.15.76:0/262362467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674cafa0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f39ec004f10 tx=0x7f39ec00f790 comp rx=0 tx=0).stop 2024-03-12T05:58:22.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.664+0000 7f3a0c980740 1 -- 172.21.15.76:0/262362467 shutdown_connections 2024-03-12T05:58:22.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.664+0000 7f3a0c980740 1 --2- 172.21.15.76:0/262362467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674cd750 0x5576674c59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.664+0000 7f3a0c980740 1 --2- 172.21.15.76:0/262362467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674cafa0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.664+0000 7f3a0c980740 1 --2- 172.21.15.76:0/262362467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674c4250 0x5576674cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.664+0000 7f3a0c980740 1 -- 172.21.15.76:0/262362467 >> 172.21.15.76:0/262362467 conn(0x5576674be980 msgr2=0x5576674e4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.664+0000 7f3a0c980740 1 -- 172.21.15.76:0/262362467 shutdown_connections 2024-03-12T05:58:22.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.664+0000 7f3a0c980740 1 -- 172.21.15.76:0/262362467 wait complete. 2024-03-12T05:58:22.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.664+0000 7f3a0c980740 1 -- start start 2024-03-12T05:58:22.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f3a0c980740 1 Processor -- start 2024-03-12T05:58:22.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f3a0c980740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674c4250 0x5576674d26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f3a0c980740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f3a0c980740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674cd750 0x557667532fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f3a0c980740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576674d3390 con 0x5576674c4b70 2024-03-12T05:58:22.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f3a0c980740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576675334f0 con 0x5576674c4250 2024-03-12T05:58:22.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f3a0c980740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576675336b0 con 0x5576674cd750 2024-03-12T05:58:22.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f3a0c980740 10 client.0 _ll_register_callbacks cb 0x7f39ec00a6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:22.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f39faffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f39faffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674d2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45590/0 (socket says 172.21.15.76:45590) 2024-03-12T05:58:22.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f39faffd640 1 -- 172.21.15.76:0/3931922876 learned_addr learned my addr 172.21.15.76:0/3931922876 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:22.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f39faffd640 1 -- 172.21.15.76:0/3931922876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674cd750 msgr2=0x557667532fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f3a03fff640 1 --2- 172.21.15.76:0/3931922876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674cd750 0x557667532fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f39faffd640 1 --2- 172.21.15.76:0/3931922876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674cd750 0x557667532fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f39faffd640 1 -- 172.21.15.76:0/3931922876 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674c4250 msgr2=0x5576674d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:22.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f39faffd640 1 --2- 172.21.15.76:0/3931922876 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674c4250 0x5576674d26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f39faffd640 1 -- 172.21.15.76:0/3931922876 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55766742a6a0 con 0x5576674c4b70 2024-03-12T05:58:22.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f3a03fff640 1 --2- 172.21.15.76:0/3931922876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674cd750 0x557667532fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:58:22.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f39faffd640 1 --2- 172.21.15.76:0/3931922876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674d2c30 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f39ec004f60 tx=0x7f39ec005cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd2cb09f0d8348e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:22.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.665+0000 7f39e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:22.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f39e2ffd640 1 -- 172.21.15.76:0/3931922876 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec00a780 con 0x5576674c4b70 2024-03-12T05:58:22.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f39e2ffd640 1 -- 172.21.15.76:0/3931922876 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f39ec000e40 con 0x5576674c4b70 2024-03-12T05:58:22.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f3a0c980740 1 -- 172.21.15.76:0/3931922876 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5576671b7890 con 0x5576674c4b70 2024-03-12T05:58:22.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f39e2ffd640 1 -- 172.21.15.76:0/3931922876 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39ec034450 con 0x5576674c4b70 2024-03-12T05:58:22.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f3a0c980740 10 client.16173 fetch_fsmap learned FSMap version 24 2024-03-12T05:58:22.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f3a0c980740 1 -- 172.21.15.76:0/3931922876 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5576673ec8c0 con 0x5576674c4b70 2024-03-12T05:58:22.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f39e2ffd640 1 -- 172.21.15.76:0/3931922876 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5576671b7890 con 0x5576674c4b70 2024-03-12T05:58:22.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f39e2ffd640 1 -- 172.21.15.76:0/3931922876 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f39ec0184b0 con 0x5576674c4b70 2024-03-12T05:58:22.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f3a0c980740 10 client.16173 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:58:22.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f3a0c980740 10 client.16173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:22.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f3a0c980740 20 client.16173 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:22.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f3a0c980740 1 --2- 172.21.15.76:0/3931922876 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557667534ae0 0x557667536f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:22.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f3a0c980740 4 client.16173 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:58:22.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f3a0c980740 1 -- 172.21.15.76:0/3931922876 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5576670d7700 con 0x557667534ae0 2024-03-12T05:58:22.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.666+0000 7f3a037fe640 1 --2- 172.21.15.76:0/3931922876 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557667534ae0 0x557667536f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:22.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f39e2ffd640 10 client.16173 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:22.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f3a037fe640 1 --2- 172.21.15.76:0/3931922876 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557667534ae0 0x557667536f00 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c174830c8acb21a server_cookie=20e59c680169131b in_seq=0 out_seq=0 2024-03-12T05:58:22.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f39e2ffd640 1 -- 172.21.15.76:0/3931922876 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5576670d7700 con 0x557667534ae0 2024-03-12T05:58:22.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f39e2ffd640 10 client.16173 handle_command_reply: tid=0 2024-03-12T05:58:22.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f3a0c980740 1 client.16173 shutdown 2024-03-12T05:58:22.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f3a0c980740 1 -- 172.21.15.76:0/3931922876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 msgr2=0x5576674d2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f3a0c980740 1 --2- 172.21.15.76:0/3931922876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674d2c30 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f39ec004f60 tx=0x7f39ec005cd0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f3a0c980740 1 -- 172.21.15.76:0/3931922876 shutdown_connections 2024-03-12T05:58:22.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f3a0c980740 1 --2- 172.21.15.76:0/3931922876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5576674cd750 0x557667532fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f3a0c980740 1 --2- 172.21.15.76:0/3931922876 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557667534ae0 0x557667536f00 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f3a0c980740 1 --2- 172.21.15.76:0/3931922876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5576674c4b70 0x5576674d2c30 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f3a0c980740 1 --2- 172.21.15.76:0/3931922876 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5576674c4250 0x5576674d26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f39e2ffd640 0 client.16173 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:22.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f3a0c980740 1 -- 172.21.15.76:0/3931922876 >> 172.21.15.76:0/3931922876 conn(0x5576674be980 msgr2=0x5576674c59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f3a0c980740 1 -- 172.21.15.76:0/3931922876 shutdown_connections 2024-03-12T05:58:22.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f3a0c980740 1 -- 172.21.15.76:0/3931922876 wait complete. 2024-03-12T05:58:22.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.667+0000 7f3a0c980740 20 client.16173 trim_cache size 0 max 16384 2024-03-12T05:58:22.686 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:58:22.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.668+0000 7f3a09dac640 1 -- 172.21.15.76:0/4063573377 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f39f00a1ba0 msgr2=0x7f39f00a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.668+0000 7f3a09dac640 1 --2- 172.21.15.76:0/4063573377 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f39f00a1ba0 0x7f39f00a4060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009fd0 tx=0x7f39fc009600 comp rx=0 tx=0).stop 2024-03-12T05:58:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.668+0000 7f3a09dac640 1 -- 172.21.15.76:0/4063573377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a041023a0 msgr2=0x7f3a040ffff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.668+0000 7f3a09dac640 1 --2- 172.21.15.76:0/4063573377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a041023a0 0x7f3a040ffff0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f39f4005940 tx=0x7f39f4004040 comp rx=0 tx=0).stop 2024-03-12T05:58:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.668+0000 7f3a09dac640 1 -- 172.21.15.76:0/4063573377 shutdown_connections 2024-03-12T05:58:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.668+0000 7f3a09dac640 1 --2- 172.21.15.76:0/4063573377 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f39f00a1ba0 0x7f39f00a4060 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.668+0000 7f3a09dac640 1 --2- 172.21.15.76:0/4063573377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a0410b770 0x7f3a04100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.668+0000 7f3a09dac640 1 --2- 172.21.15.76:0/4063573377 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a04102cc0 0x7f3a04100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.668+0000 7f3a09dac640 1 --2- 172.21.15.76:0/4063573377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a041023a0 0x7f3a040ffff0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:22.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.668+0000 7f3a09dac640 1 -- 172.21.15.76:0/4063573377 >> 172.21.15.76:0/4063573377 conn(0x7f3a040771a0 msgr2=0x7f3a040fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:22.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.668+0000 7f3a09dac640 1 -- 172.21.15.76:0/4063573377 shutdown_connections 2024-03-12T05:58:22.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:22.668+0000 7f3a09dac640 1 -- 172.21.15.76:0/4063573377 wait complete. 2024-03-12T05:58:23.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:23 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4052172301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:58:23.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:23 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4052172301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:58:23.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:23 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4052172301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:58:23.690 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:58:23.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:58:23.691 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:58:23.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.802+0000 7f09f0649640 1 -- 172.21.15.76:0/811184499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e8100280 msgr2=0x7f09e8100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:23.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.802+0000 7f09f0649640 1 --2- 172.21.15.76:0/811184499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e8100280 0x7f09e8100660 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f09e4009930 tx=0x7f09e402bc30 comp rx=0 tx=0).stop 2024-03-12T05:58:23.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.802+0000 7f09f0649640 1 -- 172.21.15.76:0/811184499 shutdown_connections 2024-03-12T05:58:23.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.802+0000 7f09f0649640 1 --2- 172.21.15.76:0/811184499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09e8105350 0x7f09e810cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.802+0000 7f09f0649640 1 --2- 172.21.15.76:0/811184499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09e8100c30 0x7f09e8104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.802+0000 7f09f0649640 1 --2- 172.21.15.76:0/811184499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e8100280 0x7f09e8100660 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.802+0000 7f09f0649640 1 -- 172.21.15.76:0/811184499 >> 172.21.15.76:0/811184499 conn(0x7f09e80fbae0 msgr2=0x7f09e80fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:23.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.802+0000 7f09f0649640 1 -- 172.21.15.76:0/811184499 shutdown_connections 2024-03-12T05:58:23.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.802+0000 7f09f0649640 1 -- 172.21.15.76:0/811184499 wait complete. 2024-03-12T05:58:23.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09f0649640 1 Processor -- start 2024-03-12T05:58:23.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09f0649640 1 -- start start 2024-03-12T05:58:23.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09f0649640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09e8100280 0x7f09e810d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09f0649640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e8105350 0x7f09e810dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09f0649640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09e810e280 0x7f09e81a5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09f0649640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e81a5fd0 con 0x7f09e8105350 2024-03-12T05:58:23.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09f0649640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e81a6140 con 0x7f09e8100280 2024-03-12T05:58:23.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09f0649640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e81a6300 con 0x7f09e810e280 2024-03-12T05:58:23.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09edbe4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e8105350 0x7f09e810dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09edbe4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e8105350 0x7f09e810dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45608/0 (socket says 172.21.15.76:45608) 2024-03-12T05:58:23.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09edbe4640 1 -- 172.21.15.76:0/1940557330 learned_addr learned my addr 172.21.15.76:0/1940557330 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:23.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09ee3e5640 1 --2- 172.21.15.76:0/1940557330 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09e8100280 0x7f09e810d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09eebe6640 1 --2- 172.21.15.76:0/1940557330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09e810e280 0x7f09e81a5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09edbe4640 1 -- 172.21.15.76:0/1940557330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09e810e280 msgr2=0x7f09e81a5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:23.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09edbe4640 1 --2- 172.21.15.76:0/1940557330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09e810e280 0x7f09e81a5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09edbe4640 1 -- 172.21.15.76:0/1940557330 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09e8100280 msgr2=0x7f09e810d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:23.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09edbe4640 1 --2- 172.21.15.76:0/1940557330 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09e8100280 0x7f09e810d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09edbe4640 1 -- 172.21.15.76:0/1940557330 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e4009560 con 0x7f09e8105350 2024-03-12T05:58:23.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.803+0000 7f09edbe4640 1 --2- 172.21.15.76:0/1940557330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e8105350 0x7f09e810dd40 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f09d800c970 tx=0x7f09d800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:23.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.804+0000 7f09df7fe640 1 -- 172.21.15.76:0/1940557330 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8004440 con 0x7f09e8105350 2024-03-12T05:58:23.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.804+0000 7f09df7fe640 1 -- 172.21.15.76:0/1940557330 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09d8007d40 con 0x7f09e8105350 2024-03-12T05:58:23.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.804+0000 7f09f0649640 1 -- 172.21.15.76:0/1940557330 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e81a65e0 con 0x7f09e8105350 2024-03-12T05:58:23.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.804+0000 7f09df7fe640 1 -- 172.21.15.76:0/1940557330 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d800f450 con 0x7f09e8105350 2024-03-12T05:58:23.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.804+0000 7f09f0649640 1 -- 172.21.15.76:0/1940557330 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e81a6af0 con 0x7f09e8105350 2024-03-12T05:58:23.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.806+0000 7f09f3217740 1 Processor -- start 2024-03-12T05:58:23.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.806+0000 7f09f3217740 1 -- start start 2024-03-12T05:58:23.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.806+0000 7f09df7fe640 1 -- 172.21.15.76:0/1940557330 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f09d8007860 con 0x7f09e8105350 2024-03-12T05:58:23.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09f3217740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c680296760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09f3217740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c68029f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09f3217740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68029f8b0 0x55c6802a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09f3217740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c680204f90 con 0x55c680296380 2024-03-12T05:58:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09f3217740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6802024c0 con 0x55c680296d30 2024-03-12T05:58:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09f3217740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6801fdca0 con 0x55c68029f8b0 2024-03-12T05:58:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09eebe6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c680296760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09eebe6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c680296760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45612/0 (socket says 172.21.15.76:45612) 2024-03-12T05:58:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09eebe6640 1 -- 172.21.15.76:0/488440522 learned_addr learned my addr 172.21.15.76:0/488440522 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09ee3e5640 1 --2- 172.21.15.76:0/488440522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c68029f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09edbe4640 1 --2- 172.21.15.76:0/488440522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68029f8b0 0x55c6802a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09ee3e5640 1 -- 172.21.15.76:0/488440522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68029f8b0 msgr2=0x55c6802a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09df7fe640 1 --2- 172.21.15.76:0/1940557330 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f09d40a1bd0 0x7f09d40a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09ee3e5640 1 --2- 172.21.15.76:0/488440522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68029f8b0 0x55c6802a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09ee3e5640 1 -- 172.21.15.76:0/488440522 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 msgr2=0x55c680296760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09ee3e5640 1 --2- 172.21.15.76:0/488440522 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c680296760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09ee3e5640 1 -- 172.21.15.76:0/488440522 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6801c00b0 con 0x55c680296d30 2024-03-12T05:58:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09eebe6640 1 --2- 172.21.15.76:0/488440522 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c680296760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:58:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09df7fe640 1 -- 172.21.15.76:0/1940557330 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f09d80ccf50 con 0x7f09e8105350 2024-03-12T05:58:23.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09eebe6640 1 --2- 172.21.15.76:0/1940557330 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f09d40a1bd0 0x7f09d40a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.807+0000 7f09ee3e5640 1 --2- 172.21.15.76:0/488440522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c68029f2e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f09e4009230 tx=0x7f09e4004800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ead83e888a85affc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.808+0000 7f09dcbfe640 1 -- 172.21.15.76:0/488440522 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e40049f0 con 0x55c680296d30 2024-03-12T05:58:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.808+0000 7f09dcbfe640 1 -- 172.21.15.76:0/488440522 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09e4004030 con 0x55c680296d30 2024-03-12T05:58:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.808+0000 7f09dcbfe640 1 -- 172.21.15.76:0/488440522 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e40052d0 con 0x55c680296d30 2024-03-12T05:58:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.808+0000 7f09f3217740 1 -- 172.21.15.76:0/488440522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 msgr2=0x55c68029f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.808+0000 7f09eebe6640 1 --2- 172.21.15.76:0/1940557330 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f09d40a1bd0 0x7f09d40a4090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f09e000f2e0 tx=0x7f09e0004b50 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.808+0000 7f09f3217740 1 --2- 172.21.15.76:0/488440522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c68029f2e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f09e4009230 tx=0x7f09e4004800 comp rx=0 tx=0).stop 2024-03-12T05:58:23.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.808+0000 7f09f3217740 1 -- 172.21.15.76:0/488440522 shutdown_connections 2024-03-12T05:58:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 1 --2- 172.21.15.76:0/488440522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68029f8b0 0x55c6802a1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 1 --2- 172.21.15.76:0/488440522 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c68029f2e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 1 --2- 172.21.15.76:0/488440522 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c680296760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 1 -- 172.21.15.76:0/488440522 >> 172.21.15.76:0/488440522 conn(0x55c680290840 msgr2=0x55c680290c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 1 -- 172.21.15.76:0/488440522 shutdown_connections 2024-03-12T05:58:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 1 -- 172.21.15.76:0/488440522 wait complete. 2024-03-12T05:58:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 1 -- start start 2024-03-12T05:58:23.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 1 Processor -- start 2024-03-12T05:58:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c68029c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c68029cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68029f8b0 0x55c68029d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6801a2f40 con 0x55c680296380 2024-03-12T05:58:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c67fed14b0 con 0x55c680296d30 2024-03-12T05:58:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c680183670 con 0x55c68029f8b0 2024-03-12T05:58:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09f3217740 10 client.0 _ll_register_callbacks cb 0x55c6801f0f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09ee3e5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c68029c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09ee3e5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c68029c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45626/0 (socket says 172.21.15.76:45626) 2024-03-12T05:58:23.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.809+0000 7f09ee3e5640 1 -- 172.21.15.76:0/3575217499 learned_addr learned my addr 172.21.15.76:0/3575217499 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09edbe4640 1 --2- 172.21.15.76:0/3575217499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c68029cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09eebe6640 1 --2- 172.21.15.76:0/3575217499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68029f8b0 0x55c68029d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09ee3e5640 1 -- 172.21.15.76:0/3575217499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68029f8b0 msgr2=0x55c68029d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09ee3e5640 1 --2- 172.21.15.76:0/3575217499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68029f8b0 0x55c68029d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09ee3e5640 1 -- 172.21.15.76:0/3575217499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 msgr2=0x55c68029cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09ee3e5640 1 --2- 172.21.15.76:0/3575217499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c68029cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09ee3e5640 1 -- 172.21.15.76:0/3575217499 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e8069860 con 0x55c680296380 2024-03-12T05:58:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09edbe4640 1 --2- 172.21.15.76:0/3575217499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c68029cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09ee3e5640 1 --2- 172.21.15.76:0/3575217499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c68029c820 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f09e4036040 tx=0x7f09e40049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f5ac291e1d11e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09d1ffb640 1 -- 172.21.15.76:0/3575217499 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e4051070 con 0x55c680296380 2024-03-12T05:58:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09d1ffb640 1 -- 172.21.15.76:0/3575217499 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f09e4004310 con 0x55c680296380 2024-03-12T05:58:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09f3217740 1 -- 172.21.15.76:0/3575217499 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c6801d7f40 con 0x55c680296380 2024-03-12T05:58:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09d1ffb640 1 -- 172.21.15.76:0/3575217499 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09e40044d0 con 0x55c680296380 2024-03-12T05:58:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09f3217740 10 client.16185 fetch_fsmap learned FSMap version 24 2024-03-12T05:58:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09f3217740 1 -- 172.21.15.76:0/3575217499 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55c6801fc320 con 0x55c680296380 2024-03-12T05:58:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.810+0000 7f09d1ffb640 1 -- 172.21.15.76:0/3575217499 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c6801d7f40 con 0x55c680296380 2024-03-12T05:58:23.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.811+0000 7f09d1ffb640 1 -- 172.21.15.76:0/3575217499 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f09e4054030 con 0x55c680296380 2024-03-12T05:58:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.811+0000 7f09f3217740 10 client.16185 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:58:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.811+0000 7f09f3217740 10 client.16185 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.811+0000 7f09f3217740 20 client.16185 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.811+0000 7f09f3217740 1 --2- 172.21.15.76:0/3575217499 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c6802af9b0 0x55c6802b1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.811+0000 7f09f3217740 4 client.16185 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:58:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.811+0000 7f09f3217740 1 -- 172.21.15.76:0/3575217499 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6801f4310 con 0x55c6802af9b0 2024-03-12T05:58:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.811+0000 7f09edbe4640 1 --2- 172.21.15.76:0/3575217499 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c6802af9b0 0x55c6802b1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.811+0000 7f09edbe4640 1 --2- 172.21.15.76:0/3575217499 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c6802af9b0 0x55c6802b1df0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98c99cf4d59b2616 server_cookie=6fa28afe721e23d2 in_seq=0 out_seq=0 2024-03-12T05:58:23.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.811+0000 7f09d1ffb640 10 client.16185 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:23.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09d1ffb640 1 -- 172.21.15.76:0/3575217499 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c6801f4310 con 0x55c6802af9b0 2024-03-12T05:58:23.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09d1ffb640 10 client.16185 handle_command_reply: tid=0 2024-03-12T05:58:23.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09f3217740 1 client.16185 shutdown 2024-03-12T05:58:23.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09f3217740 1 -- 172.21.15.76:0/3575217499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 msgr2=0x55c68029c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:23.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09f3217740 1 --2- 172.21.15.76:0/3575217499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c68029c820 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f09e4036040 tx=0x7f09e40049f0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09f3217740 1 -- 172.21.15.76:0/3575217499 shutdown_connections 2024-03-12T05:58:23.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09f3217740 1 --2- 172.21.15.76:0/3575217499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c68029f8b0 0x55c68029d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09f3217740 1 --2- 172.21.15.76:0/3575217499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c68029cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09f3217740 1 --2- 172.21.15.76:0/3575217499 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c6802af9b0 0x55c6802b1df0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09f3217740 1 --2- 172.21.15.76:0/3575217499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c68029c820 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09d1ffb640 0 client.16185 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:23.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09f3217740 1 -- 172.21.15.76:0/3575217499 >> 172.21.15.76:0/3575217499 conn(0x55c680290840 msgr2=0x55c6802927a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:23.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09f3217740 1 -- 172.21.15.76:0/3575217499 shutdown_connections 2024-03-12T05:58:23.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09f3217740 1 -- 172.21.15.76:0/3575217499 wait complete. 2024-03-12T05:58:23.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.812+0000 7f09f3217740 20 client.16185 trim_cache size 0 max 16384 2024-03-12T05:58:23.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09f3217740 1 Processor -- start 2024-03-12T05:58:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09f3217740 1 -- start start 2024-03-12T05:58:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09f3217740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c680290840 0x55c68029ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09f3217740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c68029d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09f3217740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c680299a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09f3217740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c68028c180 con 0x55c680296380 2024-03-12T05:58:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09f3217740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6802ac8a0 con 0x55c680296d30 2024-03-12T05:58:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09f3217740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c680290370 con 0x55c680290840 2024-03-12T05:58:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09edbe4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c68029d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09edbe4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c68029d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45630/0 (socket says 172.21.15.76:45630) 2024-03-12T05:58:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09edbe4640 1 -- 172.21.15.76:0/1055645326 learned_addr learned my addr 172.21.15.76:0/1055645326 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09ee3e5640 1 --2- 172.21.15.76:0/1055645326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c680290840 0x55c68029ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09edbe4640 1 -- 172.21.15.76:0/1055645326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c680290840 msgr2=0x55c68029ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09edbe4640 1 --2- 172.21.15.76:0/1055645326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c680290840 0x55c68029ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09edbe4640 1 -- 172.21.15.76:0/1055645326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 msgr2=0x55c680299a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09edbe4640 1 --2- 172.21.15.76:0/1055645326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c680299a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09edbe4640 1 -- 172.21.15.76:0/1055645326 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c680198c00 con 0x55c680296380 2024-03-12T05:58:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09ee3e5640 1 --2- 172.21.15.76:0/1055645326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c680290840 0x55c68029ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:58:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09edbe4640 1 --2- 172.21.15.76:0/1055645326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c68029d0c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f09d80cacd0 tx=0x7f09d80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a758811aa9b9c88 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09d3fff640 1 -- 172.21.15.76:0/1055645326 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d80b1070 con 0x55c680296380 2024-03-12T05:58:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09d3fff640 1 -- 172.21.15.76:0/1055645326 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09d809a2b0 con 0x55c680296380 2024-03-12T05:58:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09d3fff640 1 -- 172.21.15.76:0/1055645326 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8003680 con 0x55c680296380 2024-03-12T05:58:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09f3217740 1 -- 172.21.15.76:0/1055645326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 msgr2=0x55c68029d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.828+0000 7f09f3217740 1 --2- 172.21.15.76:0/1055645326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c68029d0c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f09d80cacd0 tx=0x7f09d80035f0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 -- 172.21.15.76:0/1055645326 shutdown_connections 2024-03-12T05:58:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 --2- 172.21.15.76:0/1055645326 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c680299a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 --2- 172.21.15.76:0/1055645326 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c68029d0c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 --2- 172.21.15.76:0/1055645326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c680290840 0x55c68029ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 -- 172.21.15.76:0/1055645326 >> 172.21.15.76:0/1055645326 conn(0x55c680297480 msgr2=0x55c68028f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 -- 172.21.15.76:0/1055645326 shutdown_connections 2024-03-12T05:58:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 -- 172.21.15.76:0/1055645326 wait complete. 2024-03-12T05:58:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 -- start start 2024-03-12T05:58:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 Processor -- start 2024-03-12T05:58:23.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c680290840 0x55c6802a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c6802a5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c680305ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6802a6240 con 0x55c680296380 2024-03-12T05:58:23.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6802a63b0 con 0x55c680296d30 2024-03-12T05:58:23.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c680306420 con 0x55c680290840 2024-03-12T05:58:23.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09f3217740 10 client.0 _ll_register_callbacks cb 0x7f09e402c630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:23.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09edbe4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c6802a5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09edbe4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c6802a5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45638/0 (socket says 172.21.15.76:45638) 2024-03-12T05:58:23.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.829+0000 7f09edbe4640 1 -- 172.21.15.76:0/271005694 learned_addr learned my addr 172.21.15.76:0/271005694 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:23.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09edbe4640 1 -- 172.21.15.76:0/271005694 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c680290840 msgr2=0x55c6802a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:23.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09eebe6640 1 --2- 172.21.15.76:0/271005694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c680305ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09edbe4640 1 --2- 172.21.15.76:0/271005694 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c680290840 0x55c6802a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09edbe4640 1 -- 172.21.15.76:0/271005694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 msgr2=0x55c680305ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:23.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09edbe4640 1 --2- 172.21.15.76:0/271005694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c680305ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09edbe4640 1 -- 172.21.15.76:0/271005694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e81a6af0 con 0x55c680296380 2024-03-12T05:58:23.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09edbe4640 1 --2- 172.21.15.76:0/271005694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c6802a5ae0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f09d8007590 tx=0x7f09d80045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c099f97877306b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:23.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:23.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09d1ffb640 1 -- 172.21.15.76:0/271005694 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d80b1030 con 0x55c680296380 2024-03-12T05:58:23.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09d1ffb640 1 -- 172.21.15.76:0/271005694 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f09d80b2070 con 0x55c680296380 2024-03-12T05:58:23.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09f3217740 1 -- 172.21.15.76:0/271005694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c680223900 con 0x55c680296380 2024-03-12T05:58:23.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09d1ffb640 1 -- 172.21.15.76:0/271005694 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09d80b8120 con 0x55c680296380 2024-03-12T05:58:23.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09f3217740 10 client.16191 fetch_fsmap learned FSMap version 24 2024-03-12T05:58:23.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09f3217740 1 -- 172.21.15.76:0/271005694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55c6801be910 con 0x55c680296380 2024-03-12T05:58:23.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09d1ffb640 1 -- 172.21.15.76:0/271005694 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c680223900 con 0x55c680296380 2024-03-12T05:58:23.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09d1ffb640 1 -- 172.21.15.76:0/271005694 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f09d8099090 con 0x55c680296380 2024-03-12T05:58:23.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09f3217740 10 client.16191 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:58:23.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09f3217740 10 client.16191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:23.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09f3217740 20 client.16191 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:23.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09f3217740 1 --2- 172.21.15.76:0/271005694 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c68029f8b0 0x55c680309900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09f3217740 4 client.16191 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:58:23.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.830+0000 7f09f3217740 1 -- 172.21.15.76:0/271005694 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c67fefd860 con 0x55c68029f8b0 2024-03-12T05:58:23.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.831+0000 7f09ee3e5640 1 --2- 172.21.15.76:0/271005694 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c68029f8b0 0x55c680309900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.831+0000 7f09d1ffb640 10 client.16191 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:23.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.831+0000 7f09ee3e5640 1 --2- 172.21.15.76:0/271005694 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c68029f8b0 0x55c680309900 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e65e86aef665df3 server_cookie=bc93039a3d59e8c8 in_seq=0 out_seq=0 2024-03-12T05:58:23.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.832+0000 7f09d1ffb640 1 -- 172.21.15.76:0/271005694 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c67fefd860 con 0x55c68029f8b0 2024-03-12T05:58:23.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.832+0000 7f09d1ffb640 10 client.16191 handle_command_reply: tid=0 2024-03-12T05:58:23.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.832+0000 7f09f3217740 1 client.16191 shutdown 2024-03-12T05:58:23.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.832+0000 7f09f3217740 1 -- 172.21.15.76:0/271005694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 msgr2=0x55c6802a5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.832+0000 7f09f3217740 1 --2- 172.21.15.76:0/271005694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c6802a5ae0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f09d8007590 tx=0x7f09d80045c0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.832+0000 7f09f3217740 1 -- 172.21.15.76:0/271005694 shutdown_connections 2024-03-12T05:58:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.832+0000 7f09f3217740 1 --2- 172.21.15.76:0/271005694 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c680296d30 0x55c680305ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.832+0000 7f09f3217740 1 --2- 172.21.15.76:0/271005694 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c68029f8b0 0x55c680309900 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.832+0000 7f09f3217740 1 --2- 172.21.15.76:0/271005694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c680296380 0x55c6802a5ae0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.832+0000 7f09f3217740 1 --2- 172.21.15.76:0/271005694 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c680290840 0x55c6802a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.832+0000 7f09d1ffb640 0 client.16191 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.832+0000 7f09f3217740 1 -- 172.21.15.76:0/271005694 >> 172.21.15.76:0/271005694 conn(0x55c680297480 msgr2=0x55c68029afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f3217740 1 -- 172.21.15.76:0/271005694 shutdown_connections 2024-03-12T05:58:23.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f3217740 1 -- 172.21.15.76:0/271005694 wait complete. 2024-03-12T05:58:23.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f3217740 20 client.16191 trim_cache size 0 max 16384 2024-03-12T05:58:23.846 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:58:23.846 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T05:58:23.846 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "7adf43df-b19c-4e79-8643-5f0493d6c444", 2024-03-12T05:58:23.846 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T05:58:23.847 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:58:23.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f0649640 1 -- 172.21.15.76:0/1940557330 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f09d40a1bd0 msgr2=0x7f09d40a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:23.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f0649640 1 --2- 172.21.15.76:0/1940557330 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f09d40a1bd0 0x7f09d40a4090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f09e000f2e0 tx=0x7f09e0004b50 comp rx=0 tx=0).stop 2024-03-12T05:58:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f0649640 1 -- 172.21.15.76:0/1940557330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e8105350 msgr2=0x7f09e810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f0649640 1 --2- 172.21.15.76:0/1940557330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e8105350 0x7f09e810dd40 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f09d800c970 tx=0x7f09d800ce30 comp rx=0 tx=0).stop 2024-03-12T05:58:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f0649640 1 -- 172.21.15.76:0/1940557330 shutdown_connections 2024-03-12T05:58:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f0649640 1 --2- 172.21.15.76:0/1940557330 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f09d40a1bd0 0x7f09d40a4090 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f0649640 1 --2- 172.21.15.76:0/1940557330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09e810e280 0x7f09e81a5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f0649640 1 --2- 172.21.15.76:0/1940557330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e8105350 0x7f09e810dd40 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f0649640 1 --2- 172.21.15.76:0/1940557330 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09e8100280 0x7f09e810d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f0649640 1 -- 172.21.15.76:0/1940557330 >> 172.21.15.76:0/1940557330 conn(0x7f09e80fbae0 msgr2=0x7f09e80fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f0649640 1 -- 172.21.15.76:0/1940557330 shutdown_connections 2024-03-12T05:58:23.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.833+0000 7f09f0649640 1 -- 172.21.15.76:0/1940557330 wait complete. 2024-03-12T05:58:23.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:58:23.850 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:58:23.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.961+0000 7fc2c0d43640 1 -- 172.21.15.76:0/2319194235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2bc100c10 msgr2=0x7fc2bc104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:23.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.961+0000 7fc2c0d43640 1 --2- 172.21.15.76:0/2319194235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2bc100c10 0x7fc2bc104c50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0009930 tx=0x7fc2b002bc30 comp rx=0 tx=0).stop 2024-03-12T05:58:23.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.961+0000 7fc2c0d43640 1 -- 172.21.15.76:0/2319194235 shutdown_connections 2024-03-12T05:58:23.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.961+0000 7fc2c0d43640 1 --2- 172.21.15.76:0/2319194235 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2bc105330 0x7fc2bc10cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.961+0000 7fc2c0d43640 1 --2- 172.21.15.76:0/2319194235 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2bc100c10 0x7fc2bc104c50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.961+0000 7fc2c0d43640 1 --2- 172.21.15.76:0/2319194235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2bc100260 0x7fc2bc100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.961+0000 7fc2c0d43640 1 -- 172.21.15.76:0/2319194235 >> 172.21.15.76:0/2319194235 conn(0x7fc2bc077180 msgr2=0x7fc2bc077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:23.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.961+0000 7fc2c0d43640 1 -- 172.21.15.76:0/2319194235 shutdown_connections 2024-03-12T05:58:23.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.961+0000 7fc2c0d43640 1 -- 172.21.15.76:0/2319194235 wait complete. 2024-03-12T05:58:23.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.962+0000 7fc2c0d43640 1 Processor -- start 2024-03-12T05:58:23.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.962+0000 7fc2c0d43640 1 -- start start 2024-03-12T05:58:23.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.962+0000 7fc2c0d43640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2bc100260 0x7fc2bc198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2c0d43640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2bc105330 0x7fc2bc198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2c0d43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2bc1990f0 0x7fc2bc1a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2c0d43640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc1a3d40 con 0x7fc2bc105330 2024-03-12T05:58:23.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2c0d43640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc1a3eb0 con 0x7fc2bc100260 2024-03-12T05:58:23.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2c0d43640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc1a4070 con 0x7fc2bc1990f0 2024-03-12T05:58:23.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2ba59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2bc100260 0x7fc2bc198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2ba59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2bc100260 0x7fc2bc198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:38288/0 (socket says 172.21.15.76:38288) 2024-03-12T05:58:23.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2ba59c640 1 -- 172.21.15.76:0/1843357161 learned_addr learned my addr 172.21.15.76:0/1843357161 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:23.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2bad9d640 1 --2- 172.21.15.76:0/1843357161 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2bc1990f0 0x7fc2bc1a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2b9d9b640 1 --2- 172.21.15.76:0/1843357161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2bc105330 0x7fc2bc198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2ba59c640 1 -- 172.21.15.76:0/1843357161 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2bc1990f0 msgr2=0x7fc2bc1a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:23.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2ba59c640 1 --2- 172.21.15.76:0/1843357161 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2bc1990f0 0x7fc2bc1a36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2ba59c640 1 -- 172.21.15.76:0/1843357161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2bc105330 msgr2=0x7fc2bc198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:23.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2ba59c640 1 --2- 172.21.15.76:0/1843357161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2bc105330 0x7fc2bc198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2ba59c640 1 -- 172.21.15.76:0/1843357161 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b4009840 con 0x7fc2bc100260 2024-03-12T05:58:23.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.963+0000 7fc2ba59c640 1 --2- 172.21.15.76:0/1843357161 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2bc100260 0x7fc2bc198620 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac00e9f0 tx=0x7fc2ac00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:23.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.964+0000 7fc2ab7fe640 1 -- 172.21.15.76:0/1843357161 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac009720 con 0x7fc2bc100260 2024-03-12T05:58:23.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.964+0000 7fc2c0d43640 1 -- 172.21.15.76:0/1843357161 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b0009560 con 0x7fc2bc100260 2024-03-12T05:58:23.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.964+0000 7fc2ab7fe640 1 -- 172.21.15.76:0/1843357161 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2ac005500 con 0x7fc2bc100260 2024-03-12T05:58:23.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.964+0000 7fc2ab7fe640 1 -- 172.21.15.76:0/1843357161 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac010430 con 0x7fc2bc100260 2024-03-12T05:58:23.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.964+0000 7fc2c0d43640 1 -- 172.21.15.76:0/1843357161 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2bc1a46a0 con 0x7fc2bc100260 2024-03-12T05:58:23.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2c39ea740 1 Processor -- start 2024-03-12T05:58:23.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2c39ea740 1 -- start start 2024-03-12T05:58:23.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2ab7fe640 1 -- 172.21.15.76:0/1843357161 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc2ac005020 con 0x7fc2bc100260 2024-03-12T05:58:23.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2c39ea740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646a2b0 0x56371646a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2c39ea740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56371646abd0 0x563716473270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2c39ea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5637164737b0 0x563716475ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2c39ea740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637163d9280 con 0x56371646abd0 2024-03-12T05:58:23.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2c39ea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637163d67b0 con 0x5637164737b0 2024-03-12T05:58:23.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2c39ea740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637163d1f90 con 0x56371646a2b0 2024-03-12T05:58:23.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2b9d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5637164737b0 0x563716475ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2b9d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5637164737b0 0x563716475ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:38296/0 (socket says 172.21.15.76:38296) 2024-03-12T05:58:23.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2b9d9b640 1 -- 172.21.15.76:0/1886870573 learned_addr learned my addr 172.21.15.76:0/1886870573 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:23.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2ba59c640 1 --2- 172.21.15.76:0/1886870573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56371646abd0 0x563716473270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2bad9d640 1 --2- 172.21.15.76:0/1886870573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646a2b0 0x56371646a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2b9d9b640 1 -- 172.21.15.76:0/1886870573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646a2b0 msgr2=0x56371646a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:23.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2b9d9b640 1 --2- 172.21.15.76:0/1886870573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646a2b0 0x56371646a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2b9d9b640 1 -- 172.21.15.76:0/1886870573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56371646abd0 msgr2=0x563716473270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:23.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2b9d9b640 1 --2- 172.21.15.76:0/1886870573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56371646abd0 0x563716473270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2ab7fe640 1 --2- 172.21.15.76:0/1843357161 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2a40a1ba0 0x7fc2a40a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2b9d9b640 1 -- 172.21.15.76:0/1886870573 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563716390570 con 0x5637164737b0 2024-03-12T05:58:23.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2bad9d640 1 --2- 172.21.15.76:0/1843357161 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2a40a1ba0 0x7fc2a40a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2ab7fe640 1 -- 172.21.15.76:0/1843357161 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc2ac014070 con 0x7fc2bc100260 2024-03-12T05:58:23.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.968+0000 7fc2b9d9b640 1 --2- 172.21.15.76:0/1886870573 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5637164737b0 0x563716475ba0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0009900 tx=0x7fc2b0005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6b35738157ba6c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:23.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.969+0000 7fc2bad9d640 1 --2- 172.21.15.76:0/1843357161 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2a40a1ba0 0x7fc2a40a4060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4005fd0 tx=0x7fc2b4009500 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:23.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.969+0000 7fc2a8ff9640 1 -- 172.21.15.76:0/1886870573 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0004800 con 0x5637164737b0 2024-03-12T05:58:23.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.969+0000 7fc2a8ff9640 1 -- 172.21.15.76:0/1886870573 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2b0004960 con 0x5637164737b0 2024-03-12T05:58:23.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.969+0000 7fc2a8ff9640 1 -- 172.21.15.76:0/1886870573 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0004800 con 0x5637164737b0 2024-03-12T05:58:23.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.970+0000 7fc2c39ea740 1 -- 172.21.15.76:0/1886870573 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5637164737b0 msgr2=0x563716475ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:23.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.970+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/1886870573 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5637164737b0 0x563716475ba0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0009900 tx=0x7fc2b0005c00 comp rx=0 tx=0).stop 2024-03-12T05:58:23.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.970+0000 7fc2c39ea740 1 -- 172.21.15.76:0/1886870573 shutdown_connections 2024-03-12T05:58:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.970+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/1886870573 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5637164737b0 0x563716475ba0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.970+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/1886870573 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56371646abd0 0x563716473270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.970+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/1886870573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646a2b0 0x56371646a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.970+0000 7fc2c39ea740 1 -- 172.21.15.76:0/1886870573 >> 172.21.15.76:0/1886870573 conn(0x563716464990 msgr2=0x563716464d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.970+0000 7fc2c39ea740 1 -- 172.21.15.76:0/1886870573 shutdown_connections 2024-03-12T05:58:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.970+0000 7fc2c39ea740 1 -- 172.21.15.76:0/1886870573 wait complete. 2024-03-12T05:58:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.970+0000 7fc2c39ea740 1 -- start start 2024-03-12T05:58:23.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.970+0000 7fc2c39ea740 1 Processor -- start 2024-03-12T05:58:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.970+0000 7fc2c39ea740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646a2b0 0x56371647d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2c39ea740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56371646abd0 0x56371647d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2c39ea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5637164737b0 0x563716477810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2c39ea740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637160a5430 con 0x56371646abd0 2024-03-12T05:58:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2c39ea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563716061e00 con 0x5637164737b0 2024-03-12T05:58:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2c39ea740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563716356fd0 con 0x56371646a2b0 2024-03-12T05:58:23.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2c39ea740 10 client.0 _ll_register_callbacks cb 0x5637163acfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2b9d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56371646abd0 0x56371647d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2b9d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56371646abd0 0x56371647d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:45688/0 (socket says 172.21.15.76:45688) 2024-03-12T05:58:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2b9d9b640 1 -- 172.21.15.76:0/3690150066 learned_addr learned my addr 172.21.15.76:0/3690150066 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2bad9d640 1 --2- 172.21.15.76:0/3690150066 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5637164737b0 0x563716477810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2ba59c640 1 --2- 172.21.15.76:0/3690150066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646a2b0 0x56371647d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2b9d9b640 1 -- 172.21.15.76:0/3690150066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646a2b0 msgr2=0x56371647d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2b9d9b640 1 --2- 172.21.15.76:0/3690150066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646a2b0 0x56371647d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2b9d9b640 1 -- 172.21.15.76:0/3690150066 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5637164737b0 msgr2=0x563716477810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2b9d9b640 1 --2- 172.21.15.76:0/3690150066 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5637164737b0 0x563716477810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2b9d9b640 1 -- 172.21.15.76:0/3690150066 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2bc10d710 con 0x56371646abd0 2024-03-12T05:58:23.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2bad9d640 1 --2- 172.21.15.76:0/3690150066 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5637164737b0 0x563716477810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:58:23.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2b9d9b640 1 --2- 172.21.15.76:0/3690150066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56371646abd0 0x56371647d980 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0036040 tx=0x7fc2b002cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b513425d329974 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:23.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.971+0000 7fc2a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:23.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.972+0000 7fc2a1ffb640 1 -- 172.21.15.76:0/3690150066 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0005300 con 0x56371646abd0 2024-03-12T05:58:23.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.972+0000 7fc2a1ffb640 1 -- 172.21.15.76:0/3690150066 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc2b0050070 con 0x56371646abd0 2024-03-12T05:58:23.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.972+0000 7fc2c39ea740 1 -- 172.21.15.76:0/3690150066 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5637160d17e0 con 0x56371646abd0 2024-03-12T05:58:23.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.972+0000 7fc2a1ffb640 1 -- 172.21.15.76:0/3690150066 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2b0004ad0 con 0x56371646abd0 2024-03-12T05:58:23.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.972+0000 7fc2c39ea740 10 client.16203 fetch_fsmap learned FSMap version 24 2024-03-12T05:58:23.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.972+0000 7fc2c39ea740 1 -- 172.21.15.76:0/3690150066 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5637163d0610 con 0x56371646abd0 2024-03-12T05:58:23.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.972+0000 7fc2a1ffb640 1 -- 172.21.15.76:0/3690150066 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5637160d17e0 con 0x56371646abd0 2024-03-12T05:58:23.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.973+0000 7fc2a1ffb640 1 -- 172.21.15.76:0/3690150066 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc2b003c050 con 0x56371646abd0 2024-03-12T05:58:23.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.973+0000 7fc2c39ea740 10 client.16203 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:58:23.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.973+0000 7fc2c39ea740 10 client.16203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:23.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.973+0000 7fc2c39ea740 20 client.16203 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:23.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.973+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/3690150066 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5637164879f0 0x563716489e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:23.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.973+0000 7fc2c39ea740 4 client.16203 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:58:23.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.973+0000 7fc2c39ea740 1 -- 172.21.15.76:0/3690150066 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56371610e720 con 0x5637164879f0 2024-03-12T05:58:23.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.973+0000 7fc2ba59c640 1 --2- 172.21.15.76:0/3690150066 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5637164879f0 0x563716489e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:23.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.973+0000 7fc2a1ffb640 10 client.16203 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:23.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.973+0000 7fc2ba59c640 1 --2- 172.21.15.76:0/3690150066 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5637164879f0 0x563716489e10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c7de6b50975696 server_cookie=b3d6eae176540f58 in_seq=0 out_seq=0 2024-03-12T05:58:23.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2a1ffb640 1 -- 172.21.15.76:0/3690150066 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56371610e720 con 0x5637164879f0 2024-03-12T05:58:23.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2a1ffb640 10 client.16203 handle_command_reply: tid=0 2024-03-12T05:58:23.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2c39ea740 1 client.16203 shutdown 2024-03-12T05:58:23.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2c39ea740 1 -- 172.21.15.76:0/3690150066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56371646abd0 msgr2=0x56371647d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:23.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/3690150066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56371646abd0 0x56371647d980 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0036040 tx=0x7fc2b002cd60 comp rx=0 tx=0).stop 2024-03-12T05:58:23.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2c39ea740 1 -- 172.21.15.76:0/3690150066 shutdown_connections 2024-03-12T05:58:23.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/3690150066 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5637164737b0 0x563716477810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/3690150066 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5637164879f0 0x563716489e10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/3690150066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56371646abd0 0x56371647d980 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/3690150066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646a2b0 0x56371647d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:23.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2c39ea740 1 -- 172.21.15.76:0/3690150066 >> 172.21.15.76:0/3690150066 conn(0x563716464990 msgr2=0x563716466730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:23.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2a1ffb640 0 client.16203 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:23.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2c39ea740 1 -- 172.21.15.76:0/3690150066 shutdown_connections 2024-03-12T05:58:23.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2c39ea740 1 -- 172.21.15.76:0/3690150066 wait complete. 2024-03-12T05:58:23.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.974+0000 7fc2c39ea740 20 client.16203 trim_cache size 0 max 16384 2024-03-12T05:58:24.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2c39ea740 1 Processor -- start 2024-03-12T05:58:24.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2c39ea740 1 -- start start 2024-03-12T05:58:24.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2c39ea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56371646a2b0 0x56371647d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:24.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2c39ea740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646abd0 0x56371647d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:24.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2c39ea740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637164737b0 0x5637164770a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:24.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2c39ea740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637164694b0 con 0x5637164737b0 2024-03-12T05:58:24.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2c39ea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563716469f10 con 0x56371646a2b0 2024-03-12T05:58:24.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2c39ea740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637164659e0 con 0x56371646abd0 2024-03-12T05:58:24.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2ba59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56371646a2b0 0x56371647d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:24.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2bad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637164737b0 0x5637164770a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:24.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2ba59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56371646a2b0 0x56371647d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:38310/0 (socket says 172.21.15.76:38310) 2024-03-12T05:58:24.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2ba59c640 1 -- 172.21.15.76:0/2269337209 learned_addr learned my addr 172.21.15.76:0/2269337209 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:24.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2b9d9b640 1 --2- 172.21.15.76:0/2269337209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646abd0 0x56371647d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:24.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2ba59c640 1 -- 172.21.15.76:0/2269337209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646abd0 msgr2=0x56371647d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:24.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2ba59c640 1 --2- 172.21.15.76:0/2269337209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646abd0 0x56371647d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2ba59c640 1 -- 172.21.15.76:0/2269337209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637164737b0 msgr2=0x5637164770a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:24.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.996+0000 7fc2ba59c640 1 --2- 172.21.15.76:0/2269337209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637164737b0 0x5637164770a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.997+0000 7fc2ba59c640 1 -- 172.21.15.76:0/2269337209 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56371636cac0 con 0x56371646a2b0 2024-03-12T05:58:24.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.997+0000 7fc2b9d9b640 1 --2- 172.21.15.76:0/2269337209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646abd0 0x56371647d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:24.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.997+0000 7fc2ba59c640 1 --2- 172.21.15.76:0/2269337209 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56371646a2b0 0x56371647d230 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0971f0 tx=0x7fc2ac0cf040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=963ff5d7a94cfe1a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:24.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.997+0000 7fc2a3fff640 1 -- 172.21.15.76:0/2269337209 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac0d7040 con 0x56371646a2b0 2024-03-12T05:58:24.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.997+0000 7fc2a3fff640 1 -- 172.21.15.76:0/2269337209 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2ac09a5f0 con 0x56371646a2b0 2024-03-12T05:58:24.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.997+0000 7fc2a3fff640 1 -- 172.21.15.76:0/2269337209 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac09e970 con 0x56371646a2b0 2024-03-12T05:58:24.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.998+0000 7fc2c39ea740 1 -- 172.21.15.76:0/2269337209 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56371646a2b0 msgr2=0x56371647d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:24.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.998+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/2269337209 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56371646a2b0 0x56371647d230 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0971f0 tx=0x7fc2ac0cf040 comp rx=0 tx=0).stop 2024-03-12T05:58:24.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.998+0000 7fc2c39ea740 1 -- 172.21.15.76:0/2269337209 shutdown_connections 2024-03-12T05:58:24.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.998+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/2269337209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637164737b0 0x5637164770a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.998+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/2269337209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646abd0 0x56371647d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.998+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/2269337209 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56371646a2b0 0x56371647d230 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.998+0000 7fc2c39ea740 1 -- 172.21.15.76:0/2269337209 >> 172.21.15.76:0/2269337209 conn(0x563716464990 msgr2=0x56371646fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:24.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.998+0000 7fc2c39ea740 1 -- 172.21.15.76:0/2269337209 shutdown_connections 2024-03-12T05:58:24.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.998+0000 7fc2c39ea740 1 -- 172.21.15.76:0/2269337209 wait complete. 2024-03-12T05:58:24.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.998+0000 7fc2c39ea740 1 -- start start 2024-03-12T05:58:24.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.998+0000 7fc2c39ea740 1 Processor -- start 2024-03-12T05:58:24.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2c39ea740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56371646a2b0 0x5637164804e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:24.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2c39ea740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646abd0 0x563716480a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:24.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2c39ea740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637164737b0 0x5637164ddeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:24.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2c39ea740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563716481320 con 0x5637164737b0 2024-03-12T05:58:24.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2c39ea740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637164de3f0 con 0x56371646a2b0 2024-03-12T05:58:24.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2c39ea740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637164de530 con 0x56371646abd0 2024-03-12T05:58:24.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2c39ea740 10 client.0 _ll_register_callbacks cb 0x7fc2ac005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:24.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2ba59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56371646a2b0 0x5637164804e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:24.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2ba59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56371646a2b0 0x5637164804e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:38318/0 (socket says 172.21.15.76:38318) 2024-03-12T05:58:24.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2ba59c640 1 -- 172.21.15.76:0/856779539 learned_addr learned my addr 172.21.15.76:0/856779539 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:24.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2bad9d640 1 --2- 172.21.15.76:0/856779539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637164737b0 0x5637164ddeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:24.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2b9d9b640 1 --2- 172.21.15.76:0/856779539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646abd0 0x563716480a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:24.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2ba59c640 1 -- 172.21.15.76:0/856779539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646abd0 msgr2=0x563716480a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:24.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2ba59c640 1 --2- 172.21.15.76:0/856779539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646abd0 0x563716480a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2ba59c640 1 -- 172.21.15.76:0/856779539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637164737b0 msgr2=0x5637164ddeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:24.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2ba59c640 1 --2- 172.21.15.76:0/856779539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637164737b0 0x5637164ddeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2ba59c640 1 -- 172.21.15.76:0/856779539 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ac0cde70 con 0x56371646a2b0 2024-03-12T05:58:24.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2ba59c640 1 --2- 172.21.15.76:0/856779539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56371646a2b0 0x5637164804e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0cb940 tx=0x7fc2ac009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff60080bf5e44f8e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:24.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T05:58:24.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:23.999+0000 7fc2a1ffb640 1 -- 172.21.15.76:0/856779539 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac005670 con 0x56371646a2b0 2024-03-12T05:58:24.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.000+0000 7fc2a1ffb640 1 -- 172.21.15.76:0/856779539 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc2ac09af20 con 0x56371646a2b0 2024-03-12T05:58:24.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.000+0000 7fc2c39ea740 1 -- 172.21.15.76:0/856779539 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2b0004960 con 0x56371646a2b0 2024-03-12T05:58:24.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.000+0000 7fc2a1ffb640 1 -- 172.21.15.76:0/856779539 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2ac005670 con 0x56371646a2b0 2024-03-12T05:58:24.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.000+0000 7fc2c39ea740 10 client.25601 fetch_fsmap learned FSMap version 24 2024-03-12T05:58:24.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.000+0000 7fc2c39ea740 1 -- 172.21.15.76:0/856779539 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5637163926a0 con 0x56371646a2b0 2024-03-12T05:58:24.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.000+0000 7fc2a1ffb640 1 -- 172.21.15.76:0/856779539 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2b0004960 con 0x56371646a2b0 2024-03-12T05:58:24.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.000+0000 7fc2a1ffb640 1 -- 172.21.15.76:0/856779539 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc2ac01f060 con 0x56371646a2b0 2024-03-12T05:58:24.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.000+0000 7fc2c39ea740 10 client.25601 fetch_fsmap finished waiting for FSMap version 24 2024-03-12T05:58:24.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.000+0000 7fc2c39ea740 10 client.25601 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:24.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.000+0000 7fc2c39ea740 20 client.25601 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:24.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.000+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/856779539 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5637164df8f0 0x5637164e1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:24.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.000+0000 7fc2c39ea740 4 client.25601 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:58:24.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.000+0000 7fc2c39ea740 1 -- 172.21.15.76:0/856779539 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56371615d890 con 0x5637164df8f0 2024-03-12T05:58:24.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.001+0000 7fc2b9d9b640 1 --2- 172.21.15.76:0/856779539 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5637164df8f0 0x5637164e1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:24.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.001+0000 7fc2b9d9b640 1 --2- 172.21.15.76:0/856779539 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5637164df8f0 0x5637164e1ce0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f99452d58b47045b server_cookie=18f10ed39044dedf in_seq=0 out_seq=0 2024-03-12T05:58:24.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.001+0000 7fc2a1ffb640 10 client.25601 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:24.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.001+0000 7fc2a1ffb640 1 -- 172.21.15.76:0/856779539 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56371615d890 con 0x5637164df8f0 2024-03-12T05:58:24.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.001+0000 7fc2a1ffb640 10 client.25601 handle_command_reply: tid=0 2024-03-12T05:58:24.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.001+0000 7fc2c39ea740 1 client.25601 shutdown 2024-03-12T05:58:24.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.002+0000 7fc2c39ea740 1 -- 172.21.15.76:0/856779539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56371646a2b0 msgr2=0x5637164804e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:24.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.002+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/856779539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56371646a2b0 0x5637164804e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0cb940 tx=0x7fc2ac009310 comp rx=0 tx=0).stop 2024-03-12T05:58:24.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.002+0000 7fc2c39ea740 1 -- 172.21.15.76:0/856779539 shutdown_connections 2024-03-12T05:58:24.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.002+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/856779539 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5637164df8f0 0x5637164e1ce0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.002+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/856779539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5637164737b0 0x5637164ddeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.002+0000 7fc2a1ffb640 0 client.25601 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:24.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.002+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/856779539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56371646abd0 0x563716480a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.002+0000 7fc2c39ea740 1 --2- 172.21.15.76:0/856779539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56371646a2b0 0x5637164804e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.002+0000 7fc2c39ea740 1 -- 172.21.15.76:0/856779539 >> 172.21.15.76:0/856779539 conn(0x563716464990 msgr2=0x56371647d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:24.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.002+0000 7fc2c39ea740 1 -- 172.21.15.76:0/856779539 shutdown_connections 2024-03-12T05:58:24.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.002+0000 7fc2c39ea740 1 -- 172.21.15.76:0/856779539 wait complete. 2024-03-12T05:58:24.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.002+0000 7fc2c39ea740 20 client.25601 trim_cache size 0 max 16384 2024-03-12T05:58:24.016 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:58:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:58:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T05:58:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "7adf43df-b19c-4e79-8643-5f0493d6c444": { 2024-03-12T05:58:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T05:58:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "7adf43df-b19c-4e79-8643-5f0493d6c444", 2024-03-12T05:58:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T05:58:24.016 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:58:24.017 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:58:24.017 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:58:24.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.003+0000 7fc2c0d43640 1 -- 172.21.15.76:0/1843357161 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2a40a1ba0 msgr2=0x7fc2a40a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:24.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.003+0000 7fc2c0d43640 1 --2- 172.21.15.76:0/1843357161 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2a40a1ba0 0x7fc2a40a4060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4005fd0 tx=0x7fc2b4009500 comp rx=0 tx=0).stop 2024-03-12T05:58:24.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.003+0000 7fc2c0d43640 1 -- 172.21.15.76:0/1843357161 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2bc100260 msgr2=0x7fc2bc198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:24.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.003+0000 7fc2c0d43640 1 --2- 172.21.15.76:0/1843357161 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2bc100260 0x7fc2bc198620 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac00e9f0 tx=0x7fc2ac00edf0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.003+0000 7fc2c0d43640 1 -- 172.21.15.76:0/1843357161 shutdown_connections 2024-03-12T05:58:24.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.003+0000 7fc2c0d43640 1 --2- 172.21.15.76:0/1843357161 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2a40a1ba0 0x7fc2a40a4060 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.003+0000 7fc2c0d43640 1 --2- 172.21.15.76:0/1843357161 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2bc1990f0 0x7fc2bc1a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.003+0000 7fc2c0d43640 1 --2- 172.21.15.76:0/1843357161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2bc105330 0x7fc2bc198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.003+0000 7fc2c0d43640 1 --2- 172.21.15.76:0/1843357161 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2bc100260 0x7fc2bc198620 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:24.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.003+0000 7fc2c0d43640 1 -- 172.21.15.76:0/1843357161 >> 172.21.15.76:0/1843357161 conn(0x7fc2bc077180 msgr2=0x7fc2bc0ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:24.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.003+0000 7fc2c0d43640 1 -- 172.21.15.76:0/1843357161 shutdown_connections 2024-03-12T05:58:24.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:24.003+0000 7fc2c0d43640 1 -- 172.21.15.76:0/1843357161 wait complete. 2024-03-12T05:58:24.021 INFO:tasks.cephfs.filesystem:scrub status for tag:7adf43df-b19c-4e79-8643-5f0493d6c444 - {'path': '/', 'tag': '7adf43df-b19c-4e79-8643-5f0493d6c444', 'options': 'recursive,force'} 2024-03-12T05:58:24.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:24 smithi082 ceph-mon[36382]: pgmap v81: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 10 op/s 2024-03-12T05:58:24.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:24 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:58:24.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:24 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:58:24.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:24 smithi088 ceph-mon[36585]: pgmap v81: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 10 op/s 2024-03-12T05:58:24.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:24 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:58:24.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:24 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:58:24.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:24 smithi076 ceph-mon[32095]: pgmap v81: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 10 op/s 2024-03-12T05:58:24.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:24 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:58:24.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:24 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:58:25.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:25 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T05:58:25.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:25 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:25.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:25 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:58:25.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:25 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:25.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:25 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:25.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:25 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:25.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:25 smithi082 ceph-mon[36382]: pgmap v82: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2024-03-12T05:58:25.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:25 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T05:58:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:25 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:25 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:58:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:25 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:25 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:25 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:25 smithi088 ceph-mon[36585]: pgmap v82: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2024-03-12T05:58:25.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:25 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T05:58:25.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:25 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:25.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:25 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:58:25.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:25 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:25.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:25 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:25.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:25 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:25.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:25 smithi076 ceph-mon[32095]: pgmap v82: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2024-03-12T05:58:28.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:28 smithi082 ceph-mon[36382]: pgmap v83: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2024-03-12T05:58:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:28 smithi088 ceph-mon[36585]: pgmap v83: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2024-03-12T05:58:28.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:28 smithi076 ceph-mon[32095]: pgmap v83: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2024-03-12T05:58:30.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:30 smithi082 ceph-mon[36382]: pgmap v84: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 122 KiB/s wr, 11 op/s 2024-03-12T05:58:30.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:30 smithi088 ceph-mon[36585]: pgmap v84: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 122 KiB/s wr, 11 op/s 2024-03-12T05:58:30.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:30 smithi076 ceph-mon[32095]: pgmap v84: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 122 KiB/s wr, 11 op/s 2024-03-12T05:58:32.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:32 smithi082 ceph-mon[36382]: pgmap v85: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 117 KiB/s wr, 7 op/s 2024-03-12T05:58:32.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:32 smithi088 ceph-mon[36585]: pgmap v85: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 117 KiB/s wr, 7 op/s 2024-03-12T05:58:32.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:32 smithi076 ceph-mon[32095]: pgmap v85: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 117 KiB/s wr, 7 op/s 2024-03-12T05:58:34.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:34 smithi082 ceph-mon[36382]: pgmap v86: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 119 KiB/s wr, 11 op/s 2024-03-12T05:58:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:34 smithi088 ceph-mon[36585]: pgmap v86: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 119 KiB/s wr, 11 op/s 2024-03-12T05:58:34.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:34 smithi076 ceph-mon[32095]: pgmap v86: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 119 KiB/s wr, 11 op/s 2024-03-12T05:58:36.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:36 smithi082 ceph-mon[36382]: pgmap v87: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2024-03-12T05:58:36.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:36 smithi088 ceph-mon[36585]: pgmap v87: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2024-03-12T05:58:36.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:36 smithi076 ceph-mon[32095]: pgmap v87: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2024-03-12T05:58:38.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:38 smithi082 ceph-mon[36382]: pgmap v88: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2024-03-12T05:58:38.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:38 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:58:38.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:38 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:58:38.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:38 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:58:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:38 smithi088 ceph-mon[36585]: pgmap v88: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2024-03-12T05:58:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:38 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:58:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:38 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:58:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:38 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:58:38.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:38 smithi076 ceph-mon[32095]: pgmap v88: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2024-03-12T05:58:38.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:38 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:58:38.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:38 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:58:38.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:38 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:58:40.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:40 smithi082 ceph-mon[36382]: pgmap v89: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 54 KiB/s wr, 11 op/s 2024-03-12T05:58:40.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:40 smithi088 ceph-mon[36585]: pgmap v89: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 54 KiB/s wr, 11 op/s 2024-03-12T05:58:40.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:40 smithi076 ceph-mon[32095]: pgmap v89: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 54 KiB/s wr, 11 op/s 2024-03-12T05:58:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:42 smithi082 ceph-mon[36382]: pgmap v90: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.4 KiB/s wr, 7 op/s 2024-03-12T05:58:42.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:42 smithi088 ceph-mon[36585]: pgmap v90: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.4 KiB/s wr, 7 op/s 2024-03-12T05:58:42.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:42 smithi076 ceph-mon[32095]: pgmap v90: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.4 KiB/s wr, 7 op/s 2024-03-12T05:58:44.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:44 smithi082 ceph-mon[36382]: pgmap v91: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 KiB/s wr, 10 op/s 2024-03-12T05:58:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:44 smithi088 ceph-mon[36585]: pgmap v91: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 KiB/s wr, 10 op/s 2024-03-12T05:58:44.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:44 smithi076 ceph-mon[32095]: pgmap v91: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 KiB/s wr, 10 op/s 2024-03-12T05:58:45.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:45 smithi082 ceph-mon[36382]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:58:45.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:45 smithi088 ceph-mon[36585]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:58:45.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:45 smithi076 ceph-mon[32095]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T05:58:46.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:46 smithi082 ceph-mon[36382]: pgmap v92: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-03-12T05:58:46.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:46 smithi088 ceph-mon[36585]: pgmap v92: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-03-12T05:58:46.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:46 smithi076 ceph-mon[32095]: pgmap v92: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-03-12T05:58:47.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:47 smithi082 ceph-mon[36382]: pgmap v93: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-03-12T05:58:47.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:47 smithi088 ceph-mon[36585]: pgmap v93: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-03-12T05:58:47.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:47 smithi076 ceph-mon[32095]: pgmap v93: 97 pgs: 97 active+clean; 81 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-03-12T05:58:50.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:50 smithi082 ceph-mon[36382]: pgmap v94: 97 pgs: 97 active+clean; 82 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 13 op/s 2024-03-12T05:58:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:50 smithi088 ceph-mon[36585]: pgmap v94: 97 pgs: 97 active+clean; 82 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 13 op/s 2024-03-12T05:58:50.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:50 smithi076 ceph-mon[32095]: pgmap v94: 97 pgs: 97 active+clean; 82 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 13 op/s 2024-03-12T05:58:52.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:52 smithi082 ceph-mon[36382]: pgmap v95: 97 pgs: 97 active+clean; 82 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 9 op/s 2024-03-12T05:58:52.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:52 smithi088 ceph-mon[36585]: pgmap v95: 97 pgs: 97 active+clean; 82 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 9 op/s 2024-03-12T05:58:52.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:52 smithi076 ceph-mon[32095]: pgmap v95: 97 pgs: 97 active+clean; 82 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 9 op/s 2024-03-12T05:58:54.022 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:58:54.023 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:58:54.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.143+0000 7f9ae0725640 1 -- 172.21.15.76:0/4275650413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ad8102cc0 msgr2=0x7f9ad810b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.143+0000 7f9ae0725640 1 --2- 172.21.15.76:0/4275650413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ad8102cc0 0x7f9ad810b1a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8009900 tx=0x7f9ac802bc30 comp rx=0 tx=0).stop 2024-03-12T05:58:54.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.143+0000 7f9ae0725640 1 -- 172.21.15.76:0/4275650413 shutdown_connections 2024-03-12T05:58:54.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.143+0000 7f9ae0725640 1 --2- 172.21.15.76:0/4275650413 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ad810b770 0x7f9ad810dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.143+0000 7f9ae0725640 1 --2- 172.21.15.76:0/4275650413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ad8102cc0 0x7f9ad810b1a0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.143+0000 7f9ae0725640 1 --2- 172.21.15.76:0/4275650413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad81023a0 0x7f9ad8102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.143+0000 7f9ae0725640 1 -- 172.21.15.76:0/4275650413 >> 172.21.15.76:0/4275650413 conn(0x7f9ad80fbac0 msgr2=0x7f9ad80fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.143+0000 7f9ae0725640 1 -- 172.21.15.76:0/4275650413 shutdown_connections 2024-03-12T05:58:54.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.143+0000 7f9ae0725640 1 -- 172.21.15.76:0/4275650413 wait complete. 2024-03-12T05:58:54.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.144+0000 7f9ae0725640 1 Processor -- start 2024-03-12T05:58:54.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.144+0000 7f9ae0725640 1 -- start start 2024-03-12T05:58:54.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.144+0000 7f9ae0725640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ad81023a0 0x7f9ad81977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.144+0000 7f9ae0725640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad8102cc0 0x7f9ad8197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.144+0000 7f9ae0725640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ad810b770 0x7f9ad819c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.144+0000 7f9ae0725640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad819c610 con 0x7f9ad810b770 2024-03-12T05:58:54.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.144+0000 7f9ae0725640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad819c780 con 0x7f9ad81023a0 2024-03-12T05:58:54.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.144+0000 7f9ae0725640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad819c940 con 0x7f9ad8102cc0 2024-03-12T05:58:54.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.144+0000 7f9adecc2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ad810b770 0x7f9ad819c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.144+0000 7f9adecc2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ad810b770 0x7f9ad819c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51456/0 (socket says 172.21.15.76:51456) 2024-03-12T05:58:54.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.144+0000 7f9adecc2640 1 -- 172.21.15.76:0/1902177880 learned_addr learned my addr 172.21.15.76:0/1902177880 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.145+0000 7f9ade4c1640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ad81023a0 0x7f9ad81977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.145+0000 7f9addcc0640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad8102cc0 0x7f9ad8197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.145+0000 7f9adecc2640 1 -- 172.21.15.76:0/1902177880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad8102cc0 msgr2=0x7f9ad8197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.145+0000 7f9adecc2640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad8102cc0 0x7f9ad8197ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.145+0000 7f9adecc2640 1 -- 172.21.15.76:0/1902177880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ad81023a0 msgr2=0x7f9ad81977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.145+0000 7f9adecc2640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ad81023a0 0x7f9ad81977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.145+0000 7f9adecc2640 1 -- 172.21.15.76:0/1902177880 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac8009560 con 0x7f9ad810b770 2024-03-12T05:58:54.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.145+0000 7f9addcc0640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad8102cc0 0x7f9ad8197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:54.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.145+0000 7f9adecc2640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ad810b770 0x7f9ad819c010 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000eaa0 tx=0x7f9ad000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.145+0000 7f9acf7fe640 1 -- 172.21.15.76:0/1902177880 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad000cdb0 con 0x7f9ad810b770 2024-03-12T05:58:54.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.145+0000 7f9acf7fe640 1 -- 172.21.15.76:0/1902177880 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ad0005020 con 0x7f9ad810b770 2024-03-12T05:58:54.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.145+0000 7f9ae0725640 1 -- 172.21.15.76:0/1902177880 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad819cc20 con 0x7f9ad810b770 2024-03-12T05:58:54.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.145+0000 7f9acf7fe640 1 -- 172.21.15.76:0/1902177880 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad00116b0 con 0x7f9ad810b770 2024-03-12T05:58:54.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.145+0000 7f9ae0725640 1 -- 172.21.15.76:0/1902177880 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad81a44a0 con 0x7f9ad810b770 2024-03-12T05:58:54.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.148+0000 7f9acf7fe640 1 -- 172.21.15.76:0/1902177880 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9ad0005520 con 0x7f9ad810b770 2024-03-12T05:58:54.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.148+0000 7f9ae33d2740 1 Processor -- start 2024-03-12T05:58:54.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9ae33d2740 1 -- start start 2024-03-12T05:58:54.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9ae33d2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d7211d2b0 0x563d7211d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9acf7fe640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9ac40a1ba0 0x7f9ac40a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9ae33d2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211dbd0 0x563d72126270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9ae33d2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d721267b0 0x563d72128ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9ae33d2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d7208c280 con 0x563d721267b0 2024-03-12T05:58:54.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9ae33d2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d720897b0 con 0x563d7211dbd0 2024-03-12T05:58:54.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9ae33d2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d72084f90 con 0x563d7211d2b0 2024-03-12T05:58:54.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9adecc2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d721267b0 0x563d72128ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9addcc0640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9ac40a1ba0 0x7f9ac40a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9adecc2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d721267b0 0x563d72128ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51468/0 (socket says 172.21.15.76:51468) 2024-03-12T05:58:54.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9adecc2640 1 -- 172.21.15.76:0/967748379 learned_addr learned my addr 172.21.15.76:0/967748379 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9ade4c1640 1 --2- 172.21.15.76:0/967748379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d7211d2b0 0x563d7211d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9acf7fe640 1 -- 172.21.15.76:0/1902177880 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9ad00cda20 con 0x7f9ad810b770 2024-03-12T05:58:54.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.149+0000 7f9addcc0640 1 --2- 172.21.15.76:0/967748379 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211dbd0 0x563d72126270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.150+0000 7f9adecc2640 1 -- 172.21.15.76:0/967748379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d7211d2b0 msgr2=0x563d7211d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.150+0000 7f9adecc2640 1 --2- 172.21.15.76:0/967748379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d7211d2b0 0x563d7211d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.150+0000 7f9adecc2640 1 -- 172.21.15.76:0/967748379 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211dbd0 msgr2=0x563d72126270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.150+0000 7f9adecc2640 1 --2- 172.21.15.76:0/967748379 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211dbd0 0x563d72126270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.150+0000 7f9adecc2640 1 -- 172.21.15.76:0/967748379 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d72043570 con 0x563d721267b0 2024-03-12T05:58:54.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.150+0000 7f9adecc2640 1 --2- 172.21.15.76:0/967748379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d721267b0 0x563d72128ba0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000efd0 tx=0x7f9ad0099fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2513737fabf5878 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.150+0000 7f9addcc0640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9ac40a1ba0 0x7f9ac40a4060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9ad80fe0e0 tx=0x7f9ac80047d0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.150+0000 7f9accff9640 1 -- 172.21.15.76:0/967748379 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad00aced0 con 0x563d721267b0 2024-03-12T05:58:54.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.150+0000 7f9accff9640 1 -- 172.21.15.76:0/967748379 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ad00accf0 con 0x563d721267b0 2024-03-12T05:58:54.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.150+0000 7f9accff9640 1 -- 172.21.15.76:0/967748379 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0098980 con 0x563d721267b0 2024-03-12T05:58:54.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.151+0000 7f9ae33d2740 1 -- 172.21.15.76:0/967748379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d721267b0 msgr2=0x563d72128ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.151+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/967748379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d721267b0 0x563d72128ba0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000efd0 tx=0x7f9ad0099fe0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.151+0000 7f9ae33d2740 1 -- 172.21.15.76:0/967748379 shutdown_connections 2024-03-12T05:58:54.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.151+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/967748379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d721267b0 0x563d72128ba0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.151+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/967748379 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211dbd0 0x563d72126270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.151+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/967748379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d7211d2b0 0x563d7211d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.151+0000 7f9ae33d2740 1 -- 172.21.15.76:0/967748379 >> 172.21.15.76:0/967748379 conn(0x563d72117990 msgr2=0x563d72117d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.151+0000 7f9ae33d2740 1 -- 172.21.15.76:0/967748379 shutdown_connections 2024-03-12T05:58:54.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.151+0000 7f9ae33d2740 1 -- 172.21.15.76:0/967748379 wait complete. 2024-03-12T05:58:54.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.151+0000 7f9ae33d2740 1 -- start start 2024-03-12T05:58:54.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.151+0000 7f9ae33d2740 1 Processor -- start 2024-03-12T05:58:54.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9ae33d2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211d2b0 0x563d721261b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9ae33d2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211dbd0 0x563d72123540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9ae33d2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d721267b0 0x563d72123b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9ae33d2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d71d58430 con 0x563d7211dbd0 2024-03-12T05:58:54.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9ae33d2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d71d14e00 con 0x563d7211d2b0 2024-03-12T05:58:54.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9ae33d2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d72009fd0 con 0x563d721267b0 2024-03-12T05:58:54.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9ae33d2740 10 client.0 _ll_register_callbacks cb 0x563d7205ffd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:54.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9addcc0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211dbd0 0x563d72123540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9addcc0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211dbd0 0x563d72123540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51480/0 (socket says 172.21.15.76:51480) 2024-03-12T05:58:54.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9addcc0640 1 -- 172.21.15.76:0/2447146998 learned_addr learned my addr 172.21.15.76:0/2447146998 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9adecc2640 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d721267b0 0x563d72123b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9ade4c1640 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211d2b0 0x563d721261b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9addcc0640 1 -- 172.21.15.76:0/2447146998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d721267b0 msgr2=0x563d72123b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9addcc0640 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d721267b0 0x563d72123b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9addcc0640 1 -- 172.21.15.76:0/2447146998 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211d2b0 msgr2=0x563d721261b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9addcc0640 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211d2b0 0x563d721261b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9addcc0640 1 -- 172.21.15.76:0/2447146998 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad81a44a0 con 0x563d7211dbd0 2024-03-12T05:58:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9ade4c1640 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211d2b0 0x563d721261b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:54.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.152+0000 7f9adecc2640 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d721267b0 0x563d72123b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:54.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9addcc0640 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211dbd0 0x563d72123540 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8009900 tx=0x7f9ac802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbc9203573dd5eb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ac1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:54.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ac1ffb640 1 -- 172.21.15.76:0/2447146998 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac802c5d0 con 0x563d7211dbd0 2024-03-12T05:58:54.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ac1ffb640 1 -- 172.21.15.76:0/2447146998 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9ac80035f0 con 0x563d7211dbd0 2024-03-12T05:58:54.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ae33d2740 1 -- 172.21.15.76:0/2447146998 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563d71d847e0 con 0x563d7211dbd0 2024-03-12T05:58:54.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ac1ffb640 1 -- 172.21.15.76:0/2447146998 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ac80037b0 con 0x563d7211dbd0 2024-03-12T05:58:54.171 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ae33d2740 10 client.16221 fetch_fsmap learned FSMap version 25 2024-03-12T05:58:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ae33d2740 1 -- 172.21.15.76:0/2447146998 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563d72083610 con 0x563d7211dbd0 2024-03-12T05:58:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ac1ffb640 1 -- 172.21.15.76:0/2447146998 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ac805c020 con 0x563d7211dbd0 2024-03-12T05:58:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ac1ffb640 1 -- 172.21.15.76:0/2447146998 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f9ac80586e0 con 0x563d7211dbd0 2024-03-12T05:58:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ae33d2740 10 client.16221 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:58:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ae33d2740 10 client.16221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ae33d2740 20 client.16221 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d7213ad50 0x563d7213d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ae33d2740 4 client.16221 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:58:54.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.153+0000 7f9ae33d2740 1 -- 172.21.15.76:0/2447146998 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d71dc1720 con 0x563d7213ad50 2024-03-12T05:58:54.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.154+0000 7f9ade4c1640 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d7213ad50 0x563d7213d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.154+0000 7f9ade4c1640 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d7213ad50 0x563d7213d170 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73d446831b7329ec server_cookie=6b097244c6364eb in_seq=0 out_seq=0 2024-03-12T05:58:54.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.154+0000 7f9ac1ffb640 10 client.16221 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:54.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ac1ffb640 1 -- 172.21.15.76:0/2447146998 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563d71dc1720 con 0x563d7213ad50 2024-03-12T05:58:54.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ac1ffb640 10 client.16221 handle_command_reply: tid=0 2024-03-12T05:58:54.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ae33d2740 1 client.16221 shutdown 2024-03-12T05:58:54.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ae33d2740 1 -- 172.21.15.76:0/2447146998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211dbd0 msgr2=0x563d72123540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211dbd0 0x563d72123540 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8009900 tx=0x7f9ac802bc30 comp rx=0 tx=0).stop 2024-03-12T05:58:54.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ae33d2740 1 -- 172.21.15.76:0/2447146998 shutdown_connections 2024-03-12T05:58:54.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d721267b0 0x563d72123b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d7213ad50 0x563d7213d170 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211dbd0 0x563d72123540 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/2447146998 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211d2b0 0x563d721261b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ac1ffb640 0 client.16221 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:54.174 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ae33d2740 1 -- 172.21.15.76:0/2447146998 >> 172.21.15.76:0/2447146998 conn(0x563d72117990 msgr2=0x563d72119730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ae33d2740 1 -- 172.21.15.76:0/2447146998 shutdown_connections 2024-03-12T05:58:54.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ae33d2740 1 -- 172.21.15.76:0/2447146998 wait complete. 2024-03-12T05:58:54.175 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.155+0000 7f9ae33d2740 20 client.16221 trim_cache size 0 max 16384 2024-03-12T05:58:54.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9ae33d2740 1 Processor -- start 2024-03-12T05:58:54.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9ae33d2740 1 -- start start 2024-03-12T05:58:54.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9ae33d2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d7211d2b0 0x563d72125c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9ae33d2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211dbd0 0x563d721261b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9ae33d2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d721267b0 0x563d72122dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9ae33d2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d721371c0 con 0x563d7211dbd0 2024-03-12T05:58:54.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9ae33d2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d721130f0 con 0x563d721267b0 2024-03-12T05:58:54.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9ae33d2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d7211c5a0 con 0x563d7211d2b0 2024-03-12T05:58:54.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9addcc0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211dbd0 0x563d721261b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9addcc0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211dbd0 0x563d721261b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51496/0 (socket says 172.21.15.76:51496) 2024-03-12T05:58:54.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9addcc0640 1 -- 172.21.15.76:0/3373138019 learned_addr learned my addr 172.21.15.76:0/3373138019 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9adecc2640 1 --2- 172.21.15.76:0/3373138019 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d721267b0 0x563d72122dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9ade4c1640 1 --2- 172.21.15.76:0/3373138019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d7211d2b0 0x563d72125c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9addcc0640 1 -- 172.21.15.76:0/3373138019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d7211d2b0 msgr2=0x563d72125c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9addcc0640 1 --2- 172.21.15.76:0/3373138019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d7211d2b0 0x563d72125c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9addcc0640 1 -- 172.21.15.76:0/3373138019 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d721267b0 msgr2=0x563d72122dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9addcc0640 1 --2- 172.21.15.76:0/3373138019 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d721267b0 0x563d72122dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9addcc0640 1 -- 172.21.15.76:0/3373138019 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d7201fac0 con 0x563d7211dbd0 2024-03-12T05:58:54.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.171+0000 7f9adecc2640 1 --2- 172.21.15.76:0/3373138019 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d721267b0 0x563d72122dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:58:54.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.172+0000 7f9addcc0640 1 --2- 172.21.15.76:0/3373138019 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211dbd0 0x563d721261b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8058ac0 tx=0x7f9ac8009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7297815aa7c3e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.172+0000 7f9ac3fff640 1 -- 172.21.15.76:0/3373138019 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8005d50 con 0x563d7211dbd0 2024-03-12T05:58:54.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.172+0000 7f9ac3fff640 1 -- 172.21.15.76:0/3373138019 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ac805c070 con 0x563d7211dbd0 2024-03-12T05:58:54.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.172+0000 7f9ac3fff640 1 -- 172.21.15.76:0/3373138019 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8005730 con 0x563d7211dbd0 2024-03-12T05:58:54.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.172+0000 7f9ae33d2740 1 -- 172.21.15.76:0/3373138019 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211dbd0 msgr2=0x563d721261b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.172+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/3373138019 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211dbd0 0x563d721261b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8058ac0 tx=0x7f9ac8009290 comp rx=0 tx=0).stop 2024-03-12T05:58:54.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.172+0000 7f9ae33d2740 1 -- 172.21.15.76:0/3373138019 shutdown_connections 2024-03-12T05:58:54.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.172+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/3373138019 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d721267b0 0x563d72122dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.172+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/3373138019 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211dbd0 0x563d721261b0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.172+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/3373138019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d7211d2b0 0x563d72125c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.172+0000 7f9ae33d2740 1 -- 172.21.15.76:0/3373138019 >> 172.21.15.76:0/3373138019 conn(0x563d72117990 msgr2=0x563d72121e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.172+0000 7f9ae33d2740 1 -- 172.21.15.76:0/3373138019 shutdown_connections 2024-03-12T05:58:54.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.172+0000 7f9ae33d2740 1 -- 172.21.15.76:0/3373138019 wait complete. 2024-03-12T05:58:54.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.173+0000 7f9ae33d2740 1 -- start start 2024-03-12T05:58:54.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.173+0000 7f9ae33d2740 1 Processor -- start 2024-03-12T05:58:54.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.173+0000 7f9ae33d2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211d2b0 0x563d7212ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.173+0000 7f9ae33d2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211dbd0 0x563d7212f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.173+0000 7f9ae33d2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d721267b0 0x563d72191270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.173+0000 7f9ae33d2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d721917b0 con 0x563d7211d2b0 2024-03-12T05:58:54.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.173+0000 7f9ae33d2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d721918f0 con 0x563d7211dbd0 2024-03-12T05:58:54.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.173+0000 7f9ae33d2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d72191a60 con 0x563d721267b0 2024-03-12T05:58:54.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.173+0000 7f9ae33d2740 10 client.0 _ll_register_callbacks cb 0x7f9ad000cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:54.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.173+0000 7f9ade4c1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211d2b0 0x563d7212ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.173+0000 7f9ade4c1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211d2b0 0x563d7212ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51502/0 (socket says 172.21.15.76:51502) 2024-03-12T05:58:54.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.173+0000 7f9ade4c1640 1 -- 172.21.15.76:0/3145880575 learned_addr learned my addr 172.21.15.76:0/3145880575 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.173+0000 7f9adecc2640 1 --2- 172.21.15.76:0/3145880575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d721267b0 0x563d72191270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.173+0000 7f9addcc0640 1 --2- 172.21.15.76:0/3145880575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211dbd0 0x563d7212f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.174+0000 7f9ade4c1640 1 -- 172.21.15.76:0/3145880575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d721267b0 msgr2=0x563d72191270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.174+0000 7f9ade4c1640 1 --2- 172.21.15.76:0/3145880575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d721267b0 0x563d72191270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.174+0000 7f9ade4c1640 1 -- 172.21.15.76:0/3145880575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211dbd0 msgr2=0x563d7212f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.174+0000 7f9ade4c1640 1 --2- 172.21.15.76:0/3145880575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211dbd0 0x563d7212f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.174+0000 7f9ade4c1640 1 -- 172.21.15.76:0/3145880575 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d72083610 con 0x563d7211d2b0 2024-03-12T05:58:54.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.174+0000 7f9ade4c1640 1 --2- 172.21.15.76:0/3145880575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211d2b0 0x563d7212ebc0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x563d7213da40 tx=0x7f9ad40045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20260c2743381c8c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.174+0000 7f9ac1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:54.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.174+0000 7f9ac1ffb640 1 -- 172.21.15.76:0/3145880575 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4005050 con 0x563d7211d2b0 2024-03-12T05:58:54.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.174+0000 7f9ac1ffb640 1 -- 172.21.15.76:0/3145880575 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9ad40051b0 con 0x563d7211d2b0 2024-03-12T05:58:54.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.174+0000 7f9ae33d2740 1 -- 172.21.15.76:0/3145880575 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9ad00accf0 con 0x563d7211d2b0 2024-03-12T05:58:54.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.174+0000 7f9ac1ffb640 1 -- 172.21.15.76:0/3145880575 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ad4005370 con 0x563d7211d2b0 2024-03-12T05:58:54.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.174+0000 7f9ae33d2740 10 client.16227 fetch_fsmap learned FSMap version 25 2024-03-12T05:58:54.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.174+0000 7f9ae33d2740 1 -- 172.21.15.76:0/3145880575 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563d720456a0 con 0x563d7211d2b0 2024-03-12T05:58:54.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.174+0000 7f9ac1ffb640 1 -- 172.21.15.76:0/3145880575 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ad00accf0 con 0x563d7211d2b0 2024-03-12T05:58:54.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.175+0000 7f9ac1ffb640 1 -- 172.21.15.76:0/3145880575 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f9ad401f1e0 con 0x563d7211d2b0 2024-03-12T05:58:54.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.175+0000 7f9ae33d2740 10 client.16227 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:58:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.175+0000 7f9ae33d2740 10 client.16227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.175+0000 7f9ae33d2740 20 client.16227 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.175+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/3145880575 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d72193300 0x563d72195720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.175+0000 7f9ae33d2740 4 client.16227 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:58:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.175+0000 7f9ae33d2740 1 -- 172.21.15.76:0/3145880575 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563d71e10890 con 0x563d72193300 2024-03-12T05:58:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.175+0000 7f9addcc0640 1 --2- 172.21.15.76:0/3145880575 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d72193300 0x563d72195720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.175+0000 7f9ac1ffb640 10 client.16227 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.175+0000 7f9addcc0640 1 --2- 172.21.15.76:0/3145880575 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d72193300 0x563d72195720 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5c1081ff0f81496 server_cookie=d1622ec5c4543e80 in_seq=0 out_seq=0 2024-03-12T05:58:54.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.175+0000 7f9ac1ffb640 1 -- 172.21.15.76:0/3145880575 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x563d71e10890 con 0x563d72193300 2024-03-12T05:58:54.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.175+0000 7f9ac1ffb640 10 client.16227 handle_command_reply: tid=0 2024-03-12T05:58:54.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.175+0000 7f9ae33d2740 1 client.16227 shutdown 2024-03-12T05:58:54.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae33d2740 1 -- 172.21.15.76:0/3145880575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211d2b0 msgr2=0x563d7212ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/3145880575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211d2b0 0x563d7212ebc0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x563d7213da40 tx=0x7f9ad40045c0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae33d2740 1 -- 172.21.15.76:0/3145880575 shutdown_connections 2024-03-12T05:58:54.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/3145880575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d721267b0 0x563d72191270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/3145880575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d7211dbd0 0x563d7212f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/3145880575 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d72193300 0x563d72195720 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae33d2740 1 --2- 172.21.15.76:0/3145880575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d7211d2b0 0x563d7212ebc0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ac1ffb640 0 client.16227 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:54.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae33d2740 1 -- 172.21.15.76:0/3145880575 >> 172.21.15.76:0/3145880575 conn(0x563d72117990 msgr2=0x563d72122d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae33d2740 1 -- 172.21.15.76:0/3145880575 shutdown_connections 2024-03-12T05:58:54.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae33d2740 1 -- 172.21.15.76:0/3145880575 wait complete. 2024-03-12T05:58:54.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae33d2740 20 client.16227 trim_cache size 0 max 16384 2024-03-12T05:58:54.192 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:58:54.192 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T05:58:54.192 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T05:58:54.192 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:58:54.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae0725640 1 -- 172.21.15.76:0/1902177880 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9ac40a1ba0 msgr2=0x7f9ac40a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae0725640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9ac40a1ba0 0x7f9ac40a4060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9ad80fe0e0 tx=0x7f9ac80047d0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae0725640 1 -- 172.21.15.76:0/1902177880 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ad810b770 msgr2=0x7f9ad819c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae0725640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ad810b770 0x7f9ad819c010 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000eaa0 tx=0x7f9ad000ef60 comp rx=0 tx=0).stop 2024-03-12T05:58:54.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae0725640 1 -- 172.21.15.76:0/1902177880 shutdown_connections 2024-03-12T05:58:54.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae0725640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9ac40a1ba0 0x7f9ac40a4060 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae0725640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ad810b770 0x7f9ad819c010 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae0725640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad8102cc0 0x7f9ad8197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae0725640 1 --2- 172.21.15.76:0/1902177880 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ad81023a0 0x7f9ad81977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.176+0000 7f9ae0725640 1 -- 172.21.15.76:0/1902177880 >> 172.21.15.76:0/1902177880 conn(0x7f9ad80fbac0 msgr2=0x7f9ad80fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.177+0000 7f9ae0725640 1 -- 172.21.15.76:0/1902177880 shutdown_connections 2024-03-12T05:58:54.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.177+0000 7f9ae0725640 1 -- 172.21.15.76:0/1902177880 wait complete. 2024-03-12T05:58:54.195 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:58:54.196 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:58:54.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.299+0000 7f103e0dd640 1 -- 172.21.15.76:0/2860656629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1038105370 msgr2=0x7f103810cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.299+0000 7f103e0dd640 1 --2- 172.21.15.76:0/2860656629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1038105370 0x7f103810cf40 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f1030009cd0 tx=0x7f103002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.299+0000 7f103e0dd640 1 -- 172.21.15.76:0/2860656629 shutdown_connections 2024-03-12T05:58:54.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.299+0000 7f103e0dd640 1 --2- 172.21.15.76:0/2860656629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1038105370 0x7f103810cf40 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.299+0000 7f103e0dd640 1 --2- 172.21.15.76:0/2860656629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1038100c50 0x7f1038104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.299+0000 7f103e0dd640 1 --2- 172.21.15.76:0/2860656629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10381002a0 0x7f1038100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.299+0000 7f103e0dd640 1 -- 172.21.15.76:0/2860656629 >> 172.21.15.76:0/2860656629 conn(0x7f10380772c0 msgr2=0x7f10380776d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.299+0000 7f103e0dd640 1 -- 172.21.15.76:0/2860656629 shutdown_connections 2024-03-12T05:58:54.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.299+0000 7f103e0dd640 1 -- 172.21.15.76:0/2860656629 wait complete. 2024-03-12T05:58:54.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.299+0000 7f103e0dd640 1 Processor -- start 2024-03-12T05:58:54.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.299+0000 7f103e0dd640 1 -- start start 2024-03-12T05:58:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f103e0dd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10381002a0 0x7f1038101d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f103e0dd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1038105370 0x7f1038102270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f103e0dd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10381027b0 0x7f10381a5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f103e0dd640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10381a5f20 con 0x7f1038105370 2024-03-12T05:58:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f103e0dd640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10381a6090 con 0x7f10381027b0 2024-03-12T05:58:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f103e0dd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10381a6250 con 0x7f10381002a0 2024-03-12T05:58:54.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f1036ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1038105370 0x7f1038102270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f1036ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1038105370 0x7f1038102270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51520/0 (socket says 172.21.15.76:51520) 2024-03-12T05:58:54.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f1036ffd640 1 -- 172.21.15.76:0/2767783035 learned_addr learned my addr 172.21.15.76:0/2767783035 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f10377fe640 1 --2- 172.21.15.76:0/2767783035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10381002a0 0x7f1038101d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f1037fff640 1 --2- 172.21.15.76:0/2767783035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10381027b0 0x7f10381a5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f1036ffd640 1 -- 172.21.15.76:0/2767783035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10381002a0 msgr2=0x7f1038101d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f1036ffd640 1 --2- 172.21.15.76:0/2767783035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10381002a0 0x7f1038101d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f1036ffd640 1 -- 172.21.15.76:0/2767783035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10381027b0 msgr2=0x7f10381a5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f1036ffd640 1 --2- 172.21.15.76:0/2767783035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10381027b0 0x7f10381a5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f1036ffd640 1 -- 172.21.15.76:0/2767783035 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1030009900 con 0x7f1038105370 2024-03-12T05:58:54.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f1036ffd640 1 --2- 172.21.15.76:0/2767783035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1038105370 0x7f1038102270 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f102c00d770 tx=0x7f102c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f1034ff9640 1 -- 172.21.15.76:0/2767783035 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c009c30 con 0x7f1038105370 2024-03-12T05:58:54.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f1034ff9640 1 -- 172.21.15.76:0/2767783035 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f102c00de80 con 0x7f1038105370 2024-03-12T05:58:54.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f103e0dd640 1 -- 172.21.15.76:0/2767783035 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10381a6530 con 0x7f1038105370 2024-03-12T05:58:54.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f1034ff9640 1 -- 172.21.15.76:0/2767783035 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c00f6b0 con 0x7f1038105370 2024-03-12T05:58:54.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.300+0000 7f103e0dd640 1 -- 172.21.15.76:0/2767783035 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10381a6a40 con 0x7f1038105370 2024-03-12T05:58:54.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.301+0000 7f103e0dd640 1 -- 172.21.15.76:0/2767783035 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1038101770 con 0x7f1038105370 2024-03-12T05:58:54.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.302+0000 7f1034ff9640 1 -- 172.21.15.76:0/2767783035 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f102c005030 con 0x7f1038105370 2024-03-12T05:58:54.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.303+0000 7f1034ff9640 1 --2- 172.21.15.76:0/2767783035 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f10200a1bd0 0x7f10200a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.303+0000 7f1034ff9640 1 -- 172.21.15.76:0/2767783035 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f102c0cd3f0 con 0x7f1038105370 2024-03-12T05:58:54.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.303+0000 7f10377fe640 1 --2- 172.21.15.76:0/2767783035 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f10200a1bd0 0x7f10200a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.303+0000 7f10377fe640 1 --2- 172.21.15.76:0/2767783035 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f10200a1bd0 0x7f10200a4090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1028005d00 tx=0x7f102800b410 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.307+0000 7f1034ff9640 1 -- 172.21.15.76:0/2767783035 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f102c0963a0 con 0x7f1038105370 2024-03-12T05:58:54.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:54 smithi082 ceph-mon[36382]: pgmap v96: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 14 op/s 2024-03-12T05:58:54.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:54 smithi088 ceph-mon[36585]: pgmap v96: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 14 op/s 2024-03-12T05:58:54.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:54 smithi076 ceph-mon[32095]: pgmap v96: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 14 op/s 2024-03-12T05:58:54.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.535+0000 7f103e0dd640 1 -- 172.21.15.76:0/2767783035 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1038100c50 con 0x7f1038105370 2024-03-12T05:58:54.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.536+0000 7f1034ff9640 1 -- 172.21.15.76:0/2767783035 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11332 (secure 0 0 0) 0x7f102c021020 con 0x7f1038105370 2024-03-12T05:58:54.541 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:58:54.542 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:58:43.838072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:58:54.542 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 25 2024-03-12T05:58:54.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.538+0000 7f103e0dd640 1 -- 172.21.15.76:0/2767783035 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f10200a1bd0 msgr2=0x7f10200a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.538+0000 7f103e0dd640 1 --2- 172.21.15.76:0/2767783035 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f10200a1bd0 0x7f10200a4090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1028005d00 tx=0x7f102800b410 comp rx=0 tx=0).stop 2024-03-12T05:58:54.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.538+0000 7f103e0dd640 1 -- 172.21.15.76:0/2767783035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1038105370 msgr2=0x7f1038102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.538+0000 7f103e0dd640 1 --2- 172.21.15.76:0/2767783035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1038105370 0x7f1038102270 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f102c00d770 tx=0x7f102c00dc30 comp rx=0 tx=0).stop 2024-03-12T05:58:54.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.538+0000 7f103e0dd640 1 -- 172.21.15.76:0/2767783035 shutdown_connections 2024-03-12T05:58:54.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.538+0000 7f103e0dd640 1 --2- 172.21.15.76:0/2767783035 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f10200a1bd0 0x7f10200a4090 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.538+0000 7f103e0dd640 1 --2- 172.21.15.76:0/2767783035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f10381027b0 0x7f10381a5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.538+0000 7f103e0dd640 1 --2- 172.21.15.76:0/2767783035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1038105370 0x7f1038102270 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.538+0000 7f103e0dd640 1 --2- 172.21.15.76:0/2767783035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10381002a0 0x7f1038101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.538+0000 7f103e0dd640 1 -- 172.21.15.76:0/2767783035 >> 172.21.15.76:0/2767783035 conn(0x7f10380772c0 msgr2=0x7f1038103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.538+0000 7f103e0dd640 1 -- 172.21.15.76:0/2767783035 shutdown_connections 2024-03-12T05:58:54.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.538+0000 7f103e0dd640 1 -- 172.21.15.76:0/2767783035 wait complete. 2024-03-12T05:58:54.555 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:58:54.555 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:58:54.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.656+0000 7f80b584e640 1 -- 172.21.15.76:0/2148477610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b0100d80 msgr2=0x7f80b0101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.656+0000 7f80b584e640 1 --2- 172.21.15.76:0/2148477610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b0100d80 0x7f80b0101160 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f80a4009930 tx=0x7f80a402bc30 comp rx=0 tx=0).stop 2024-03-12T05:58:54.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 -- 172.21.15.76:0/2148477610 shutdown_connections 2024-03-12T05:58:54.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 --2- 172.21.15.76:0/2148477610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80b0105340 0x7f80b010cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 --2- 172.21.15.76:0/2148477610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f80b0101730 0x7f80b0104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 --2- 172.21.15.76:0/2148477610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b0100d80 0x7f80b0101160 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 -- 172.21.15.76:0/2148477610 >> 172.21.15.76:0/2148477610 conn(0x7f80b00752e0 msgr2=0x7f80b00756f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 -- 172.21.15.76:0/2148477610 shutdown_connections 2024-03-12T05:58:54.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 -- 172.21.15.76:0/2148477610 wait complete. 2024-03-12T05:58:54.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 Processor -- start 2024-03-12T05:58:54.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 -- start start 2024-03-12T05:58:54.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b0100d80 0x7f80b019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f80b0105340 0x7f80b019d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80b01975c0 0x7f80b0197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b0197fb0 con 0x7f80b0100d80 2024-03-12T05:58:54.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b0198120 con 0x7f80b0105340 2024-03-12T05:58:54.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.657+0000 7f80b584e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b01982e0 con 0x7f80b01975c0 2024-03-12T05:58:54.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80aeffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b0100d80 0x7f80b019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80aeffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b0100d80 0x7f80b019d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51536/0 (socket says 172.21.15.76:51536) 2024-03-12T05:58:54.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80aeffd640 1 -- 172.21.15.76:0/694277610 learned_addr learned my addr 172.21.15.76:0/694277610 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80ae7fc640 1 --2- 172.21.15.76:0/694277610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f80b0105340 0x7f80b019d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80af7fe640 1 --2- 172.21.15.76:0/694277610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80b01975c0 0x7f80b0197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80aeffd640 1 -- 172.21.15.76:0/694277610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80b01975c0 msgr2=0x7f80b0197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80aeffd640 1 --2- 172.21.15.76:0/694277610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80b01975c0 0x7f80b0197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80aeffd640 1 -- 172.21.15.76:0/694277610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f80b0105340 msgr2=0x7f80b019d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80aeffd640 1 --2- 172.21.15.76:0/694277610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f80b0105340 0x7f80b019d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80aeffd640 1 -- 172.21.15.76:0/694277610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a4009560 con 0x7f80b0100d80 2024-03-12T05:58:54.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80aeffd640 1 --2- 172.21.15.76:0/694277610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b0100d80 0x7f80b019d410 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f80a4000c00 tx=0x7f80a4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80b484c640 1 -- 172.21.15.76:0/694277610 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a4004440 con 0x7f80b0100d80 2024-03-12T05:58:54.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80b584e640 1 -- 172.21.15.76:0/694277610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b01a42d0 con 0x7f80b0100d80 2024-03-12T05:58:54.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80b484c640 1 -- 172.21.15.76:0/694277610 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80a40052d0 con 0x7f80b0100d80 2024-03-12T05:58:54.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80b484c640 1 -- 172.21.15.76:0/694277610 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a4003da0 con 0x7f80b0100d80 2024-03-12T05:58:54.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.658+0000 7f80b584e640 1 -- 172.21.15.76:0/694277610 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b01a47d0 con 0x7f80b0100d80 2024-03-12T05:58:54.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.661+0000 7f80b841d740 1 Processor -- start 2024-03-12T05:58:54.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.661+0000 7f80b484c640 1 -- 172.21.15.76:0/694277610 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f80a4005440 con 0x7f80b0100d80 2024-03-12T05:58:54.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.661+0000 7f80b841d740 1 -- start start 2024-03-12T05:58:54.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.661+0000 7f80b841d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034250 0x55adf6034630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.661+0000 7f80b841d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034b70 0x55adf603d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.661+0000 7f80b841d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf603fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.661+0000 7f80b841d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf5fef960 con 0x55adf603d750 2024-03-12T05:58:54.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.661+0000 7f80b841d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf5fa3310 con 0x55adf6034250 2024-03-12T05:58:54.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80b841d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf5fa0840 con 0x55adf6034b70 2024-03-12T05:58:54.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80ae7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf603fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80af7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034250 0x55adf6034630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80aeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034b70 0x55adf603d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80ae7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf603fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51544/0 (socket says 172.21.15.76:51544) 2024-03-12T05:58:54.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80ae7fc640 1 -- 172.21.15.76:0/2794874632 learned_addr learned my addr 172.21.15.76:0/2794874632 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80b484c640 1 --2- 172.21.15.76:0/694277610 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f80980a1ba0 0x7f80980a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80ae7fc640 1 -- 172.21.15.76:0/2794874632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034b70 msgr2=0x55adf603d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80ae7fc640 1 --2- 172.21.15.76:0/2794874632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034b70 0x55adf603d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80ae7fc640 1 -- 172.21.15.76:0/2794874632 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034250 msgr2=0x55adf6034630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80ae7fc640 1 --2- 172.21.15.76:0/2794874632 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034250 0x55adf6034630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80ae7fc640 1 -- 172.21.15.76:0/2794874632 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55adf5f5e060 con 0x55adf603d750 2024-03-12T05:58:54.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80aeffd640 1 --2- 172.21.15.76:0/2794874632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034b70 0x55adf603d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:58:54.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80af7fe640 1 --2- 172.21.15.76:0/2794874632 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034250 0x55adf6034630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:58:54.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80b484c640 1 -- 172.21.15.76:0/694277610 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f80a40efe40 con 0x7f80b0100d80 2024-03-12T05:58:54.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80af7fe640 1 --2- 172.21.15.76:0/694277610 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f80980a1ba0 0x7f80980a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f80ae7fc640 1 --2- 172.21.15.76:0/2794874632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf603fb40 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f80a000bc60 tx=0x7f80a00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c67ebd4c3b93de5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f8095ffb640 1 -- 172.21.15.76:0/2794874632 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a001c480 con 0x55adf603d750 2024-03-12T05:58:54.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.662+0000 7f8095ffb640 1 -- 172.21.15.76:0/2794874632 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80a0015070 con 0x55adf603d750 2024-03-12T05:58:54.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.663+0000 7f8095ffb640 1 -- 172.21.15.76:0/2794874632 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0011ac0 con 0x55adf603d750 2024-03-12T05:58:54.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.663+0000 7f80af7fe640 1 --2- 172.21.15.76:0/694277610 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f80980a1ba0 0x7f80980a4060 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f80a8005fd0 tx=0x7f80a8004a40 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.663+0000 7f80b841d740 1 -- 172.21.15.76:0/2794874632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 msgr2=0x55adf603fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.663+0000 7f80b841d740 1 --2- 172.21.15.76:0/2794874632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf603fb40 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f80a000bc60 tx=0x7f80a00107a0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.663+0000 7f80b841d740 1 -- 172.21.15.76:0/2794874632 shutdown_connections 2024-03-12T05:58:54.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.663+0000 7f80b841d740 1 --2- 172.21.15.76:0/2794874632 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf603fb40 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.663+0000 7f80b841d740 1 --2- 172.21.15.76:0/2794874632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034b70 0x55adf603d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.663+0000 7f80b841d740 1 --2- 172.21.15.76:0/2794874632 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034250 0x55adf6034630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.663+0000 7f80b841d740 1 -- 172.21.15.76:0/2794874632 >> 172.21.15.76:0/2794874632 conn(0x55adf602e980 msgr2=0x55adf602ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.663+0000 7f80b841d740 1 -- 172.21.15.76:0/2794874632 shutdown_connections 2024-03-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.663+0000 7f80b841d740 1 -- 172.21.15.76:0/2794874632 wait complete. 2024-03-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.664+0000 7f80b841d740 1 -- start start 2024-03-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.664+0000 7f80b841d740 1 Processor -- start 2024-03-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.664+0000 7f80b841d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034250 0x55adf603a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.664+0000 7f80b841d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034b70 0x55adf603ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.664+0000 7f80b841d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf603b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.664+0000 7f80b841d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf5f9c020 con 0x55adf603d750 2024-03-12T05:58:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.664+0000 7f80b841d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf5c6f430 con 0x55adf6034b70 2024-03-12T05:58:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.664+0000 7f80b841d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf5c2be00 con 0x55adf6034250 2024-03-12T05:58:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.664+0000 7f80b841d740 10 client.0 _ll_register_callbacks cb 0x55adf5f43cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.664+0000 7f80af7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf603b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.664+0000 7f80af7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf603b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51558/0 (socket says 172.21.15.76:51558) 2024-03-12T05:58:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.664+0000 7f80af7fe640 1 -- 172.21.15.76:0/1063920058 learned_addr learned my addr 172.21.15.76:0/1063920058 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.664+0000 7f80aeffd640 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034250 0x55adf603a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.664+0000 7f80ae7fc640 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034b70 0x55adf603ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f80af7fe640 1 -- 172.21.15.76:0/1063920058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034250 msgr2=0x55adf603a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f80af7fe640 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034250 0x55adf603a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f80af7fe640 1 -- 172.21.15.76:0/1063920058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034b70 msgr2=0x55adf603ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f80af7fe640 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034b70 0x55adf603ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f80af7fe640 1 -- 172.21.15.76:0/1063920058 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a000b8c0 con 0x55adf603d750 2024-03-12T05:58:54.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f80aeffd640 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034250 0x55adf603a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:58:54.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f80ae7fc640 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034b70 0x55adf603ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:58:54.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f80af7fe640 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf603b180 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f80a8009d60 tx=0x7f80a8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256d625acaf36759 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f808effd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f808effd640 1 -- 172.21.15.76:0/1063920058 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a8003970 con 0x55adf603d750 2024-03-12T05:58:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f808effd640 1 -- 172.21.15.76:0/1063920058 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f80a8003ad0 con 0x55adf603d750 2024-03-12T05:58:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f80b841d740 1 -- 172.21.15.76:0/1063920058 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55adf5c9b7e0 con 0x55adf603d750 2024-03-12T05:58:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f808effd640 1 -- 172.21.15.76:0/1063920058 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80a8003c90 con 0x55adf603d750 2024-03-12T05:58:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f80b841d740 10 client.16245 fetch_fsmap learned FSMap version 25 2024-03-12T05:58:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f80b841d740 1 -- 172.21.15.76:0/1063920058 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55adf5f9a6a0 con 0x55adf603d750 2024-03-12T05:58:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f808effd640 1 -- 172.21.15.76:0/1063920058 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55adf5c9b7e0 con 0x55adf603d750 2024-03-12T05:58:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.665+0000 7f808effd640 1 -- 172.21.15.76:0/1063920058 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f80a8004520 con 0x55adf603d750 2024-03-12T05:58:54.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.666+0000 7f80b841d740 10 client.16245 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:58:54.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.666+0000 7f80b841d740 10 client.16245 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:58:54.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.666+0000 7f80b841d740 20 client.16245 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:54.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.666+0000 7f80b841d740 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55adf604db90 0x55adf604ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.666+0000 7f80b841d740 4 client.16245 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:58:54.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.666+0000 7f80b841d740 1 -- 172.21.15.76:0/1063920058 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55adf5cd8720 con 0x55adf604db90 2024-03-12T05:58:54.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.666+0000 7f80aeffd640 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55adf604db90 0x55adf604ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.666+0000 7f80aeffd640 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55adf604db90 0x55adf604ffb0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2a471296970f536 server_cookie=cc963b3c83c07755 in_seq=0 out_seq=0 2024-03-12T05:58:54.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.666+0000 7f808effd640 10 client.16245 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:58:54.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f808effd640 1 -- 172.21.15.76:0/1063920058 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55adf5cd8720 con 0x55adf604db90 2024-03-12T05:58:54.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f808effd640 10 client.16245 handle_command_reply: tid=0 2024-03-12T05:58:54.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f80b841d740 1 client.16245 shutdown 2024-03-12T05:58:54.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f80b841d740 1 -- 172.21.15.76:0/1063920058 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 msgr2=0x55adf603b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f80b841d740 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf603b180 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f80a8009d60 tx=0x7f80a8004ab0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f80b841d740 1 -- 172.21.15.76:0/1063920058 shutdown_connections 2024-03-12T05:58:54.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f80b841d740 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55adf604db90 0x55adf604ffb0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f80b841d740 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf603b180 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f80b841d740 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034b70 0x55adf603ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f808effd640 0 client.16245 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:58:54.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f80b841d740 1 --2- 172.21.15.76:0/1063920058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034250 0x55adf603a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f80b841d740 1 -- 172.21.15.76:0/1063920058 >> 172.21.15.76:0/1063920058 conn(0x55adf602e980 msgr2=0x55adf60306d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f80b841d740 1 -- 172.21.15.76:0/1063920058 shutdown_connections 2024-03-12T05:58:54.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f80b841d740 1 -- 172.21.15.76:0/1063920058 wait complete. 2024-03-12T05:58:54.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.667+0000 7f80b841d740 20 client.16245 trim_cache size 0 max 16384 2024-03-12T05:58:54.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.679+0000 7f80b841d740 1 Processor -- start 2024-03-12T05:58:54.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.679+0000 7f80b841d740 1 -- start start 2024-03-12T05:58:54.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.679+0000 7f80b841d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034250 0x55adf603afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.679+0000 7f80b841d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf6034b70 0x55adf6030460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.679+0000 7f80b841d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf603d750 0x55adf6044600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.679+0000 7f80b841d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf5f21130 con 0x55adf6034b70 2024-03-12T05:58:54.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.679+0000 7f80b841d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf6033cc0 con 0x55adf6034250 2024-03-12T05:58:54.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.679+0000 7f80b841d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf60333c0 con 0x55adf603d750 2024-03-12T05:58:54.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.680+0000 7f80ae7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf6034b70 0x55adf6030460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.680+0000 7f80ae7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf6034b70 0x55adf6030460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51574/0 (socket says 172.21.15.76:51574) 2024-03-12T05:58:54.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.680+0000 7f80ae7fc640 1 -- 172.21.15.76:0/1939585539 learned_addr learned my addr 172.21.15.76:0/1939585539 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.680+0000 7f80af7fe640 1 --2- 172.21.15.76:0/1939585539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf603d750 0x55adf6044600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.680+0000 7f80aeffd640 1 --2- 172.21.15.76:0/1939585539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034250 0x55adf603afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.680+0000 7f80af7fe640 1 -- 172.21.15.76:0/1939585539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034250 msgr2=0x55adf603afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.680+0000 7f80af7fe640 1 --2- 172.21.15.76:0/1939585539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034250 0x55adf603afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.680+0000 7f80af7fe640 1 -- 172.21.15.76:0/1939585539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf6034b70 msgr2=0x55adf6030460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.680+0000 7f80af7fe640 1 --2- 172.21.15.76:0/1939585539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf6034b70 0x55adf6030460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.680+0000 7f80af7fe640 1 -- 172.21.15.76:0/1939585539 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55adf5f36c20 con 0x55adf603d750 2024-03-12T05:58:54.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.680+0000 7f80af7fe640 1 --2- 172.21.15.76:0/1939585539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf603d750 0x55adf6044600 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f80a8013f60 tx=0x7f80a8005cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d3ec4f9f2f23419 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.680+0000 7f80957fa640 1 -- 172.21.15.76:0/1939585539 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a801e9a0 con 0x55adf603d750 2024-03-12T05:58:54.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.680+0000 7f80957fa640 1 -- 172.21.15.76:0/1939585539 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80a8013040 con 0x55adf603d750 2024-03-12T05:58:54.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.680+0000 7f80957fa640 1 -- 172.21.15.76:0/1939585539 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a802f450 con 0x55adf603d750 2024-03-12T05:58:54.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.681+0000 7f80b841d740 1 -- 172.21.15.76:0/1939585539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf603d750 msgr2=0x55adf6044600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.681+0000 7f80b841d740 1 --2- 172.21.15.76:0/1939585539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf603d750 0x55adf6044600 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f80a8013f60 tx=0x7f80a8005cf0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.681+0000 7f80b841d740 1 -- 172.21.15.76:0/1939585539 shutdown_connections 2024-03-12T05:58:54.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.681+0000 7f80b841d740 1 --2- 172.21.15.76:0/1939585539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf603d750 0x55adf6044600 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.681+0000 7f80b841d740 1 --2- 172.21.15.76:0/1939585539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf6034b70 0x55adf6030460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.681+0000 7f80b841d740 1 --2- 172.21.15.76:0/1939585539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034250 0x55adf603afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.681+0000 7f80b841d740 1 -- 172.21.15.76:0/1939585539 >> 172.21.15.76:0/1939585539 conn(0x55adf602e980 msgr2=0x55adf602fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.681+0000 7f80b841d740 1 -- 172.21.15.76:0/1939585539 shutdown_connections 2024-03-12T05:58:54.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.681+0000 7f80b841d740 1 -- 172.21.15.76:0/1939585539 wait complete. 2024-03-12T05:58:54.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.681+0000 7f80b841d740 1 -- start start 2024-03-12T05:58:54.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80b841d740 1 Processor -- start 2024-03-12T05:58:54.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80b841d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034250 0x55adf6043650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80b841d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034b70 0x55adf6043b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80b841d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf60a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80b841d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf6044290 con 0x55adf603d750 2024-03-12T05:58:54.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80b841d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf60a4620 con 0x55adf6034b70 2024-03-12T05:58:54.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80b841d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adf60a4760 con 0x55adf6034250 2024-03-12T05:58:54.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80b841d740 10 client.0 _ll_register_callbacks cb 0x7f80a000b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:54.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80af7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf60a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80af7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf60a40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51590/0 (socket says 172.21.15.76:51590) 2024-03-12T05:58:54.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80af7fe640 1 -- 172.21.15.76:0/1737880437 learned_addr learned my addr 172.21.15.76:0/1737880437 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80ae7fc640 1 --2- 172.21.15.76:0/1737880437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034b70 0x55adf6043b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80aeffd640 1 --2- 172.21.15.76:0/1737880437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034250 0x55adf6043650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80af7fe640 1 -- 172.21.15.76:0/1737880437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034250 msgr2=0x55adf6043650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80af7fe640 1 --2- 172.21.15.76:0/1737880437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034250 0x55adf6043650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80af7fe640 1 -- 172.21.15.76:0/1737880437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034b70 msgr2=0x55adf6043b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80af7fe640 1 --2- 172.21.15.76:0/1737880437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034b70 0x55adf6043b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80af7fe640 1 -- 172.21.15.76:0/1737880437 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55adf5f9a6a0 con 0x55adf603d750 2024-03-12T05:58:54.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80ae7fc640 1 --2- 172.21.15.76:0/1737880437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034b70 0x55adf6043b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:58:54.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f80af7fe640 1 --2- 172.21.15.76:0/1737880437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf60a40e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f80a80048d0 tx=0x7f80a801fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9a5aeb57272551 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.682+0000 7f808effd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f808effd640 1 -- 172.21.15.76:0/1737880437 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a8012730 con 0x55adf603d750 2024-03-12T05:58:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f808effd640 1 -- 172.21.15.76:0/1737880437 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f80a8012890 con 0x55adf603d750 2024-03-12T05:58:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f80b841d740 1 -- 172.21.15.76:0/1737880437 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f80a0015070 con 0x55adf603d750 2024-03-12T05:58:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f808effd640 1 -- 172.21.15.76:0/1737880437 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80a801fe90 con 0x55adf603d750 2024-03-12T05:58:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f80b841d740 10 client.16251 fetch_fsmap learned FSMap version 25 2024-03-12T05:58:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f80b841d740 1 -- 172.21.15.76:0/1737880437 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55adf5f5c8c0 con 0x55adf603d750 2024-03-12T05:58:54.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f808effd640 1 -- 172.21.15.76:0/1737880437 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80a0015070 con 0x55adf603d750 2024-03-12T05:58:54.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f808effd640 1 -- 172.21.15.76:0/1737880437 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f80a8038cb0 con 0x55adf603d750 2024-03-12T05:58:54.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f80b841d740 10 client.16251 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:58:54.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f80b841d740 10 client.16251 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:58:54.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f80b841d740 20 client.16251 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:54.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f80b841d740 1 --2- 172.21.15.76:0/1737880437 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55adf60a5bd0 0x55adf60a7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f80b841d740 4 client.16251 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:58:54.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f80b841d740 1 -- 172.21.15.76:0/1737880437 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55adf5d27890 con 0x55adf60a5bd0 2024-03-12T05:58:54.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.683+0000 7f80aeffd640 1 --2- 172.21.15.76:0/1737880437 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55adf60a5bd0 0x55adf60a7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f80aeffd640 1 --2- 172.21.15.76:0/1737880437 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55adf60a5bd0 0x55adf60a7ff0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7f003d895590f0c3 server_cookie=a190fc0064df8aea in_seq=0 out_seq=0 2024-03-12T05:58:54.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f808effd640 10 client.16251 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:58:54.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f808effd640 1 -- 172.21.15.76:0/1737880437 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55adf5d27890 con 0x55adf60a5bd0 2024-03-12T05:58:54.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f808effd640 10 client.16251 handle_command_reply: tid=0 2024-03-12T05:58:54.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f80b841d740 1 client.16251 shutdown 2024-03-12T05:58:54.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f80b841d740 1 -- 172.21.15.76:0/1737880437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 msgr2=0x55adf60a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f80b841d740 1 --2- 172.21.15.76:0/1737880437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf60a40e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f80a80048d0 tx=0x7f80a801fa60 comp rx=0 tx=0).stop 2024-03-12T05:58:54.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f80b841d740 1 -- 172.21.15.76:0/1737880437 shutdown_connections 2024-03-12T05:58:54.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f80b841d740 1 --2- 172.21.15.76:0/1737880437 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55adf60a5bd0 0x55adf60a7ff0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f80b841d740 1 --2- 172.21.15.76:0/1737880437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55adf603d750 0x55adf60a40e0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f808effd640 0 client.16251 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:58:54.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f80b841d740 1 --2- 172.21.15.76:0/1737880437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55adf6034b70 0x55adf6043b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f80b841d740 1 --2- 172.21.15.76:0/1737880437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55adf6034250 0x55adf6043650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f80b841d740 1 -- 172.21.15.76:0/1737880437 >> 172.21.15.76:0/1737880437 conn(0x55adf602e980 msgr2=0x55adf602db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f80b841d740 1 -- 172.21.15.76:0/1737880437 shutdown_connections 2024-03-12T05:58:54.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f80b841d740 1 -- 172.21.15.76:0/1737880437 wait complete. 2024-03-12T05:58:54.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.684+0000 7f80b841d740 20 client.16251 trim_cache size 0 max 16384 2024-03-12T05:58:54.702 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:58:54.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.685+0000 7f80b584e640 1 -- 172.21.15.76:0/694277610 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f80980a1ba0 msgr2=0x7f80980a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.685+0000 7f80b584e640 1 --2- 172.21.15.76:0/694277610 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f80980a1ba0 0x7f80980a4060 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f80a8005fd0 tx=0x7f80a8004a40 comp rx=0 tx=0).stop 2024-03-12T05:58:54.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.685+0000 7f80b584e640 1 -- 172.21.15.76:0/694277610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b0100d80 msgr2=0x7f80b019d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.685+0000 7f80b584e640 1 --2- 172.21.15.76:0/694277610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b0100d80 0x7f80b019d410 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f80a4000c00 tx=0x7f80a4004040 comp rx=0 tx=0).stop 2024-03-12T05:58:54.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.685+0000 7f80b584e640 1 -- 172.21.15.76:0/694277610 shutdown_connections 2024-03-12T05:58:54.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.685+0000 7f80b584e640 1 --2- 172.21.15.76:0/694277610 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f80980a1ba0 0x7f80980a4060 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.685+0000 7f80b584e640 1 --2- 172.21.15.76:0/694277610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80b01975c0 0x7f80b0197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.685+0000 7f80b584e640 1 --2- 172.21.15.76:0/694277610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f80b0105340 0x7f80b019d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.685+0000 7f80b584e640 1 --2- 172.21.15.76:0/694277610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b0100d80 0x7f80b019d410 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.685+0000 7f80b584e640 1 -- 172.21.15.76:0/694277610 >> 172.21.15.76:0/694277610 conn(0x7f80b00752e0 msgr2=0x7f80b0104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.685+0000 7f80b584e640 1 -- 172.21.15.76:0/694277610 shutdown_connections 2024-03-12T05:58:54.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.685+0000 7f80b584e640 1 -- 172.21.15.76:0/694277610 wait complete. 2024-03-12T05:58:54.705 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:58:54.705 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:58:54.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.807+0000 7fc2e089d640 1 -- 172.21.15.76:0/770069411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2dc105360 msgr2=0x7fc2dc10cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.807+0000 7fc2e089d640 1 --2- 172.21.15.76:0/770069411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2dc105360 0x7fc2dc10cf30 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009cd0 tx=0x7fc2d402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.808+0000 7fc2e089d640 1 -- 172.21.15.76:0/770069411 shutdown_connections 2024-03-12T05:58:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.808+0000 7fc2e089d640 1 --2- 172.21.15.76:0/770069411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2dc105360 0x7fc2dc10cf30 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.808+0000 7fc2e089d640 1 --2- 172.21.15.76:0/770069411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2dc100c40 0x7fc2dc104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.808+0000 7fc2e089d640 1 --2- 172.21.15.76:0/770069411 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2dc100290 0x7fc2dc100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.808+0000 7fc2e089d640 1 -- 172.21.15.76:0/770069411 >> 172.21.15.76:0/770069411 conn(0x7fc2dc077230 msgr2=0x7fc2dc077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.808+0000 7fc2e089d640 1 -- 172.21.15.76:0/770069411 shutdown_connections 2024-03-12T05:58:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.808+0000 7fc2e089d640 1 -- 172.21.15.76:0/770069411 wait complete. 2024-03-12T05:58:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.808+0000 7fc2e089d640 1 Processor -- start 2024-03-12T05:58:54.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.808+0000 7fc2e089d640 1 -- start start 2024-03-12T05:58:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.808+0000 7fc2e089d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2dc100290 0x7fc2dc19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2e089d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2dc105360 0x7fc2dc19db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2e089d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2dc197680 0x7fc2dc197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2e089d640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc198150 con 0x7fc2dc105360 2024-03-12T05:58:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2e089d640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc1982c0 con 0x7fc2dc197680 2024-03-12T05:58:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2e089d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc198480 con 0x7fc2dc100290 2024-03-12T05:58:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2d9d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2dc105360 0x7fc2dc19db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2d9d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2dc105360 0x7fc2dc19db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51610/0 (socket says 172.21.15.76:51610) 2024-03-12T05:58:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2d9d9b640 1 -- 172.21.15.76:0/4121410205 learned_addr learned my addr 172.21.15.76:0/4121410205 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2da59c640 1 --2- 172.21.15.76:0/4121410205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2dc100290 0x7fc2dc19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2dad9d640 1 --2- 172.21.15.76:0/4121410205 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2dc197680 0x7fc2dc197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2d9d9b640 1 -- 172.21.15.76:0/4121410205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2dc100290 msgr2=0x7fc2dc19d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2d9d9b640 1 --2- 172.21.15.76:0/4121410205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2dc100290 0x7fc2dc19d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2d9d9b640 1 -- 172.21.15.76:0/4121410205 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2dc197680 msgr2=0x7fc2dc197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2d9d9b640 1 --2- 172.21.15.76:0/4121410205 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2dc197680 0x7fc2dc197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2d9d9b640 1 -- 172.21.15.76:0/4121410205 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d4009900 con 0x7fc2dc105360 2024-03-12T05:58:54.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2d9d9b640 1 --2- 172.21.15.76:0/4121410205 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2dc105360 0x7fc2dc19db00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000e9f0 tx=0x7fc2d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2cb7fe640 1 -- 172.21.15.76:0/4121410205 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00096c0 con 0x7fc2dc105360 2024-03-12T05:58:54.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2cb7fe640 1 -- 172.21.15.76:0/4121410205 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2d0005500 con 0x7fc2dc105360 2024-03-12T05:58:54.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2e089d640 1 -- 172.21.15.76:0/4121410205 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2dc1a4420 con 0x7fc2dc105360 2024-03-12T05:58:54.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2cb7fe640 1 -- 172.21.15.76:0/4121410205 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0010430 con 0x7fc2dc105360 2024-03-12T05:58:54.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.809+0000 7fc2e089d640 1 -- 172.21.15.76:0/4121410205 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2dc1a48b0 con 0x7fc2dc105360 2024-03-12T05:58:54.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.812+0000 7fc2e347b740 1 Processor -- start 2024-03-12T05:58:54.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.812+0000 7fc2e347b740 1 -- start start 2024-03-12T05:58:54.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.812+0000 7fc2e347b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fb250 0x563cd63fb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.812+0000 7fc2e347b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd63fbb70 0x563cd6404210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.812+0000 7fc2e347b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd6404750 0x563cd6406b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.826 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.812+0000 7fc2e347b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd63b6960 con 0x563cd63fb250 2024-03-12T05:58:54.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.812+0000 7fc2e347b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd636a310 con 0x563cd6404750 2024-03-12T05:58:54.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.812+0000 7fc2e347b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd6367840 con 0x563cd63fbb70 2024-03-12T05:58:54.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.812+0000 7fc2dad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fb250 0x563cd63fb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.812+0000 7fc2dad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fb250 0x563cd63fb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51626/0 (socket says 172.21.15.76:51626) 2024-03-12T05:58:54.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.812+0000 7fc2dad9d640 1 -- 172.21.15.76:0/2359008686 learned_addr learned my addr 172.21.15.76:0/2359008686 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.812+0000 7fc2cb7fe640 1 -- 172.21.15.76:0/4121410205 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc2d0005020 con 0x7fc2dc105360 2024-03-12T05:58:54.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.812+0000 7fc2da59c640 1 --2- 172.21.15.76:0/2359008686 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd63fbb70 0x563cd6404210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.813+0000 7fc2dad9d640 1 -- 172.21.15.76:0/2359008686 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd63fbb70 msgr2=0x563cd6404210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.813+0000 7fc2dad9d640 1 --2- 172.21.15.76:0/2359008686 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd63fbb70 0x563cd6404210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.813+0000 7fc2dad9d640 1 -- 172.21.15.76:0/2359008686 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd6404750 msgr2=0x563cd6406b40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-12T05:58:54.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.813+0000 7fc2dad9d640 1 --2- 172.21.15.76:0/2359008686 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd6404750 0x563cd6406b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.813+0000 7fc2dad9d640 1 -- 172.21.15.76:0/2359008686 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cd6325060 con 0x563cd63fb250 2024-03-12T05:58:54.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.813+0000 7fc2da59c640 1 --2- 172.21.15.76:0/2359008686 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd63fbb70 0x563cd6404210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:54.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.813+0000 7fc2dad9d640 1 --2- 172.21.15.76:0/2359008686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fb250 0x563cd63fb630 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009cd0 tx=0x7fc2d4005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f976effbf938878 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.813+0000 7fc2c8ff9640 1 -- 172.21.15.76:0/2359008686 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4034440 con 0x563cd63fb250 2024-03-12T05:58:54.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.813+0000 7fc2c8ff9640 1 -- 172.21.15.76:0/2359008686 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2d40345a0 con 0x563cd63fb250 2024-03-12T05:58:54.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.813+0000 7fc2c8ff9640 1 -- 172.21.15.76:0/2359008686 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4003bb0 con 0x563cd63fb250 2024-03-12T05:58:54.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.813+0000 7fc2cb7fe640 1 --2- 172.21.15.76:0/4121410205 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2c40a1be0 0x7fc2c40a40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.813+0000 7fc2da59c640 1 --2- 172.21.15.76:0/4121410205 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2c40a1be0 0x7fc2c40a40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.813+0000 7fc2cb7fe640 1 -- 172.21.15.76:0/4121410205 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc2d0014070 con 0x7fc2dc105360 2024-03-12T05:58:54.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.814+0000 7fc2e347b740 1 -- 172.21.15.76:0/2359008686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fb250 msgr2=0x563cd63fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.814+0000 7fc2da59c640 1 --2- 172.21.15.76:0/4121410205 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2c40a1be0 0x7fc2c40a40a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x563cd63fad20 tx=0x7fc2cc00b390 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.814+0000 7fc2e347b740 1 --2- 172.21.15.76:0/2359008686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fb250 0x563cd63fb630 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009cd0 tx=0x7fc2d4005fa0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.814+0000 7fc2e347b740 1 -- 172.21.15.76:0/2359008686 shutdown_connections 2024-03-12T05:58:54.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.814+0000 7fc2e347b740 1 --2- 172.21.15.76:0/2359008686 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd6404750 0x563cd6406b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.814+0000 7fc2e347b740 1 --2- 172.21.15.76:0/2359008686 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd63fbb70 0x563cd6404210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.814+0000 7fc2e347b740 1 --2- 172.21.15.76:0/2359008686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fb250 0x563cd63fb630 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.814+0000 7fc2e347b740 1 -- 172.21.15.76:0/2359008686 >> 172.21.15.76:0/2359008686 conn(0x563cd63f5980 msgr2=0x563cd63f5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.814+0000 7fc2e347b740 1 -- 172.21.15.76:0/2359008686 shutdown_connections 2024-03-12T05:58:54.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.814+0000 7fc2e347b740 1 -- 172.21.15.76:0/2359008686 wait complete. 2024-03-12T05:58:54.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.814+0000 7fc2e347b740 1 -- start start 2024-03-12T05:58:54.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.814+0000 7fc2e347b740 1 Processor -- start 2024-03-12T05:58:54.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2e347b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd63fb250 0x563cd6401820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2e347b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fbb70 0x563cd6401d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2e347b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd6404750 0x563cd64022a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2e347b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd6363020 con 0x563cd6404750 2024-03-12T05:58:54.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2e347b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd6036430 con 0x563cd63fbb70 2024-03-12T05:58:54.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2e347b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd5ff2e00 con 0x563cd63fb250 2024-03-12T05:58:54.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2e347b740 10 client.0 _ll_register_callbacks cb 0x563cd630acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:54.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2dad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd6404750 0x563cd64022a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2dad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd6404750 0x563cd64022a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51638/0 (socket says 172.21.15.76:51638) 2024-03-12T05:58:54.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2dad9d640 1 -- 172.21.15.76:0/383643893 learned_addr learned my addr 172.21.15.76:0/383643893 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2d9d9b640 1 --2- 172.21.15.76:0/383643893 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fbb70 0x563cd6401d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2da59c640 1 --2- 172.21.15.76:0/383643893 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd63fb250 0x563cd6401820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2dad9d640 1 -- 172.21.15.76:0/383643893 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd63fb250 msgr2=0x563cd6401820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2dad9d640 1 --2- 172.21.15.76:0/383643893 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd63fb250 0x563cd6401820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2dad9d640 1 -- 172.21.15.76:0/383643893 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fbb70 msgr2=0x563cd6401d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2dad9d640 1 --2- 172.21.15.76:0/383643893 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fbb70 0x563cd6401d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2dad9d640 1 -- 172.21.15.76:0/383643893 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2dc068450 con 0x563cd6404750 2024-03-12T05:58:54.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2d9d9b640 1 --2- 172.21.15.76:0/383643893 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fbb70 0x563cd6401d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:58:54.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2dad9d640 1 --2- 172.21.15.76:0/383643893 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd6404750 0x563cd64022a0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009620 tx=0x7fc2d4032a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e88e344dff63b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:54.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2c1ffb640 1 -- 172.21.15.76:0/383643893 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4034ea0 con 0x563cd6404750 2024-03-12T05:58:54.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2c1ffb640 1 -- 172.21.15.76:0/383643893 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc2d4003970 con 0x563cd6404750 2024-03-12T05:58:54.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2e347b740 1 -- 172.21.15.76:0/383643893 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563cd60627e0 con 0x563cd6404750 2024-03-12T05:58:54.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2c1ffb640 1 -- 172.21.15.76:0/383643893 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2d4003b70 con 0x563cd6404750 2024-03-12T05:58:54.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2e347b740 10 client.16263 fetch_fsmap learned FSMap version 25 2024-03-12T05:58:54.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.815+0000 7fc2e347b740 1 -- 172.21.15.76:0/383643893 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563cd63616a0 con 0x563cd6404750 2024-03-12T05:58:54.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.816+0000 7fc2c1ffb640 1 -- 172.21.15.76:0/383643893 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563cd60627e0 con 0x563cd6404750 2024-03-12T05:58:54.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.816+0000 7fc2c1ffb640 1 -- 172.21.15.76:0/383643893 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fc2d404a4e0 con 0x563cd6404750 2024-03-12T05:58:54.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.816+0000 7fc2e347b740 10 client.16263 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:58:54.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.816+0000 7fc2e347b740 10 client.16263 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:58:54.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.816+0000 7fc2e347b740 20 client.16263 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:54.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.816+0000 7fc2e347b740 1 --2- 172.21.15.76:0/383643893 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x563cd6414890 0x563cd6416c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.816+0000 7fc2e347b740 4 client.16263 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:58:54.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.816+0000 7fc2e347b740 1 -- 172.21.15.76:0/383643893 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563cd609f720 con 0x563cd6414890 2024-03-12T05:58:54.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.816+0000 7fc2da59c640 1 --2- 172.21.15.76:0/383643893 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x563cd6414890 0x563cd6416c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.816+0000 7fc2da59c640 1 --2- 172.21.15.76:0/383643893 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x563cd6414890 0x563cd6416c80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=addfc5b844df0eba server_cookie=7db9d86d717a615c in_seq=0 out_seq=0 2024-03-12T05:58:54.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.816+0000 7fc2c1ffb640 10 client.16263 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:58:54.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.817+0000 7fc2c1ffb640 1 -- 172.21.15.76:0/383643893 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563cd609f720 con 0x563cd6414890 2024-03-12T05:58:54.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.817+0000 7fc2c1ffb640 10 client.16263 handle_command_reply: tid=0 2024-03-12T05:58:54.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.817+0000 7fc2e347b740 1 client.16263 shutdown 2024-03-12T05:58:54.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.817+0000 7fc2e347b740 1 -- 172.21.15.76:0/383643893 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd6404750 msgr2=0x563cd64022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.817+0000 7fc2e347b740 1 --2- 172.21.15.76:0/383643893 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd6404750 0x563cd64022a0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4009620 tx=0x7fc2d4032a20 comp rx=0 tx=0).stop 2024-03-12T05:58:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.817+0000 7fc2e347b740 1 -- 172.21.15.76:0/383643893 shutdown_connections 2024-03-12T05:58:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.817+0000 7fc2e347b740 1 --2- 172.21.15.76:0/383643893 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x563cd6414890 0x563cd6416c80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.817+0000 7fc2e347b740 1 --2- 172.21.15.76:0/383643893 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd6404750 0x563cd64022a0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.817+0000 7fc2e347b740 1 --2- 172.21.15.76:0/383643893 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fbb70 0x563cd6401d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.817+0000 7fc2e347b740 1 --2- 172.21.15.76:0/383643893 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd63fb250 0x563cd6401820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.817+0000 7fc2e347b740 1 -- 172.21.15.76:0/383643893 >> 172.21.15.76:0/383643893 conn(0x563cd63f5980 msgr2=0x563cd63f76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.817+0000 7fc2c1ffb640 0 client.16263 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:58:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.817+0000 7fc2e347b740 1 -- 172.21.15.76:0/383643893 shutdown_connections 2024-03-12T05:58:54.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.817+0000 7fc2e347b740 1 -- 172.21.15.76:0/383643893 wait complete. 2024-03-12T05:58:54.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.818+0000 7fc2e347b740 20 client.16263 trim_cache size 0 max 16384 2024-03-12T05:58:54.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.829+0000 7fc2e347b740 1 Processor -- start 2024-03-12T05:58:54.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.829+0000 7fc2e347b740 1 -- start start 2024-03-12T05:58:54.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.829+0000 7fc2e347b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fb250 0x563cd6403f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.829+0000 7fc2e347b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fbb70 0x563cd64020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.829+0000 7fc2e347b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd6404750 0x563cd63fcb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.829+0000 7fc2e347b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd62e8130 con 0x563cd63fbb70 2024-03-12T05:58:54.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.829+0000 7fc2e347b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd6414690 con 0x563cd63fb250 2024-03-12T05:58:54.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.829+0000 7fc2e347b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd63f1080 con 0x563cd6404750 2024-03-12T05:58:54.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.829+0000 7fc2d9d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fbb70 0x563cd64020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.829+0000 7fc2d9d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fbb70 0x563cd64020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51648/0 (socket says 172.21.15.76:51648) 2024-03-12T05:58:54.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.829+0000 7fc2d9d9b640 1 -- 172.21.15.76:0/1213153300 learned_addr learned my addr 172.21.15.76:0/1213153300 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.829+0000 7fc2da59c640 1 --2- 172.21.15.76:0/1213153300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fb250 0x563cd6403f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.829+0000 7fc2dad9d640 1 --2- 172.21.15.76:0/1213153300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd6404750 0x563cd63fcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.830+0000 7fc2d9d9b640 1 -- 172.21.15.76:0/1213153300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd6404750 msgr2=0x563cd63fcb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.830+0000 7fc2d9d9b640 1 --2- 172.21.15.76:0/1213153300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd6404750 0x563cd63fcb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.830+0000 7fc2d9d9b640 1 -- 172.21.15.76:0/1213153300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fb250 msgr2=0x563cd6403f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.830+0000 7fc2d9d9b640 1 --2- 172.21.15.76:0/1213153300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fb250 0x563cd6403f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.830+0000 7fc2d9d9b640 1 -- 172.21.15.76:0/1213153300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cd62fdc20 con 0x563cd63fbb70 2024-03-12T05:58:54.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.830+0000 7fc2dad9d640 1 --2- 172.21.15.76:0/1213153300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd6404750 0x563cd63fcb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:54.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.830+0000 7fc2d9d9b640 1 --2- 172.21.15.76:0/1213153300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fbb70 0x563cd64020c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00049d0 tx=0x7fc2d0099ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae8a76e5cee0d0b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.830+0000 7fc2c3fff640 1 -- 172.21.15.76:0/1213153300 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d009a460 con 0x563cd63fbb70 2024-03-12T05:58:54.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.830+0000 7fc2c3fff640 1 -- 172.21.15.76:0/1213153300 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2d009ad90 con 0x563cd63fbb70 2024-03-12T05:58:54.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.830+0000 7fc2c3fff640 1 -- 172.21.15.76:0/1213153300 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0098990 con 0x563cd63fbb70 2024-03-12T05:58:54.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 -- 172.21.15.76:0/1213153300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fbb70 msgr2=0x563cd64020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 --2- 172.21.15.76:0/1213153300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fbb70 0x563cd64020c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00049d0 tx=0x7fc2d0099ff0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 -- 172.21.15.76:0/1213153300 shutdown_connections 2024-03-12T05:58:54.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 --2- 172.21.15.76:0/1213153300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd6404750 0x563cd63fcb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 --2- 172.21.15.76:0/1213153300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fbb70 0x563cd64020c0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 --2- 172.21.15.76:0/1213153300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fb250 0x563cd6403f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 -- 172.21.15.76:0/1213153300 >> 172.21.15.76:0/1213153300 conn(0x563cd63f5980 msgr2=0x563cd641b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 -- 172.21.15.76:0/1213153300 shutdown_connections 2024-03-12T05:58:54.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 -- 172.21.15.76:0/1213153300 wait complete. 2024-03-12T05:58:54.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 -- start start 2024-03-12T05:58:54.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 Processor -- start 2024-03-12T05:58:54.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fb250 0x563cd640baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fbb70 0x563cd6409e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd6404750 0x563cd640a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd640ac80 con 0x563cd63fb250 2024-03-12T05:58:54.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd646ad40 con 0x563cd63fbb70 2024-03-12T05:58:54.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd646ae80 con 0x563cd6404750 2024-03-12T05:58:54.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2e347b740 10 client.0 _ll_register_callbacks cb 0x7fc2d4037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:54.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2da59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fb250 0x563cd640baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2da59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fb250 0x563cd640baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51650/0 (socket says 172.21.15.76:51650) 2024-03-12T05:58:54.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2da59c640 1 -- 172.21.15.76:0/3857433276 learned_addr learned my addr 172.21.15.76:0/3857433276 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2d9d9b640 1 --2- 172.21.15.76:0/3857433276 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fbb70 0x563cd6409e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.831+0000 7fc2dad9d640 1 --2- 172.21.15.76:0/3857433276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd6404750 0x563cd640a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.832+0000 7fc2da59c640 1 -- 172.21.15.76:0/3857433276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd6404750 msgr2=0x563cd640a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.832+0000 7fc2da59c640 1 --2- 172.21.15.76:0/3857433276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd6404750 0x563cd640a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.832+0000 7fc2da59c640 1 -- 172.21.15.76:0/3857433276 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fbb70 msgr2=0x563cd6409e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.832+0000 7fc2da59c640 1 --2- 172.21.15.76:0/3857433276 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fbb70 0x563cd6409e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.832+0000 7fc2da59c640 1 -- 172.21.15.76:0/3857433276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2dc1a48b0 con 0x563cd63fb250 2024-03-12T05:58:54.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.832+0000 7fc2da59c640 1 --2- 172.21.15.76:0/3857433276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fb250 0x563cd640baa0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc003e50 tx=0x7fc2cc014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a66ef40ee2f8af99 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.832+0000 7fc2c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:54.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.832+0000 7fc2c1ffb640 1 -- 172.21.15.76:0/3857433276 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc022040 con 0x563cd63fb250 2024-03-12T05:58:54.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.832+0000 7fc2e347b740 1 -- 172.21.15.76:0/3857433276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2d40345a0 con 0x563cd63fb250 2024-03-12T05:58:54.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.832+0000 7fc2c1ffb640 1 -- 172.21.15.76:0/3857433276 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc2cc027370 con 0x563cd63fb250 2024-03-12T05:58:54.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.832+0000 7fc2c1ffb640 1 -- 172.21.15.76:0/3857433276 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2cc012ea0 con 0x563cd63fb250 2024-03-12T05:58:54.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.832+0000 7fc2e347b740 10 client.16269 fetch_fsmap learned FSMap version 25 2024-03-12T05:58:54.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.832+0000 7fc2e347b740 1 -- 172.21.15.76:0/3857433276 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563cd63238c0 con 0x563cd63fb250 2024-03-12T05:58:54.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.832+0000 7fc2c1ffb640 1 -- 172.21.15.76:0/3857433276 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2d40345a0 con 0x563cd63fb250 2024-03-12T05:58:54.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2c1ffb640 1 -- 172.21.15.76:0/3857433276 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fc2cc02d070 con 0x563cd63fb250 2024-03-12T05:58:54.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2e347b740 10 client.16269 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:58:54.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2e347b740 10 client.16269 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:58:54.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2e347b740 20 client.16269 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:54.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2e347b740 1 --2- 172.21.15.76:0/3857433276 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x563cd646c730 0x563cd646eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2e347b740 4 client.16269 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:58:54.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2e347b740 1 -- 172.21.15.76:0/3857433276 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563cd60ee890 con 0x563cd646c730 2024-03-12T05:58:54.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2d9d9b640 1 --2- 172.21.15.76:0/3857433276 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x563cd646c730 0x563cd646eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2d9d9b640 1 --2- 172.21.15.76:0/3857433276 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x563cd646c730 0x563cd646eb50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8d23cda11e952fdb server_cookie=caf6ea2f853960e9 in_seq=0 out_seq=0 2024-03-12T05:58:54.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2c1ffb640 10 client.16269 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:58:54.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2c1ffb640 1 -- 172.21.15.76:0/3857433276 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563cd60ee890 con 0x563cd646c730 2024-03-12T05:58:54.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2c1ffb640 10 client.16269 handle_command_reply: tid=0 2024-03-12T05:58:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2e347b740 1 client.16269 shutdown 2024-03-12T05:58:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2e347b740 1 -- 172.21.15.76:0/3857433276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fb250 msgr2=0x563cd640baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2e347b740 1 --2- 172.21.15.76:0/3857433276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fb250 0x563cd640baa0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc003e50 tx=0x7fc2cc014040 comp rx=0 tx=0).stop 2024-03-12T05:58:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2e347b740 1 -- 172.21.15.76:0/3857433276 shutdown_connections 2024-03-12T05:58:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2e347b740 1 --2- 172.21.15.76:0/3857433276 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x563cd646c730 0x563cd646eb50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.833+0000 7fc2e347b740 1 --2- 172.21.15.76:0/3857433276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563cd6404750 0x563cd640a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2c1ffb640 0 client.16269 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:58:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e347b740 1 --2- 172.21.15.76:0/3857433276 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563cd63fbb70 0x563cd6409e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e347b740 1 --2- 172.21.15.76:0/3857433276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563cd63fb250 0x563cd640baa0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e347b740 1 -- 172.21.15.76:0/3857433276 >> 172.21.15.76:0/3857433276 conn(0x563cd63f5980 msgr2=0x563cd63fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e347b740 1 -- 172.21.15.76:0/3857433276 shutdown_connections 2024-03-12T05:58:54.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e347b740 1 -- 172.21.15.76:0/3857433276 wait complete. 2024-03-12T05:58:54.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e347b740 20 client.16269 trim_cache size 0 max 16384 2024-03-12T05:58:54.854 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:58:54.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e089d640 1 -- 172.21.15.76:0/4121410205 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2c40a1be0 msgr2=0x7fc2c40a40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e089d640 1 --2- 172.21.15.76:0/4121410205 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2c40a1be0 0x7fc2c40a40a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x563cd63fad20 tx=0x7fc2cc00b390 comp rx=0 tx=0).stop 2024-03-12T05:58:54.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e089d640 1 -- 172.21.15.76:0/4121410205 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2dc105360 msgr2=0x7fc2dc19db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e089d640 1 --2- 172.21.15.76:0/4121410205 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2dc105360 0x7fc2dc19db00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000e9f0 tx=0x7fc2d000eeb0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e089d640 1 -- 172.21.15.76:0/4121410205 shutdown_connections 2024-03-12T05:58:54.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e089d640 1 --2- 172.21.15.76:0/4121410205 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2c40a1be0 0x7fc2c40a40a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e089d640 1 --2- 172.21.15.76:0/4121410205 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2dc197680 0x7fc2dc197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e089d640 1 --2- 172.21.15.76:0/4121410205 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2dc105360 0x7fc2dc19db00 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e089d640 1 --2- 172.21.15.76:0/4121410205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2dc100290 0x7fc2dc19d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e089d640 1 -- 172.21.15.76:0/4121410205 >> 172.21.15.76:0/4121410205 conn(0x7fc2dc077230 msgr2=0x7fc2dc0fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e089d640 1 -- 172.21.15.76:0/4121410205 shutdown_connections 2024-03-12T05:58:54.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.834+0000 7fc2e089d640 1 -- 172.21.15.76:0/4121410205 wait complete. 2024-03-12T05:58:54.858 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:58:54.858 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:58:54.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.959+0000 7f8f21adf640 1 -- 172.21.15.76:0/184810370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f1c100290 msgr2=0x7f8f1c100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 --2- 172.21.15.76:0/184810370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f1c100290 0x7f8f1c100670 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c009930 tx=0x7f8f0c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:58:54.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 -- 172.21.15.76:0/184810370 shutdown_connections 2024-03-12T05:58:54.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 --2- 172.21.15.76:0/184810370 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c105360 0x7f8f1c10cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 --2- 172.21.15.76:0/184810370 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f1c100c40 0x7f8f1c104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 --2- 172.21.15.76:0/184810370 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f1c100290 0x7f8f1c100670 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 -- 172.21.15.76:0/184810370 >> 172.21.15.76:0/184810370 conn(0x7f8f1c077230 msgr2=0x7f8f1c077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 -- 172.21.15.76:0/184810370 shutdown_connections 2024-03-12T05:58:54.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 -- 172.21.15.76:0/184810370 wait complete. 2024-03-12T05:58:54.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 Processor -- start 2024-03-12T05:58:54.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 -- start start 2024-03-12T05:58:54.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c100290 0x7f8f1c19d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f1c105360 0x7f8f1c19d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f1c197640 0x7f8f1c197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c198030 con 0x7f8f1c197640 2024-03-12T05:58:54.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c1981a0 con 0x7f8f1c105360 2024-03-12T05:58:54.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.960+0000 7f8f21adf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c198360 con 0x7f8f1c100290 2024-03-12T05:58:54.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f1bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f1c197640 0x7f8f1c197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f1bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f1c197640 0x7f8f1c197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51680/0 (socket says 172.21.15.76:51680) 2024-03-12T05:58:54.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f1bfff640 1 -- 172.21.15.76:0/984967045 learned_addr learned my addr 172.21.15.76:0/984967045 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f1b7fe640 1 --2- 172.21.15.76:0/984967045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c100290 0x7f8f1c19d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f13fff640 1 --2- 172.21.15.76:0/984967045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f1c105360 0x7f8f1c19d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f1bfff640 1 -- 172.21.15.76:0/984967045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c100290 msgr2=0x7f8f1c19d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f1bfff640 1 --2- 172.21.15.76:0/984967045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c100290 0x7f8f1c19d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f1bfff640 1 -- 172.21.15.76:0/984967045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f1c105360 msgr2=0x7f8f1c19d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f1bfff640 1 --2- 172.21.15.76:0/984967045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f1c105360 0x7f8f1c19d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f1bfff640 1 -- 172.21.15.76:0/984967045 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0c009560 con 0x7f8f1c197640 2024-03-12T05:58:54.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f1bfff640 1 --2- 172.21.15.76:0/984967045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f1c197640 0x7f8f1c197af0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8f1400e960 tx=0x7f8f1400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f197fa640 1 -- 172.21.15.76:0/984967045 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f14004d20 con 0x7f8f1c197640 2024-03-12T05:58:54.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f197fa640 1 -- 172.21.15.76:0/984967045 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f14004e80 con 0x7f8f1c197640 2024-03-12T05:58:54.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f21adf640 1 -- 172.21.15.76:0/984967045 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1c1a4300 con 0x7f8f1c197640 2024-03-12T05:58:54.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f197fa640 1 -- 172.21.15.76:0/984967045 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f140116d0 con 0x7f8f1c197640 2024-03-12T05:58:54.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.961+0000 7f8f21adf640 1 -- 172.21.15.76:0/984967045 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c1a47c0 con 0x7f8f1c197640 2024-03-12T05:58:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.964+0000 7f8f24793740 1 Processor -- start 2024-03-12T05:58:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.964+0000 7f8f197fa640 1 -- 172.21.15.76:0/984967045 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8f14005340 con 0x7f8f1c197640 2024-03-12T05:58:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.964+0000 7f8f24793740 1 -- start start 2024-03-12T05:58:54.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.964+0000 7f8f24793740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 0x55a59b155630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f24793740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155b70 0x55a59b15e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f24793740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b160b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f24793740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59b110960 con 0x55a59b15e750 2024-03-12T05:58:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f24793740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59b0c4310 con 0x55a59b155b70 2024-03-12T05:58:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f24793740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59b0c1840 con 0x55a59b155250 2024-03-12T05:58:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f13fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b160b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f13fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b160b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51684/0 (socket says 172.21.15.76:51684) 2024-03-12T05:58:54.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f13fff640 1 -- 172.21.15.76:0/4223891968 learned_addr learned my addr 172.21.15.76:0/4223891968 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f1b7fe640 1 --2- 172.21.15.76:0/4223891968 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155b70 0x55a59b15e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f197fa640 1 --2- 172.21.15.76:0/984967045 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8f080a1ba0 0x7f8f080a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f13fff640 1 --2- 172.21.15.76:0/4223891968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 0x55a59b155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f1bfff640 1 --2- 172.21.15.76:0/984967045 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8f080a1ba0 0x7f8f080a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f13fff640 1 -- 172.21.15.76:0/4223891968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 msgr2=0x55a59b155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f13fff640 1 --2- 172.21.15.76:0/4223891968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 0x55a59b155630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f13fff640 1 -- 172.21.15.76:0/4223891968 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155b70 msgr2=0x55a59b15e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f13fff640 1 --2- 172.21.15.76:0/4223891968 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155b70 0x55a59b15e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f13fff640 1 -- 172.21.15.76:0/4223891968 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a59b07f060 con 0x55a59b15e750 2024-03-12T05:58:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f1b7fe640 1 --2- 172.21.15.76:0/4223891968 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155b70 0x55a59b15e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:58:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f197fa640 1 -- 172.21.15.76:0/984967045 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8f140cd8f0 con 0x7f8f1c197640 2024-03-12T05:58:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f13fff640 1 --2- 172.21.15.76:0/4223891968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b160b40 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c06a870 tx=0x7f8f0400b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74071885ab9b2659 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f11ffb640 1 -- 172.21.15.76:0/4223891968 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04005020 con 0x55a59b15e750 2024-03-12T05:58:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.965+0000 7f8f1bfff640 1 --2- 172.21.15.76:0/984967045 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8f080a1ba0 0x7f8f080a4060 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8f14020000 tx=0x7f8f14095040 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.966+0000 7f8f11ffb640 1 -- 172.21.15.76:0/4223891968 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f04005660 con 0x55a59b15e750 2024-03-12T05:58:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.966+0000 7f8f11ffb640 1 -- 172.21.15.76:0/4223891968 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04004a00 con 0x55a59b15e750 2024-03-12T05:58:54.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.966+0000 7f8f24793740 1 -- 172.21.15.76:0/4223891968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 msgr2=0x55a59b160b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.966+0000 7f8f24793740 1 --2- 172.21.15.76:0/4223891968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b160b40 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c06a870 tx=0x7f8f0400b9b0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.966+0000 7f8f24793740 1 -- 172.21.15.76:0/4223891968 shutdown_connections 2024-03-12T05:58:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.966+0000 7f8f24793740 1 --2- 172.21.15.76:0/4223891968 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b160b40 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.966+0000 7f8f24793740 1 --2- 172.21.15.76:0/4223891968 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155b70 0x55a59b15e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.966+0000 7f8f24793740 1 --2- 172.21.15.76:0/4223891968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 0x55a59b155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.966+0000 7f8f24793740 1 -- 172.21.15.76:0/4223891968 >> 172.21.15.76:0/4223891968 conn(0x55a59b14f980 msgr2=0x55a59b14fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.966+0000 7f8f24793740 1 -- 172.21.15.76:0/4223891968 shutdown_connections 2024-03-12T05:58:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.966+0000 7f8f24793740 1 -- 172.21.15.76:0/4223891968 wait complete. 2024-03-12T05:58:54.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.966+0000 7f8f24793740 1 -- start start 2024-03-12T05:58:54.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f24793740 1 Processor -- start 2024-03-12T05:58:54.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f24793740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155250 0x55a59b15b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f24793740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155b70 0x55a59b15bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f24793740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b15c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f24793740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59b0bd020 con 0x55a59b15e750 2024-03-12T05:58:54.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f24793740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59ad90430 con 0x55a59b155250 2024-03-12T05:58:54.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f24793740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59ad4ce00 con 0x55a59b155b70 2024-03-12T05:58:54.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f24793740 10 client.0 _ll_register_callbacks cb 0x55a59b064cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:54.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f13fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b15c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f13fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b15c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51686/0 (socket says 172.21.15.76:51686) 2024-03-12T05:58:54.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f13fff640 1 -- 172.21.15.76:0/280970352 learned_addr learned my addr 172.21.15.76:0/280970352 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f13fff640 1 --2- 172.21.15.76:0/280970352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155250 0x55a59b15b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f1b7fe640 1 --2- 172.21.15.76:0/280970352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155b70 0x55a59b15bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f13fff640 1 -- 172.21.15.76:0/280970352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155b70 msgr2=0x55a59b15bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f13fff640 1 --2- 172.21.15.76:0/280970352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155b70 0x55a59b15bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f13fff640 1 -- 172.21.15.76:0/280970352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155250 msgr2=0x55a59b15b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f13fff640 1 --2- 172.21.15.76:0/280970352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155250 0x55a59b15b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.967+0000 7f8f13fff640 1 -- 172.21.15.76:0/280970352 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0401a040 con 0x55a59b15e750 2024-03-12T05:58:54.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8f1b7fe640 1 --2- 172.21.15.76:0/280970352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155b70 0x55a59b15bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:54.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8f13fff640 1 --2- 172.21.15.76:0/280970352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b15c230 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400b470 tx=0x7f8f04010e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=734598606f192be1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8efaffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:54.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8efaffd640 1 -- 172.21.15.76:0/280970352 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0400b520 con 0x55a59b15e750 2024-03-12T05:58:54.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8efaffd640 1 -- 172.21.15.76:0/280970352 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8f040048d0 con 0x55a59b15e750 2024-03-12T05:58:54.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8f24793740 1 -- 172.21.15.76:0/280970352 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a59adbc7e0 con 0x55a59b15e750 2024-03-12T05:58:54.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8efaffd640 1 -- 172.21.15.76:0/280970352 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f04010830 con 0x55a59b15e750 2024-03-12T05:58:54.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8f24793740 10 client.16281 fetch_fsmap learned FSMap version 25 2024-03-12T05:58:54.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8f24793740 1 -- 172.21.15.76:0/280970352 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a59b0bb6a0 con 0x55a59b15e750 2024-03-12T05:58:54.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8efaffd640 1 -- 172.21.15.76:0/280970352 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a59adbc7e0 con 0x55a59b15e750 2024-03-12T05:58:54.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8efaffd640 1 -- 172.21.15.76:0/280970352 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f8f0401f5e0 con 0x55a59b15e750 2024-03-12T05:58:54.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8f24793740 10 client.16281 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:58:54.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8f24793740 10 client.16281 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:54.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8f24793740 20 client.16281 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:54.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8f24793740 1 --2- 172.21.15.76:0/280970352 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a59b16ead0 0x55a59b170ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8f24793740 4 client.16281 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:58:54.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.968+0000 7f8f24793740 1 -- 172.21.15.76:0/280970352 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a59adf9720 con 0x55a59b16ead0 2024-03-12T05:58:54.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.969+0000 7f8f1b7fe640 1 --2- 172.21.15.76:0/280970352 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a59b16ead0 0x55a59b170ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.969+0000 7f8efaffd640 10 client.16281 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:54.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.969+0000 7f8f1b7fe640 1 --2- 172.21.15.76:0/280970352 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a59b16ead0 0x55a59b170ef0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c9fc25fe6a6a4f2 server_cookie=24fd26493bc51557 in_seq=0 out_seq=0 2024-03-12T05:58:54.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8efaffd640 1 -- 172.21.15.76:0/280970352 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a59adf9720 con 0x55a59b16ead0 2024-03-12T05:58:54.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8efaffd640 10 client.16281 handle_command_reply: tid=0 2024-03-12T05:58:54.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8f24793740 1 client.16281 shutdown 2024-03-12T05:58:54.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8f24793740 1 -- 172.21.15.76:0/280970352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 msgr2=0x55a59b15c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8f24793740 1 --2- 172.21.15.76:0/280970352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b15c230 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400b470 tx=0x7f8f04010e70 comp rx=0 tx=0).stop 2024-03-12T05:58:54.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8f24793740 1 -- 172.21.15.76:0/280970352 shutdown_connections 2024-03-12T05:58:54.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8f24793740 1 --2- 172.21.15.76:0/280970352 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a59b16ead0 0x55a59b170ef0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8f24793740 1 --2- 172.21.15.76:0/280970352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b15c230 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8f24793740 1 --2- 172.21.15.76:0/280970352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155b70 0x55a59b15bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8efaffd640 0 client.16281 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:54.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8f24793740 1 --2- 172.21.15.76:0/280970352 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155250 0x55a59b15b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8f24793740 1 -- 172.21.15.76:0/280970352 >> 172.21.15.76:0/280970352 conn(0x55a59b14f980 msgr2=0x55a59b1516d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8f24793740 1 -- 172.21.15.76:0/280970352 shutdown_connections 2024-03-12T05:58:54.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8f24793740 1 -- 172.21.15.76:0/280970352 wait complete. 2024-03-12T05:58:54.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.970+0000 7f8f24793740 20 client.16281 trim_cache size 0 max 16384 2024-03-12T05:58:54.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.981+0000 7f8f24793740 1 Processor -- start 2024-03-12T05:58:54.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.981+0000 7f8f24793740 1 -- start start 2024-03-12T05:58:54.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f24793740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 0x55a59b15dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f24793740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b155b70 0x55a59b156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f24793740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b15e750 0x55a59b156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.994 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f24793740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59b042130 con 0x55a59b155b70 2024-03-12T05:58:54.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f24793740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59b154f10 con 0x55a59b15e750 2024-03-12T05:58:54.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f24793740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59b150a00 con 0x55a59b155250 2024-03-12T05:58:54.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f1b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b155b70 0x55a59b156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f1b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b155b70 0x55a59b156a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51698/0 (socket says 172.21.15.76:51698) 2024-03-12T05:58:54.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f1b7fe640 1 -- 172.21.15.76:0/1705442785 learned_addr learned my addr 172.21.15.76:0/1705442785 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f13fff640 1 --2- 172.21.15.76:0/1705442785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b15e750 0x55a59b156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f13fff640 1 --2- 172.21.15.76:0/1705442785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 0x55a59b15dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f1b7fe640 1 -- 172.21.15.76:0/1705442785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 msgr2=0x55a59b15dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.995 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f1b7fe640 1 --2- 172.21.15.76:0/1705442785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 0x55a59b15dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f1b7fe640 1 -- 172.21.15.76:0/1705442785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b15e750 msgr2=0x55a59b156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f1b7fe640 1 --2- 172.21.15.76:0/1705442785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b15e750 0x55a59b156fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f1b7fe640 1 -- 172.21.15.76:0/1705442785 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a59b057c20 con 0x55a59b155b70 2024-03-12T05:58:54.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f13fff640 1 --2- 172.21.15.76:0/1705442785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b15e750 0x55a59b156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:54.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f1b7fe640 1 --2- 172.21.15.76:0/1705442785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b155b70 0x55a59b156a90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c005ab0 tx=0x7f8f0c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a606c19df30a892a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:54.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f117fa640 1 -- 172.21.15.76:0/1705442785 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c048be0 con 0x55a59b155b70 2024-03-12T05:58:54.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f117fa640 1 -- 172.21.15.76:0/1705442785 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f0c048d40 con 0x55a59b155b70 2024-03-12T05:58:54.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.982+0000 7f8f117fa640 1 -- 172.21.15.76:0/1705442785 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c048600 con 0x55a59b155b70 2024-03-12T05:58:54.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.983+0000 7f8f24793740 1 -- 172.21.15.76:0/1705442785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b155b70 msgr2=0x55a59b156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:54.996 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.983+0000 7f8f24793740 1 --2- 172.21.15.76:0/1705442785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b155b70 0x55a59b156a90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c005ab0 tx=0x7f8f0c009260 comp rx=0 tx=0).stop 2024-03-12T05:58:54.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.983+0000 7f8f24793740 1 -- 172.21.15.76:0/1705442785 shutdown_connections 2024-03-12T05:58:54.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.983+0000 7f8f24793740 1 --2- 172.21.15.76:0/1705442785 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b15e750 0x55a59b156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.983+0000 7f8f24793740 1 --2- 172.21.15.76:0/1705442785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b155b70 0x55a59b156a90 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.983+0000 7f8f24793740 1 --2- 172.21.15.76:0/1705442785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 0x55a59b15dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.983+0000 7f8f24793740 1 -- 172.21.15.76:0/1705442785 >> 172.21.15.76:0/1705442785 conn(0x55a59b14f980 msgr2=0x55a59b14aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:54.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.983+0000 7f8f24793740 1 -- 172.21.15.76:0/1705442785 shutdown_connections 2024-03-12T05:58:54.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.983+0000 7f8f24793740 1 -- 172.21.15.76:0/1705442785 wait complete. 2024-03-12T05:58:54.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.983+0000 7f8f24793740 1 -- start start 2024-03-12T05:58:54.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.983+0000 7f8f24793740 1 Processor -- start 2024-03-12T05:58:54.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f24793740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 0x55a59b164640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f24793740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155b70 0x55a59b164b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f24793740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b1c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:54.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f24793740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59b165280 con 0x55a59b15e750 2024-03-12T05:58:54.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f24793740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59b1c5520 con 0x55a59b155b70 2024-03-12T05:58:54.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f24793740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59b1c5660 con 0x55a59b155250 2024-03-12T05:58:54.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f24793740 10 client.0 _ll_register_callbacks cb 0x7f8f0400b860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:54.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f13fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b1c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f1b7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155b70 0x55a59b164b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f1b7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155b70 0x55a59b164b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:40594/0 (socket says 172.21.15.76:40594) 2024-03-12T05:58:54.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f1b7fe640 1 -- 172.21.15.76:0/3149596739 learned_addr learned my addr 172.21.15.76:0/3149596739 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:54.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f13fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b1c4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51704/0 (socket says 172.21.15.76:51704) 2024-03-12T05:58:54.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f13fff640 1 --2- 172.21.15.76:0/3149596739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 0x55a59b164640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:54.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f1b7fe640 1 -- 172.21.15.76:0/3149596739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 msgr2=0x55a59b164640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f1b7fe640 1 --2- 172.21.15.76:0/3149596739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 0x55a59b164640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:54.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f1b7fe640 1 -- 172.21.15.76:0/3149596739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 msgr2=0x55a59b1c4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:54.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f1b7fe640 1 --2- 172.21.15.76:0/3149596739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b1c4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:55.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f1b7fe640 1 -- 172.21.15.76:0/3149596739 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0c030070 con 0x55a59b155b70 2024-03-12T05:58:55.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f13fff640 1 --2- 172.21.15.76:0/3149596739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b1c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:58:55.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8f1b7fe640 1 --2- 172.21.15.76:0/3149596739 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155b70 0x55a59b164b80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c009230 tx=0x7f8f0c02c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c519900f0e32295 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:55.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.984+0000 7f8efaffd640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T05:58:55.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8efaffd640 1 -- 172.21.15.76:0/3149596739 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c0485a0 con 0x55a59b155b70 2024-03-12T05:58:55.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8f24793740 1 -- 172.21.15.76:0/3149596739 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f04005660 con 0x55a59b155b70 2024-03-12T05:58:55.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8efaffd640 1 -- 172.21.15.76:0/3149596739 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8f0c045040 con 0x55a59b155b70 2024-03-12T05:58:55.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8efaffd640 1 -- 172.21.15.76:0/3149596739 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f0c02c7e0 con 0x55a59b155b70 2024-03-12T05:58:55.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8f24793740 10 client.25673 fetch_fsmap learned FSMap version 25 2024-03-12T05:58:55.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8f24793740 1 -- 172.21.15.76:0/3149596739 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a59b07d8c0 con 0x55a59b155b70 2024-03-12T05:58:55.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8efaffd640 1 -- 172.21.15.76:0/3149596739 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f04005660 con 0x55a59b155b70 2024-03-12T05:58:55.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8efaffd640 1 -- 172.21.15.76:0/3149596739 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f8f0c03c040 con 0x55a59b155b70 2024-03-12T05:58:55.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8f24793740 10 client.25673 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:58:55.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8f24793740 10 client.25673 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:55.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8f24793740 20 client.25673 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:55.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8f24793740 1 --2- 172.21.15.76:0/3149596739 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a59b1c6c70 0x55a59b1c9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:55.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8f24793740 4 client.25673 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:58:55.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8f24793740 1 -- 172.21.15.76:0/3149596739 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a59ae48890 con 0x55a59b1c6c70 2024-03-12T05:58:55.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.985+0000 7f8f13fff640 1 --2- 172.21.15.76:0/3149596739 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a59b1c6c70 0x55a59b1c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:55.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8efaffd640 10 client.25673 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:55.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8f13fff640 1 --2- 172.21.15.76:0/3149596739 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a59b1c6c70 0x55a59b1c9090 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b84407c613c206e server_cookie=6b281d4e009918b5 in_seq=0 out_seq=0 2024-03-12T05:58:55.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8efaffd640 1 -- 172.21.15.76:0/3149596739 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a59ae48890 con 0x55a59b1c6c70 2024-03-12T05:58:55.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8efaffd640 10 client.25673 handle_command_reply: tid=0 2024-03-12T05:58:55.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8f24793740 1 client.25673 shutdown 2024-03-12T05:58:55.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8f24793740 1 -- 172.21.15.76:0/3149596739 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155b70 msgr2=0x55a59b164b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:55.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8f24793740 1 --2- 172.21.15.76:0/3149596739 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155b70 0x55a59b164b80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c009230 tx=0x7f8f0c02c740 comp rx=0 tx=0).stop 2024-03-12T05:58:55.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8f24793740 1 -- 172.21.15.76:0/3149596739 shutdown_connections 2024-03-12T05:58:55.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8f24793740 1 --2- 172.21.15.76:0/3149596739 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a59b1c6c70 0x55a59b1c9090 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:55.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8f24793740 1 --2- 172.21.15.76:0/3149596739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a59b15e750 0x55a59b1c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:55.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8f24793740 1 --2- 172.21.15.76:0/3149596739 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a59b155b70 0x55a59b164b80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:55.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8efaffd640 0 client.25673 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:55.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8f24793740 1 --2- 172.21.15.76:0/3149596739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a59b155250 0x55a59b164640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:55.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8f24793740 1 -- 172.21.15.76:0/3149596739 >> 172.21.15.76:0/3149596739 conn(0x55a59b14f980 msgr2=0x55a59b159c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:55.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8f24793740 1 -- 172.21.15.76:0/3149596739 shutdown_connections 2024-03-12T05:58:55.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8f24793740 1 -- 172.21.15.76:0/3149596739 wait complete. 2024-03-12T05:58:55.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.986+0000 7f8f24793740 20 client.25673 trim_cache size 0 max 16384 2024-03-12T05:58:55.004 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:58:55.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.987+0000 7f8f21adf640 1 -- 172.21.15.76:0/984967045 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8f080a1ba0 msgr2=0x7f8f080a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:55.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.987+0000 7f8f21adf640 1 --2- 172.21.15.76:0/984967045 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8f080a1ba0 0x7f8f080a4060 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8f14020000 tx=0x7f8f14095040 comp rx=0 tx=0).stop 2024-03-12T05:58:55.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.987+0000 7f8f21adf640 1 -- 172.21.15.76:0/984967045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f1c197640 msgr2=0x7f8f1c197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:55.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.987+0000 7f8f21adf640 1 --2- 172.21.15.76:0/984967045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f1c197640 0x7f8f1c197af0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8f1400e960 tx=0x7f8f1400ee20 comp rx=0 tx=0).stop 2024-03-12T05:58:55.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.987+0000 7f8f21adf640 1 -- 172.21.15.76:0/984967045 shutdown_connections 2024-03-12T05:58:55.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.987+0000 7f8f21adf640 1 --2- 172.21.15.76:0/984967045 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8f080a1ba0 0x7f8f080a4060 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:55.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.987+0000 7f8f21adf640 1 --2- 172.21.15.76:0/984967045 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f1c197640 0x7f8f1c197af0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:55.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.987+0000 7f8f21adf640 1 --2- 172.21.15.76:0/984967045 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8f1c105360 0x7f8f1c19d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:55.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.987+0000 7f8f21adf640 1 --2- 172.21.15.76:0/984967045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c100290 0x7f8f1c19d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:55.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.987+0000 7f8f21adf640 1 -- 172.21.15.76:0/984967045 >> 172.21.15.76:0/984967045 conn(0x7f8f1c077230 msgr2=0x7f8f1c068830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:55.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.987+0000 7f8f21adf640 1 -- 172.21.15.76:0/984967045 shutdown_connections 2024-03-12T05:58:55.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:54.987+0000 7f8f21adf640 1 -- 172.21.15.76:0/984967045 wait complete. 2024-03-12T05:58:55.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:55 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:58:55.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:55 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2767783035' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:58:55.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:55 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:58:55.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:55 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:58:55.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:55 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:58:55.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:55 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:58:55.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:55 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2767783035' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:58:55.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:55 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:58:55.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:55 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:58:55.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:55 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:58:55.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:55 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:58:55.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:55 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2767783035' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:58:55.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:55 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:58:55.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:55 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:58:55.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:55 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:58:56.008 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:58:56.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:58:56.008 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:58:56.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.119+0000 7f240c393640 1 -- 172.21.15.76:0/1171855658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2404105360 msgr2=0x7f240410cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.119+0000 7f240c393640 1 --2- 172.21.15.76:0/1171855658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2404105360 0x7f240410cf30 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009cd0 tx=0x7f23fc02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.119+0000 7f240c393640 1 -- 172.21.15.76:0/1171855658 shutdown_connections 2024-03-12T05:58:56.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.119+0000 7f240c393640 1 --2- 172.21.15.76:0/1171855658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2404105360 0x7f240410cf30 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.119+0000 7f240c393640 1 --2- 172.21.15.76:0/1171855658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2404100c40 0x7f2404104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.119+0000 7f240c393640 1 --2- 172.21.15.76:0/1171855658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2404100290 0x7f2404100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.119+0000 7f240c393640 1 -- 172.21.15.76:0/1171855658 >> 172.21.15.76:0/1171855658 conn(0x7f2404077230 msgr2=0x7f2404077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:56.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.119+0000 7f240c393640 1 -- 172.21.15.76:0/1171855658 shutdown_connections 2024-03-12T05:58:56.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.119+0000 7f240c393640 1 -- 172.21.15.76:0/1171855658 wait complete. 2024-03-12T05:58:56.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240c393640 1 Processor -- start 2024-03-12T05:58:56.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240c393640 1 -- start start 2024-03-12T05:58:56.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240c393640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2404100290 0x7f240419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240c393640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2404105360 0x7f240419fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240c393640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24041997e0 0x7f2404199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240c393640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240419a290 con 0x7f2404100290 2024-03-12T05:58:56.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240c393640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240419a400 con 0x7f2404105360 2024-03-12T05:58:56.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240c393640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240419a570 con 0x7f24041997e0 2024-03-12T05:58:56.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240a12f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2404100290 0x7f240419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240a12f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2404100290 0x7f240419f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51726/0 (socket says 172.21.15.76:51726) 2024-03-12T05:58:56.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240a12f640 1 -- 172.21.15.76:0/2686794223 learned_addr learned my addr 172.21.15.76:0/2686794223 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:56.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240992e640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2404105360 0x7f240419fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240a930640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24041997e0 0x7f2404199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240a12f640 1 -- 172.21.15.76:0/2686794223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24041997e0 msgr2=0x7f2404199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240a12f640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24041997e0 0x7f2404199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240a12f640 1 -- 172.21.15.76:0/2686794223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2404105360 msgr2=0x7f240419fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240a12f640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2404105360 0x7f240419fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240a12f640 1 -- 172.21.15.76:0/2686794223 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc009900 con 0x7f2404100290 2024-03-12T05:58:56.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.120+0000 7f240992e640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2404105360 0x7f240419fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:58:56.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.121+0000 7f240a12f640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2404100290 0x7f240419f670 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f23f400c890 tx=0x7f23f400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:56.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.121+0000 7f23fb7fe640 1 -- 172.21.15.76:0/2686794223 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4004220 con 0x7f2404100290 2024-03-12T05:58:56.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.121+0000 7f23fb7fe640 1 -- 172.21.15.76:0/2686794223 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23f4007e90 con 0x7f2404100290 2024-03-12T05:58:56.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.121+0000 7f23fb7fe640 1 -- 172.21.15.76:0/2686794223 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f4004a00 con 0x7f2404100290 2024-03-12T05:58:56.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.121+0000 7f240c393640 1 -- 172.21.15.76:0/2686794223 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24041a6520 con 0x7f2404100290 2024-03-12T05:58:56.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.121+0000 7f240c393640 1 -- 172.21.15.76:0/2686794223 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24041a6a00 con 0x7f2404100290 2024-03-12T05:58:56.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.123+0000 7f23fb7fe640 1 -- 172.21.15.76:0/2686794223 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f23f40079b0 con 0x7f2404100290 2024-03-12T05:58:56.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.123+0000 7f240ef69740 1 Processor -- start 2024-03-12T05:58:56.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f240ef69740 1 -- start start 2024-03-12T05:58:56.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f240ef69740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b0380 0x55eb4e4b0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f240ef69740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4b0d30 0x55eb4e4b92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f23fb7fe640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f23f00a1ba0 0x7f23f00a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f240ef69740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b98b0 0x55eb4e4bbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f240ef69740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb4e41ef90 con 0x55eb4e4b98b0 2024-03-12T05:58:56.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f240ef69740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb4e41c4c0 con 0x55eb4e4b0380 2024-03-12T05:58:56.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f240ef69740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb4e417ca0 con 0x55eb4e4b0d30 2024-03-12T05:58:56.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f240992e640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f23f00a1ba0 0x7f23f00a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f240a930640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b98b0 0x55eb4e4bbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f240a930640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b98b0 0x55eb4e4bbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51738/0 (socket says 172.21.15.76:51738) 2024-03-12T05:58:56.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f240a930640 1 -- 172.21.15.76:0/4293684997 learned_addr learned my addr 172.21.15.76:0/4293684997 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:56.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f23fb7fe640 1 -- 172.21.15.76:0/2686794223 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f23f40cd590 con 0x7f2404100290 2024-03-12T05:58:56.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f240a12f640 1 --2- 172.21.15.76:0/4293684997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4b0d30 0x55eb4e4b92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f240a930640 1 --2- 172.21.15.76:0/4293684997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b0380 0x55eb4e4b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.124+0000 7f240a930640 1 -- 172.21.15.76:0/4293684997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4b0d30 msgr2=0x55eb4e4b92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.125+0000 7f240a930640 1 --2- 172.21.15.76:0/4293684997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4b0d30 0x55eb4e4b92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.125+0000 7f240a930640 1 -- 172.21.15.76:0/4293684997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b0380 msgr2=0x55eb4e4b0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.125+0000 7f240a930640 1 --2- 172.21.15.76:0/4293684997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b0380 0x55eb4e4b0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.125+0000 7f240a930640 1 -- 172.21.15.76:0/4293684997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb4e3da0b0 con 0x55eb4e4b98b0 2024-03-12T05:58:56.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.125+0000 7f240a12f640 1 --2- 172.21.15.76:0/4293684997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4b0d30 0x55eb4e4b92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:58:56.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.125+0000 7f240992e640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f23f00a1ba0 0x7f23f00a4060 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f240410d350 tx=0x7f2400005cb0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:56.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.125+0000 7f240a930640 1 --2- 172.21.15.76:0/4293684997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b98b0 0x55eb4e4bbd00 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f23fc033000 tx=0x7f23fc009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44908d1849fd5ddd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:56.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.125+0000 7f23f8bfe640 1 -- 172.21.15.76:0/4293684997 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc0347a0 con 0x55eb4e4b98b0 2024-03-12T05:58:56.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.125+0000 7f23f8bfe640 1 -- 172.21.15.76:0/4293684997 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23fc003970 con 0x55eb4e4b98b0 2024-03-12T05:58:56.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.125+0000 7f23f8bfe640 1 -- 172.21.15.76:0/4293684997 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc004310 con 0x55eb4e4b98b0 2024-03-12T05:58:56.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 -- 172.21.15.76:0/4293684997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b98b0 msgr2=0x55eb4e4bbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 --2- 172.21.15.76:0/4293684997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b98b0 0x55eb4e4bbd00 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f23fc033000 tx=0x7f23fc009360 comp rx=0 tx=0).stop 2024-03-12T05:58:56.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 -- 172.21.15.76:0/4293684997 shutdown_connections 2024-03-12T05:58:56.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 --2- 172.21.15.76:0/4293684997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b98b0 0x55eb4e4bbd00 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 --2- 172.21.15.76:0/4293684997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4b0d30 0x55eb4e4b92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 --2- 172.21.15.76:0/4293684997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b0380 0x55eb4e4b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 -- 172.21.15.76:0/4293684997 >> 172.21.15.76:0/4293684997 conn(0x55eb4e4aa840 msgr2=0x55eb4e4aac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:56.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 -- 172.21.15.76:0/4293684997 shutdown_connections 2024-03-12T05:58:56.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 -- 172.21.15.76:0/4293684997 wait complete. 2024-03-12T05:58:56.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 -- start start 2024-03-12T05:58:56.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 Processor -- start 2024-03-12T05:58:56.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e4b6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4b0d30 0x55eb4e4b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.140 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b98b0 0x55eb4e4b72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb4e3bcf40 con 0x55eb4e4b0380 2024-03-12T05:58:56.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb4e0eb4b0 con 0x55eb4e4b98b0 2024-03-12T05:58:56.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb4e39d670 con 0x55eb4e4b0d30 2024-03-12T05:58:56.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240ef69740 10 client.0 _ll_register_callbacks cb 0x55eb4e40af70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:56.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240a12f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e4b6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240a12f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e4b6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51750/0 (socket says 172.21.15.76:51750) 2024-03-12T05:58:56.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.126+0000 7f240a12f640 1 -- 172.21.15.76:0/489809768 learned_addr learned my addr 172.21.15.76:0/489809768 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:56.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f240992e640 1 --2- 172.21.15.76:0/489809768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4b0d30 0x55eb4e4b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f240a930640 1 --2- 172.21.15.76:0/489809768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b98b0 0x55eb4e4b72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f240a12f640 1 -- 172.21.15.76:0/489809768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4b0d30 msgr2=0x55eb4e4b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f240a12f640 1 --2- 172.21.15.76:0/489809768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4b0d30 0x55eb4e4b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f240a12f640 1 -- 172.21.15.76:0/489809768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b98b0 msgr2=0x55eb4e4b72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f240a12f640 1 --2- 172.21.15.76:0/489809768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b98b0 0x55eb4e4b72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f240a12f640 1 -- 172.21.15.76:0/489809768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2404068450 con 0x55eb4e4b0380 2024-03-12T05:58:56.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f240992e640 1 --2- 172.21.15.76:0/489809768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4b0d30 0x55eb4e4b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:58:56.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f240a12f640 1 --2- 172.21.15.76:0/489809768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e4b6870 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f23f4005fd0 tx=0x7f23f409a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4545fba0d5ff2b8c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:56.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f23edffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:56.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f23edffb640 1 -- 172.21.15.76:0/489809768 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f409e170 con 0x55eb4e4b0380 2024-03-12T05:58:56.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f23edffb640 1 -- 172.21.15.76:0/489809768 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f23f40a2990 con 0x55eb4e4b0380 2024-03-12T05:58:56.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f240ef69740 1 -- 172.21.15.76:0/489809768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eb4e3f1f40 con 0x55eb4e4b0380 2024-03-12T05:58:56.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f23edffb640 1 -- 172.21.15.76:0/489809768 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23f4096860 con 0x55eb4e4b0380 2024-03-12T05:58:56.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f240ef69740 10 client.16299 fetch_fsmap learned FSMap version 25 2024-03-12T05:58:56.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f240ef69740 1 -- 172.21.15.76:0/489809768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55eb4e416320 con 0x55eb4e4b0380 2024-03-12T05:58:56.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f23edffb640 1 -- 172.21.15.76:0/489809768 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55eb4e3f1f40 con 0x55eb4e4b0380 2024-03-12T05:58:56.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f23edffb640 1 -- 172.21.15.76:0/489809768 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f23f4016030 con 0x55eb4e4b0380 2024-03-12T05:58:56.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f240ef69740 10 client.16299 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:58:56.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.127+0000 7f240ef69740 10 client.16299 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:56.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.128+0000 7f240ef69740 20 client.16299 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:56.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.128+0000 7f240ef69740 1 --2- 172.21.15.76:0/489809768 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55eb4e4c9db0 0x55eb4e4cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.128+0000 7f240ef69740 4 client.16299 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:58:56.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.128+0000 7f240ef69740 1 -- 172.21.15.76:0/489809768 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb4e40e310 con 0x55eb4e4c9db0 2024-03-12T05:58:56.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.128+0000 7f240992e640 1 --2- 172.21.15.76:0/489809768 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55eb4e4c9db0 0x55eb4e4cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.128+0000 7f23edffb640 10 client.16299 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:56.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.128+0000 7f240992e640 1 --2- 172.21.15.76:0/489809768 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55eb4e4c9db0 0x55eb4e4cc1f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edcb6cfbd5963ae6 server_cookie=7e364943433081b8 in_seq=0 out_seq=0 2024-03-12T05:58:56.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f23edffb640 1 -- 172.21.15.76:0/489809768 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55eb4e40e310 con 0x55eb4e4c9db0 2024-03-12T05:58:56.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f23edffb640 10 client.16299 handle_command_reply: tid=0 2024-03-12T05:58:56.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f240ef69740 1 client.16299 shutdown 2024-03-12T05:58:56.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f240ef69740 1 -- 172.21.15.76:0/489809768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 msgr2=0x55eb4e4b6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f240ef69740 1 --2- 172.21.15.76:0/489809768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e4b6870 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f23f4005fd0 tx=0x7f23f409a1f0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f240ef69740 1 -- 172.21.15.76:0/489809768 shutdown_connections 2024-03-12T05:58:56.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f240ef69740 1 --2- 172.21.15.76:0/489809768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b98b0 0x55eb4e4b72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f240ef69740 1 --2- 172.21.15.76:0/489809768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4b0d30 0x55eb4e4b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f240ef69740 1 --2- 172.21.15.76:0/489809768 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55eb4e4c9db0 0x55eb4e4cc1f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f240ef69740 1 --2- 172.21.15.76:0/489809768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e4b6870 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f23edffb640 0 client.16299 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:56.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f240ef69740 1 -- 172.21.15.76:0/489809768 >> 172.21.15.76:0/489809768 conn(0x55eb4e4aa840 msgr2=0x55eb4e4ac7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:56.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f240ef69740 1 -- 172.21.15.76:0/489809768 shutdown_connections 2024-03-12T05:58:56.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f240ef69740 1 -- 172.21.15.76:0/489809768 wait complete. 2024-03-12T05:58:56.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.129+0000 7f240ef69740 20 client.16299 trim_cache size 0 max 16384 2024-03-12T05:58:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.144+0000 7f240ef69740 1 Processor -- start 2024-03-12T05:58:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.144+0000 7f240ef69740 1 -- start start 2024-03-12T05:58:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.144+0000 7f240ef69740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b1480 0x55eb4e4b7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.144+0000 7f240ef69740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4aa840 0x55eb4e4a9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.144+0000 7f240ef69740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e4c0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.144+0000 7f240ef69740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb4e40e340 con 0x55eb4e4b0380 2024-03-12T05:58:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.144+0000 7f240ef69740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb4e3f1f70 con 0x55eb4e4b1480 2024-03-12T05:58:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.144+0000 7f240ef69740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb4e4aa370 con 0x55eb4e4aa840 2024-03-12T05:58:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f240a930640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e4c0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f240a930640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e4c0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51762/0 (socket says 172.21.15.76:51762) 2024-03-12T05:58:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f240a930640 1 -- 172.21.15.76:0/2326197950 learned_addr learned my addr 172.21.15.76:0/2326197950 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f240a12f640 1 --2- 172.21.15.76:0/2326197950 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b1480 0x55eb4e4b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f240a930640 1 -- 172.21.15.76:0/2326197950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4aa840 msgr2=0x55eb4e4a9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f240992e640 1 --2- 172.21.15.76:0/2326197950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4aa840 0x55eb4e4a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f240a930640 1 --2- 172.21.15.76:0/2326197950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4aa840 0x55eb4e4a9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f240a930640 1 -- 172.21.15.76:0/2326197950 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b1480 msgr2=0x55eb4e4b7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f240a930640 1 --2- 172.21.15.76:0/2326197950 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b1480 0x55eb4e4b7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f240a930640 1 -- 172.21.15.76:0/2326197950 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb4e3b2c00 con 0x55eb4e4b0380 2024-03-12T05:58:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f240992e640 1 --2- 172.21.15.76:0/2326197950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4aa840 0x55eb4e4a9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:58:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f240a930640 1 --2- 172.21.15.76:0/2326197950 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e4c0880 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f23fc033000 tx=0x7f23fc009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff2fb888af3d4a7f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f23effff640 1 -- 172.21.15.76:0/2326197950 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc0347a0 con 0x55eb4e4b0380 2024-03-12T05:58:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f23effff640 1 -- 172.21.15.76:0/2326197950 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23fc032340 con 0x55eb4e4b0380 2024-03-12T05:58:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.145+0000 7f23effff640 1 -- 172.21.15.76:0/2326197950 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc032a30 con 0x55eb4e4b0380 2024-03-12T05:58:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 -- 172.21.15.76:0/2326197950 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 msgr2=0x55eb4e4c0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 --2- 172.21.15.76:0/2326197950 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e4c0880 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f23fc033000 tx=0x7f23fc009540 comp rx=0 tx=0).stop 2024-03-12T05:58:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 -- 172.21.15.76:0/2326197950 shutdown_connections 2024-03-12T05:58:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 --2- 172.21.15.76:0/2326197950 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e4c0880 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 --2- 172.21.15.76:0/2326197950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4aa840 0x55eb4e4a9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 --2- 172.21.15.76:0/2326197950 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b1480 0x55eb4e4b7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 -- 172.21.15.76:0/2326197950 >> 172.21.15.76:0/2326197950 conn(0x55eb4e4b2240 msgr2=0x55eb4e4a9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 -- 172.21.15.76:0/2326197950 shutdown_connections 2024-03-12T05:58:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 -- 172.21.15.76:0/2326197950 wait complete. 2024-03-12T05:58:56.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 -- start start 2024-03-12T05:58:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 Processor -- start 2024-03-12T05:58:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4aa840 0x55eb4e5195b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e519af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b1480 0x55eb4e4bfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb4e4c00a0 con 0x55eb4e4b0380 2024-03-12T05:58:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb4e4c0210 con 0x55eb4e4b1480 2024-03-12T05:58:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb4e4c03d0 con 0x55eb4e4aa840 2024-03-12T05:58:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240ef69740 10 client.0 _ll_register_callbacks cb 0x7f240406aa10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:56.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240992e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e519af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.146+0000 7f240992e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e519af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51770/0 (socket says 172.21.15.76:51770) 2024-03-12T05:58:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f240992e640 1 -- 172.21.15.76:0/2502024534 learned_addr learned my addr 172.21.15.76:0/2502024534 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f240a12f640 1 --2- 172.21.15.76:0/2502024534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4aa840 0x55eb4e5195b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f240a930640 1 --2- 172.21.15.76:0/2502024534 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b1480 0x55eb4e4bfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f240992e640 1 -- 172.21.15.76:0/2502024534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4aa840 msgr2=0x55eb4e5195b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f240992e640 1 --2- 172.21.15.76:0/2502024534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4aa840 0x55eb4e5195b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f240992e640 1 -- 172.21.15.76:0/2502024534 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b1480 msgr2=0x55eb4e4bfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f240992e640 1 --2- 172.21.15.76:0/2502024534 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b1480 0x55eb4e4bfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f240992e640 1 -- 172.21.15.76:0/2502024534 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb4e3da0b0 con 0x55eb4e4b0380 2024-03-12T05:58:56.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f240992e640 1 --2- 172.21.15.76:0/2502024534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e519af0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f240419ab30 tx=0x7f2400004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eddaabf8a739a572 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:56.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f23edffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:56.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f23edffb640 1 -- 172.21.15.76:0/2502024534 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2400023d00 con 0x55eb4e4b0380 2024-03-12T05:58:56.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f23edffb640 1 -- 172.21.15.76:0/2502024534 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f240001d900 con 0x55eb4e4b0380 2024-03-12T05:58:56.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f240ef69740 1 -- 172.21.15.76:0/2502024534 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eb4e43d900 con 0x55eb4e4b0380 2024-03-12T05:58:56.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f23edffb640 1 -- 172.21.15.76:0/2502024534 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2400023e60 con 0x55eb4e4b0380 2024-03-12T05:58:56.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f240ef69740 10 client.16305 fetch_fsmap learned FSMap version 25 2024-03-12T05:58:56.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f240ef69740 1 -- 172.21.15.76:0/2502024534 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55eb4e3d8910 con 0x55eb4e4b0380 2024-03-12T05:58:56.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.147+0000 7f23edffb640 1 -- 172.21.15.76:0/2502024534 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55eb4e43d900 con 0x55eb4e4b0380 2024-03-12T05:58:56.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.148+0000 7f23edffb640 1 -- 172.21.15.76:0/2502024534 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f240000b050 con 0x55eb4e4b0380 2024-03-12T05:58:56.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.148+0000 7f240ef69740 10 client.16305 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:58:56.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.148+0000 7f240ef69740 10 client.16305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:56.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.148+0000 7f240ef69740 20 client.16305 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:56.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.148+0000 7f240ef69740 1 --2- 172.21.15.76:0/2502024534 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55eb4e4b0d30 0x55eb4e524580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.148+0000 7f240ef69740 4 client.16305 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:58:56.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.148+0000 7f240ef69740 1 -- 172.21.15.76:0/2502024534 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55eb4e117860 con 0x55eb4e4b0d30 2024-03-12T05:58:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.148+0000 7f240a12f640 1 --2- 172.21.15.76:0/2502024534 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55eb4e4b0d30 0x55eb4e524580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.148+0000 7f23edffb640 10 client.16305 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.148+0000 7f240a12f640 1 --2- 172.21.15.76:0/2502024534 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55eb4e4b0d30 0x55eb4e524580 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1eefca73a628938 server_cookie=75a6f75372d4f990 in_seq=0 out_seq=0 2024-03-12T05:58:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.149+0000 7f23edffb640 1 -- 172.21.15.76:0/2502024534 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55eb4e117860 con 0x55eb4e4b0d30 2024-03-12T05:58:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.149+0000 7f23edffb640 10 client.16305 handle_command_reply: tid=0 2024-03-12T05:58:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.149+0000 7f240ef69740 1 client.16305 shutdown 2024-03-12T05:58:56.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.149+0000 7f240ef69740 1 -- 172.21.15.76:0/2502024534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 msgr2=0x55eb4e519af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.149+0000 7f240ef69740 1 --2- 172.21.15.76:0/2502024534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e519af0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f240419ab30 tx=0x7f2400004730 comp rx=0 tx=0).stop 2024-03-12T05:58:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.149+0000 7f240ef69740 1 -- 172.21.15.76:0/2502024534 shutdown_connections 2024-03-12T05:58:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.149+0000 7f240ef69740 1 --2- 172.21.15.76:0/2502024534 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55eb4e4b1480 0x55eb4e4bfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.149+0000 7f240ef69740 1 --2- 172.21.15.76:0/2502024534 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55eb4e4b0d30 0x55eb4e524580 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240ef69740 1 --2- 172.21.15.76:0/2502024534 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55eb4e4b0380 0x55eb4e519af0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f23edffb640 0 client.16305 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240ef69740 1 --2- 172.21.15.76:0/2502024534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eb4e4aa840 0x55eb4e5195b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240ef69740 1 -- 172.21.15.76:0/2502024534 >> 172.21.15.76:0/2502024534 conn(0x55eb4e4b2240 msgr2=0x55eb4e4a9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:56.163 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240ef69740 1 -- 172.21.15.76:0/2502024534 shutdown_connections 2024-03-12T05:58:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240ef69740 1 -- 172.21.15.76:0/2502024534 wait complete. 2024-03-12T05:58:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240ef69740 20 client.16305 trim_cache size 0 max 16384 2024-03-12T05:58:56.164 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:58:56.164 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T05:58:56.164 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "3ee2f9f7-7b2d-4a21-95d4-d204430ddf11", 2024-03-12T05:58:56.164 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T05:58:56.165 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:58:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240c393640 1 -- 172.21.15.76:0/2686794223 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f23f00a1ba0 msgr2=0x7f23f00a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240c393640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f23f00a1ba0 0x7f23f00a4060 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f240410d350 tx=0x7f2400005cb0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240c393640 1 -- 172.21.15.76:0/2686794223 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2404100290 msgr2=0x7f240419f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240c393640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2404100290 0x7f240419f670 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f23f400c890 tx=0x7f23f400cd50 comp rx=0 tx=0).stop 2024-03-12T05:58:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240c393640 1 -- 172.21.15.76:0/2686794223 shutdown_connections 2024-03-12T05:58:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240c393640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f23f00a1ba0 0x7f23f00a4060 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240c393640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24041997e0 0x7f2404199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240c393640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2404105360 0x7f240419fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240c393640 1 --2- 172.21.15.76:0/2686794223 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2404100290 0x7f240419f670 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240c393640 1 -- 172.21.15.76:0/2686794223 >> 172.21.15.76:0/2686794223 conn(0x7f2404077230 msgr2=0x7f24040fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240c393640 1 -- 172.21.15.76:0/2686794223 shutdown_connections 2024-03-12T05:58:56.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.150+0000 7f240c393640 1 -- 172.21.15.76:0/2686794223 wait complete. 2024-03-12T05:58:56.168 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:58:56.168 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:58:56.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.287+0000 7f899ad20640 1 -- 172.21.15.76:0/699694870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8994100260 msgr2=0x7f8994100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.287+0000 7f899ad20640 1 --2- 172.21.15.76:0/699694870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8994100260 0x7f8994100640 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f8988009930 tx=0x7f898802bc30 comp rx=0 tx=0).stop 2024-03-12T05:58:56.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.287+0000 7f899ad20640 1 -- 172.21.15.76:0/699694870 shutdown_connections 2024-03-12T05:58:56.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.287+0000 7f899ad20640 1 --2- 172.21.15.76:0/699694870 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8994105330 0x7f899410cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.287+0000 7f899ad20640 1 --2- 172.21.15.76:0/699694870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8994100c10 0x7f8994104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.287+0000 7f899ad20640 1 --2- 172.21.15.76:0/699694870 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8994100260 0x7f8994100640 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.287+0000 7f899ad20640 1 -- 172.21.15.76:0/699694870 >> 172.21.15.76:0/699694870 conn(0x7f8994077180 msgr2=0x7f8994077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:56.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.288+0000 7f899ad20640 1 -- 172.21.15.76:0/699694870 shutdown_connections 2024-03-12T05:58:56.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.288+0000 7f899ad20640 1 -- 172.21.15.76:0/699694870 wait complete. 2024-03-12T05:58:56.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.288+0000 7f899ad20640 1 Processor -- start 2024-03-12T05:58:56.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.288+0000 7f899ad20640 1 -- start start 2024-03-12T05:58:56.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.288+0000 7f899ad20640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8994100260 0x7f899419f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.288+0000 7f899ad20640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8994105330 0x7f899419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.288+0000 7f899ad20640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89941997b0 0x7f8994199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.289+0000 7f899ad20640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899419a260 con 0x7f8994100260 2024-03-12T05:58:56.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.289+0000 7f899ad20640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899419a3d0 con 0x7f89941997b0 2024-03-12T05:58:56.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.289+0000 7f899ad20640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899419a540 con 0x7f8994105330 2024-03-12T05:58:56.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.289+0000 7f8998abc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8994100260 0x7f899419f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.289+0000 7f8998abc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8994100260 0x7f899419f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51796/0 (socket says 172.21.15.76:51796) 2024-03-12T05:58:56.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.289+0000 7f8998abc640 1 -- 172.21.15.76:0/275970262 learned_addr learned my addr 172.21.15.76:0/275970262 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:56.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.289+0000 7f8993fff640 1 --2- 172.21.15.76:0/275970262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8994105330 0x7f899419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.289+0000 7f89992bd640 1 --2- 172.21.15.76:0/275970262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89941997b0 0x7f8994199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.289+0000 7f8998abc640 1 -- 172.21.15.76:0/275970262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8994105330 msgr2=0x7f899419fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.289+0000 7f8998abc640 1 --2- 172.21.15.76:0/275970262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8994105330 0x7f899419fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.289+0000 7f8998abc640 1 -- 172.21.15.76:0/275970262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89941997b0 msgr2=0x7f8994199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.289+0000 7f8998abc640 1 --2- 172.21.15.76:0/275970262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89941997b0 0x7f8994199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.289+0000 7f8998abc640 1 -- 172.21.15.76:0/275970262 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8988009560 con 0x7f8994100260 2024-03-12T05:58:56.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.290+0000 7f8998abc640 1 --2- 172.21.15.76:0/275970262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8994100260 0x7f899419f640 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8988009230 tx=0x7f898802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:56.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.290+0000 7f8991ffb640 1 -- 172.21.15.76:0/275970262 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8988004350 con 0x7f8994100260 2024-03-12T05:58:56.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.290+0000 7f8991ffb640 1 -- 172.21.15.76:0/275970262 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89880044b0 con 0x7f8994100260 2024-03-12T05:58:56.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.290+0000 7f899ad20640 1 -- 172.21.15.76:0/275970262 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89941a6490 con 0x7f8994100260 2024-03-12T05:58:56.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.290+0000 7f8991ffb640 1 -- 172.21.15.76:0/275970262 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8988003d00 con 0x7f8994100260 2024-03-12T05:58:56.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.290+0000 7f899ad20640 1 -- 172.21.15.76:0/275970262 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89941a69a0 con 0x7f8994100260 2024-03-12T05:58:56.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.293+0000 7f8991ffb640 1 -- 172.21.15.76:0/275970262 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8988004df0 con 0x7f8994100260 2024-03-12T05:58:56.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.293+0000 7f899d9cb740 1 Processor -- start 2024-03-12T05:58:56.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.293+0000 7f899d9cb740 1 -- start start 2024-03-12T05:58:56.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f899d9cb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56213038e2b0 0x56213038e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f899d9cb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038ebd0 0x562130397270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f8991ffb640 1 --2- 172.21.15.76:0/275970262 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f89840a1c20 0x7f89840a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f899d9cb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621303977b0 0x562130399ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f899d9cb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621302fd280 con 0x56213038ebd0 2024-03-12T05:58:56.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f899d9cb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621302fa7b0 con 0x5621303977b0 2024-03-12T05:58:56.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f899d9cb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621302f5f90 con 0x56213038e2b0 2024-03-12T05:58:56.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f8998abc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038ebd0 0x562130397270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f8993fff640 1 --2- 172.21.15.76:0/275970262 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f89840a1c20 0x7f89840a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f8998abc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038ebd0 0x562130397270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51800/0 (socket says 172.21.15.76:51800) 2024-03-12T05:58:56.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f8998abc640 1 -- 172.21.15.76:0/2345456083 learned_addr learned my addr 172.21.15.76:0/2345456083 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:56.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f89992bd640 1 --2- 172.21.15.76:0/2345456083 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621303977b0 0x562130399ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f89992bd640 1 --2- 172.21.15.76:0/2345456083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56213038e2b0 0x56213038e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f8991ffb640 1 -- 172.21.15.76:0/275970262 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f89880ef700 con 0x7f8994100260 2024-03-12T05:58:56.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f89992bd640 1 -- 172.21.15.76:0/2345456083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56213038e2b0 msgr2=0x56213038e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f89992bd640 1 --2- 172.21.15.76:0/2345456083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56213038e2b0 0x56213038e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f89992bd640 1 -- 172.21.15.76:0/2345456083 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038ebd0 msgr2=0x562130397270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f89992bd640 1 --2- 172.21.15.76:0/2345456083 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038ebd0 0x562130397270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f89992bd640 1 -- 172.21.15.76:0/2345456083 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621302b4570 con 0x5621303977b0 2024-03-12T05:58:56.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.294+0000 7f89992bd640 1 --2- 172.21.15.76:0/2345456083 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621303977b0 0x562130399ba0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f898c0119f0 tx=0x7f898c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=666a0ac8de2d68c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:56.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.295+0000 7f8993fff640 1 --2- 172.21.15.76:0/275970262 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f89840a1c20 0x7f89840a40e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f899410d2c0 tx=0x7f8980005f90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:56.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.295+0000 7f897effd640 1 -- 172.21.15.76:0/2345456083 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898c011130 con 0x5621303977b0 2024-03-12T05:58:56.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.295+0000 7f897effd640 1 -- 172.21.15.76:0/2345456083 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f898c011290 con 0x5621303977b0 2024-03-12T05:58:56.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.295+0000 7f897effd640 1 -- 172.21.15.76:0/2345456083 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898c005280 con 0x5621303977b0 2024-03-12T05:58:56.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.295+0000 7f899d9cb740 1 -- 172.21.15.76:0/2345456083 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621303977b0 msgr2=0x562130399ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.295+0000 7f899d9cb740 1 --2- 172.21.15.76:0/2345456083 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621303977b0 0x562130399ba0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f898c0119f0 tx=0x7f898c004a40 comp rx=0 tx=0).stop 2024-03-12T05:58:56.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.295+0000 7f899d9cb740 1 -- 172.21.15.76:0/2345456083 shutdown_connections 2024-03-12T05:58:56.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.295+0000 7f899d9cb740 1 --2- 172.21.15.76:0/2345456083 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621303977b0 0x562130399ba0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.295+0000 7f899d9cb740 1 --2- 172.21.15.76:0/2345456083 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038ebd0 0x562130397270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.295+0000 7f899d9cb740 1 --2- 172.21.15.76:0/2345456083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56213038e2b0 0x56213038e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.295+0000 7f899d9cb740 1 -- 172.21.15.76:0/2345456083 >> 172.21.15.76:0/2345456083 conn(0x562130388990 msgr2=0x562130388d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:56.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.295+0000 7f899d9cb740 1 -- 172.21.15.76:0/2345456083 shutdown_connections 2024-03-12T05:58:56.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.296+0000 7f899d9cb740 1 -- 172.21.15.76:0/2345456083 wait complete. 2024-03-12T05:58:56.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.296+0000 7f899d9cb740 1 -- start start 2024-03-12T05:58:56.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.296+0000 7f899d9cb740 1 Processor -- start 2024-03-12T05:58:56.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f899d9cb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x562130397210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f899d9cb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56213038ebd0 0x5621303945a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f899d9cb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621303977b0 0x562130394cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f899d9cb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56212ffc9430 con 0x56213038e2b0 2024-03-12T05:58:56.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f899d9cb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56212ff85e00 con 0x5621303977b0 2024-03-12T05:58:56.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f899d9cb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56213027afd0 con 0x56213038ebd0 2024-03-12T05:58:56.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f899d9cb740 10 client.0 _ll_register_callbacks cb 0x5621302d0fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:56.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f8998abc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x562130397210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f8993fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56213038ebd0 0x5621303945a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f8998abc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x562130397210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51804/0 (socket says 172.21.15.76:51804) 2024-03-12T05:58:56.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f8998abc640 1 -- 172.21.15.76:0/1671397163 learned_addr learned my addr 172.21.15.76:0/1671397163 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:56.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f89992bd640 1 --2- 172.21.15.76:0/1671397163 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621303977b0 0x562130394cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f8998abc640 1 -- 172.21.15.76:0/1671397163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56213038ebd0 msgr2=0x5621303945a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f8998abc640 1 --2- 172.21.15.76:0/1671397163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56213038ebd0 0x5621303945a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f8998abc640 1 -- 172.21.15.76:0/1671397163 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621303977b0 msgr2=0x562130394cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f8998abc640 1 --2- 172.21.15.76:0/1671397163 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621303977b0 0x562130394cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f8998abc640 1 -- 172.21.15.76:0/1671397163 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898c019040 con 0x56213038e2b0 2024-03-12T05:58:56.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.297+0000 7f8993fff640 1 --2- 172.21.15.76:0/1671397163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56213038ebd0 0x5621303945a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:58:56.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.298+0000 7f8998abc640 1 --2- 172.21.15.76:0/1671397163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x562130397210 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8988005ec0 tx=0x7f89880dc0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185fc31c8c865664 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:56.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.298+0000 7f8973fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:58:56.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.298+0000 7f8973fff640 1 -- 172.21.15.76:0/1671397163 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89880bbfb0 con 0x56213038e2b0 2024-03-12T05:58:56.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.298+0000 7f8973fff640 1 -- 172.21.15.76:0/1671397163 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f89880035f0 con 0x56213038e2b0 2024-03-12T05:58:56.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.298+0000 7f899d9cb740 1 -- 172.21.15.76:0/1671397163 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56212fff57e0 con 0x56213038e2b0 2024-03-12T05:58:56.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.298+0000 7f8973fff640 1 -- 172.21.15.76:0/1671397163 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89880bbdd0 con 0x56213038e2b0 2024-03-12T05:58:56.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.298+0000 7f899d9cb740 10 client.16317 fetch_fsmap learned FSMap version 25 2024-03-12T05:58:56.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.298+0000 7f899d9cb740 1 -- 172.21.15.76:0/1671397163 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5621302f4610 con 0x56213038e2b0 2024-03-12T05:58:56.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.298+0000 7f8973fff640 1 -- 172.21.15.76:0/1671397163 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56212fff57e0 con 0x56213038e2b0 2024-03-12T05:58:56.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.298+0000 7f8973fff640 1 -- 172.21.15.76:0/1671397163 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f898803c070 con 0x56213038e2b0 2024-03-12T05:58:56.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.299+0000 7f899d9cb740 10 client.16317 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:58:56.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.299+0000 7f899d9cb740 10 client.16317 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:56.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.299+0000 7f899d9cb740 20 client.16317 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:56.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.299+0000 7f899d9cb740 1 --2- 172.21.15.76:0/1671397163 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621303afff0 0x5621303b2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.299+0000 7f899d9cb740 4 client.16317 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:58:56.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.299+0000 7f899d9cb740 1 -- 172.21.15.76:0/1671397163 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562130032720 con 0x5621303afff0 2024-03-12T05:58:56.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.299+0000 7f8993fff640 1 --2- 172.21.15.76:0/1671397163 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621303afff0 0x5621303b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.299+0000 7f8993fff640 1 --2- 172.21.15.76:0/1671397163 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621303afff0 0x5621303b2410 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55e96d42c8643d17 server_cookie=10157cfc252b66fe in_seq=0 out_seq=0 2024-03-12T05:58:56.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.299+0000 7f8973fff640 10 client.16317 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:56.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f8973fff640 1 -- 172.21.15.76:0/1671397163 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562130032720 con 0x5621303afff0 2024-03-12T05:58:56.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f8973fff640 10 client.16317 handle_command_reply: tid=0 2024-03-12T05:58:56.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f899d9cb740 1 client.16317 shutdown 2024-03-12T05:58:56.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f899d9cb740 1 -- 172.21.15.76:0/1671397163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 msgr2=0x562130397210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f899d9cb740 1 --2- 172.21.15.76:0/1671397163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x562130397210 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8988005ec0 tx=0x7f89880dc0b0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f899d9cb740 1 -- 172.21.15.76:0/1671397163 shutdown_connections 2024-03-12T05:58:56.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f899d9cb740 1 --2- 172.21.15.76:0/1671397163 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621303977b0 0x562130394cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f899d9cb740 1 --2- 172.21.15.76:0/1671397163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56213038ebd0 0x5621303945a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f899d9cb740 1 --2- 172.21.15.76:0/1671397163 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621303afff0 0x5621303b2410 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f8973fff640 0 client.16317 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:56.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f899d9cb740 1 --2- 172.21.15.76:0/1671397163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x562130397210 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f899d9cb740 1 -- 172.21.15.76:0/1671397163 >> 172.21.15.76:0/1671397163 conn(0x562130388990 msgr2=0x56213038a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:56.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f899d9cb740 1 -- 172.21.15.76:0/1671397163 shutdown_connections 2024-03-12T05:58:56.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f899d9cb740 1 -- 172.21.15.76:0/1671397163 wait complete. 2024-03-12T05:58:56.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.300+0000 7f899d9cb740 20 client.16317 trim_cache size 0 max 16384 2024-03-12T05:58:56.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.320+0000 7f899d9cb740 1 Processor -- start 2024-03-12T05:58:56.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.320+0000 7f899d9cb740 1 -- start start 2024-03-12T05:58:56.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f899d9cb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x562130396cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f899d9cb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56213038ebd0 0x562130397210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f899d9cb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621303977b0 0x562130393e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f899d9cb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621303840f0 con 0x56213038e2b0 2024-03-12T05:58:56.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f899d9cb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562130382eb0 con 0x56213038ebd0 2024-03-12T05:58:56.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f899d9cb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562130384320 con 0x5621303977b0 2024-03-12T05:58:56.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f8998abc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x562130396cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f8998abc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x562130396cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51818/0 (socket says 172.21.15.76:51818) 2024-03-12T05:58:56.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f8998abc640 1 -- 172.21.15.76:0/762661559 learned_addr learned my addr 172.21.15.76:0/762661559 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:56.329 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f8993fff640 1 --2- 172.21.15.76:0/762661559 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56213038ebd0 0x562130397210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f89992bd640 1 --2- 172.21.15.76:0/762661559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621303977b0 0x562130393e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f8993fff640 1 -- 172.21.15.76:0/762661559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621303977b0 msgr2=0x562130393e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f8993fff640 1 --2- 172.21.15.76:0/762661559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621303977b0 0x562130393e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f8993fff640 1 -- 172.21.15.76:0/762661559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 msgr2=0x562130396cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f8993fff640 1 --2- 172.21.15.76:0/762661559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x562130396cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f8993fff640 1 -- 172.21.15.76:0/762661559 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562130290ac0 con 0x56213038ebd0 2024-03-12T05:58:56.330 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f8998abc640 1 --2- 172.21.15.76:0/762661559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x562130396cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:58:56.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.321+0000 7f8993fff640 1 --2- 172.21.15.76:0/762661559 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56213038ebd0 0x562130397210 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x5621303b2d20 tx=0x7f8980014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=602932de60f1f90d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:56.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.322+0000 7f897e7fc640 1 -- 172.21.15.76:0/762661559 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980026070 con 0x56213038ebd0 2024-03-12T05:58:56.331 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.322+0000 7f897e7fc640 1 -- 172.21.15.76:0/762661559 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8980016070 con 0x56213038ebd0 2024-03-12T05:58:56.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.322+0000 7f897e7fc640 1 -- 172.21.15.76:0/762661559 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898001fa20 con 0x56213038ebd0 2024-03-12T05:58:56.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.322+0000 7f899d9cb740 1 -- 172.21.15.76:0/762661559 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56213038ebd0 msgr2=0x562130397210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.322+0000 7f899d9cb740 1 --2- 172.21.15.76:0/762661559 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56213038ebd0 0x562130397210 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x5621303b2d20 tx=0x7f8980014000 comp rx=0 tx=0).stop 2024-03-12T05:58:56.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.322+0000 7f899d9cb740 1 -- 172.21.15.76:0/762661559 shutdown_connections 2024-03-12T05:58:56.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.322+0000 7f899d9cb740 1 --2- 172.21.15.76:0/762661559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621303977b0 0x562130393e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.322+0000 7f899d9cb740 1 --2- 172.21.15.76:0/762661559 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56213038ebd0 0x562130397210 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.322+0000 7f899d9cb740 1 --2- 172.21.15.76:0/762661559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x562130396cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.322+0000 7f899d9cb740 1 -- 172.21.15.76:0/762661559 >> 172.21.15.76:0/762661559 conn(0x562130388990 msgr2=0x5621303b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:56.332 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.322+0000 7f899d9cb740 1 -- 172.21.15.76:0/762661559 shutdown_connections 2024-03-12T05:58:56.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.322+0000 7f899d9cb740 1 -- 172.21.15.76:0/762661559 wait complete. 2024-03-12T05:58:56.333 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.323+0000 7f899d9cb740 1 -- start start 2024-03-12T05:58:56.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.323+0000 7f899d9cb740 1 Processor -- start 2024-03-12T05:58:56.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.323+0000 7f899d9cb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x5621303a6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.323+0000 7f899d9cb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56213038ebd0 0x5621303a7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.323+0000 7f899d9cb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621303977b0 0x5621303a3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.323+0000 7f899d9cb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621303a3b30 con 0x56213038e2b0 2024-03-12T05:58:56.334 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.323+0000 7f899d9cb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621303a3ca0 con 0x56213038ebd0 2024-03-12T05:58:56.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.323+0000 7f899d9cb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621303a3e60 con 0x5621303977b0 2024-03-12T05:58:56.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.323+0000 7f899d9cb740 10 client.0 _ll_register_callbacks cb 0x7f898c009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:58:56.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.323+0000 7f8998abc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x5621303a6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.323+0000 7f8993fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56213038ebd0 0x5621303a7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.323+0000 7f8993fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56213038ebd0 0x5621303a7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:40728/0 (socket says 172.21.15.76:40728) 2024-03-12T05:58:56.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.324+0000 7f8993fff640 1 -- 172.21.15.76:0/1556333133 learned_addr learned my addr 172.21.15.76:0/1556333133 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:58:56.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.324+0000 7f89992bd640 1 --2- 172.21.15.76:0/1556333133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621303977b0 0x5621303a3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.335 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.324+0000 7f8993fff640 1 -- 172.21.15.76:0/1556333133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621303977b0 msgr2=0x5621303a3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.324+0000 7f8993fff640 1 --2- 172.21.15.76:0/1556333133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621303977b0 0x5621303a3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.324+0000 7f8993fff640 1 -- 172.21.15.76:0/1556333133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 msgr2=0x5621303a6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:58:56.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.324+0000 7f8993fff640 1 --2- 172.21.15.76:0/1556333133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x5621303a6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.324+0000 7f8993fff640 1 -- 172.21.15.76:0/1556333133 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5621302f4610 con 0x56213038ebd0 2024-03-12T05:58:56.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.324+0000 7f8993fff640 1 --2- 172.21.15.76:0/1556333133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56213038ebd0 0x5621303a7430 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8980021040 tx=0x7f8980012e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de6354d11d212daa server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:58:56.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.324+0000 7f8973fff640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T05:58:56.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.324+0000 7f8973fff640 1 -- 172.21.15.76:0/1556333133 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980026050 con 0x56213038ebd0 2024-03-12T05:58:56.336 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.324+0000 7f899d9cb740 1 -- 172.21.15.76:0/1556333133 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f898c011290 con 0x56213038ebd0 2024-03-12T05:58:56.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.324+0000 7f8973fff640 1 -- 172.21.15.76:0/1556333133 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8980018070 con 0x56213038ebd0 2024-03-12T05:58:56.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.324+0000 7f8973fff640 1 -- 172.21.15.76:0/1556333133 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8980005390 con 0x56213038ebd0 2024-03-12T05:58:56.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.325+0000 7f899d9cb740 10 client.25703 fetch_fsmap learned FSMap version 25 2024-03-12T05:58:56.337 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.325+0000 7f899d9cb740 1 -- 172.21.15.76:0/1556333133 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f8980012450 con 0x56213038ebd0 2024-03-12T05:58:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.325+0000 7f8973fff640 1 -- 172.21.15.76:0/1556333133 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f898c011290 con 0x56213038ebd0 2024-03-12T05:58:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.325+0000 7f8973fff640 1 -- 172.21.15.76:0/1556333133 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f898001b040 con 0x56213038ebd0 2024-03-12T05:58:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.325+0000 7f899d9cb740 10 client.25703 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:58:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.325+0000 7f899d9cb740 10 client.25703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:58:56.339 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.325+0000 7f899d9cb740 20 client.25703 populate_metadata read hostname 'smithi076' 2024-03-12T05:58:56.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.325+0000 7f899d9cb740 1 --2- 172.21.15.76:0/1556333133 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621303afff0 0x562130409950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:58:56.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.325+0000 7f899d9cb740 4 client.25703 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:58:56.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.325+0000 7f899d9cb740 1 -- 172.21.15.76:0/1556333133 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562130081890 con 0x5621303afff0 2024-03-12T05:58:56.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.325+0000 7f8998abc640 1 --2- 172.21.15.76:0/1556333133 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621303afff0 0x562130409950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:58:56.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f8973fff640 10 client.25703 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:56.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f8998abc640 1 --2- 172.21.15.76:0/1556333133 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621303afff0 0x562130409950 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8f80975648ca763 server_cookie=26d74a3565357f1b in_seq=0 out_seq=0 2024-03-12T05:58:56.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f8973fff640 1 -- 172.21.15.76:0/1556333133 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x562130081890 con 0x5621303afff0 2024-03-12T05:58:56.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f8973fff640 10 client.25703 handle_command_reply: tid=0 2024-03-12T05:58:56.340 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f899d9cb740 1 client.25703 shutdown 2024-03-12T05:58:56.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f899d9cb740 1 -- 172.21.15.76:0/1556333133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56213038ebd0 msgr2=0x5621303a7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f899d9cb740 1 --2- 172.21.15.76:0/1556333133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56213038ebd0 0x5621303a7430 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8980021040 tx=0x7f8980012e30 comp rx=0 tx=0).stop 2024-03-12T05:58:56.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f899d9cb740 1 -- 172.21.15.76:0/1556333133 shutdown_connections 2024-03-12T05:58:56.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f899d9cb740 1 --2- 172.21.15.76:0/1556333133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621303977b0 0x5621303a3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f899d9cb740 1 --2- 172.21.15.76:0/1556333133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56213038ebd0 0x5621303a7430 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f899d9cb740 1 --2- 172.21.15.76:0/1556333133 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621303afff0 0x562130409950 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f899d9cb740 1 --2- 172.21.15.76:0/1556333133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56213038e2b0 0x5621303a6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f8973fff640 0 client.25703 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:58:56.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f899d9cb740 1 -- 172.21.15.76:0/1556333133 >> 172.21.15.76:0/1556333133 conn(0x562130388990 msgr2=0x562130389e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:56.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f899d9cb740 1 -- 172.21.15.76:0/1556333133 shutdown_connections 2024-03-12T05:58:56.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.326+0000 7f899d9cb740 1 -- 172.21.15.76:0/1556333133 wait complete. 2024-03-12T05:58:56.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.327+0000 7f899d9cb740 20 client.25703 trim_cache size 0 max 16384 2024-03-12T05:58:56.342 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:58:56.342 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:58:56.342 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T05:58:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: "3ee2f9f7-7b2d-4a21-95d4-d204430ddf11": { 2024-03-12T05:58:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T05:58:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "3ee2f9f7-7b2d-4a21-95d4-d204430ddf11", 2024-03-12T05:58:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T05:58:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:58:56.343 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:58:56.343 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:58:56.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.327+0000 7f899ad20640 1 -- 172.21.15.76:0/275970262 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f89840a1c20 msgr2=0x7f89840a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.327+0000 7f899ad20640 1 --2- 172.21.15.76:0/275970262 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f89840a1c20 0x7f89840a40e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f899410d2c0 tx=0x7f8980005f90 comp rx=0 tx=0).stop 2024-03-12T05:58:56.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.327+0000 7f899ad20640 1 -- 172.21.15.76:0/275970262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8994100260 msgr2=0x7f899419f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:58:56.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.327+0000 7f899ad20640 1 --2- 172.21.15.76:0/275970262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8994100260 0x7f899419f640 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8988009230 tx=0x7f898802c650 comp rx=0 tx=0).stop 2024-03-12T05:58:56.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.327+0000 7f899ad20640 1 -- 172.21.15.76:0/275970262 shutdown_connections 2024-03-12T05:58:56.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.327+0000 7f899ad20640 1 --2- 172.21.15.76:0/275970262 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f89840a1c20 0x7f89840a40e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.327+0000 7f899ad20640 1 --2- 172.21.15.76:0/275970262 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89941997b0 0x7f8994199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.327+0000 7f899ad20640 1 --2- 172.21.15.76:0/275970262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8994105330 0x7f899419fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.327+0000 7f899ad20640 1 --2- 172.21.15.76:0/275970262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8994100260 0x7f899419f640 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:58:56.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.327+0000 7f899ad20640 1 -- 172.21.15.76:0/275970262 >> 172.21.15.76:0/275970262 conn(0x7f8994077180 msgr2=0x7f8994068670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:58:56.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.328+0000 7f899ad20640 1 -- 172.21.15.76:0/275970262 shutdown_connections 2024-03-12T05:58:56.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:58:56.328+0000 7f899ad20640 1 -- 172.21.15.76:0/275970262 wait complete. 2024-03-12T05:58:56.346 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:56 smithi076 ceph-mon[32095]: pgmap v97: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 10 op/s 2024-03-12T05:58:56.347 INFO:tasks.cephfs.filesystem:scrub status for tag:3ee2f9f7-7b2d-4a21-95d4-d204430ddf11 - {'path': '/', 'tag': '3ee2f9f7-7b2d-4a21-95d4-d204430ddf11', 'options': 'recursive,force'} 2024-03-12T05:58:56.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:56 smithi082 ceph-mon[36382]: pgmap v97: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 10 op/s 2024-03-12T05:58:56.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:56 smithi088 ceph-mon[36585]: pgmap v97: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 10 op/s 2024-03-12T05:58:57.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:57 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T05:58:57.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:57 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:57.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:57 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:58:57.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:57 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:58:57.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:57 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:58:57.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:57 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:57.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:57 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:57.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:57 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T05:58:57.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:57 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:57.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:57 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:58:57.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:57 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:58:57.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:57 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:58:57.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:57 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:57.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:57 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:57.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:57 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T05:58:57.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:57 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:57.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:57 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:58:57.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:57 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:58:57.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:57 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:58:57.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:57 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:57.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:57 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:58.386 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:58 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:58.386 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:58:58 smithi088 ceph-mon[36585]: pgmap v98: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 10 op/s 2024-03-12T05:58:58.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:58 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:58.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:58:58 smithi082 ceph-mon[36382]: pgmap v98: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 10 op/s 2024-03-12T05:58:58.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:58 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:58:58.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:58:58 smithi076 ceph-mon[32095]: pgmap v98: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 10 op/s 2024-03-12T05:59:00.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:00 smithi082 ceph-mon[36382]: pgmap v99: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 14 op/s 2024-03-12T05:59:00.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:00 smithi088 ceph-mon[36585]: pgmap v99: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 14 op/s 2024-03-12T05:59:00.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:00 smithi076 ceph-mon[32095]: pgmap v99: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 14 op/s 2024-03-12T05:59:02.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:02 smithi082 ceph-mon[36382]: pgmap v100: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-03-12T05:59:02.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:02 smithi088 ceph-mon[36585]: pgmap v100: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-03-12T05:59:02.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:02 smithi076 ceph-mon[32095]: pgmap v100: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-03-12T05:59:04.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:04 smithi082 ceph-mon[36382]: pgmap v101: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 34 KiB/s wr, 25 op/s 2024-03-12T05:59:04.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:04 smithi088 ceph-mon[36585]: pgmap v101: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 34 KiB/s wr, 25 op/s 2024-03-12T05:59:04.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:04 smithi076 ceph-mon[32095]: pgmap v101: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 34 KiB/s wr, 25 op/s 2024-03-12T05:59:06.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:06 smithi082 ceph-mon[36382]: pgmap v102: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-03-12T05:59:06.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:06 smithi088 ceph-mon[36585]: pgmap v102: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-03-12T05:59:06.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:06 smithi076 ceph-mon[32095]: pgmap v102: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-03-12T05:59:08.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:08 smithi082 ceph-mon[36382]: pgmap v103: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-03-12T05:59:08.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:08 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:59:08.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:08 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:59:08.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:08 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:59:08.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:08 smithi088 ceph-mon[36585]: pgmap v103: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-03-12T05:59:08.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:08 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:59:08.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:08 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:59:08.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:08 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:59:08.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:08 smithi076 ceph-mon[32095]: pgmap v103: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 27 KiB/s wr, 21 op/s 2024-03-12T05:59:08.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:08 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:59:08.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:08 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:59:08.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:08 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:59:10.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:10 smithi082 ceph-mon[36382]: pgmap v104: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 27 KiB/s wr, 23 op/s 2024-03-12T05:59:10.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:10 smithi088 ceph-mon[36585]: pgmap v104: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 27 KiB/s wr, 23 op/s 2024-03-12T05:59:10.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:10 smithi076 ceph-mon[32095]: pgmap v104: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 27 KiB/s wr, 23 op/s 2024-03-12T05:59:11.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:11 smithi082 ceph-mon[36382]: pgmap v105: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-12T05:59:11.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:11 smithi088 ceph-mon[36585]: pgmap v105: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-12T05:59:11.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:11 smithi076 ceph-mon[32095]: pgmap v105: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-12T05:59:14.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:14 smithi082 ceph-mon[36382]: pgmap v106: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-12T05:59:14.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:14 smithi088 ceph-mon[36585]: pgmap v106: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-12T05:59:14.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:14 smithi076 ceph-mon[32095]: pgmap v106: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-12T05:59:16.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:16 smithi082 ceph-mon[36382]: pgmap v107: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:59:16.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:16 smithi088 ceph-mon[36585]: pgmap v107: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:59:16.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:16 smithi076 ceph-mon[32095]: pgmap v107: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:59:18.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:18 smithi082 ceph-mon[36382]: pgmap v108: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:59:18.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:18 smithi088 ceph-mon[36585]: pgmap v108: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:59:18.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:18 smithi076 ceph-mon[32095]: pgmap v108: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-12T05:59:20.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:20 smithi082 ceph-mon[36382]: pgmap v109: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-12T05:59:20.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:20 smithi088 ceph-mon[36585]: pgmap v109: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-12T05:59:20.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:20 smithi076 ceph-mon[32095]: pgmap v109: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-12T05:59:22.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:22 smithi082 ceph-mon[36382]: pgmap v110: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-12T05:59:22.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:22 smithi088 ceph-mon[36585]: pgmap v110: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-12T05:59:22.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:22 smithi076 ceph-mon[32095]: pgmap v110: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-12T05:59:24.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:24 smithi082 ceph-mon[36382]: pgmap v111: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-12T05:59:24.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:24 smithi088 ceph-mon[36585]: pgmap v111: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-12T05:59:24.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:24 smithi076 ceph-mon[32095]: pgmap v111: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-12T05:59:26.347 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:59:26.348 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:59:26.378 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:26 smithi076 ceph-mon[32095]: pgmap v112: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:26.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:26 smithi082 ceph-mon[36382]: pgmap v112: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:26.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.457+0000 7f51b0d7a640 1 -- 172.21.15.76:0/1914717076 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51ac106440 msgr2=0x7f51ac10e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.457+0000 7f51b0d7a640 1 --2- 172.21.15.76:0/1914717076 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51ac106440 0x7f51ac10e010 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f519c009930 tx=0x7f519c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:59:26.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.457+0000 7f51b0d7a640 1 -- 172.21.15.76:0/1914717076 shutdown_connections 2024-03-12T05:59:26.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.457+0000 7f51b0d7a640 1 --2- 172.21.15.76:0/1914717076 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51ac106440 0x7f51ac10e010 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.457+0000 7f51b0d7a640 1 --2- 172.21.15.76:0/1914717076 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f51ac101d20 0x7f51ac105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.457+0000 7f51b0d7a640 1 --2- 172.21.15.76:0/1914717076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51ac101370 0x7f51ac101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.457+0000 7f51b0d7a640 1 -- 172.21.15.76:0/1914717076 >> 172.21.15.76:0/1914717076 conn(0x7f51ac0fcc10 msgr2=0x7f51ac0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:26.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.457+0000 7f51b0d7a640 1 -- 172.21.15.76:0/1914717076 shutdown_connections 2024-03-12T05:59:26.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.457+0000 7f51b0d7a640 1 -- 172.21.15.76:0/1914717076 wait complete. 2024-03-12T05:59:26.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51b0d7a640 1 Processor -- start 2024-03-12T05:59:26.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51b0d7a640 1 -- start start 2024-03-12T05:59:26.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51b0d7a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51ac101370 0x7f51ac19d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51b0d7a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51ac106440 0x7f51ac19d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51b0d7a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f51ac197650 0x7f51ac197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51b0d7a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac198040 con 0x7f51ac106440 2024-03-12T05:59:26.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51b0d7a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac1981b0 con 0x7f51ac197650 2024-03-12T05:59:26.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51b0d7a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ac198370 con 0x7f51ac101370 2024-03-12T05:59:26.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51a9d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51ac106440 0x7f51ac19d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51a9d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51ac106440 0x7f51ac19d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48014/0 (socket says 172.21.15.76:48014) 2024-03-12T05:59:26.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51a9d9b640 1 -- 172.21.15.76:0/1314477499 learned_addr learned my addr 172.21.15.76:0/1314477499 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:26.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51aa59c640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51ac101370 0x7f51ac19d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51aad9d640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f51ac197650 0x7f51ac197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51a9d9b640 1 -- 172.21.15.76:0/1314477499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51ac101370 msgr2=0x7f51ac19d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51a9d9b640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51ac101370 0x7f51ac19d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51a9d9b640 1 -- 172.21.15.76:0/1314477499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f51ac197650 msgr2=0x7f51ac197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.467 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51a9d9b640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f51ac197650 0x7f51ac197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51a9d9b640 1 -- 172.21.15.76:0/1314477499 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519c009560 con 0x7f51ac106440 2024-03-12T05:59:26.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.458+0000 7f51aad9d640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f51ac197650 0x7f51ac197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:59:26.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.459+0000 7f51a9d9b640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51ac106440 0x7f51ac19d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f519400e9c0 tx=0x7f519400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:26.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.459+0000 7f51a37fe640 1 -- 172.21.15.76:0/1314477499 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194009700 con 0x7f51ac106440 2024-03-12T05:59:26.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.459+0000 7f51a37fe640 1 -- 172.21.15.76:0/1314477499 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f519400cd80 con 0x7f51ac106440 2024-03-12T05:59:26.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.459+0000 7f51b0d7a640 1 -- 172.21.15.76:0/1314477499 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ac1a4310 con 0x7f51ac106440 2024-03-12T05:59:26.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.459+0000 7f51a37fe640 1 -- 172.21.15.76:0/1314477499 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5194010430 con 0x7f51ac106440 2024-03-12T05:59:26.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.459+0000 7f51b0d7a640 1 -- 172.21.15.76:0/1314477499 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ac1a4820 con 0x7f51ac106440 2024-03-12T05:59:26.468 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.461+0000 7f51a37fe640 1 -- 172.21.15.76:0/1314477499 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5194005000 con 0x7f51ac106440 2024-03-12T05:59:26.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.461+0000 7f51a37fe640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f51980a1ba0 0x7f51980a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.461+0000 7f51b3a2d740 1 Processor -- start 2024-03-12T05:59:26.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51a37fe640 1 -- 172.21.15.76:0/1314477499 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5194014070 con 0x7f51ac106440 2024-03-12T05:59:26.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51aa59c640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f51980a1ba0 0x7f51980a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51b3a2d740 1 -- start start 2024-03-12T05:59:26.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51b3a2d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7af2b0 0x55facc7af690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51b3a2d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7afbd0 0x55facc7b8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51b3a2d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7b87b0 0x55facc7baba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51b3a2d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facc71e280 con 0x55facc7afbd0 2024-03-12T05:59:26.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51b3a2d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facc71b7b0 con 0x55facc7b87b0 2024-03-12T05:59:26.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51b3a2d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facc716f90 con 0x55facc7af2b0 2024-03-12T05:59:26.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51a9d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7afbd0 0x55facc7b8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51a9d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7afbd0 0x55facc7b8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48026/0 (socket says 172.21.15.76:48026) 2024-03-12T05:59:26.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51a9d9b640 1 -- 172.21.15.76:0/3205313120 learned_addr learned my addr 172.21.15.76:0/3205313120 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:26.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51aa59c640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f51980a1ba0 0x7f51980a4060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f51a4005fd0 tx=0x7f51a4009500 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:26.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51aad9d640 1 --2- 172.21.15.76:0/3205313120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7b87b0 0x55facc7baba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51a9d9b640 1 -- 172.21.15.76:0/3205313120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7af2b0 msgr2=0x55facc7af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:26.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51a9d9b640 1 --2- 172.21.15.76:0/3205313120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7af2b0 0x55facc7af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51a9d9b640 1 -- 172.21.15.76:0/3205313120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7b87b0 msgr2=0x55facc7baba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:26.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51a9d9b640 1 --2- 172.21.15.76:0/3205313120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7b87b0 0x55facc7baba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51a9d9b640 1 -- 172.21.15.76:0/3205313120 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55facc6d5570 con 0x55facc7afbd0 2024-03-12T05:59:26.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.462+0000 7f51a9d9b640 1 --2- 172.21.15.76:0/3205313120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7afbd0 0x55facc7b8270 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f5194096d70 tx=0x7f519409a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66e06be0944fbd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:26.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.463+0000 7f51a0ff9640 1 -- 172.21.15.76:0/3205313120 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51940b2070 con 0x55facc7afbd0 2024-03-12T05:59:26.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.463+0000 7f51a0ff9640 1 -- 172.21.15.76:0/3205313120 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f519409ea60 con 0x55facc7afbd0 2024-03-12T05:59:26.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.463+0000 7f51a0ff9640 1 -- 172.21.15.76:0/3205313120 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51940cb840 con 0x55facc7afbd0 2024-03-12T05:59:26.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.463+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3205313120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7afbd0 msgr2=0x55facc7b8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.463+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3205313120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7afbd0 0x55facc7b8270 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f5194096d70 tx=0x7f519409a810 comp rx=0 tx=0).stop 2024-03-12T05:59:26.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.463+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3205313120 shutdown_connections 2024-03-12T05:59:26.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.463+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3205313120 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7b87b0 0x55facc7baba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.463+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3205313120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7afbd0 0x55facc7b8270 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.463+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3205313120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7af2b0 0x55facc7af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.463+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3205313120 >> 172.21.15.76:0/3205313120 conn(0x55facc7a9990 msgr2=0x55facc7a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:26.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.463+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3205313120 shutdown_connections 2024-03-12T05:59:26.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.463+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3205313120 wait complete. 2024-03-12T05:59:26.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.464+0000 7f51b3a2d740 1 -- start start 2024-03-12T05:59:26.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.464+0000 7f51b3a2d740 1 Processor -- start 2024-03-12T05:59:26.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.464+0000 7f51b3a2d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7af2b0 0x55facc7b5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.464+0000 7f51b3a2d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.464+0000 7f51b3a2d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7b87b0 0x55facc7b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.464+0000 7f51b3a2d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facc3ea430 con 0x55facc7b87b0 2024-03-12T05:59:26.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.464+0000 7f51b3a2d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facc3a6e00 con 0x55facc7afbd0 2024-03-12T05:59:26.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.464+0000 7f51b3a2d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facc69bfd0 con 0x55facc7af2b0 2024-03-12T05:59:26.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.464+0000 7f51b3a2d740 10 client.0 _ll_register_callbacks cb 0x55facc6f1fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:26.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.464+0000 7f51aad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7b87b0 0x55facc7b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.464+0000 7f51aad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7b87b0 0x55facc7b60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48042/0 (socket says 172.21.15.76:48042) 2024-03-12T05:59:26.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.464+0000 7f51aad9d640 1 -- 172.21.15.76:0/3977809528 learned_addr learned my addr 172.21.15.76:0/3977809528 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:26.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.464+0000 7f51a9d9b640 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.464+0000 7f51aa59c640 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7af2b0 0x55facc7b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51aad9d640 1 -- 172.21.15.76:0/3977809528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7af2b0 msgr2=0x55facc7b5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:26.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51aad9d640 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7af2b0 0x55facc7b5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51aad9d640 1 -- 172.21.15.76:0/3977809528 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 msgr2=0x55facc7b5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:26.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51aad9d640 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7b5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51aad9d640 1 -- 172.21.15.76:0/3977809528 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ac1a4820 con 0x55facc7b87b0 2024-03-12T05:59:26.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51aa59c640 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7af2b0 0x55facc7b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:26.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51a9d9b640 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:59:26.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51aad9d640 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7b87b0 0x55facc7b60a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f519c005ab0 tx=0x7f519c02ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa936181d9a49d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:26.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f5191ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:26.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f5191ffb640 1 -- 172.21.15.76:0/3977809528 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519c004450 con 0x55facc7b87b0 2024-03-12T05:59:26.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f5191ffb640 1 -- 172.21.15.76:0/3977809528 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f519c02cc00 con 0x55facc7b87b0 2024-03-12T05:59:26.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3977809528 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55facc4167e0 con 0x55facc7b87b0 2024-03-12T05:59:26.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f5191ffb640 1 -- 172.21.15.76:0/3977809528 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f519c005d50 con 0x55facc7b87b0 2024-03-12T05:59:26.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51b3a2d740 10 client.16335 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:26.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3977809528 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55facc715610 con 0x55facc7b87b0 2024-03-12T05:59:26.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f5191ffb640 1 -- 172.21.15.76:0/3977809528 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55facc4167e0 con 0x55facc7b87b0 2024-03-12T05:59:26.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f5191ffb640 1 -- 172.21.15.76:0/3977809528 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f519c03c040 con 0x55facc7b87b0 2024-03-12T05:59:26.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51b3a2d740 10 client.16335 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:26.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51b3a2d740 10 client.16335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:59:26.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51b3a2d740 20 client.16335 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:26.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55facc7c89b0 0x55facc7cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51b3a2d740 4 client.16335 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:59:26.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.465+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3977809528 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55facc453720 con 0x55facc7c89b0 2024-03-12T05:59:26.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.466+0000 7f51aa59c640 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55facc7c89b0 0x55facc7cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.466+0000 7f5191ffb640 10 client.16335 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:26.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.466+0000 7f51aa59c640 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55facc7c89b0 0x55facc7cadd0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c202585947ec93d server_cookie=6ca93a1781a688ff in_seq=0 out_seq=0 2024-03-12T05:59:26.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f5191ffb640 1 -- 172.21.15.76:0/3977809528 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55facc453720 con 0x55facc7c89b0 2024-03-12T05:59:26.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f5191ffb640 10 client.16335 handle_command_reply: tid=0 2024-03-12T05:59:26.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f51b3a2d740 1 client.16335 shutdown 2024-03-12T05:59:26.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3977809528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7b87b0 msgr2=0x55facc7b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7b87b0 0x55facc7b60a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f519c005ab0 tx=0x7f519c02ca10 comp rx=0 tx=0).stop 2024-03-12T05:59:26.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3977809528 shutdown_connections 2024-03-12T05:59:26.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55facc7c89b0 0x55facc7cadd0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7b87b0 0x55facc7b60a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f5191ffb640 0 client.16335 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:26.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3977809528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7af2b0 0x55facc7b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3977809528 >> 172.21.15.76:0/3977809528 conn(0x55facc7a9990 msgr2=0x55facc7ab730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:26.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3977809528 shutdown_connections 2024-03-12T05:59:26.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3977809528 wait complete. 2024-03-12T05:59:26.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.467+0000 7f51b3a2d740 20 client.16335 trim_cache size 0 max 16384 2024-03-12T05:59:26.489 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:26 smithi088 ceph-mon[36585]: pgmap v112: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:26.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51b3a2d740 1 Processor -- start 2024-03-12T05:59:26.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51b3a2d740 1 -- start start 2024-03-12T05:59:26.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51b3a2d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7af2b0 0x55facc7b4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51b3a2d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51b3a2d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7b87b0 0x55facc7a88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51b3a2d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facc7bafa0 con 0x55facc7af2b0 2024-03-12T05:59:26.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51b3a2d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facc453750 con 0x55facc7afbd0 2024-03-12T05:59:26.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51b3a2d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facc7aef10 con 0x55facc7b87b0 2024-03-12T05:59:26.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51aa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7af2b0 0x55facc7b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51aa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7af2b0 0x55facc7b4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48056/0 (socket says 172.21.15.76:48056) 2024-03-12T05:59:26.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51aa59c640 1 -- 172.21.15.76:0/3846482391 learned_addr learned my addr 172.21.15.76:0/3846482391 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:26.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51aad9d640 1 --2- 172.21.15.76:0/3846482391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7b87b0 0x55facc7a88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51a9d9b640 1 --2- 172.21.15.76:0/3846482391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51aa59c640 1 -- 172.21.15.76:0/3846482391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7b87b0 msgr2=0x55facc7a88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:26.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51aa59c640 1 --2- 172.21.15.76:0/3846482391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7b87b0 0x55facc7a88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51aa59c640 1 -- 172.21.15.76:0/3846482391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 msgr2=0x55facc7b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:26.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51aa59c640 1 --2- 172.21.15.76:0/3846482391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7b5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51aa59c640 1 -- 172.21.15.76:0/3846482391 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55facc6b1ac0 con 0x55facc7af2b0 2024-03-12T05:59:26.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51a9d9b640 1 --2- 172.21.15.76:0/3846482391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:59:26.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f51aa59c640 1 --2- 172.21.15.76:0/3846482391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7af2b0 0x55facc7b4780 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x55facc7cb6e0 tx=0x7f51a4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0355074979790b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:26.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.482+0000 7f5193fff640 1 -- 172.21.15.76:0/3846482391 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a4027de0 con 0x55facc7af2b0 2024-03-12T05:59:26.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.483+0000 7f5193fff640 1 -- 172.21.15.76:0/3846482391 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51a4015070 con 0x55facc7af2b0 2024-03-12T05:59:26.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.483+0000 7f5193fff640 1 -- 172.21.15.76:0/3846482391 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a4027580 con 0x55facc7af2b0 2024-03-12T05:59:26.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.483+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3846482391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7af2b0 msgr2=0x55facc7b4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.483+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3846482391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7af2b0 0x55facc7b4780 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x55facc7cb6e0 tx=0x7f51a4013000 comp rx=0 tx=0).stop 2024-03-12T05:59:26.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.483+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3846482391 shutdown_connections 2024-03-12T05:59:26.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.483+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3846482391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7b87b0 0x55facc7a88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.483+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3846482391 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.483+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3846482391 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7af2b0 0x55facc7b4780 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.483+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3846482391 >> 172.21.15.76:0/3846482391 conn(0x55facc7a9990 msgr2=0x55facc7a81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:26.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.483+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3846482391 shutdown_connections 2024-03-12T05:59:26.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.483+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3846482391 wait complete. 2024-03-12T05:59:26.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.483+0000 7f51b3a2d740 1 -- start start 2024-03-12T05:59:26.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.483+0000 7f51b3a2d740 1 Processor -- start 2024-03-12T05:59:26.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51b3a2d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7af2b0 0x55facc7be8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51b3a2d740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7bede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51b3a2d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7b87b0 0x55facc818f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51b3a2d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facc819440 con 0x55facc7af2b0 2024-03-12T05:59:26.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51b3a2d740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facc8195b0 con 0x55facc7afbd0 2024-03-12T05:59:26.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51b3a2d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facc819770 con 0x55facc7b87b0 2024-03-12T05:59:26.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51b3a2d740 10 client.0 _ll_register_callbacks cb 0x7f519400cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:26.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51aa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7af2b0 0x55facc7be8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51aa59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7af2b0 0x55facc7be8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48066/0 (socket says 172.21.15.76:48066) 2024-03-12T05:59:26.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51aa59c640 1 -- 172.21.15.76:0/3941587800 learned_addr learned my addr 172.21.15.76:0/3941587800 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:26.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51aad9d640 1 --2- 172.21.15.76:0/3941587800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7b87b0 0x55facc818f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51a9d9b640 1 --2- 172.21.15.76:0/3941587800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7bede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51aa59c640 1 -- 172.21.15.76:0/3941587800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7b87b0 msgr2=0x55facc818f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:26.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51aa59c640 1 --2- 172.21.15.76:0/3941587800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7b87b0 0x55facc818f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51aa59c640 1 -- 172.21.15.76:0/3941587800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 msgr2=0x55facc7bede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:26.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51aa59c640 1 --2- 172.21.15.76:0/3941587800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7bede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51aa59c640 1 -- 172.21.15.76:0/3941587800 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a4011520 con 0x55facc7af2b0 2024-03-12T05:59:26.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51a9d9b640 1 --2- 172.21.15.76:0/3941587800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7bede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:26.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51aa59c640 1 --2- 172.21.15.76:0/3941587800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7af2b0 0x55facc7be8a0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f51a401fe50 tx=0x7f51a401f990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84711fca2c37af1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:26.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f5191ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:26.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f5191ffb640 1 -- 172.21.15.76:0/3941587800 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a4003970 con 0x55facc7af2b0 2024-03-12T05:59:26.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f5191ffb640 1 -- 172.21.15.76:0/3941587800 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f51a4024070 con 0x55facc7af2b0 2024-03-12T05:59:26.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.484+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3941587800 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f519409ea60 con 0x55facc7af2b0 2024-03-12T05:59:26.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.485+0000 7f5191ffb640 1 -- 172.21.15.76:0/3941587800 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51a4003ad0 con 0x55facc7af2b0 2024-03-12T05:59:26.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.485+0000 7f51b3a2d740 10 client.16341 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:26.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.485+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3941587800 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55facc6d76a0 con 0x55facc7af2b0 2024-03-12T05:59:26.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.485+0000 7f5191ffb640 1 -- 172.21.15.76:0/3941587800 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f519409ea60 con 0x55facc7af2b0 2024-03-12T05:59:26.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.485+0000 7f5191ffb640 1 -- 172.21.15.76:0/3941587800 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f51a402d040 con 0x55facc7af2b0 2024-03-12T05:59:26.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.485+0000 7f51b3a2d740 10 client.16341 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:26.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.485+0000 7f51b3a2d740 10 client.16341 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:59:26.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.485+0000 7f51b3a2d740 20 client.16341 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:26.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.485+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3941587800 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55facc820d40 0x55facc823160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.485+0000 7f51b3a2d740 4 client.16341 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:59:26.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.485+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3941587800 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55facc4a2890 con 0x55facc820d40 2024-03-12T05:59:26.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.485+0000 7f51a9d9b640 1 --2- 172.21.15.76:0/3941587800 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55facc820d40 0x55facc823160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f5191ffb640 10 client.16341 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:26.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f51a9d9b640 1 --2- 172.21.15.76:0/3941587800 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55facc820d40 0x55facc823160 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=384207928cf6e4da server_cookie=a92c9045b4b7ff4f in_seq=0 out_seq=0 2024-03-12T05:59:26.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f5191ffb640 1 -- 172.21.15.76:0/3941587800 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55facc4a2890 con 0x55facc820d40 2024-03-12T05:59:26.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f5191ffb640 10 client.16341 handle_command_reply: tid=0 2024-03-12T05:59:26.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f51b3a2d740 1 client.16341 shutdown 2024-03-12T05:59:26.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3941587800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7af2b0 msgr2=0x55facc7be8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3941587800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7af2b0 0x55facc7be8a0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f51a401fe50 tx=0x7f51a401f990 comp rx=0 tx=0).stop 2024-03-12T05:59:26.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3941587800 shutdown_connections 2024-03-12T05:59:26.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3941587800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55facc7b87b0 0x55facc818f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3941587800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55facc7afbd0 0x55facc7bede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3941587800 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55facc820d40 0x55facc823160 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f51b3a2d740 1 --2- 172.21.15.76:0/3941587800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55facc7af2b0 0x55facc7be8a0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f5191ffb640 0 client.16341 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:26.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3941587800 >> 172.21.15.76:0/3941587800 conn(0x55facc7a9990 msgr2=0x55facc7a8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:26.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3941587800 shutdown_connections 2024-03-12T05:59:26.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f51b3a2d740 1 -- 172.21.15.76:0/3941587800 wait complete. 2024-03-12T05:59:26.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.486+0000 7f51b3a2d740 20 client.16341 trim_cache size 0 max 16384 2024-03-12T05:59:26.504 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:59:26.504 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T05:59:26.504 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T05:59:26.504 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:59:26.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.487+0000 7f51b0d7a640 1 -- 172.21.15.76:0/1314477499 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f51980a1ba0 msgr2=0x7f51980a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.487+0000 7f51b0d7a640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f51980a1ba0 0x7f51980a4060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f51a4005fd0 tx=0x7f51a4009500 comp rx=0 tx=0).stop 2024-03-12T05:59:26.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.487+0000 7f51b0d7a640 1 -- 172.21.15.76:0/1314477499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51ac106440 msgr2=0x7f51ac19d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.487+0000 7f51b0d7a640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51ac106440 0x7f51ac19d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f519400e9c0 tx=0x7f519400ee80 comp rx=0 tx=0).stop 2024-03-12T05:59:26.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.487+0000 7f51b0d7a640 1 -- 172.21.15.76:0/1314477499 shutdown_connections 2024-03-12T05:59:26.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.487+0000 7f51b0d7a640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f51980a1ba0 0x7f51980a4060 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.487+0000 7f51b0d7a640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f51ac197650 0x7f51ac197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.487+0000 7f51b0d7a640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51ac106440 0x7f51ac19d990 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.487+0000 7f51b0d7a640 1 --2- 172.21.15.76:0/1314477499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51ac101370 0x7f51ac19d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.487+0000 7f51b0d7a640 1 -- 172.21.15.76:0/1314477499 >> 172.21.15.76:0/1314477499 conn(0x7f51ac0fcc10 msgr2=0x7f51ac0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:26.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.487+0000 7f51b0d7a640 1 -- 172.21.15.76:0/1314477499 shutdown_connections 2024-03-12T05:59:26.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.487+0000 7f51b0d7a640 1 -- 172.21.15.76:0/1314477499 wait complete. 2024-03-12T05:59:26.507 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:59:26.507 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:59:26.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.644+0000 7fd83eb65640 1 -- 172.21.15.76:0/140681555 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd8381048d0 msgr2=0x7fd838104cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.644+0000 7fd83eb65640 1 --2- 172.21.15.76:0/140681555 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd8381048d0 0x7fd838104cb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00aa40 tx=0x7fd82c02cf50 comp rx=0 tx=0).stop 2024-03-12T05:59:26.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.644+0000 7fd83eb65640 1 -- 172.21.15.76:0/140681555 shutdown_connections 2024-03-12T05:59:26.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.644+0000 7fd83eb65640 1 --2- 172.21.15.76:0/140681555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8381003a0 0x7fd83810c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.644+0000 7fd83eb65640 1 --2- 172.21.15.76:0/140681555 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd8380ff9e0 0x7fd8380ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.644+0000 7fd83eb65640 1 --2- 172.21.15.76:0/140681555 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd8381048d0 0x7fd838104cb0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.644+0000 7fd83eb65640 1 -- 172.21.15.76:0/140681555 >> 172.21.15.76:0/140681555 conn(0x7fd838075390 msgr2=0x7fd8380757a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:26.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.644+0000 7fd83eb65640 1 -- 172.21.15.76:0/140681555 shutdown_connections 2024-03-12T05:59:26.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.644+0000 7fd83eb65640 1 -- 172.21.15.76:0/140681555 wait complete. 2024-03-12T05:59:26.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd83eb65640 1 Processor -- start 2024-03-12T05:59:26.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd83eb65640 1 -- start start 2024-03-12T05:59:26.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd83eb65640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd8380ff9e0 0x7fd83819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd83eb65640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd8381003a0 0x7fd83819dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd83eb65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8381048d0 0x7fd838197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd83eb65640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd838197ed0 con 0x7fd8381003a0 2024-03-12T05:59:26.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd83eb65640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd838198040 con 0x7fd8380ff9e0 2024-03-12T05:59:26.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd83eb65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd838198200 con 0x7fd8381048d0 2024-03-12T05:59:26.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd837fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd8381003a0 0x7fd83819dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd83c901640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd8380ff9e0 0x7fd83819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd837fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd8381003a0 0x7fd83819dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48090/0 (socket says 172.21.15.76:48090) 2024-03-12T05:59:26.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd837fff640 1 -- 172.21.15.76:0/4159821790 learned_addr learned my addr 172.21.15.76:0/4159821790 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:26.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd837fff640 1 -- 172.21.15.76:0/4159821790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8381048d0 msgr2=0x7fd838197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd83d102640 1 --2- 172.21.15.76:0/4159821790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8381048d0 0x7fd838197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd837fff640 1 --2- 172.21.15.76:0/4159821790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8381048d0 0x7fd838197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd837fff640 1 -- 172.21.15.76:0/4159821790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd8380ff9e0 msgr2=0x7fd83819d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd837fff640 1 --2- 172.21.15.76:0/4159821790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd8380ff9e0 0x7fd83819d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd837fff640 1 -- 172.21.15.76:0/4159821790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82c00a670 con 0x7fd8381003a0 2024-03-12T05:59:26.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.645+0000 7fd837fff640 1 --2- 172.21.15.76:0/4159821790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd8381003a0 0x7fd83819dbc0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd82400e9f0 tx=0x7fd82400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:26.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.646+0000 7fd835ffb640 1 -- 172.21.15.76:0/4159821790 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824009700 con 0x7fd8381003a0 2024-03-12T05:59:26.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.646+0000 7fd83eb65640 1 -- 172.21.15.76:0/4159821790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8381984e0 con 0x7fd8381003a0 2024-03-12T05:59:26.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.646+0000 7fd835ffb640 1 -- 172.21.15.76:0/4159821790 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd82400cd80 con 0x7fd8381003a0 2024-03-12T05:59:26.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.646+0000 7fd835ffb640 1 -- 172.21.15.76:0/4159821790 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824018920 con 0x7fd8381003a0 2024-03-12T05:59:26.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.646+0000 7fd83eb65640 1 -- 172.21.15.76:0/4159821790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8381a4600 con 0x7fd8381003a0 2024-03-12T05:59:26.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.647+0000 7fd83eb65640 1 -- 172.21.15.76:0/4159821790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd838100ef0 con 0x7fd8381003a0 2024-03-12T05:59:26.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.649+0000 7fd835ffb640 1 -- 172.21.15.76:0/4159821790 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd824018a80 con 0x7fd8381003a0 2024-03-12T05:59:26.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.649+0000 7fd835ffb640 1 --2- 172.21.15.76:0/4159821790 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8280a1ba0 0x7fd8280a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.649+0000 7fd83c901640 1 --2- 172.21.15.76:0/4159821790 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8280a1ba0 0x7fd8280a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.649+0000 7fd835ffb640 1 -- 172.21.15.76:0/4159821790 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd824014070 con 0x7fd8381003a0 2024-03-12T05:59:26.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.649+0000 7fd83c901640 1 --2- 172.21.15.76:0/4159821790 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8280a1ba0 0x7fd8280a4060 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00a340 tx=0x7fd82c005fd0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:26.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.653+0000 7fd835ffb640 1 -- 172.21.15.76:0/4159821790 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd824052d20 con 0x7fd8381003a0 2024-03-12T05:59:26.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.862+0000 7fd83eb65640 1 -- 172.21.15.76:0/4159821790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd838068450 con 0x7fd8381003a0 2024-03-12T05:59:26.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.862+0000 7fd835ffb640 1 -- 172.21.15.76:0/4159821790 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11332 (secure 0 0 0) 0x7fd824096ae0 con 0x7fd8381003a0 2024-03-12T05:59:26.868 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:59:26.868 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:58:43.838072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:59:26.868 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 25 2024-03-12T05:59:26.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.865+0000 7fd83eb65640 1 -- 172.21.15.76:0/4159821790 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8280a1ba0 msgr2=0x7fd8280a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.865+0000 7fd83eb65640 1 --2- 172.21.15.76:0/4159821790 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8280a1ba0 0x7fd8280a4060 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00a340 tx=0x7fd82c005fd0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.865+0000 7fd83eb65640 1 -- 172.21.15.76:0/4159821790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd8381003a0 msgr2=0x7fd83819dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.865+0000 7fd83eb65640 1 --2- 172.21.15.76:0/4159821790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd8381003a0 0x7fd83819dbc0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd82400e9f0 tx=0x7fd82400eeb0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.865+0000 7fd83eb65640 1 -- 172.21.15.76:0/4159821790 shutdown_connections 2024-03-12T05:59:26.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.865+0000 7fd83eb65640 1 --2- 172.21.15.76:0/4159821790 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8280a1ba0 0x7fd8280a4060 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.865+0000 7fd83eb65640 1 --2- 172.21.15.76:0/4159821790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8381048d0 0x7fd838197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.865+0000 7fd83eb65640 1 --2- 172.21.15.76:0/4159821790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd8381003a0 0x7fd83819dbc0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.865+0000 7fd83eb65640 1 --2- 172.21.15.76:0/4159821790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd8380ff9e0 0x7fd83819d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.865+0000 7fd83eb65640 1 -- 172.21.15.76:0/4159821790 >> 172.21.15.76:0/4159821790 conn(0x7fd838075390 msgr2=0x7fd8380fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:26.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.865+0000 7fd83eb65640 1 -- 172.21.15.76:0/4159821790 shutdown_connections 2024-03-12T05:59:26.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.865+0000 7fd83eb65640 1 -- 172.21.15.76:0/4159821790 wait complete. 2024-03-12T05:59:26.881 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:59:26.881 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:59:26.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.980+0000 7f2414779640 1 -- 172.21.15.76:0/803835987 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f240c105360 msgr2=0x7f240c10cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.980+0000 7f2414779640 1 --2- 172.21.15.76:0/803835987 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f240c105360 0x7f240c10cf30 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f2404009cd0 tx=0x7f240402c0f0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.980+0000 7f2414779640 1 -- 172.21.15.76:0/803835987 shutdown_connections 2024-03-12T05:59:26.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.980+0000 7f2414779640 1 --2- 172.21.15.76:0/803835987 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f240c105360 0x7f240c10cf30 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.980+0000 7f2414779640 1 --2- 172.21.15.76:0/803835987 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f240c100c40 0x7f240c104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.980+0000 7f2414779640 1 --2- 172.21.15.76:0/803835987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f240c100290 0x7f240c100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.980+0000 7f2414779640 1 -- 172.21.15.76:0/803835987 >> 172.21.15.76:0/803835987 conn(0x7f240c077230 msgr2=0x7f240c077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:26.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.980+0000 7f2414779640 1 -- 172.21.15.76:0/803835987 shutdown_connections 2024-03-12T05:59:26.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.980+0000 7f2414779640 1 -- 172.21.15.76:0/803835987 wait complete. 2024-03-12T05:59:26.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.981+0000 7f2414779640 1 Processor -- start 2024-03-12T05:59:26.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.981+0000 7f2414779640 1 -- start start 2024-03-12T05:59:26.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.981+0000 7f2414779640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f240c100290 0x7f240c19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.981+0000 7f2414779640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f240c105360 0x7f240c19d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.981+0000 7f2414779640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f240c1975e0 0x7f240c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.981+0000 7f2414779640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c197fd0 con 0x7f240c100290 2024-03-12T05:59:26.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.981+0000 7f2414779640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c198140 con 0x7f240c105360 2024-03-12T05:59:26.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.981+0000 7f2414779640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c198300 con 0x7f240c1975e0 2024-03-12T05:59:26.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.981+0000 7f2412515640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f240c100290 0x7f240c19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.981+0000 7f2412515640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f240c100290 0x7f240c19d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48100/0 (socket says 172.21.15.76:48100) 2024-03-12T05:59:26.989 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.981+0000 7f2412515640 1 -- 172.21.15.76:0/2563005028 learned_addr learned my addr 172.21.15.76:0/2563005028 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:26.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.981+0000 7f2412d16640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f240c1975e0 0x7f240c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.981+0000 7f2411d14640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f240c105360 0x7f240c19d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.982+0000 7f2412515640 1 -- 172.21.15.76:0/2563005028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f240c1975e0 msgr2=0x7f240c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.982+0000 7f2412515640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f240c1975e0 0x7f240c197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.982+0000 7f2412515640 1 -- 172.21.15.76:0/2563005028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f240c105360 msgr2=0x7f240c19d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:26.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.982+0000 7f2412515640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f240c105360 0x7f240c19d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:26.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.982+0000 7f2412515640 1 -- 172.21.15.76:0/2563005028 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2404009900 con 0x7f240c100290 2024-03-12T05:59:26.990 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.982+0000 7f2412d16640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f240c1975e0 0x7f240c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:26.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.982+0000 7f2412515640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f240c100290 0x7f240c19d430 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f240800d770 tx=0x7f240800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:26.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.982+0000 7f24037fe640 1 -- 172.21.15.76:0/2563005028 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408009c30 con 0x7f240c100290 2024-03-12T05:59:26.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.982+0000 7f24037fe640 1 -- 172.21.15.76:0/2563005028 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f240800de80 con 0x7f240c100290 2024-03-12T05:59:26.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.982+0000 7f2414779640 1 -- 172.21.15.76:0/2563005028 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c1a42f0 con 0x7f240c100290 2024-03-12T05:59:26.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.982+0000 7f24037fe640 1 -- 172.21.15.76:0/2563005028 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408004e80 con 0x7f240c100290 2024-03-12T05:59:26.991 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.982+0000 7f2414779640 1 -- 172.21.15.76:0/2563005028 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240c1a47b0 con 0x7f240c100290 2024-03-12T05:59:26.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.984+0000 7f24037fe640 1 -- 172.21.15.76:0/2563005028 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f2408005020 con 0x7f240c100290 2024-03-12T05:59:26.997 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.984+0000 7f241741f740 1 Processor -- start 2024-03-12T05:59:26.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f241741f740 1 -- start start 2024-03-12T05:59:26.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f241741f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fc250 0x55faf84fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f241741f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8505210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f241741f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 0x55faf8507b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f241741f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf84b7960 con 0x55faf84fcb70 2024-03-12T05:59:26.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f241741f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf846b310 con 0x55faf8505750 2024-03-12T05:59:26.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f241741f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf8468840 con 0x55faf84fc250 2024-03-12T05:59:26.998 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f24037fe640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f23f80a1ba0 0x7f23f80a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:26.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f2412515640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8505210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f2412d16640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f23f80a1ba0 0x7f23f80a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f2412515640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8505210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48104/0 (socket says 172.21.15.76:48104) 2024-03-12T05:59:26.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f2412515640 1 -- 172.21.15.76:0/3066254956 learned_addr learned my addr 172.21.15.76:0/3066254956 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:26.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f2411d14640 1 --2- 172.21.15.76:0/3066254956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 0x55faf8507b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f2412d16640 1 --2- 172.21.15.76:0/3066254956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fc250 0x55faf84fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:26.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f24037fe640 1 -- 172.21.15.76:0/2563005028 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f24080cd2d0 con 0x7f240c100290 2024-03-12T05:59:26.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f2411d14640 1 -- 172.21.15.76:0/3066254956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fc250 msgr2=0x55faf84fc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:26.999 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f2411d14640 1 --2- 172.21.15.76:0/3066254956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fc250 0x55faf84fc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f2411d14640 1 -- 172.21.15.76:0/3066254956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 msgr2=0x55faf8505210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f2411d14640 1 --2- 172.21.15.76:0/3066254956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8505210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f2411d14640 1 -- 172.21.15.76:0/3066254956 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55faf8426060 con 0x55faf8505750 2024-03-12T05:59:27.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.985+0000 7f2412515640 1 --2- 172.21.15.76:0/3066254956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8505210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:59:27.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.986+0000 7f2412d16640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f23f80a1ba0 0x7f23f80a4060 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2404006010 tx=0x7f2404032340 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.986+0000 7f2411d14640 1 --2- 172.21.15.76:0/3066254956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 0x55faf8507b40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f240c198720 tx=0x7f23fc00f360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ec801ccbebedaac server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.986+0000 7f2400ff9640 1 -- 172.21.15.76:0/3066254956 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00fd60 con 0x55faf8505750 2024-03-12T05:59:27.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.986+0000 7f2400ff9640 1 -- 172.21.15.76:0/3066254956 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23fc00fec0 con 0x55faf8505750 2024-03-12T05:59:27.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.986+0000 7f2400ff9640 1 -- 172.21.15.76:0/3066254956 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc0116a0 con 0x55faf8505750 2024-03-12T05:59:27.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 -- 172.21.15.76:0/3066254956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 msgr2=0x55faf8507b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 --2- 172.21.15.76:0/3066254956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 0x55faf8507b40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f240c198720 tx=0x7f23fc00f360 comp rx=0 tx=0).stop 2024-03-12T05:59:27.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 -- 172.21.15.76:0/3066254956 shutdown_connections 2024-03-12T05:59:27.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 --2- 172.21.15.76:0/3066254956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 0x55faf8507b40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 --2- 172.21.15.76:0/3066254956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8505210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 --2- 172.21.15.76:0/3066254956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fc250 0x55faf84fc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 -- 172.21.15.76:0/3066254956 >> 172.21.15.76:0/3066254956 conn(0x55faf84f6980 msgr2=0x55faf84f6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 -- 172.21.15.76:0/3066254956 shutdown_connections 2024-03-12T05:59:27.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 -- 172.21.15.76:0/3066254956 wait complete. 2024-03-12T05:59:27.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 -- start start 2024-03-12T05:59:27.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 Processor -- start 2024-03-12T05:59:27.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fc250 0x55faf85027b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fcb70 0x55faf8502cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 0x55faf8503230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf8464020 con 0x55faf84fc250 2024-03-12T05:59:27.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf8137430 con 0x55faf8505750 2024-03-12T05:59:27.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf80f3e00 con 0x55faf84fcb70 2024-03-12T05:59:27.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f241741f740 10 client.0 _ll_register_callbacks cb 0x55faf840bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:27.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f2412515640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fc250 0x55faf85027b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f2412515640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fc250 0x55faf85027b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48116/0 (socket says 172.21.15.76:48116) 2024-03-12T05:59:27.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f2412515640 1 -- 172.21.15.76:0/1441520265 learned_addr learned my addr 172.21.15.76:0/1441520265 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:27.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.987+0000 7f2412d16640 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 0x55faf8503230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f2411d14640 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fcb70 0x55faf8502cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f2412515640 1 -- 172.21.15.76:0/1441520265 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fcb70 msgr2=0x55faf8502cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f2412515640 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fcb70 0x55faf8502cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f2412515640 1 -- 172.21.15.76:0/1441520265 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 msgr2=0x55faf8503230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f2412515640 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 0x55faf8503230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f2412515640 1 -- 172.21.15.76:0/1441520265 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23fc00b790 con 0x55faf84fc250 2024-03-12T05:59:27.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f2411d14640 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fcb70 0x55faf8502cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:27.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f2412d16640 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 0x55faf8503230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:59:27.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f2412515640 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fc250 0x55faf85027b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f2408009610 tx=0x7f2408069310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383637a9fe28d219 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f23f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:27.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f23f5ffb640 1 -- 172.21.15.76:0/1441520265 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240809a630 con 0x55faf84fc250 2024-03-12T05:59:27.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f23f5ffb640 1 -- 172.21.15.76:0/1441520265 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2408099f30 con 0x55faf84fc250 2024-03-12T05:59:27.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f241741f740 1 -- 172.21.15.76:0/1441520265 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55faf81a0720 con 0x55faf84fc250 2024-03-12T05:59:27.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f23f5ffb640 1 -- 172.21.15.76:0/1441520265 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2408014020 con 0x55faf84fc250 2024-03-12T05:59:27.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f241741f740 10 client.16359 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:27.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f241741f740 1 -- 172.21.15.76:0/1441520265 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55faf84626a0 con 0x55faf84fc250 2024-03-12T05:59:27.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.988+0000 7f23f5ffb640 1 -- 172.21.15.76:0/1441520265 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24080b1080 con 0x55faf84fc250 2024-03-12T05:59:27.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.989+0000 7f23f5ffb640 1 -- 172.21.15.76:0/1441520265 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f2408099380 con 0x55faf84fc250 2024-03-12T05:59:27.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.989+0000 7f241741f740 10 client.16359 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:27.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.989+0000 7f241741f740 10 client.16359 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:59:27.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.989+0000 7f241741f740 20 client.16359 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:27.005 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.989+0000 7f241741f740 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55faf8515c30 0x55faf8518050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.989+0000 7f241741f740 4 client.16359 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:59:27.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.989+0000 7f241741f740 1 -- 172.21.15.76:0/1441520265 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55faf81ef890 con 0x55faf8515c30 2024-03-12T05:59:27.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.989+0000 7f2411d14640 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55faf8515c30 0x55faf8518050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.989+0000 7f2411d14640 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55faf8515c30 0x55faf8518050 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ecc6f96f432bac18 server_cookie=fe28e3d5646bf30e in_seq=0 out_seq=0 2024-03-12T05:59:27.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.989+0000 7f23f5ffb640 10 client.16359 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:59:27.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f23f5ffb640 1 -- 172.21.15.76:0/1441520265 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55faf81ef890 con 0x55faf8515c30 2024-03-12T05:59:27.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f23f5ffb640 10 client.16359 handle_command_reply: tid=0 2024-03-12T05:59:27.006 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f241741f740 1 client.16359 shutdown 2024-03-12T05:59:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f241741f740 1 -- 172.21.15.76:0/1441520265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fc250 msgr2=0x55faf85027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f241741f740 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fc250 0x55faf85027b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f2408009610 tx=0x7f2408069310 comp rx=0 tx=0).stop 2024-03-12T05:59:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f241741f740 1 -- 172.21.15.76:0/1441520265 shutdown_connections 2024-03-12T05:59:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f241741f740 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55faf8515c30 0x55faf8518050 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f241741f740 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 0x55faf8503230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f241741f740 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fcb70 0x55faf8502cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f23f5ffb640 0 client.16359 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:59:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f241741f740 1 --2- 172.21.15.76:0/1441520265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fc250 0x55faf85027b0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.007 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f241741f740 1 -- 172.21.15.76:0/1441520265 >> 172.21.15.76:0/1441520265 conn(0x55faf84f6980 msgr2=0x55faf84f86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f241741f740 1 -- 172.21.15.76:0/1441520265 shutdown_connections 2024-03-12T05:59:27.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f241741f740 1 -- 172.21.15.76:0/1441520265 wait complete. 2024-03-12T05:59:27.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:26.990+0000 7f241741f740 20 client.16359 trim_cache size 0 max 16384 2024-03-12T05:59:27.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f241741f740 1 Processor -- start 2024-03-12T05:59:27.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f241741f740 1 -- start start 2024-03-12T05:59:27.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f241741f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fc250 0x55faf8504ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f241741f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8503050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f241741f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 0x55faf84ffa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f241741f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf83e9130 con 0x55faf84fcb70 2024-03-12T05:59:27.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f241741f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf84f2080 con 0x55faf8505750 2024-03-12T05:59:27.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f241741f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf8507e70 con 0x55faf84fc250 2024-03-12T05:59:27.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f2411d14640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8503050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f2411d14640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8503050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48118/0 (socket says 172.21.15.76:48118) 2024-03-12T05:59:27.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f2411d14640 1 -- 172.21.15.76:0/1519388007 learned_addr learned my addr 172.21.15.76:0/1519388007 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:27.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f2412515640 1 --2- 172.21.15.76:0/1519388007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fc250 0x55faf8504ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f2411d14640 1 -- 172.21.15.76:0/1519388007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fc250 msgr2=0x55faf8504ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f2411d14640 1 --2- 172.21.15.76:0/1519388007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fc250 0x55faf8504ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f2412d16640 1 --2- 172.21.15.76:0/1519388007 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 0x55faf84ffa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f2411d14640 1 -- 172.21.15.76:0/1519388007 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 msgr2=0x55faf84ffa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f2411d14640 1 --2- 172.21.15.76:0/1519388007 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 0x55faf84ffa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f2411d14640 1 -- 172.21.15.76:0/1519388007 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55faf83fec20 con 0x55faf84fcb70 2024-03-12T05:59:27.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f2411d14640 1 --2- 172.21.15.76:0/1519388007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8503050 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x55faf84f7b10 tx=0x7f23fc00b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6734c96bd893795 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.002+0000 7f23f7fff640 1 -- 172.21.15.76:0/1519388007 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00fd60 con 0x55faf84fcb70 2024-03-12T05:59:27.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.003+0000 7f23f7fff640 1 -- 172.21.15.76:0/1519388007 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23fc021040 con 0x55faf84fcb70 2024-03-12T05:59:27.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.003+0000 7f23f7fff640 1 -- 172.21.15.76:0/1519388007 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc0040c0 con 0x55faf84fcb70 2024-03-12T05:59:27.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.003+0000 7f241741f740 1 -- 172.21.15.76:0/1519388007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 msgr2=0x55faf8503050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.003+0000 7f241741f740 1 --2- 172.21.15.76:0/1519388007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8503050 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x55faf84f7b10 tx=0x7f23fc00b4d0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.003+0000 7f241741f740 1 -- 172.21.15.76:0/1519388007 shutdown_connections 2024-03-12T05:59:27.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.003+0000 7f241741f740 1 --2- 172.21.15.76:0/1519388007 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf8505750 0x55faf84ffa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.003+0000 7f241741f740 1 --2- 172.21.15.76:0/1519388007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8503050 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.003+0000 7f241741f740 1 --2- 172.21.15.76:0/1519388007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf84fc250 0x55faf8504ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.003+0000 7f241741f740 1 -- 172.21.15.76:0/1519388007 >> 172.21.15.76:0/1519388007 conn(0x55faf84f6980 msgr2=0x55faf851c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.003+0000 7f241741f740 1 -- 172.21.15.76:0/1519388007 shutdown_connections 2024-03-12T05:59:27.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.003+0000 7f241741f740 1 -- 172.21.15.76:0/1519388007 wait complete. 2024-03-12T05:59:27.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.003+0000 7f241741f740 1 -- start start 2024-03-12T05:59:27.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.003+0000 7f241741f740 1 Processor -- start 2024-03-12T05:59:27.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f241741f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf84fc250 0x55faf8565410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f241741f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8565950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f241741f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf8505750 0x55faf850b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f241741f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf850be60 con 0x55faf84fcb70 2024-03-12T05:59:27.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f241741f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf850bfd0 con 0x55faf84fc250 2024-03-12T05:59:27.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f241741f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faf850c190 con 0x55faf8505750 2024-03-12T05:59:27.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f241741f740 10 client.0 _ll_register_callbacks cb 0x7f23fc00b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:27.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f2411d14640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8565950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f2411d14640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8565950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48126/0 (socket says 172.21.15.76:48126) 2024-03-12T05:59:27.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f2411d14640 1 -- 172.21.15.76:0/503422938 learned_addr learned my addr 172.21.15.76:0/503422938 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:27.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f2412515640 1 --2- 172.21.15.76:0/503422938 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf84fc250 0x55faf8565410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f2412d16640 1 --2- 172.21.15.76:0/503422938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf8505750 0x55faf850b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f2411d14640 1 -- 172.21.15.76:0/503422938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf8505750 msgr2=0x55faf850b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f2411d14640 1 --2- 172.21.15.76:0/503422938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf8505750 0x55faf850b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f2411d14640 1 -- 172.21.15.76:0/503422938 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf84fc250 msgr2=0x55faf8565410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f2411d14640 1 --2- 172.21.15.76:0/503422938 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf84fc250 0x55faf8565410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f2411d14640 1 -- 172.21.15.76:0/503422938 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55faf8426060 con 0x55faf84fcb70 2024-03-12T05:59:27.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f2411d14640 1 --2- 172.21.15.76:0/503422938 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8565950 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f23fc003a20 tx=0x7f23fc003b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f28aa8d311f1b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f23f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f23f5ffb640 1 -- 172.21.15.76:0/503422938 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc037070 con 0x55faf84fcb70 2024-03-12T05:59:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f23f5ffb640 1 -- 172.21.15.76:0/503422938 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f23fc005050 con 0x55faf84fcb70 2024-03-12T05:59:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.004+0000 7f241741f740 1 -- 172.21.15.76:0/503422938 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55faf81637e0 con 0x55faf84fcb70 2024-03-12T05:59:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.005+0000 7f23f5ffb640 1 -- 172.21.15.76:0/503422938 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23fc01e070 con 0x55faf84fcb70 2024-03-12T05:59:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.005+0000 7f241741f740 10 client.16365 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.005+0000 7f241741f740 1 -- 172.21.15.76:0/503422938 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55faf84248c0 con 0x55faf84fcb70 2024-03-12T05:59:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.005+0000 7f23f5ffb640 1 -- 172.21.15.76:0/503422938 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55faf81637e0 con 0x55faf84fcb70 2024-03-12T05:59:27.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.005+0000 7f23f5ffb640 1 -- 172.21.15.76:0/503422938 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f23fc03a030 con 0x55faf84fcb70 2024-03-12T05:59:27.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.005+0000 7f241741f740 10 client.16365 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:27.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.005+0000 7f241741f740 10 client.16365 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:59:27.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.005+0000 7f241741f740 20 client.16365 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:27.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.005+0000 7f241741f740 1 --2- 172.21.15.76:0/503422938 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55faf856dff0 0x55faf8570410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.005+0000 7f241741f740 4 client.16365 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:59:27.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.005+0000 7f241741f740 1 -- 172.21.15.76:0/503422938 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55faf810f700 con 0x55faf856dff0 2024-03-12T05:59:27.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.005+0000 7f2412515640 1 --2- 172.21.15.76:0/503422938 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55faf856dff0 0x55faf8570410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.005+0000 7f23f5ffb640 10 client.16365 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:59:27.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.005+0000 7f2412515640 1 --2- 172.21.15.76:0/503422938 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55faf856dff0 0x55faf8570410 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9c43d98663ef2036 server_cookie=6b58a930e636b185 in_seq=0 out_seq=0 2024-03-12T05:59:27.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f23f5ffb640 1 -- 172.21.15.76:0/503422938 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55faf810f700 con 0x55faf856dff0 2024-03-12T05:59:27.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f23f5ffb640 10 client.16365 handle_command_reply: tid=0 2024-03-12T05:59:27.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f241741f740 1 client.16365 shutdown 2024-03-12T05:59:27.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f241741f740 1 -- 172.21.15.76:0/503422938 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 msgr2=0x55faf8565950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f241741f740 1 --2- 172.21.15.76:0/503422938 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8565950 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f23fc003a20 tx=0x7f23fc003b70 comp rx=0 tx=0).stop 2024-03-12T05:59:27.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f241741f740 1 -- 172.21.15.76:0/503422938 shutdown_connections 2024-03-12T05:59:27.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f241741f740 1 --2- 172.21.15.76:0/503422938 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55faf856dff0 0x55faf8570410 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f241741f740 1 --2- 172.21.15.76:0/503422938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55faf8505750 0x55faf850b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f241741f740 1 --2- 172.21.15.76:0/503422938 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55faf84fcb70 0x55faf8565950 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f23f5ffb640 0 client.16365 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:59:27.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f241741f740 1 --2- 172.21.15.76:0/503422938 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55faf84fc250 0x55faf8565410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f241741f740 1 -- 172.21.15.76:0/503422938 >> 172.21.15.76:0/503422938 conn(0x55faf84f6980 msgr2=0x55faf84f52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f241741f740 1 -- 172.21.15.76:0/503422938 shutdown_connections 2024-03-12T05:59:27.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f241741f740 1 -- 172.21.15.76:0/503422938 wait complete. 2024-03-12T05:59:27.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f241741f740 20 client.16365 trim_cache size 0 max 16384 2024-03-12T05:59:27.024 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:59:27.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f2414779640 1 -- 172.21.15.76:0/2563005028 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f23f80a1ba0 msgr2=0x7f23f80a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f2414779640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f23f80a1ba0 0x7f23f80a4060 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2404006010 tx=0x7f2404032340 comp rx=0 tx=0).stop 2024-03-12T05:59:27.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f2414779640 1 -- 172.21.15.76:0/2563005028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f240c100290 msgr2=0x7f240c19d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f2414779640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f240c100290 0x7f240c19d430 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f240800d770 tx=0x7f240800dc30 comp rx=0 tx=0).stop 2024-03-12T05:59:27.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f2414779640 1 -- 172.21.15.76:0/2563005028 shutdown_connections 2024-03-12T05:59:27.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f2414779640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f23f80a1ba0 0x7f23f80a4060 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f2414779640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f240c1975e0 0x7f240c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f2414779640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f240c105360 0x7f240c19d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f2414779640 1 --2- 172.21.15.76:0/2563005028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f240c100290 0x7f240c19d430 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f2414779640 1 -- 172.21.15.76:0/2563005028 >> 172.21.15.76:0/2563005028 conn(0x7f240c077230 msgr2=0x7f240c068790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f2414779640 1 -- 172.21.15.76:0/2563005028 shutdown_connections 2024-03-12T05:59:27.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.006+0000 7f2414779640 1 -- 172.21.15.76:0/2563005028 wait complete. 2024-03-12T05:59:27.026 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:59:27.027 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:59:27.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.127+0000 7fcf5d2b5640 1 -- 172.21.15.76:0/3020645487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf58106460 msgr2=0x7fcf5810e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.127+0000 7fcf5d2b5640 1 --2- 172.21.15.76:0/3020645487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf58106460 0x7fcf5810e030 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009cd0 tx=0x7fcf5002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.127+0000 7fcf5d2b5640 1 -- 172.21.15.76:0/3020645487 shutdown_connections 2024-03-12T05:59:27.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.127+0000 7fcf5d2b5640 1 --2- 172.21.15.76:0/3020645487 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf58106460 0x7fcf5810e030 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.127+0000 7fcf5d2b5640 1 --2- 172.21.15.76:0/3020645487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf58101d40 0x7fcf58105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.127+0000 7fcf5d2b5640 1 --2- 172.21.15.76:0/3020645487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf58101390 0x7fcf58101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.127+0000 7fcf5d2b5640 1 -- 172.21.15.76:0/3020645487 >> 172.21.15.76:0/3020645487 conn(0x7fcf580fcc10 msgr2=0x7fcf580ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.127+0000 7fcf5d2b5640 1 -- 172.21.15.76:0/3020645487 shutdown_connections 2024-03-12T05:59:27.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.127+0000 7fcf5d2b5640 1 -- 172.21.15.76:0/3020645487 wait complete. 2024-03-12T05:59:27.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.127+0000 7fcf5d2b5640 1 Processor -- start 2024-03-12T05:59:27.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf5d2b5640 1 -- start start 2024-03-12T05:59:27.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf5d2b5640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf58101390 0x7fcf5819d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf5d2b5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf58106460 0x7fcf5819d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf5d2b5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf581975d0 0x7fcf58197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf5d2b5640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf58197fc0 con 0x7fcf58106460 2024-03-12T05:59:27.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf5d2b5640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf58198130 con 0x7fcf58101390 2024-03-12T05:59:27.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf5d2b5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf581982f0 con 0x7fcf581975d0 2024-03-12T05:59:27.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf567fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf58106460 0x7fcf5819d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf567fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf58106460 0x7fcf5819d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48148/0 (socket says 172.21.15.76:48148) 2024-03-12T05:59:27.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf567fc640 1 -- 172.21.15.76:0/2474697447 learned_addr learned my addr 172.21.15.76:0/2474697447 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:27.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf56ffd640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf58101390 0x7fcf5819d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf577fe640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf581975d0 0x7fcf58197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf567fc640 1 -- 172.21.15.76:0/2474697447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf581975d0 msgr2=0x7fcf58197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.136 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf567fc640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf581975d0 0x7fcf58197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf567fc640 1 -- 172.21.15.76:0/2474697447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf58101390 msgr2=0x7fcf5819d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf567fc640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf58101390 0x7fcf5819d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf567fc640 1 -- 172.21.15.76:0/2474697447 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf50009900 con 0x7fcf58106460 2024-03-12T05:59:27.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.128+0000 7fcf56ffd640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf58101390 0x7fcf5819d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:59:27.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.129+0000 7fcf567fc640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf58106460 0x7fcf5819d960 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00e9f0 tx=0x7fcf4c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.130+0000 7fcf3ffff640 1 -- 172.21.15.76:0/2474697447 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c0096c0 con 0x7fcf58106460 2024-03-12T05:59:27.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.130+0000 7fcf3ffff640 1 -- 172.21.15.76:0/2474697447 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf4c005500 con 0x7fcf58106460 2024-03-12T05:59:27.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.130+0000 7fcf5d2b5640 1 -- 172.21.15.76:0/2474697447 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf581a42e0 con 0x7fcf58106460 2024-03-12T05:59:27.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.130+0000 7fcf3ffff640 1 -- 172.21.15.76:0/2474697447 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c010430 con 0x7fcf58106460 2024-03-12T05:59:27.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.130+0000 7fcf5d2b5640 1 -- 172.21.15.76:0/2474697447 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf581a47f0 con 0x7fcf58106460 2024-03-12T05:59:27.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.132+0000 7fcf3ffff640 1 -- 172.21.15.76:0/2474697447 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fcf4c005020 con 0x7fcf58106460 2024-03-12T05:59:27.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.132+0000 7fcf3ffff640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcf400a1c20 0x7fcf400a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf5ff63740 1 Processor -- start 2024-03-12T05:59:27.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf5ff63740 1 -- start start 2024-03-12T05:59:27.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf56ffd640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcf400a1c20 0x7fcf400a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf3ffff640 1 -- 172.21.15.76:0/2474697447 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcf4c014070 con 0x7fcf58106460 2024-03-12T05:59:27.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf5ff63740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d406894c30 0x55d406895010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf5ff63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 0x55d4068a2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf5ff63740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d4068a2b60 0x55d4068a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf5ff63740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d406850960 con 0x55d4068a2b60 2024-03-12T05:59:27.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf5ff63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d406804310 con 0x55d4068955e0 2024-03-12T05:59:27.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf5ff63740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d406801840 con 0x55d406894c30 2024-03-12T05:59:27.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf577fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d4068a2b60 0x55d4068a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf577fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d4068a2b60 0x55d4068a4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48162/0 (socket says 172.21.15.76:48162) 2024-03-12T05:59:27.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf577fe640 1 -- 172.21.15.76:0/2727764030 learned_addr learned my addr 172.21.15.76:0/2727764030 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:27.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf56ffd640 1 --2- 172.21.15.76:0/2727764030 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d406894c30 0x55d406895010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf567fc640 1 --2- 172.21.15.76:0/2727764030 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 0x55d4068a2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf56ffd640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcf400a1c20 0x7fcf400a40e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcf48011490 tx=0x7fcf48005f90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf577fe640 1 -- 172.21.15.76:0/2727764030 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d406894c30 msgr2=0x55d406895010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf577fe640 1 --2- 172.21.15.76:0/2727764030 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d406894c30 0x55d406895010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf577fe640 1 -- 172.21.15.76:0/2727764030 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 msgr2=0x55d4068a2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf577fe640 1 --2- 172.21.15.76:0/2727764030 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 0x55d4068a2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf577fe640 1 -- 172.21.15.76:0/2727764030 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4067bf060 con 0x55d4068a2b60 2024-03-12T05:59:27.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.133+0000 7fcf577fe640 1 --2- 172.21.15.76:0/2727764030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d4068a2b60 0x55d4068a4f50 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fcf5002c700 tx=0x7fcf50032d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c2ab33d1e49639 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.134+0000 7fcf3d7fa640 1 -- 172.21.15.76:0/2727764030 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf50034440 con 0x55d4068a2b60 2024-03-12T05:59:27.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.134+0000 7fcf3d7fa640 1 -- 172.21.15.76:0/2727764030 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf500345a0 con 0x55d4068a2b60 2024-03-12T05:59:27.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.134+0000 7fcf3d7fa640 1 -- 172.21.15.76:0/2727764030 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf50003b70 con 0x55d4068a2b60 2024-03-12T05:59:27.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.134+0000 7fcf5ff63740 1 -- 172.21.15.76:0/2727764030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d4068a2b60 msgr2=0x55d4068a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.134+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/2727764030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d4068a2b60 0x55d4068a4f50 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fcf5002c700 tx=0x7fcf50032d70 comp rx=0 tx=0).stop 2024-03-12T05:59:27.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.134+0000 7fcf5ff63740 1 -- 172.21.15.76:0/2727764030 shutdown_connections 2024-03-12T05:59:27.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.134+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/2727764030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d4068a2b60 0x55d4068a4f50 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.134+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/2727764030 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 0x55d4068a2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.134+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/2727764030 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d406894c30 0x55d406895010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.134+0000 7fcf5ff63740 1 -- 172.21.15.76:0/2727764030 >> 172.21.15.76:0/2727764030 conn(0x55d40688f980 msgr2=0x55d40688fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.134+0000 7fcf5ff63740 1 -- 172.21.15.76:0/2727764030 shutdown_connections 2024-03-12T05:59:27.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.134+0000 7fcf5ff63740 1 -- 172.21.15.76:0/2727764030 wait complete. 2024-03-12T05:59:27.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf5ff63740 1 -- start start 2024-03-12T05:59:27.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf5ff63740 1 Processor -- start 2024-03-12T05:59:27.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf5ff63740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d406894c30 0x55d40689fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf5ff63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 0x55d4068a0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf5ff63740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4068a2b60 0x55d4068a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf5ff63740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4067fd020 con 0x55d406894c30 2024-03-12T05:59:27.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf5ff63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4064d0430 con 0x55d4068955e0 2024-03-12T05:59:27.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf5ff63740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d40648ce00 con 0x55d4068a2b60 2024-03-12T05:59:27.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf5ff63740 10 client.0 _ll_register_callbacks cb 0x55d4067a4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:27.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf56ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d406894c30 0x55d40689fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf56ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d406894c30 0x55d40689fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48164/0 (socket says 172.21.15.76:48164) 2024-03-12T05:59:27.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf56ffd640 1 -- 172.21.15.76:0/3062510228 learned_addr learned my addr 172.21.15.76:0/3062510228 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:27.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf567fc640 1 --2- 172.21.15.76:0/3062510228 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 0x55d4068a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf577fe640 1 --2- 172.21.15.76:0/3062510228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4068a2b60 0x55d4068a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf56ffd640 1 -- 172.21.15.76:0/3062510228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4068a2b60 msgr2=0x55d4068a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf56ffd640 1 --2- 172.21.15.76:0/3062510228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4068a2b60 0x55d4068a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf56ffd640 1 -- 172.21.15.76:0/3062510228 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 msgr2=0x55d4068a0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf56ffd640 1 --2- 172.21.15.76:0/3062510228 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 0x55d4068a0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf56ffd640 1 -- 172.21.15.76:0/3062510228 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5810e840 con 0x55d406894c30 2024-03-12T05:59:27.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.135+0000 7fcf577fe640 1 --2- 172.21.15.76:0/3062510228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4068a2b60 0x55d4068a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:59:27.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf56ffd640 1 --2- 172.21.15.76:0/3062510228 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d406894c30 0x55d40689fb20 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fcf48003740 tx=0x7fcf48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0a3e04a2b4e913 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf3aff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:27.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf3aff5640 1 -- 172.21.15.76:0/3062510228 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf48023070 con 0x55d406894c30 2024-03-12T05:59:27.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf3aff5640 1 -- 172.21.15.76:0/3062510228 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcf4800f970 con 0x55d406894c30 2024-03-12T05:59:27.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf5ff63740 1 -- 172.21.15.76:0/3062510228 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4064fc7e0 con 0x55d406894c30 2024-03-12T05:59:27.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf3aff5640 1 -- 172.21.15.76:0/3062510228 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf4800fb30 con 0x55d406894c30 2024-03-12T05:59:27.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf5ff63740 10 client.16377 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:27.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf5ff63740 1 -- 172.21.15.76:0/3062510228 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d4067fb6a0 con 0x55d406894c30 2024-03-12T05:59:27.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf3aff5640 1 -- 172.21.15.76:0/3062510228 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d4064fc7e0 con 0x55d406894c30 2024-03-12T05:59:27.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf3aff5640 1 -- 172.21.15.76:0/3062510228 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fcf4801b410 con 0x55d406894c30 2024-03-12T05:59:27.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf5ff63740 10 client.16377 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:27.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf5ff63740 10 client.16377 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:59:27.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf5ff63740 20 client.16377 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:27.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/3062510228 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d4068b2fc0 0x55d4068b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf5ff63740 4 client.16377 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:59:27.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.136+0000 7fcf5ff63740 1 -- 172.21.15.76:0/3062510228 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d406539720 con 0x55d4068b2fc0 2024-03-12T05:59:27.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.137+0000 7fcf567fc640 1 --2- 172.21.15.76:0/3062510228 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d4068b2fc0 0x55d4068b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.137+0000 7fcf3aff5640 10 client.16377 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:59:27.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.137+0000 7fcf567fc640 1 --2- 172.21.15.76:0/3062510228 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d4068b2fc0 0x55d4068b53e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=df00e4bbeb51c794 server_cookie=e58bbb38db35c56c in_seq=0 out_seq=0 2024-03-12T05:59:27.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf3aff5640 1 -- 172.21.15.76:0/3062510228 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d406539720 con 0x55d4068b2fc0 2024-03-12T05:59:27.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf3aff5640 10 client.16377 handle_command_reply: tid=0 2024-03-12T05:59:27.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf5ff63740 1 client.16377 shutdown 2024-03-12T05:59:27.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf5ff63740 1 -- 172.21.15.76:0/3062510228 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d406894c30 msgr2=0x55d40689fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/3062510228 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d406894c30 0x55d40689fb20 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fcf48003740 tx=0x7fcf48004730 comp rx=0 tx=0).stop 2024-03-12T05:59:27.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf5ff63740 1 -- 172.21.15.76:0/3062510228 shutdown_connections 2024-03-12T05:59:27.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/3062510228 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d4068b2fc0 0x55d4068b53e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/3062510228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4068a2b60 0x55d4068a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf3aff5640 0 client.16377 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:59:27.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/3062510228 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 0x55d4068a0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/3062510228 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d406894c30 0x55d40689fb20 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf5ff63740 1 -- 172.21.15.76:0/3062510228 >> 172.21.15.76:0/3062510228 conn(0x55d40688f980 msgr2=0x55d4068916d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf5ff63740 1 -- 172.21.15.76:0/3062510228 shutdown_connections 2024-03-12T05:59:27.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf5ff63740 1 -- 172.21.15.76:0/3062510228 wait complete. 2024-03-12T05:59:27.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.138+0000 7fcf5ff63740 20 client.16377 trim_cache size 0 max 16384 2024-03-12T05:59:27.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.149+0000 7fcf5ff63740 1 Processor -- start 2024-03-12T05:59:27.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf5ff63740 1 -- start start 2024-03-12T05:59:27.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf5ff63740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d406894c30 0x55d4068a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf5ff63740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d4068955e0 0x55d4068914e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf5ff63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068a2b60 0x55d4068a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf5ff63740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d406782130 con 0x55d4068955e0 2024-03-12T05:59:27.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf5ff63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d406890780 con 0x55d4068a2b60 2024-03-12T05:59:27.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf5ff63740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d406890980 con 0x55d406894c30 2024-03-12T05:59:27.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf567fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d4068955e0 0x55d4068914e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf567fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d4068955e0 0x55d4068914e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48176/0 (socket says 172.21.15.76:48176) 2024-03-12T05:59:27.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf567fc640 1 -- 172.21.15.76:0/471254199 learned_addr learned my addr 172.21.15.76:0/471254199 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:27.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf56ffd640 1 --2- 172.21.15.76:0/471254199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d406894c30 0x55d4068a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf577fe640 1 --2- 172.21.15.76:0/471254199 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068a2b60 0x55d4068a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf567fc640 1 -- 172.21.15.76:0/471254199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d406894c30 msgr2=0x55d4068a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf567fc640 1 --2- 172.21.15.76:0/471254199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d406894c30 0x55d4068a03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf567fc640 1 -- 172.21.15.76:0/471254199 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068a2b60 msgr2=0x55d4068a99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf567fc640 1 --2- 172.21.15.76:0/471254199 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068a2b60 0x55d4068a99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf567fc640 1 -- 172.21.15.76:0/471254199 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d406797c20 con 0x55d4068955e0 2024-03-12T05:59:27.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf577fe640 1 --2- 172.21.15.76:0/471254199 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068a2b60 0x55d4068a99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:59:27.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf567fc640 1 --2- 172.21.15.76:0/471254199 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d4068955e0 0x55d4068914e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c0cb230 tx=0x7fcf4c0cd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d934024c68c4dc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf3cff9640 1 -- 172.21.15.76:0/471254199 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c09e800 con 0x55d4068955e0 2024-03-12T05:59:27.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf3cff9640 1 -- 172.21.15.76:0/471254199 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf4c0a2e50 con 0x55d4068955e0 2024-03-12T05:59:27.159 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.150+0000 7fcf3cff9640 1 -- 172.21.15.76:0/471254199 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c098ad0 con 0x55d4068955e0 2024-03-12T05:59:27.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.151+0000 7fcf5ff63740 1 -- 172.21.15.76:0/471254199 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d4068955e0 msgr2=0x55d4068914e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.151+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/471254199 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d4068955e0 0x55d4068914e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c0cb230 tx=0x7fcf4c0cd040 comp rx=0 tx=0).stop 2024-03-12T05:59:27.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.151+0000 7fcf5ff63740 1 -- 172.21.15.76:0/471254199 shutdown_connections 2024-03-12T05:59:27.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.151+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/471254199 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068a2b60 0x55d4068a99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.151+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/471254199 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d4068955e0 0x55d4068914e0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.151+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/471254199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d406894c30 0x55d4068a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.151+0000 7fcf5ff63740 1 -- 172.21.15.76:0/471254199 >> 172.21.15.76:0/471254199 conn(0x55d40688f980 msgr2=0x55d406890dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.151+0000 7fcf5ff63740 1 -- 172.21.15.76:0/471254199 shutdown_connections 2024-03-12T05:59:27.160 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.151+0000 7fcf5ff63740 1 -- 172.21.15.76:0/471254199 wait complete. 2024-03-12T05:59:27.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.151+0000 7fcf5ff63740 1 -- start start 2024-03-12T05:59:27.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.151+0000 7fcf5ff63740 1 Processor -- start 2024-03-12T05:59:27.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf5ff63740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d406894c30 0x55d4068a8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.164 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf5ff63740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 0x55d4068a9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf5ff63740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4068a2b60 0x55d4069094f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf5ff63740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4068a97e0 con 0x55d406894c30 2024-03-12T05:59:27.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf5ff63740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d406909a60 con 0x55d4068955e0 2024-03-12T05:59:27.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf5ff63740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d406909c20 con 0x55d4068a2b60 2024-03-12T05:59:27.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf5ff63740 10 client.0 _ll_register_callbacks cb 0x7fcf48005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:27.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf56ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d406894c30 0x55d4068a8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf56ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d406894c30 0x55d4068a8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48178/0 (socket says 172.21.15.76:48178) 2024-03-12T05:59:27.165 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf56ffd640 1 -- 172.21.15.76:0/1405700204 learned_addr learned my addr 172.21.15.76:0/1405700204 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:27.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf567fc640 1 --2- 172.21.15.76:0/1405700204 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 0x55d4068a9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf577fe640 1 --2- 172.21.15.76:0/1405700204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4068a2b60 0x55d4069094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf56ffd640 1 -- 172.21.15.76:0/1405700204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4068a2b60 msgr2=0x55d4069094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf56ffd640 1 --2- 172.21.15.76:0/1405700204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4068a2b60 0x55d4069094f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf56ffd640 1 -- 172.21.15.76:0/1405700204 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 msgr2=0x55d4068a9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf56ffd640 1 --2- 172.21.15.76:0/1405700204 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 0x55d4068a9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf56ffd640 1 -- 172.21.15.76:0/1405700204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf581a47f0 con 0x55d406894c30 2024-03-12T05:59:27.166 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf56ffd640 1 --2- 172.21.15.76:0/1405700204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d406894c30 0x55d4068a8b10 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009ed0 tx=0x7fcf480035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d695a1a89904d0a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf3aff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:27.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf3aff5640 1 -- 172.21.15.76:0/1405700204 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf48023070 con 0x55d406894c30 2024-03-12T05:59:27.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf3aff5640 1 -- 172.21.15.76:0/1405700204 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcf4801b410 con 0x55d406894c30 2024-03-12T05:59:27.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf5ff63740 1 -- 172.21.15.76:0/1405700204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf500345a0 con 0x55d406894c30 2024-03-12T05:59:27.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf3aff5640 1 -- 172.21.15.76:0/1405700204 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf4801b5d0 con 0x55d406894c30 2024-03-12T05:59:27.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf5ff63740 10 client.16383 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:27.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.152+0000 7fcf5ff63740 1 -- 172.21.15.76:0/1405700204 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d4067bd8c0 con 0x55d406894c30 2024-03-12T05:59:27.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.153+0000 7fcf3aff5640 1 -- 172.21.15.76:0/1405700204 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf500345a0 con 0x55d406894c30 2024-03-12T05:59:27.167 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.153+0000 7fcf3aff5640 1 -- 172.21.15.76:0/1405700204 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fcf48043a40 con 0x55d406894c30 2024-03-12T05:59:27.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.153+0000 7fcf5ff63740 10 client.16383 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:27.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.153+0000 7fcf5ff63740 10 client.16383 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:59:27.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.153+0000 7fcf5ff63740 20 client.16383 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:27.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.153+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/1405700204 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d40690b410 0x55d40690d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.153+0000 7fcf5ff63740 4 client.16383 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:59:27.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.153+0000 7fcf5ff63740 1 -- 172.21.15.76:0/1405700204 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d406588890 con 0x55d40690b410 2024-03-12T05:59:27.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.153+0000 7fcf567fc640 1 --2- 172.21.15.76:0/1405700204 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d40690b410 0x55d40690d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.168 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.153+0000 7fcf567fc640 1 --2- 172.21.15.76:0/1405700204 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d40690b410 0x55d40690d830 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=843a17fef314bfe3 server_cookie=3c038fbaab94fdea in_seq=0 out_seq=0 2024-03-12T05:59:27.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.153+0000 7fcf3aff5640 10 client.16383 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:59:27.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf3aff5640 1 -- 172.21.15.76:0/1405700204 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d406588890 con 0x55d40690b410 2024-03-12T05:59:27.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf3aff5640 10 client.16383 handle_command_reply: tid=0 2024-03-12T05:59:27.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5ff63740 1 client.16383 shutdown 2024-03-12T05:59:27.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5ff63740 1 -- 172.21.15.76:0/1405700204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d406894c30 msgr2=0x55d4068a8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/1405700204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d406894c30 0x55d4068a8b10 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009ed0 tx=0x7fcf480035f0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5ff63740 1 -- 172.21.15.76:0/1405700204 shutdown_connections 2024-03-12T05:59:27.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/1405700204 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d40690b410 0x55d40690d830 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.169 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/1405700204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d4068a2b60 0x55d4069094f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/1405700204 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d4068955e0 0x55d4068a9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5ff63740 1 --2- 172.21.15.76:0/1405700204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d406894c30 0x55d4068a8b10 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf3aff5640 0 client.16383 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:59:27.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5ff63740 1 -- 172.21.15.76:0/1405700204 >> 172.21.15.76:0/1405700204 conn(0x55d40688f980 msgr2=0x55d406891660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5ff63740 1 -- 172.21.15.76:0/1405700204 shutdown_connections 2024-03-12T05:59:27.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5ff63740 1 -- 172.21.15.76:0/1405700204 wait complete. 2024-03-12T05:59:27.170 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5ff63740 20 client.16383 trim_cache size 0 max 16384 2024-03-12T05:59:27.171 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:59:27.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5d2b5640 1 -- 172.21.15.76:0/2474697447 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcf400a1c20 msgr2=0x7fcf400a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5d2b5640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcf400a1c20 0x7fcf400a40e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcf48011490 tx=0x7fcf48005f90 comp rx=0 tx=0).stop 2024-03-12T05:59:27.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5d2b5640 1 -- 172.21.15.76:0/2474697447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf58106460 msgr2=0x7fcf5819d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5d2b5640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf58106460 0x7fcf5819d960 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00e9f0 tx=0x7fcf4c00eeb0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.172 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5d2b5640 1 -- 172.21.15.76:0/2474697447 shutdown_connections 2024-03-12T05:59:27.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5d2b5640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcf400a1c20 0x7fcf400a40e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5d2b5640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf581975d0 0x7fcf58197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5d2b5640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcf58106460 0x7fcf5819d960 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5d2b5640 1 --2- 172.21.15.76:0/2474697447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcf58101390 0x7fcf5819d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.154+0000 7fcf5d2b5640 1 -- 172.21.15.76:0/2474697447 >> 172.21.15.76:0/2474697447 conn(0x7fcf580fcc10 msgr2=0x7fcf580fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.155+0000 7fcf5d2b5640 1 -- 172.21.15.76:0/2474697447 shutdown_connections 2024-03-12T05:59:27.173 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.155+0000 7fcf5d2b5640 1 -- 172.21.15.76:0/2474697447 wait complete. 2024-03-12T05:59:27.173 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:59:27.174 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:59:27.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.272+0000 7f7984a4e640 1 -- 172.21.15.76:0/2843579790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7980106450 msgr2=0x7f798010e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.272+0000 7f7984a4e640 1 --2- 172.21.15.76:0/2843579790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7980106450 0x7f798010e020 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f7978009cd0 tx=0x7f797802c0f0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.273+0000 7f7984a4e640 1 -- 172.21.15.76:0/2843579790 shutdown_connections 2024-03-12T05:59:27.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.273+0000 7f7984a4e640 1 --2- 172.21.15.76:0/2843579790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7980106450 0x7f798010e020 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.273+0000 7f7984a4e640 1 --2- 172.21.15.76:0/2843579790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7980101d30 0x7f7980105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.273+0000 7f7984a4e640 1 --2- 172.21.15.76:0/2843579790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7980101380 0x7f7980101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.273+0000 7f7984a4e640 1 -- 172.21.15.76:0/2843579790 >> 172.21.15.76:0/2843579790 conn(0x7f79800fcc00 msgr2=0x7f79800ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.273+0000 7f7984a4e640 1 -- 172.21.15.76:0/2843579790 shutdown_connections 2024-03-12T05:59:27.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.273+0000 7f7984a4e640 1 -- 172.21.15.76:0/2843579790 wait complete. 2024-03-12T05:59:27.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.273+0000 7f7984a4e640 1 Processor -- start 2024-03-12T05:59:27.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.273+0000 7f7984a4e640 1 -- start start 2024-03-12T05:59:27.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.273+0000 7f7984a4e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7980101380 0x7f798019d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f7984a4e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7980106450 0x7f798019dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f7984a4e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7980197650 0x7f7980197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f7984a4e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7980198120 con 0x7f7980197650 2024-03-12T05:59:27.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f7984a4e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7980198290 con 0x7f7980106450 2024-03-12T05:59:27.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f7984a4e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7980198450 con 0x7f7980101380 2024-03-12T05:59:27.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f797ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7980197650 0x7f7980197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f797ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7980197650 0x7f7980197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48190/0 (socket says 172.21.15.76:48190) 2024-03-12T05:59:27.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f797ed9d640 1 -- 172.21.15.76:0/1332803467 learned_addr learned my addr 172.21.15.76:0/1332803467 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:27.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f797e59c640 1 --2- 172.21.15.76:0/1332803467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7980101380 0x7f798019d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f797dd9b640 1 --2- 172.21.15.76:0/1332803467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7980106450 0x7f798019dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f797ed9d640 1 -- 172.21.15.76:0/1332803467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7980101380 msgr2=0x7f798019d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f797ed9d640 1 --2- 172.21.15.76:0/1332803467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7980101380 0x7f798019d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f797ed9d640 1 -- 172.21.15.76:0/1332803467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7980106450 msgr2=0x7f798019dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f797ed9d640 1 --2- 172.21.15.76:0/1332803467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7980106450 0x7f798019dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f797ed9d640 1 -- 172.21.15.76:0/1332803467 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7978009900 con 0x7f7980197650 2024-03-12T05:59:27.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f797ed9d640 1 --2- 172.21.15.76:0/1332803467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7980197650 0x7f7980197ad0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f79780348f0 tx=0x7f7978034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f796f7fe640 1 -- 172.21.15.76:0/1332803467 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780328c0 con 0x7f7980197650 2024-03-12T05:59:27.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f7984a4e640 1 -- 172.21.15.76:0/1332803467 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79801a4390 con 0x7f7980197650 2024-03-12T05:59:27.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f796f7fe640 1 -- 172.21.15.76:0/1332803467 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7978003e50 con 0x7f7980197650 2024-03-12T05:59:27.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f796f7fe640 1 -- 172.21.15.76:0/1332803467 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978040450 con 0x7f7980197650 2024-03-12T05:59:27.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.274+0000 7f7984a4e640 1 -- 172.21.15.76:0/1332803467 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79801a4860 con 0x7f7980197650 2024-03-12T05:59:27.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.277+0000 7f796f7fe640 1 -- 172.21.15.76:0/1332803467 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f797800a040 con 0x7f7980197650 2024-03-12T05:59:27.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.277+0000 7f7987626740 1 Processor -- start 2024-03-12T05:59:27.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.277+0000 7f7987626740 1 -- start start 2024-03-12T05:59:27.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.277+0000 7f796f7fe640 1 --2- 172.21.15.76:0/1332803467 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f79680a1ba0 0x7f79680a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.277+0000 7f7987626740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c2619e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.277+0000 7f7987626740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.277+0000 7f7987626740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 0x563c261a9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.277+0000 7f7987626740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c26159960 con 0x563c2619eb70 2024-03-12T05:59:27.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.277+0000 7f7987626740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c2610d310 con 0x563c261a7750 2024-03-12T05:59:27.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.277+0000 7f7987626740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c2610a840 con 0x563c2619e250 2024-03-12T05:59:27.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.277+0000 7f797dd9b640 1 --2- 172.21.15.76:0/1332803467 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f79680a1ba0 0x7f79680a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f797dd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f796f7fe640 1 -- 172.21.15.76:0/1332803467 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f79780f0500 con 0x7f7980197650 2024-03-12T05:59:27.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f797dd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48202/0 (socket says 172.21.15.76:48202) 2024-03-12T05:59:27.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f797dd9b640 1 -- 172.21.15.76:0/4091060757 learned_addr learned my addr 172.21.15.76:0/4091060757 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:27.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f797e59c640 1 --2- 172.21.15.76:0/4091060757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c2619e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f797ed9d640 1 --2- 172.21.15.76:0/4091060757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 0x563c261a9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f797dd9b640 1 -- 172.21.15.76:0/4091060757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 msgr2=0x563c2619e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f797dd9b640 1 --2- 172.21.15.76:0/4091060757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c2619e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f797dd9b640 1 -- 172.21.15.76:0/4091060757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 msgr2=0x563c261a9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f797dd9b640 1 --2- 172.21.15.76:0/4091060757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 0x563c261a9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f797dd9b640 1 -- 172.21.15.76:0/4091060757 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c260c8060 con 0x563c2619eb70 2024-03-12T05:59:27.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f797ed9d640 1 --2- 172.21.15.76:0/4091060757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 0x563c261a9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:27.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f797dd9b640 1 --2- 172.21.15.76:0/4091060757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a7210 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7970014950 tx=0x7f7970014e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70deb96219407859 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f797dd9b640 1 --2- 172.21.15.76:0/1332803467 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f79680a1ba0 0x7f79680a4060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7970006fd0 tx=0x7f797001b000 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f796cff9640 1 -- 172.21.15.76:0/4091060757 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7970018a20 con 0x563c2619eb70 2024-03-12T05:59:27.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f796cff9640 1 -- 172.21.15.76:0/4091060757 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7970014350 con 0x563c2619eb70 2024-03-12T05:59:27.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.278+0000 7f796cff9640 1 -- 172.21.15.76:0/4091060757 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7970020bb0 con 0x563c2619eb70 2024-03-12T05:59:27.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.279+0000 7f7987626740 1 -- 172.21.15.76:0/4091060757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 msgr2=0x563c261a7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.279+0000 7f7987626740 1 --2- 172.21.15.76:0/4091060757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a7210 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7970014950 tx=0x7f7970014e10 comp rx=0 tx=0).stop 2024-03-12T05:59:27.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.279+0000 7f7987626740 1 -- 172.21.15.76:0/4091060757 shutdown_connections 2024-03-12T05:59:27.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.279+0000 7f7987626740 1 --2- 172.21.15.76:0/4091060757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 0x563c261a9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.279+0000 7f7987626740 1 --2- 172.21.15.76:0/4091060757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a7210 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.279+0000 7f7987626740 1 --2- 172.21.15.76:0/4091060757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c2619e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.279+0000 7f7987626740 1 -- 172.21.15.76:0/4091060757 >> 172.21.15.76:0/4091060757 conn(0x563c26198980 msgr2=0x563c26198d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.279+0000 7f7987626740 1 -- 172.21.15.76:0/4091060757 shutdown_connections 2024-03-12T05:59:27.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.279+0000 7f7987626740 1 -- 172.21.15.76:0/4091060757 wait complete. 2024-03-12T05:59:27.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.279+0000 7f7987626740 1 -- start start 2024-03-12T05:59:27.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.279+0000 7f7987626740 1 Processor -- start 2024-03-12T05:59:27.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f7987626740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c261a4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f7987626740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f7987626740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 0x563c261a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f7987626740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c26106020 con 0x563c2619eb70 2024-03-12T05:59:27.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f7987626740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c25dd9430 con 0x563c261a7750 2024-03-12T05:59:27.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f7987626740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c25d95e00 con 0x563c2619e250 2024-03-12T05:59:27.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f7987626740 10 client.0 _ll_register_callbacks cb 0x563c260adcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:27.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f797dd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f797dd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48206/0 (socket says 172.21.15.76:48206) 2024-03-12T05:59:27.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f797dd9b640 1 -- 172.21.15.76:0/3986563551 learned_addr learned my addr 172.21.15.76:0/3986563551 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:27.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f797ed9d640 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 0x563c261a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f797e59c640 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c261a4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f797dd9b640 1 -- 172.21.15.76:0/3986563551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 msgr2=0x563c261a4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f797dd9b640 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c261a4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f797dd9b640 1 -- 172.21.15.76:0/3986563551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 msgr2=0x563c261a51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f797dd9b640 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 0x563c261a51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f797dd9b640 1 -- 172.21.15.76:0/3986563551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7970014530 con 0x563c2619eb70 2024-03-12T05:59:27.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f797ed9d640 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 0x563c261a51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:27.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f797e59c640 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c261a4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:59:27.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f797dd9b640 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a4c80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f797000c820 tx=0x7f7970018d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6f22f67f047489f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f7965ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:27.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f7965ffb640 1 -- 172.21.15.76:0/3986563551 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7970020880 con 0x563c2619eb70 2024-03-12T05:59:27.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f7965ffb640 1 -- 172.21.15.76:0/3986563551 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f797001c070 con 0x563c2619eb70 2024-03-12T05:59:27.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.280+0000 7f7987626740 1 -- 172.21.15.76:0/3986563551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563c25e057e0 con 0x563c2619eb70 2024-03-12T05:59:27.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.281+0000 7f7965ffb640 1 -- 172.21.15.76:0/3986563551 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7970018ad0 con 0x563c2619eb70 2024-03-12T05:59:27.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.281+0000 7f7987626740 10 client.16395 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:27.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.281+0000 7f7987626740 1 -- 172.21.15.76:0/3986563551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563c261046a0 con 0x563c2619eb70 2024-03-12T05:59:27.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.281+0000 7f7965ffb640 1 -- 172.21.15.76:0/3986563551 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563c25e057e0 con 0x563c2619eb70 2024-03-12T05:59:27.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.281+0000 7f7965ffb640 1 -- 172.21.15.76:0/3986563551 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f7970032060 con 0x563c2619eb70 2024-03-12T05:59:27.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.281+0000 7f7987626740 10 client.16395 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:27.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.281+0000 7f7987626740 10 client.16395 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:59:27.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.281+0000 7f7987626740 20 client.16395 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:27.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.281+0000 7f7987626740 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563c261b7b20 0x563c261b9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.281+0000 7f7987626740 4 client.16395 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:59:27.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.281+0000 7f7987626740 1 -- 172.21.15.76:0/3986563551 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563c25e42720 con 0x563c261b7b20 2024-03-12T05:59:27.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.281+0000 7f797e59c640 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563c261b7b20 0x563c261b9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f797e59c640 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563c261b7b20 0x563c261b9f80 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49d7dad7cd62db22 server_cookie=cc165ea7f0eab665 in_seq=0 out_seq=0 2024-03-12T05:59:27.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7965ffb640 10 client.16395 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:27.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7965ffb640 1 -- 172.21.15.76:0/3986563551 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563c25e42720 con 0x563c261b7b20 2024-03-12T05:59:27.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7965ffb640 10 client.16395 handle_command_reply: tid=0 2024-03-12T05:59:27.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7987626740 1 client.16395 shutdown 2024-03-12T05:59:27.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7987626740 1 -- 172.21.15.76:0/3986563551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 msgr2=0x563c261a4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7987626740 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a4c80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f797000c820 tx=0x7f7970018d30 comp rx=0 tx=0).stop 2024-03-12T05:59:27.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7987626740 1 -- 172.21.15.76:0/3986563551 shutdown_connections 2024-03-12T05:59:27.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7987626740 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 0x563c261a51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7987626740 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563c261b7b20 0x563c261b9f80 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7987626740 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a4c80 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7987626740 1 --2- 172.21.15.76:0/3986563551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c261a4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7965ffb640 0 client.16395 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:27.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7987626740 1 -- 172.21.15.76:0/3986563551 >> 172.21.15.76:0/3986563551 conn(0x563c26198980 msgr2=0x563c2619a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7987626740 1 -- 172.21.15.76:0/3986563551 shutdown_connections 2024-03-12T05:59:27.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7987626740 1 -- 172.21.15.76:0/3986563551 wait complete. 2024-03-12T05:59:27.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.282+0000 7f7987626740 20 client.16395 trim_cache size 0 max 16384 2024-03-12T05:59:27.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 Processor -- start 2024-03-12T05:59:27.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 -- start start 2024-03-12T05:59:27.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c261a6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 0x563c261a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c2608b130 con 0x563c2619eb70 2024-03-12T05:59:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c2619def0 con 0x563c261a7750 2024-03-12T05:59:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c261b45b0 con 0x563c2619e250 2024-03-12T05:59:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f797dd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f797dd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48212/0 (socket says 172.21.15.76:48212) 2024-03-12T05:59:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f797dd9b640 1 -- 172.21.15.76:0/820144385 learned_addr learned my addr 172.21.15.76:0/820144385 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f797ed9d640 1 --2- 172.21.15.76:0/820144385 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 0x563c261a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f797dd9b640 1 -- 172.21.15.76:0/820144385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 msgr2=0x563c261a6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f797dd9b640 1 --2- 172.21.15.76:0/820144385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c261a6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f797dd9b640 1 -- 172.21.15.76:0/820144385 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 msgr2=0x563c261a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f797dd9b640 1 --2- 172.21.15.76:0/820144385 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 0x563c261a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f797dd9b640 1 -- 172.21.15.76:0/820144385 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c260a0c20 con 0x563c2619eb70 2024-03-12T05:59:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f797dd9b640 1 --2- 172.21.15.76:0/820144385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a4fe0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f7970004810 tx=0x7f7970014fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65e0cd6b7c537f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7967fff640 1 -- 172.21.15.76:0/820144385 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797000c3b0 con 0x563c2619eb70 2024-03-12T05:59:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7967fff640 1 -- 172.21.15.76:0/820144385 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7970006bc0 con 0x563c2619eb70 2024-03-12T05:59:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7967fff640 1 -- 172.21.15.76:0/820144385 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797002b910 con 0x563c2619eb70 2024-03-12T05:59:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 -- 172.21.15.76:0/820144385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 msgr2=0x563c261a4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 --2- 172.21.15.76:0/820144385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a4fe0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f7970004810 tx=0x7f7970014fc0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 -- 172.21.15.76:0/820144385 shutdown_connections 2024-03-12T05:59:27.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 --2- 172.21.15.76:0/820144385 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c261a7750 0x563c261a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 --2- 172.21.15.76:0/820144385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c2619eb70 0x563c261a4fe0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 --2- 172.21.15.76:0/820144385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c261a6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 -- 172.21.15.76:0/820144385 >> 172.21.15.76:0/820144385 conn(0x563c26198980 msgr2=0x563c2619fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 -- 172.21.15.76:0/820144385 shutdown_connections 2024-03-12T05:59:27.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.295+0000 7f7987626740 1 -- 172.21.15.76:0/820144385 wait complete. 2024-03-12T05:59:27.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f7987626740 1 -- start start 2024-03-12T05:59:27.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f7987626740 1 Processor -- start 2024-03-12T05:59:27.312 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f7987626740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c261ad390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f7987626740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c2619eb70 0x563c261ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f7987626740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c261a7750 0x563c2620e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f7987626740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c261ae1d0 con 0x563c261a7750 2024-03-12T05:59:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f7987626740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c2620e5b0 con 0x563c2619eb70 2024-03-12T05:59:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f7987626740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c2620e6f0 con 0x563c2619e250 2024-03-12T05:59:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f7987626740 10 client.0 _ll_register_callbacks cb 0x7f797000ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f797ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c261a7750 0x563c2620e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f797ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c261a7750 0x563c2620e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48226/0 (socket says 172.21.15.76:48226) 2024-03-12T05:59:27.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f797ed9d640 1 -- 172.21.15.76:0/3213378073 learned_addr learned my addr 172.21.15.76:0/3213378073 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f797ed9d640 1 -- 172.21.15.76:0/3213378073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 msgr2=0x563c261ad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f797ed9d640 1 --2- 172.21.15.76:0/3213378073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c261ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f797ed9d640 1 -- 172.21.15.76:0/3213378073 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c2619eb70 msgr2=0x563c261ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f797ed9d640 1 --2- 172.21.15.76:0/3213378073 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c2619eb70 0x563c261ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f797ed9d640 1 -- 172.21.15.76:0/3213378073 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c261046a0 con 0x563c261a7750 2024-03-12T05:59:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f797ed9d640 1 --2- 172.21.15.76:0/3213378073 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c261a7750 0x563c2620e070 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f79780b9c40 tx=0x7f79780bdab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f5940b45beacc23 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.296+0000 7f7965ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f7965ffb640 1 -- 172.21.15.76:0/3213378073 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780c13c0 con 0x563c261a7750 2024-03-12T05:59:27.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f7965ffb640 1 -- 172.21.15.76:0/3213378073 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f79780c1190 con 0x563c261a7750 2024-03-12T05:59:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f7987626740 1 -- 172.21.15.76:0/3213378073 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7970014350 con 0x563c261a7750 2024-03-12T05:59:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f7965ffb640 1 -- 172.21.15.76:0/3213378073 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79780c5a10 con 0x563c261a7750 2024-03-12T05:59:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f7987626740 10 client.16401 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f7987626740 1 -- 172.21.15.76:0/3213378073 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563c260c68c0 con 0x563c261a7750 2024-03-12T05:59:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f7965ffb640 1 -- 172.21.15.76:0/3213378073 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7970014350 con 0x563c261a7750 2024-03-12T05:59:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f7965ffb640 1 -- 172.21.15.76:0/3213378073 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f7978004040 con 0x563c261a7750 2024-03-12T05:59:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f7987626740 10 client.16401 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f7987626740 10 client.16401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:59:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f7987626740 20 client.16401 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:27.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f7987626740 1 --2- 172.21.15.76:0/3213378073 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563c2620fdd0 0x563c26212210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f7987626740 4 client.16401 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f7987626740 1 -- 172.21.15.76:0/3213378073 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563c25e91890 con 0x563c2620fdd0 2024-03-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.297+0000 7f797e59c640 1 --2- 172.21.15.76:0/3213378073 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563c2620fdd0 0x563c26212210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7965ffb640 10 client.16401 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f797e59c640 1 --2- 172.21.15.76:0/3213378073 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563c2620fdd0 0x563c26212210 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e0777b1806adb73 server_cookie=2e731f39d64267b6 in_seq=0 out_seq=0 2024-03-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7965ffb640 1 -- 172.21.15.76:0/3213378073 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563c25e91890 con 0x563c2620fdd0 2024-03-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7965ffb640 10 client.16401 handle_command_reply: tid=0 2024-03-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7987626740 1 client.16401 shutdown 2024-03-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7987626740 1 -- 172.21.15.76:0/3213378073 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c261a7750 msgr2=0x563c2620e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7987626740 1 --2- 172.21.15.76:0/3213378073 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c261a7750 0x563c2620e070 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f79780b9c40 tx=0x7f79780bdab0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7987626740 1 -- 172.21.15.76:0/3213378073 shutdown_connections 2024-03-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7987626740 1 --2- 172.21.15.76:0/3213378073 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563c2620fdd0 0x563c26212210 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7987626740 1 --2- 172.21.15.76:0/3213378073 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563c261a7750 0x563c2620e070 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7987626740 1 --2- 172.21.15.76:0/3213378073 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563c2619eb70 0x563c261ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7965ffb640 0 client.16401 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7987626740 1 --2- 172.21.15.76:0/3213378073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563c2619e250 0x563c261ad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7987626740 1 -- 172.21.15.76:0/3213378073 >> 172.21.15.76:0/3213378073 conn(0x563c26198980 msgr2=0x563c261972f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7987626740 1 -- 172.21.15.76:0/3213378073 shutdown_connections 2024-03-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7987626740 1 -- 172.21.15.76:0/3213378073 wait complete. 2024-03-12T05:59:27.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.298+0000 7f7987626740 20 client.16401 trim_cache size 0 max 16384 2024-03-12T05:59:27.318 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:59:27.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.299+0000 7f7984a4e640 1 -- 172.21.15.76:0/1332803467 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f79680a1ba0 msgr2=0x7f79680a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.299+0000 7f7984a4e640 1 --2- 172.21.15.76:0/1332803467 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f79680a1ba0 0x7f79680a4060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7970006fd0 tx=0x7f797001b000 comp rx=0 tx=0).stop 2024-03-12T05:59:27.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.299+0000 7f7984a4e640 1 -- 172.21.15.76:0/1332803467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7980197650 msgr2=0x7f7980197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:27.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.299+0000 7f7984a4e640 1 --2- 172.21.15.76:0/1332803467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7980197650 0x7f7980197ad0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f79780348f0 tx=0x7f7978034cc0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.299+0000 7f7984a4e640 1 -- 172.21.15.76:0/1332803467 shutdown_connections 2024-03-12T05:59:27.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.299+0000 7f7984a4e640 1 --2- 172.21.15.76:0/1332803467 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f79680a1ba0 0x7f79680a4060 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.299+0000 7f7984a4e640 1 --2- 172.21.15.76:0/1332803467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7980197650 0x7f7980197ad0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.299+0000 7f7984a4e640 1 --2- 172.21.15.76:0/1332803467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7980106450 0x7f798019dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.299+0000 7f7984a4e640 1 --2- 172.21.15.76:0/1332803467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7980101380 0x7f798019d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:27.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.299+0000 7f7984a4e640 1 -- 172.21.15.76:0/1332803467 >> 172.21.15.76:0/1332803467 conn(0x7f79800fcc00 msgr2=0x7f79800fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:27.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.299+0000 7f7984a4e640 1 -- 172.21.15.76:0/1332803467 shutdown_connections 2024-03-12T05:59:27.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:27.299+0000 7f7984a4e640 1 -- 172.21.15.76:0/1332803467 wait complete. 2024-03-12T05:59:27.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:27 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4159821790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:59:27.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:27 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4159821790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:59:27.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:27 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4159821790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T05:59:28.322 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T05:59:28.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T05:59:28.322 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T05:59:28.385 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:28 smithi088 ceph-mon[36585]: pgmap v113: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.436+0000 7f3a1cdd0640 1 -- 172.21.15.76:0/239109672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a18100290 msgr2=0x7f3a18100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.436+0000 7f3a1cdd0640 1 --2- 172.21.15.76:0/239109672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a18100290 0x7f3a18100670 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009930 tx=0x7f3a0c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:59:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.436+0000 7f3a1cdd0640 1 -- 172.21.15.76:0/239109672 shutdown_connections 2024-03-12T05:59:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.436+0000 7f3a1cdd0640 1 --2- 172.21.15.76:0/239109672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a18105360 0x7f3a1810cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.436+0000 7f3a1cdd0640 1 --2- 172.21.15.76:0/239109672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a18100c40 0x7f3a18104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.436+0000 7f3a1cdd0640 1 --2- 172.21.15.76:0/239109672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a18100290 0x7f3a18100670 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.436+0000 7f3a1cdd0640 1 -- 172.21.15.76:0/239109672 >> 172.21.15.76:0/239109672 conn(0x7f3a18077230 msgr2=0x7f3a18077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.436+0000 7f3a1cdd0640 1 -- 172.21.15.76:0/239109672 shutdown_connections 2024-03-12T05:59:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.436+0000 7f3a1cdd0640 1 -- 172.21.15.76:0/239109672 wait complete. 2024-03-12T05:59:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.437+0000 7f3a1cdd0640 1 Processor -- start 2024-03-12T05:59:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.437+0000 7f3a1cdd0640 1 -- start start 2024-03-12T05:59:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.437+0000 7f3a1cdd0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a18100290 0x7f3a1819d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.437+0000 7f3a1cdd0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a18105360 0x7f3a1819dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.437+0000 7f3a1cdd0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a18197650 0x7f3a18197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.437+0000 7f3a1cdd0640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a18198120 con 0x7f3a18100290 2024-03-12T05:59:28.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.437+0000 7f3a1cdd0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a18198290 con 0x7f3a18105360 2024-03-12T05:59:28.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.437+0000 7f3a1cdd0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a18198450 con 0x7f3a18197650 2024-03-12T05:59:28.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.437+0000 7f3a1659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a18100290 0x7f3a1819d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.437+0000 7f3a1659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a18100290 0x7f3a1819d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48244/0 (socket says 172.21.15.76:48244) 2024-03-12T05:59:28.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.437+0000 7f3a1659c640 1 -- 172.21.15.76:0/2663519827 learned_addr learned my addr 172.21.15.76:0/2663519827 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:28.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a18197650 0x7f3a18197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a1659c640 1 -- 172.21.15.76:0/2663519827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a18197650 msgr2=0x7f3a18197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a1659c640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a18197650 0x7f3a18197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a1659c640 1 -- 172.21.15.76:0/2663519827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a18105360 msgr2=0x7f3a1819dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a15d9b640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a18105360 0x7f3a1819dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a1659c640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a18105360 0x7f3a1819dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a1659c640 1 -- 172.21.15.76:0/2663519827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c009560 con 0x7f3a18100290 2024-03-12T05:59:28.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a18197650 0x7f3a18197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:28.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a15d9b640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a18105360 0x7f3a1819dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:59:28.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a1659c640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a18100290 0x7f3a1819d590 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009230 tx=0x7f3a0c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:28.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a077fe640 1 -- 172.21.15.76:0/2663519827 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c004440 con 0x7f3a18100290 2024-03-12T05:59:28.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a077fe640 1 -- 172.21.15.76:0/2663519827 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a0c0052d0 con 0x7f3a18100290 2024-03-12T05:59:28.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a077fe640 1 -- 172.21.15.76:0/2663519827 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c003d70 con 0x7f3a18100290 2024-03-12T05:59:28.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a1cdd0640 1 -- 172.21.15.76:0/2663519827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a181a4390 con 0x7f3a18100290 2024-03-12T05:59:28.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.438+0000 7f3a1cdd0640 1 -- 172.21.15.76:0/2663519827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a181a48c0 con 0x7f3a18100290 2024-03-12T05:59:28.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:28 smithi082 ceph-mon[36382]: pgmap v113: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:28.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.440+0000 7f3a077fe640 1 -- 172.21.15.76:0/2663519827 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3a0c005440 con 0x7f3a18100290 2024-03-12T05:59:28.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.440+0000 7f3a1f99b740 1 Processor -- start 2024-03-12T05:59:28.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a1f99b740 1 -- start start 2024-03-12T05:59:28.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a077fe640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3a000a1c20 0x7f3a000a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a1f99b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17260 0x56347fc17640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a1f99b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc17c10 0x56347fc20740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a1f99b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc21170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a1f99b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347fb85f90 con 0x56347fc20d10 2024-03-12T05:59:28.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a1f99b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347fb834c0 con 0x56347fc17c10 2024-03-12T05:59:28.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a1f99b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347fb7eca0 con 0x56347fc17260 2024-03-12T05:59:28.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3a000a1c20 0x7f3a000a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a077fe640 1 -- 172.21.15.76:0/2663519827 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3a0c0ef960 con 0x7f3a18100290 2024-03-12T05:59:28.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a16d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc21170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a16d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc21170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48256/0 (socket says 172.21.15.76:48256) 2024-03-12T05:59:28.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a16d9d640 1 -- 172.21.15.76:0/1703904125 learned_addr learned my addr 172.21.15.76:0/1703904125 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:28.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a1659c640 1 --2- 172.21.15.76:0/1703904125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17260 0x56347fc17640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a15d9b640 1 --2- 172.21.15.76:0/1703904125 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc17c10 0x56347fc20740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a16d9d640 1 -- 172.21.15.76:0/1703904125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17260 msgr2=0x56347fc17640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/1703904125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17260 0x56347fc17640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a16d9d640 1 -- 172.21.15.76:0/1703904125 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc17c10 msgr2=0x56347fc20740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/1703904125 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc17c10 0x56347fc20740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.441+0000 7f3a16d9d640 1 -- 172.21.15.76:0/1703904125 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56347fb410b0 con 0x56347fc20d10 2024-03-12T05:59:28.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.442+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/1703904125 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc21170 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3a10012ce0 tx=0x7f3a1000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347b261bb01a113a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:28.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.442+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3a000a1c20 0x7f3a000a40e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3a1000fc30 tx=0x7f3a1001d000 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:28.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.442+0000 7f3a04ff9640 1 -- 172.21.15.76:0/1703904125 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10012dd0 con 0x56347fc20d10 2024-03-12T05:59:28.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.442+0000 7f3a04ff9640 1 -- 172.21.15.76:0/1703904125 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a10009d20 con 0x56347fc20d10 2024-03-12T05:59:28.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.442+0000 7f3a04ff9640 1 -- 172.21.15.76:0/1703904125 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10005280 con 0x56347fc20d10 2024-03-12T05:59:28.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 -- 172.21.15.76:0/1703904125 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 msgr2=0x56347fc21170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/1703904125 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc21170 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3a10012ce0 tx=0x7f3a1000f730 comp rx=0 tx=0).stop 2024-03-12T05:59:28.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 -- 172.21.15.76:0/1703904125 shutdown_connections 2024-03-12T05:59:28.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/1703904125 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc21170 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/1703904125 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc17c10 0x56347fc20740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/1703904125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17260 0x56347fc17640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 -- 172.21.15.76:0/1703904125 >> 172.21.15.76:0/1703904125 conn(0x56347fc11840 msgr2=0x56347fc11c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:28.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 -- 172.21.15.76:0/1703904125 shutdown_connections 2024-03-12T05:59:28.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 -- 172.21.15.76:0/1703904125 wait complete. 2024-03-12T05:59:28.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 -- start start 2024-03-12T05:59:28.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 Processor -- start 2024-03-12T05:59:28.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc17260 0x56347fc24840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17c10 0x56347fc24d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc252c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347fb23f40 con 0x56347fc20d10 2024-03-12T05:59:28.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f8524b0 con 0x56347fc17260 2024-03-12T05:59:28.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347fb04670 con 0x56347fc17c10 2024-03-12T05:59:28.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a1f99b740 10 client.0 _ll_register_callbacks cb 0x56347fb71f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:28.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a16d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc252c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a16d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc252c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48262/0 (socket says 172.21.15.76:48262) 2024-03-12T05:59:28.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.443+0000 7f3a16d9d640 1 -- 172.21.15.76:0/2347747682 learned_addr learned my addr 172.21.15.76:0/2347747682 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:28.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f3a1659c640 1 --2- 172.21.15.76:0/2347747682 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc17260 0x56347fc24840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f3a16d9d640 1 -- 172.21.15.76:0/2347747682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17c10 msgr2=0x56347fc24d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/2347747682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17c10 0x56347fc24d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f3a16d9d640 1 -- 172.21.15.76:0/2347747682 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc17260 msgr2=0x56347fc24840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/2347747682 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc17260 0x56347fc24840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f3a16d9d640 1 -- 172.21.15.76:0/2347747682 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a1001a040 con 0x56347fc20d10 2024-03-12T05:59:28.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/2347747682 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc252c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3a10006010 tx=0x7f3a1001cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad6387620d1b568 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:28.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f39fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:28.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f39fdffb640 1 -- 172.21.15.76:0/2347747682 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10012dd0 con 0x56347fc20d10 2024-03-12T05:59:28.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f39fdffb640 1 -- 172.21.15.76:0/2347747682 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3a100054b0 con 0x56347fc20d10 2024-03-12T05:59:28.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2347747682 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56347fb58f40 con 0x56347fc20d10 2024-03-12T05:59:28.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f39fdffb640 1 -- 172.21.15.76:0/2347747682 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a10005670 con 0x56347fc20d10 2024-03-12T05:59:28.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f3a1f99b740 10 client.16413 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:28.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2347747682 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56347fb7d320 con 0x56347fc20d10 2024-03-12T05:59:28.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f39fdffb640 1 -- 172.21.15.76:0/2347747682 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56347fb58f40 con 0x56347fc20d10 2024-03-12T05:59:28.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.444+0000 7f39fdffb640 1 -- 172.21.15.76:0/2347747682 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f3a10024030 con 0x56347fc20d10 2024-03-12T05:59:28.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.445+0000 7f3a1f99b740 10 client.16413 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:28.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.445+0000 7f3a1f99b740 10 client.16413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:59:28.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.445+0000 7f3a1f99b740 20 client.16413 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:28.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.445+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/2347747682 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56347fc35320 0x56347fc37740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.445+0000 7f3a1f99b740 4 client.16413 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:59:28.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.445+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2347747682 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56347fb75310 con 0x56347fc35320 2024-03-12T05:59:28.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.445+0000 7f3a1659c640 1 --2- 172.21.15.76:0/2347747682 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56347fc35320 0x56347fc37740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.445+0000 7f39fdffb640 10 client.16413 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:28.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.445+0000 7f3a1659c640 1 --2- 172.21.15.76:0/2347747682 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56347fc35320 0x56347fc37740 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d715d2b7d7267e4d server_cookie=8ed71bbadd61131e in_seq=0 out_seq=0 2024-03-12T05:59:28.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f39fdffb640 1 -- 172.21.15.76:0/2347747682 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56347fb75310 con 0x56347fc35320 2024-03-12T05:59:28.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f39fdffb640 10 client.16413 handle_command_reply: tid=0 2024-03-12T05:59:28.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f3a1f99b740 1 client.16413 shutdown 2024-03-12T05:59:28.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2347747682 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 msgr2=0x56347fc252c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/2347747682 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc252c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3a10006010 tx=0x7f3a1001cf90 comp rx=0 tx=0).stop 2024-03-12T05:59:28.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2347747682 shutdown_connections 2024-03-12T05:59:28.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/2347747682 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56347fc35320 0x56347fc37740 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/2347747682 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc252c0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f39fdffb640 0 client.16413 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:28.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/2347747682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17c10 0x56347fc24d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/2347747682 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc17260 0x56347fc24840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2347747682 >> 172.21.15.76:0/2347747682 conn(0x56347fc11840 msgr2=0x56347fc137a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:28.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2347747682 shutdown_connections 2024-03-12T05:59:28.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2347747682 wait complete. 2024-03-12T05:59:28.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.446+0000 7f3a1f99b740 20 client.16413 trim_cache size 0 max 16384 2024-03-12T05:59:28.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.461+0000 7f3a1f99b740 1 Processor -- start 2024-03-12T05:59:28.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.461+0000 7f3a1f99b740 1 -- start start 2024-03-12T05:59:28.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.461+0000 7f3a1f99b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17260 0x56347fc250e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.461+0000 7f3a1f99b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc17c10 0x56347fc22100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.461+0000 7f3a1f99b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc2bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.461+0000 7f3a1f99b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347fc27900 con 0x56347fc20d10 2024-03-12T05:59:28.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.461+0000 7f3a1f99b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347fc0bcf0 con 0x56347fc17c10 2024-03-12T05:59:28.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.461+0000 7f3a1f99b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347fc16690 con 0x56347fc17260 2024-03-12T05:59:28.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.461+0000 7f3a16d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc2bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.461+0000 7f3a16d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc2bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48264/0 (socket says 172.21.15.76:48264) 2024-03-12T05:59:28.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.461+0000 7f3a16d9d640 1 -- 172.21.15.76:0/288190577 learned_addr learned my addr 172.21.15.76:0/288190577 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:28.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.461+0000 7f3a1659c640 1 --2- 172.21.15.76:0/288190577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17260 0x56347fc250e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a16d9d640 1 -- 172.21.15.76:0/288190577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17260 msgr2=0x56347fc250e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/288190577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17260 0x56347fc250e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a16d9d640 1 -- 172.21.15.76:0/288190577 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc17c10 msgr2=0x56347fc22100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/288190577 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc17c10 0x56347fc22100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a16d9d640 1 -- 172.21.15.76:0/288190577 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56347fb19c00 con 0x56347fc20d10 2024-03-12T05:59:28.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a1659c640 1 --2- 172.21.15.76:0/288190577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17260 0x56347fc250e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:59:28.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/288190577 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc2bcf0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f3a10012670 tx=0x7f3a10009670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d520e34aefd47fc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:28.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f39fffff640 1 -- 172.21.15.76:0/288190577 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1001cb40 con 0x56347fc20d10 2024-03-12T05:59:28.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f39fffff640 1 -- 172.21.15.76:0/288190577 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a1002fe30 con 0x56347fc20d10 2024-03-12T05:59:28.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f39fffff640 1 -- 172.21.15.76:0/288190577 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a10005420 con 0x56347fc20d10 2024-03-12T05:59:28.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a1f99b740 1 -- 172.21.15.76:0/288190577 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 msgr2=0x56347fc2bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/288190577 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc2bcf0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f3a10012670 tx=0x7f3a10009670 comp rx=0 tx=0).stop 2024-03-12T05:59:28.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a1f99b740 1 -- 172.21.15.76:0/288190577 shutdown_connections 2024-03-12T05:59:28.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/288190577 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc20d10 0x56347fc2bcf0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/288190577 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc17c10 0x56347fc22100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/288190577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17260 0x56347fc250e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a1f99b740 1 -- 172.21.15.76:0/288190577 >> 172.21.15.76:0/288190577 conn(0x56347fc11840 msgr2=0x56347fc10050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:28.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a1f99b740 1 -- 172.21.15.76:0/288190577 shutdown_connections 2024-03-12T05:59:28.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.462+0000 7f3a1f99b740 1 -- 172.21.15.76:0/288190577 wait complete. 2024-03-12T05:59:28.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1f99b740 1 -- start start 2024-03-12T05:59:28.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1f99b740 1 Processor -- start 2024-03-12T05:59:28.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1f99b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc17260 0x56347fc2b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1f99b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17c10 0x56347fc2b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1f99b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc20d10 0x56347fc8b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1f99b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347fc8bd90 con 0x56347fc17260 2024-03-12T05:59:28.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1f99b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347fc8bed0 con 0x56347fc20d10 2024-03-12T05:59:28.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1f99b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347fc8c060 con 0x56347fc17c10 2024-03-12T05:59:28.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1f99b740 10 client.0 _ll_register_callbacks cb 0x7f3a1806aa10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:28.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc17260 0x56347fc2b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc17260 0x56347fc2b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48280/0 (socket says 172.21.15.76:48280) 2024-03-12T05:59:28.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1659c640 1 -- 172.21.15.76:0/2071867201 learned_addr learned my addr 172.21.15.76:0/2071867201 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:28.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a15d9b640 1 --2- 172.21.15.76:0/2071867201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17c10 0x56347fc2b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/2071867201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc20d10 0x56347fc8b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1659c640 1 -- 172.21.15.76:0/2071867201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17c10 msgr2=0x56347fc2b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1659c640 1 --2- 172.21.15.76:0/2071867201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17c10 0x56347fc2b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1659c640 1 -- 172.21.15.76:0/2071867201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc20d10 msgr2=0x56347fc8b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1659c640 1 --2- 172.21.15.76:0/2071867201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc20d10 0x56347fc8b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a1659c640 1 -- 172.21.15.76:0/2071867201 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56347fb7d320 con 0x56347fc17260 2024-03-12T05:59:28.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.463+0000 7f3a16d9d640 1 --2- 172.21.15.76:0/2071867201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc20d10 0x56347fc8b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:59:28.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f3a1659c640 1 --2- 172.21.15.76:0/2071867201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc17260 0x56347fc2b030 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0bc3d0 tx=0x7f3a0c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a07e065ae1c82d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:28.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f39fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:28.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f39fdffb640 1 -- 172.21.15.76:0/2071867201 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c0d9060 con 0x56347fc17260 2024-03-12T05:59:28.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f39fdffb640 1 -- 172.21.15.76:0/2071867201 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3a0c0c0820 con 0x56347fc17260 2024-03-12T05:59:28.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2071867201 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56347fba4900 con 0x56347fc17260 2024-03-12T05:59:28.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f39fdffb640 1 -- 172.21.15.76:0/2071867201 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a0c0c4e70 con 0x56347fc17260 2024-03-12T05:59:28.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f3a1f99b740 10 client.16419 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:28.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2071867201 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56347fb3f910 con 0x56347fc17260 2024-03-12T05:59:28.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f39fdffb640 1 -- 172.21.15.76:0/2071867201 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56347fba4900 con 0x56347fc17260 2024-03-12T05:59:28.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f39fdffb640 1 -- 172.21.15.76:0/2071867201 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f3a0c03c070 con 0x56347fc17260 2024-03-12T05:59:28.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f3a1f99b740 10 client.16419 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:28.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f3a1f99b740 10 client.16419 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:59:28.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f3a1f99b740 20 client.16419 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:28.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/2071867201 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56347fc8d6a0 0x56347fc8fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f3a1f99b740 4 client.16419 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T05:59:28.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2071867201 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56347f87e860 con 0x56347fc8d6a0 2024-03-12T05:59:28.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.464+0000 7f3a15d9b640 1 --2- 172.21.15.76:0/2071867201 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56347fc8d6a0 0x56347fc8fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.465+0000 7f3a15d9b640 1 --2- 172.21.15.76:0/2071867201 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56347fc8d6a0 0x56347fc8fac0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42d52ddbe817df19 server_cookie=8c164fb2693dbc56 in_seq=0 out_seq=0 2024-03-12T05:59:28.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.465+0000 7f39fdffb640 10 client.16419 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:28.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f39fdffb640 1 -- 172.21.15.76:0/2071867201 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56347f87e860 con 0x56347fc8d6a0 2024-03-12T05:59:28.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f39fdffb640 10 client.16419 handle_command_reply: tid=0 2024-03-12T05:59:28.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f3a1f99b740 1 client.16419 shutdown 2024-03-12T05:59:28.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2071867201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc17260 msgr2=0x56347fc2b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/2071867201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc17260 0x56347fc2b030 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0bc3d0 tx=0x7f3a0c009260 comp rx=0 tx=0).stop 2024-03-12T05:59:28.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2071867201 shutdown_connections 2024-03-12T05:59:28.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/2071867201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56347fc20d10 0x56347fc8b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/2071867201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56347fc17c10 0x56347fc2b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/2071867201 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56347fc8d6a0 0x56347fc8fac0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f3a1f99b740 1 --2- 172.21.15.76:0/2071867201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56347fc17260 0x56347fc2b030 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2071867201 >> 172.21.15.76:0/2071867201 conn(0x56347fc11840 msgr2=0x56347fc108e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:28.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f39fdffb640 0 client.16419 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:28.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2071867201 shutdown_connections 2024-03-12T05:59:28.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f3a1f99b740 1 -- 172.21.15.76:0/2071867201 wait complete. 2024-03-12T05:59:28.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.466+0000 7f3a1f99b740 20 client.16419 trim_cache size 0 max 16384 2024-03-12T05:59:28.483 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:59:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T05:59:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "8e4fd6df-87c6-48f5-9526-cc7474ef2c17", 2024-03-12T05:59:28.483 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T05:59:28.483 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:59:28.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.467+0000 7f3a1cdd0640 1 -- 172.21.15.76:0/2663519827 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3a000a1c20 msgr2=0x7f3a000a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.467+0000 7f3a1cdd0640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3a000a1c20 0x7f3a000a40e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3a1000fc30 tx=0x7f3a1001d000 comp rx=0 tx=0).stop 2024-03-12T05:59:28.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.467+0000 7f3a1cdd0640 1 -- 172.21.15.76:0/2663519827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a18100290 msgr2=0x7f3a1819d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.467+0000 7f3a1cdd0640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a18100290 0x7f3a1819d590 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009230 tx=0x7f3a0c004040 comp rx=0 tx=0).stop 2024-03-12T05:59:28.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.467+0000 7f3a1cdd0640 1 -- 172.21.15.76:0/2663519827 shutdown_connections 2024-03-12T05:59:28.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.467+0000 7f3a1cdd0640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3a000a1c20 0x7f3a000a40e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.467+0000 7f3a1cdd0640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a18197650 0x7f3a18197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.467+0000 7f3a1cdd0640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3a18105360 0x7f3a1819dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.467+0000 7f3a1cdd0640 1 --2- 172.21.15.76:0/2663519827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a18100290 0x7f3a1819d590 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.467+0000 7f3a1cdd0640 1 -- 172.21.15.76:0/2663519827 >> 172.21.15.76:0/2663519827 conn(0x7f3a18077230 msgr2=0x7f3a180687f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:28.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.467+0000 7f3a1cdd0640 1 -- 172.21.15.76:0/2663519827 shutdown_connections 2024-03-12T05:59:28.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.467+0000 7f3a1cdd0640 1 -- 172.21.15.76:0/2663519827 wait complete. 2024-03-12T05:59:28.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:28 smithi076 ceph-mon[32095]: pgmap v113: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:28.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:59:28.487 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:59:28.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.607+0000 7fc3af348640 1 -- 172.21.15.76:0/1298336209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3a8100c10 msgr2=0x7fc3a8104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.607+0000 7fc3af348640 1 --2- 172.21.15.76:0/1298336209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3a8100c10 0x7fc3a8104c50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc398009860 tx=0x7fc39802bb70 comp rx=0 tx=0).stop 2024-03-12T05:59:28.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.607+0000 7fc3af348640 1 -- 172.21.15.76:0/1298336209 shutdown_connections 2024-03-12T05:59:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.607+0000 7fc3af348640 1 --2- 172.21.15.76:0/1298336209 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3a8105330 0x7fc3a810cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.607+0000 7fc3af348640 1 --2- 172.21.15.76:0/1298336209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3a8100c10 0x7fc3a8104c50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.607+0000 7fc3af348640 1 --2- 172.21.15.76:0/1298336209 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3a8100260 0x7fc3a8100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.607+0000 7fc3af348640 1 -- 172.21.15.76:0/1298336209 >> 172.21.15.76:0/1298336209 conn(0x7fc3a8077180 msgr2=0x7fc3a8077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.607+0000 7fc3af348640 1 -- 172.21.15.76:0/1298336209 shutdown_connections 2024-03-12T05:59:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.607+0000 7fc3af348640 1 -- 172.21.15.76:0/1298336209 wait complete. 2024-03-12T05:59:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.608+0000 7fc3af348640 1 Processor -- start 2024-03-12T05:59:28.614 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.608+0000 7fc3af348640 1 -- start start 2024-03-12T05:59:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.608+0000 7fc3af348640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3a8100260 0x7fc3a819f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.608+0000 7fc3af348640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3a8105330 0x7fc3a819fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.608+0000 7fc3af348640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3a81997e0 0x7fc3a8199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.608+0000 7fc3af348640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a819a290 con 0x7fc3a8100260 2024-03-12T05:59:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.608+0000 7fc3af348640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a819a400 con 0x7fc3a81997e0 2024-03-12T05:59:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.608+0000 7fc3af348640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a819a570 con 0x7fc3a8105330 2024-03-12T05:59:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.608+0000 7fc3ad0e4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3a8100260 0x7fc3a819f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.608+0000 7fc3ad0e4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3a8100260 0x7fc3a819f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48298/0 (socket says 172.21.15.76:48298) 2024-03-12T05:59:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.608+0000 7fc3ad0e4640 1 -- 172.21.15.76:0/2115896461 learned_addr learned my addr 172.21.15.76:0/2115896461 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.608+0000 7fc3ad8e5640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3a81997e0 0x7fc3a8199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.608+0000 7fc3ac8e3640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3a8105330 0x7fc3a819fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.609+0000 7fc3ad0e4640 1 -- 172.21.15.76:0/2115896461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3a8105330 msgr2=0x7fc3a819fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.609+0000 7fc3ad0e4640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3a8105330 0x7fc3a819fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.609+0000 7fc3ad0e4640 1 -- 172.21.15.76:0/2115896461 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3a81997e0 msgr2=0x7fc3a8199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.609+0000 7fc3ad0e4640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3a81997e0 0x7fc3a8199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.609+0000 7fc3ad0e4640 1 -- 172.21.15.76:0/2115896461 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc398009490 con 0x7fc3a8100260 2024-03-12T05:59:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.609+0000 7fc3ad8e5640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3a81997e0 0x7fc3a8199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:59:28.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.609+0000 7fc3ad0e4640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3a8100260 0x7fc3a819f670 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400e9f0 tx=0x7fc3a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:28.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.610+0000 7fc39e7fc640 1 -- 172.21.15.76:0/2115896461 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a400ccc0 con 0x7fc3a8100260 2024-03-12T05:59:28.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.610+0000 7fc39e7fc640 1 -- 172.21.15.76:0/2115896461 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3a400ce20 con 0x7fc3a8100260 2024-03-12T05:59:28.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.610+0000 7fc3af348640 1 -- 172.21.15.76:0/2115896461 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a81a6520 con 0x7fc3a8100260 2024-03-12T05:59:28.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.610+0000 7fc39e7fc640 1 -- 172.21.15.76:0/2115896461 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4010690 con 0x7fc3a8100260 2024-03-12T05:59:28.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.610+0000 7fc3af348640 1 -- 172.21.15.76:0/2115896461 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a81a69b0 con 0x7fc3a8100260 2024-03-12T05:59:28.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.612+0000 7fc39e7fc640 1 -- 172.21.15.76:0/2115896461 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc3a40046c0 con 0x7fc3a8100260 2024-03-12T05:59:28.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.613+0000 7fc3b1ff8740 1 Processor -- start 2024-03-12T05:59:28.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.613+0000 7fc3b1ff8740 1 -- start start 2024-03-12T05:59:28.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.613+0000 7fc39e7fc640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc3940a1c20 0x7fc3940a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.613+0000 7fc3b1ff8740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbb52b0 0x56159cbb5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.613+0000 7fc3b1ff8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb5bd0 0x56159cbbe270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.613+0000 7fc3b1ff8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbbe7b0 0x56159cbc0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.613+0000 7fc3b1ff8740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56159cb24280 con 0x56159cbb52b0 2024-03-12T05:59:28.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.613+0000 7fc3b1ff8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56159cb217b0 con 0x56159cbb5bd0 2024-03-12T05:59:28.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.613+0000 7fc3b1ff8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56159cb1cf90 con 0x56159cbbe7b0 2024-03-12T05:59:28.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ad8e5640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc3940a1c20 0x7fc3940a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ac8e3640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb5bd0 0x56159cbbe270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ac8e3640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb5bd0 0x56159cbbe270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:48160/0 (socket says 172.21.15.76:48160) 2024-03-12T05:59:28.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ac8e3640 1 -- 172.21.15.76:0/3660407394 learned_addr learned my addr 172.21.15.76:0/3660407394 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:28.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc39e7fc640 1 -- 172.21.15.76:0/2115896461 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc3a4014070 con 0x7fc3a8100260 2024-03-12T05:59:28.622 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ad8e5640 1 --2- 172.21.15.76:0/3660407394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbbe7b0 0x56159cbc0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ad0e4640 1 --2- 172.21.15.76:0/3660407394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbb52b0 0x56159cbb5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ac8e3640 1 -- 172.21.15.76:0/3660407394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbbe7b0 msgr2=0x56159cbc0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ac8e3640 1 --2- 172.21.15.76:0/3660407394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbbe7b0 0x56159cbc0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ac8e3640 1 -- 172.21.15.76:0/3660407394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbb52b0 msgr2=0x56159cbb5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ac8e3640 1 --2- 172.21.15.76:0/3660407394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbb52b0 0x56159cbb5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ac8e3640 1 -- 172.21.15.76:0/3660407394 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56159cadb570 con 0x56159cbb5bd0 2024-03-12T05:59:28.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ad0e4640 1 --2- 172.21.15.76:0/3660407394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbb52b0 0x56159cbb5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:28.623 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ad8e5640 1 --2- 172.21.15.76:0/3660407394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbbe7b0 0x56159cbc0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:28.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ac8e3640 1 --2- 172.21.15.76:0/3660407394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb5bd0 0x56159cbbe270 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc39802c180 tx=0x7fc3980323c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=909ec8018e40f4d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:28.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3ad8e5640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc3940a1c20 0x7fc3940a40e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x56159cbb0b70 tx=0x7fc3a000b710 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:28.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3937fe640 1 -- 172.21.15.76:0/3660407394 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398004180 con 0x56159cbb5bd0 2024-03-12T05:59:28.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.614+0000 7fc3937fe640 1 -- 172.21.15.76:0/3660407394 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc398034920 con 0x56159cbb5bd0 2024-03-12T05:59:28.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.615+0000 7fc3937fe640 1 -- 172.21.15.76:0/3660407394 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39802cb60 con 0x56159cbb5bd0 2024-03-12T05:59:28.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.615+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/3660407394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb5bd0 msgr2=0x56159cbbe270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.615+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/3660407394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb5bd0 0x56159cbbe270 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc39802c180 tx=0x7fc3980323c0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.616+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/3660407394 shutdown_connections 2024-03-12T05:59:28.624 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.616+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/3660407394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbbe7b0 0x56159cbc0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.616+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/3660407394 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb5bd0 0x56159cbbe270 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.616+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/3660407394 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbb52b0 0x56159cbb5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.616+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/3660407394 >> 172.21.15.76:0/3660407394 conn(0x56159cbaf990 msgr2=0x56159cbafd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:28.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.616+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/3660407394 shutdown_connections 2024-03-12T05:59:28.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.616+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/3660407394 wait complete. 2024-03-12T05:59:28.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.616+0000 7fc3b1ff8740 1 -- start start 2024-03-12T05:59:28.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.616+0000 7fc3b1ff8740 1 Processor -- start 2024-03-12T05:59:28.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.616+0000 7fc3b1ff8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb52b0 0x56159cbbe1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.616+0000 7fc3b1ff8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb5bd0 0x56159cbbb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3b1ff8740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbbe7b0 0x56159cbbbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3b1ff8740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56159c7f0430 con 0x56159cbbe7b0 2024-03-12T05:59:28.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3b1ff8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56159c7ace00 con 0x56159cbb52b0 2024-03-12T05:59:28.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3b1ff8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56159caa1fd0 con 0x56159cbb5bd0 2024-03-12T05:59:28.629 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3ac8e3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb5bd0 0x56159cbbb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3b1ff8740 10 client.0 _ll_register_callbacks cb 0x56159caf7fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:28.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3ac8e3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb5bd0 0x56159cbbb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.76:44352/0 (socket says 172.21.15.76:44352) 2024-03-12T05:59:28.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3ac8e3640 1 -- 172.21.15.76:0/2761729784 learned_addr learned my addr 172.21.15.76:0/2761729784 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:28.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3ad0e4640 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb52b0 0x56159cbbe1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3ad8e5640 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbbe7b0 0x56159cbbbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3ad0e4640 1 -- 172.21.15.76:0/2761729784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb5bd0 msgr2=0x56159cbbb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3ad0e4640 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb5bd0 0x56159cbbb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.630 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3ad0e4640 1 -- 172.21.15.76:0/2761729784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbbe7b0 msgr2=0x56159cbbbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3ad0e4640 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbbe7b0 0x56159cbbbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3ad0e4640 1 -- 172.21.15.76:0/2761729784 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a810d710 con 0x56159cbb52b0 2024-03-12T05:59:28.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3ac8e3640 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb5bd0 0x56159cbbb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:59:28.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3ad8e5640 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbbe7b0 0x56159cbbbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:28.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc3ad0e4640 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb52b0 0x56159cbbe1c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40cb1a0 tx=0x7fc3a409a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bceab234b5100f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:28.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.617+0000 7fc390ff9640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T05:59:28.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc390ff9640 1 -- 172.21.15.76:0/2761729784 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4013b20 con 0x56159cbb52b0 2024-03-12T05:59:28.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc390ff9640 1 -- 172.21.15.76:0/2761729784 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc3a409a680 con 0x56159cbb52b0 2024-03-12T05:59:28.631 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/2761729784 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56159c81c7e0 con 0x56159cbb52b0 2024-03-12T05:59:28.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc390ff9640 1 -- 172.21.15.76:0/2761729784 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3a409e7d0 con 0x56159cbb52b0 2024-03-12T05:59:28.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc3b1ff8740 10 client.25778 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:28.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/2761729784 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56159cb1b610 con 0x56159cbb52b0 2024-03-12T05:59:28.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc390ff9640 1 -- 172.21.15.76:0/2761729784 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56159c81c7e0 con 0x56159cbb52b0 2024-03-12T05:59:28.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc390ff9640 1 -- 172.21.15.76:0/2761729784 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fc3a401f060 con 0x56159cbb52b0 2024-03-12T05:59:28.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc3b1ff8740 10 client.25778 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:28.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc3b1ff8740 10 client.25778 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:59:28.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc3b1ff8740 20 client.25778 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:28.632 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56159cbce940 0x56159cbd0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc3b1ff8740 4 client.25778 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:59:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/2761729784 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56159c859720 con 0x56159cbce940 2024-03-12T05:59:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.618+0000 7fc3ac8e3640 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56159cbce940 0x56159cbd0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.619+0000 7fc390ff9640 10 client.25778 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.619+0000 7fc3ac8e3640 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56159cbce940 0x56159cbd0d60 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8432bbb22cf677e server_cookie=216b9a1fb8f9512f in_seq=0 out_seq=0 2024-03-12T05:59:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc390ff9640 1 -- 172.21.15.76:0/2761729784 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56159c859720 con 0x56159cbce940 2024-03-12T05:59:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc390ff9640 10 client.25778 handle_command_reply: tid=0 2024-03-12T05:59:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc3b1ff8740 1 client.25778 shutdown 2024-03-12T05:59:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/2761729784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb52b0 msgr2=0x56159cbbe1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb52b0 0x56159cbbe1c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40cb1a0 tx=0x7fc3a409a0a0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/2761729784 shutdown_connections 2024-03-12T05:59:28.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56159cbce940 0x56159cbd0d60 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbbe7b0 0x56159cbbbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb5bd0 0x56159cbbb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/2761729784 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb52b0 0x56159cbbe1c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc390ff9640 0 client.25778 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:28.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/2761729784 >> 172.21.15.76:0/2761729784 conn(0x56159cbaf990 msgr2=0x56159cbb1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:28.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/2761729784 shutdown_connections 2024-03-12T05:59:28.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/2761729784 wait complete. 2024-03-12T05:59:28.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.620+0000 7fc3b1ff8740 20 client.25778 trim_cache size 0 max 16384 2024-03-12T05:59:28.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3b1ff8740 1 Processor -- start 2024-03-12T05:59:28.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3b1ff8740 1 -- start start 2024-03-12T05:59:28.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3b1ff8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb52b0 0x56159cbb5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3b1ff8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb5bd0 0x56159cbbdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3b1ff8740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbbe7b0 0x56159cbbe1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3b1ff8740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56159cbcadb0 con 0x56159cbbe7b0 2024-03-12T05:59:28.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3b1ff8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56159cbab0f0 con 0x56159cbb5bd0 2024-03-12T05:59:28.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3b1ff8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56159cbcb3f0 con 0x56159cbb52b0 2024-03-12T05:59:28.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3ad8e5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbbe7b0 0x56159cbbe1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3ad8e5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbbe7b0 0x56159cbbe1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:48322/0 (socket says 172.21.15.76:48322) 2024-03-12T05:59:28.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3ac8e3640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb5bd0 0x56159cbbdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3ad8e5640 1 -- 172.21.15.76:0/1597145610 learned_addr learned my addr 172.21.15.76:0/1597145610 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:28.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3ad0e4640 1 --2- 172.21.15.76:0/1597145610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb52b0 0x56159cbb5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3ac8e3640 1 -- 172.21.15.76:0/1597145610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb52b0 msgr2=0x56159cbb5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3ac8e3640 1 --2- 172.21.15.76:0/1597145610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb52b0 0x56159cbb5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3ac8e3640 1 -- 172.21.15.76:0/1597145610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbbe7b0 msgr2=0x56159cbbe1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3ac8e3640 1 --2- 172.21.15.76:0/1597145610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbbe7b0 0x56159cbbe1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3ac8e3640 1 -- 172.21.15.76:0/1597145610 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56159cab7ac0 con 0x56159cbb5bd0 2024-03-12T05:59:28.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.640+0000 7fc3ad8e5640 1 --2- 172.21.15.76:0/1597145610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbbe7b0 0x56159cbbe1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T05:59:28.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.641+0000 7fc3ac8e3640 1 --2- 172.21.15.76:0/1597145610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb5bd0 0x56159cbbdc80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc39802bbb0 tx=0x7fc398052040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d15d8f6be5732077 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:28.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.641+0000 7fc392ffd640 1 -- 172.21.15.76:0/1597145610 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39802cbf0 con 0x56159cbb5bd0 2024-03-12T05:59:28.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.641+0000 7fc392ffd640 1 -- 172.21.15.76:0/1597145610 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc39802c610 con 0x56159cbb5bd0 2024-03-12T05:59:28.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.641+0000 7fc392ffd640 1 -- 172.21.15.76:0/1597145610 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3980038e0 con 0x56159cbb5bd0 2024-03-12T05:59:28.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.642+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/1597145610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb5bd0 msgr2=0x56159cbbdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.642+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/1597145610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb5bd0 0x56159cbbdc80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc39802bbb0 tx=0x7fc398052040 comp rx=0 tx=0).stop 2024-03-12T05:59:28.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.642+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/1597145610 shutdown_connections 2024-03-12T05:59:28.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.642+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/1597145610 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbbe7b0 0x56159cbbe1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.642+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/1597145610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbb5bd0 0x56159cbbdc80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.642+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/1597145610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb52b0 0x56159cbb5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.642+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/1597145610 >> 172.21.15.76:0/1597145610 conn(0x56159cbaf990 msgr2=0x56159cbd5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:28.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.642+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/1597145610 shutdown_connections 2024-03-12T05:59:28.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.642+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/1597145610 wait complete. 2024-03-12T05:59:28.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.642+0000 7fc3b1ff8740 1 -- start start 2024-03-12T05:59:28.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.642+0000 7fc3b1ff8740 1 Processor -- start 2024-03-12T05:59:28.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3b1ff8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb52b0 0x56159cbc1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3b1ff8740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbb5bd0 0x56159cbc1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3b1ff8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbbe7b0 0x56159cc1de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3b1ff8740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56159cc1e380 con 0x56159cbb5bd0 2024-03-12T05:59:28.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3b1ff8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56159cc1e4c0 con 0x56159cbbe7b0 2024-03-12T05:59:28.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3b1ff8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56159cc1e680 con 0x56159cbb52b0 2024-03-12T05:59:28.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3ad8e5640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbbe7b0 0x56159cc1de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3ad8e5640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbbe7b0 0x56159cc1de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:48182/0 (socket says 172.21.15.76:48182) 2024-03-12T05:59:28.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3ad8e5640 1 -- 172.21.15.76:0/1125784424 learned_addr learned my addr 172.21.15.76:0/1125784424 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:28.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3ad0e4640 1 --2- 172.21.15.76:0/1125784424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb52b0 0x56159cbc1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3b1ff8740 10 client.0 _ll_register_callbacks cb 0x7fc3a4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:28.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3ac8e3640 1 --2- 172.21.15.76:0/1125784424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbb5bd0 0x56159cbc1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3ad8e5640 1 -- 172.21.15.76:0/1125784424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb52b0 msgr2=0x56159cbc1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3ad8e5640 1 --2- 172.21.15.76:0/1125784424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb52b0 0x56159cbc1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3ad8e5640 1 -- 172.21.15.76:0/1125784424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbb5bd0 msgr2=0x56159cbc1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:28.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3ad8e5640 1 --2- 172.21.15.76:0/1125784424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbb5bd0 0x56159cbc1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.643+0000 7fc3ad8e5640 1 -- 172.21.15.76:0/1125784424 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a00187d0 con 0x56159cbbe7b0 2024-03-12T05:59:28.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.644+0000 7fc3ad8e5640 1 --2- 172.21.15.76:0/1125784424 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbbe7b0 0x56159cc1de40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0018cb0 tx=0x7fc3a000d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f82c88b086dbccd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:28.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.644+0000 7fc390ff9640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T05:59:28.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.644+0000 7fc390ff9640 1 -- 172.21.15.76:0/1125784424 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0010040 con 0x56159cbbe7b0 2024-03-12T05:59:28.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.644+0000 7fc390ff9640 1 -- 172.21.15.76:0/1125784424 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc3a0028480 con 0x56159cbbe7b0 2024-03-12T05:59:28.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.644+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/1125784424 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc398034920 con 0x56159cbbe7b0 2024-03-12T05:59:28.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.644+0000 7fc390ff9640 1 -- 172.21.15.76:0/1125784424 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3a0028640 con 0x56159cbbe7b0 2024-03-12T05:59:28.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.644+0000 7fc3b1ff8740 10 client.25784 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:28.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.644+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/1125784424 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56159cadb570 con 0x56159cbbe7b0 2024-03-12T05:59:28.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.644+0000 7fc390ff9640 1 -- 172.21.15.76:0/1125784424 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3a0022040 con 0x56159cbbe7b0 2024-03-12T05:59:28.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.645+0000 7fc390ff9640 1 -- 172.21.15.76:0/1125784424 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fc3a0030070 con 0x56159cbbe7b0 2024-03-12T05:59:28.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.645+0000 7fc3b1ff8740 10 client.25784 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:28.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.645+0000 7fc3b1ff8740 10 client.25784 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:59:28.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.645+0000 7fc3b1ff8740 20 client.25784 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:28.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.645+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/1125784424 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56159cc25fa0 0x56159cc283c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.645+0000 7fc3b1ff8740 4 client.25784 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.645+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/1125784424 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fc398034920 con 0x56159cc25fa0 2024-03-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.645+0000 7fc3ad0e4640 1 --2- 172.21.15.76:0/1125784424 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56159cc25fa0 0x56159cc283c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc390ff9640 10 client.25784 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc3ad0e4640 1 --2- 172.21.15.76:0/1125784424 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56159cc25fa0 0x56159cc283c0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d26ae9d38897195 server_cookie=d7d824f67f6840a5 in_seq=0 out_seq=0 2024-03-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc390ff9640 1 -- 172.21.15.76:0/1125784424 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7fc398034920 con 0x56159cc25fa0 2024-03-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc390ff9640 10 client.25784 handle_command_reply: tid=0 2024-03-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc3b1ff8740 1 client.25784 shutdown 2024-03-12T05:59:28.657 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/1125784424 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbbe7b0 msgr2=0x56159cc1de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/1125784424 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbbe7b0 0x56159cc1de40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0018cb0 tx=0x7fc3a000d040 comp rx=0 tx=0).stop 2024-03-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/1125784424 shutdown_connections 2024-03-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/1125784424 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56159cbbe7b0 0x56159cc1de40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/1125784424 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56159cc25fa0 0x56159cc283c0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/1125784424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56159cbb5bd0 0x56159cbc1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc3b1ff8740 1 --2- 172.21.15.76:0/1125784424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56159cbb52b0 0x56159cbc1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc390ff9640 0 client.25784 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/1125784424 >> 172.21.15.76:0/1125784424 conn(0x56159cbaf990 msgr2=0x56159cbb6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:28.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/1125784424 shutdown_connections 2024-03-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc3b1ff8740 1 -- 172.21.15.76:0/1125784424 wait complete. 2024-03-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.646+0000 7fc3b1ff8740 20 client.25784 trim_cache size 0 max 16384 2024-03-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T05:59:28.659 INFO:teuthology.orchestra.run.smithi076.stdout: "8e4fd6df-87c6-48f5-9526-cc7474ef2c17": { 2024-03-12T05:59:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T05:59:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "8e4fd6df-87c6-48f5-9526-cc7474ef2c17", 2024-03-12T05:59:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T05:59:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:59:28.660 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T05:59:28.660 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.647+0000 7fc3af348640 1 -- 172.21.15.76:0/2115896461 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc3940a1c20 msgr2=0x7fc3940a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.647+0000 7fc3af348640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc3940a1c20 0x7fc3940a40e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x56159cbb0b70 tx=0x7fc3a000b710 comp rx=0 tx=0).stop 2024-03-12T05:59:28.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.647+0000 7fc3af348640 1 -- 172.21.15.76:0/2115896461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3a8100260 msgr2=0x7fc3a819f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:28.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.647+0000 7fc3af348640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3a8100260 0x7fc3a819f670 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400e9f0 tx=0x7fc3a400eeb0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.647+0000 7fc3af348640 1 -- 172.21.15.76:0/2115896461 shutdown_connections 2024-03-12T05:59:28.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.647+0000 7fc3af348640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc3940a1c20 0x7fc3940a40e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.647+0000 7fc3af348640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc3a81997e0 0x7fc3a8199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.647+0000 7fc3af348640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3a8105330 0x7fc3a819fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.647+0000 7fc3af348640 1 --2- 172.21.15.76:0/2115896461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc3a8100260 0x7fc3a819f670 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:28.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.647+0000 7fc3af348640 1 -- 172.21.15.76:0/2115896461 >> 172.21.15.76:0/2115896461 conn(0x7fc3a8077180 msgr2=0x7fc3a80ff730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:28.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.647+0000 7fc3af348640 1 -- 172.21.15.76:0/2115896461 shutdown_connections 2024-03-12T05:59:28.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:28.647+0000 7fc3af348640 1 -- 172.21.15.76:0/2115896461 wait complete. 2024-03-12T05:59:28.664 INFO:tasks.cephfs.filesystem:scrub status for tag:8e4fd6df-87c6-48f5-9526-cc7474ef2c17 - {'path': '/', 'tag': '8e4fd6df-87c6-48f5-9526-cc7474ef2c17', 'options': 'recursive,force'} 2024-03-12T05:59:29.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:29 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:59:29.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:29 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:59:29.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:29 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:59:29.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:29 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:59:29.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:29 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:59:29.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:29 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:59:29.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:29 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:59:29.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:29 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:59:29.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:29 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:59:29.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:29 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:59:29.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:29 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:59:29.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:29 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:59:30.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:30 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T05:59:30.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:30 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:59:30.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:30 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T05:59:30.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:30 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T05:59:30.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:30 smithi082 ceph-mon[36382]: pgmap v114: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:59:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:30 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T05:59:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:30 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:59:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:30 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T05:59:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:30 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T05:59:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:30 smithi088 ceph-mon[36585]: pgmap v114: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:59:30.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:30 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T05:59:30.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:30 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:59:30.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:30 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T05:59:30.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:30 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T05:59:30.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:30 smithi076 ceph-mon[32095]: pgmap v114: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:59:31.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:31 smithi082 ceph-mon[36382]: pgmap v115: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:31.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:31 smithi088 ceph-mon[36585]: pgmap v115: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:31.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:31 smithi076 ceph-mon[32095]: pgmap v115: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:34.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:34 smithi082 ceph-mon[36382]: pgmap v116: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:59:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:34 smithi088 ceph-mon[36585]: pgmap v116: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:59:34.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:34 smithi076 ceph-mon[32095]: pgmap v116: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:59:36.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:36 smithi082 ceph-mon[36382]: pgmap v117: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:59:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:36 smithi088 ceph-mon[36585]: pgmap v117: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:59:36.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:36 smithi076 ceph-mon[32095]: pgmap v117: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:59:38.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:38 smithi082 ceph-mon[36382]: pgmap v118: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:59:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:38 smithi088 ceph-mon[36585]: pgmap v118: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:59:38.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:38 smithi076 ceph-mon[32095]: pgmap v118: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:59:40.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:40 smithi082 ceph-mon[36382]: pgmap v119: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:59:40.484 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:40 smithi088 ceph-mon[36585]: pgmap v119: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:59:40.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:40 smithi076 ceph-mon[32095]: pgmap v119: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:59:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:42 smithi082 ceph-mon[36382]: pgmap v120: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:59:42.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:42 smithi088 ceph-mon[36585]: pgmap v120: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:59:42.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:42 smithi076 ceph-mon[32095]: pgmap v120: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T05:59:43.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:43 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:59:43.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:43 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:59:43.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:43 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T05:59:43.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:43 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:59:43.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:43 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:59:43.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:43 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T05:59:43.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:43 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:59:43.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:43 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:59:43.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:43 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T05:59:44.390 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:44 smithi076 ceph-mon[32095]: pgmap v121: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:59:44.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:44 smithi082 ceph-mon[36382]: pgmap v121: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:59:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:44 smithi088 ceph-mon[36585]: pgmap v121: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T05:59:46.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:46 smithi082 ceph-mon[36382]: pgmap v122: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:46.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:46 smithi088 ceph-mon[36585]: pgmap v122: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:46.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:46 smithi076 ceph-mon[32095]: pgmap v122: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:48.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:48 smithi082 ceph-mon[36382]: pgmap v123: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:48 smithi088 ceph-mon[36585]: pgmap v123: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:48.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:48 smithi076 ceph-mon[32095]: pgmap v123: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:50.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:50 smithi082 ceph-mon[36382]: pgmap v124: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:59:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:50 smithi088 ceph-mon[36585]: pgmap v124: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:59:50.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:50 smithi076 ceph-mon[32095]: pgmap v124: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:59:52.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:52 smithi082 ceph-mon[36382]: pgmap v125: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:52.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:52 smithi088 ceph-mon[36585]: pgmap v125: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:52.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:52 smithi076 ceph-mon[32095]: pgmap v125: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:53 smithi088 ceph-mon[36585]: pgmap v126: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:59:53.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:53 smithi076 ceph-mon[32095]: pgmap v126: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:59:53.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:53 smithi082 ceph-mon[36382]: pgmap v126: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T05:59:56.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:56 smithi082 ceph-mon[36382]: pgmap v127: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:56.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:56 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:59:56.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:56 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:59:56.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:56 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:59:56.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:56 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:59:56.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:56 smithi088 ceph-mon[36585]: pgmap v127: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:56.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:56 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:59:56.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:56 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:59:56.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:56 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:59:56.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:56 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:59:56.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:56 smithi076 ceph-mon[32095]: pgmap v127: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:56.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:56 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T05:59:56.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:56 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T05:59:56.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:56 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T05:59:56.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:56 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T05:59:58.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 05:59:58 smithi082 ceph-mon[36382]: pgmap v128: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:58.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 05:59:58 smithi088 ceph-mon[36585]: pgmap v128: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:58.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 05:59:58 smithi076 ceph-mon[32095]: pgmap v128: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T05:59:58.665 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T05:59:58.665 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T05:59:58.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.779+0000 7f57c898c640 1 -- 172.21.15.76:0/1765525236 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c4105330 msgr2=0x7f57c410cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:58.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.779+0000 7f57c898c640 1 --2- 172.21.15.76:0/1765525236 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c4105330 0x7f57c410cf00 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009cd0 tx=0x7f57bc02c0f0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.779+0000 7f57c898c640 1 -- 172.21.15.76:0/1765525236 shutdown_connections 2024-03-12T05:59:58.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.779+0000 7f57c898c640 1 --2- 172.21.15.76:0/1765525236 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c4105330 0x7f57c410cf00 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.779+0000 7f57c898c640 1 --2- 172.21.15.76:0/1765525236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57c4100c10 0x7f57c4104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.779+0000 7f57c898c640 1 --2- 172.21.15.76:0/1765525236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c4100260 0x7f57c4100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.779+0000 7f57c898c640 1 -- 172.21.15.76:0/1765525236 >> 172.21.15.76:0/1765525236 conn(0x7f57c4077180 msgr2=0x7f57c4077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:58.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.779+0000 7f57c898c640 1 -- 172.21.15.76:0/1765525236 shutdown_connections 2024-03-12T05:59:58.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.779+0000 7f57c898c640 1 -- 172.21.15.76:0/1765525236 wait complete. 2024-03-12T05:59:58.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.780+0000 7f57c898c640 1 Processor -- start 2024-03-12T05:59:58.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.780+0000 7f57c898c640 1 -- start start 2024-03-12T05:59:58.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.780+0000 7f57c898c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57c4100260 0x7f57c41965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.780+0000 7f57c898c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c4105330 0x7f57c4196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.780+0000 7f57c898c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c419aea0 0x7f57c419b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.780+0000 7f57c898c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c419b890 con 0x7f57c419aea0 2024-03-12T05:59:58.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.780+0000 7f57c898c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c419ba00 con 0x7f57c4100260 2024-03-12T05:59:58.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.780+0000 7f57c898c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c419bb70 con 0x7f57c4105330 2024-03-12T05:59:58.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.780+0000 7f57c2d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c419aea0 0x7f57c419b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.780+0000 7f57c2d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c419aea0 0x7f57c419b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39730/0 (socket says 172.21.15.76:39730) 2024-03-12T05:59:58.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.780+0000 7f57c2d9d640 1 -- 172.21.15.76:0/2604398526 learned_addr learned my addr 172.21.15.76:0/2604398526 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:58.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.780+0000 7f57c259c640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57c4100260 0x7f57c41965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.780+0000 7f57bbfff640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c4105330 0x7f57c4196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.781+0000 7f57c2d9d640 1 -- 172.21.15.76:0/2604398526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c4105330 msgr2=0x7f57c4196b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:58.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.781+0000 7f57c2d9d640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c4105330 0x7f57c4196b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.781+0000 7f57c2d9d640 1 -- 172.21.15.76:0/2604398526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57c4100260 msgr2=0x7f57c41965e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:58.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.781+0000 7f57c2d9d640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57c4100260 0x7f57c41965e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.781+0000 7f57c2d9d640 1 -- 172.21.15.76:0/2604398526 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc009900 con 0x7f57c419aea0 2024-03-12T05:59:58.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.781+0000 7f57bbfff640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c4105330 0x7f57c4196b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:58.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.781+0000 7f57c2d9d640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c419aea0 0x7f57c419b350 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f57bc004af0 tx=0x7f57bc034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:58.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.781+0000 7f57bb7fe640 1 -- 172.21.15.76:0/2604398526 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0328c0 con 0x7f57c419aea0 2024-03-12T05:59:58.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.781+0000 7f57bb7fe640 1 -- 172.21.15.76:0/2604398526 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57bc003e50 con 0x7f57c419aea0 2024-03-12T05:59:58.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.781+0000 7f57c898c640 1 -- 172.21.15.76:0/2604398526 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c41a3150 con 0x7f57c419aea0 2024-03-12T05:59:58.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.781+0000 7f57bb7fe640 1 -- 172.21.15.76:0/2604398526 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc004860 con 0x7f57c419aea0 2024-03-12T05:59:58.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.781+0000 7f57c898c640 1 -- 172.21.15.76:0/2604398526 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c41a35d0 con 0x7f57c419aea0 2024-03-12T05:59:58.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.783+0000 7f57bb7fe640 1 -- 172.21.15.76:0/2604398526 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f57bc00a040 con 0x7f57c419aea0 2024-03-12T05:59:58.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57cb1fd740 1 Processor -- start 2024-03-12T05:59:58.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57bb7fe640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f57b00a1c20 0x7f57b00a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57bbfff640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f57b00a1c20 0x7f57b00a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57bb7fe640 1 -- 172.21.15.76:0/2604398526 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f57bc0f03a0 con 0x7f57c419aea0 2024-03-12T05:59:58.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57cb1fd740 1 -- start start 2024-03-12T05:59:58.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57cb1fd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876e04d0 0x564d876e08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57cb1fd740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876dbab0 0x564d876dbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57cb1fd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 0x564d876e8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57cb1fd740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d87647280 con 0x564d876dbab0 2024-03-12T05:59:58.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57cb1fd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d876447b0 con 0x564d876e04d0 2024-03-12T05:59:58.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57cb1fd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d8763ff90 con 0x564d876dc470 2024-03-12T05:59:58.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57bbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876dbab0 0x564d876dbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57bbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876dbab0 0x564d876dbf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39746/0 (socket says 172.21.15.76:39746) 2024-03-12T05:59:58.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57bbfff640 1 -- 172.21.15.76:0/889752834 learned_addr learned my addr 172.21.15.76:0/889752834 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:58.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.784+0000 7f57bbfff640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f57b00a1c20 0x7f57b00a40e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f57ac005fd0 tx=0x7f57ac005850 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:58.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.785+0000 7f57c259c640 1 --2- 172.21.15.76:0/889752834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876e04d0 0x564d876e08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.785+0000 7f57c2d9d640 1 --2- 172.21.15.76:0/889752834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 0x564d876e8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.785+0000 7f57bbfff640 1 -- 172.21.15.76:0/889752834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 msgr2=0x564d876e8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:58.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.785+0000 7f57bbfff640 1 --2- 172.21.15.76:0/889752834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 0x564d876e8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.785+0000 7f57bbfff640 1 -- 172.21.15.76:0/889752834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876e04d0 msgr2=0x564d876e08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:58.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.785+0000 7f57bbfff640 1 --2- 172.21.15.76:0/889752834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876e04d0 0x564d876e08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.785+0000 7f57bbfff640 1 -- 172.21.15.76:0/889752834 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d875fe570 con 0x564d876dbab0 2024-03-12T05:59:58.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.785+0000 7f57c259c640 1 --2- 172.21.15.76:0/889752834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876e04d0 0x564d876e08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:58.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.785+0000 7f57bbfff640 1 --2- 172.21.15.76:0/889752834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876dbab0 0x564d876dbf30 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00bf90 tx=0x7f57ac012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43311acc8ce9b022 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:58.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.785+0000 7f57b8ff9640 1 -- 172.21.15.76:0/889752834 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac004030 con 0x564d876dbab0 2024-03-12T05:59:58.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.785+0000 7f57b8ff9640 1 -- 172.21.15.76:0/889752834 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57ac004190 con 0x564d876dbab0 2024-03-12T05:59:58.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.785+0000 7f57b8ff9640 1 -- 172.21.15.76:0/889752834 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac01e740 con 0x564d876dbab0 2024-03-12T05:59:58.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.786+0000 7f57cb1fd740 1 -- 172.21.15.76:0/889752834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876dbab0 msgr2=0x564d876dbf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:58.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.786+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/889752834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876dbab0 0x564d876dbf30 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00bf90 tx=0x7f57ac012000 comp rx=0 tx=0).stop 2024-03-12T05:59:58.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.786+0000 7f57cb1fd740 1 -- 172.21.15.76:0/889752834 shutdown_connections 2024-03-12T05:59:58.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.786+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/889752834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 0x564d876e8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.786+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/889752834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876dbab0 0x564d876dbf30 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.786+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/889752834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876e04d0 0x564d876e08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.786+0000 7f57cb1fd740 1 -- 172.21.15.76:0/889752834 >> 172.21.15.76:0/889752834 conn(0x564d876d2690 msgr2=0x564d876d2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:58.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.786+0000 7f57cb1fd740 1 -- 172.21.15.76:0/889752834 shutdown_connections 2024-03-12T05:59:58.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.786+0000 7f57cb1fd740 1 -- 172.21.15.76:0/889752834 wait complete. 2024-03-12T05:59:58.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.786+0000 7f57cb1fd740 1 -- start start 2024-03-12T05:59:58.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.786+0000 7f57cb1fd740 1 Processor -- start 2024-03-12T05:59:58.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57cb1fd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876dbab0 0x564d876e82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57cb1fd740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876dc470 0x564d876e5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57cb1fd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876e04d0 0x564d876e5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57cb1fd740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d87313430 con 0x564d876dc470 2024-03-12T05:59:58.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57cb1fd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d872cfe00 con 0x564d876dbab0 2024-03-12T05:59:58.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57cb1fd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d875c4fd0 con 0x564d876e04d0 2024-03-12T05:59:58.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57cb1fd740 10 client.0 _ll_register_callbacks cb 0x564d8761afd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:58.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57bbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876dc470 0x564d876e5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57bbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876dc470 0x564d876e5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39758/0 (socket says 172.21.15.76:39758) 2024-03-12T05:59:58.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57bbfff640 1 -- 172.21.15.76:0/3902109119 learned_addr learned my addr 172.21.15.76:0/3902109119 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:58.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57c2d9d640 1 --2- 172.21.15.76:0/3902109119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876e04d0 0x564d876e5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57c259c640 1 --2- 172.21.15.76:0/3902109119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876dbab0 0x564d876e82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57bbfff640 1 -- 172.21.15.76:0/3902109119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876e04d0 msgr2=0x564d876e5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:58.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57bbfff640 1 --2- 172.21.15.76:0/3902109119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876e04d0 0x564d876e5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57bbfff640 1 -- 172.21.15.76:0/3902109119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876dbab0 msgr2=0x564d876e82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:58.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57bbfff640 1 --2- 172.21.15.76:0/3902109119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876dbab0 0x564d876e82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57bbfff640 1 -- 172.21.15.76:0/3902109119 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57ac00bc40 con 0x564d876dc470 2024-03-12T05:59:58.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57bbfff640 1 --2- 172.21.15.76:0/3902109119 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876dc470 0x564d876e5640 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f57ac003e70 tx=0x7f57ac004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bff947b80ddd539 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:58.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:58.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57a9ffb640 1 -- 172.21.15.76:0/3902109119 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac02e070 con 0x564d876dc470 2024-03-12T05:59:58.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57a9ffb640 1 -- 172.21.15.76:0/3902109119 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f57ac003880 con 0x564d876dc470 2024-03-12T05:59:58.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.787+0000 7f57cb1fd740 1 -- 172.21.15.76:0/3902109119 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d8733f7e0 con 0x564d876dc470 2024-03-12T05:59:58.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.788+0000 7f57a9ffb640 1 -- 172.21.15.76:0/3902109119 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57ac003a40 con 0x564d876dc470 2024-03-12T05:59:58.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.788+0000 7f57cb1fd740 10 client.16449 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:58.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.788+0000 7f57cb1fd740 1 -- 172.21.15.76:0/3902109119 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564d8763e610 con 0x564d876dc470 2024-03-12T05:59:58.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.788+0000 7f57a9ffb640 1 -- 172.21.15.76:0/3902109119 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564d8733f7e0 con 0x564d876dc470 2024-03-12T05:59:58.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.788+0000 7f57a9ffb640 1 -- 172.21.15.76:0/3902109119 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f57ac033540 con 0x564d876dc470 2024-03-12T05:59:58.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.788+0000 7f57cb1fd740 10 client.16449 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:58.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.788+0000 7f57cb1fd740 10 client.16449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:59:58.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.788+0000 7f57cb1fd740 20 client.16449 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:58.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.788+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/3902109119 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x564d876f5c40 0x564d876f8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.788+0000 7f57cb1fd740 4 client.16449 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:59:58.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.788+0000 7f57cb1fd740 1 -- 172.21.15.76:0/3902109119 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d8737c720 con 0x564d876f5c40 2024-03-12T05:59:58.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.788+0000 7f57c259c640 1 --2- 172.21.15.76:0/3902109119 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x564d876f5c40 0x564d876f8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.789+0000 7f57c259c640 1 --2- 172.21.15.76:0/3902109119 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x564d876f5c40 0x564d876f8060 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4317f6a5a82547c2 server_cookie=d1f0a963d6428abe in_seq=0 out_seq=0 2024-03-12T05:59:58.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.789+0000 7f57a9ffb640 10 client.16449 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:58.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.789+0000 7f57a9ffb640 1 -- 172.21.15.76:0/3902109119 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564d8737c720 con 0x564d876f5c40 2024-03-12T05:59:58.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.789+0000 7f57a9ffb640 10 client.16449 handle_command_reply: tid=0 2024-03-12T05:59:58.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.790+0000 7f57cb1fd740 1 client.16449 shutdown 2024-03-12T05:59:58.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.790+0000 7f57cb1fd740 1 -- 172.21.15.76:0/3902109119 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876dc470 msgr2=0x564d876e5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:58.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.790+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/3902109119 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876dc470 0x564d876e5640 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f57ac003e70 tx=0x7f57ac004530 comp rx=0 tx=0).stop 2024-03-12T05:59:58.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.790+0000 7f57cb1fd740 1 -- 172.21.15.76:0/3902109119 shutdown_connections 2024-03-12T05:59:58.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.790+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/3902109119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876e04d0 0x564d876e5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.790+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/3902109119 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x564d876f5c40 0x564d876f8060 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.790+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/3902109119 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876dc470 0x564d876e5640 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.790+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/3902109119 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876dbab0 0x564d876e82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.790+0000 7f57a9ffb640 0 client.16449 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:58.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.790+0000 7f57cb1fd740 1 -- 172.21.15.76:0/3902109119 >> 172.21.15.76:0/3902109119 conn(0x564d876d2690 msgr2=0x564d876e1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:58.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.790+0000 7f57cb1fd740 1 -- 172.21.15.76:0/3902109119 shutdown_connections 2024-03-12T05:59:58.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.790+0000 7f57cb1fd740 1 -- 172.21.15.76:0/3902109119 wait complete. 2024-03-12T05:59:58.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.790+0000 7f57cb1fd740 20 client.16449 trim_cache size 0 max 16384 2024-03-12T05:59:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57cb1fd740 1 Processor -- start 2024-03-12T05:59:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57cb1fd740 1 -- start start 2024-03-12T05:59:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57cb1fd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876dbab0 0x564d876e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57cb1fd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 0x564d876e82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57cb1fd740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876e04d0 0x564d876ecaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57cb1fd740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d876fd230 con 0x564d876e04d0 2024-03-12T05:59:58.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57cb1fd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d876d39a0 con 0x564d876dbab0 2024-03-12T05:59:58.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57cb1fd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d876e3b90 con 0x564d876dc470 2024-03-12T05:59:58.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57c2d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876e04d0 0x564d876ecaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57c2d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876e04d0 0x564d876ecaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39770/0 (socket says 172.21.15.76:39770) 2024-03-12T05:59:58.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57c2d9d640 1 -- 172.21.15.76:0/2270598200 learned_addr learned my addr 172.21.15.76:0/2270598200 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:58.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57c2d9d640 1 -- 172.21.15.76:0/2270598200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 msgr2=0x564d876e82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:58.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57c2d9d640 1 --2- 172.21.15.76:0/2270598200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 0x564d876e82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57c2d9d640 1 -- 172.21.15.76:0/2270598200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876dbab0 msgr2=0x564d876e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:58.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57c2d9d640 1 --2- 172.21.15.76:0/2270598200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876dbab0 0x564d876e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57c2d9d640 1 -- 172.21.15.76:0/2270598200 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d875daac0 con 0x564d876e04d0 2024-03-12T05:59:58.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.805+0000 7f57c2d9d640 1 --2- 172.21.15.76:0/2270598200 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876e04d0 0x564d876ecaf0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0edce0 tx=0x7f57bc004300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=858c37475ad3a40d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:58.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.806+0000 7f57abfff640 1 -- 172.21.15.76:0/2270598200 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0bcb30 con 0x564d876e04d0 2024-03-12T05:59:58.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.806+0000 7f57abfff640 1 -- 172.21.15.76:0/2270598200 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57bc0bd8d0 con 0x564d876e04d0 2024-03-12T05:59:58.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.806+0000 7f57abfff640 1 -- 172.21.15.76:0/2270598200 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0b9840 con 0x564d876e04d0 2024-03-12T05:59:58.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.806+0000 7f57cb1fd740 1 -- 172.21.15.76:0/2270598200 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876e04d0 msgr2=0x564d876ecaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:58.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.806+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/2270598200 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876e04d0 0x564d876ecaf0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0edce0 tx=0x7f57bc004300 comp rx=0 tx=0).stop 2024-03-12T05:59:58.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.806+0000 7f57cb1fd740 1 -- 172.21.15.76:0/2270598200 shutdown_connections 2024-03-12T05:59:58.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.806+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/2270598200 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876e04d0 0x564d876ecaf0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.806+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/2270598200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 0x564d876e82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.806+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/2270598200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876dbab0 0x564d876e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.806+0000 7f57cb1fd740 1 -- 172.21.15.76:0/2270598200 >> 172.21.15.76:0/2270598200 conn(0x564d876d2690 msgr2=0x564d876de800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:58.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.806+0000 7f57cb1fd740 1 -- 172.21.15.76:0/2270598200 shutdown_connections 2024-03-12T05:59:58.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.806+0000 7f57cb1fd740 1 -- 172.21.15.76:0/2270598200 wait complete. 2024-03-12T05:59:58.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57cb1fd740 1 -- start start 2024-03-12T05:59:58.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57cb1fd740 1 Processor -- start 2024-03-12T05:59:58.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57cb1fd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876dbab0 0x564d876ebc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57cb1fd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 0x564d876ec1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57cb1fd740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876e04d0 0x564d8774c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57cb1fd740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d8774c670 con 0x564d876e04d0 2024-03-12T05:59:58.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57cb1fd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d8774c7b0 con 0x564d876dbab0 2024-03-12T05:59:58.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57cb1fd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d8774c8f0 con 0x564d876dc470 2024-03-12T05:59:58.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57cb1fd740 10 client.0 _ll_register_callbacks cb 0x7f57ac015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:58.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57c2d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876e04d0 0x564d8774c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57c2d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876e04d0 0x564d8774c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39780/0 (socket says 172.21.15.76:39780) 2024-03-12T05:59:58.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57c2d9d640 1 -- 172.21.15.76:0/4097372997 learned_addr learned my addr 172.21.15.76:0/4097372997 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:58.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57c259c640 1 --2- 172.21.15.76:0/4097372997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876dbab0 0x564d876ebc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57bbfff640 1 --2- 172.21.15.76:0/4097372997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 0x564d876ec1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57c2d9d640 1 -- 172.21.15.76:0/4097372997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 msgr2=0x564d876ec1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:58.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57c2d9d640 1 --2- 172.21.15.76:0/4097372997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 0x564d876ec1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57c2d9d640 1 -- 172.21.15.76:0/4097372997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876dbab0 msgr2=0x564d876ebc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:58.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57c2d9d640 1 --2- 172.21.15.76:0/4097372997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876dbab0 0x564d876ebc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57c2d9d640 1 -- 172.21.15.76:0/4097372997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c41a35d0 con 0x564d876e04d0 2024-03-12T05:59:58.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57bbfff640 1 --2- 172.21.15.76:0/4097372997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 0x564d876ec1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:58.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57c2d9d640 1 --2- 172.21.15.76:0/4097372997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876e04d0 0x564d8774c130 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0348f0 tx=0x7f57bc0c1230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9d761beef7d971 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:58.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.807+0000 7f57a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:58.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57a9ffb640 1 -- 172.21.15.76:0/4097372997 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0bd1d0 con 0x564d876e04d0 2024-03-12T05:59:58.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57a9ffb640 1 -- 172.21.15.76:0/4097372997 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f57bc0d5e40 con 0x564d876e04d0 2024-03-12T05:59:58.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57cb1fd740 1 -- 172.21.15.76:0/4097372997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57ac004190 con 0x564d876e04d0 2024-03-12T05:59:58.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57a9ffb640 1 -- 172.21.15.76:0/4097372997 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57bc0c1320 con 0x564d876e04d0 2024-03-12T05:59:58.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57cb1fd740 10 client.16455 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:58.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57cb1fd740 1 -- 172.21.15.76:0/4097372997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564d876006a0 con 0x564d876e04d0 2024-03-12T05:59:58.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57a9ffb640 1 -- 172.21.15.76:0/4097372997 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57ac004190 con 0x564d876e04d0 2024-03-12T05:59:58.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57a9ffb640 1 -- 172.21.15.76:0/4097372997 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f57bc004b40 con 0x564d876e04d0 2024-03-12T05:59:58.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57cb1fd740 10 client.16455 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:58.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57cb1fd740 10 client.16455 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:59:58.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57cb1fd740 20 client.16455 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:58.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/4097372997 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x564d8774dd00 0x564d87750120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.822 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57cb1fd740 4 client.16455 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T05:59:58.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57cb1fd740 1 -- 172.21.15.76:0/4097372997 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564d873cb890 con 0x564d8774dd00 2024-03-12T05:59:58.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.808+0000 7f57c259c640 1 --2- 172.21.15.76:0/4097372997 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x564d8774dd00 0x564d87750120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57a9ffb640 10 client.16455 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:58.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57c259c640 1 --2- 172.21.15.76:0/4097372997 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x564d8774dd00 0x564d87750120 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b612b9c636870f2 server_cookie=e6427b67946160be in_seq=0 out_seq=0 2024-03-12T05:59:58.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57a9ffb640 1 -- 172.21.15.76:0/4097372997 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x564d873cb890 con 0x564d8774dd00 2024-03-12T05:59:58.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57a9ffb640 10 client.16455 handle_command_reply: tid=0 2024-03-12T05:59:58.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57cb1fd740 1 client.16455 shutdown 2024-03-12T05:59:58.823 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57cb1fd740 1 -- 172.21.15.76:0/4097372997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876e04d0 msgr2=0x564d8774c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:58.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/4097372997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876e04d0 0x564d8774c130 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0348f0 tx=0x7f57bc0c1230 comp rx=0 tx=0).stop 2024-03-12T05:59:58.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57cb1fd740 1 -- 172.21.15.76:0/4097372997 shutdown_connections 2024-03-12T05:59:58.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/4097372997 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x564d8774dd00 0x564d87750120 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/4097372997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564d876e04d0 0x564d8774c130 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57a9ffb640 0 client.16455 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:58.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/4097372997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d876dc470 0x564d876ec1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57cb1fd740 1 --2- 172.21.15.76:0/4097372997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564d876dbab0 0x564d876ebc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57cb1fd740 1 -- 172.21.15.76:0/4097372997 >> 172.21.15.76:0/4097372997 conn(0x564d876d2690 msgr2=0x564d876df760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:58.824 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57cb1fd740 1 -- 172.21.15.76:0/4097372997 shutdown_connections 2024-03-12T05:59:58.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57cb1fd740 1 -- 172.21.15.76:0/4097372997 wait complete. 2024-03-12T05:59:58.825 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.809+0000 7f57cb1fd740 20 client.16455 trim_cache size 0 max 16384 2024-03-12T05:59:58.825 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T05:59:58.825 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T05:59:58.825 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T05:59:58.826 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T05:59:58.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.810+0000 7f57c898c640 1 -- 172.21.15.76:0/2604398526 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f57b00a1c20 msgr2=0x7f57b00a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:58.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.810+0000 7f57c898c640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f57b00a1c20 0x7f57b00a40e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f57ac005fd0 tx=0x7f57ac005850 comp rx=0 tx=0).stop 2024-03-12T05:59:58.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.810+0000 7f57c898c640 1 -- 172.21.15.76:0/2604398526 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c419aea0 msgr2=0x7f57c419b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:58.827 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.810+0000 7f57c898c640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c419aea0 0x7f57c419b350 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f57bc004af0 tx=0x7f57bc034cc0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.810+0000 7f57c898c640 1 -- 172.21.15.76:0/2604398526 shutdown_connections 2024-03-12T05:59:58.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.810+0000 7f57c898c640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f57b00a1c20 0x7f57b00a40e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.810+0000 7f57c898c640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c419aea0 0x7f57c419b350 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.810+0000 7f57c898c640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c4105330 0x7f57c4196b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.810+0000 7f57c898c640 1 --2- 172.21.15.76:0/2604398526 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f57c4100260 0x7f57c41965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.810+0000 7f57c898c640 1 -- 172.21.15.76:0/2604398526 >> 172.21.15.76:0/2604398526 conn(0x7f57c4077180 msgr2=0x7f57c40fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:58.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.810+0000 7f57c898c640 1 -- 172.21.15.76:0/2604398526 shutdown_connections 2024-03-12T05:59:58.828 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.810+0000 7f57c898c640 1 -- 172.21.15.76:0/2604398526 wait complete. 2024-03-12T05:59:58.829 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T05:59:58.829 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T05:59:58.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.926+0000 7f9fd0096640 1 -- 172.21.15.76:0/1917716042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fc8101d40 msgr2=0x7f9fc8105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:58.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.926+0000 7f9fd0096640 1 --2- 172.21.15.76:0/1917716042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fc8101d40 0x7f9fc8105d80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8009930 tx=0x7f9fb802bc30 comp rx=0 tx=0).stop 2024-03-12T05:59:58.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.926+0000 7f9fd0096640 1 -- 172.21.15.76:0/1917716042 shutdown_connections 2024-03-12T05:59:58.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.926+0000 7f9fd0096640 1 --2- 172.21.15.76:0/1917716042 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fc8106460 0x7f9fc810e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.926+0000 7f9fd0096640 1 --2- 172.21.15.76:0/1917716042 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fc8101d40 0x7f9fc8105d80 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.926+0000 7f9fd0096640 1 --2- 172.21.15.76:0/1917716042 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9fc8101390 0x7f9fc8101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.926+0000 7f9fd0096640 1 -- 172.21.15.76:0/1917716042 >> 172.21.15.76:0/1917716042 conn(0x7f9fc80fcc30 msgr2=0x7f9fc80ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:58.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.926+0000 7f9fd0096640 1 -- 172.21.15.76:0/1917716042 shutdown_connections 2024-03-12T05:59:58.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.926+0000 7f9fd0096640 1 -- 172.21.15.76:0/1917716042 wait complete. 2024-03-12T05:59:58.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fd0096640 1 Processor -- start 2024-03-12T05:59:58.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fd0096640 1 -- start start 2024-03-12T05:59:58.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fd0096640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fc8101390 0x7f9fc819d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fd0096640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9fc8106460 0x7f9fc819dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fd0096640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fc8197640 0x7f9fc8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fd0096640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc8198110 con 0x7f9fc8197640 2024-03-12T05:59:58.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fd0096640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc8198280 con 0x7f9fc8106460 2024-03-12T05:59:58.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fd0096640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc8198440 con 0x7f9fc8101390 2024-03-12T05:59:58.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fce633640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fc8197640 0x7f9fc8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fce633640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fc8197640 0x7f9fc8197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39796/0 (socket says 172.21.15.76:39796) 2024-03-12T05:59:58.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fce633640 1 -- 172.21.15.76:0/1878921596 learned_addr learned my addr 172.21.15.76:0/1878921596 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:58.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fcd631640 1 --2- 172.21.15.76:0/1878921596 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9fc8106460 0x7f9fc819dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fcde32640 1 --2- 172.21.15.76:0/1878921596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fc8101390 0x7f9fc819d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fce633640 1 -- 172.21.15.76:0/1878921596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fc8101390 msgr2=0x7f9fc819d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:58.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fce633640 1 --2- 172.21.15.76:0/1878921596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fc8101390 0x7f9fc819d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fce633640 1 -- 172.21.15.76:0/1878921596 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9fc8106460 msgr2=0x7f9fc819dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:58.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fce633640 1 --2- 172.21.15.76:0/1878921596 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9fc8106460 0x7f9fc819dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:58.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fce633640 1 -- 172.21.15.76:0/1878921596 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb8009560 con 0x7f9fc8197640 2024-03-12T05:59:58.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.927+0000 7f9fce633640 1 --2- 172.21.15.76:0/1878921596 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fc8197640 0x7f9fc8197ac0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f9fc000cad0 tx=0x7f9fc000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:58.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.928+0000 7f9fbeffd640 1 -- 172.21.15.76:0/1878921596 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc000ae20 con 0x7f9fc8197640 2024-03-12T05:59:58.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.928+0000 7f9fbeffd640 1 -- 172.21.15.76:0/1878921596 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fc0005020 con 0x7f9fc8197640 2024-03-12T05:59:58.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.928+0000 7f9fd0096640 1 -- 172.21.15.76:0/1878921596 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc81a43e0 con 0x7f9fc8197640 2024-03-12T05:59:58.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.928+0000 7f9fbeffd640 1 -- 172.21.15.76:0/1878921596 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc000f6b0 con 0x7f9fc8197640 2024-03-12T05:59:58.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.928+0000 7f9fd0096640 1 -- 172.21.15.76:0/1878921596 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc81a4910 con 0x7f9fc8197640 2024-03-12T05:59:58.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.928+0000 7f9fd0096640 1 -- 172.21.15.76:0/1878921596 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc8102860 con 0x7f9fc8197640 2024-03-12T05:59:58.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.930+0000 7f9fbeffd640 1 -- 172.21.15.76:0/1878921596 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9fc0005520 con 0x7f9fc8197640 2024-03-12T05:59:58.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.930+0000 7f9fbeffd640 1 --2- 172.21.15.76:0/1878921596 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9fb40a1c20 0x7f9fb40a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:58.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.930+0000 7f9fcde32640 1 --2- 172.21.15.76:0/1878921596 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9fb40a1c20 0x7f9fb40a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:58.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.930+0000 7f9fbeffd640 1 -- 172.21.15.76:0/1878921596 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9fc00cda60 con 0x7f9fc8197640 2024-03-12T05:59:58.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.931+0000 7f9fcde32640 1 --2- 172.21.15.76:0/1878921596 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9fb40a1c20 0x7f9fb40a40e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4005fd0 tx=0x7f9fc4005850 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:58.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:58.934+0000 7f9fbeffd640 1 -- 172.21.15.76:0/1878921596 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9fc0096960 con 0x7f9fc8197640 2024-03-12T05:59:59.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.145+0000 7f9fd0096640 1 -- 172.21.15.76:0/1878921596 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fc8109690 con 0x7f9fc8197640 2024-03-12T05:59:59.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.146+0000 7f9fbeffd640 1 -- 172.21.15.76:0/1878921596 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11332 (secure 0 0 0) 0x7f9fc009a020 con 0x7f9fc8197640 2024-03-12T05:59:59.152 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T05:59:59.152 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:58:43.838072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T05:59:59.152 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 25 2024-03-12T05:59:59.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.148+0000 7f9fd0096640 1 -- 172.21.15.76:0/1878921596 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9fb40a1c20 msgr2=0x7f9fb40a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.148+0000 7f9fd0096640 1 --2- 172.21.15.76:0/1878921596 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9fb40a1c20 0x7f9fb40a40e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4005fd0 tx=0x7f9fc4005850 comp rx=0 tx=0).stop 2024-03-12T05:59:59.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.148+0000 7f9fd0096640 1 -- 172.21.15.76:0/1878921596 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fc8197640 msgr2=0x7f9fc8197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.148+0000 7f9fd0096640 1 --2- 172.21.15.76:0/1878921596 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fc8197640 0x7f9fc8197ac0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f9fc000cad0 tx=0x7f9fc000cf90 comp rx=0 tx=0).stop 2024-03-12T05:59:59.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.148+0000 7f9fd0096640 1 -- 172.21.15.76:0/1878921596 shutdown_connections 2024-03-12T05:59:59.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.148+0000 7f9fd0096640 1 --2- 172.21.15.76:0/1878921596 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9fb40a1c20 0x7f9fb40a40e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.148+0000 7f9fd0096640 1 --2- 172.21.15.76:0/1878921596 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fc8197640 0x7f9fc8197ac0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.148+0000 7f9fd0096640 1 --2- 172.21.15.76:0/1878921596 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9fc8106460 0x7f9fc819dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.148+0000 7f9fd0096640 1 --2- 172.21.15.76:0/1878921596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fc8101390 0x7f9fc819d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.148+0000 7f9fd0096640 1 -- 172.21.15.76:0/1878921596 >> 172.21.15.76:0/1878921596 conn(0x7f9fc80fcc30 msgr2=0x7f9fc80fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.148+0000 7f9fd0096640 1 -- 172.21.15.76:0/1878921596 shutdown_connections 2024-03-12T05:59:59.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.148+0000 7f9fd0096640 1 -- 172.21.15.76:0/1878921596 wait complete. 2024-03-12T05:59:59.165 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:59:59.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T05:59:59.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.264+0000 7ff9b28d4640 1 -- 172.21.15.76:0/2461176366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9ac101380 msgr2=0x7ff9ac101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.264+0000 7ff9b28d4640 1 --2- 172.21.15.76:0/2461176366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9ac101380 0x7ff9ac101760 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0009930 tx=0x7ff9a002bc30 comp rx=0 tx=0).stop 2024-03-12T05:59:59.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.264+0000 7ff9b28d4640 1 -- 172.21.15.76:0/2461176366 shutdown_connections 2024-03-12T05:59:59.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.264+0000 7ff9b28d4640 1 --2- 172.21.15.76:0/2461176366 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9ac106450 0x7ff9ac10e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.264+0000 7ff9b28d4640 1 --2- 172.21.15.76:0/2461176366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9ac101d30 0x7ff9ac105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.264+0000 7ff9b28d4640 1 --2- 172.21.15.76:0/2461176366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9ac101380 0x7ff9ac101760 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.264+0000 7ff9b28d4640 1 -- 172.21.15.76:0/2461176366 >> 172.21.15.76:0/2461176366 conn(0x7ff9ac0fcc20 msgr2=0x7ff9ac0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.264+0000 7ff9b28d4640 1 -- 172.21.15.76:0/2461176366 shutdown_connections 2024-03-12T05:59:59.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.264+0000 7ff9b28d4640 1 -- 172.21.15.76:0/2461176366 wait complete. 2024-03-12T05:59:59.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.264+0000 7ff9b28d4640 1 Processor -- start 2024-03-12T05:59:59.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.264+0000 7ff9b28d4640 1 -- start start 2024-03-12T05:59:59.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9b28d4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9ac101380 0x7ff9ac19f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9b28d4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9ac106450 0x7ff9ac19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9b28d4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9ac1997b0 0x7ff9ac199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9b28d4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac19a260 con 0x7ff9ac106450 2024-03-12T05:59:59.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9b28d4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac19a3d0 con 0x7ff9ac101380 2024-03-12T05:59:59.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9b28d4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac19a540 con 0x7ff9ac1997b0 2024-03-12T05:59:59.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9ab7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9ac106450 0x7ff9ac19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9ab7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9ac106450 0x7ff9ac19fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39816/0 (socket says 172.21.15.76:39816) 2024-03-12T05:59:59.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9ab7fe640 1 -- 172.21.15.76:0/3087012630 learned_addr learned my addr 172.21.15.76:0/3087012630 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9b0e71640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9ac1997b0 0x7ff9ac199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9abfff640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9ac101380 0x7ff9ac19f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9ab7fe640 1 -- 172.21.15.76:0/3087012630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9ac1997b0 msgr2=0x7ff9ac199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9ab7fe640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9ac1997b0 0x7ff9ac199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9ab7fe640 1 -- 172.21.15.76:0/3087012630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9ac101380 msgr2=0x7ff9ac19f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9ab7fe640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9ac101380 0x7ff9ac19f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9ab7fe640 1 -- 172.21.15.76:0/3087012630 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a0009560 con 0x7ff9ac106450 2024-03-12T05:59:59.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9abfff640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9ac101380 0x7ff9ac19f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:59.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9ab7fe640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9ac106450 0x7ff9ac19fb80 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00d770 tx=0x7ff99c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9a97fa640 1 -- 172.21.15.76:0/3087012630 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c009c30 con 0x7ff9ac106450 2024-03-12T05:59:59.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9a97fa640 1 -- 172.21.15.76:0/3087012630 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff99c00de80 con 0x7ff9ac106450 2024-03-12T05:59:59.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9b28d4640 1 -- 172.21.15.76:0/3087012630 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ac1a64f0 con 0x7ff9ac106450 2024-03-12T05:59:59.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9a97fa640 1 -- 172.21.15.76:0/3087012630 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c00f6b0 con 0x7ff9ac106450 2024-03-12T05:59:59.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.265+0000 7ff9b28d4640 1 -- 172.21.15.76:0/3087012630 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ac1a69d0 con 0x7ff9ac106450 2024-03-12T05:59:59.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9a97fa640 1 -- 172.21.15.76:0/3087012630 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff99c010460 con 0x7ff9ac106450 2024-03-12T05:59:59.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9b5587740 1 Processor -- start 2024-03-12T05:59:59.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9b5587740 1 -- start start 2024-03-12T05:59:59.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9b5587740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653250 0x55839e653630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9b5587740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e65c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9a97fa640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff9940a1ba0 0x7ff9940a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9b5587740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e65eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9b5587740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55839e60e960 con 0x55839e653b70 2024-03-12T05:59:59.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9b5587740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55839e5c2310 con 0x55839e65c750 2024-03-12T05:59:59.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9b5587740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55839e5bf840 con 0x55839e653250 2024-03-12T05:59:59.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9ab7fe640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff9940a1ba0 0x7ff9940a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9abfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e65c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9abfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e65c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39832/0 (socket says 172.21.15.76:39832) 2024-03-12T05:59:59.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9abfff640 1 -- 172.21.15.76:0/2575324478 learned_addr learned my addr 172.21.15.76:0/2575324478 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.268+0000 7ff9b0e71640 1 --2- 172.21.15.76:0/2575324478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653250 0x55839e653630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.269+0000 7ff9a97fa640 1 -- 172.21.15.76:0/3087012630 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff99c0cd580 con 0x7ff9ac106450 2024-03-12T05:59:59.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.269+0000 7ff9b0e71640 1 --2- 172.21.15.76:0/2575324478 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e65eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.269+0000 7ff9abfff640 1 -- 172.21.15.76:0/2575324478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653250 msgr2=0x55839e653630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.269+0000 7ff9abfff640 1 --2- 172.21.15.76:0/2575324478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653250 0x55839e653630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.269+0000 7ff9abfff640 1 -- 172.21.15.76:0/2575324478 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 msgr2=0x55839e65eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.284 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.269+0000 7ff9abfff640 1 --2- 172.21.15.76:0/2575324478 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e65eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.269+0000 7ff9abfff640 1 -- 172.21.15.76:0/2575324478 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55839e57d060 con 0x55839e653b70 2024-03-12T05:59:59.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.269+0000 7ff9b0e71640 1 --2- 172.21.15.76:0/2575324478 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e65eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:59:59.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.269+0000 7ff9ab7fe640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff9940a1ba0 0x7ff9940a4060 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff99c096640 tx=0x7ff99c06d490 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.269+0000 7ff9abfff640 1 --2- 172.21.15.76:0/2575324478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e65c210 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0009930 tx=0x7ff9a0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0627c40a2105262 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.269+0000 7ff9927fc640 1 -- 172.21.15.76:0/2575324478 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a002c7e0 con 0x55839e653b70 2024-03-12T05:59:59.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.269+0000 7ff9927fc640 1 -- 172.21.15.76:0/2575324478 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9a0004510 con 0x55839e653b70 2024-03-12T05:59:59.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.269+0000 7ff9927fc640 1 -- 172.21.15.76:0/2575324478 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0005460 con 0x55839e653b70 2024-03-12T05:59:59.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.270+0000 7ff9b5587740 1 -- 172.21.15.76:0/2575324478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 msgr2=0x55839e65c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.270+0000 7ff9b5587740 1 --2- 172.21.15.76:0/2575324478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e65c210 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0009930 tx=0x7ff9a0004740 comp rx=0 tx=0).stop 2024-03-12T05:59:59.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.270+0000 7ff9b5587740 1 -- 172.21.15.76:0/2575324478 shutdown_connections 2024-03-12T05:59:59.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.270+0000 7ff9b5587740 1 --2- 172.21.15.76:0/2575324478 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e65eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.270+0000 7ff9b5587740 1 --2- 172.21.15.76:0/2575324478 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e65c210 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.270+0000 7ff9b5587740 1 --2- 172.21.15.76:0/2575324478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653250 0x55839e653630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.270+0000 7ff9b5587740 1 -- 172.21.15.76:0/2575324478 >> 172.21.15.76:0/2575324478 conn(0x55839e64d980 msgr2=0x55839e64dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.270+0000 7ff9b5587740 1 -- 172.21.15.76:0/2575324478 shutdown_connections 2024-03-12T05:59:59.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.270+0000 7ff9b5587740 1 -- 172.21.15.76:0/2575324478 wait complete. 2024-03-12T05:59:59.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.270+0000 7ff9b5587740 1 -- start start 2024-03-12T05:59:59.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.270+0000 7ff9b5587740 1 Processor -- start 2024-03-12T05:59:59.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9b5587740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653250 0x55839e6597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9b5587740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e659cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9b5587740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e65a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9b5587740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55839e5bb020 con 0x55839e653b70 2024-03-12T05:59:59.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9b5587740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55839e28e430 con 0x55839e65c750 2024-03-12T05:59:59.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9b5587740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55839e24ae00 con 0x55839e653250 2024-03-12T05:59:59.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9b5587740 10 client.0 _ll_register_callbacks cb 0x55839e562cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:59.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9abfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e659cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9abfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e659cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39834/0 (socket says 172.21.15.76:39834) 2024-03-12T05:59:59.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9abfff640 1 -- 172.21.15.76:0/3054395688 learned_addr learned my addr 172.21.15.76:0/3054395688 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9b0e71640 1 --2- 172.21.15.76:0/3054395688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653250 0x55839e6597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9b0e71640 1 --2- 172.21.15.76:0/3054395688 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e65a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9abfff640 1 -- 172.21.15.76:0/3054395688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653250 msgr2=0x55839e6597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9abfff640 1 --2- 172.21.15.76:0/3054395688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653250 0x55839e6597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9abfff640 1 -- 172.21.15.76:0/3054395688 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 msgr2=0x55839e65a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9abfff640 1 --2- 172.21.15.76:0/3054395688 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e65a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9abfff640 1 -- 172.21.15.76:0/3054395688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ac10e830 con 0x55839e653b70 2024-03-12T05:59:59.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9b0e71640 1 --2- 172.21.15.76:0/3054395688 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e65a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9abfff640 1 --2- 172.21.15.76:0/3054395688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e659cf0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff9a002c170 tx=0x7ff9a0004fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc05f93fa8493eae server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff98b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff98b7fe640 1 -- 172.21.15.76:0/3054395688 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0005610 con 0x55839e653b70 2024-03-12T05:59:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff98b7fe640 1 -- 172.21.15.76:0/3054395688 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff9a002cbe0 con 0x55839e653b70 2024-03-12T05:59:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9b5587740 1 -- 172.21.15.76:0/3054395688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55839e2ba7e0 con 0x55839e653b70 2024-03-12T05:59:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff98b7fe640 1 -- 172.21.15.76:0/3054395688 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9a0034440 con 0x55839e653b70 2024-03-12T05:59:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9b5587740 10 client.16473 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.271+0000 7ff9b5587740 1 -- 172.21.15.76:0/3054395688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55839e5b96a0 con 0x55839e653b70 2024-03-12T05:59:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.272+0000 7ff98b7fe640 1 -- 172.21.15.76:0/3054395688 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55839e2ba7e0 con 0x55839e653b70 2024-03-12T05:59:59.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.272+0000 7ff98b7fe640 1 -- 172.21.15.76:0/3054395688 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7ff9a003c040 con 0x55839e653b70 2024-03-12T05:59:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.272+0000 7ff9b5587740 10 client.16473 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.272+0000 7ff9b5587740 10 client.16473 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:59:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.272+0000 7ff9b5587740 20 client.16473 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.272+0000 7ff9b5587740 1 --2- 172.21.15.76:0/3054395688 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55839e66c710 0x55839e66eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.272+0000 7ff9b5587740 4 client.16473 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:59:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.272+0000 7ff9b5587740 1 -- 172.21.15.76:0/3054395688 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55839e2f7720 con 0x55839e66c710 2024-03-12T05:59:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.272+0000 7ff9b0e71640 1 --2- 172.21.15.76:0/3054395688 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55839e66c710 0x55839e66eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.273+0000 7ff98b7fe640 10 client.16473 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:59:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.273+0000 7ff9b0e71640 1 --2- 172.21.15.76:0/3054395688 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55839e66c710 0x55839e66eb30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=94d544b7a7d227c0 server_cookie=a4c30f4071332057 in_seq=0 out_seq=0 2024-03-12T05:59:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.273+0000 7ff98b7fe640 1 -- 172.21.15.76:0/3054395688 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55839e2f7720 con 0x55839e66c710 2024-03-12T05:59:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.273+0000 7ff98b7fe640 10 client.16473 handle_command_reply: tid=0 2024-03-12T05:59:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.273+0000 7ff9b5587740 1 client.16473 shutdown 2024-03-12T05:59:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.273+0000 7ff9b5587740 1 -- 172.21.15.76:0/3054395688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 msgr2=0x55839e659cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.273+0000 7ff9b5587740 1 --2- 172.21.15.76:0/3054395688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e659cf0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff9a002c170 tx=0x7ff9a0004fe0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.273+0000 7ff9b5587740 1 -- 172.21.15.76:0/3054395688 shutdown_connections 2024-03-12T05:59:59.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.273+0000 7ff9b5587740 1 --2- 172.21.15.76:0/3054395688 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55839e66c710 0x55839e66eb30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.273+0000 7ff9b5587740 1 --2- 172.21.15.76:0/3054395688 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e65a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.273+0000 7ff98b7fe640 0 client.16473 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:59:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.273+0000 7ff9b5587740 1 --2- 172.21.15.76:0/3054395688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e659cf0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.273+0000 7ff9b5587740 1 --2- 172.21.15.76:0/3054395688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653250 0x55839e6597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.273+0000 7ff9b5587740 1 -- 172.21.15.76:0/3054395688 >> 172.21.15.76:0/3054395688 conn(0x55839e64d980 msgr2=0x55839e64f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.274+0000 7ff9b5587740 1 -- 172.21.15.76:0/3054395688 shutdown_connections 2024-03-12T05:59:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.274+0000 7ff9b5587740 1 -- 172.21.15.76:0/3054395688 wait complete. 2024-03-12T05:59:59.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.274+0000 7ff9b5587740 20 client.16473 trim_cache size 0 max 16384 2024-03-12T05:59:59.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.285+0000 7ff9b5587740 1 Processor -- start 2024-03-12T05:59:59.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.285+0000 7ff9b5587740 1 -- start start 2024-03-12T05:59:59.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.285+0000 7ff9b5587740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653250 0x55839e65bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.285+0000 7ff9b5587740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e65a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.285+0000 7ff9b5587740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e656a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.285+0000 7ff9b5587740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55839e540130 con 0x55839e653b70 2024-03-12T05:59:59.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.285+0000 7ff9b5587740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55839e65f0c0 con 0x55839e65c750 2024-03-12T05:59:59.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.285+0000 7ff9b5587740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55839e652c90 con 0x55839e653250 2024-03-12T05:59:59.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.285+0000 7ff9abfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e65a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.285+0000 7ff9abfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e65a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39844/0 (socket says 172.21.15.76:39844) 2024-03-12T05:59:59.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.285+0000 7ff9abfff640 1 -- 172.21.15.76:0/1975553736 learned_addr learned my addr 172.21.15.76:0/1975553736 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9b0e71640 1 --2- 172.21.15.76:0/1975553736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e656a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9abfff640 1 -- 172.21.15.76:0/1975553736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653250 msgr2=0x55839e65bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9abfff640 1 --2- 172.21.15.76:0/1975553736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653250 0x55839e65bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9abfff640 1 -- 172.21.15.76:0/1975553736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 msgr2=0x55839e656a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9abfff640 1 --2- 172.21.15.76:0/1975553736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e656a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9abfff640 1 -- 172.21.15.76:0/1975553736 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55839e555c20 con 0x55839e653b70 2024-03-12T05:59:59.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9abfff640 1 --2- 172.21.15.76:0/1975553736 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e65a050 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0004fb0 tx=0x7ff9a0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568c8a89ef860944 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff991ffb640 1 -- 172.21.15.76:0/1975553736 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0005d50 con 0x55839e653b70 2024-03-12T05:59:59.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff991ffb640 1 -- 172.21.15.76:0/1975553736 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9a0004df0 con 0x55839e653b70 2024-03-12T05:59:59.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff991ffb640 1 -- 172.21.15.76:0/1975553736 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a00347e0 con 0x55839e653b70 2024-03-12T05:59:59.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9b5587740 1 -- 172.21.15.76:0/1975553736 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 msgr2=0x55839e65a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9b5587740 1 --2- 172.21.15.76:0/1975553736 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e65a050 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0004fb0 tx=0x7ff9a0004740 comp rx=0 tx=0).stop 2024-03-12T05:59:59.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9b5587740 1 -- 172.21.15.76:0/1975553736 shutdown_connections 2024-03-12T05:59:59.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9b5587740 1 --2- 172.21.15.76:0/1975553736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e656a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9b5587740 1 --2- 172.21.15.76:0/1975553736 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653b70 0x55839e65a050 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9b5587740 1 --2- 172.21.15.76:0/1975553736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653250 0x55839e65bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9b5587740 1 -- 172.21.15.76:0/1975553736 >> 172.21.15.76:0/1975553736 conn(0x55839e64d980 msgr2=0x55839e673490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9b5587740 1 -- 172.21.15.76:0/1975553736 shutdown_connections 2024-03-12T05:59:59.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.286+0000 7ff9b5587740 1 -- 172.21.15.76:0/1975553736 wait complete. 2024-03-12T05:59:59.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b5587740 1 -- start start 2024-03-12T05:59:59.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b5587740 1 Processor -- start 2024-03-12T05:59:59.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b5587740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653250 0x55839e663970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b5587740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653b70 0x55839e661d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b5587740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e662250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b5587740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55839e662930 con 0x55839e653250 2024-03-12T05:59:59.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b5587740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55839e662aa0 con 0x55839e65c750 2024-03-12T05:59:59.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b5587740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55839e6c2c20 con 0x55839e653b70 2024-03-12T05:59:59.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b5587740 10 client.0 _ll_register_callbacks cb 0x7ff9a0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:59.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b0e71640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653250 0x55839e663970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b0e71640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653250 0x55839e663970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39852/0 (socket says 172.21.15.76:39852) 2024-03-12T05:59:59.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b0e71640 1 -- 172.21.15.76:0/3634983232 learned_addr learned my addr 172.21.15.76:0/3634983232 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b0e71640 1 -- 172.21.15.76:0/3634983232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653b70 msgr2=0x55839e661d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b0e71640 1 --2- 172.21.15.76:0/3634983232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653b70 0x55839e661d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b0e71640 1 -- 172.21.15.76:0/3634983232 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 msgr2=0x55839e662250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b0e71640 1 --2- 172.21.15.76:0/3634983232 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e662250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.287+0000 7ff9b0e71640 1 -- 172.21.15.76:0/3634983232 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55839e5b96a0 con 0x55839e653250 2024-03-12T05:59:59.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff9b0e71640 1 --2- 172.21.15.76:0/3634983232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653250 0x55839e663970 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4013530 tx=0x7ff9a4020930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41d8f7d70fc1cd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff98b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:59.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff98b7fe640 1 -- 172.21.15.76:0/3634983232 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4018070 con 0x55839e653250 2024-03-12T05:59:59.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff98b7fe640 1 -- 172.21.15.76:0/3634983232 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff9a4003970 con 0x55839e653250 2024-03-12T05:59:59.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff9b5587740 1 -- 172.21.15.76:0/3634983232 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff9a0004510 con 0x55839e653250 2024-03-12T05:59:59.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff98b7fe640 1 -- 172.21.15.76:0/3634983232 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9a4011440 con 0x55839e653250 2024-03-12T05:59:59.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff9b5587740 10 client.16479 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:59.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff9b5587740 1 -- 172.21.15.76:0/3634983232 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55839e57b8c0 con 0x55839e653250 2024-03-12T05:59:59.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff98b7fe640 1 -- 172.21.15.76:0/3634983232 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9a0004510 con 0x55839e653250 2024-03-12T05:59:59.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff98b7fe640 1 -- 172.21.15.76:0/3634983232 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7ff9a4028780 con 0x55839e653250 2024-03-12T05:59:59.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff9b5587740 10 client.16479 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:59.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff9b5587740 10 client.16479 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T05:59:59.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff9b5587740 20 client.16479 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:59.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff9b5587740 1 --2- 172.21.15.76:0/3634983232 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55839e66c710 0x55839e6c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff9b5587740 4 client.16479 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:59:59.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.288+0000 7ff9b5587740 1 -- 172.21.15.76:0/3634983232 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55839e346890 con 0x55839e66c710 2024-03-12T05:59:59.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff9abfff640 1 --2- 172.21.15.76:0/3634983232 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55839e66c710 0x55839e6c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff98b7fe640 10 client.16479 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T05:59:59.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff9abfff640 1 --2- 172.21.15.76:0/3634983232 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55839e66c710 0x55839e6c64c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d5b11a5c65b25fec server_cookie=ce102fdb592e8bcf in_seq=0 out_seq=0 2024-03-12T05:59:59.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff98b7fe640 1 -- 172.21.15.76:0/3634983232 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55839e346890 con 0x55839e66c710 2024-03-12T05:59:59.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff98b7fe640 10 client.16479 handle_command_reply: tid=0 2024-03-12T05:59:59.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff9b5587740 1 client.16479 shutdown 2024-03-12T05:59:59.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff9b5587740 1 -- 172.21.15.76:0/3634983232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653250 msgr2=0x55839e663970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff9b5587740 1 --2- 172.21.15.76:0/3634983232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653250 0x55839e663970 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4013530 tx=0x7ff9a4020930 comp rx=0 tx=0).stop 2024-03-12T05:59:59.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff9b5587740 1 -- 172.21.15.76:0/3634983232 shutdown_connections 2024-03-12T05:59:59.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff9b5587740 1 --2- 172.21.15.76:0/3634983232 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55839e66c710 0x55839e6c64c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff9b5587740 1 --2- 172.21.15.76:0/3634983232 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55839e65c750 0x55839e662250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff98b7fe640 0 client.16479 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T05:59:59.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff9b5587740 1 --2- 172.21.15.76:0/3634983232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55839e653b70 0x55839e661d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff9b5587740 1 --2- 172.21.15.76:0/3634983232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55839e653250 0x55839e663970 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff9b5587740 1 -- 172.21.15.76:0/3634983232 >> 172.21.15.76:0/3634983232 conn(0x55839e64d980 msgr2=0x55839e64c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff9b5587740 1 -- 172.21.15.76:0/3634983232 shutdown_connections 2024-03-12T05:59:59.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff9b5587740 1 -- 172.21.15.76:0/3634983232 wait complete. 2024-03-12T05:59:59.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.289+0000 7ff9b5587740 20 client.16479 trim_cache size 0 max 16384 2024-03-12T05:59:59.308 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:59:59.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.290+0000 7ff9b28d4640 1 -- 172.21.15.76:0/3087012630 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff9940a1ba0 msgr2=0x7ff9940a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.290+0000 7ff9b28d4640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff9940a1ba0 0x7ff9940a4060 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff99c096640 tx=0x7ff99c06d490 comp rx=0 tx=0).stop 2024-03-12T05:59:59.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.290+0000 7ff9b28d4640 1 -- 172.21.15.76:0/3087012630 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9ac106450 msgr2=0x7ff9ac19fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.290+0000 7ff9b28d4640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9ac106450 0x7ff9ac19fb80 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00d770 tx=0x7ff99c00dc30 comp rx=0 tx=0).stop 2024-03-12T05:59:59.310 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.290+0000 7ff9b28d4640 1 -- 172.21.15.76:0/3087012630 shutdown_connections 2024-03-12T05:59:59.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.290+0000 7ff9b28d4640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff9940a1ba0 0x7ff9940a4060 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.290+0000 7ff9b28d4640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9ac1997b0 0x7ff9ac199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.290+0000 7ff9b28d4640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9ac106450 0x7ff9ac19fb80 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.290+0000 7ff9b28d4640 1 --2- 172.21.15.76:0/3087012630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff9ac101380 0x7ff9ac19f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.290+0000 7ff9b28d4640 1 -- 172.21.15.76:0/3087012630 >> 172.21.15.76:0/3087012630 conn(0x7ff9ac0fcc20 msgr2=0x7ff9ac0fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.290+0000 7ff9b28d4640 1 -- 172.21.15.76:0/3087012630 shutdown_connections 2024-03-12T05:59:59.311 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.290+0000 7ff9b28d4640 1 -- 172.21.15.76:0/3087012630 wait complete. 2024-03-12T05:59:59.312 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:59:59.312 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T05:59:59.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.411+0000 7ff5537a7640 1 -- 172.21.15.76:0/3652516614 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff54c106150 msgr2=0x7ff54c106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.411+0000 7ff5537a7640 1 --2- 172.21.15.76:0/3652516614 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff54c106150 0x7ff54c106530 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff53c009930 tx=0x7ff53c02bc30 comp rx=0 tx=0).stop 2024-03-12T05:59:59.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.411+0000 7ff5537a7640 1 -- 172.21.15.76:0/3652516614 shutdown_connections 2024-03-12T05:59:59.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.411+0000 7ff5537a7640 1 --2- 172.21.15.76:0/3652516614 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0fffb0 0x7ff54c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.411+0000 7ff5537a7640 1 --2- 172.21.15.76:0/3652516614 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff54c0ff5f0 0x7ff54c0ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.411+0000 7ff5537a7640 1 --2- 172.21.15.76:0/3652516614 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff54c106150 0x7ff54c106530 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.411+0000 7ff5537a7640 1 -- 172.21.15.76:0/3652516614 >> 172.21.15.76:0/3652516614 conn(0x7ff54c076580 msgr2=0x7ff54c0fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.411+0000 7ff5537a7640 1 -- 172.21.15.76:0/3652516614 shutdown_connections 2024-03-12T05:59:59.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.411+0000 7ff5537a7640 1 -- 172.21.15.76:0/3652516614 wait complete. 2024-03-12T05:59:59.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.411+0000 7ff5537a7640 1 Processor -- start 2024-03-12T05:59:59.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.411+0000 7ff5537a7640 1 -- start start 2024-03-12T05:59:59.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff5537a7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0ff5f0 0x7ff54c19d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff5537a7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff54c0fffb0 0x7ff54c19d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff5537a7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff54c106150 0x7ff54c1975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff5537a7640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c197c00 con 0x7ff54c106150 2024-03-12T05:59:59.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff5537a7640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c197d70 con 0x7ff54c0fffb0 2024-03-12T05:59:59.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff5537a7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c197f30 con 0x7ff54c0ff5f0 2024-03-12T05:59:59.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff551d44640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff54c106150 0x7ff54c1975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff551d44640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff54c106150 0x7ff54c1975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39866/0 (socket says 172.21.15.76:39866) 2024-03-12T05:59:59.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff551d44640 1 -- 172.21.15.76:0/1883166956 learned_addr learned my addr 172.21.15.76:0/1883166956 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff551543640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0ff5f0 0x7ff54c19d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff550d42640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff54c0fffb0 0x7ff54c19d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff551d44640 1 -- 172.21.15.76:0/1883166956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0ff5f0 msgr2=0x7ff54c19d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff551d44640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0ff5f0 0x7ff54c19d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff551d44640 1 -- 172.21.15.76:0/1883166956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff54c0fffb0 msgr2=0x7ff54c19d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff551d44640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff54c0fffb0 0x7ff54c19d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff551d44640 1 -- 172.21.15.76:0/1883166956 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c009560 con 0x7ff54c106150 2024-03-12T05:59:59.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff550d42640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff54c0fffb0 0x7ff54c19d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:59.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff551d44640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff54c106150 0x7ff54c1975b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff54400cce0 tx=0x7ff54400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff5427fc640 1 -- 172.21.15.76:0/1883166956 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54400f450 con 0x7ff54c106150 2024-03-12T05:59:59.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff5427fc640 1 -- 172.21.15.76:0/1883166956 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff54400fa90 con 0x7ff54c106150 2024-03-12T05:59:59.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff5427fc640 1 -- 172.21.15.76:0/1883166956 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544018b90 con 0x7ff54c106150 2024-03-12T05:59:59.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff5537a7640 1 -- 172.21.15.76:0/1883166956 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54c198210 con 0x7ff54c106150 2024-03-12T05:59:59.422 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.412+0000 7ff5537a7640 1 -- 172.21.15.76:0/1883166956 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54c1a44a0 con 0x7ff54c106150 2024-03-12T05:59:59.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.414+0000 7ff5427fc640 1 -- 172.21.15.76:0/1883166956 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff54400fc00 con 0x7ff54c106150 2024-03-12T05:59:59.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff556377740 1 Processor -- start 2024-03-12T05:59:59.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff556377740 1 -- start start 2024-03-12T05:59:59.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff556377740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3da2250 0x55d8e3da2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff556377740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2b70 0x55d8e3dab210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff556377740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3dab750 0x55d8e3dadb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff556377740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e3d5d960 con 0x55d8e3dab750 2024-03-12T05:59:59.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff556377740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e3d11310 con 0x55d8e3da2250 2024-03-12T05:59:59.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff556377740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e3d0e840 con 0x55d8e3da2b70 2024-03-12T05:59:59.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff5427fc640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff5380a1ba0 0x7ff5380a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff551d44640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff5380a1ba0 0x7ff5380a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff550d42640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3dab750 0x55d8e3dadb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff550d42640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3dab750 0x55d8e3dadb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39878/0 (socket says 172.21.15.76:39878) 2024-03-12T05:59:59.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff550d42640 1 -- 172.21.15.76:0/3044345175 learned_addr learned my addr 172.21.15.76:0/3044345175 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff551543640 1 --2- 172.21.15.76:0/3044345175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2b70 0x55d8e3dab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.415+0000 7ff5427fc640 1 -- 172.21.15.76:0/1883166956 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff5440cdfc0 con 0x7ff54c106150 2024-03-12T05:59:59.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.416+0000 7ff550d42640 1 --2- 172.21.15.76:0/3044345175 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3da2250 0x55d8e3da2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.416+0000 7ff550d42640 1 -- 172.21.15.76:0/3044345175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2b70 msgr2=0x55d8e3dab210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.416+0000 7ff550d42640 1 --2- 172.21.15.76:0/3044345175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2b70 0x55d8e3dab210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.416+0000 7ff550d42640 1 -- 172.21.15.76:0/3044345175 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3da2250 msgr2=0x55d8e3da2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.416+0000 7ff550d42640 1 --2- 172.21.15.76:0/3044345175 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3da2250 0x55d8e3da2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.416+0000 7ff550d42640 1 -- 172.21.15.76:0/3044345175 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8e3ccc060 con 0x55d8e3dab750 2024-03-12T05:59:59.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.416+0000 7ff551d44640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff5380a1ba0 0x7ff5380a4060 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff544097080 tx=0x7ff54406ded0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.416+0000 7ff550d42640 1 --2- 172.21.15.76:0/3044345175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3dab750 0x55d8e3dadb40 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff548012600 tx=0x7ff5480097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c7367d2a7a06c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.416+0000 7ff5377fe640 1 -- 172.21.15.76:0/3044345175 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548010710 con 0x55d8e3dab750 2024-03-12T05:59:59.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.416+0000 7ff5377fe640 1 -- 172.21.15.76:0/3044345175 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff548010d50 con 0x55d8e3dab750 2024-03-12T05:59:59.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.416+0000 7ff5377fe640 1 -- 172.21.15.76:0/3044345175 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54801dd10 con 0x55d8e3dab750 2024-03-12T05:59:59.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.417+0000 7ff556377740 1 -- 172.21.15.76:0/3044345175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3dab750 msgr2=0x55d8e3dadb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.417+0000 7ff556377740 1 --2- 172.21.15.76:0/3044345175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3dab750 0x55d8e3dadb40 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff548012600 tx=0x7ff5480097b0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.417+0000 7ff556377740 1 -- 172.21.15.76:0/3044345175 shutdown_connections 2024-03-12T05:59:59.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.417+0000 7ff556377740 1 --2- 172.21.15.76:0/3044345175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3dab750 0x55d8e3dadb40 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.417+0000 7ff556377740 1 --2- 172.21.15.76:0/3044345175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2b70 0x55d8e3dab210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.417+0000 7ff556377740 1 --2- 172.21.15.76:0/3044345175 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3da2250 0x55d8e3da2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.417+0000 7ff556377740 1 -- 172.21.15.76:0/3044345175 >> 172.21.15.76:0/3044345175 conn(0x55d8e3d9c980 msgr2=0x55d8e3d9cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.417+0000 7ff556377740 1 -- 172.21.15.76:0/3044345175 shutdown_connections 2024-03-12T05:59:59.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.417+0000 7ff556377740 1 -- 172.21.15.76:0/3044345175 wait complete. 2024-03-12T05:59:59.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.417+0000 7ff556377740 1 -- start start 2024-03-12T05:59:59.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.417+0000 7ff556377740 1 Processor -- start 2024-03-12T05:59:59.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff556377740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2250 0x55d8e3da87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff556377740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2b70 0x55d8e3da8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff556377740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3dab750 0x55d8e3da9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff556377740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e3d0a020 con 0x55d8e3da2250 2024-03-12T05:59:59.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff556377740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e39dd430 con 0x55d8e3dab750 2024-03-12T05:59:59.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff556377740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e3999e00 con 0x55d8e3da2b70 2024-03-12T05:59:59.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff556377740 10 client.0 _ll_register_callbacks cb 0x55d8e3cb1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:59.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff550d42640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2250 0x55d8e3da87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff550d42640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2250 0x55d8e3da87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39886/0 (socket says 172.21.15.76:39886) 2024-03-12T05:59:59.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff550d42640 1 -- 172.21.15.76:0/1206786713 learned_addr learned my addr 172.21.15.76:0/1206786713 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff551543640 1 --2- 172.21.15.76:0/1206786713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2b70 0x55d8e3da8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff550d42640 1 --2- 172.21.15.76:0/1206786713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3dab750 0x55d8e3da9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff550d42640 1 -- 172.21.15.76:0/1206786713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2b70 msgr2=0x55d8e3da8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff550d42640 1 --2- 172.21.15.76:0/1206786713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2b70 0x55d8e3da8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff550d42640 1 -- 172.21.15.76:0/1206786713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3dab750 msgr2=0x55d8e3da9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff550d42640 1 --2- 172.21.15.76:0/1206786713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3dab750 0x55d8e3da9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff550d42640 1 -- 172.21.15.76:0/1206786713 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff548009d30 con 0x55d8e3da2250 2024-03-12T05:59:59.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff550d42640 1 --2- 172.21.15.76:0/1206786713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2250 0x55d8e3da87b0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff54801d930 tx=0x7ff5480058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1417abdbebf77ef server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff534ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:59.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff534ff9640 1 -- 172.21.15.76:0/1206786713 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548004260 con 0x55d8e3da2250 2024-03-12T05:59:59.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff556377740 1 -- 172.21.15.76:0/1206786713 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d8e3a097e0 con 0x55d8e3da2250 2024-03-12T05:59:59.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.418+0000 7ff534ff9640 1 -- 172.21.15.76:0/1206786713 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff548000e40 con 0x55d8e3da2250 2024-03-12T05:59:59.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.419+0000 7ff534ff9640 1 -- 172.21.15.76:0/1206786713 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5480043c0 con 0x55d8e3da2250 2024-03-12T05:59:59.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.419+0000 7ff556377740 10 client.16491 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:59.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.419+0000 7ff556377740 1 -- 172.21.15.76:0/1206786713 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d8e3d086a0 con 0x55d8e3da2250 2024-03-12T05:59:59.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.419+0000 7ff534ff9640 1 -- 172.21.15.76:0/1206786713 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d8e3a097e0 con 0x55d8e3da2250 2024-03-12T05:59:59.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.419+0000 7ff534ff9640 1 -- 172.21.15.76:0/1206786713 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7ff548010a60 con 0x55d8e3da2250 2024-03-12T05:59:59.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.419+0000 7ff556377740 10 client.16491 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:59.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.419+0000 7ff556377740 10 client.16491 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:59:59.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.419+0000 7ff556377740 20 client.16491 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:59.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.419+0000 7ff556377740 1 --2- 172.21.15.76:0/1206786713 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d8e3dbbb60 0x55d8e3dbdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.419+0000 7ff556377740 4 client.16491 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:59:59.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.419+0000 7ff556377740 1 -- 172.21.15.76:0/1206786713 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d8e3a46720 con 0x55d8e3dbbb60 2024-03-12T05:59:59.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.419+0000 7ff551543640 1 --2- 172.21.15.76:0/1206786713 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d8e3dbbb60 0x55d8e3dbdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff534ff9640 10 client.16491 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:59:59.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff551543640 1 --2- 172.21.15.76:0/1206786713 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d8e3dbbb60 0x55d8e3dbdf80 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e849753f3c3ee19d server_cookie=588395386b666d39 in_seq=0 out_seq=0 2024-03-12T05:59:59.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff534ff9640 1 -- 172.21.15.76:0/1206786713 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d8e3a46720 con 0x55d8e3dbbb60 2024-03-12T05:59:59.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff534ff9640 10 client.16491 handle_command_reply: tid=0 2024-03-12T05:59:59.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff556377740 1 client.16491 shutdown 2024-03-12T05:59:59.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff556377740 1 -- 172.21.15.76:0/1206786713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2250 msgr2=0x55d8e3da87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff556377740 1 --2- 172.21.15.76:0/1206786713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2250 0x55d8e3da87b0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff54801d930 tx=0x7ff5480058d0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff556377740 1 -- 172.21.15.76:0/1206786713 shutdown_connections 2024-03-12T05:59:59.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff556377740 1 --2- 172.21.15.76:0/1206786713 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d8e3dbbb60 0x55d8e3dbdf80 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff556377740 1 --2- 172.21.15.76:0/1206786713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3dab750 0x55d8e3da9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff534ff9640 0 client.16491 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:59:59.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff556377740 1 --2- 172.21.15.76:0/1206786713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2b70 0x55d8e3da8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff556377740 1 --2- 172.21.15.76:0/1206786713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2250 0x55d8e3da87b0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff556377740 1 -- 172.21.15.76:0/1206786713 >> 172.21.15.76:0/1206786713 conn(0x55d8e3d9c980 msgr2=0x55d8e3d9e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff556377740 1 -- 172.21.15.76:0/1206786713 shutdown_connections 2024-03-12T05:59:59.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.420+0000 7ff556377740 1 -- 172.21.15.76:0/1206786713 wait complete. 2024-03-12T05:59:59.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.421+0000 7ff556377740 20 client.16491 trim_cache size 0 max 16384 2024-03-12T05:59:59.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 Processor -- start 2024-03-12T05:59:59.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 -- start start 2024-03-12T05:59:59.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2250 0x55d8e3daaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2b70 0x55d8e3da3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3dab750 0x55d8e3da3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e3c8f130 con 0x55d8e3da2b70 2024-03-12T05:59:59.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e3db7610 con 0x55d8e3dab750 2024-03-12T05:59:59.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e3dbb8e0 con 0x55d8e3da2250 2024-03-12T05:59:59.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff551543640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2b70 0x55d8e3da3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff551543640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2b70 0x55d8e3da3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39888/0 (socket says 172.21.15.76:39888) 2024-03-12T05:59:59.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff551543640 1 -- 172.21.15.76:0/3785949807 learned_addr learned my addr 172.21.15.76:0/3785949807 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff551543640 1 -- 172.21.15.76:0/3785949807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2250 msgr2=0x55d8e3daaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff551543640 1 --2- 172.21.15.76:0/3785949807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2250 0x55d8e3daaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff551543640 1 -- 172.21.15.76:0/3785949807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3dab750 msgr2=0x55d8e3da3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff551543640 1 --2- 172.21.15.76:0/3785949807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3dab750 0x55d8e3da3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff551543640 1 -- 172.21.15.76:0/3785949807 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8e3ca4c20 con 0x55d8e3da2b70 2024-03-12T05:59:59.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff551543640 1 --2- 172.21.15.76:0/3785949807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2b70 0x55d8e3da3a90 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff53c005ab0 tx=0x7ff53c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f942c99d3dd5b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff536ffd640 1 -- 172.21.15.76:0/3785949807 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c02cbf0 con 0x55d8e3da2b70 2024-03-12T05:59:59.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff536ffd640 1 -- 172.21.15.76:0/3785949807 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff53c049be0 con 0x55d8e3da2b70 2024-03-12T05:59:59.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff536ffd640 1 -- 172.21.15.76:0/3785949807 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c049600 con 0x55d8e3da2b70 2024-03-12T05:59:59.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 -- 172.21.15.76:0/3785949807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2b70 msgr2=0x55d8e3da3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 --2- 172.21.15.76:0/3785949807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2b70 0x55d8e3da3a90 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff53c005ab0 tx=0x7ff53c009260 comp rx=0 tx=0).stop 2024-03-12T05:59:59.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 -- 172.21.15.76:0/3785949807 shutdown_connections 2024-03-12T05:59:59.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 --2- 172.21.15.76:0/3785949807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3dab750 0x55d8e3da3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 --2- 172.21.15.76:0/3785949807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2b70 0x55d8e3da3a90 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 --2- 172.21.15.76:0/3785949807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2250 0x55d8e3daaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 -- 172.21.15.76:0/3785949807 >> 172.21.15.76:0/3785949807 conn(0x55d8e3d9c980 msgr2=0x55d8e3d97a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 -- 172.21.15.76:0/3785949807 shutdown_connections 2024-03-12T05:59:59.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.433+0000 7ff556377740 1 -- 172.21.15.76:0/3785949807 wait complete. 2024-03-12T05:59:59.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.434+0000 7ff556377740 1 -- start start 2024-03-12T05:59:59.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.434+0000 7ff556377740 1 Processor -- start 2024-03-12T05:59:59.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.434+0000 7ff556377740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2250 0x55d8e3db1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.434+0000 7ff556377740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2b70 0x55d8e3db1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.434+0000 7ff556377740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3dab750 0x55d8e3db1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.434+0000 7ff556377740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e3e1a8b0 con 0x55d8e3da2250 2024-03-12T05:59:59.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.434+0000 7ff556377740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e3e1aa20 con 0x55d8e3dab750 2024-03-12T05:59:59.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.434+0000 7ff556377740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e3e1abe0 con 0x55d8e3da2b70 2024-03-12T05:59:59.449 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.434+0000 7ff556377740 10 client.0 _ll_register_callbacks cb 0x7ff548009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:59.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.434+0000 7ff550d42640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2250 0x55d8e3db1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.434+0000 7ff550d42640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2250 0x55d8e3db1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39904/0 (socket says 172.21.15.76:39904) 2024-03-12T05:59:59.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.434+0000 7ff550d42640 1 -- 172.21.15.76:0/3147444155 learned_addr learned my addr 172.21.15.76:0/3147444155 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff550d42640 1 -- 172.21.15.76:0/3147444155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2b70 msgr2=0x55d8e3db1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff550d42640 1 --2- 172.21.15.76:0/3147444155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2b70 0x55d8e3db1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff550d42640 1 -- 172.21.15.76:0/3147444155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3dab750 msgr2=0x55d8e3db1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff550d42640 1 --2- 172.21.15.76:0/3147444155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3dab750 0x55d8e3db1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.450 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff550d42640 1 -- 172.21.15.76:0/3147444155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54c10e930 con 0x55d8e3da2250 2024-03-12T05:59:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff550d42640 1 --2- 172.21.15.76:0/3147444155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2250 0x55d8e3db1110 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7ff548004ae0 tx=0x7ff54801d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a027c9ee27d7da44 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff534ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff534ff9640 1 -- 172.21.15.76:0/3147444155 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff548010ec0 con 0x55d8e3da2250 2024-03-12T05:59:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff534ff9640 1 -- 172.21.15.76:0/3147444155 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff548000e40 con 0x55d8e3da2250 2024-03-12T05:59:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff556377740 1 -- 172.21.15.76:0/3147444155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff548010d50 con 0x55d8e3da2250 2024-03-12T05:59:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff534ff9640 1 -- 172.21.15.76:0/3147444155 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff54801d7a0 con 0x55d8e3da2250 2024-03-12T05:59:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff556377740 10 client.16497 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff556377740 1 -- 172.21.15.76:0/3147444155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d8e3cca8c0 con 0x55d8e3da2250 2024-03-12T05:59:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff534ff9640 1 -- 172.21.15.76:0/3147444155 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff548010d50 con 0x55d8e3da2250 2024-03-12T05:59:59.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff534ff9640 1 -- 172.21.15.76:0/3147444155 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7ff548010680 con 0x55d8e3da2250 2024-03-12T05:59:59.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff556377740 10 client.16497 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:59.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff556377740 10 client.16497 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T05:59:59.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff556377740 20 client.16497 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:59.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff556377740 1 --2- 172.21.15.76:0/3147444155 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d8e3dbbb60 0x55d8e3e1e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff556377740 4 client.16497 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:59:59.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.435+0000 7ff556377740 1 -- 172.21.15.76:0/3147444155 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d8e3a95890 con 0x55d8e3dbbb60 2024-03-12T05:59:59.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff551543640 1 --2- 172.21.15.76:0/3147444155 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d8e3dbbb60 0x55d8e3e1e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff551543640 1 --2- 172.21.15.76:0/3147444155 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d8e3dbbb60 0x55d8e3e1e1c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=52b66d194290f150 server_cookie=4f8000267b1516c6 in_seq=0 out_seq=0 2024-03-12T05:59:59.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff534ff9640 10 client.16497 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T05:59:59.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff534ff9640 1 -- 172.21.15.76:0/3147444155 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d8e3a95890 con 0x55d8e3dbbb60 2024-03-12T05:59:59.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff534ff9640 10 client.16497 handle_command_reply: tid=0 2024-03-12T05:59:59.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff556377740 1 client.16497 shutdown 2024-03-12T05:59:59.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff556377740 1 -- 172.21.15.76:0/3147444155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2250 msgr2=0x55d8e3db1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff556377740 1 --2- 172.21.15.76:0/3147444155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2250 0x55d8e3db1110 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7ff548004ae0 tx=0x7ff54801d960 comp rx=0 tx=0).stop 2024-03-12T05:59:59.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff556377740 1 -- 172.21.15.76:0/3147444155 shutdown_connections 2024-03-12T05:59:59.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff556377740 1 --2- 172.21.15.76:0/3147444155 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55d8e3dbbb60 0x55d8e3e1e1c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff556377740 1 --2- 172.21.15.76:0/3147444155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d8e3dab750 0x55d8e3db1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff534ff9640 0 client.16497 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T05:59:59.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff556377740 1 --2- 172.21.15.76:0/3147444155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3da2b70 0x55d8e3db1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff556377740 1 --2- 172.21.15.76:0/3147444155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d8e3da2250 0x55d8e3db1110 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.436+0000 7ff556377740 1 -- 172.21.15.76:0/3147444155 >> 172.21.15.76:0/3147444155 conn(0x55d8e3d9c980 msgr2=0x55d8e3da3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff556377740 1 -- 172.21.15.76:0/3147444155 shutdown_connections 2024-03-12T05:59:59.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff556377740 1 -- 172.21.15.76:0/3147444155 wait complete. 2024-03-12T05:59:59.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff556377740 20 client.16497 trim_cache size 0 max 16384 2024-03-12T05:59:59.454 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:59:59.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff5537a7640 1 -- 172.21.15.76:0/1883166956 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff5380a1ba0 msgr2=0x7ff5380a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff5537a7640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff5380a1ba0 0x7ff5380a4060 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff544097080 tx=0x7ff54406ded0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff5537a7640 1 -- 172.21.15.76:0/1883166956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff54c106150 msgr2=0x7ff54c1975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff5537a7640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff54c106150 0x7ff54c1975b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff54400cce0 tx=0x7ff54400a620 comp rx=0 tx=0).stop 2024-03-12T05:59:59.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff5537a7640 1 -- 172.21.15.76:0/1883166956 shutdown_connections 2024-03-12T05:59:59.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff5537a7640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff5380a1ba0 0x7ff5380a4060 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff5537a7640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff54c106150 0x7ff54c1975b0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff5537a7640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff54c0fffb0 0x7ff54c19d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff5537a7640 1 --2- 172.21.15.76:0/1883166956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff54c0ff5f0 0x7ff54c19d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff5537a7640 1 -- 172.21.15.76:0/1883166956 >> 172.21.15.76:0/1883166956 conn(0x7ff54c076580 msgr2=0x7ff54c104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff5537a7640 1 -- 172.21.15.76:0/1883166956 shutdown_connections 2024-03-12T05:59:59.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.437+0000 7ff5537a7640 1 -- 172.21.15.76:0/1883166956 wait complete. 2024-03-12T05:59:59.457 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T05:59:59.458 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T05:59:59.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.557+0000 7f0cac8df640 1 -- 172.21.15.76:0/3764222202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ca8105350 msgr2=0x7f0ca810cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.557+0000 7f0cac8df640 1 --2- 172.21.15.76:0/3764222202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ca8105350 0x7f0ca810cf20 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009cd0 tx=0x7f0ca002c0f0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.557+0000 7f0cac8df640 1 -- 172.21.15.76:0/3764222202 shutdown_connections 2024-03-12T05:59:59.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.557+0000 7f0cac8df640 1 --2- 172.21.15.76:0/3764222202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ca8105350 0x7f0ca810cf20 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.557+0000 7f0cac8df640 1 --2- 172.21.15.76:0/3764222202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ca8100c30 0x7f0ca8104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.557+0000 7f0cac8df640 1 --2- 172.21.15.76:0/3764222202 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0ca8100280 0x7f0ca8100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.557+0000 7f0cac8df640 1 -- 172.21.15.76:0/3764222202 >> 172.21.15.76:0/3764222202 conn(0x7f0ca80771a0 msgr2=0x7f0ca80775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.557+0000 7f0cac8df640 1 -- 172.21.15.76:0/3764222202 shutdown_connections 2024-03-12T05:59:59.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.557+0000 7f0cac8df640 1 -- 172.21.15.76:0/3764222202 wait complete. 2024-03-12T05:59:59.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0cac8df640 1 Processor -- start 2024-03-12T05:59:59.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0cac8df640 1 -- start start 2024-03-12T05:59:59.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0cac8df640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ca8100280 0x7f0ca819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0cac8df640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ca8105350 0x7f0ca819d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0cac8df640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0ca8197610 0x7f0ca8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0cac8df640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca8198000 con 0x7f0ca8100280 2024-03-12T05:59:59.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0cac8df640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca8198170 con 0x7f0ca8197610 2024-03-12T05:59:59.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0cac8df640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca8198330 con 0x7f0ca8105350 2024-03-12T05:59:59.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0ca659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ca8100280 0x7f0ca819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0ca659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ca8100280 0x7f0ca819d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39916/0 (socket says 172.21.15.76:39916) 2024-03-12T05:59:59.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0ca659c640 1 -- 172.21.15.76:0/1050315965 learned_addr learned my addr 172.21.15.76:0/1050315965 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0ca6d9d640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0ca8197610 0x7f0ca8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0ca5d9b640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ca8105350 0x7f0ca819d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0ca659c640 1 -- 172.21.15.76:0/1050315965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ca8105350 msgr2=0x7f0ca819d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0ca659c640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ca8105350 0x7f0ca819d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0ca659c640 1 -- 172.21.15.76:0/1050315965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0ca8197610 msgr2=0x7f0ca8197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0ca659c640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0ca8197610 0x7f0ca8197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0ca659c640 1 -- 172.21.15.76:0/1050315965 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca0009900 con 0x7f0ca8100280 2024-03-12T05:59:59.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0ca6d9d640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0ca8197610 0x7f0ca8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T05:59:59.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.558+0000 7f0ca659c640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ca8100280 0x7f0ca819d410 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c00e9f0 tx=0x7f0c9c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.559+0000 7f0c977fe640 1 -- 172.21.15.76:0/1050315965 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c009700 con 0x7f0ca8100280 2024-03-12T05:59:59.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.559+0000 7f0c977fe640 1 -- 172.21.15.76:0/1050315965 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c9c00cd80 con 0x7f0ca8100280 2024-03-12T05:59:59.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.559+0000 7f0cac8df640 1 -- 172.21.15.76:0/1050315965 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca81a42d0 con 0x7f0ca8100280 2024-03-12T05:59:59.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.559+0000 7f0c977fe640 1 -- 172.21.15.76:0/1050315965 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c018920 con 0x7f0ca8100280 2024-03-12T05:59:59.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.559+0000 7f0cac8df640 1 -- 172.21.15.76:0/1050315965 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca81a47e0 con 0x7f0ca8100280 2024-03-12T05:59:59.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.561+0000 7f0c977fe640 1 -- 172.21.15.76:0/1050315965 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f0c9c018a80 con 0x7f0ca8100280 2024-03-12T05:59:59.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.561+0000 7f0caf594740 1 Processor -- start 2024-03-12T05:59:59.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.561+0000 7f0caf594740 1 -- start start 2024-03-12T05:59:59.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0c977fe640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f0c900a1ba0 0x7f0c900a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0caf594740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b387250 0x557d7b387630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0caf594740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b390210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0caf594740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b390750 0x557d7b392b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0caf594740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7b342960 con 0x557d7b387b70 2024-03-12T05:59:59.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0caf594740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7b2f6310 con 0x557d7b387250 2024-03-12T05:59:59.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0caf594740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7b2f3840 con 0x557d7b390750 2024-03-12T05:59:59.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0ca6d9d640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f0c900a1ba0 0x7f0c900a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0ca5d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b390210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0ca5d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b390210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39932/0 (socket says 172.21.15.76:39932) 2024-03-12T05:59:59.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0ca659c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b387250 0x557d7b387630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0c977fe640 1 -- 172.21.15.76:0/1050315965 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0c9c014070 con 0x7f0ca8100280 2024-03-12T05:59:59.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/3962513879 learned_addr learned my addr 172.21.15.76:0/3962513879 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0ca6d9d640 1 --2- 172.21.15.76:0/3962513879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b390750 0x557d7b392b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/3962513879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b390750 msgr2=0x557d7b392b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0ca5d9b640 1 --2- 172.21.15.76:0/3962513879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b390750 0x557d7b392b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/3962513879 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b387250 msgr2=0x557d7b387630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0ca5d9b640 1 --2- 172.21.15.76:0/3962513879 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b387250 0x557d7b387630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/3962513879 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d7b2b1060 con 0x557d7b387b70 2024-03-12T05:59:59.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0ca6d9d640 1 --2- 172.21.15.76:0/3962513879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b390750 0x557d7b392b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:59.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0ca6d9d640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f0c900a1ba0 0x7f0c900a4060 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0037040 tx=0x7f0ca00324f0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0ca5d9b640 1 --2- 172.21.15.76:0/3962513879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b390210 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f0c98009b40 tx=0x7f0c98010370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7999402e92261dea server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.562+0000 7f0c94ff9640 1 -- 172.21.15.76:0/3962513879 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c98010d20 con 0x557d7b387b70 2024-03-12T05:59:59.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.563+0000 7f0c94ff9640 1 -- 172.21.15.76:0/3962513879 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c98010e80 con 0x557d7b387b70 2024-03-12T05:59:59.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.563+0000 7f0c94ff9640 1 -- 172.21.15.76:0/3962513879 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c98012a00 con 0x557d7b387b70 2024-03-12T05:59:59.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.563+0000 7f0caf594740 1 -- 172.21.15.76:0/3962513879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 msgr2=0x557d7b390210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.563+0000 7f0caf594740 1 --2- 172.21.15.76:0/3962513879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b390210 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f0c98009b40 tx=0x7f0c98010370 comp rx=0 tx=0).stop 2024-03-12T05:59:59.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.563+0000 7f0caf594740 1 -- 172.21.15.76:0/3962513879 shutdown_connections 2024-03-12T05:59:59.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.563+0000 7f0caf594740 1 --2- 172.21.15.76:0/3962513879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b390750 0x557d7b392b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.563+0000 7f0caf594740 1 --2- 172.21.15.76:0/3962513879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b390210 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.563+0000 7f0caf594740 1 --2- 172.21.15.76:0/3962513879 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b387250 0x557d7b387630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.563+0000 7f0caf594740 1 -- 172.21.15.76:0/3962513879 >> 172.21.15.76:0/3962513879 conn(0x557d7b381980 msgr2=0x557d7b381d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.563+0000 7f0caf594740 1 -- 172.21.15.76:0/3962513879 shutdown_connections 2024-03-12T05:59:59.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.563+0000 7f0caf594740 1 -- 172.21.15.76:0/3962513879 wait complete. 2024-03-12T05:59:59.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0caf594740 1 -- start start 2024-03-12T05:59:59.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0caf594740 1 Processor -- start 2024-03-12T05:59:59.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0caf594740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b387250 0x557d7b38d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0caf594740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b38dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0caf594740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b390750 0x557d7b38e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0caf594740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7b2ef020 con 0x557d7b387b70 2024-03-12T05:59:59.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0caf594740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7afc2430 con 0x557d7b387250 2024-03-12T05:59:59.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0caf594740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7af7ee00 con 0x557d7b390750 2024-03-12T05:59:59.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0caf594740 10 client.0 _ll_register_callbacks cb 0x557d7b296cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:59.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0ca5d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b38dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0ca5d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b38dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39940/0 (socket says 172.21.15.76:39940) 2024-03-12T05:59:59.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/2098671231 learned_addr learned my addr 172.21.15.76:0/2098671231 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0ca659c640 1 --2- 172.21.15.76:0/2098671231 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b387250 0x557d7b38d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/2098671231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b390750 msgr2=0x557d7b38e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0ca5d9b640 1 --2- 172.21.15.76:0/2098671231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b390750 0x557d7b38e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/2098671231 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b387250 msgr2=0x557d7b38d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0ca5d9b640 1 --2- 172.21.15.76:0/2098671231 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b387250 0x557d7b38d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/2098671231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c980097a0 con 0x557d7b387b70 2024-03-12T05:59:59.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0ca659c640 1 --2- 172.21.15.76:0/2098671231 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b387250 0x557d7b38d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T05:59:59.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0ca5d9b640 1 --2- 172.21.15.76:0/2098671231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b38dca0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f0c98011010 tx=0x7f0c980052e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db0f668e33fb7ee server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.564+0000 7f0c8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:59.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0c8dffb640 1 -- 172.21.15.76:0/2098671231 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c98021070 con 0x557d7b387b70 2024-03-12T05:59:59.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0c8dffb640 1 -- 172.21.15.76:0/2098671231 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0c98016070 con 0x557d7b387b70 2024-03-12T05:59:59.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0caf594740 1 -- 172.21.15.76:0/2098671231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d7afee7e0 con 0x557d7b387b70 2024-03-12T05:59:59.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0c8dffb640 1 -- 172.21.15.76:0/2098671231 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c98004050 con 0x557d7b387b70 2024-03-12T05:59:59.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0caf594740 10 client.16509 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:59.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0caf594740 1 -- 172.21.15.76:0/2098671231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557d7b2ed6a0 con 0x557d7b387b70 2024-03-12T05:59:59.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0c8dffb640 1 -- 172.21.15.76:0/2098671231 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c98033020 con 0x557d7b387b70 2024-03-12T05:59:59.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0c8dffb640 1 -- 172.21.15.76:0/2098671231 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f0c980047c0 con 0x557d7b387b70 2024-03-12T05:59:59.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0caf594740 10 client.16509 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:59.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0caf594740 10 client.16509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:59:59.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0caf594740 20 client.16509 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:59.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0caf594740 1 --2- 172.21.15.76:0/2098671231 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557d7b3a0af0 0x557d7b3a2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0caf594740 4 client.16509 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T05:59:59.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0caf594740 1 -- 172.21.15.76:0/2098671231 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d7b02b720 con 0x557d7b3a0af0 2024-03-12T05:59:59.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.565+0000 7f0ca659c640 1 --2- 172.21.15.76:0/2098671231 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557d7b3a0af0 0x557d7b3a2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.566+0000 7f0c8dffb640 10 client.16509 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:59.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.566+0000 7f0ca659c640 1 --2- 172.21.15.76:0/2098671231 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557d7b3a0af0 0x557d7b3a2f30 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94000ef1cfac8b4f server_cookie=e22f21ecffffcfa7 in_seq=0 out_seq=0 2024-03-12T05:59:59.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.566+0000 7f0c8dffb640 1 -- 172.21.15.76:0/2098671231 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557d7b02b720 con 0x557d7b3a0af0 2024-03-12T05:59:59.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.566+0000 7f0c8dffb640 10 client.16509 handle_command_reply: tid=0 2024-03-12T05:59:59.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.567+0000 7f0caf594740 1 client.16509 shutdown 2024-03-12T05:59:59.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.567+0000 7f0caf594740 1 -- 172.21.15.76:0/2098671231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 msgr2=0x557d7b38dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.567+0000 7f0caf594740 1 --2- 172.21.15.76:0/2098671231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b38dca0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f0c98011010 tx=0x7f0c980052e0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.567+0000 7f0caf594740 1 -- 172.21.15.76:0/2098671231 shutdown_connections 2024-03-12T05:59:59.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.567+0000 7f0caf594740 1 --2- 172.21.15.76:0/2098671231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b390750 0x557d7b38e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.567+0000 7f0caf594740 1 --2- 172.21.15.76:0/2098671231 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557d7b3a0af0 0x557d7b3a2f30 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.567+0000 7f0caf594740 1 --2- 172.21.15.76:0/2098671231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b38dca0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.567+0000 7f0caf594740 1 --2- 172.21.15.76:0/2098671231 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b387250 0x557d7b38d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.567+0000 7f0c8dffb640 0 client.16509 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:59.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.567+0000 7f0caf594740 1 -- 172.21.15.76:0/2098671231 >> 172.21.15.76:0/2098671231 conn(0x557d7b381980 msgr2=0x557d7b3836d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.567+0000 7f0caf594740 1 -- 172.21.15.76:0/2098671231 shutdown_connections 2024-03-12T05:59:59.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.567+0000 7f0caf594740 1 -- 172.21.15.76:0/2098671231 wait complete. 2024-03-12T05:59:59.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.567+0000 7f0caf594740 20 client.16509 trim_cache size 0 max 16384 2024-03-12T05:59:59.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0caf594740 1 Processor -- start 2024-03-12T05:59:59.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0caf594740 1 -- start start 2024-03-12T05:59:59.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0caf594740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b387250 0x557d7b38fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0caf594740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b38e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0caf594740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b390750 0x557d7b38a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0caf594740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7b274130 con 0x557d7b387b70 2024-03-12T05:59:59.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0caf594740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7b39d640 con 0x557d7b390750 2024-03-12T05:59:59.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0caf594740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7b3a81b0 con 0x557d7b387250 2024-03-12T05:59:59.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0ca5d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b38e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0ca5d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b38e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39944/0 (socket says 172.21.15.76:39944) 2024-03-12T05:59:59.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/220605619 learned_addr learned my addr 172.21.15.76:0/220605619 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0ca659c640 1 --2- 172.21.15.76:0/220605619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b387250 0x557d7b38fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/220605619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b387250 msgr2=0x557d7b38fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0ca5d9b640 1 --2- 172.21.15.76:0/220605619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b387250 0x557d7b38fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/220605619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b390750 msgr2=0x557d7b38a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0ca5d9b640 1 --2- 172.21.15.76:0/220605619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b390750 0x557d7b38a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/220605619 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d7b289c20 con 0x557d7b387b70 2024-03-12T05:59:59.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0ca5d9b640 1 --2- 172.21.15.76:0/220605619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b38e000 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0c98005940 tx=0x7f0c98010370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1014ca73c99c33ae server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0c8ffff640 1 -- 172.21.15.76:0/220605619 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c98021020 con 0x557d7b387b70 2024-03-12T05:59:59.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0c8ffff640 1 -- 172.21.15.76:0/220605619 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c98033030 con 0x557d7b387b70 2024-03-12T05:59:59.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0c8ffff640 1 -- 172.21.15.76:0/220605619 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9801b480 con 0x557d7b387b70 2024-03-12T05:59:59.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.579+0000 7f0caf594740 1 -- 172.21.15.76:0/220605619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 msgr2=0x557d7b38e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 --2- 172.21.15.76:0/220605619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b38e000 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0c98005940 tx=0x7f0c98010370 comp rx=0 tx=0).stop 2024-03-12T05:59:59.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 -- 172.21.15.76:0/220605619 shutdown_connections 2024-03-12T05:59:59.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 --2- 172.21.15.76:0/220605619 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b390750 0x557d7b38a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 --2- 172.21.15.76:0/220605619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b38e000 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 --2- 172.21.15.76:0/220605619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b387250 0x557d7b38fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 -- 172.21.15.76:0/220605619 >> 172.21.15.76:0/220605619 conn(0x557d7b381980 msgr2=0x557d7b388a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 -- 172.21.15.76:0/220605619 shutdown_connections 2024-03-12T05:59:59.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 -- 172.21.15.76:0/220605619 wait complete. 2024-03-12T05:59:59.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 -- start start 2024-03-12T05:59:59.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 Processor -- start 2024-03-12T05:59:59.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b387250 0x557d7b396840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b396d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b390750 0x557d7b3f7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7b3f75a0 con 0x557d7b387b70 2024-03-12T05:59:59.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7b3f76e0 con 0x557d7b390750 2024-03-12T05:59:59.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7b3f7870 con 0x557d7b387250 2024-03-12T05:59:59.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0caf594740 10 client.0 _ll_register_callbacks cb 0x7f0c98009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T05:59:59.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0ca5d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b396d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0ca5d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b396d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39960/0 (socket says 172.21.15.76:39960) 2024-03-12T05:59:59.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/95569755 learned_addr learned my addr 172.21.15.76:0/95569755 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T05:59:59.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0ca659c640 1 --2- 172.21.15.76:0/95569755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b387250 0x557d7b396840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0ca6d9d640 1 --2- 172.21.15.76:0/95569755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b390750 0x557d7b3f7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/95569755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b387250 msgr2=0x557d7b396840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0ca5d9b640 1 --2- 172.21.15.76:0/95569755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b387250 0x557d7b396840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.580+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/95569755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b390750 msgr2=0x557d7b3f7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T05:59:59.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0ca5d9b640 1 --2- 172.21.15.76:0/95569755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b390750 0x557d7b3f7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0ca5d9b640 1 -- 172.21.15.76:0/95569755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d7b2ed6a0 con 0x557d7b387b70 2024-03-12T05:59:59.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0ca5d9b640 1 --2- 172.21.15.76:0/95569755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b396d80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0c98009770 tx=0x7f0c98012760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5c8c30028180b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T05:59:59.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0c8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T05:59:59.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0c8dffb640 1 -- 172.21.15.76:0/95569755 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c980099c0 con 0x557d7b387b70 2024-03-12T05:59:59.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0c8dffb640 1 -- 172.21.15.76:0/95569755 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0c98000e40 con 0x557d7b387b70 2024-03-12T05:59:59.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0caf594740 1 -- 172.21.15.76:0/95569755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c98010e80 con 0x557d7b387b70 2024-03-12T05:59:59.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0c8dffb640 1 -- 172.21.15.76:0/95569755 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c9801b690 con 0x557d7b387b70 2024-03-12T05:59:59.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0caf594740 10 client.16515 fetch_fsmap learned FSMap version 25 2024-03-12T05:59:59.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0caf594740 1 -- 172.21.15.76:0/95569755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557d7b2af8c0 con 0x557d7b387b70 2024-03-12T05:59:59.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0c8dffb640 1 -- 172.21.15.76:0/95569755 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c98010e80 con 0x557d7b387b70 2024-03-12T05:59:59.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0c8dffb640 1 -- 172.21.15.76:0/95569755 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f0c9802d070 con 0x557d7b387b70 2024-03-12T05:59:59.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0caf594740 10 client.16515 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T05:59:59.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0caf594740 10 client.16515 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T05:59:59.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0caf594740 20 client.16515 populate_metadata read hostname 'smithi076' 2024-03-12T05:59:59.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0caf594740 1 --2- 172.21.15.76:0/95569755 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557d7b3f8cf0 0x557d7b3fb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T05:59:59.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0caf594740 4 client.16515 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T05:59:59.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0caf594740 1 -- 172.21.15.76:0/95569755 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557d7b07a890 con 0x557d7b3f8cf0 2024-03-12T05:59:59.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.581+0000 7f0ca659c640 1 --2- 172.21.15.76:0/95569755 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557d7b3f8cf0 0x557d7b3fb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T05:59:59.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0c8dffb640 10 client.16515 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:59.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0ca659c640 1 --2- 172.21.15.76:0/95569755 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557d7b3f8cf0 0x557d7b3fb110 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40f1af07f26eb7e1 server_cookie=4041ed6c4f8f346 in_seq=0 out_seq=0 2024-03-12T05:59:59.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0c8dffb640 1 -- 172.21.15.76:0/95569755 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557d7b07a890 con 0x557d7b3f8cf0 2024-03-12T05:59:59.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0c8dffb640 10 client.16515 handle_command_reply: tid=0 2024-03-12T05:59:59.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0caf594740 1 client.16515 shutdown 2024-03-12T05:59:59.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0caf594740 1 -- 172.21.15.76:0/95569755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 msgr2=0x557d7b396d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0caf594740 1 --2- 172.21.15.76:0/95569755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b396d80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0c98009770 tx=0x7f0c98012760 comp rx=0 tx=0).stop 2024-03-12T05:59:59.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0caf594740 1 -- 172.21.15.76:0/95569755 shutdown_connections 2024-03-12T05:59:59.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0caf594740 1 --2- 172.21.15.76:0/95569755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557d7b390750 0x557d7b3f7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0caf594740 1 --2- 172.21.15.76:0/95569755 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x557d7b3f8cf0 0x557d7b3fb110 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0caf594740 1 --2- 172.21.15.76:0/95569755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557d7b387b70 0x557d7b396d80 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0caf594740 1 --2- 172.21.15.76:0/95569755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d7b387250 0x557d7b396840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0c8dffb640 0 client.16515 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T05:59:59.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0caf594740 1 -- 172.21.15.76:0/95569755 >> 172.21.15.76:0/95569755 conn(0x557d7b381980 msgr2=0x557d7b3802f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0caf594740 1 -- 172.21.15.76:0/95569755 shutdown_connections 2024-03-12T05:59:59.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0caf594740 1 -- 172.21.15.76:0/95569755 wait complete. 2024-03-12T05:59:59.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.582+0000 7f0caf594740 20 client.16515 trim_cache size 0 max 16384 2024-03-12T05:59:59.603 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T05:59:59.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.583+0000 7f0cac8df640 1 -- 172.21.15.76:0/1050315965 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f0c900a1ba0 msgr2=0x7f0c900a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.583+0000 7f0cac8df640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f0c900a1ba0 0x7f0c900a4060 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0037040 tx=0x7f0ca00324f0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.583+0000 7f0cac8df640 1 -- 172.21.15.76:0/1050315965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ca8100280 msgr2=0x7f0ca819d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T05:59:59.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.583+0000 7f0cac8df640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ca8100280 0x7f0ca819d410 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c00e9f0 tx=0x7f0c9c00eeb0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.583+0000 7f0cac8df640 1 -- 172.21.15.76:0/1050315965 shutdown_connections 2024-03-12T05:59:59.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.583+0000 7f0cac8df640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f0c900a1ba0 0x7f0c900a4060 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.583+0000 7f0cac8df640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0ca8197610 0x7f0ca8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.583+0000 7f0cac8df640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ca8105350 0x7f0ca819d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.583+0000 7f0cac8df640 1 --2- 172.21.15.76:0/1050315965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0ca8100280 0x7f0ca819d410 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T05:59:59.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.583+0000 7f0cac8df640 1 -- 172.21.15.76:0/1050315965 >> 172.21.15.76:0/1050315965 conn(0x7f0ca80771a0 msgr2=0x7f0ca80fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T05:59:59.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.583+0000 7f0cac8df640 1 -- 172.21.15.76:0/1050315965 shutdown_connections 2024-03-12T05:59:59.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T05:59:59.583+0000 7f0cac8df640 1 -- 172.21.15.76:0/1050315965 wait complete. 2024-03-12T06:00:00.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:00 smithi082 ceph-mon[36382]: pgmap v129: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:00.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:00 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1878921596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:00:00.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:00 smithi082 ceph-mon[36382]: overall HEALTH_OK 2024-03-12T06:00:00.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:00 smithi088 ceph-mon[36585]: pgmap v129: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:00.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:00 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1878921596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:00:00.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:00 smithi088 ceph-mon[36585]: overall HEALTH_OK 2024-03-12T06:00:00.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:00 smithi076 ceph-mon[32095]: pgmap v129: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:00.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:00 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1878921596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:00:00.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:00 smithi076 ceph-mon[32095]: overall HEALTH_OK 2024-03-12T06:00:00.607 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T06:00:00.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:00:00.607 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T06:00:00.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.726+0000 7f3d0619a640 1 -- 172.21.15.76:0/3001063753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d00105350 msgr2=0x7f3d0010cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.726+0000 7f3d0619a640 1 --2- 172.21.15.76:0/3001063753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d00105350 0x7f3d0010cf20 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8009cd0 tx=0x7f3cf802c0f0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.726+0000 7f3d0619a640 1 -- 172.21.15.76:0/3001063753 shutdown_connections 2024-03-12T06:00:00.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.726+0000 7f3d0619a640 1 --2- 172.21.15.76:0/3001063753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d00105350 0x7f3d0010cf20 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.726+0000 7f3d0619a640 1 --2- 172.21.15.76:0/3001063753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d00100c30 0x7f3d00104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.726+0000 7f3d0619a640 1 --2- 172.21.15.76:0/3001063753 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d00100280 0x7f3d00100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.726+0000 7f3d0619a640 1 -- 172.21.15.76:0/3001063753 >> 172.21.15.76:0/3001063753 conn(0x7f3d000fbae0 msgr2=0x7f3d000fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:00.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.726+0000 7f3d0619a640 1 -- 172.21.15.76:0/3001063753 shutdown_connections 2024-03-12T06:00:00.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.726+0000 7f3d0619a640 1 -- 172.21.15.76:0/3001063753 wait complete. 2024-03-12T06:00:00.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.727+0000 7f3d0619a640 1 Processor -- start 2024-03-12T06:00:00.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.727+0000 7f3d0619a640 1 -- start start 2024-03-12T06:00:00.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.727+0000 7f3d0619a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d00100280 0x7f3d00198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.727+0000 7f3d0619a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d00105350 0x7f3d00198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.727+0000 7f3d0619a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d00199110 0x7f3d001a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.727+0000 7f3d0619a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d001a3d60 con 0x7f3d00199110 2024-03-12T06:00:00.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.727+0000 7f3d0619a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d001a3ed0 con 0x7f3d00105350 2024-03-12T06:00:00.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.727+0000 7f3d0619a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d001a4090 con 0x7f3d00100280 2024-03-12T06:00:00.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.727+0000 7f3cfffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d00199110 0x7f3d001a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.727+0000 7f3cfffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d00199110 0x7f3d001a36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39972/0 (socket says 172.21.15.76:39972) 2024-03-12T06:00:00.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.727+0000 7f3cfffff640 1 -- 172.21.15.76:0/1030295400 learned_addr learned my addr 172.21.15.76:0/1030295400 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:00.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.727+0000 7f3cff7fe640 1 --2- 172.21.15.76:0/1030295400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d00100280 0x7f3d00198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.728+0000 7f3cfeffd640 1 --2- 172.21.15.76:0/1030295400 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d00105350 0x7f3d00198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.728+0000 7f3cfffff640 1 -- 172.21.15.76:0/1030295400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d00100280 msgr2=0x7f3d00198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.728+0000 7f3cfffff640 1 --2- 172.21.15.76:0/1030295400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d00100280 0x7f3d00198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.728+0000 7f3cfffff640 1 -- 172.21.15.76:0/1030295400 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d00105350 msgr2=0x7f3d00198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.728+0000 7f3cfffff640 1 --2- 172.21.15.76:0/1030295400 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d00105350 0x7f3d00198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.728+0000 7f3cfffff640 1 -- 172.21.15.76:0/1030295400 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf8009900 con 0x7f3d00199110 2024-03-12T06:00:00.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.728+0000 7f3cfffff640 1 --2- 172.21.15.76:0/1030295400 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d00199110 0x7f3d001a36c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8004af0 tx=0x7f3cf8032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:00.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.728+0000 7f3cfcff9640 1 -- 172.21.15.76:0/1030295400 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf80328b0 con 0x7f3d00199110 2024-03-12T06:00:00.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.728+0000 7f3cfcff9640 1 -- 172.21.15.76:0/1030295400 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cf8003e50 con 0x7f3d00199110 2024-03-12T06:00:00.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.728+0000 7f3d0619a640 1 -- 172.21.15.76:0/1030295400 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d001a4310 con 0x7f3d00199110 2024-03-12T06:00:00.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.728+0000 7f3cfcff9640 1 -- 172.21.15.76:0/1030295400 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8004740 con 0x7f3d00199110 2024-03-12T06:00:00.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.728+0000 7f3d0619a640 1 -- 172.21.15.76:0/1030295400 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d001a47c0 con 0x7f3d00199110 2024-03-12T06:00:00.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.730+0000 7f3cfcff9640 1 -- 172.21.15.76:0/1030295400 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3cf800a040 con 0x7f3d00199110 2024-03-12T06:00:00.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3d08d74740 1 Processor -- start 2024-03-12T06:00:00.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3d08d74740 1 -- start start 2024-03-12T06:00:00.745 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3d08d74740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 0x55e8942e5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3d08d74740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942e5d30 0x55e8942ee2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3cfcff9640 1 --2- 172.21.15.76:0/1030295400 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3ce80a1ba0 0x7f3ce80a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3d08d74740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942ee8b0 0x55e8942f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3d08d74740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e894253f90 con 0x55e8942ee8b0 2024-03-12T06:00:00.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3d08d74740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8942514c0 con 0x55e8942e5d30 2024-03-12T06:00:00.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3d08d74740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89424cca0 con 0x55e8942e5380 2024-03-12T06:00:00.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3cfffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942ee8b0 0x55e8942f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3cfeffd640 1 --2- 172.21.15.76:0/1030295400 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3ce80a1ba0 0x7f3ce80a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3cfffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942ee8b0 0x55e8942f0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39978/0 (socket says 172.21.15.76:39978) 2024-03-12T06:00:00.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3cfffff640 1 -- 172.21.15.76:0/2293058540 learned_addr learned my addr 172.21.15.76:0/2293058540 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:00.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3cff7fe640 1 --2- 172.21.15.76:0/2293058540 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942e5d30 0x55e8942ee2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3cfcff9640 1 -- 172.21.15.76:0/1030295400 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3cf80f0320 con 0x7f3d00199110 2024-03-12T06:00:00.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3cfeffd640 1 --2- 172.21.15.76:0/2293058540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 0x55e8942e5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3cfffff640 1 -- 172.21.15.76:0/2293058540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 msgr2=0x55e8942e5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.731+0000 7f3cfffff640 1 --2- 172.21.15.76:0/2293058540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 0x55e8942e5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.732+0000 7f3cfffff640 1 -- 172.21.15.76:0/2293058540 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942e5d30 msgr2=0x55e8942ee2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.732+0000 7f3cfffff640 1 --2- 172.21.15.76:0/2293058540 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942e5d30 0x55e8942ee2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.732+0000 7f3cfffff640 1 -- 172.21.15.76:0/2293058540 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e89420f0b0 con 0x55e8942ee8b0 2024-03-12T06:00:00.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.732+0000 7f3cfeffd640 1 --2- 172.21.15.76:0/1030295400 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3ce80a1ba0 0x7f3ce80a4060 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3d0006a3b0 tx=0x7f3cf4012820 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:00.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.732+0000 7f3cfffff640 1 --2- 172.21.15.76:0/2293058540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942ee8b0 0x55e8942f0d00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80bd020 tx=0x7f3cf80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=740acb1f911194e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:00.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.732+0000 7f3ce5ffb640 1 -- 172.21.15.76:0/2293058540 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf80be1c0 con 0x55e8942ee8b0 2024-03-12T06:00:00.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.732+0000 7f3ce5ffb640 1 -- 172.21.15.76:0/2293058540 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cf80bd790 con 0x55e8942ee8b0 2024-03-12T06:00:00.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.732+0000 7f3ce5ffb640 1 -- 172.21.15.76:0/2293058540 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf80bda20 con 0x55e8942ee8b0 2024-03-12T06:00:00.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 -- 172.21.15.76:0/2293058540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942ee8b0 msgr2=0x55e8942f0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 --2- 172.21.15.76:0/2293058540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942ee8b0 0x55e8942f0d00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80bd020 tx=0x7f3cf80095c0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 -- 172.21.15.76:0/2293058540 shutdown_connections 2024-03-12T06:00:00.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 --2- 172.21.15.76:0/2293058540 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942ee8b0 0x55e8942f0d00 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 --2- 172.21.15.76:0/2293058540 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942e5d30 0x55e8942ee2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 --2- 172.21.15.76:0/2293058540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 0x55e8942e5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 -- 172.21.15.76:0/2293058540 >> 172.21.15.76:0/2293058540 conn(0x55e8942df840 msgr2=0x55e8942dfc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:00.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 -- 172.21.15.76:0/2293058540 shutdown_connections 2024-03-12T06:00:00.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 -- 172.21.15.76:0/2293058540 wait complete. 2024-03-12T06:00:00.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 -- start start 2024-03-12T06:00:00.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 Processor -- start 2024-03-12T06:00:00.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 0x55e8942eb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942e5d30 0x55e8942ebd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942ee8b0 0x55e8942ec250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.733+0000 7f3d08d74740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8941f1f40 con 0x55e8942ee8b0 2024-03-12T06:00:00.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3d08d74740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e893f204b0 con 0x55e8942e5d30 2024-03-12T06:00:00.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3d08d74740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8941d2670 con 0x55e8942e5380 2024-03-12T06:00:00.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3d08d74740 10 client.0 _ll_register_callbacks cb 0x55e89423ff70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:00.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cfffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942ee8b0 0x55e8942ec250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cfffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942ee8b0 0x55e8942ec250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39986/0 (socket says 172.21.15.76:39986) 2024-03-12T06:00:00.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cfffff640 1 -- 172.21.15.76:0/1277092757 learned_addr learned my addr 172.21.15.76:0/1277092757 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:00.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cfeffd640 1 --2- 172.21.15.76:0/1277092757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942e5d30 0x55e8942ebd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cff7fe640 1 --2- 172.21.15.76:0/1277092757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 0x55e8942eb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cfffff640 1 -- 172.21.15.76:0/1277092757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 msgr2=0x55e8942eb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cfffff640 1 --2- 172.21.15.76:0/1277092757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 0x55e8942eb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cfffff640 1 -- 172.21.15.76:0/1277092757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942e5d30 msgr2=0x55e8942ebd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cfffff640 1 --2- 172.21.15.76:0/1277092757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942e5d30 0x55e8942ebd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cfffff640 1 -- 172.21.15.76:0/1277092757 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d001a47c0 con 0x55e8942ee8b0 2024-03-12T06:00:00.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cfffff640 1 --2- 172.21.15.76:0/1277092757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942ee8b0 0x55e8942ec250 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80b9a60 tx=0x7f3cf80edb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d9e39d7a756b0e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:00.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cdeffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:00.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cdeffd640 1 -- 172.21.15.76:0/1277092757 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf80c11e0 con 0x55e8942ee8b0 2024-03-12T06:00:00.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cdeffd640 1 -- 172.21.15.76:0/1277092757 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3cf80c5830 con 0x55e8942ee8b0 2024-03-12T06:00:00.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3d08d74740 1 -- 172.21.15.76:0/1277092757 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e894226f40 con 0x55e8942ee8b0 2024-03-12T06:00:00.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3cdeffd640 1 -- 172.21.15.76:0/1277092757 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3cf80b9700 con 0x55e8942ee8b0 2024-03-12T06:00:00.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3d08d74740 10 client.16527 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:00.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.734+0000 7f3d08d74740 1 -- 172.21.15.76:0/1277092757 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e89424b320 con 0x55e8942ee8b0 2024-03-12T06:00:00.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.735+0000 7f3cdeffd640 1 -- 172.21.15.76:0/1277092757 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e894226f40 con 0x55e8942ee8b0 2024-03-12T06:00:00.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.735+0000 7f3cdeffd640 1 -- 172.21.15.76:0/1277092757 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f3cf80bd2f0 con 0x55e8942ee8b0 2024-03-12T06:00:00.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.735+0000 7f3d08d74740 10 client.16527 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:00.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.735+0000 7f3d08d74740 10 client.16527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:00:00.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.735+0000 7f3d08d74740 20 client.16527 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:00.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.735+0000 7f3d08d74740 1 --2- 172.21.15.76:0/1277092757 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e8942fe7a0 0x55e894300bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.735+0000 7f3d08d74740 4 client.16527 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:00:00.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.735+0000 7f3d08d74740 1 -- 172.21.15.76:0/1277092757 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e894243310 con 0x55e8942fe7a0 2024-03-12T06:00:00.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.735+0000 7f3cff7fe640 1 --2- 172.21.15.76:0/1277092757 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e8942fe7a0 0x55e894300bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.735+0000 7f3cdeffd640 10 client.16527 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:00.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.735+0000 7f3cff7fe640 1 --2- 172.21.15.76:0/1277092757 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e8942fe7a0 0x55e894300bc0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e3bfd2c4fe751ce server_cookie=daf072a9d4319435 in_seq=0 out_seq=0 2024-03-12T06:00:00.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.736+0000 7f3cdeffd640 1 -- 172.21.15.76:0/1277092757 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e894243310 con 0x55e8942fe7a0 2024-03-12T06:00:00.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.736+0000 7f3cdeffd640 10 client.16527 handle_command_reply: tid=0 2024-03-12T06:00:00.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.736+0000 7f3d08d74740 1 client.16527 shutdown 2024-03-12T06:00:00.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.736+0000 7f3d08d74740 1 -- 172.21.15.76:0/1277092757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942ee8b0 msgr2=0x55e8942ec250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.736+0000 7f3d08d74740 1 --2- 172.21.15.76:0/1277092757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942ee8b0 0x55e8942ec250 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80b9a60 tx=0x7f3cf80edb50 comp rx=0 tx=0).stop 2024-03-12T06:00:00.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.736+0000 7f3d08d74740 1 -- 172.21.15.76:0/1277092757 shutdown_connections 2024-03-12T06:00:00.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.736+0000 7f3d08d74740 1 --2- 172.21.15.76:0/1277092757 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e8942fe7a0 0x55e894300bc0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.736+0000 7f3d08d74740 1 --2- 172.21.15.76:0/1277092757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942ee8b0 0x55e8942ec250 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.736+0000 7f3d08d74740 1 --2- 172.21.15.76:0/1277092757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942e5d30 0x55e8942ebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.736+0000 7f3cdeffd640 0 client.16527 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:00.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.736+0000 7f3d08d74740 1 --2- 172.21.15.76:0/1277092757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 0x55e8942eb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.736+0000 7f3d08d74740 1 -- 172.21.15.76:0/1277092757 >> 172.21.15.76:0/1277092757 conn(0x55e8942df840 msgr2=0x55e8942e17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:00.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.736+0000 7f3d08d74740 1 -- 172.21.15.76:0/1277092757 shutdown_connections 2024-03-12T06:00:00.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.736+0000 7f3d08d74740 1 -- 172.21.15.76:0/1277092757 wait complete. 2024-03-12T06:00:00.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.737+0000 7f3d08d74740 20 client.16527 trim_cache size 0 max 16384 2024-03-12T06:00:00.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.751+0000 7f3d08d74740 1 Processor -- start 2024-03-12T06:00:00.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3d08d74740 1 -- start start 2024-03-12T06:00:00.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3d08d74740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 0x55e8942ede90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3d08d74740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942e5d30 0x55e8942ec070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3d08d74740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942ee8b0 0x55e8942e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3d08d74740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e894305dd0 con 0x55e8942e5d30 2024-03-12T06:00:00.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3d08d74740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8942f1160 con 0x55e8942ee8b0 2024-03-12T06:00:00.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3d08d74740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8942fb610 con 0x55e8942e5380 2024-03-12T06:00:00.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3cfeffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942e5d30 0x55e8942ec070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3cfeffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942e5d30 0x55e8942ec070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39998/0 (socket says 172.21.15.76:39998) 2024-03-12T06:00:00.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3cfeffd640 1 -- 172.21.15.76:0/3944263946 learned_addr learned my addr 172.21.15.76:0/3944263946 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:00.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3cfffff640 1 --2- 172.21.15.76:0/3944263946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942ee8b0 0x55e8942e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3cff7fe640 1 --2- 172.21.15.76:0/3944263946 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 0x55e8942ede90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3cfeffd640 1 -- 172.21.15.76:0/3944263946 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 msgr2=0x55e8942ede90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3cfeffd640 1 --2- 172.21.15.76:0/3944263946 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 0x55e8942ede90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3cfeffd640 1 -- 172.21.15.76:0/3944263946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942ee8b0 msgr2=0x55e8942e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3cfeffd640 1 --2- 172.21.15.76:0/3944263946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942ee8b0 0x55e8942e8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3cfeffd640 1 -- 172.21.15.76:0/3944263946 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e8941e7c00 con 0x55e8942e5d30 2024-03-12T06:00:00.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3cfffff640 1 --2- 172.21.15.76:0/3944263946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942ee8b0 0x55e8942e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:00:00.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3cff7fe640 1 --2- 172.21.15.76:0/3944263946 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 0x55e8942ede90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:00:00.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3cfeffd640 1 --2- 172.21.15.76:0/3944263946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942e5d30 0x55e8942ec070 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f3d00069630 tx=0x7f3cf401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41557a95a0713954 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:00.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.752+0000 7f3ce57fa640 1 -- 172.21.15.76:0/3944263946 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4029670 con 0x55e8942e5d30 2024-03-12T06:00:00.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.753+0000 7f3ce57fa640 1 -- 172.21.15.76:0/3944263946 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cf40044a0 con 0x55e8942e5d30 2024-03-12T06:00:00.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.753+0000 7f3ce57fa640 1 -- 172.21.15.76:0/3944263946 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4010bc0 con 0x55e8942e5d30 2024-03-12T06:00:00.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.753+0000 7f3d08d74740 1 -- 172.21.15.76:0/3944263946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942e5d30 msgr2=0x55e8942ec070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.753+0000 7f3d08d74740 1 --2- 172.21.15.76:0/3944263946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942e5d30 0x55e8942ec070 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f3d00069630 tx=0x7f3cf401d000 comp rx=0 tx=0).stop 2024-03-12T06:00:00.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.753+0000 7f3d08d74740 1 -- 172.21.15.76:0/3944263946 shutdown_connections 2024-03-12T06:00:00.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.753+0000 7f3d08d74740 1 --2- 172.21.15.76:0/3944263946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942ee8b0 0x55e8942e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.753+0000 7f3d08d74740 1 --2- 172.21.15.76:0/3944263946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942e5d30 0x55e8942ec070 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.753+0000 7f3d08d74740 1 --2- 172.21.15.76:0/3944263946 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5380 0x55e8942ede90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.753+0000 7f3d08d74740 1 -- 172.21.15.76:0/3944263946 >> 172.21.15.76:0/3944263946 conn(0x55e8942df840 msgr2=0x55e8942e6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:00.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.753+0000 7f3d08d74740 1 -- 172.21.15.76:0/3944263946 shutdown_connections 2024-03-12T06:00:00.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.753+0000 7f3d08d74740 1 -- 172.21.15.76:0/3944263946 wait complete. 2024-03-12T06:00:00.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.753+0000 7f3d08d74740 1 -- start start 2024-03-12T06:00:00.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3d08d74740 1 Processor -- start 2024-03-12T06:00:00.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3d08d74740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942e5380 0x55e8942f4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3d08d74740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5d30 0x55e8942f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3d08d74740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942ee8b0 0x55e894354cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3d08d74740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8942f5360 con 0x55e8942e5380 2024-03-12T06:00:00.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3d08d74740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8943551f0 con 0x55e8942ee8b0 2024-03-12T06:00:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3d08d74740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e894355330 con 0x55e8942e5d30 2024-03-12T06:00:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3d08d74740 10 client.0 _ll_register_callbacks cb 0x7f3cf80048a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3cff7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942e5380 0x55e8942f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3cff7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942e5380 0x55e8942f4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40010/0 (socket says 172.21.15.76:40010) 2024-03-12T06:00:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3cff7fe640 1 -- 172.21.15.76:0/3018295372 learned_addr learned my addr 172.21.15.76:0/3018295372 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:00.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3cfeffd640 1 --2- 172.21.15.76:0/3018295372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5d30 0x55e8942f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3cfffff640 1 --2- 172.21.15.76:0/3018295372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942ee8b0 0x55e894354cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3cff7fe640 1 -- 172.21.15.76:0/3018295372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5d30 msgr2=0x55e8942f4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3cff7fe640 1 --2- 172.21.15.76:0/3018295372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5d30 0x55e8942f4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3cff7fe640 1 -- 172.21.15.76:0/3018295372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942ee8b0 msgr2=0x55e894354cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3cff7fe640 1 --2- 172.21.15.76:0/3018295372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942ee8b0 0x55e894354cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3cff7fe640 1 -- 172.21.15.76:0/3018295372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf40041a0 con 0x55e8942e5380 2024-03-12T06:00:00.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3cfffff640 1 --2- 172.21.15.76:0/3018295372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942ee8b0 0x55e894354cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:00:00.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3cff7fe640 1 --2- 172.21.15.76:0/3018295372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942e5380 0x55e8942f4720 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0009fd0 tx=0x7f3cf0012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7eab624c5105f8a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:00.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.754+0000 7f3cdeffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:00.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3cdeffd640 1 -- 172.21.15.76:0/3018295372 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0004230 con 0x55e8942e5380 2024-03-12T06:00:00.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3cdeffd640 1 -- 172.21.15.76:0/3018295372 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3cf0004390 con 0x55e8942e5380 2024-03-12T06:00:00.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3d08d74740 1 -- 172.21.15.76:0/3018295372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e894272900 con 0x55e8942e5380 2024-03-12T06:00:00.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3cdeffd640 1 -- 172.21.15.76:0/3018295372 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3cf00035f0 con 0x55e8942e5380 2024-03-12T06:00:00.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3d08d74740 10 client.16533 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:00.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3d08d74740 1 -- 172.21.15.76:0/3018295372 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e89420d910 con 0x55e8942e5380 2024-03-12T06:00:00.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3cdeffd640 1 -- 172.21.15.76:0/3018295372 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e894272900 con 0x55e8942e5380 2024-03-12T06:00:00.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3cdeffd640 1 -- 172.21.15.76:0/3018295372 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f3cf001f3e0 con 0x55e8942e5380 2024-03-12T06:00:00.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3d08d74740 10 client.16533 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:00.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3d08d74740 10 client.16533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:00:00.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3d08d74740 20 client.16533 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:00.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3d08d74740 1 --2- 172.21.15.76:0/3018295372 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e894356b50 0x55e894358f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3d08d74740 4 client.16533 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T06:00:00.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3d08d74740 1 -- 172.21.15.76:0/3018295372 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e893f4c860 con 0x55e894356b50 2024-03-12T06:00:00.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.755+0000 7f3cfeffd640 1 --2- 172.21.15.76:0/3018295372 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e894356b50 0x55e894358f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.756+0000 7f3cdeffd640 10 client.16533 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:00.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.756+0000 7f3cfeffd640 1 --2- 172.21.15.76:0/3018295372 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e894356b50 0x55e894358f70 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b37d10c6e1bf73d3 server_cookie=23664bc494581be9 in_seq=0 out_seq=0 2024-03-12T06:00:00.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3cdeffd640 1 -- 172.21.15.76:0/3018295372 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e893f4c860 con 0x55e894356b50 2024-03-12T06:00:00.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3cdeffd640 10 client.16533 handle_command_reply: tid=0 2024-03-12T06:00:00.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3d08d74740 1 client.16533 shutdown 2024-03-12T06:00:00.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3d08d74740 1 -- 172.21.15.76:0/3018295372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942e5380 msgr2=0x55e8942f4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3d08d74740 1 --2- 172.21.15.76:0/3018295372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942e5380 0x55e8942f4720 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0009fd0 tx=0x7f3cf0012820 comp rx=0 tx=0).stop 2024-03-12T06:00:00.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3d08d74740 1 -- 172.21.15.76:0/3018295372 shutdown_connections 2024-03-12T06:00:00.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3d08d74740 1 --2- 172.21.15.76:0/3018295372 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e8942ee8b0 0x55e894354cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3d08d74740 1 --2- 172.21.15.76:0/3018295372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e8942e5d30 0x55e8942f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3d08d74740 1 --2- 172.21.15.76:0/3018295372 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e894356b50 0x55e894358f70 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3d08d74740 1 --2- 172.21.15.76:0/3018295372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e8942e5380 0x55e8942f4720 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3cdeffd640 0 client.16533 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:00.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3d08d74740 1 -- 172.21.15.76:0/3018295372 >> 172.21.15.76:0/3018295372 conn(0x55e8942df840 msgr2=0x55e8942de050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:00.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3d08d74740 1 -- 172.21.15.76:0/3018295372 shutdown_connections 2024-03-12T06:00:00.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3d08d74740 1 -- 172.21.15.76:0/3018295372 wait complete. 2024-03-12T06:00:00.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.757+0000 7f3d08d74740 20 client.16533 trim_cache size 0 max 16384 2024-03-12T06:00:00.775 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:00:00.775 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T06:00:00.775 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "df275c43-2ae0-4489-b2e0-da233e54602c", 2024-03-12T06:00:00.775 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T06:00:00.775 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:00:00.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.758+0000 7f3d0619a640 1 -- 172.21.15.76:0/1030295400 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3ce80a1ba0 msgr2=0x7f3ce80a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.758+0000 7f3d0619a640 1 --2- 172.21.15.76:0/1030295400 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3ce80a1ba0 0x7f3ce80a4060 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3d0006a3b0 tx=0x7f3cf4012820 comp rx=0 tx=0).stop 2024-03-12T06:00:00.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.758+0000 7f3d0619a640 1 -- 172.21.15.76:0/1030295400 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d00199110 msgr2=0x7f3d001a36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.758+0000 7f3d0619a640 1 --2- 172.21.15.76:0/1030295400 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d00199110 0x7f3d001a36c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8004af0 tx=0x7f3cf8032570 comp rx=0 tx=0).stop 2024-03-12T06:00:00.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.758+0000 7f3d0619a640 1 -- 172.21.15.76:0/1030295400 shutdown_connections 2024-03-12T06:00:00.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.758+0000 7f3d0619a640 1 --2- 172.21.15.76:0/1030295400 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f3ce80a1ba0 0x7f3ce80a4060 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.758+0000 7f3d0619a640 1 --2- 172.21.15.76:0/1030295400 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d00199110 0x7f3d001a36c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.777 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.758+0000 7f3d0619a640 1 --2- 172.21.15.76:0/1030295400 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3d00105350 0x7f3d00198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.758+0000 7f3d0619a640 1 --2- 172.21.15.76:0/1030295400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3d00100280 0x7f3d00198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.758+0000 7f3d0619a640 1 -- 172.21.15.76:0/1030295400 >> 172.21.15.76:0/1030295400 conn(0x7f3d000fbae0 msgr2=0x7f3d000fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:00.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.758+0000 7f3d0619a640 1 -- 172.21.15.76:0/1030295400 shutdown_connections 2024-03-12T06:00:00.778 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.758+0000 7f3d0619a640 1 -- 172.21.15.76:0/1030295400 wait complete. 2024-03-12T06:00:00.778 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:00:00.778 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:00:00.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.903+0000 7fb6d472c640 1 -- 172.21.15.76:0/2046401979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6cc10d720 msgr2=0x7fb6cc10db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.903+0000 7fb6d472c640 1 --2- 172.21.15.76:0/2046401979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6cc10d720 0x7fb6cc10db00 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009930 tx=0x7fb6c802bc30 comp rx=0 tx=0).stop 2024-03-12T06:00:00.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.903+0000 7fb6d472c640 1 -- 172.21.15.76:0/2046401979 shutdown_connections 2024-03-12T06:00:00.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.903+0000 7fb6d472c640 1 --2- 172.21.15.76:0/2046401979 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6cc1080e0 0x7fb6cc1145e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.903+0000 7fb6d472c640 1 --2- 172.21.15.76:0/2046401979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6cc107720 0x7fb6cc107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.903+0000 7fb6d472c640 1 --2- 172.21.15.76:0/2046401979 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6cc10d720 0x7fb6cc10db00 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.903+0000 7fb6d472c640 1 -- 172.21.15.76:0/2046401979 >> 172.21.15.76:0/2046401979 conn(0x7fb6cc076f90 msgr2=0x7fb6cc105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:00.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.903+0000 7fb6d472c640 1 -- 172.21.15.76:0/2046401979 shutdown_connections 2024-03-12T06:00:00.910 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.903+0000 7fb6d472c640 1 -- 172.21.15.76:0/2046401979 wait complete. 2024-03-12T06:00:00.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.904+0000 7fb6d472c640 1 Processor -- start 2024-03-12T06:00:00.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.904+0000 7fb6d472c640 1 -- start start 2024-03-12T06:00:00.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.904+0000 7fb6d472c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6cc107720 0x7fb6cc19de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.904+0000 7fb6d472c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6cc1080e0 0x7fb6cc19e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.904+0000 7fb6d472c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6cc10d720 0x7fb6cc1a2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.904+0000 7fb6d472c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc1a2d20 con 0x7fb6cc107720 2024-03-12T06:00:00.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.904+0000 7fb6d472c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc1a2e90 con 0x7fb6cc1080e0 2024-03-12T06:00:00.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.904+0000 7fb6d472c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc1a3050 con 0x7fb6cc10d720 2024-03-12T06:00:00.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.904+0000 7fb6d24c8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6cc107720 0x7fb6cc19de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.904+0000 7fb6d24c8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6cc107720 0x7fb6cc19de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40036/0 (socket says 172.21.15.76:40036) 2024-03-12T06:00:00.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.904+0000 7fb6d24c8640 1 -- 172.21.15.76:0/3040906973 learned_addr learned my addr 172.21.15.76:0/3040906973 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:00.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.904+0000 7fb6d1cc7640 1 --2- 172.21.15.76:0/3040906973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6cc1080e0 0x7fb6cc19e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.904+0000 7fb6d2cc9640 1 --2- 172.21.15.76:0/3040906973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6cc10d720 0x7fb6cc1a2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.905+0000 7fb6d24c8640 1 -- 172.21.15.76:0/3040906973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6cc10d720 msgr2=0x7fb6cc1a2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.905+0000 7fb6d24c8640 1 --2- 172.21.15.76:0/3040906973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6cc10d720 0x7fb6cc1a2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.905+0000 7fb6d24c8640 1 -- 172.21.15.76:0/3040906973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6cc1080e0 msgr2=0x7fb6cc19e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.905+0000 7fb6d24c8640 1 --2- 172.21.15.76:0/3040906973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6cc1080e0 0x7fb6cc19e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.905+0000 7fb6d24c8640 1 -- 172.21.15.76:0/3040906973 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c8009560 con 0x7fb6cc107720 2024-03-12T06:00:00.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.905+0000 7fb6d24c8640 1 --2- 172.21.15.76:0/3040906973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6cc107720 0x7fb6cc19de60 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8000c00 tx=0x7fb6c802cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:00.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.905+0000 7fb6c37fe640 1 -- 172.21.15.76:0/3040906973 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8004350 con 0x7fb6cc107720 2024-03-12T06:00:00.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.905+0000 7fb6c37fe640 1 -- 172.21.15.76:0/3040906973 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6c80044b0 con 0x7fb6cc107720 2024-03-12T06:00:00.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.905+0000 7fb6d472c640 1 -- 172.21.15.76:0/3040906973 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6cc1a32d0 con 0x7fb6cc107720 2024-03-12T06:00:00.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.905+0000 7fb6c37fe640 1 -- 172.21.15.76:0/3040906973 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8003d00 con 0x7fb6cc107720 2024-03-12T06:00:00.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.905+0000 7fb6d472c640 1 -- 172.21.15.76:0/3040906973 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6cc1aabb0 con 0x7fb6cc107720 2024-03-12T06:00:00.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.908+0000 7fb6c37fe640 1 -- 172.21.15.76:0/3040906973 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb6c8004df0 con 0x7fb6cc107720 2024-03-12T06:00:00.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.909+0000 7fb6d73dc740 1 Processor -- start 2024-03-12T06:00:00.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.909+0000 7fb6d73dc740 1 -- start start 2024-03-12T06:00:00.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.910+0000 7fb6c37fe640 1 --2- 172.21.15.76:0/3040906973 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb6b80a1c20 0x7fb6b80a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.910+0000 7fb6d73dc740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec772b0 0x56145ec77690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.910+0000 7fb6d73dc740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec77bd0 0x56145ec80270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.910+0000 7fb6d73dc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 0x56145ec82ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.910+0000 7fb6d73dc740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145ebe6280 con 0x56145ec77bd0 2024-03-12T06:00:00.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.910+0000 7fb6d73dc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145ebe37b0 con 0x56145ec807b0 2024-03-12T06:00:00.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.910+0000 7fb6d73dc740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145ebdef90 con 0x56145ec772b0 2024-03-12T06:00:00.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.910+0000 7fb6d1cc7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec77bd0 0x56145ec80270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.910+0000 7fb6c37fe640 1 -- 172.21.15.76:0/3040906973 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb6c80efdd0 con 0x7fb6cc107720 2024-03-12T06:00:00.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.910+0000 7fb6d1cc7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec77bd0 0x56145ec80270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40038/0 (socket says 172.21.15.76:40038) 2024-03-12T06:00:00.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.910+0000 7fb6d1cc7640 1 -- 172.21.15.76:0/1093282627 learned_addr learned my addr 172.21.15.76:0/1093282627 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:00.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.910+0000 7fb6d2cc9640 1 --2- 172.21.15.76:0/3040906973 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb6b80a1c20 0x7fb6b80a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.911+0000 7fb6d24c8640 1 --2- 172.21.15.76:0/1093282627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec772b0 0x56145ec77690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.911+0000 7fb6d2cc9640 1 --2- 172.21.15.76:0/1093282627 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 0x56145ec82ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.911+0000 7fb6d1cc7640 1 -- 172.21.15.76:0/1093282627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec772b0 msgr2=0x56145ec77690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.911+0000 7fb6d1cc7640 1 --2- 172.21.15.76:0/1093282627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec772b0 0x56145ec77690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.911+0000 7fb6d1cc7640 1 -- 172.21.15.76:0/1093282627 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 msgr2=0x56145ec82ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.911+0000 7fb6d1cc7640 1 --2- 172.21.15.76:0/1093282627 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 0x56145ec82ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.911+0000 7fb6d1cc7640 1 -- 172.21.15.76:0/1093282627 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56145eb9d570 con 0x56145ec77bd0 2024-03-12T06:00:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.911+0000 7fb6d24c8640 1 --2- 172.21.15.76:0/1093282627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec772b0 0x56145ec77690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:00:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.911+0000 7fb6d2cc9640 1 --2- 172.21.15.76:0/1093282627 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 0x56145ec82ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:00:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.911+0000 7fb6d2cc9640 1 --2- 172.21.15.76:0/3040906973 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb6b80a1c20 0x7fb6b80a40e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4005fd0 tx=0x7fb6c4009e90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.911+0000 7fb6d1cc7640 1 --2- 172.21.15.76:0/1093282627 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec77bd0 0x56145ec80270 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00b740 tx=0x7fb6bc00bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=492d96987b3ee33e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.911+0000 7fb6c0ff9640 1 -- 172.21.15.76:0/1093282627 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc003bd0 con 0x56145ec77bd0 2024-03-12T06:00:00.920 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.911+0000 7fb6c0ff9640 1 -- 172.21.15.76:0/1093282627 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6bc003d30 con 0x56145ec77bd0 2024-03-12T06:00:00.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.911+0000 7fb6c0ff9640 1 -- 172.21.15.76:0/1093282627 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc004e00 con 0x56145ec77bd0 2024-03-12T06:00:00.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.912+0000 7fb6d73dc740 1 -- 172.21.15.76:0/1093282627 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec77bd0 msgr2=0x56145ec80270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.912+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/1093282627 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec77bd0 0x56145ec80270 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00b740 tx=0x7fb6bc00bb40 comp rx=0 tx=0).stop 2024-03-12T06:00:00.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.912+0000 7fb6d73dc740 1 -- 172.21.15.76:0/1093282627 shutdown_connections 2024-03-12T06:00:00.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.912+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/1093282627 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 0x56145ec82ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.912+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/1093282627 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec77bd0 0x56145ec80270 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.912+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/1093282627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec772b0 0x56145ec77690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.921 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.912+0000 7fb6d73dc740 1 -- 172.21.15.76:0/1093282627 >> 172.21.15.76:0/1093282627 conn(0x56145ec71990 msgr2=0x56145ec71d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:00.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.912+0000 7fb6d73dc740 1 -- 172.21.15.76:0/1093282627 shutdown_connections 2024-03-12T06:00:00.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.912+0000 7fb6d73dc740 1 -- 172.21.15.76:0/1093282627 wait complete. 2024-03-12T06:00:00.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.913+0000 7fb6d73dc740 1 -- start start 2024-03-12T06:00:00.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.913+0000 7fb6d73dc740 1 Processor -- start 2024-03-12T06:00:00.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.913+0000 7fb6d73dc740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec7d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.913+0000 7fb6d73dc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec77bd0 0x56145ec7db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.913+0000 7fb6d73dc740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec807b0 0x56145ec7e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.913+0000 7fb6d73dc740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145e8b2430 con 0x56145ec772b0 2024-03-12T06:00:00.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.913+0000 7fb6d73dc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145e86ee00 con 0x56145ec77bd0 2024-03-12T06:00:00.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.913+0000 7fb6d73dc740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145eb63fd0 con 0x56145ec807b0 2024-03-12T06:00:00.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.913+0000 7fb6d73dc740 10 client.0 _ll_register_callbacks cb 0x56145ebb9fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:00.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.913+0000 7fb6d1cc7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec77bd0 0x56145ec7db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.913+0000 7fb6d1cc7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec77bd0 0x56145ec7db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:34542/0 (socket says 172.21.15.76:34542) 2024-03-12T06:00:00.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.913+0000 7fb6d1cc7640 1 -- 172.21.15.76:0/2246467232 learned_addr learned my addr 172.21.15.76:0/2246467232 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:00.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.914+0000 7fb6d2cc9640 1 --2- 172.21.15.76:0/2246467232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec807b0 0x56145ec7e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.914+0000 7fb6d24c8640 1 --2- 172.21.15.76:0/2246467232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec7d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.914+0000 7fb6d1cc7640 1 -- 172.21.15.76:0/2246467232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec807b0 msgr2=0x56145ec7e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.914+0000 7fb6d1cc7640 1 --2- 172.21.15.76:0/2246467232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec807b0 0x56145ec7e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.914+0000 7fb6d1cc7640 1 -- 172.21.15.76:0/2246467232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 msgr2=0x56145ec7d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.914+0000 7fb6d1cc7640 1 --2- 172.21.15.76:0/2246467232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec7d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.914+0000 7fb6d1cc7640 1 -- 172.21.15.76:0/2246467232 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6bc004310 con 0x56145ec77bd0 2024-03-12T06:00:00.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.914+0000 7fb6d2cc9640 1 --2- 172.21.15.76:0/2246467232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec807b0 0x56145ec7e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:00:00.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.914+0000 7fb6d1cc7640 1 --2- 172.21.15.76:0/2246467232 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec77bd0 0x56145ec7db70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0057c0 tx=0x7fb6bc004a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b821ddb48e4b44a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:00.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.914+0000 7fb6b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T06:00:00.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.915+0000 7fb6b5ffb640 1 -- 172.21.15.76:0/2246467232 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc004d40 con 0x56145ec77bd0 2024-03-12T06:00:00.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.915+0000 7fb6b5ffb640 1 -- 172.21.15.76:0/2246467232 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb6bc004ea0 con 0x56145ec77bd0 2024-03-12T06:00:00.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.915+0000 7fb6d73dc740 1 -- 172.21.15.76:0/2246467232 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56145e8de7e0 con 0x56145ec77bd0 2024-03-12T06:00:00.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.915+0000 7fb6b5ffb640 1 -- 172.21.15.76:0/2246467232 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6bc005060 con 0x56145ec77bd0 2024-03-12T06:00:00.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.915+0000 7fb6d73dc740 10 client.25859 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:00.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.915+0000 7fb6d73dc740 1 -- 172.21.15.76:0/2246467232 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56145ebdd610 con 0x56145ec77bd0 2024-03-12T06:00:00.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.915+0000 7fb6b5ffb640 1 -- 172.21.15.76:0/2246467232 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56145e8de7e0 con 0x56145ec77bd0 2024-03-12T06:00:00.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.916+0000 7fb6b5ffb640 1 -- 172.21.15.76:0/2246467232 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fb6bc019420 con 0x56145ec77bd0 2024-03-12T06:00:00.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.916+0000 7fb6d73dc740 10 client.25859 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:00.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.916+0000 7fb6d73dc740 10 client.25859 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:00:00.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.916+0000 7fb6d73dc740 20 client.25859 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:00.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.916+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/2246467232 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56145ec90a70 0x56145ec92e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.916+0000 7fb6d73dc740 4 client.25859 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:00:00.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.916+0000 7fb6d73dc740 1 -- 172.21.15.76:0/2246467232 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56145e91b720 con 0x56145ec90a70 2024-03-12T06:00:00.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.916+0000 7fb6d24c8640 1 --2- 172.21.15.76:0/2246467232 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56145ec90a70 0x56145ec92e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.917+0000 7fb6d24c8640 1 --2- 172.21.15.76:0/2246467232 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56145ec90a70 0x56145ec92e90 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=452e752a4563a9b3 server_cookie=f9dd50a9fa442193 in_seq=0 out_seq=0 2024-03-12T06:00:00.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.917+0000 7fb6b5ffb640 10 client.25859 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:00.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.917+0000 7fb6b5ffb640 1 -- 172.21.15.76:0/2246467232 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56145e91b720 con 0x56145ec90a70 2024-03-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.917+0000 7fb6b5ffb640 10 client.25859 handle_command_reply: tid=0 2024-03-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.917+0000 7fb6d73dc740 1 client.25859 shutdown 2024-03-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.918+0000 7fb6d73dc740 1 -- 172.21.15.76:0/2246467232 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec77bd0 msgr2=0x56145ec7db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.918+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/2246467232 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec77bd0 0x56145ec7db70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0057c0 tx=0x7fb6bc004a30 comp rx=0 tx=0).stop 2024-03-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.918+0000 7fb6d73dc740 1 -- 172.21.15.76:0/2246467232 shutdown_connections 2024-03-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.918+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/2246467232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec807b0 0x56145ec7e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.918+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/2246467232 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec77bd0 0x56145ec7db70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.918+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/2246467232 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56145ec90a70 0x56145ec92e90 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.918+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/2246467232 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec7d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.918+0000 7fb6d73dc740 1 -- 172.21.15.76:0/2246467232 >> 172.21.15.76:0/2246467232 conn(0x56145ec71990 msgr2=0x56145ec73730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.918+0000 7fb6b5ffb640 0 client.25859 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.918+0000 7fb6d73dc740 1 -- 172.21.15.76:0/2246467232 shutdown_connections 2024-03-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.918+0000 7fb6d73dc740 1 -- 172.21.15.76:0/2246467232 wait complete. 2024-03-12T06:00:00.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.918+0000 7fb6d73dc740 20 client.25859 trim_cache size 0 max 16384 2024-03-12T06:00:00.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.942+0000 7fb6d73dc740 1 Processor -- start 2024-03-12T06:00:00.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.942+0000 7fb6d73dc740 1 -- start start 2024-03-12T06:00:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.942+0000 7fb6d73dc740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec7fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.942+0000 7fb6d73dc740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec77bd0 0x56145ec7ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.942+0000 7fb6d73dc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 0x56145ec7a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.942+0000 7fb6d73dc740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145ec76f10 con 0x56145ec772b0 2024-03-12T06:00:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.942+0000 7fb6d73dc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145ec98080 con 0x56145ec807b0 2024-03-12T06:00:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.942+0000 7fb6d73dc740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145ec78410 con 0x56145ec77bd0 2024-03-12T06:00:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.942+0000 7fb6d24c8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec7fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.942+0000 7fb6d24c8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec7fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40056/0 (socket says 172.21.15.76:40056) 2024-03-12T06:00:00.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.942+0000 7fb6d24c8640 1 -- 172.21.15.76:0/792697413 learned_addr learned my addr 172.21.15.76:0/792697413 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:00.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.942+0000 7fb6d2cc9640 1 --2- 172.21.15.76:0/792697413 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 0x56145ec7a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.942+0000 7fb6d1cc7640 1 --2- 172.21.15.76:0/792697413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec77bd0 0x56145ec7ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.943+0000 7fb6d24c8640 1 -- 172.21.15.76:0/792697413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec77bd0 msgr2=0x56145ec7ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.943+0000 7fb6d24c8640 1 --2- 172.21.15.76:0/792697413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec77bd0 0x56145ec7ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.943+0000 7fb6d24c8640 1 -- 172.21.15.76:0/792697413 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 msgr2=0x56145ec7a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.943+0000 7fb6d24c8640 1 --2- 172.21.15.76:0/792697413 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 0x56145ec7a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.943+0000 7fb6d24c8640 1 -- 172.21.15.76:0/792697413 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56145eb79ac0 con 0x56145ec772b0 2024-03-12T06:00:00.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.943+0000 7fb6d24c8640 1 --2- 172.21.15.76:0/792697413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec7fd60 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb6c80b90f0 tx=0x7fb6c8034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba40e06909996597 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:00.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.943+0000 7fb6b7fff640 1 -- 172.21.15.76:0/792697413 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c80c30b0 con 0x56145ec772b0 2024-03-12T06:00:00.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.943+0000 7fb6b7fff640 1 -- 172.21.15.76:0/792697413 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6c80bd390 con 0x56145ec772b0 2024-03-12T06:00:00.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.943+0000 7fb6b7fff640 1 -- 172.21.15.76:0/792697413 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c80bb020 con 0x56145ec772b0 2024-03-12T06:00:00.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.944+0000 7fb6d73dc740 1 -- 172.21.15.76:0/792697413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 msgr2=0x56145ec7fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.944+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/792697413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec7fd60 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb6c80b90f0 tx=0x7fb6c8034000 comp rx=0 tx=0).stop 2024-03-12T06:00:00.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.944+0000 7fb6d73dc740 1 -- 172.21.15.76:0/792697413 shutdown_connections 2024-03-12T06:00:00.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.944+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/792697413 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 0x56145ec7a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.944+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/792697413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec77bd0 0x56145ec7ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.944+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/792697413 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec7fd60 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.944+0000 7fb6d73dc740 1 -- 172.21.15.76:0/792697413 >> 172.21.15.76:0/792697413 conn(0x56145ec71990 msgr2=0x56145ec78910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:00.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.945+0000 7fb6d73dc740 1 -- 172.21.15.76:0/792697413 shutdown_connections 2024-03-12T06:00:00.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.945+0000 7fb6d73dc740 1 -- 172.21.15.76:0/792697413 wait complete. 2024-03-12T06:00:00.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.945+0000 7fb6d73dc740 1 -- start start 2024-03-12T06:00:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.945+0000 7fb6d73dc740 1 Processor -- start 2024-03-12T06:00:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.946+0000 7fb6d73dc740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec852d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.946+0000 7fb6d73dc740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec77bd0 0x56145ec85810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.946+0000 7fb6d73dc740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 0x56145ec85d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.946+0000 7fb6d73dc740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145ece60e0 con 0x56145ec772b0 2024-03-12T06:00:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.946+0000 7fb6d73dc740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145ece6250 con 0x56145ec807b0 2024-03-12T06:00:00.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.946+0000 7fb6d73dc740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56145ece6410 con 0x56145ec77bd0 2024-03-12T06:00:00.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.946+0000 7fb6d73dc740 10 client.0 _ll_register_callbacks cb 0x7fb6bc004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:00.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.946+0000 7fb6d24c8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.946+0000 7fb6d24c8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec852d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:40058/0 (socket says 172.21.15.76:40058) 2024-03-12T06:00:00.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.946+0000 7fb6d24c8640 1 -- 172.21.15.76:0/4102072547 learned_addr learned my addr 172.21.15.76:0/4102072547 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.946+0000 7fb6d1cc7640 1 --2- 172.21.15.76:0/4102072547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec77bd0 0x56145ec85810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.946+0000 7fb6d2cc9640 1 --2- 172.21.15.76:0/4102072547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 0x56145ec85d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.947+0000 7fb6d24c8640 1 -- 172.21.15.76:0/4102072547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec77bd0 msgr2=0x56145ec85810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.947+0000 7fb6d24c8640 1 --2- 172.21.15.76:0/4102072547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec77bd0 0x56145ec85810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.947+0000 7fb6d24c8640 1 -- 172.21.15.76:0/4102072547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 msgr2=0x56145ec85d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.947+0000 7fb6d24c8640 1 --2- 172.21.15.76:0/4102072547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 0x56145ec85d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.947+0000 7fb6d24c8640 1 -- 172.21.15.76:0/4102072547 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6cc1a32d0 con 0x56145ec772b0 2024-03-12T06:00:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.947+0000 7fb6d24c8640 1 --2- 172.21.15.76:0/4102072547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec852d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8001eb0 tx=0x7fb6c8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192251a353b29f32 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.947+0000 7fb6b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.947+0000 7fb6b5ffb640 1 -- 172.21.15.76:0/4102072547 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c80ed970 con 0x56145ec772b0 2024-03-12T06:00:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.947+0000 7fb6b5ffb640 1 -- 172.21.15.76:0/4102072547 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb6c80c0de0 con 0x56145ec772b0 2024-03-12T06:00:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.947+0000 7fb6d73dc740 1 -- 172.21.15.76:0/4102072547 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb6bc003d30 con 0x56145ec772b0 2024-03-12T06:00:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.947+0000 7fb6b5ffb640 1 -- 172.21.15.76:0/4102072547 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6c80c4f10 con 0x56145ec772b0 2024-03-12T06:00:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.947+0000 7fb6d73dc740 10 client.16551 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.947+0000 7fb6d73dc740 1 -- 172.21.15.76:0/4102072547 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56145eb9f6a0 con 0x56145ec772b0 2024-03-12T06:00:00.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.948+0000 7fb6b5ffb640 1 -- 172.21.15.76:0/4102072547 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6bc003d30 con 0x56145ec772b0 2024-03-12T06:00:00.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.948+0000 7fb6b5ffb640 1 -- 172.21.15.76:0/4102072547 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fb6c803c070 con 0x56145ec772b0 2024-03-12T06:00:00.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.948+0000 7fb6d73dc740 10 client.16551 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:00.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.948+0000 7fb6d73dc740 10 client.16551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:00:00.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.948+0000 7fb6d73dc740 20 client.16551 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:00.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.948+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/4102072547 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56145ece7830 0x56145ece9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:00.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.948+0000 7fb6d73dc740 4 client.16551 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:00:00.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.948+0000 7fb6d73dc740 1 -- 172.21.15.76:0/4102072547 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56145e96a890 con 0x56145ece7830 2024-03-12T06:00:00.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.948+0000 7fb6d1cc7640 1 --2- 172.21.15.76:0/4102072547 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56145ece7830 0x56145ece9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.949+0000 7fb6d1cc7640 1 --2- 172.21.15.76:0/4102072547 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56145ece7830 0x56145ece9c50 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a434740e4ea2e90 server_cookie=f91057135b4bf7c0 in_seq=0 out_seq=0 2024-03-12T06:00:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.949+0000 7fb6b5ffb640 10 client.16551 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.949+0000 7fb6b5ffb640 1 -- 172.21.15.76:0/4102072547 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56145e96a890 con 0x56145ece7830 2024-03-12T06:00:00.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.949+0000 7fb6b5ffb640 10 client.16551 handle_command_reply: tid=0 2024-03-12T06:00:00.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.949+0000 7fb6d73dc740 1 client.16551 shutdown 2024-03-12T06:00:00.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.949+0000 7fb6d73dc740 1 -- 172.21.15.76:0/4102072547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 msgr2=0x56145ec852d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.949+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/4102072547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec852d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8001eb0 tx=0x7fb6c8009260 comp rx=0 tx=0).stop 2024-03-12T06:00:00.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.950+0000 7fb6d73dc740 1 -- 172.21.15.76:0/4102072547 shutdown_connections 2024-03-12T06:00:00.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.950+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/4102072547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56145ec807b0 0x56145ec85d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.950+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/4102072547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145ec77bd0 0x56145ec85810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.950+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/4102072547 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56145ece7830 0x56145ece9c50 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.950+0000 7fb6d73dc740 1 --2- 172.21.15.76:0/4102072547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56145ec772b0 0x56145ec852d0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.950+0000 7fb6b5ffb640 0 client.16551 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:00.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.950+0000 7fb6d73dc740 1 -- 172.21.15.76:0/4102072547 >> 172.21.15.76:0/4102072547 conn(0x56145ec71990 msgr2=0x56145ec7bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:00.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.950+0000 7fb6d73dc740 1 -- 172.21.15.76:0/4102072547 shutdown_connections 2024-03-12T06:00:00.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.950+0000 7fb6d73dc740 1 -- 172.21.15.76:0/4102072547 wait complete. 2024-03-12T06:00:00.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.950+0000 7fb6d73dc740 20 client.16551 trim_cache size 0 max 16384 2024-03-12T06:00:00.962 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:00:00.962 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T06:00:00.963 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T06:00:00.963 INFO:teuthology.orchestra.run.smithi076.stdout: "df275c43-2ae0-4489-b2e0-da233e54602c": { 2024-03-12T06:00:00.963 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T06:00:00.963 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "df275c43-2ae0-4489-b2e0-da233e54602c", 2024-03-12T06:00:00.963 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T06:00:00.963 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:00:00.963 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:00:00.963 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:00:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.950+0000 7fb6d472c640 1 -- 172.21.15.76:0/3040906973 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb6b80a1c20 msgr2=0x7fb6b80a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.950+0000 7fb6d472c640 1 --2- 172.21.15.76:0/3040906973 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb6b80a1c20 0x7fb6b80a40e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4005fd0 tx=0x7fb6c4009e90 comp rx=0 tx=0).stop 2024-03-12T06:00:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.950+0000 7fb6d472c640 1 -- 172.21.15.76:0/3040906973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6cc107720 msgr2=0x7fb6cc19de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.950+0000 7fb6d472c640 1 --2- 172.21.15.76:0/3040906973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6cc107720 0x7fb6cc19de60 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8000c00 tx=0x7fb6c802cd80 comp rx=0 tx=0).stop 2024-03-12T06:00:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.951+0000 7fb6d472c640 1 -- 172.21.15.76:0/3040906973 shutdown_connections 2024-03-12T06:00:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.951+0000 7fb6d472c640 1 --2- 172.21.15.76:0/3040906973 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb6b80a1c20 0x7fb6b80a40e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.951+0000 7fb6d472c640 1 --2- 172.21.15.76:0/3040906973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6cc10d720 0x7fb6cc1a2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.951+0000 7fb6d472c640 1 --2- 172.21.15.76:0/3040906973 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb6cc1080e0 0x7fb6cc19e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.951+0000 7fb6d472c640 1 --2- 172.21.15.76:0/3040906973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6cc107720 0x7fb6cc19de60 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.951+0000 7fb6d472c640 1 -- 172.21.15.76:0/3040906973 >> 172.21.15.76:0/3040906973 conn(0x7fb6cc076f90 msgr2=0x7fb6cc0775f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.951+0000 7fb6d472c640 1 -- 172.21.15.76:0/3040906973 shutdown_connections 2024-03-12T06:00:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:00.951+0000 7fb6d472c640 1 -- 172.21.15.76:0/3040906973 wait complete. 2024-03-12T06:00:00.969 INFO:tasks.cephfs.filesystem:scrub status for tag:df275c43-2ae0-4489-b2e0-da233e54602c - {'path': '/', 'tag': 'df275c43-2ae0-4489-b2e0-da233e54602c', 'options': 'recursive,force'} 2024-03-12T06:00:01.238 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:01 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:00:01.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:01 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:00:01.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:01 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:00:01.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:01 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:00:01.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:01 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:00:01.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:01 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:00:02.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:02 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T06:00:02.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:02 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:02.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:02 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:00:02.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:02 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:02.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:02 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:02.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:02 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:02.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:02 smithi082 ceph-mon[36382]: pgmap v130: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:02.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:02 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T06:00:02.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:02 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:02.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:02 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:00:02.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:02 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:02.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:02 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:02.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:02 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:02.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:02 smithi088 ceph-mon[36585]: pgmap v130: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:02.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:02 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T06:00:02.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:02 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:02.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:02 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:00:02.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:02 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:02.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:02 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:02.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:02 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:02.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:02 smithi076 ceph-mon[32095]: pgmap v130: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:04.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:04 smithi082 ceph-mon[36382]: pgmap v131: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:00:04.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:04 smithi088 ceph-mon[36585]: pgmap v131: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:00:04.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:04 smithi076 ceph-mon[32095]: pgmap v131: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:00:06.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:06 smithi082 ceph-mon[36382]: pgmap v132: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:00:06.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:06 smithi088 ceph-mon[36585]: pgmap v132: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:00:06.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:06 smithi076 ceph-mon[32095]: pgmap v132: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:00:08.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:08 smithi082 ceph-mon[36382]: pgmap v133: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:00:08.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:08 smithi088 ceph-mon[36585]: pgmap v133: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:00:08.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:08 smithi076 ceph-mon[32095]: pgmap v133: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:00:10.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:10 smithi082 ceph-mon[36382]: pgmap v134: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:00:10.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:10 smithi088 ceph-mon[36585]: pgmap v134: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:00:10.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:10 smithi076 ceph-mon[32095]: pgmap v134: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:00:12.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:12 smithi082 ceph-mon[36382]: pgmap v135: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:00:12.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:12 smithi088 ceph-mon[36585]: pgmap v135: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:00:12.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:12 smithi076 ceph-mon[32095]: pgmap v135: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:00:13.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:13 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:00:13.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:13 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:00:13.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:13 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:00:13.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:13 smithi082 ceph-mon[36382]: pgmap v136: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:00:13.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:13 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:00:13.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:13 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:00:13.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:13 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:00:13.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:13 smithi088 ceph-mon[36585]: pgmap v136: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:00:13.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:13 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:00:13.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:13 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:00:13.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:13 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:00:13.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:13 smithi076 ceph-mon[32095]: pgmap v136: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:00:16.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:16 smithi082 ceph-mon[36382]: pgmap v137: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:00:16.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:16 smithi088 ceph-mon[36585]: pgmap v137: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:00:16.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:16 smithi076 ceph-mon[32095]: pgmap v137: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:00:18.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:18 smithi082 ceph-mon[36382]: pgmap v138: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:00:18.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:18 smithi088 ceph-mon[36585]: pgmap v138: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:00:18.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:18 smithi076 ceph-mon[32095]: pgmap v138: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:00:20.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:20 smithi082 ceph-mon[36382]: pgmap v139: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-12T06:00:20.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:20 smithi088 ceph-mon[36585]: pgmap v139: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-12T06:00:20.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:20 smithi076 ceph-mon[32095]: pgmap v139: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-12T06:00:22.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:22 smithi082 ceph-mon[36382]: pgmap v140: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:22.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:22 smithi088 ceph-mon[36585]: pgmap v140: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:22.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:22 smithi076 ceph-mon[32095]: pgmap v140: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:24.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:24 smithi082 ceph-mon[36382]: pgmap v141: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:24.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:24 smithi088 ceph-mon[36585]: pgmap v141: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:24.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:24 smithi076 ceph-mon[32095]: pgmap v141: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:26.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:26 smithi082 ceph-mon[36382]: pgmap v142: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:26 smithi088 ceph-mon[36585]: pgmap v142: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:26.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:26 smithi076 ceph-mon[32095]: pgmap v142: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:28.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:28 smithi082 ceph-mon[36382]: pgmap v143: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:28 smithi088 ceph-mon[36585]: pgmap v143: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:28.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:28 smithi076 ceph-mon[32095]: pgmap v143: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:30.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:30 smithi082 ceph-mon[36382]: pgmap v144: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:30 smithi088 ceph-mon[36585]: pgmap v144: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:30.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:30 smithi076 ceph-mon[32095]: pgmap v144: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:30.970 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:00:30.971 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:00:31.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.080+0000 7fd9106d4640 1 -- 172.21.15.76:0/1162470882 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908101d30 msgr2=0x7fd908105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.080+0000 7fd9106d4640 1 --2- 172.21.15.76:0/1162470882 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908101d30 0x7fd908105d70 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd904009930 tx=0x7fd90402bc30 comp rx=0 tx=0).stop 2024-03-12T06:00:31.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.080+0000 7fd9106d4640 1 -- 172.21.15.76:0/1162470882 shutdown_connections 2024-03-12T06:00:31.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.080+0000 7fd9106d4640 1 --2- 172.21.15.76:0/1162470882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd908106450 0x7fd90810e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.080+0000 7fd9106d4640 1 --2- 172.21.15.76:0/1162470882 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908101d30 0x7fd908105d70 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.080+0000 7fd9106d4640 1 --2- 172.21.15.76:0/1162470882 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd908101380 0x7fd908101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.080+0000 7fd9106d4640 1 -- 172.21.15.76:0/1162470882 >> 172.21.15.76:0/1162470882 conn(0x7fd9080fcbe0 msgr2=0x7fd9080ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.080+0000 7fd9106d4640 1 -- 172.21.15.76:0/1162470882 shutdown_connections 2024-03-12T06:00:31.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.080+0000 7fd9106d4640 1 -- 172.21.15.76:0/1162470882 wait complete. 2024-03-12T06:00:31.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.080+0000 7fd9106d4640 1 Processor -- start 2024-03-12T06:00:31.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.080+0000 7fd9106d4640 1 -- start start 2024-03-12T06:00:31.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd9106d4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd908101380 0x7fd90819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd9106d4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd908106450 0x7fd90819d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd9106d4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908197610 0x7fd908197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd9106d4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd908198000 con 0x7fd908197610 2024-03-12T06:00:31.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd9106d4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd908198170 con 0x7fd908101380 2024-03-12T06:00:31.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd9106d4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd908198330 con 0x7fd908106450 2024-03-12T06:00:31.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd90ec71640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908197610 0x7fd908197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd90ec71640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908197610 0x7fd908197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50598/0 (socket says 172.21.15.76:50598) 2024-03-12T06:00:31.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd90ec71640 1 -- 172.21.15.76:0/1626675797 learned_addr learned my addr 172.21.15.76:0/1626675797 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd90e470640 1 --2- 172.21.15.76:0/1626675797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd908101380 0x7fd90819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd90dc6f640 1 --2- 172.21.15.76:0/1626675797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd908106450 0x7fd90819d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd90ec71640 1 -- 172.21.15.76:0/1626675797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd908106450 msgr2=0x7fd90819d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd90ec71640 1 --2- 172.21.15.76:0/1626675797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd908106450 0x7fd90819d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd90ec71640 1 -- 172.21.15.76:0/1626675797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd908101380 msgr2=0x7fd90819d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd90ec71640 1 --2- 172.21.15.76:0/1626675797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd908101380 0x7fd90819d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd90ec71640 1 -- 172.21.15.76:0/1626675797 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd904009560 con 0x7fd908197610 2024-03-12T06:00:31.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd90ec71640 1 --2- 172.21.15.76:0/1626675797 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908197610 0x7fd908197ac0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd90000b970 tx=0x7fd90000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd8ff7fe640 1 -- 172.21.15.76:0/1626675797 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90000cb40 con 0x7fd908197610 2024-03-12T06:00:31.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd8ff7fe640 1 -- 172.21.15.76:0/1626675797 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd90000cca0 con 0x7fd908197610 2024-03-12T06:00:31.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd9106d4640 1 -- 172.21.15.76:0/1626675797 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9081a4320 con 0x7fd908197610 2024-03-12T06:00:31.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.081+0000 7fd8ff7fe640 1 -- 172.21.15.76:0/1626675797 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9000077d0 con 0x7fd908197610 2024-03-12T06:00:31.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.082+0000 7fd9106d4640 1 -- 172.21.15.76:0/1626675797 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9081a47b0 con 0x7fd908197610 2024-03-12T06:00:31.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.084+0000 7fd8ff7fe640 1 -- 172.21.15.76:0/1626675797 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd900014440 con 0x7fd908197610 2024-03-12T06:00:31.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.084+0000 7fd913387740 1 Processor -- start 2024-03-12T06:00:31.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.084+0000 7fd913387740 1 -- start start 2024-03-12T06:00:31.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.084+0000 7fd913387740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5589237842b0 0x558923784690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.084+0000 7fd913387740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558923784bd0 0x55892378d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.084+0000 7fd913387740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55892378d7b0 0x55892378fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.084+0000 7fd913387740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589236f3280 con 0x55892378d7b0 2024-03-12T06:00:31.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.084+0000 7fd913387740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589236f07b0 con 0x5589237842b0 2024-03-12T06:00:31.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.084+0000 7fd913387740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589236ebf90 con 0x558923784bd0 2024-03-12T06:00:31.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd8ff7fe640 1 --2- 172.21.15.76:0/1626675797 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8f40a1ba0 0x7fd8f40a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd90dc6f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55892378d7b0 0x55892378fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd90dc6f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55892378d7b0 0x55892378fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50608/0 (socket says 172.21.15.76:50608) 2024-03-12T06:00:31.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd90dc6f640 1 -- 172.21.15.76:0/1555983956 learned_addr learned my addr 172.21.15.76:0/1555983956 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd90e470640 1 --2- 172.21.15.76:0/1555983956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558923784bd0 0x55892378d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd90ec71640 1 --2- 172.21.15.76:0/1626675797 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8f40a1ba0 0x7fd8f40a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd90dc6f640 1 --2- 172.21.15.76:0/1555983956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5589237842b0 0x558923784690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd8ff7fe640 1 -- 172.21.15.76:0/1626675797 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd9000cd040 con 0x7fd908197610 2024-03-12T06:00:31.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd90dc6f640 1 -- 172.21.15.76:0/1555983956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558923784bd0 msgr2=0x55892378d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd90dc6f640 1 --2- 172.21.15.76:0/1555983956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558923784bd0 0x55892378d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd90dc6f640 1 -- 172.21.15.76:0/1555983956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5589237842b0 msgr2=0x558923784690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd90dc6f640 1 --2- 172.21.15.76:0/1555983956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5589237842b0 0x558923784690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd90dc6f640 1 -- 172.21.15.76:0/1555983956 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589236aa570 con 0x55892378d7b0 2024-03-12T06:00:31.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd90dc6f640 1 --2- 172.21.15.76:0/1555983956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55892378d7b0 0x55892378fba0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd904005940 tx=0x7fd904005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9afbddcfaf51f8a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd90ec71640 1 --2- 172.21.15.76:0/1626675797 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8f40a1ba0 0x7fd8f40a4060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd900096210 tx=0x7fd900004a40 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd8fcff9640 1 -- 172.21.15.76:0/1555983956 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90402ca60 con 0x55892378d7b0 2024-03-12T06:00:31.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd8fcff9640 1 -- 172.21.15.76:0/1555983956 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd904004510 con 0x55892378d7b0 2024-03-12T06:00:31.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.085+0000 7fd8fcff9640 1 -- 172.21.15.76:0/1555983956 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9040055f0 con 0x55892378d7b0 2024-03-12T06:00:31.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.086+0000 7fd913387740 1 -- 172.21.15.76:0/1555983956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55892378d7b0 msgr2=0x55892378fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.086+0000 7fd913387740 1 --2- 172.21.15.76:0/1555983956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55892378d7b0 0x55892378fba0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd904005940 tx=0x7fd904005fd0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.086+0000 7fd913387740 1 -- 172.21.15.76:0/1555983956 shutdown_connections 2024-03-12T06:00:31.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.086+0000 7fd913387740 1 --2- 172.21.15.76:0/1555983956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55892378d7b0 0x55892378fba0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.086+0000 7fd913387740 1 --2- 172.21.15.76:0/1555983956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558923784bd0 0x55892378d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.086+0000 7fd913387740 1 --2- 172.21.15.76:0/1555983956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5589237842b0 0x558923784690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.086+0000 7fd913387740 1 -- 172.21.15.76:0/1555983956 >> 172.21.15.76:0/1555983956 conn(0x55892377e990 msgr2=0x55892377ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.086+0000 7fd913387740 1 -- 172.21.15.76:0/1555983956 shutdown_connections 2024-03-12T06:00:31.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.086+0000 7fd913387740 1 -- 172.21.15.76:0/1555983956 wait complete. 2024-03-12T06:00:31.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd913387740 1 -- start start 2024-03-12T06:00:31.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd913387740 1 Processor -- start 2024-03-12T06:00:31.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd913387740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5589237842b0 0x55892378d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd913387740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558923784bd0 0x55892378a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd913387740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55892378d7b0 0x55892378acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd913387740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589233bf430 con 0x558923784bd0 2024-03-12T06:00:31.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd913387740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55892337be00 con 0x55892378d7b0 2024-03-12T06:00:31.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd913387740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558923670fd0 con 0x5589237842b0 2024-03-12T06:00:31.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd913387740 10 client.0 _ll_register_callbacks cb 0x5589236c6fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:31.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd90e470640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558923784bd0 0x55892378a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd90dc6f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5589237842b0 0x55892378d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd90e470640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558923784bd0 0x55892378a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50616/0 (socket says 172.21.15.76:50616) 2024-03-12T06:00:31.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd90e470640 1 -- 172.21.15.76:0/2656516620 learned_addr learned my addr 172.21.15.76:0/2656516620 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd90dc6f640 1 --2- 172.21.15.76:0/2656516620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55892378d7b0 0x55892378acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd90e470640 1 -- 172.21.15.76:0/2656516620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5589237842b0 msgr2=0x55892378d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd90e470640 1 --2- 172.21.15.76:0/2656516620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5589237842b0 0x55892378d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd90e470640 1 -- 172.21.15.76:0/2656516620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55892378d7b0 msgr2=0x55892378acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd90e470640 1 --2- 172.21.15.76:0/2656516620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55892378d7b0 0x55892378acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.087+0000 7fd90e470640 1 -- 172.21.15.76:0/2656516620 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90810e830 con 0x558923784bd0 2024-03-12T06:00:31.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd90e470640 1 --2- 172.21.15.76:0/2656516620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558923784bd0 0x55892378a5d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800cec0 tx=0x7fd8f80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26884b8f6210af28 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd8f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:31.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd8f1ffb640 1 -- 172.21.15.76:0/2656516620 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f80041d0 con 0x558923784bd0 2024-03-12T06:00:31.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd8f1ffb640 1 -- 172.21.15.76:0/2656516620 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd8f8011a50 con 0x558923784bd0 2024-03-12T06:00:31.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd913387740 1 -- 172.21.15.76:0/2656516620 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5589233eb7e0 con 0x558923784bd0 2024-03-12T06:00:31.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd8f1ffb640 1 -- 172.21.15.76:0/2656516620 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8f8011c10 con 0x558923784bd0 2024-03-12T06:00:31.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd913387740 10 client.16563 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:31.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd913387740 1 -- 172.21.15.76:0/2656516620 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5589236ea610 con 0x558923784bd0 2024-03-12T06:00:31.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd8f1ffb640 1 -- 172.21.15.76:0/2656516620 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5589233eb7e0 con 0x558923784bd0 2024-03-12T06:00:31.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd8f1ffb640 1 -- 172.21.15.76:0/2656516620 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fd8f8019500 con 0x558923784bd0 2024-03-12T06:00:31.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd913387740 10 client.16563 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:31.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd913387740 10 client.16563 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:00:31.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd913387740 20 client.16563 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:31.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd913387740 1 --2- 172.21.15.76:0/2656516620 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55892379dae0 0x55892379ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd913387740 4 client.16563 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:00:31.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.088+0000 7fd913387740 1 -- 172.21.15.76:0/2656516620 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558923428720 con 0x55892379dae0 2024-03-12T06:00:31.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.089+0000 7fd90dc6f640 1 --2- 172.21.15.76:0/2656516620 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55892379dae0 0x55892379ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.089+0000 7fd90dc6f640 1 --2- 172.21.15.76:0/2656516620 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55892379dae0 0x55892379ff00 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=185e108bada8c71e server_cookie=f86f472e419b2fe9 in_seq=0 out_seq=0 2024-03-12T06:00:31.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.089+0000 7fd8f1ffb640 10 client.16563 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:31.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd8f1ffb640 1 -- 172.21.15.76:0/2656516620 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7fd90405aa90 con 0x55892379dae0 2024-03-12T06:00:31.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd8f1ffb640 10 client.16563 handle_command_reply: tid=0 2024-03-12T06:00:31.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd913387740 1 client.16563 shutdown 2024-03-12T06:00:31.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd913387740 1 -- 172.21.15.76:0/2656516620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558923784bd0 msgr2=0x55892378a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd913387740 1 --2- 172.21.15.76:0/2656516620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558923784bd0 0x55892378a5d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800cec0 tx=0x7fd8f80039c0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd913387740 1 -- 172.21.15.76:0/2656516620 shutdown_connections 2024-03-12T06:00:31.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd913387740 1 --2- 172.21.15.76:0/2656516620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55892378d7b0 0x55892378acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd913387740 1 --2- 172.21.15.76:0/2656516620 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55892379dae0 0x55892379ff00 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd913387740 1 --2- 172.21.15.76:0/2656516620 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558923784bd0 0x55892378a5d0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd913387740 1 --2- 172.21.15.76:0/2656516620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5589237842b0 0x55892378d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd8f1ffb640 0 client.16563 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:31.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd913387740 1 -- 172.21.15.76:0/2656516620 >> 172.21.15.76:0/2656516620 conn(0x55892377e990 msgr2=0x558923780730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd913387740 1 -- 172.21.15.76:0/2656516620 shutdown_connections 2024-03-12T06:00:31.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd913387740 1 -- 172.21.15.76:0/2656516620 wait complete. 2024-03-12T06:00:31.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.090+0000 7fd913387740 20 client.16563 trim_cache size 0 max 16384 2024-03-12T06:00:31.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd913387740 1 Processor -- start 2024-03-12T06:00:31.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd913387740 1 -- start start 2024-03-12T06:00:31.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd913387740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5589237842b0 0x55892378cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd913387740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558923784bd0 0x55892378d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd913387740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55892378d7b0 0x558923788ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd913387740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558923785440 con 0x558923784bd0 2024-03-12T06:00:31.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd913387740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558923783f10 con 0x55892378d7b0 2024-03-12T06:00:31.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd913387740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589237a5120 con 0x5589237842b0 2024-03-12T06:00:31.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd90e470640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558923784bd0 0x55892378d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd90e470640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558923784bd0 0x55892378d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50632/0 (socket says 172.21.15.76:50632) 2024-03-12T06:00:31.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd90e470640 1 -- 172.21.15.76:0/3834127549 learned_addr learned my addr 172.21.15.76:0/3834127549 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd90dc6f640 1 --2- 172.21.15.76:0/3834127549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55892378d7b0 0x558923788ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd90e470640 1 -- 172.21.15.76:0/3834127549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5589237842b0 msgr2=0x55892378cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd90e470640 1 --2- 172.21.15.76:0/3834127549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5589237842b0 0x55892378cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.105+0000 7fd90e470640 1 -- 172.21.15.76:0/3834127549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55892378d7b0 msgr2=0x558923788ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd90e470640 1 --2- 172.21.15.76:0/3834127549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55892378d7b0 0x558923788ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd90e470640 1 -- 172.21.15.76:0/3834127549 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558923686ac0 con 0x558923784bd0 2024-03-12T06:00:31.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd90dc6f640 1 --2- 172.21.15.76:0/3834127549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55892378d7b0 0x558923788ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:00:31.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd90e470640 1 --2- 172.21.15.76:0/3834127549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558923784bd0 0x55892378d240 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800ce90 tx=0x7fd8f8003ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a389460e853da1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd8f3fff640 1 -- 172.21.15.76:0/3834127549 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8019e50 con 0x558923784bd0 2024-03-12T06:00:31.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd8f3fff640 1 -- 172.21.15.76:0/3834127549 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8f8011d30 con 0x558923784bd0 2024-03-12T06:00:31.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd8f3fff640 1 -- 172.21.15.76:0/3834127549 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f802b450 con 0x558923784bd0 2024-03-12T06:00:31.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd913387740 1 -- 172.21.15.76:0/3834127549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558923784bd0 msgr2=0x55892378d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd913387740 1 --2- 172.21.15.76:0/3834127549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558923784bd0 0x55892378d240 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800ce90 tx=0x7fd8f8003ec0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd913387740 1 -- 172.21.15.76:0/3834127549 shutdown_connections 2024-03-12T06:00:31.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd913387740 1 --2- 172.21.15.76:0/3834127549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55892378d7b0 0x558923788ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd913387740 1 --2- 172.21.15.76:0/3834127549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558923784bd0 0x55892378d240 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd913387740 1 --2- 172.21.15.76:0/3834127549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5589237842b0 0x55892378cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd913387740 1 -- 172.21.15.76:0/3834127549 >> 172.21.15.76:0/3834127549 conn(0x55892377e990 msgr2=0x5589237858b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd913387740 1 -- 172.21.15.76:0/3834127549 shutdown_connections 2024-03-12T06:00:31.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.106+0000 7fd913387740 1 -- 172.21.15.76:0/3834127549 wait complete. 2024-03-12T06:00:31.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd913387740 1 -- start start 2024-03-12T06:00:31.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd913387740 1 Processor -- start 2024-03-12T06:00:31.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd913387740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5589237842b0 0x558923790f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd913387740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558923784bd0 0x5589237914d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd913387740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55892378d7b0 0x558923791a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd913387740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558923794f50 con 0x5589237842b0 2024-03-12T06:00:31.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd913387740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589237f3fe0 con 0x558923784bd0 2024-03-12T06:00:31.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd913387740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589237f4170 con 0x55892378d7b0 2024-03-12T06:00:31.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd913387740 10 client.0 _ll_register_callbacks cb 0x7fd904009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:31.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd90dc6f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5589237842b0 0x558923790f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd90dc6f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5589237842b0 0x558923790f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50646/0 (socket says 172.21.15.76:50646) 2024-03-12T06:00:31.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd90dc6f640 1 -- 172.21.15.76:0/4100218221 learned_addr learned my addr 172.21.15.76:0/4100218221 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd90e470640 1 --2- 172.21.15.76:0/4100218221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558923784bd0 0x5589237914d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd90dc6f640 1 -- 172.21.15.76:0/4100218221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55892378d7b0 msgr2=0x558923791a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd90dc6f640 1 --2- 172.21.15.76:0/4100218221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55892378d7b0 0x558923791a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd90dc6f640 1 -- 172.21.15.76:0/4100218221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558923784bd0 msgr2=0x5589237914d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd90dc6f640 1 --2- 172.21.15.76:0/4100218221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558923784bd0 0x5589237914d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd90dc6f640 1 -- 172.21.15.76:0/4100218221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5589236ea610 con 0x5589237842b0 2024-03-12T06:00:31.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd90e470640 1 --2- 172.21.15.76:0/4100218221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558923784bd0 0x5589237914d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:00:31.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd90dc6f640 1 --2- 172.21.15.76:0/4100218221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5589237842b0 0x558923790f90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x55892377fb70 tx=0x7fd904000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea35746a8e09e8a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.107+0000 7fd8f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:31.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.108+0000 7fd8f1ffb640 1 -- 172.21.15.76:0/4100218221 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90402c860 con 0x5589237842b0 2024-03-12T06:00:31.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.108+0000 7fd8f1ffb640 1 -- 172.21.15.76:0/4100218221 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd904000e40 con 0x5589237842b0 2024-03-12T06:00:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.108+0000 7fd913387740 1 -- 172.21.15.76:0/4100218221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd904004510 con 0x5589237842b0 2024-03-12T06:00:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.108+0000 7fd8f1ffb640 1 -- 172.21.15.76:0/4100218221 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd90402c9c0 con 0x5589237842b0 2024-03-12T06:00:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.108+0000 7fd913387740 10 client.16569 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.108+0000 7fd913387740 1 -- 172.21.15.76:0/4100218221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5589236ac6a0 con 0x5589237842b0 2024-03-12T06:00:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.108+0000 7fd8f1ffb640 1 -- 172.21.15.76:0/4100218221 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd904004510 con 0x5589237842b0 2024-03-12T06:00:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.108+0000 7fd8f1ffb640 1 -- 172.21.15.76:0/4100218221 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fd904041d10 con 0x5589237842b0 2024-03-12T06:00:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.108+0000 7fd913387740 10 client.16569 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.108+0000 7fd913387740 10 client.16569 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:00:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.108+0000 7fd913387740 20 client.16569 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:31.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.108+0000 7fd913387740 1 --2- 172.21.15.76:0/4100218221 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5589237f57c0 0x5589237f7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.108+0000 7fd913387740 4 client.16569 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:00:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.108+0000 7fd913387740 1 -- 172.21.15.76:0/4100218221 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558923477890 con 0x5589237f57c0 2024-03-12T06:00:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd90e470640 1 --2- 172.21.15.76:0/4100218221 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5589237f57c0 0x5589237f7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd8f1ffb640 10 client.16569 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd90e470640 1 --2- 172.21.15.76:0/4100218221 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5589237f57c0 0x5589237f7be0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f67fe6c2b422744 server_cookie=29b7f1002f7f5817 in_seq=0 out_seq=0 2024-03-12T06:00:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd8f1ffb640 1 -- 172.21.15.76:0/4100218221 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558923477890 con 0x5589237f57c0 2024-03-12T06:00:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd8f1ffb640 10 client.16569 handle_command_reply: tid=0 2024-03-12T06:00:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd913387740 1 client.16569 shutdown 2024-03-12T06:00:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd913387740 1 -- 172.21.15.76:0/4100218221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5589237842b0 msgr2=0x558923790f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd913387740 1 --2- 172.21.15.76:0/4100218221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5589237842b0 0x558923790f90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x55892377fb70 tx=0x7fd904000bc0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd913387740 1 -- 172.21.15.76:0/4100218221 shutdown_connections 2024-03-12T06:00:31.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd913387740 1 --2- 172.21.15.76:0/4100218221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55892378d7b0 0x558923791a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd913387740 1 --2- 172.21.15.76:0/4100218221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558923784bd0 0x5589237914d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd913387740 1 --2- 172.21.15.76:0/4100218221 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5589237f57c0 0x5589237f7be0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd913387740 1 --2- 172.21.15.76:0/4100218221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5589237842b0 0x558923790f90 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd8f1ffb640 0 client.16569 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:31.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd913387740 1 -- 172.21.15.76:0/4100218221 >> 172.21.15.76:0/4100218221 conn(0x55892377e990 msgr2=0x558923788de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd913387740 1 -- 172.21.15.76:0/4100218221 shutdown_connections 2024-03-12T06:00:31.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd913387740 1 -- 172.21.15.76:0/4100218221 wait complete. 2024-03-12T06:00:31.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.109+0000 7fd913387740 20 client.16569 trim_cache size 0 max 16384 2024-03-12T06:00:31.126 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:00:31.126 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T06:00:31.126 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T06:00:31.126 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:00:31.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.110+0000 7fd9106d4640 1 -- 172.21.15.76:0/1626675797 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8f40a1ba0 msgr2=0x7fd8f40a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.110+0000 7fd9106d4640 1 --2- 172.21.15.76:0/1626675797 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8f40a1ba0 0x7fd8f40a4060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd900096210 tx=0x7fd900004a40 comp rx=0 tx=0).stop 2024-03-12T06:00:31.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.110+0000 7fd9106d4640 1 -- 172.21.15.76:0/1626675797 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908197610 msgr2=0x7fd908197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.110+0000 7fd9106d4640 1 --2- 172.21.15.76:0/1626675797 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908197610 0x7fd908197ac0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd90000b970 tx=0x7fd90000be30 comp rx=0 tx=0).stop 2024-03-12T06:00:31.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.110+0000 7fd9106d4640 1 -- 172.21.15.76:0/1626675797 shutdown_connections 2024-03-12T06:00:31.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.110+0000 7fd9106d4640 1 --2- 172.21.15.76:0/1626675797 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8f40a1ba0 0x7fd8f40a4060 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.110+0000 7fd9106d4640 1 --2- 172.21.15.76:0/1626675797 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd908197610 0x7fd908197ac0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.110+0000 7fd9106d4640 1 --2- 172.21.15.76:0/1626675797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd908106450 0x7fd90819d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.110+0000 7fd9106d4640 1 --2- 172.21.15.76:0/1626675797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd908101380 0x7fd90819d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.110+0000 7fd9106d4640 1 -- 172.21.15.76:0/1626675797 >> 172.21.15.76:0/1626675797 conn(0x7fd9080fcbe0 msgr2=0x7fd9080feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.110+0000 7fd9106d4640 1 -- 172.21.15.76:0/1626675797 shutdown_connections 2024-03-12T06:00:31.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.110+0000 7fd9106d4640 1 -- 172.21.15.76:0/1626675797 wait complete. 2024-03-12T06:00:31.129 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T06:00:31.129 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T06:00:31.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.227+0000 7fd34aa98640 1 -- 172.21.15.76:0/2760340740 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd344100c30 msgr2=0x7fd344104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.227+0000 7fd34aa98640 1 --2- 172.21.15.76:0/2760340740 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd344100c30 0x7fd344104c70 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd330009930 tx=0x7fd33002bc30 comp rx=0 tx=0).stop 2024-03-12T06:00:31.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.227+0000 7fd34aa98640 1 -- 172.21.15.76:0/2760340740 shutdown_connections 2024-03-12T06:00:31.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.227+0000 7fd34aa98640 1 --2- 172.21.15.76:0/2760340740 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd344105350 0x7fd34410cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.227+0000 7fd34aa98640 1 --2- 172.21.15.76:0/2760340740 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd344100c30 0x7fd344104c70 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.227+0000 7fd34aa98640 1 --2- 172.21.15.76:0/2760340740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd344100280 0x7fd344100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.227+0000 7fd34aa98640 1 -- 172.21.15.76:0/2760340740 >> 172.21.15.76:0/2760340740 conn(0x7fd3440fbb00 msgr2=0x7fd3440fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.227+0000 7fd34aa98640 1 -- 172.21.15.76:0/2760340740 shutdown_connections 2024-03-12T06:00:31.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.227+0000 7fd34aa98640 1 -- 172.21.15.76:0/2760340740 wait complete. 2024-03-12T06:00:31.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd34aa98640 1 Processor -- start 2024-03-12T06:00:31.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd34aa98640 1 -- start start 2024-03-12T06:00:31.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd34aa98640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd344100280 0x7fd344198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd34aa98640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd344105350 0x7fd344198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd34aa98640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd344199120 0x7fd3441a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd34aa98640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3441a3da0 con 0x7fd344105350 2024-03-12T06:00:31.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd34aa98640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3441a3f10 con 0x7fd344100280 2024-03-12T06:00:31.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd34aa98640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3441a40d0 con 0x7fd344199120 2024-03-12T06:00:31.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd343fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd344105350 0x7fd344198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd343fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd344105350 0x7fd344198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:50652/0 (socket says 172.21.15.76:50652) 2024-03-12T06:00:31.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd343fff640 1 -- 172.21.15.76:0/1941578416 learned_addr learned my addr 172.21.15.76:0/1941578416 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd349035640 1 --2- 172.21.15.76:0/1941578416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd344199120 0x7fd3441a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd348834640 1 --2- 172.21.15.76:0/1941578416 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd344100280 0x7fd344198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd343fff640 1 -- 172.21.15.76:0/1941578416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd344199120 msgr2=0x7fd3441a36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd343fff640 1 --2- 172.21.15.76:0/1941578416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd344199120 0x7fd3441a36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd343fff640 1 -- 172.21.15.76:0/1941578416 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd344100280 msgr2=0x7fd344198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd343fff640 1 --2- 172.21.15.76:0/1941578416 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd344100280 0x7fd344198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd343fff640 1 -- 172.21.15.76:0/1941578416 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd330009560 con 0x7fd344105350 2024-03-12T06:00:31.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.228+0000 7fd343fff640 1 --2- 172.21.15.76:0/1941578416 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd344105350 0x7fd344198b90 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd330000c00 tx=0x7fd330004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.229+0000 7fd341ffb640 1 -- 172.21.15.76:0/1941578416 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd330004df0 con 0x7fd344105350 2024-03-12T06:00:31.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.229+0000 7fd341ffb640 1 -- 172.21.15.76:0/1941578416 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd330005430 con 0x7fd344105350 2024-03-12T06:00:31.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.229+0000 7fd34aa98640 1 -- 172.21.15.76:0/1941578416 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3441a4350 con 0x7fd344105350 2024-03-12T06:00:31.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.229+0000 7fd341ffb640 1 -- 172.21.15.76:0/1941578416 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33003e450 con 0x7fd344105350 2024-03-12T06:00:31.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.229+0000 7fd34aa98640 1 -- 172.21.15.76:0/1941578416 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3441a4860 con 0x7fd344105350 2024-03-12T06:00:31.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.230+0000 7fd34aa98640 1 -- 172.21.15.76:0/1941578416 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd344101750 con 0x7fd344105350 2024-03-12T06:00:31.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.231+0000 7fd341ffb640 1 -- 172.21.15.76:0/1941578416 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd330003680 con 0x7fd344105350 2024-03-12T06:00:31.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.232+0000 7fd341ffb640 1 --2- 172.21.15.76:0/1941578416 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd3340a1ba0 0x7fd3340a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.232+0000 7fd348834640 1 --2- 172.21.15.76:0/1941578416 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd3340a1ba0 0x7fd3340a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.232+0000 7fd341ffb640 1 -- 172.21.15.76:0/1941578416 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd3300ef970 con 0x7fd344105350 2024-03-12T06:00:31.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.232+0000 7fd348834640 1 --2- 172.21.15.76:0/1941578416 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd3340a1ba0 0x7fd3340a4060 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd3380058c0 tx=0x7fd33800a250 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.236+0000 7fd341ffb640 1 -- 172.21.15.76:0/1941578416 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd3300b8980 con 0x7fd344105350 2024-03-12T06:00:31.452 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.446+0000 7fd34aa98640 1 -- 172.21.15.76:0/1941578416 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd344108580 con 0x7fd344105350 2024-03-12T06:00:31.453 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.447+0000 7fd341ffb640 1 -- 172.21.15.76:0/1941578416 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11332 (secure 0 0 0) 0x7fd3300bc040 con 0x7fd344105350 2024-03-12T06:00:31.453 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:00:31.453 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T05:58:43.838072+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T06:00:31.453 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 25 2024-03-12T06:00:31.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.449+0000 7fd34aa98640 1 -- 172.21.15.76:0/1941578416 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd3340a1ba0 msgr2=0x7fd3340a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.449+0000 7fd34aa98640 1 --2- 172.21.15.76:0/1941578416 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd3340a1ba0 0x7fd3340a4060 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd3380058c0 tx=0x7fd33800a250 comp rx=0 tx=0).stop 2024-03-12T06:00:31.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.449+0000 7fd34aa98640 1 -- 172.21.15.76:0/1941578416 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd344105350 msgr2=0x7fd344198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.449+0000 7fd34aa98640 1 --2- 172.21.15.76:0/1941578416 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd344105350 0x7fd344198b90 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd330000c00 tx=0x7fd330004270 comp rx=0 tx=0).stop 2024-03-12T06:00:31.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.449+0000 7fd34aa98640 1 -- 172.21.15.76:0/1941578416 shutdown_connections 2024-03-12T06:00:31.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.449+0000 7fd34aa98640 1 --2- 172.21.15.76:0/1941578416 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd3340a1ba0 0x7fd3340a4060 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.449+0000 7fd34aa98640 1 --2- 172.21.15.76:0/1941578416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd344199120 0x7fd3441a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.449+0000 7fd34aa98640 1 --2- 172.21.15.76:0/1941578416 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd344105350 0x7fd344198b90 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.449+0000 7fd34aa98640 1 --2- 172.21.15.76:0/1941578416 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd344100280 0x7fd344198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.449+0000 7fd34aa98640 1 -- 172.21.15.76:0/1941578416 >> 172.21.15.76:0/1941578416 conn(0x7fd3440fbb00 msgr2=0x7fd3440fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.450+0000 7fd34aa98640 1 -- 172.21.15.76:0/1941578416 shutdown_connections 2024-03-12T06:00:31.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.450+0000 7fd34aa98640 1 -- 172.21.15.76:0/1941578416 wait complete. 2024-03-12T06:00:31.466 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:00:31.466 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T06:00:31.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.565+0000 7f46d976c640 1 -- 172.21.15.76:0/187638566 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46d4106450 msgr2=0x7f46d410e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.565+0000 7f46d976c640 1 --2- 172.21.15.76:0/187638566 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46d4106450 0x7f46d410e020 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009cd0 tx=0x7f46cc02c0f0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.565+0000 7f46d976c640 1 -- 172.21.15.76:0/187638566 shutdown_connections 2024-03-12T06:00:31.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.565+0000 7f46d976c640 1 --2- 172.21.15.76:0/187638566 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46d4106450 0x7f46d410e020 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.565+0000 7f46d976c640 1 --2- 172.21.15.76:0/187638566 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46d4101d30 0x7f46d4105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.565+0000 7f46d976c640 1 --2- 172.21.15.76:0/187638566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46d4101380 0x7f46d4101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.565+0000 7f46d976c640 1 -- 172.21.15.76:0/187638566 >> 172.21.15.76:0/187638566 conn(0x7f46d40fcc20 msgr2=0x7f46d40ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.565+0000 7f46d976c640 1 -- 172.21.15.76:0/187638566 shutdown_connections 2024-03-12T06:00:31.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.565+0000 7f46d976c640 1 -- 172.21.15.76:0/187638566 wait complete. 2024-03-12T06:00:31.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d976c640 1 Processor -- start 2024-03-12T06:00:31.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d976c640 1 -- start start 2024-03-12T06:00:31.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d976c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46d4101380 0x7f46d4197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d976c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46d4106450 0x7f46d4197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d976c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46d419bfe0 0x7f46d419c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d976c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d419c9d0 con 0x7f46d419bfe0 2024-03-12T06:00:31.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d976c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d419cb40 con 0x7f46d4106450 2024-03-12T06:00:31.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d976c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d419ccb0 con 0x7f46d4101380 2024-03-12T06:00:31.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46d419bfe0 0x7f46d419c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46d419bfe0 0x7f46d419c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58004/0 (socket says 172.21.15.76:58004) 2024-03-12T06:00:31.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d37fe640 1 -- 172.21.15.76:0/3403787519 learned_addr learned my addr 172.21.15.76:0/3403787519 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46cbfff640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46d4106450 0x7f46d4197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46d4101380 0x7f46d4197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d37fe640 1 -- 172.21.15.76:0/3403787519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46d4101380 msgr2=0x7f46d4197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d37fe640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46d4101380 0x7f46d4197720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d37fe640 1 -- 172.21.15.76:0/3403787519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46d4106450 msgr2=0x7f46d4197c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d37fe640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46d4106450 0x7f46d4197c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d37fe640 1 -- 172.21.15.76:0/3403787519 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc009900 con 0x7f46d419bfe0 2024-03-12T06:00:31.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46d4101380 0x7f46d4197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:00:31.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.566+0000 7f46d37fe640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46d419bfe0 0x7f46d419c490 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f46cc004af0 tx=0x7f46cc034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.567+0000 7f46d0ff9640 1 -- 172.21.15.76:0/3403787519 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc0328c0 con 0x7f46d419bfe0 2024-03-12T06:00:31.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.567+0000 7f46d0ff9640 1 -- 172.21.15.76:0/3403787519 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46cc003e50 con 0x7f46d419bfe0 2024-03-12T06:00:31.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.567+0000 7f46d0ff9640 1 -- 172.21.15.76:0/3403787519 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc004860 con 0x7f46d419bfe0 2024-03-12T06:00:31.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.567+0000 7f46d976c640 1 -- 172.21.15.76:0/3403787519 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d41a4290 con 0x7f46d419bfe0 2024-03-12T06:00:31.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.567+0000 7f46d976c640 1 -- 172.21.15.76:0/3403787519 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d41a46b0 con 0x7f46d419bfe0 2024-03-12T06:00:31.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.569+0000 7f46d0ff9640 1 -- 172.21.15.76:0/3403787519 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f46cc00a040 con 0x7f46d419bfe0 2024-03-12T06:00:31.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.569+0000 7f46dc41b740 1 Processor -- start 2024-03-12T06:00:31.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.569+0000 7f46dc41b740 1 -- start start 2024-03-12T06:00:31.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46dc41b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2b250 0x564196b2b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46dc41b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2bb70 0x564196b34210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46dc41b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b36b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46dc41b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196ae6960 con 0x564196b34750 2024-03-12T06:00:31.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46dc41b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196a9a310 con 0x564196b2bb70 2024-03-12T06:00:31.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46dc41b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196a97840 con 0x564196b2b250 2024-03-12T06:00:31.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46d0ff9640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46c00a1c20 0x7f46c00a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46cbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b36b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46d37fe640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46c00a1c20 0x7f46c00a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46cbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b36b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58014/0 (socket says 172.21.15.76:58014) 2024-03-12T06:00:31.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46d2ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2bb70 0x564196b34210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46cbfff640 1 -- 172.21.15.76:0/2500808346 learned_addr learned my addr 172.21.15.76:0/2500808346 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46cbfff640 1 --2- 172.21.15.76:0/2500808346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2b250 0x564196b2b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46d0ff9640 1 -- 172.21.15.76:0/3403787519 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f46cc0f04b0 con 0x7f46d419bfe0 2024-03-12T06:00:31.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46d2ffd640 1 -- 172.21.15.76:0/2500808346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2b250 msgr2=0x564196b2b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/2500808346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2b250 0x564196b2b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46d2ffd640 1 -- 172.21.15.76:0/2500808346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 msgr2=0x564196b36b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/2500808346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b36b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46d2ffd640 1 -- 172.21.15.76:0/2500808346 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564196a55060 con 0x564196b2bb70 2024-03-12T06:00:31.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46cbfff640 1 --2- 172.21.15.76:0/2500808346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b36b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:00:31.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46d37fe640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46c00a1c20 0x7f46c00a40e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0393d0 tx=0x7f46cc0b28f0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.570+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/2500808346 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2bb70 0x564196b34210 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f46c400bb30 tx=0x7f46c400d340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c368fd1f35e56e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.571+0000 7f46c97fa640 1 -- 172.21.15.76:0/2500808346 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4017420 con 0x564196b2bb70 2024-03-12T06:00:31.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.571+0000 7f46c97fa640 1 -- 172.21.15.76:0/2500808346 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46c4017a60 con 0x564196b2bb70 2024-03-12T06:00:31.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.571+0000 7f46c97fa640 1 -- 172.21.15.76:0/2500808346 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c400faf0 con 0x564196b2bb70 2024-03-12T06:00:31.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.571+0000 7f46dc41b740 1 -- 172.21.15.76:0/2500808346 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2bb70 msgr2=0x564196b34210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.571+0000 7f46dc41b740 1 --2- 172.21.15.76:0/2500808346 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2bb70 0x564196b34210 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f46c400bb30 tx=0x7f46c400d340 comp rx=0 tx=0).stop 2024-03-12T06:00:31.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.571+0000 7f46dc41b740 1 -- 172.21.15.76:0/2500808346 shutdown_connections 2024-03-12T06:00:31.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.571+0000 7f46dc41b740 1 --2- 172.21.15.76:0/2500808346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b36b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.571+0000 7f46dc41b740 1 --2- 172.21.15.76:0/2500808346 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2bb70 0x564196b34210 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.571+0000 7f46dc41b740 1 --2- 172.21.15.76:0/2500808346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2b250 0x564196b2b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.571+0000 7f46dc41b740 1 -- 172.21.15.76:0/2500808346 >> 172.21.15.76:0/2500808346 conn(0x564196b25980 msgr2=0x564196b25d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.571+0000 7f46dc41b740 1 -- 172.21.15.76:0/2500808346 shutdown_connections 2024-03-12T06:00:31.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.571+0000 7f46dc41b740 1 -- 172.21.15.76:0/2500808346 wait complete. 2024-03-12T06:00:31.588 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46dc41b740 1 -- start start 2024-03-12T06:00:31.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46dc41b740 1 Processor -- start 2024-03-12T06:00:31.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46dc41b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 0x564196b316f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46dc41b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2bb70 0x564196b31c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46dc41b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b32170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46dc41b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196a93020 con 0x564196b34750 2024-03-12T06:00:31.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46dc41b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196766430 con 0x564196b2b250 2024-03-12T06:00:31.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46dc41b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196722e00 con 0x564196b2bb70 2024-03-12T06:00:31.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46dc41b740 10 client.0 _ll_register_callbacks cb 0x564196a3acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:31.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46cbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b32170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.589 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46cbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b32170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58018/0 (socket says 172.21.15.76:58018) 2024-03-12T06:00:31.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46cbfff640 1 -- 172.21.15.76:0/963970246 learned_addr learned my addr 172.21.15.76:0/963970246 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46cbfff640 1 --2- 172.21.15.76:0/963970246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 0x564196b316f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/963970246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2bb70 0x564196b31c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46cbfff640 1 -- 172.21.15.76:0/963970246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2bb70 msgr2=0x564196b31c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46cbfff640 1 --2- 172.21.15.76:0/963970246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2bb70 0x564196b31c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46cbfff640 1 -- 172.21.15.76:0/963970246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 msgr2=0x564196b316f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.590 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46cbfff640 1 --2- 172.21.15.76:0/963970246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 0x564196b316f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.572+0000 7f46cbfff640 1 -- 172.21.15.76:0/963970246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c400b790 con 0x564196b34750 2024-03-12T06:00:31.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46cbfff640 1 --2- 172.21.15.76:0/963970246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b32170 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0104b0 tx=0x7f46bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69adaa726aa057b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46ba7fc640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:31.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46ba7fc640 1 -- 172.21.15.76:0/963970246 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc004030 con 0x564196b34750 2024-03-12T06:00:31.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46ba7fc640 1 -- 172.21.15.76:0/963970246 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f46bc004190 con 0x564196b34750 2024-03-12T06:00:31.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46dc41b740 1 -- 172.21.15.76:0/963970246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5641967927e0 con 0x564196b34750 2024-03-12T06:00:31.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46ba7fc640 1 -- 172.21.15.76:0/963970246 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46bc004350 con 0x564196b34750 2024-03-12T06:00:31.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46dc41b740 10 client.16587 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:31.591 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46dc41b740 1 -- 172.21.15.76:0/963970246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564196a916a0 con 0x564196b34750 2024-03-12T06:00:31.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46ba7fc640 1 -- 172.21.15.76:0/963970246 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5641967927e0 con 0x564196b34750 2024-03-12T06:00:31.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46ba7fc640 1 -- 172.21.15.76:0/963970246 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f46bc011440 con 0x564196b34750 2024-03-12T06:00:31.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46dc41b740 10 client.16587 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:31.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46dc41b740 10 client.16587 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:00:31.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46dc41b740 20 client.16587 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:31.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46dc41b740 1 --2- 172.21.15.76:0/963970246 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564196b44a70 0x564196b46e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46dc41b740 4 client.16587 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:00:31.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.573+0000 7f46dc41b740 1 -- 172.21.15.76:0/963970246 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5641967cf720 con 0x564196b44a70 2024-03-12T06:00:31.592 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.574+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/963970246 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564196b44a70 0x564196b46e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.574+0000 7f46ba7fc640 10 client.16587 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:00:31.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.574+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/963970246 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564196b44a70 0x564196b46e90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d07fb3161d916cb7 server_cookie=5112c09b07fc5790 in_seq=0 out_seq=0 2024-03-12T06:00:31.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46ba7fc640 1 -- 172.21.15.76:0/963970246 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5641967cf720 con 0x564196b44a70 2024-03-12T06:00:31.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46ba7fc640 10 client.16587 handle_command_reply: tid=0 2024-03-12T06:00:31.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46dc41b740 1 client.16587 shutdown 2024-03-12T06:00:31.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46dc41b740 1 -- 172.21.15.76:0/963970246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 msgr2=0x564196b32170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46dc41b740 1 --2- 172.21.15.76:0/963970246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b32170 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0104b0 tx=0x7f46bc004730 comp rx=0 tx=0).stop 2024-03-12T06:00:31.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46dc41b740 1 -- 172.21.15.76:0/963970246 shutdown_connections 2024-03-12T06:00:31.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46dc41b740 1 --2- 172.21.15.76:0/963970246 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564196b44a70 0x564196b46e90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46dc41b740 1 --2- 172.21.15.76:0/963970246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b32170 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46ba7fc640 0 client.16587 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:00:31.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46dc41b740 1 --2- 172.21.15.76:0/963970246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2bb70 0x564196b31c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46dc41b740 1 --2- 172.21.15.76:0/963970246 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 0x564196b316f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46dc41b740 1 -- 172.21.15.76:0/963970246 >> 172.21.15.76:0/963970246 conn(0x564196b25980 msgr2=0x564196b276d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46dc41b740 1 -- 172.21.15.76:0/963970246 shutdown_connections 2024-03-12T06:00:31.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46dc41b740 1 -- 172.21.15.76:0/963970246 wait complete. 2024-03-12T06:00:31.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.575+0000 7f46dc41b740 20 client.16587 trim_cache size 0 max 16384 2024-03-12T06:00:31.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.586+0000 7f46dc41b740 1 Processor -- start 2024-03-12T06:00:31.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.586+0000 7f46dc41b740 1 -- start start 2024-03-12T06:00:31.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46dc41b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 0x564196b33e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46dc41b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b2bb70 0x564196b2e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46dc41b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b34750 0x564196b2eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46dc41b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196a18130 con 0x564196b2bb70 2024-03-12T06:00:31.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46dc41b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196b36f00 con 0x564196b2b250 2024-03-12T06:00:31.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46dc41b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196b26760 con 0x564196b34750 2024-03-12T06:00:31.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46d2ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b2bb70 0x564196b2e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46d2ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b2bb70 0x564196b2e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58030/0 (socket says 172.21.15.76:58030) 2024-03-12T06:00:31.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46d2ffd640 1 -- 172.21.15.76:0/1153690140 learned_addr learned my addr 172.21.15.76:0/1153690140 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46cbfff640 1 --2- 172.21.15.76:0/1153690140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 0x564196b33e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46cbfff640 1 --2- 172.21.15.76:0/1153690140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b34750 0x564196b2eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46d2ffd640 1 -- 172.21.15.76:0/1153690140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b34750 msgr2=0x564196b2eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/1153690140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b34750 0x564196b2eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46d2ffd640 1 -- 172.21.15.76:0/1153690140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 msgr2=0x564196b33e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/1153690140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 0x564196b33e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46d2ffd640 1 -- 172.21.15.76:0/1153690140 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564196a2dc20 con 0x564196b2bb70 2024-03-12T06:00:31.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46cbfff640 1 --2- 172.21.15.76:0/1153690140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 0x564196b33e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:00:31.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/1153690140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b2bb70 0x564196b2e970 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f46c400de20 tx=0x7f46c400b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2b188d7005b5e3a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.587+0000 7f46c8ff9640 1 -- 172.21.15.76:0/1153690140 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4017dd0 con 0x564196b2bb70 2024-03-12T06:00:31.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.588+0000 7f46c8ff9640 1 -- 172.21.15.76:0/1153690140 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46c4005040 con 0x564196b2bb70 2024-03-12T06:00:31.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.588+0000 7f46c8ff9640 1 -- 172.21.15.76:0/1153690140 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c40049e0 con 0x564196b2bb70 2024-03-12T06:00:31.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.588+0000 7f46dc41b740 1 -- 172.21.15.76:0/1153690140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b2bb70 msgr2=0x564196b2e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.588+0000 7f46dc41b740 1 --2- 172.21.15.76:0/1153690140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b2bb70 0x564196b2e970 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f46c400de20 tx=0x7f46c400b540 comp rx=0 tx=0).stop 2024-03-12T06:00:31.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.588+0000 7f46dc41b740 1 -- 172.21.15.76:0/1153690140 shutdown_connections 2024-03-12T06:00:31.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.588+0000 7f46dc41b740 1 --2- 172.21.15.76:0/1153690140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b34750 0x564196b2eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.588+0000 7f46dc41b740 1 --2- 172.21.15.76:0/1153690140 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b2bb70 0x564196b2e970 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.588+0000 7f46dc41b740 1 --2- 172.21.15.76:0/1153690140 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 0x564196b33e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.588+0000 7f46dc41b740 1 -- 172.21.15.76:0/1153690140 >> 172.21.15.76:0/1153690140 conn(0x564196b25980 msgr2=0x564196b2c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.588+0000 7f46dc41b740 1 -- 172.21.15.76:0/1153690140 shutdown_connections 2024-03-12T06:00:31.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.588+0000 7f46dc41b740 1 -- 172.21.15.76:0/1153690140 wait complete. 2024-03-12T06:00:31.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46dc41b740 1 -- start start 2024-03-12T06:00:31.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46dc41b740 1 Processor -- start 2024-03-12T06:00:31.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46dc41b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 0x564196b3a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46dc41b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2bb70 0x564196b3ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46dc41b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b9af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46dc41b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196b9b4a0 con 0x564196b34750 2024-03-12T06:00:31.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46dc41b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196b9b5e0 con 0x564196b2b250 2024-03-12T06:00:31.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46dc41b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196b9b7a0 con 0x564196b2bb70 2024-03-12T06:00:31.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46dc41b740 10 client.0 _ll_register_callbacks cb 0x7f46c400b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:31.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46cbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b9af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46cbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b9af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58044/0 (socket says 172.21.15.76:58044) 2024-03-12T06:00:31.605 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46cbfff640 1 -- 172.21.15.76:0/3706449812 learned_addr learned my addr 172.21.15.76:0/3706449812 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46cbfff640 1 --2- 172.21.15.76:0/3706449812 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 0x564196b3a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/3706449812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2bb70 0x564196b3ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46cbfff640 1 -- 172.21.15.76:0/3706449812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2bb70 msgr2=0x564196b3ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46cbfff640 1 --2- 172.21.15.76:0/3706449812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2bb70 0x564196b3ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46cbfff640 1 -- 172.21.15.76:0/3706449812 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 msgr2=0x564196b3a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46cbfff640 1 --2- 172.21.15.76:0/3706449812 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 0x564196b3a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46cbfff640 1 -- 172.21.15.76:0/3706449812 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564196a55060 con 0x564196b34750 2024-03-12T06:00:31.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46cbfff640 1 --2- 172.21.15.76:0/3706449812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b9af60 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f46bc011010 tx=0x7f46bc014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a046e1d69a2a205 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.589+0000 7f46ba7fc640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:31.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46ba7fc640 1 -- 172.21.15.76:0/3706449812 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc009530 con 0x564196b34750 2024-03-12T06:00:31.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46ba7fc640 1 -- 172.21.15.76:0/3706449812 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f46bc000e40 con 0x564196b34750 2024-03-12T06:00:31.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46dc41b740 1 -- 172.21.15.76:0/3706449812 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564196b26300 con 0x564196b34750 2024-03-12T06:00:31.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46ba7fc640 1 -- 172.21.15.76:0/3706449812 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46bc011da0 con 0x564196b34750 2024-03-12T06:00:31.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46dc41b740 10 client.16593 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:31.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46dc41b740 1 -- 172.21.15.76:0/3706449812 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564196a538c0 con 0x564196b34750 2024-03-12T06:00:31.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46ba7fc640 1 -- 172.21.15.76:0/3706449812 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564196b26300 con 0x564196b34750 2024-03-12T06:00:31.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46ba7fc640 1 -- 172.21.15.76:0/3706449812 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f46bc031bf0 con 0x564196b34750 2024-03-12T06:00:31.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46dc41b740 10 client.16593 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:31.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46dc41b740 10 client.16593 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:00:31.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46dc41b740 20 client.16593 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:31.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46dc41b740 1 --2- 172.21.15.76:0/3706449812 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564196b44a70 0x564196b9edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46dc41b740 4 client.16593 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:00:31.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46dc41b740 1 -- 172.21.15.76:0/3706449812 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f46c4017a60 con 0x564196b44a70 2024-03-12T06:00:31.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.590+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/3706449812 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564196b44a70 0x564196b9edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46ba7fc640 10 client.16593 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:00:31.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/3706449812 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564196b44a70 0x564196b9edf0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3d72111921b9c033 server_cookie=dfcc27ce2965a33e in_seq=0 out_seq=0 2024-03-12T06:00:31.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46ba7fc640 1 -- 172.21.15.76:0/3706449812 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f46c4017a60 con 0x564196b44a70 2024-03-12T06:00:31.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46ba7fc640 10 client.16593 handle_command_reply: tid=0 2024-03-12T06:00:31.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46dc41b740 1 client.16593 shutdown 2024-03-12T06:00:31.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46dc41b740 1 -- 172.21.15.76:0/3706449812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 msgr2=0x564196b9af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46dc41b740 1 --2- 172.21.15.76:0/3706449812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b9af60 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f46bc011010 tx=0x7f46bc014040 comp rx=0 tx=0).stop 2024-03-12T06:00:31.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46dc41b740 1 -- 172.21.15.76:0/3706449812 shutdown_connections 2024-03-12T06:00:31.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46dc41b740 1 --2- 172.21.15.76:0/3706449812 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x564196b44a70 0x564196b9edf0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46dc41b740 1 --2- 172.21.15.76:0/3706449812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564196b34750 0x564196b9af60 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46ba7fc640 0 client.16593 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:00:31.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46dc41b740 1 --2- 172.21.15.76:0/3706449812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564196b2bb70 0x564196b3ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46dc41b740 1 --2- 172.21.15.76:0/3706449812 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x564196b2b250 0x564196b3a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46dc41b740 1 -- 172.21.15.76:0/3706449812 >> 172.21.15.76:0/3706449812 conn(0x564196b25980 msgr2=0x564196b31690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46dc41b740 1 -- 172.21.15.76:0/3706449812 shutdown_connections 2024-03-12T06:00:31.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46dc41b740 1 -- 172.21.15.76:0/3706449812 wait complete. 2024-03-12T06:00:31.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.591+0000 7f46dc41b740 20 client.16593 trim_cache size 0 max 16384 2024-03-12T06:00:31.610 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:00:31.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.592+0000 7f46d976c640 1 -- 172.21.15.76:0/3403787519 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46c00a1c20 msgr2=0x7f46c00a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.592+0000 7f46d976c640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46c00a1c20 0x7f46c00a40e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0393d0 tx=0x7f46cc0b28f0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.592+0000 7f46d976c640 1 -- 172.21.15.76:0/3403787519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46d419bfe0 msgr2=0x7f46d419c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.592+0000 7f46d976c640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46d419bfe0 0x7f46d419c490 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f46cc004af0 tx=0x7f46cc034cc0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.592+0000 7f46d976c640 1 -- 172.21.15.76:0/3403787519 shutdown_connections 2024-03-12T06:00:31.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.592+0000 7f46d976c640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46c00a1c20 0x7f46c00a40e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.592+0000 7f46d976c640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46d419bfe0 0x7f46d419c490 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.592+0000 7f46d976c640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46d4106450 0x7f46d4197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.592+0000 7f46d976c640 1 --2- 172.21.15.76:0/3403787519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46d4101380 0x7f46d4197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.592+0000 7f46d976c640 1 -- 172.21.15.76:0/3403787519 >> 172.21.15.76:0/3403787519 conn(0x7f46d40fcc20 msgr2=0x7f46d40fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.592+0000 7f46d976c640 1 -- 172.21.15.76:0/3403787519 shutdown_connections 2024-03-12T06:00:31.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.592+0000 7f46d976c640 1 -- 172.21.15.76:0/3403787519 wait complete. 2024-03-12T06:00:31.614 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:00:31.614 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T06:00:31.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.713+0000 7fe2d946f640 1 -- 172.21.15.76:0/589154164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2d4105350 msgr2=0x7fe2d410cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.713+0000 7fe2d946f640 1 --2- 172.21.15.76:0/589154164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2d4105350 0x7fe2d410cf20 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009cd0 tx=0x7fe2cc02c0f0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.713+0000 7fe2d946f640 1 -- 172.21.15.76:0/589154164 shutdown_connections 2024-03-12T06:00:31.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.713+0000 7fe2d946f640 1 --2- 172.21.15.76:0/589154164 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2d4105350 0x7fe2d410cf20 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.713+0000 7fe2d946f640 1 --2- 172.21.15.76:0/589154164 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe2d4100c30 0x7fe2d4104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.713+0000 7fe2d946f640 1 --2- 172.21.15.76:0/589154164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d4100280 0x7fe2d4100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.713+0000 7fe2d946f640 1 -- 172.21.15.76:0/589154164 >> 172.21.15.76:0/589154164 conn(0x7fe2d40fbae0 msgr2=0x7fe2d40fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.713+0000 7fe2d946f640 1 -- 172.21.15.76:0/589154164 shutdown_connections 2024-03-12T06:00:31.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.713+0000 7fe2d946f640 1 -- 172.21.15.76:0/589154164 wait complete. 2024-03-12T06:00:31.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.713+0000 7fe2d946f640 1 Processor -- start 2024-03-12T06:00:31.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d946f640 1 -- start start 2024-03-12T06:00:31.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d946f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2d4100280 0x7fe2d41985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d946f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d4105350 0x7fe2d4198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d946f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe2d4199100 0x7fe2d41a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d946f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d41a3d50 con 0x7fe2d4100280 2024-03-12T06:00:31.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d946f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d41a3ec0 con 0x7fe2d4199100 2024-03-12T06:00:31.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d946f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d41a4080 con 0x7fe2d4105350 2024-03-12T06:00:31.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d2ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2d4100280 0x7fe2d41985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d2ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2d4100280 0x7fe2d41985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58058/0 (socket says 172.21.15.76:58058) 2024-03-12T06:00:31.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d2ffd640 1 -- 172.21.15.76:0/2959519186 learned_addr learned my addr 172.21.15.76:0/2959519186 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d37fe640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe2d4199100 0x7fe2d41a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d27fc640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d4105350 0x7fe2d4198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d2ffd640 1 -- 172.21.15.76:0/2959519186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d4105350 msgr2=0x7fe2d4198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d2ffd640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d4105350 0x7fe2d4198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d2ffd640 1 -- 172.21.15.76:0/2959519186 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe2d4199100 msgr2=0x7fe2d41a3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d2ffd640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe2d4199100 0x7fe2d41a3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d2ffd640 1 -- 172.21.15.76:0/2959519186 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc009900 con 0x7fe2d4100280 2024-03-12T06:00:31.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d27fc640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d4105350 0x7fe2d4198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:00:31.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d37fe640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe2d4199100 0x7fe2d41a3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:00:31.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d2ffd640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2d4100280 0x7fe2d41985e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800d770 tx=0x7fe2c800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2bbfff640 1 -- 172.21.15.76:0/2959519186 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8009c30 con 0x7fe2d4100280 2024-03-12T06:00:31.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2bbfff640 1 -- 172.21.15.76:0/2959519186 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2c800de80 con 0x7fe2d4100280 2024-03-12T06:00:31.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d946f640 1 -- 172.21.15.76:0/2959519186 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d41a4360 con 0x7fe2d4100280 2024-03-12T06:00:31.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2bbfff640 1 -- 172.21.15.76:0/2959519186 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8004e80 con 0x7fe2d4100280 2024-03-12T06:00:31.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.714+0000 7fe2d946f640 1 -- 172.21.15.76:0/2959519186 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d41a4820 con 0x7fe2d4100280 2024-03-12T06:00:31.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.717+0000 7fe2bbfff640 1 -- 172.21.15.76:0/2959519186 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe2c8005020 con 0x7fe2d4100280 2024-03-12T06:00:31.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.717+0000 7fe2dc040740 1 Processor -- start 2024-03-12T06:00:31.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.717+0000 7fe2dc040740 1 -- start start 2024-03-12T06:00:31.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.717+0000 7fe2dc040740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83250 0x55ce4bf83630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.717+0000 7fe2dc040740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83b70 0x55ce4bf8c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.717+0000 7fe2dc040740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bf8eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.717+0000 7fe2dc040740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4bf3e960 con 0x55ce4bf8c750 2024-03-12T06:00:31.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.717+0000 7fe2dc040740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4bef2310 con 0x55ce4bf83b70 2024-03-12T06:00:31.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.717+0000 7fe2dc040740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4beef840 con 0x55ce4bf83250 2024-03-12T06:00:31.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.717+0000 7fe2bbfff640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe2bc0a1ba0 0x7fe2bc0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2d27fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bf8eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2d27fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bf8eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58060/0 (socket says 172.21.15.76:58060) 2024-03-12T06:00:31.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2d27fc640 1 -- 172.21.15.76:0/1148343051 learned_addr learned my addr 172.21.15.76:0/1148343051 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2d37fe640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe2bc0a1ba0 0x7fe2bc0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2d2ffd640 1 --2- 172.21.15.76:0/1148343051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83b70 0x55ce4bf8c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2d37fe640 1 --2- 172.21.15.76:0/1148343051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83250 0x55ce4bf83630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2bbfff640 1 -- 172.21.15.76:0/2959519186 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe2c80cd330 con 0x7fe2d4100280 2024-03-12T06:00:31.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2d27fc640 1 -- 172.21.15.76:0/1148343051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83250 msgr2=0x55ce4bf83630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2d27fc640 1 --2- 172.21.15.76:0/1148343051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83250 0x55ce4bf83630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2d27fc640 1 -- 172.21.15.76:0/1148343051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83b70 msgr2=0x55ce4bf8c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2d27fc640 1 --2- 172.21.15.76:0/1148343051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83b70 0x55ce4bf8c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2d27fc640 1 -- 172.21.15.76:0/1148343051 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce4bead060 con 0x55ce4bf8c750 2024-03-12T06:00:31.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2d37fe640 1 --2- 172.21.15.76:0/1148343051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83250 0x55ce4bf83630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:00:31.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2d27fc640 1 --2- 172.21.15.76:0/1148343051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bf8eb40 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400ba70 tx=0x7fe2c400bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3e92f4794d0b9b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2d37fe640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe2bc0a1ba0 0x7fe2bc0a4060 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc006010 tx=0x7fe2cc004a40 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2b97fa640 1 -- 172.21.15.76:0/1148343051 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4017420 con 0x55ce4bf8c750 2024-03-12T06:00:31.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2b97fa640 1 -- 172.21.15.76:0/1148343051 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2c4017a60 con 0x55ce4bf8c750 2024-03-12T06:00:31.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.718+0000 7fe2b97fa640 1 -- 172.21.15.76:0/1148343051 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c400f9d0 con 0x55ce4bf8c750 2024-03-12T06:00:31.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.719+0000 7fe2dc040740 1 -- 172.21.15.76:0/1148343051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 msgr2=0x55ce4bf8eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.719+0000 7fe2dc040740 1 --2- 172.21.15.76:0/1148343051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bf8eb40 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400ba70 tx=0x7fe2c400bf30 comp rx=0 tx=0).stop 2024-03-12T06:00:31.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.719+0000 7fe2dc040740 1 -- 172.21.15.76:0/1148343051 shutdown_connections 2024-03-12T06:00:31.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.719+0000 7fe2dc040740 1 --2- 172.21.15.76:0/1148343051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bf8eb40 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.719+0000 7fe2dc040740 1 --2- 172.21.15.76:0/1148343051 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83b70 0x55ce4bf8c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.719+0000 7fe2dc040740 1 --2- 172.21.15.76:0/1148343051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83250 0x55ce4bf83630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.719+0000 7fe2dc040740 1 -- 172.21.15.76:0/1148343051 >> 172.21.15.76:0/1148343051 conn(0x55ce4bf7d980 msgr2=0x55ce4bf7dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.719+0000 7fe2dc040740 1 -- 172.21.15.76:0/1148343051 shutdown_connections 2024-03-12T06:00:31.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.719+0000 7fe2dc040740 1 -- 172.21.15.76:0/1148343051 wait complete. 2024-03-12T06:00:31.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.719+0000 7fe2dc040740 1 -- start start 2024-03-12T06:00:31.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.719+0000 7fe2dc040740 1 Processor -- start 2024-03-12T06:00:31.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2dc040740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83250 0x55ce4bf89770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2dc040740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83b70 0x55ce4bf89cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2dc040740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bf8a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2dc040740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4beeb020 con 0x55ce4bf8c750 2024-03-12T06:00:31.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2dc040740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4bbbe430 con 0x55ce4bf83250 2024-03-12T06:00:31.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2dc040740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4bb7ae00 con 0x55ce4bf83b70 2024-03-12T06:00:31.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2dc040740 10 client.0 _ll_register_callbacks cb 0x55ce4be92cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:31.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2d37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bf8a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2d37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bf8a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58068/0 (socket says 172.21.15.76:58068) 2024-03-12T06:00:31.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2d37fe640 1 -- 172.21.15.76:0/1090171243 learned_addr learned my addr 172.21.15.76:0/1090171243 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2d27fc640 1 --2- 172.21.15.76:0/1090171243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83b70 0x55ce4bf89cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2d2ffd640 1 --2- 172.21.15.76:0/1090171243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83250 0x55ce4bf89770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2d37fe640 1 -- 172.21.15.76:0/1090171243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83b70 msgr2=0x55ce4bf89cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2d37fe640 1 --2- 172.21.15.76:0/1090171243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83b70 0x55ce4bf89cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2d37fe640 1 -- 172.21.15.76:0/1090171243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83250 msgr2=0x55ce4bf89770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2d37fe640 1 --2- 172.21.15.76:0/1090171243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83250 0x55ce4bf89770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2d37fe640 1 -- 172.21.15.76:0/1090171243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c400b6d0 con 0x55ce4bf8c750 2024-03-12T06:00:31.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2d2ffd640 1 --2- 172.21.15.76:0/1090171243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83250 0x55ce4bf89770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:00:31.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2d37fe640 1 --2- 172.21.15.76:0/1090171243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bf8a1f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009cd0 tx=0x7fe2cc02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835e9070dd41a450 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:31.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2b6ff5640 1 -- 172.21.15.76:0/1090171243 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc004240 con 0x55ce4bf8c750 2024-03-12T06:00:31.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2b6ff5640 1 -- 172.21.15.76:0/1090171243 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe2cc048040 con 0x55ce4bf8c750 2024-03-12T06:00:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2dc040740 1 -- 172.21.15.76:0/1090171243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ce4bbea7e0 con 0x55ce4bf8c750 2024-03-12T06:00:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2b6ff5640 1 -- 172.21.15.76:0/1090171243 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2cc003e40 con 0x55ce4bf8c750 2024-03-12T06:00:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2dc040740 10 client.16605 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.720+0000 7fe2dc040740 1 -- 172.21.15.76:0/1090171243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ce4bee96a0 con 0x55ce4bf8c750 2024-03-12T06:00:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.721+0000 7fe2b6ff5640 1 -- 172.21.15.76:0/1090171243 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ce4bbea7e0 con 0x55ce4bf8c750 2024-03-12T06:00:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.721+0000 7fe2b6ff5640 1 -- 172.21.15.76:0/1090171243 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fe2cc03e040 con 0x55ce4bf8c750 2024-03-12T06:00:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.721+0000 7fe2dc040740 10 client.16605 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.721+0000 7fe2dc040740 10 client.16605 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:00:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.721+0000 7fe2dc040740 20 client.16605 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:31.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.721+0000 7fe2dc040740 1 --2- 172.21.15.76:0/1090171243 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55ce4bf9cc20 0x55ce4bf9f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.721+0000 7fe2dc040740 4 client.16605 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:00:31.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.721+0000 7fe2dc040740 1 -- 172.21.15.76:0/1090171243 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ce4bc27720 con 0x55ce4bf9cc20 2024-03-12T06:00:31.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.721+0000 7fe2d2ffd640 1 --2- 172.21.15.76:0/1090171243 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55ce4bf9cc20 0x55ce4bf9f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2d2ffd640 1 --2- 172.21.15.76:0/1090171243 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55ce4bf9cc20 0x55ce4bf9f040 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7658fd753f6e261b server_cookie=2ca0eccd3fb20b55 in_seq=0 out_seq=0 2024-03-12T06:00:31.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2b6ff5640 10 client.16605 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:00:31.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2b6ff5640 1 -- 172.21.15.76:0/1090171243 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ce4bc27720 con 0x55ce4bf9cc20 2024-03-12T06:00:31.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2b6ff5640 10 client.16605 handle_command_reply: tid=0 2024-03-12T06:00:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2dc040740 1 client.16605 shutdown 2024-03-12T06:00:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2dc040740 1 -- 172.21.15.76:0/1090171243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 msgr2=0x55ce4bf8a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2dc040740 1 --2- 172.21.15.76:0/1090171243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bf8a1f0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc009cd0 tx=0x7fe2cc02c0f0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2dc040740 1 -- 172.21.15.76:0/1090171243 shutdown_connections 2024-03-12T06:00:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2dc040740 1 --2- 172.21.15.76:0/1090171243 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55ce4bf9cc20 0x55ce4bf9f040 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2dc040740 1 --2- 172.21.15.76:0/1090171243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bf8a1f0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2dc040740 1 --2- 172.21.15.76:0/1090171243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83b70 0x55ce4bf89cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2b6ff5640 0 client.16605 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:00:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2dc040740 1 --2- 172.21.15.76:0/1090171243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83250 0x55ce4bf89770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2dc040740 1 -- 172.21.15.76:0/1090171243 >> 172.21.15.76:0/1090171243 conn(0x55ce4bf7d980 msgr2=0x55ce4bf7f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.722+0000 7fe2dc040740 1 -- 172.21.15.76:0/1090171243 shutdown_connections 2024-03-12T06:00:31.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.723+0000 7fe2dc040740 1 -- 172.21.15.76:0/1090171243 wait complete. 2024-03-12T06:00:31.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.723+0000 7fe2dc040740 20 client.16605 trim_cache size 0 max 16384 2024-03-12T06:00:31.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.734+0000 7fe2dc040740 1 Processor -- start 2024-03-12T06:00:31.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.734+0000 7fe2dc040740 1 -- start start 2024-03-12T06:00:31.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.734+0000 7fe2dc040740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf83250 0x55ce4bf8a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.734+0000 7fe2dc040740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83b70 0x55ce4bf7ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.734+0000 7fe2dc040740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf8c750 0x55ce4bf93600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.734+0000 7fe2dc040740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4be70130 con 0x55ce4bf83250 2024-03-12T06:00:31.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.734+0000 7fe2dc040740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4bf82c90 con 0x55ce4bf8c750 2024-03-12T06:00:31.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.734+0000 7fe2dc040740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4bf82600 con 0x55ce4bf83b70 2024-03-12T06:00:31.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.734+0000 7fe2d2ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf83250 0x55ce4bf8a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.734+0000 7fe2d2ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf83250 0x55ce4bf8a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58076/0 (socket says 172.21.15.76:58076) 2024-03-12T06:00:31.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.734+0000 7fe2d2ffd640 1 -- 172.21.15.76:0/1629342317 learned_addr learned my addr 172.21.15.76:0/1629342317 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.735+0000 7fe2d27fc640 1 --2- 172.21.15.76:0/1629342317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83b70 0x55ce4bf7ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.735+0000 7fe2d37fe640 1 --2- 172.21.15.76:0/1629342317 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf8c750 0x55ce4bf93600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.735+0000 7fe2d2ffd640 1 -- 172.21.15.76:0/1629342317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83b70 msgr2=0x55ce4bf7ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.735+0000 7fe2d2ffd640 1 --2- 172.21.15.76:0/1629342317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83b70 0x55ce4bf7ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.735+0000 7fe2d2ffd640 1 -- 172.21.15.76:0/1629342317 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf8c750 msgr2=0x55ce4bf93600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.735+0000 7fe2d2ffd640 1 --2- 172.21.15.76:0/1629342317 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf8c750 0x55ce4bf93600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.735+0000 7fe2d2ffd640 1 -- 172.21.15.76:0/1629342317 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce4be85c20 con 0x55ce4bf83250 2024-03-12T06:00:31.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.735+0000 7fe2d27fc640 1 --2- 172.21.15.76:0/1629342317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83b70 0x55ce4bf7ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:00:31.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.735+0000 7fe2d2ffd640 1 --2- 172.21.15.76:0/1629342317 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf83250 0x55ce4bf8a010 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8099ec0 tx=0x7fe2c801ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=681803a360cf6ac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.735+0000 7fe2b8ff9640 1 -- 172.21.15.76:0/1629342317 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c80a26d0 con 0x55ce4bf83250 2024-03-12T06:00:31.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.735+0000 7fe2b8ff9640 1 -- 172.21.15.76:0/1629342317 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2c8009d90 con 0x55ce4bf83250 2024-03-12T06:00:31.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.735+0000 7fe2b8ff9640 1 -- 172.21.15.76:0/1629342317 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8099380 con 0x55ce4bf83250 2024-03-12T06:00:31.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.735+0000 7fe2dc040740 1 -- 172.21.15.76:0/1629342317 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf83250 msgr2=0x55ce4bf8a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.735+0000 7fe2dc040740 1 --2- 172.21.15.76:0/1629342317 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf83250 0x55ce4bf8a010 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8099ec0 tx=0x7fe2c801ef20 comp rx=0 tx=0).stop 2024-03-12T06:00:31.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 -- 172.21.15.76:0/1629342317 shutdown_connections 2024-03-12T06:00:31.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 --2- 172.21.15.76:0/1629342317 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf8c750 0x55ce4bf93600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 --2- 172.21.15.76:0/1629342317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83b70 0x55ce4bf7ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 --2- 172.21.15.76:0/1629342317 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf83250 0x55ce4bf8a010 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 -- 172.21.15.76:0/1629342317 >> 172.21.15.76:0/1629342317 conn(0x55ce4bf7d980 msgr2=0x55ce4bf7c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 -- 172.21.15.76:0/1629342317 shutdown_connections 2024-03-12T06:00:31.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 -- 172.21.15.76:0/1629342317 wait complete. 2024-03-12T06:00:31.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 -- start start 2024-03-12T06:00:31.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 Processor -- start 2024-03-12T06:00:31.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83250 0x55ce4bf92830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83b70 0x55ce4bf92d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bff3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4bff3690 con 0x55ce4bf8c750 2024-03-12T06:00:31.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4bff37d0 con 0x55ce4bf83b70 2024-03-12T06:00:31.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce4bff3990 con 0x55ce4bf83250 2024-03-12T06:00:31.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2dc040740 10 client.0 _ll_register_callbacks cb 0x7fe2c400b2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:31.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2d37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bff3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2d37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bff3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58080/0 (socket says 172.21.15.76:58080) 2024-03-12T06:00:31.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2d37fe640 1 -- 172.21.15.76:0/3067747329 learned_addr learned my addr 172.21.15.76:0/3067747329 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2d2ffd640 1 --2- 172.21.15.76:0/3067747329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83250 0x55ce4bf92830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2d27fc640 1 --2- 172.21.15.76:0/3067747329 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83b70 0x55ce4bf92d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2d37fe640 1 -- 172.21.15.76:0/3067747329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83250 msgr2=0x55ce4bf92830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2d37fe640 1 --2- 172.21.15.76:0/3067747329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83250 0x55ce4bf92830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2d37fe640 1 -- 172.21.15.76:0/3067747329 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83b70 msgr2=0x55ce4bf92d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2d37fe640 1 --2- 172.21.15.76:0/3067747329 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83b70 0x55ce4bf92d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2d37fe640 1 -- 172.21.15.76:0/3067747329 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d41a4820 con 0x55ce4bf8c750 2024-03-12T06:00:31.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.736+0000 7fe2d27fc640 1 --2- 172.21.15.76:0/3067747329 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83b70 0x55ce4bf92d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:00:31.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2d37fe640 1 --2- 172.21.15.76:0/3067747329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bff3150 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc032650 tx=0x7fe2cc005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185f5b31d73b4d1b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:31.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2b6ff5640 1 -- 172.21.15.76:0/3067747329 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc04e660 con 0x55ce4bf8c750 2024-03-12T06:00:31.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2b6ff5640 1 -- 172.21.15.76:0/3067747329 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe2cc04e7c0 con 0x55ce4bf8c750 2024-03-12T06:00:31.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2dc040740 1 -- 172.21.15.76:0/3067747329 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe2c4017a60 con 0x55ce4bf8c750 2024-03-12T06:00:31.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2b6ff5640 1 -- 172.21.15.76:0/3067747329 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2cc043430 con 0x55ce4bf8c750 2024-03-12T06:00:31.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2dc040740 10 client.16611 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:31.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2dc040740 1 -- 172.21.15.76:0/3067747329 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ce4beab8c0 con 0x55ce4bf8c750 2024-03-12T06:00:31.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2b6ff5640 1 -- 172.21.15.76:0/3067747329 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2c4017a60 con 0x55ce4bf8c750 2024-03-12T06:00:31.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2b6ff5640 1 -- 172.21.15.76:0/3067747329 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fe2cc040580 con 0x55ce4bf8c750 2024-03-12T06:00:31.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2dc040740 10 client.16611 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:31.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2dc040740 10 client.16611 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:00:31.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2dc040740 20 client.16611 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:31.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2dc040740 1 --2- 172.21.15.76:0/3067747329 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55ce4bff4d30 0x55ce4bff7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2dc040740 4 client.16611 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:00:31.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.737+0000 7fe2dc040740 1 -- 172.21.15.76:0/3067747329 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ce4bc76890 con 0x55ce4bff4d30 2024-03-12T06:00:31.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2d2ffd640 1 --2- 172.21.15.76:0/3067747329 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55ce4bff4d30 0x55ce4bff7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2b6ff5640 10 client.16611 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:00:31.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2d2ffd640 1 --2- 172.21.15.76:0/3067747329 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55ce4bff4d30 0x55ce4bff7150 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1a8d098758a5748e server_cookie=5c7dde8c25459f96 in_seq=0 out_seq=0 2024-03-12T06:00:31.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2b6ff5640 1 -- 172.21.15.76:0/3067747329 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ce4bc76890 con 0x55ce4bff4d30 2024-03-12T06:00:31.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2b6ff5640 10 client.16611 handle_command_reply: tid=0 2024-03-12T06:00:31.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2dc040740 1 client.16611 shutdown 2024-03-12T06:00:31.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2dc040740 1 -- 172.21.15.76:0/3067747329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 msgr2=0x55ce4bff3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2dc040740 1 --2- 172.21.15.76:0/3067747329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bff3150 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc032650 tx=0x7fe2cc005bd0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2dc040740 1 -- 172.21.15.76:0/3067747329 shutdown_connections 2024-03-12T06:00:31.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2dc040740 1 --2- 172.21.15.76:0/3067747329 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55ce4bff4d30 0x55ce4bff7150 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2dc040740 1 --2- 172.21.15.76:0/3067747329 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ce4bf8c750 0x55ce4bff3150 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2b6ff5640 0 client.16611 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:00:31.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2dc040740 1 --2- 172.21.15.76:0/3067747329 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ce4bf83b70 0x55ce4bf92d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2dc040740 1 --2- 172.21.15.76:0/3067747329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ce4bf83250 0x55ce4bf92830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2dc040740 1 -- 172.21.15.76:0/3067747329 >> 172.21.15.76:0/3067747329 conn(0x55ce4bf7d980 msgr2=0x55ce4bf7cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2dc040740 1 -- 172.21.15.76:0/3067747329 shutdown_connections 2024-03-12T06:00:31.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2dc040740 1 -- 172.21.15.76:0/3067747329 wait complete. 2024-03-12T06:00:31.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.738+0000 7fe2dc040740 20 client.16611 trim_cache size 0 max 16384 2024-03-12T06:00:31.758 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:00:31.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.739+0000 7fe2d946f640 1 -- 172.21.15.76:0/2959519186 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe2bc0a1ba0 msgr2=0x7fe2bc0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.739+0000 7fe2d946f640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe2bc0a1ba0 0x7fe2bc0a4060 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc006010 tx=0x7fe2cc004a40 comp rx=0 tx=0).stop 2024-03-12T06:00:31.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.739+0000 7fe2d946f640 1 -- 172.21.15.76:0/2959519186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2d4100280 msgr2=0x7fe2d41985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.739+0000 7fe2d946f640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2d4100280 0x7fe2d41985e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800d770 tx=0x7fe2c800dc30 comp rx=0 tx=0).stop 2024-03-12T06:00:31.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.739+0000 7fe2d946f640 1 -- 172.21.15.76:0/2959519186 shutdown_connections 2024-03-12T06:00:31.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.739+0000 7fe2d946f640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe2bc0a1ba0 0x7fe2bc0a4060 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.739+0000 7fe2d946f640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe2d4199100 0x7fe2d41a3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.739+0000 7fe2d946f640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d4105350 0x7fe2d4198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.739+0000 7fe2d946f640 1 --2- 172.21.15.76:0/2959519186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2d4100280 0x7fe2d41985e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.739+0000 7fe2d946f640 1 -- 172.21.15.76:0/2959519186 >> 172.21.15.76:0/2959519186 conn(0x7fe2d40fbae0 msgr2=0x7fe2d40fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.739+0000 7fe2d946f640 1 -- 172.21.15.76:0/2959519186 shutdown_connections 2024-03-12T06:00:31.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.739+0000 7fe2d946f640 1 -- 172.21.15.76:0/2959519186 wait complete. 2024-03-12T06:00:31.761 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:00:31.761 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T06:00:31.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.860+0000 7fc2a5193640 1 -- 172.21.15.76:0/1471072201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2a0102ca0 msgr2=0x7fc2a010b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.860+0000 7fc2a5193640 1 --2- 172.21.15.76:0/1471072201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2a0102ca0 0x7fc2a010b180 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc290009900 tx=0x7fc29002bcd0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.860+0000 7fc2a5193640 1 -- 172.21.15.76:0/1471072201 shutdown_connections 2024-03-12T06:00:31.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.860+0000 7fc2a5193640 1 --2- 172.21.15.76:0/1471072201 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2a010b750 0x7fc2a010dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.860+0000 7fc2a5193640 1 --2- 172.21.15.76:0/1471072201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2a0102ca0 0x7fc2a010b180 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.860+0000 7fc2a5193640 1 --2- 172.21.15.76:0/1471072201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2a0102380 0x7fc2a0102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.860+0000 7fc2a5193640 1 -- 172.21.15.76:0/1471072201 >> 172.21.15.76:0/1471072201 conn(0x7fc2a00fbae0 msgr2=0x7fc2a00fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.860+0000 7fc2a5193640 1 -- 172.21.15.76:0/1471072201 shutdown_connections 2024-03-12T06:00:31.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.860+0000 7fc2a5193640 1 -- 172.21.15.76:0/1471072201 wait complete. 2024-03-12T06:00:31.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.860+0000 7fc2a5193640 1 Processor -- start 2024-03-12T06:00:31.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc2a5193640 1 -- start start 2024-03-12T06:00:31.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc2a5193640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2a0102380 0x7fc2a019d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc2a5193640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2a0102ca0 0x7fc2a019dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc2a5193640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2a010b750 0x7fc2a0197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc2a5193640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a0197dd0 con 0x7fc2a0102ca0 2024-03-12T06:00:31.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc2a5193640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a0197f40 con 0x7fc2a010b750 2024-03-12T06:00:31.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc2a5193640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a0198100 con 0x7fc2a0102380 2024-03-12T06:00:31.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc29e59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2a0102ca0 0x7fc2a019dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc29e59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2a0102ca0 0x7fc2a019dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58098/0 (socket says 172.21.15.76:58098) 2024-03-12T06:00:31.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc29e59c640 1 -- 172.21.15.76:0/3510352435 learned_addr learned my addr 172.21.15.76:0/3510352435 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc29ed9d640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2a0102380 0x7fc2a019d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc29f59e640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2a010b750 0x7fc2a0197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc29e59c640 1 -- 172.21.15.76:0/3510352435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2a0102380 msgr2=0x7fc2a019d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc29e59c640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2a0102380 0x7fc2a019d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc29e59c640 1 -- 172.21.15.76:0/3510352435 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2a010b750 msgr2=0x7fc2a0197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc29e59c640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2a010b750 0x7fc2a0197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc29e59c640 1 -- 172.21.15.76:0/3510352435 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc290009560 con 0x7fc2a0102ca0 2024-03-12T06:00:31.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc29ed9d640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2a0102380 0x7fc2a019d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:00:31.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.861+0000 7fc29e59c640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2a0102ca0 0x7fc2a019dac0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc290005940 tx=0x7fc29002c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.862+0000 7fc287fff640 1 -- 172.21.15.76:0/3510352435 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc290004350 con 0x7fc2a0102ca0 2024-03-12T06:00:31.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.862+0000 7fc287fff640 1 -- 172.21.15.76:0/3510352435 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2900044b0 con 0x7fc2a0102ca0 2024-03-12T06:00:31.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.862+0000 7fc2a5193640 1 -- 172.21.15.76:0/3510352435 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a0198380 con 0x7fc2a0102ca0 2024-03-12T06:00:31.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.862+0000 7fc287fff640 1 -- 172.21.15.76:0/3510352435 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc290003d30 con 0x7fc2a0102ca0 2024-03-12T06:00:31.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.862+0000 7fc2a5193640 1 -- 172.21.15.76:0/3510352435 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a01a4500 con 0x7fc2a0102ca0 2024-03-12T06:00:31.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.864+0000 7fc287fff640 1 -- 172.21.15.76:0/3510352435 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc290004df0 con 0x7fc2a0102ca0 2024-03-12T06:00:31.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.864+0000 7fc2a7d6c740 1 Processor -- start 2024-03-12T06:00:31.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc2a7d6c740 1 -- start start 2024-03-12T06:00:31.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc2a7d6c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556195901f80 0x556195902360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc2a7d6c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561959028a0 0x55619590f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc2a7d6c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55619590fb60 0x556195911f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc2a7d6c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561958bd960 con 0x5561959028a0 2024-03-12T06:00:31.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc2a7d6c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x556195871310 con 0x556195901f80 2024-03-12T06:00:31.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc2a7d6c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55619586e840 con 0x55619590fb60 2024-03-12T06:00:31.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc29ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561959028a0 0x55619590f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc29ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561959028a0 0x55619590f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58114/0 (socket says 172.21.15.76:58114) 2024-03-12T06:00:31.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc29ed9d640 1 -- 172.21.15.76:0/2052102856 learned_addr learned my addr 172.21.15.76:0/2052102856 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc29f59e640 1 --2- 172.21.15.76:0/2052102856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556195901f80 0x556195902360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc287fff640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2880a1ba0 0x7fc2880a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc29e59c640 1 --2- 172.21.15.76:0/2052102856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55619590fb60 0x556195911f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc29ed9d640 1 -- 172.21.15.76:0/2052102856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55619590fb60 msgr2=0x556195911f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc29ed9d640 1 --2- 172.21.15.76:0/2052102856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55619590fb60 0x556195911f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc29ed9d640 1 -- 172.21.15.76:0/2052102856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556195901f80 msgr2=0x556195902360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc29ed9d640 1 --2- 172.21.15.76:0/2052102856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556195901f80 0x556195902360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc29ed9d640 1 -- 172.21.15.76:0/2052102856 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55619582c060 con 0x5561959028a0 2024-03-12T06:00:31.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc29f59e640 1 --2- 172.21.15.76:0/2052102856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556195901f80 0x556195902360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:00:31.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc29f59e640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2880a1ba0 0x7fc2880a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc287fff640 1 -- 172.21.15.76:0/3510352435 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc2900efec0 con 0x7fc2a0102ca0 2024-03-12T06:00:31.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.865+0000 7fc29ed9d640 1 --2- 172.21.15.76:0/2052102856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561959028a0 0x55619590f620 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc294009c30 tx=0x7fc294010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5750e475a288515 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.866+0000 7fc2857fa640 1 -- 172.21.15.76:0/2052102856 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc294004020 con 0x5561959028a0 2024-03-12T06:00:31.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.866+0000 7fc2857fa640 1 -- 172.21.15.76:0/2052102856 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc294004180 con 0x5561959028a0 2024-03-12T06:00:31.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.866+0000 7fc29f59e640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2880a1ba0 0x7fc2880a4060 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc298005c40 tx=0x7fc298004a40 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.866+0000 7fc2857fa640 1 -- 172.21.15.76:0/2052102856 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2940126b0 con 0x5561959028a0 2024-03-12T06:00:31.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.866+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/2052102856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561959028a0 msgr2=0x55619590f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.866+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/2052102856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561959028a0 0x55619590f620 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc294009c30 tx=0x7fc294010740 comp rx=0 tx=0).stop 2024-03-12T06:00:31.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.866+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/2052102856 shutdown_connections 2024-03-12T06:00:31.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.866+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/2052102856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55619590fb60 0x556195911f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.866+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/2052102856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561959028a0 0x55619590f620 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.866+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/2052102856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556195901f80 0x556195902360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.866+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/2052102856 >> 172.21.15.76:0/2052102856 conn(0x5561958fc980 msgr2=0x5561958fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.866+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/2052102856 shutdown_connections 2024-03-12T06:00:31.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.866+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/2052102856 wait complete. 2024-03-12T06:00:31.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc2a7d6c740 1 -- start start 2024-03-12T06:00:31.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc2a7d6c740 1 Processor -- start 2024-03-12T06:00:31.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc2a7d6c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556195901f80 0x55619590cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc2a7d6c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561959028a0 0x55619590d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc2a7d6c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55619590fb60 0x55619590d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc2a7d6c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55619586a020 con 0x55619590fb60 2024-03-12T06:00:31.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc2a7d6c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55619553d430 con 0x556195901f80 2024-03-12T06:00:31.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc2a7d6c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561954f9e00 con 0x5561959028a0 2024-03-12T06:00:31.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc2a7d6c740 10 client.0 _ll_register_callbacks cb 0x556195811cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:31.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc29f59e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55619590fb60 0x55619590d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc29f59e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55619590fb60 0x55619590d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58122/0 (socket says 172.21.15.76:58122) 2024-03-12T06:00:31.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc29f59e640 1 -- 172.21.15.76:0/754084680 learned_addr learned my addr 172.21.15.76:0/754084680 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc29ed9d640 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556195901f80 0x55619590cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc29e59c640 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561959028a0 0x55619590d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc29f59e640 1 -- 172.21.15.76:0/754084680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561959028a0 msgr2=0x55619590d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc29f59e640 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561959028a0 0x55619590d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc29f59e640 1 -- 172.21.15.76:0/754084680 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556195901f80 msgr2=0x55619590cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc29f59e640 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556195901f80 0x55619590cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc29f59e640 1 -- 172.21.15.76:0/754084680 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc294009890 con 0x55619590fb60 2024-03-12T06:00:31.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc29e59c640 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561959028a0 0x55619590d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:00:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.867+0000 7fc29ed9d640 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556195901f80 0x55619590cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:00:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc29f59e640 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55619590fb60 0x55619590d630 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc298012880 tx=0x7fc298004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1447d6559f720bab server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc282ff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc282ff5640 1 -- 172.21.15.76:0/754084680 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298010be0 con 0x55619590fb60 2024-03-12T06:00:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc282ff5640 1 -- 172.21.15.76:0/754084680 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc298010d40 con 0x55619590fb60 2024-03-12T06:00:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/754084680 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5561955697e0 con 0x55619590fb60 2024-03-12T06:00:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc282ff5640 1 -- 172.21.15.76:0/754084680 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc298003970 con 0x55619590fb60 2024-03-12T06:00:31.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc2a7d6c740 10 client.16623 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/754084680 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5561958686a0 con 0x55619590fb60 2024-03-12T06:00:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc282ff5640 1 -- 172.21.15.76:0/754084680 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5561955697e0 con 0x55619590fb60 2024-03-12T06:00:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc282ff5640 1 -- 172.21.15.76:0/754084680 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fc29801e420 con 0x55619590fb60 2024-03-12T06:00:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc2a7d6c740 10 client.16623 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc2a7d6c740 10 client.16623 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:00:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc2a7d6c740 20 client.16623 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5561959200b0 0x5561959224d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc2a7d6c740 4 client.16623 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:00:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/754084680 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5561955a6720 con 0x5561959200b0 2024-03-12T06:00:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.868+0000 7fc29ed9d640 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5561959200b0 0x5561959224d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.869+0000 7fc282ff5640 10 client.16623 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.869+0000 7fc29ed9d640 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5561959200b0 0x5561959224d0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62cd469b9a7aafb server_cookie=e81b05bcebc31fda in_seq=0 out_seq=0 2024-03-12T06:00:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.869+0000 7fc282ff5640 1 -- 172.21.15.76:0/754084680 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5561955a6720 con 0x5561959200b0 2024-03-12T06:00:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.870+0000 7fc282ff5640 10 client.16623 handle_command_reply: tid=0 2024-03-12T06:00:31.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.870+0000 7fc2a7d6c740 1 client.16623 shutdown 2024-03-12T06:00:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.870+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/754084680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55619590fb60 msgr2=0x55619590d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.870+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55619590fb60 0x55619590d630 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc298012880 tx=0x7fc298004ab0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.870+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/754084680 shutdown_connections 2024-03-12T06:00:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.870+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5561959200b0 0x5561959224d0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.870+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55619590fb60 0x55619590d630 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.870+0000 7fc282ff5640 0 client.16623 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.870+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561959028a0 0x55619590d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.870+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/754084680 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556195901f80 0x55619590cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.870+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/754084680 >> 172.21.15.76:0/754084680 conn(0x5561958fc980 msgr2=0x5561958fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.870+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/754084680 shutdown_connections 2024-03-12T06:00:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.870+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/754084680 wait complete. 2024-03-12T06:00:31.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.870+0000 7fc2a7d6c740 20 client.16623 trim_cache size 0 max 16384 2024-03-12T06:00:31.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc2a7d6c740 1 Processor -- start 2024-03-12T06:00:31.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc2a7d6c740 1 -- start start 2024-03-12T06:00:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc2a7d6c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556195901f80 0x55619590f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc2a7d6c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561959028a0 0x55619590d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc2a7d6c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55619590fb60 0x556195909e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc2a7d6c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561957ef130 con 0x5561959028a0 2024-03-12T06:00:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc2a7d6c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55619591e7e0 con 0x55619590fb60 2024-03-12T06:00:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc2a7d6c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561959015c0 con 0x556195901f80 2024-03-12T06:00:31.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc29e59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561959028a0 0x55619590d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc29e59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561959028a0 0x55619590d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58134/0 (socket says 172.21.15.76:58134) 2024-03-12T06:00:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc29e59c640 1 -- 172.21.15.76:0/2448427601 learned_addr learned my addr 172.21.15.76:0/2448427601 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc29e59c640 1 -- 172.21.15.76:0/2448427601 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556195901f80 msgr2=0x55619590f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc29e59c640 1 --2- 172.21.15.76:0/2448427601 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556195901f80 0x55619590f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc29e59c640 1 -- 172.21.15.76:0/2448427601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55619590fb60 msgr2=0x556195909e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc29e59c640 1 --2- 172.21.15.76:0/2448427601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55619590fb60 0x556195909e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc29e59c640 1 -- 172.21.15.76:0/2448427601 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556195804c20 con 0x5561959028a0 2024-03-12T06:00:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc29e59c640 1 --2- 172.21.15.76:0/2448427601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561959028a0 0x55619590d450 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc2900b9090 tx=0x7fc2900bc830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84cc5dcef7165e99 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc284ff9640 1 -- 172.21.15.76:0/2448427601 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2900c0d80 con 0x5561959028a0 2024-03-12T06:00:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc284ff9640 1 -- 172.21.15.76:0/2448427601 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2900c53d0 con 0x5561959028a0 2024-03-12T06:00:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.882+0000 7fc284ff9640 1 -- 172.21.15.76:0/2448427601 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2900bce90 con 0x5561959028a0 2024-03-12T06:00:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/2448427601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561959028a0 msgr2=0x55619590d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/2448427601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561959028a0 0x55619590d450 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc2900b9090 tx=0x7fc2900bc830 comp rx=0 tx=0).stop 2024-03-12T06:00:31.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/2448427601 shutdown_connections 2024-03-12T06:00:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/2448427601 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55619590fb60 0x556195909e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/2448427601 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561959028a0 0x55619590d450 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/2448427601 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556195901f80 0x55619590f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/2448427601 >> 172.21.15.76:0/2448427601 conn(0x5561958fc980 msgr2=0x556195907ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/2448427601 shutdown_connections 2024-03-12T06:00:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/2448427601 wait complete. 2024-03-12T06:00:31.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 -- start start 2024-03-12T06:00:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 Processor -- start 2024-03-12T06:00:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556195901f80 0x5561959157b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561959028a0 0x556195915cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55619590fb60 0x5561959765e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561959164d0 con 0x556195901f80 2024-03-12T06:00:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x556195916640 con 0x5561959028a0 2024-03-12T06:00:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556195976b70 con 0x55619590fb60 2024-03-12T06:00:31.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc2a7d6c740 10 client.0 _ll_register_callbacks cb 0x7fc2940095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc29ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556195901f80 0x5561959157b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc29ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556195901f80 0x5561959157b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58144/0 (socket says 172.21.15.76:58144) 2024-03-12T06:00:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc29ed9d640 1 -- 172.21.15.76:0/3750242504 learned_addr learned my addr 172.21.15.76:0/3750242504 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc29f59e640 1 --2- 172.21.15.76:0/3750242504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55619590fb60 0x5561959765e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc29ed9d640 1 -- 172.21.15.76:0/3750242504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55619590fb60 msgr2=0x5561959765e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.883+0000 7fc29ed9d640 1 --2- 172.21.15.76:0/3750242504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55619590fb60 0x5561959765e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc29ed9d640 1 -- 172.21.15.76:0/3750242504 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561959028a0 msgr2=0x556195915cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc29ed9d640 1 --2- 172.21.15.76:0/3750242504 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561959028a0 0x556195915cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc29ed9d640 1 -- 172.21.15.76:0/3750242504 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a01a4500 con 0x556195901f80 2024-03-12T06:00:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc29ed9d640 1 --2- 172.21.15.76:0/3750242504 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556195901f80 0x5561959157b0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x556195903160 tx=0x7fc2940040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13be59ee9d0b6058 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc282ff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc282ff5640 1 -- 172.21.15.76:0/3750242504 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc294027670 con 0x556195901f80 2024-03-12T06:00:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc282ff5640 1 -- 172.21.15.76:0/3750242504 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc294016070 con 0x556195901f80 2024-03-12T06:00:31.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/3750242504 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc294004180 con 0x556195901f80 2024-03-12T06:00:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc282ff5640 1 -- 172.21.15.76:0/3750242504 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2940277d0 con 0x556195901f80 2024-03-12T06:00:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc2a7d6c740 10 client.16629 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/3750242504 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55619582a8c0 con 0x556195901f80 2024-03-12T06:00:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc282ff5640 1 -- 172.21.15.76:0/3750242504 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc294034020 con 0x556195901f80 2024-03-12T06:00:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc282ff5640 1 -- 172.21.15.76:0/3750242504 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fc294012450 con 0x556195901f80 2024-03-12T06:00:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc2a7d6c740 10 client.16629 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc2a7d6c740 10 client.16629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:00:31.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc2a7d6c740 20 client.16629 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/3750242504 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556195978420 0x55619597a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc2a7d6c740 4 client.16629 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:00:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.884+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/3750242504 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5561955f5890 con 0x556195978420 2024-03-12T06:00:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc29e59c640 1 --2- 172.21.15.76:0/3750242504 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556195978420 0x55619597a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc282ff5640 10 client.16629 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:31.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc29e59c640 1 --2- 172.21.15.76:0/3750242504 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556195978420 0x55619597a840 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ace3c4d70793a9ae server_cookie=eeb9f868abe30eb7 in_seq=0 out_seq=0 2024-03-12T06:00:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc282ff5640 1 -- 172.21.15.76:0/3750242504 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5561955f5890 con 0x556195978420 2024-03-12T06:00:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc282ff5640 10 client.16629 handle_command_reply: tid=0 2024-03-12T06:00:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc2a7d6c740 1 client.16629 shutdown 2024-03-12T06:00:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/3750242504 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556195901f80 msgr2=0x5561959157b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/3750242504 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556195901f80 0x5561959157b0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x556195903160 tx=0x7fc2940040c0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/3750242504 shutdown_connections 2024-03-12T06:00:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/3750242504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55619590fb60 0x5561959765e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/3750242504 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561959028a0 0x556195915cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/3750242504 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556195978420 0x55619597a840 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc2a7d6c740 1 --2- 172.21.15.76:0/3750242504 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556195901f80 0x5561959157b0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc282ff5640 0 client.16629 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/3750242504 >> 172.21.15.76:0/3750242504 conn(0x5561958fc980 msgr2=0x5561958fb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/3750242504 shutdown_connections 2024-03-12T06:00:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc2a7d6c740 1 -- 172.21.15.76:0/3750242504 wait complete. 2024-03-12T06:00:31.909 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.885+0000 7fc2a7d6c740 20 client.16629 trim_cache size 0 max 16384 2024-03-12T06:00:31.910 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:00:31.911 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.886+0000 7fc2a5193640 1 -- 172.21.15.76:0/3510352435 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2880a1ba0 msgr2=0x7fc2880a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.886+0000 7fc2a5193640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2880a1ba0 0x7fc2880a4060 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc298005c40 tx=0x7fc298004a40 comp rx=0 tx=0).stop 2024-03-12T06:00:31.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.886+0000 7fc2a5193640 1 -- 172.21.15.76:0/3510352435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2a0102ca0 msgr2=0x7fc2a019dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:31.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.886+0000 7fc2a5193640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2a0102ca0 0x7fc2a019dac0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc290005940 tx=0x7fc29002c710 comp rx=0 tx=0).stop 2024-03-12T06:00:31.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.886+0000 7fc2a5193640 1 -- 172.21.15.76:0/3510352435 shutdown_connections 2024-03-12T06:00:31.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.886+0000 7fc2a5193640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc2880a1ba0 0x7fc2880a4060 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.886+0000 7fc2a5193640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc2a010b750 0x7fc2a0197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.886+0000 7fc2a5193640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc2a0102ca0 0x7fc2a019dac0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.886+0000 7fc2a5193640 1 --2- 172.21.15.76:0/3510352435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2a0102380 0x7fc2a019d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:31.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.886+0000 7fc2a5193640 1 -- 172.21.15.76:0/3510352435 >> 172.21.15.76:0/3510352435 conn(0x7fc2a00fbae0 msgr2=0x7fc2a00fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:31.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.886+0000 7fc2a5193640 1 -- 172.21.15.76:0/3510352435 shutdown_connections 2024-03-12T06:00:31.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:31.886+0000 7fc2a5193640 1 -- 172.21.15.76:0/3510352435 wait complete. 2024-03-12T06:00:32.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:32 smithi082 ceph-mon[36382]: pgmap v145: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:32.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:32 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1941578416' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:00:32.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:32 smithi088 ceph-mon[36585]: pgmap v145: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:32.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:32 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1941578416' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:00:32.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:32 smithi076 ceph-mon[32095]: pgmap v145: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:32.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:32 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1941578416' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:00:32.914 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T06:00:32.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:00:32.915 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T06:00:33.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.030+0000 7fba87913640 1 -- 172.21.15.76:0/4184069310 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba80105360 msgr2=0x7fba8010cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.030+0000 7fba87913640 1 --2- 172.21.15.76:0/4184069310 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba80105360 0x7fba8010cf30 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009930 tx=0x7fba7c02bc30 comp rx=0 tx=0).stop 2024-03-12T06:00:33.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.030+0000 7fba87913640 1 -- 172.21.15.76:0/4184069310 shutdown_connections 2024-03-12T06:00:33.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.030+0000 7fba87913640 1 --2- 172.21.15.76:0/4184069310 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba80105360 0x7fba8010cf30 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.030+0000 7fba87913640 1 --2- 172.21.15.76:0/4184069310 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba80100c40 0x7fba80104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.030+0000 7fba87913640 1 --2- 172.21.15.76:0/4184069310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba80100290 0x7fba80100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.030+0000 7fba87913640 1 -- 172.21.15.76:0/4184069310 >> 172.21.15.76:0/4184069310 conn(0x7fba80077230 msgr2=0x7fba80077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:33.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.030+0000 7fba87913640 1 -- 172.21.15.76:0/4184069310 shutdown_connections 2024-03-12T06:00:33.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.030+0000 7fba87913640 1 -- 172.21.15.76:0/4184069310 wait complete. 2024-03-12T06:00:33.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba87913640 1 Processor -- start 2024-03-12T06:00:33.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba87913640 1 -- start start 2024-03-12T06:00:33.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba87913640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba80100290 0x7fba80198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba87913640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba80105360 0x7fba80198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba87913640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba80199050 0x7fba801a35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba87913640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba801a3ca0 con 0x7fba80105360 2024-03-12T06:00:33.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba87913640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba801a3e10 con 0x7fba80100290 2024-03-12T06:00:33.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba87913640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba801a3fd0 con 0x7fba80199050 2024-03-12T06:00:33.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba84eae640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba80105360 0x7fba80198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba84eae640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba80105360 0x7fba80198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58166/0 (socket says 172.21.15.76:58166) 2024-03-12T06:00:33.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba84eae640 1 -- 172.21.15.76:0/51663709 learned_addr learned my addr 172.21.15.76:0/51663709 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:33.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba856af640 1 --2- 172.21.15.76:0/51663709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba80100290 0x7fba80198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba85eb0640 1 --2- 172.21.15.76:0/51663709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba80199050 0x7fba801a35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba84eae640 1 -- 172.21.15.76:0/51663709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba80199050 msgr2=0x7fba801a35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba84eae640 1 --2- 172.21.15.76:0/51663709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba80199050 0x7fba801a35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba84eae640 1 -- 172.21.15.76:0/51663709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba80100290 msgr2=0x7fba80198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba84eae640 1 --2- 172.21.15.76:0/51663709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba80100290 0x7fba80198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.031+0000 7fba84eae640 1 -- 172.21.15.76:0/51663709 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7c009560 con 0x7fba80105360 2024-03-12T06:00:33.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.032+0000 7fba84eae640 1 --2- 172.21.15.76:0/51663709 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba80105360 0x7fba80198a70 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fba7000d770 tx=0x7fba7000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:33.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.032+0000 7fba767fc640 1 -- 172.21.15.76:0/51663709 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba70009c30 con 0x7fba80105360 2024-03-12T06:00:33.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.032+0000 7fba767fc640 1 -- 172.21.15.76:0/51663709 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba7000de80 con 0x7fba80105360 2024-03-12T06:00:33.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.032+0000 7fba87913640 1 -- 172.21.15.76:0/51663709 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba801a42b0 con 0x7fba80105360 2024-03-12T06:00:33.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.032+0000 7fba767fc640 1 -- 172.21.15.76:0/51663709 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7000f6b0 con 0x7fba80105360 2024-03-12T06:00:33.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.032+0000 7fba87913640 1 -- 172.21.15.76:0/51663709 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba801a4810 con 0x7fba80105360 2024-03-12T06:00:33.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.034+0000 7fba767fc640 1 -- 172.21.15.76:0/51663709 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fba70010460 con 0x7fba80105360 2024-03-12T06:00:33.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.035+0000 7fba8a5c9740 1 Processor -- start 2024-03-12T06:00:33.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.035+0000 7fba8a5c9740 1 -- start start 2024-03-12T06:00:33.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.035+0000 7fba8a5c9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a511e0 0x5570e4a515c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.035+0000 7fba8a5c9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a51b90 0x5570e4a5a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.035+0000 7fba8a5c9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a5acf0 0x5570e4a5b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.035+0000 7fba8a5c9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570e49bff90 con 0x5570e4a511e0 2024-03-12T06:00:33.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.035+0000 7fba8a5c9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570e49bd4c0 con 0x5570e4a51b90 2024-03-12T06:00:33.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.035+0000 7fba8a5c9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570e49b8ca0 con 0x5570e4a5acf0 2024-03-12T06:00:33.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.035+0000 7fba767fc640 1 --2- 172.21.15.76:0/51663709 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fba6c0a1c20 0x7fba6c0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.035+0000 7fba85eb0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a511e0 0x5570e4a515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.035+0000 7fba85eb0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a511e0 0x5570e4a515c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58172/0 (socket says 172.21.15.76:58172) 2024-03-12T06:00:33.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.035+0000 7fba85eb0640 1 -- 172.21.15.76:0/991880997 learned_addr learned my addr 172.21.15.76:0/991880997 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:33.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.035+0000 7fba85eb0640 1 --2- 172.21.15.76:0/51663709 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fba6c0a1c20 0x7fba6c0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.036+0000 7fba856af640 1 --2- 172.21.15.76:0/991880997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a51b90 0x5570e4a5a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.036+0000 7fba767fc640 1 -- 172.21.15.76:0/51663709 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fba700cd3a0 con 0x7fba80105360 2024-03-12T06:00:33.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.036+0000 7fba84eae640 1 --2- 172.21.15.76:0/991880997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a5acf0 0x5570e4a5b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.036+0000 7fba85eb0640 1 -- 172.21.15.76:0/991880997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a5acf0 msgr2=0x5570e4a5b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.036+0000 7fba85eb0640 1 --2- 172.21.15.76:0/991880997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a5acf0 0x5570e4a5b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.036+0000 7fba85eb0640 1 -- 172.21.15.76:0/991880997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a51b90 msgr2=0x5570e4a5a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.036+0000 7fba85eb0640 1 --2- 172.21.15.76:0/991880997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a51b90 0x5570e4a5a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.036+0000 7fba85eb0640 1 -- 172.21.15.76:0/991880997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570e497b0b0 con 0x5570e4a511e0 2024-03-12T06:00:33.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.036+0000 7fba85eb0640 1 --2- 172.21.15.76:0/991880997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a511e0 0x5570e4a515c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009230 tx=0x7fba7c02cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdefb1082ee3a80a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:33.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.036+0000 7fba85eb0640 1 --2- 172.21.15.76:0/51663709 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fba6c0a1c20 0x7fba6c0a40e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fba7c005ab0 tx=0x7fba7c049000 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:33.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.036+0000 7fba6b7fe640 1 -- 172.21.15.76:0/991880997 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7c048070 con 0x5570e4a511e0 2024-03-12T06:00:33.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.036+0000 7fba6b7fe640 1 -- 172.21.15.76:0/991880997 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba7c02c900 con 0x5570e4a511e0 2024-03-12T06:00:33.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.036+0000 7fba6b7fe640 1 -- 172.21.15.76:0/991880997 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7c0038b0 con 0x5570e4a511e0 2024-03-12T06:00:33.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.037+0000 7fba8a5c9740 1 -- 172.21.15.76:0/991880997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a511e0 msgr2=0x5570e4a515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.037+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/991880997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a511e0 0x5570e4a515c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009230 tx=0x7fba7c02cc40 comp rx=0 tx=0).stop 2024-03-12T06:00:33.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.037+0000 7fba8a5c9740 1 -- 172.21.15.76:0/991880997 shutdown_connections 2024-03-12T06:00:33.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.037+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/991880997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a5acf0 0x5570e4a5b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.037+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/991880997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a51b90 0x5570e4a5a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.037+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/991880997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a511e0 0x5570e4a515c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.037+0000 7fba8a5c9740 1 -- 172.21.15.76:0/991880997 >> 172.21.15.76:0/991880997 conn(0x5570e4a4b840 msgr2=0x5570e4a4bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:33.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.037+0000 7fba8a5c9740 1 -- 172.21.15.76:0/991880997 shutdown_connections 2024-03-12T06:00:33.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.037+0000 7fba8a5c9740 1 -- 172.21.15.76:0/991880997 wait complete. 2024-03-12T06:00:33.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.037+0000 7fba8a5c9740 1 -- start start 2024-03-12T06:00:33.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.037+0000 7fba8a5c9740 1 Processor -- start 2024-03-12T06:00:33.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba8a5c9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a511e0 0x5570e4a5e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba8a5c9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a51b90 0x5570e4a5ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba8a5c9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 0x5570e4a5f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba8a5c9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570e495df40 con 0x5570e4a511e0 2024-03-12T06:00:33.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba8a5c9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570e468c4b0 con 0x5570e4a5acf0 2024-03-12T06:00:33.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba8a5c9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570e493e670 con 0x5570e4a51b90 2024-03-12T06:00:33.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba8a5c9740 10 client.0 _ll_register_callbacks cb 0x5570e49abf70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:33.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba856af640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a511e0 0x5570e4a5e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba856af640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a511e0 0x5570e4a5e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58182/0 (socket says 172.21.15.76:58182) 2024-03-12T06:00:33.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba856af640 1 -- 172.21.15.76:0/1586649559 learned_addr learned my addr 172.21.15.76:0/1586649559 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:33.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba85eb0640 1 --2- 172.21.15.76:0/1586649559 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 0x5570e4a5f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba84eae640 1 --2- 172.21.15.76:0/1586649559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a51b90 0x5570e4a5ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba856af640 1 -- 172.21.15.76:0/1586649559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a51b90 msgr2=0x5570e4a5ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba856af640 1 --2- 172.21.15.76:0/1586649559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a51b90 0x5570e4a5ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba856af640 1 -- 172.21.15.76:0/1586649559 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 msgr2=0x5570e4a5f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba856af640 1 --2- 172.21.15.76:0/1586649559 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 0x5570e4a5f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba856af640 1 -- 172.21.15.76:0/1586649559 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba8010d740 con 0x5570e4a511e0 2024-03-12T06:00:33.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba856af640 1 --2- 172.21.15.76:0/1586649559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a511e0 0x5570e4a5e7c0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fba78010b30 tx=0x7fba7800f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7610c96bb7ee3e92 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba68ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.038+0000 7fba68ff9640 1 -- 172.21.15.76:0/1586649559 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78015070 con 0x5570e4a511e0 2024-03-12T06:00:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.039+0000 7fba8a5c9740 1 -- 172.21.15.76:0/1586649559 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5570e4992f40 con 0x5570e4a511e0 2024-03-12T06:00:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.039+0000 7fba68ff9640 1 -- 172.21.15.76:0/1586649559 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fba7800fde0 con 0x5570e4a511e0 2024-03-12T06:00:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.039+0000 7fba68ff9640 1 -- 172.21.15.76:0/1586649559 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba7801a420 con 0x5570e4a511e0 2024-03-12T06:00:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.039+0000 7fba8a5c9740 10 client.16641 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.039+0000 7fba8a5c9740 1 -- 172.21.15.76:0/1586649559 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5570e49b7320 con 0x5570e4a511e0 2024-03-12T06:00:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.039+0000 7fba68ff9640 1 -- 172.21.15.76:0/1586649559 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5570e4992f40 con 0x5570e4a511e0 2024-03-12T06:00:33.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.039+0000 7fba68ff9640 1 -- 172.21.15.76:0/1586649559 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fba7800a070 con 0x5570e4a511e0 2024-03-12T06:00:33.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.039+0000 7fba8a5c9740 10 client.16641 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:33.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.039+0000 7fba8a5c9740 10 client.16641 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:00:33.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.039+0000 7fba8a5c9740 20 client.16641 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:33.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.039+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/1586649559 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5570e4a6f450 0x5570e4a71870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.039+0000 7fba8a5c9740 4 client.16641 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:00:33.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.039+0000 7fba8a5c9740 1 -- 172.21.15.76:0/1586649559 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5570e49af310 con 0x5570e4a6f450 2024-03-12T06:00:33.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.039+0000 7fba84eae640 1 --2- 172.21.15.76:0/1586649559 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5570e4a6f450 0x5570e4a71870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.040+0000 7fba68ff9640 10 client.16641 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:33.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.040+0000 7fba84eae640 1 --2- 172.21.15.76:0/1586649559 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5570e4a6f450 0x5570e4a71870 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bd2652574953498 server_cookie=199187d0dc91914b in_seq=0 out_seq=0 2024-03-12T06:00:33.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.040+0000 7fba68ff9640 1 -- 172.21.15.76:0/1586649559 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5570e49af310 con 0x5570e4a6f450 2024-03-12T06:00:33.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.040+0000 7fba68ff9640 10 client.16641 handle_command_reply: tid=0 2024-03-12T06:00:33.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.041+0000 7fba8a5c9740 1 client.16641 shutdown 2024-03-12T06:00:33.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.041+0000 7fba8a5c9740 1 -- 172.21.15.76:0/1586649559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a511e0 msgr2=0x5570e4a5e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.041+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/1586649559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a511e0 0x5570e4a5e7c0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fba78010b30 tx=0x7fba7800f420 comp rx=0 tx=0).stop 2024-03-12T06:00:33.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.041+0000 7fba8a5c9740 1 -- 172.21.15.76:0/1586649559 shutdown_connections 2024-03-12T06:00:33.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.041+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/1586649559 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 0x5570e4a5f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.041+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/1586649559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a51b90 0x5570e4a5ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.041+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/1586649559 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5570e4a6f450 0x5570e4a71870 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.041+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/1586649559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a511e0 0x5570e4a5e7c0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.041+0000 7fba68ff9640 0 client.16641 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:33.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.041+0000 7fba8a5c9740 1 -- 172.21.15.76:0/1586649559 >> 172.21.15.76:0/1586649559 conn(0x5570e4a4b840 msgr2=0x5570e4a4d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:33.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.041+0000 7fba8a5c9740 1 -- 172.21.15.76:0/1586649559 shutdown_connections 2024-03-12T06:00:33.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.041+0000 7fba8a5c9740 1 -- 172.21.15.76:0/1586649559 wait complete. 2024-03-12T06:00:33.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.041+0000 7fba8a5c9740 20 client.16641 trim_cache size 0 max 16384 2024-03-12T06:00:33.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba8a5c9740 1 Processor -- start 2024-03-12T06:00:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba8a5c9740 1 -- start start 2024-03-12T06:00:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba8a5c9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a511e0 0x5570e4a60ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba8a5c9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a51b90 0x5570e4a5c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba8a5c9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 0x5570e4a5f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba8a5c9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570e4a6dad0 con 0x5570e4a51b90 2024-03-12T06:00:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba8a5c9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570e4a45cf0 con 0x5570e4a5acf0 2024-03-12T06:00:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba8a5c9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570e4a518f0 con 0x5570e4a511e0 2024-03-12T06:00:33.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba84eae640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a51b90 0x5570e4a5c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba84eae640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a51b90 0x5570e4a5c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58186/0 (socket says 172.21.15.76:58186) 2024-03-12T06:00:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba84eae640 1 -- 172.21.15.76:0/4234808595 learned_addr learned my addr 172.21.15.76:0/4234808595 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba856af640 1 --2- 172.21.15.76:0/4234808595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a511e0 0x5570e4a60ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba85eb0640 1 --2- 172.21.15.76:0/4234808595 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 0x5570e4a5f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba84eae640 1 -- 172.21.15.76:0/4234808595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a511e0 msgr2=0x5570e4a60ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba84eae640 1 --2- 172.21.15.76:0/4234808595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a511e0 0x5570e4a60ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba84eae640 1 -- 172.21.15.76:0/4234808595 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 msgr2=0x5570e4a5f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba84eae640 1 --2- 172.21.15.76:0/4234808595 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 0x5570e4a5f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba84eae640 1 -- 172.21.15.76:0/4234808595 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570e4953c00 con 0x5570e4a51b90 2024-03-12T06:00:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.056+0000 7fba85eb0640 1 --2- 172.21.15.76:0/4234808595 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 0x5570e4a5f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:00:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.057+0000 7fba84eae640 1 --2- 172.21.15.76:0/4234808595 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a51b90 0x5570e4a5c0e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fba700cd6c0 tx=0x7fba70099e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fbee1383a9244 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.057+0000 7fba6affd640 1 -- 172.21.15.76:0/4234808595 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba70020ea0 con 0x5570e4a51b90 2024-03-12T06:00:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.057+0000 7fba6affd640 1 -- 172.21.15.76:0/4234808595 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba7009e150 con 0x5570e4a51b90 2024-03-12T06:00:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.057+0000 7fba6affd640 1 -- 172.21.15.76:0/4234808595 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba700984b0 con 0x5570e4a51b90 2024-03-12T06:00:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.057+0000 7fba8a5c9740 1 -- 172.21.15.76:0/4234808595 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a51b90 msgr2=0x5570e4a5c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.057+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/4234808595 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a51b90 0x5570e4a5c0e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fba700cd6c0 tx=0x7fba70099e80 comp rx=0 tx=0).stop 2024-03-12T06:00:33.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.057+0000 7fba8a5c9740 1 -- 172.21.15.76:0/4234808595 shutdown_connections 2024-03-12T06:00:33.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.057+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/4234808595 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 0x5570e4a5f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.057+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/4234808595 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a51b90 0x5570e4a5c0e0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.057+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/4234808595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a511e0 0x5570e4a60ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.057+0000 7fba8a5c9740 1 -- 172.21.15.76:0/4234808595 >> 172.21.15.76:0/4234808595 conn(0x5570e4a4b840 msgr2=0x5570e4a5d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:33.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.057+0000 7fba8a5c9740 1 -- 172.21.15.76:0/4234808595 shutdown_connections 2024-03-12T06:00:33.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.057+0000 7fba8a5c9740 1 -- 172.21.15.76:0/4234808595 wait complete. 2024-03-12T06:00:33.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba8a5c9740 1 -- start start 2024-03-12T06:00:33.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba8a5c9740 1 Processor -- start 2024-03-12T06:00:33.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba8a5c9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a511e0 0x5570e4ac0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba8a5c9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a51b90 0x5570e4ac0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba8a5c9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 0x5570e4a65400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba8a5c9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570e4a659c0 con 0x5570e4a51b90 2024-03-12T06:00:33.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba8a5c9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570e4a65b30 con 0x5570e4a5acf0 2024-03-12T06:00:33.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba8a5c9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570e4a65cf0 con 0x5570e4a511e0 2024-03-12T06:00:33.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba8a5c9740 10 client.0 _ll_register_callbacks cb 0x7fba7c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:33.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba84eae640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a51b90 0x5570e4ac0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba84eae640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a51b90 0x5570e4ac0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58200/0 (socket says 172.21.15.76:58200) 2024-03-12T06:00:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba84eae640 1 -- 172.21.15.76:0/3442811374 learned_addr learned my addr 172.21.15.76:0/3442811374 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba84eae640 1 -- 172.21.15.76:0/3442811374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a511e0 msgr2=0x5570e4ac0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba84eae640 1 --2- 172.21.15.76:0/3442811374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a511e0 0x5570e4ac0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba84eae640 1 -- 172.21.15.76:0/3442811374 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 msgr2=0x5570e4a65400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba84eae640 1 --2- 172.21.15.76:0/3442811374 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 0x5570e4a65400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.058+0000 7fba84eae640 1 -- 172.21.15.76:0/3442811374 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba801a4810 con 0x5570e4a51b90 2024-03-12T06:00:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba84eae640 1 --2- 172.21.15.76:0/3442811374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a51b90 0x5570e4ac0680 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fba70009610 tx=0x7fba70005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845863c57c9d3ae3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba68ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:33.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba68ff9640 1 -- 172.21.15.76:0/3442811374 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba700a27a0 con 0x5570e4a51b90 2024-03-12T06:00:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba68ff9640 1 -- 172.21.15.76:0/3442811374 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fba7007d070 con 0x5570e4a51b90 2024-03-12T06:00:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba8a5c9740 1 -- 172.21.15.76:0/3442811374 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5570e49de900 con 0x5570e4a51b90 2024-03-12T06:00:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba68ff9640 1 -- 172.21.15.76:0/3442811374 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba700cada0 con 0x5570e4a51b90 2024-03-12T06:00:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba8a5c9740 10 client.16647 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba8a5c9740 1 -- 172.21.15.76:0/3442811374 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5570e4979910 con 0x5570e4a51b90 2024-03-12T06:00:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba68ff9640 1 -- 172.21.15.76:0/3442811374 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5570e49de900 con 0x5570e4a51b90 2024-03-12T06:00:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba68ff9640 1 -- 172.21.15.76:0/3442811374 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fba700b82a0 con 0x5570e4a51b90 2024-03-12T06:00:33.134 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba8a5c9740 10 client.16647 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba8a5c9740 10 client.16647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:00:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba8a5c9740 20 client.16647 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/3442811374 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5570e4ac7cd0 0x5570e4aca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba8a5c9740 4 client.16647 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T06:00:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba8a5c9740 1 -- 172.21.15.76:0/3442811374 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5570e46b8860 con 0x5570e4ac7cd0 2024-03-12T06:00:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.059+0000 7fba856af640 1 --2- 172.21.15.76:0/3442811374 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5570e4ac7cd0 0x5570e4aca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.060+0000 7fba68ff9640 10 client.16647 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.060+0000 7fba856af640 1 --2- 172.21.15.76:0/3442811374 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5570e4ac7cd0 0x5570e4aca0f0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62c47f2999e0ac0d server_cookie=edb13902656a882c in_seq=0 out_seq=0 2024-03-12T06:00:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba68ff9640 1 -- 172.21.15.76:0/3442811374 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5570e46b8860 con 0x5570e4ac7cd0 2024-03-12T06:00:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba68ff9640 10 client.16647 handle_command_reply: tid=0 2024-03-12T06:00:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba8a5c9740 1 client.16647 shutdown 2024-03-12T06:00:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba8a5c9740 1 -- 172.21.15.76:0/3442811374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a51b90 msgr2=0x5570e4ac0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.137 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/3442811374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a51b90 0x5570e4ac0680 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fba70009610 tx=0x7fba70005b90 comp rx=0 tx=0).stop 2024-03-12T06:00:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba8a5c9740 1 -- 172.21.15.76:0/3442811374 shutdown_connections 2024-03-12T06:00:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/3442811374 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5570e4a5acf0 0x5570e4a65400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/3442811374 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5570e4ac7cd0 0x5570e4aca0f0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/3442811374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5570e4a51b90 0x5570e4ac0680 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba8a5c9740 1 --2- 172.21.15.76:0/3442811374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5570e4a511e0 0x5570e4ac0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba68ff9640 0 client.16647 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba8a5c9740 1 -- 172.21.15.76:0/3442811374 >> 172.21.15.76:0/3442811374 conn(0x5570e4a4b840 msgr2=0x5570e4a4a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba8a5c9740 1 -- 172.21.15.76:0/3442811374 shutdown_connections 2024-03-12T06:00:33.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba8a5c9740 1 -- 172.21.15.76:0/3442811374 wait complete. 2024-03-12T06:00:33.139 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.061+0000 7fba8a5c9740 20 client.16647 trim_cache size 0 max 16384 2024-03-12T06:00:33.139 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:00:33.139 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T06:00:33.139 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "af40f471-da39-4bae-ad45-1b9f97b9db2e", 2024-03-12T06:00:33.139 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T06:00:33.139 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:00:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.062+0000 7fba87913640 1 -- 172.21.15.76:0/51663709 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fba6c0a1c20 msgr2=0x7fba6c0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.062+0000 7fba87913640 1 --2- 172.21.15.76:0/51663709 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fba6c0a1c20 0x7fba6c0a40e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fba7c005ab0 tx=0x7fba7c049000 comp rx=0 tx=0).stop 2024-03-12T06:00:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.062+0000 7fba87913640 1 -- 172.21.15.76:0/51663709 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba80105360 msgr2=0x7fba80198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.062+0000 7fba87913640 1 --2- 172.21.15.76:0/51663709 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba80105360 0x7fba80198a70 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fba7000d770 tx=0x7fba7000dc30 comp rx=0 tx=0).stop 2024-03-12T06:00:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.062+0000 7fba87913640 1 -- 172.21.15.76:0/51663709 shutdown_connections 2024-03-12T06:00:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.062+0000 7fba87913640 1 --2- 172.21.15.76:0/51663709 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fba6c0a1c20 0x7fba6c0a40e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.062+0000 7fba87913640 1 --2- 172.21.15.76:0/51663709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba80199050 0x7fba801a35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.062+0000 7fba87913640 1 --2- 172.21.15.76:0/51663709 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba80105360 0x7fba80198a70 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.062+0000 7fba87913640 1 --2- 172.21.15.76:0/51663709 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba80100290 0x7fba80198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.062+0000 7fba87913640 1 -- 172.21.15.76:0/51663709 >> 172.21.15.76:0/51663709 conn(0x7fba80077230 msgr2=0x7fba800fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.062+0000 7fba87913640 1 -- 172.21.15.76:0/51663709 shutdown_connections 2024-03-12T06:00:33.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.062+0000 7fba87913640 1 -- 172.21.15.76:0/51663709 wait complete. 2024-03-12T06:00:33.142 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:00:33.143 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:00:33.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.267+0000 7f09e73f4640 1 -- 172.21.15.76:0/1478998310 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e0100c10 msgr2=0x7f09e0104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.267+0000 7f09e73f4640 1 --2- 172.21.15.76:0/1478998310 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e0100c10 0x7f09e0104c50 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f09d0009930 tx=0x7f09d002bc30 comp rx=0 tx=0).stop 2024-03-12T06:00:33.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.268+0000 7f09e73f4640 1 -- 172.21.15.76:0/1478998310 shutdown_connections 2024-03-12T06:00:33.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.268+0000 7f09e73f4640 1 --2- 172.21.15.76:0/1478998310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09e0105330 0x7f09e010cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.268+0000 7f09e73f4640 1 --2- 172.21.15.76:0/1478998310 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e0100c10 0x7f09e0104c50 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.268+0000 7f09e73f4640 1 --2- 172.21.15.76:0/1478998310 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09e0100260 0x7f09e0100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.268+0000 7f09e73f4640 1 -- 172.21.15.76:0/1478998310 >> 172.21.15.76:0/1478998310 conn(0x7f09e0077180 msgr2=0x7f09e0077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:33.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.268+0000 7f09e73f4640 1 -- 172.21.15.76:0/1478998310 shutdown_connections 2024-03-12T06:00:33.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.268+0000 7f09e73f4640 1 -- 172.21.15.76:0/1478998310 wait complete. 2024-03-12T06:00:33.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.268+0000 7f09e73f4640 1 Processor -- start 2024-03-12T06:00:33.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.268+0000 7f09e73f4640 1 -- start start 2024-03-12T06:00:33.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.268+0000 7f09e73f4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09e0100260 0x7f09e019f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e73f4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e0105330 0x7f09e019fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e73f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09e01997e0 0x7f09e0199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e73f4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e019a290 con 0x7f09e0105330 2024-03-12T06:00:33.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e73f4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e019a400 con 0x7f09e0100260 2024-03-12T06:00:33.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e73f4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e019a570 con 0x7f09e01997e0 2024-03-12T06:00:33.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e5190640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09e0100260 0x7f09e019f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e5190640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09e0100260 0x7f09e019f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:38670/0 (socket says 172.21.15.76:38670) 2024-03-12T06:00:33.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e5190640 1 -- 172.21.15.76:0/870541373 learned_addr learned my addr 172.21.15.76:0/870541373 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:33.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e498f640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e0105330 0x7f09e019fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e5991640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09e01997e0 0x7f09e0199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e5190640 1 -- 172.21.15.76:0/870541373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09e01997e0 msgr2=0x7f09e0199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e5190640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09e01997e0 0x7f09e0199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e5190640 1 -- 172.21.15.76:0/870541373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e0105330 msgr2=0x7f09e019fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e5190640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e0105330 0x7f09e019fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e5190640 1 -- 172.21.15.76:0/870541373 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc009490 con 0x7f09e0100260 2024-03-12T06:00:33.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e498f640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e0105330 0x7f09e019fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:00:33.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09e5190640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09e0100260 0x7f09e019f670 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00ea60 tx=0x7f09dc00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:33.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.269+0000 7f09d67fc640 1 -- 172.21.15.76:0/870541373 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc00cde0 con 0x7f09e0100260 2024-03-12T06:00:33.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.270+0000 7f09d67fc640 1 -- 172.21.15.76:0/870541373 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09dc0035f0 con 0x7f09e0100260 2024-03-12T06:00:33.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.270+0000 7f09e73f4640 1 -- 172.21.15.76:0/870541373 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d0009560 con 0x7f09e0100260 2024-03-12T06:00:33.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.270+0000 7f09d67fc640 1 -- 172.21.15.76:0/870541373 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc010690 con 0x7f09e0100260 2024-03-12T06:00:33.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.270+0000 7f09e73f4640 1 -- 172.21.15.76:0/870541373 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e01a6820 con 0x7f09e0100260 2024-03-12T06:00:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.272+0000 7f09d67fc640 1 -- 172.21.15.76:0/870541373 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f09dc003af0 con 0x7f09e0100260 2024-03-12T06:00:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09e9fc8740 1 Processor -- start 2024-03-12T06:00:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09e9fc8740 1 -- start start 2024-03-12T06:00:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09d67fc640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f09cc0a1c20 0x7f09cc0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09e9fc8740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31eb4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09e9fc8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31eb4bd0 0x559d31ebd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09e9fc8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31ebd7b0 0x559d31ebfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09e9fc8740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d31e23280 con 0x559d31eb42b0 2024-03-12T06:00:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09e9fc8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d31e207b0 con 0x559d31eb4bd0 2024-03-12T06:00:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09e9fc8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d31e1bf90 con 0x559d31ebd7b0 2024-03-12T06:00:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09e5991640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f09cc0a1c20 0x7f09cc0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09d67fc640 1 -- 172.21.15.76:0/870541373 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f09dc014070 con 0x7f09e0100260 2024-03-12T06:00:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09e498f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31eb4bd0 0x559d31ebd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09e5991640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31ebd7b0 0x559d31ebfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09e498f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31eb4bd0 0x559d31ebd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:38676/0 (socket says 172.21.15.76:38676) 2024-03-12T06:00:33.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.273+0000 7f09e498f640 1 -- 172.21.15.76:0/2820505021 learned_addr learned my addr 172.21.15.76:0/2820505021 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:33.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.274+0000 7f09e5190640 1 --2- 172.21.15.76:0/2820505021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31eb4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.274+0000 7f09e498f640 1 -- 172.21.15.76:0/2820505021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31ebd7b0 msgr2=0x559d31ebfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.274+0000 7f09e498f640 1 --2- 172.21.15.76:0/2820505021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31ebd7b0 0x559d31ebfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.274+0000 7f09e498f640 1 -- 172.21.15.76:0/2820505021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 msgr2=0x559d31eb4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.274+0000 7f09e498f640 1 --2- 172.21.15.76:0/2820505021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31eb4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.274+0000 7f09e498f640 1 -- 172.21.15.76:0/2820505021 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d31dda570 con 0x559d31eb4bd0 2024-03-12T06:00:33.287 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.274+0000 7f09e5991640 1 --2- 172.21.15.76:0/2820505021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31ebd7b0 0x559d31ebfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:00:33.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.274+0000 7f09e5991640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f09cc0a1c20 0x7f09cc0a40e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f09d8005fd0 tx=0x7f09d8009b00 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:33.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.274+0000 7f09e498f640 1 --2- 172.21.15.76:0/2820505021 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31eb4bd0 0x559d31ebd270 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f09e006a920 tx=0x7f09d002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5cf67c3901043a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:33.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.274+0000 7f09cb7fe640 1 -- 172.21.15.76:0/2820505021 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d002c7e0 con 0x559d31eb4bd0 2024-03-12T06:00:33.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.274+0000 7f09cb7fe640 1 -- 172.21.15.76:0/2820505021 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09d002c940 con 0x559d31eb4bd0 2024-03-12T06:00:33.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.274+0000 7f09cb7fe640 1 -- 172.21.15.76:0/2820505021 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0005420 con 0x559d31eb4bd0 2024-03-12T06:00:33.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.275+0000 7f09e9fc8740 1 -- 172.21.15.76:0/2820505021 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31eb4bd0 msgr2=0x559d31ebd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.275+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/2820505021 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31eb4bd0 0x559d31ebd270 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f09e006a920 tx=0x7f09d002c650 comp rx=0 tx=0).stop 2024-03-12T06:00:33.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.275+0000 7f09e9fc8740 1 -- 172.21.15.76:0/2820505021 shutdown_connections 2024-03-12T06:00:33.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.275+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/2820505021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31ebd7b0 0x559d31ebfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.275+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/2820505021 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31eb4bd0 0x559d31ebd270 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.275+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/2820505021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31eb4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.275+0000 7f09e9fc8740 1 -- 172.21.15.76:0/2820505021 >> 172.21.15.76:0/2820505021 conn(0x559d31eae990 msgr2=0x559d31eaed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:33.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.275+0000 7f09e9fc8740 1 -- 172.21.15.76:0/2820505021 shutdown_connections 2024-03-12T06:00:33.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.275+0000 7f09e9fc8740 1 -- 172.21.15.76:0/2820505021 wait complete. 2024-03-12T06:00:33.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.275+0000 7f09e9fc8740 1 -- start start 2024-03-12T06:00:33.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e9fc8740 1 Processor -- start 2024-03-12T06:00:33.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e9fc8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31eb42b0 0x559d31ebd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e9fc8740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb4bd0 0x559d31eba4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e9fc8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31ebd7b0 0x559d31ebaa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e9fc8740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d31aef430 con 0x559d31eb4bd0 2024-03-12T06:00:33.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e9fc8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d31aabe00 con 0x559d31eb42b0 2024-03-12T06:00:33.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e9fc8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d31da0fd0 con 0x559d31ebd7b0 2024-03-12T06:00:33.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e9fc8740 10 client.0 _ll_register_callbacks cb 0x559d31df6fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:33.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e498f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb4bd0 0x559d31eba4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e498f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb4bd0 0x559d31eba4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58238/0 (socket says 172.21.15.76:58238) 2024-03-12T06:00:33.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e5190640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31eb42b0 0x559d31ebd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e498f640 1 -- 172.21.15.76:0/1886774823 learned_addr learned my addr 172.21.15.76:0/1886774823 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:33.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e498f640 1 -- 172.21.15.76:0/1886774823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31ebd7b0 msgr2=0x559d31ebaa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e5991640 1 --2- 172.21.15.76:0/1886774823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31ebd7b0 0x559d31ebaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e498f640 1 --2- 172.21.15.76:0/1886774823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31ebd7b0 0x559d31ebaa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e498f640 1 -- 172.21.15.76:0/1886774823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31eb42b0 msgr2=0x559d31ebd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e498f640 1 --2- 172.21.15.76:0/1886774823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31eb42b0 0x559d31ebd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e498f640 1 -- 172.21.15.76:0/1886774823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e010d710 con 0x559d31eb4bd0 2024-03-12T06:00:33.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e5190640 1 --2- 172.21.15.76:0/1886774823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31eb42b0 0x559d31ebd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:00:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.276+0000 7f09e5991640 1 --2- 172.21.15.76:0/1886774823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31ebd7b0 0x559d31ebaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:00:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.277+0000 7f09e498f640 1 --2- 172.21.15.76:0/1886774823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb4bd0 0x559d31eba4f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f09d0004c50 tx=0x7f09d002c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96b1172c2e4229 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.277+0000 7f09c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.277+0000 7f09c8ff9640 1 -- 172.21.15.76:0/1886774823 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d004a650 con 0x559d31eb4bd0 2024-03-12T06:00:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.277+0000 7f09c8ff9640 1 -- 172.21.15.76:0/1886774823 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f09d004a7b0 con 0x559d31eb4bd0 2024-03-12T06:00:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.277+0000 7f09e9fc8740 1 -- 172.21.15.76:0/1886774823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d31b1b7e0 con 0x559d31eb4bd0 2024-03-12T06:00:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.277+0000 7f09c8ff9640 1 -- 172.21.15.76:0/1886774823 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09d004a970 con 0x559d31eb4bd0 2024-03-12T06:00:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.277+0000 7f09e9fc8740 10 client.16659 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:33.292 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.277+0000 7f09e9fc8740 1 -- 172.21.15.76:0/1886774823 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559d31e1a610 con 0x559d31eb4bd0 2024-03-12T06:00:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.278+0000 7f09c8ff9640 1 -- 172.21.15.76:0/1886774823 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559d31b1b7e0 con 0x559d31eb4bd0 2024-03-12T06:00:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.278+0000 7f09c8ff9640 1 -- 172.21.15.76:0/1886774823 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f09d003c050 con 0x559d31eb4bd0 2024-03-12T06:00:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.278+0000 7f09e9fc8740 10 client.16659 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.278+0000 7f09e9fc8740 10 client.16659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:00:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.278+0000 7f09e9fc8740 20 client.16659 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.278+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/1886774823 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x559d31ecd4e0 0x559d31ecf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.278+0000 7f09e9fc8740 4 client.16659 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:00:33.293 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.278+0000 7f09e9fc8740 1 -- 172.21.15.76:0/1886774823 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d31b58720 con 0x559d31ecd4e0 2024-03-12T06:00:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.278+0000 7f09e5190640 1 --2- 172.21.15.76:0/1886774823 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x559d31ecd4e0 0x559d31ecf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.278+0000 7f09e5190640 1 --2- 172.21.15.76:0/1886774823 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x559d31ecd4e0 0x559d31ecf900 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60d6e8c4773d403b server_cookie=acee26f45ac91ec0 in_seq=0 out_seq=0 2024-03-12T06:00:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.278+0000 7f09c8ff9640 10 client.16659 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.279+0000 7f09c8ff9640 1 -- 172.21.15.76:0/1886774823 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559d31b58720 con 0x559d31ecd4e0 2024-03-12T06:00:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.279+0000 7f09c8ff9640 10 client.16659 handle_command_reply: tid=0 2024-03-12T06:00:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.279+0000 7f09e9fc8740 1 client.16659 shutdown 2024-03-12T06:00:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.279+0000 7f09e9fc8740 1 -- 172.21.15.76:0/1886774823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb4bd0 msgr2=0x559d31eba4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.279+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/1886774823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb4bd0 0x559d31eba4f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f09d0004c50 tx=0x7f09d002c2b0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.279+0000 7f09e9fc8740 1 -- 172.21.15.76:0/1886774823 shutdown_connections 2024-03-12T06:00:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.279+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/1886774823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31ebd7b0 0x559d31ebaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.279+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/1886774823 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x559d31ecd4e0 0x559d31ecf900 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.279+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/1886774823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb4bd0 0x559d31eba4f0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.279+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/1886774823 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31eb42b0 0x559d31ebd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.279+0000 7f09e9fc8740 1 -- 172.21.15.76:0/1886774823 >> 172.21.15.76:0/1886774823 conn(0x559d31eae990 msgr2=0x559d31eb0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.279+0000 7f09c8ff9640 0 client.16659 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.280+0000 7f09e9fc8740 1 -- 172.21.15.76:0/1886774823 shutdown_connections 2024-03-12T06:00:33.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.280+0000 7f09e9fc8740 1 -- 172.21.15.76:0/1886774823 wait complete. 2024-03-12T06:00:33.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.280+0000 7f09e9fc8740 20 client.16659 trim_cache size 0 max 16384 2024-03-12T06:00:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.300+0000 7f09e9fc8740 1 Processor -- start 2024-03-12T06:00:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.300+0000 7f09e9fc8740 1 -- start start 2024-03-12T06:00:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.300+0000 7f09e9fc8740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31ebcc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.300+0000 7f09e9fc8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31eb4bd0 0x559d31ebd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.300+0000 7f09e9fc8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31ebd7b0 0x559d31eb8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.300+0000 7f09e9fc8740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d31eaf9e0 con 0x559d31eb42b0 2024-03-12T06:00:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.300+0000 7f09e9fc8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d31eca370 con 0x559d31ebd7b0 2024-03-12T06:00:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.300+0000 7f09e9fc8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d31eb3600 con 0x559d31eb4bd0 2024-03-12T06:00:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.301+0000 7f09e5190640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31ebcc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.301+0000 7f09e5190640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31ebcc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58250/0 (socket says 172.21.15.76:58250) 2024-03-12T06:00:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.301+0000 7f09e498f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31eb4bd0 0x559d31ebd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.301+0000 7f09e5190640 1 -- 172.21.15.76:0/3157322479 learned_addr learned my addr 172.21.15.76:0/3157322479 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:33.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.301+0000 7f09e5991640 1 --2- 172.21.15.76:0/3157322479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31ebd7b0 0x559d31eb8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.301+0000 7f09e498f640 1 -- 172.21.15.76:0/3157322479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31ebd7b0 msgr2=0x559d31eb8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.301+0000 7f09e498f640 1 --2- 172.21.15.76:0/3157322479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31ebd7b0 0x559d31eb8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.301+0000 7f09e498f640 1 -- 172.21.15.76:0/3157322479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 msgr2=0x559d31ebcc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.301+0000 7f09e498f640 1 --2- 172.21.15.76:0/3157322479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31ebcc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.301+0000 7f09e498f640 1 -- 172.21.15.76:0/3157322479 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d31db6ac0 con 0x559d31eb4bd0 2024-03-12T06:00:33.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.301+0000 7f09e5190640 1 --2- 172.21.15.76:0/3157322479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31ebcc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:00:33.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.301+0000 7f09e5991640 1 --2- 172.21.15.76:0/3157322479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31ebd7b0 0x559d31eb8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:00:33.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.301+0000 7f09e498f640 1 --2- 172.21.15.76:0/3157322479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31eb4bd0 0x559d31ebd160 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f09d002c240 tx=0x7f09d002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1366e541c30f60a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:33.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.302+0000 7f09caffd640 1 -- 172.21.15.76:0/3157322479 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0062070 con 0x559d31eb4bd0 2024-03-12T06:00:33.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.302+0000 7f09caffd640 1 -- 172.21.15.76:0/3157322479 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09d00057f0 con 0x559d31eb4bd0 2024-03-12T06:00:33.309 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.302+0000 7f09caffd640 1 -- 172.21.15.76:0/3157322479 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d003f880 con 0x559d31eb4bd0 2024-03-12T06:00:33.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.303+0000 7f09e9fc8740 1 -- 172.21.15.76:0/3157322479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31eb4bd0 msgr2=0x559d31ebd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.303+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/3157322479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31eb4bd0 0x559d31ebd160 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f09d002c240 tx=0x7f09d002bc30 comp rx=0 tx=0).stop 2024-03-12T06:00:33.313 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.303+0000 7f09e9fc8740 1 -- 172.21.15.76:0/3157322479 shutdown_connections 2024-03-12T06:00:33.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.303+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/3157322479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31ebd7b0 0x559d31eb8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.303+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/3157322479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31eb4bd0 0x559d31ebd160 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.303+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/3157322479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31ebcc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.303+0000 7f09e9fc8740 1 -- 172.21.15.76:0/3157322479 >> 172.21.15.76:0/3157322479 conn(0x559d31eae990 msgr2=0x559d31eb57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:33.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.303+0000 7f09e9fc8740 1 -- 172.21.15.76:0/3157322479 shutdown_connections 2024-03-12T06:00:33.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.303+0000 7f09e9fc8740 1 -- 172.21.15.76:0/3157322479 wait complete. 2024-03-12T06:00:33.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.303+0000 7f09e9fc8740 1 -- start start 2024-03-12T06:00:33.314 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.303+0000 7f09e9fc8740 1 Processor -- start 2024-03-12T06:00:33.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.303+0000 7f09e9fc8740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31ec46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.303+0000 7f09e9fc8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31eb4bd0 0x559d31ec0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.303+0000 7f09e9fc8740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31ebd7b0 0x559d31ec10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e9fc8740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d31ec1610 con 0x559d31eb42b0 2024-03-12T06:00:33.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e5190640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31ec46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e9fc8740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d31ec1780 con 0x559d31ebd7b0 2024-03-12T06:00:33.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e9fc8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d31ec1940 con 0x559d31eb4bd0 2024-03-12T06:00:33.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e5190640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31ec46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58256/0 (socket says 172.21.15.76:58256) 2024-03-12T06:00:33.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e5190640 1 -- 172.21.15.76:0/4238216732 learned_addr learned my addr 172.21.15.76:0/4238216732 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:00:33.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e5991640 1 --2- 172.21.15.76:0/4238216732 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31ebd7b0 0x559d31ec10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e9fc8740 10 client.0 _ll_register_callbacks cb 0x7f09d80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:00:33.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e498f640 1 --2- 172.21.15.76:0/4238216732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31eb4bd0 0x559d31ec0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e5190640 1 -- 172.21.15.76:0/4238216732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31eb4bd0 msgr2=0x559d31ec0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e5190640 1 --2- 172.21.15.76:0/4238216732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31eb4bd0 0x559d31ec0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e5190640 1 -- 172.21.15.76:0/4238216732 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31ebd7b0 msgr2=0x559d31ec10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:00:33.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e5190640 1 --2- 172.21.15.76:0/4238216732 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31ebd7b0 0x559d31ec10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e5190640 1 -- 172.21.15.76:0/4238216732 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e01a6820 con 0x559d31eb42b0 2024-03-12T06:00:33.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e5991640 1 --2- 172.21.15.76:0/4238216732 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31ebd7b0 0x559d31ec10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:00:33.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e5190640 1 --2- 172.21.15.76:0/4238216732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31ec46d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f09dc09a840 tx=0x7f09dc09a520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722f3d95ae2d0b15 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:00:33.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:00:33.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09c8ff9640 1 -- 172.21.15.76:0/4238216732 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc094170 con 0x559d31eb42b0 2024-03-12T06:00:33.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09c8ff9640 1 -- 172.21.15.76:0/4238216732 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f09dc096f20 con 0x559d31eb42b0 2024-03-12T06:00:33.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.304+0000 7f09e9fc8740 1 -- 172.21.15.76:0/4238216732 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d31ba7890 con 0x559d31eb42b0 2024-03-12T06:00:33.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.305+0000 7f09c8ff9640 1 -- 172.21.15.76:0/4238216732 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09dc0970e0 con 0x559d31eb42b0 2024-03-12T06:00:33.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.305+0000 7f09e9fc8740 10 client.16665 fetch_fsmap learned FSMap version 25 2024-03-12T06:00:33.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.305+0000 7f09e9fc8740 1 -- 172.21.15.76:0/4238216732 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559d31e1a610 con 0x559d31eb42b0 2024-03-12T06:00:33.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.305+0000 7f09c8ff9640 1 -- 172.21.15.76:0/4238216732 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559d31ba7890 con 0x559d31eb42b0 2024-03-12T06:00:33.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.305+0000 7f09c8ff9640 1 -- 172.21.15.76:0/4238216732 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 25) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f09dc0b3200 con 0x559d31eb42b0 2024-03-12T06:00:33.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.305+0000 7f09e9fc8740 10 client.16665 fetch_fsmap finished waiting for FSMap version 25 2024-03-12T06:00:33.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.305+0000 7f09e9fc8740 10 client.16665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:00:33.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.305+0000 7f09e9fc8740 20 client.16665 populate_metadata read hostname 'smithi076' 2024-03-12T06:00:33.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.305+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/4238216732 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x559d31f24f30 0x559d31f27350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:00:33.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.305+0000 7f09e9fc8740 4 client.16665 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:00:33.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.305+0000 7f09e9fc8740 1 -- 172.21.15.76:0/4238216732 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559d31b0a4e0 con 0x559d31f24f30 2024-03-12T06:00:33.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.305+0000 7f09e498f640 1 --2- 172.21.15.76:0/4238216732 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x559d31f24f30 0x559d31f27350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:00:33.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09c8ff9640 10 client.16665 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:33.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09e498f640 1 --2- 172.21.15.76:0/4238216732 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x559d31f24f30 0x559d31f27350 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a2f981bcc995d24 server_cookie=e27d9d75bfe65eec in_seq=0 out_seq=0 2024-03-12T06:00:33.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09c8ff9640 1 -- 172.21.15.76:0/4238216732 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x559d31b0a4e0 con 0x559d31f24f30 2024-03-12T06:00:33.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09c8ff9640 10 client.16665 handle_command_reply: tid=0 2024-03-12T06:00:33.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09e9fc8740 1 client.16665 shutdown 2024-03-12T06:00:33.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09e9fc8740 1 -- 172.21.15.76:0/4238216732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 msgr2=0x559d31ec46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/4238216732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31ec46d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f09dc09a840 tx=0x7f09dc09a520 comp rx=0 tx=0).stop 2024-03-12T06:00:33.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09e9fc8740 1 -- 172.21.15.76:0/4238216732 shutdown_connections 2024-03-12T06:00:33.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/4238216732 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559d31ebd7b0 0x559d31ec10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/4238216732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d31eb4bd0 0x559d31ec0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/4238216732 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x559d31f24f30 0x559d31f27350 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09e9fc8740 1 --2- 172.21.15.76:0/4238216732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559d31eb42b0 0x559d31ec46d0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09e9fc8740 1 -- 172.21.15.76:0/4238216732 >> 172.21.15.76:0/4238216732 conn(0x559d31eae990 msgr2=0x559d31eb9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:33.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09c8ff9640 0 client.16665 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:00:33.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09e9fc8740 1 -- 172.21.15.76:0/4238216732 shutdown_connections 2024-03-12T06:00:33.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09e9fc8740 1 -- 172.21.15.76:0/4238216732 wait complete. 2024-03-12T06:00:33.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.306+0000 7f09e9fc8740 20 client.16665 trim_cache size 0 max 16384 2024-03-12T06:00:33.321 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:00:33.321 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T06:00:33.321 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T06:00:33.322 INFO:teuthology.orchestra.run.smithi076.stdout: "af40f471-da39-4bae-ad45-1b9f97b9db2e": { 2024-03-12T06:00:33.322 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T06:00:33.322 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "af40f471-da39-4bae-ad45-1b9f97b9db2e", 2024-03-12T06:00:33.322 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T06:00:33.322 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:00:33.322 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:00:33.322 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:00:33.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.307+0000 7f09e73f4640 1 -- 172.21.15.76:0/870541373 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f09cc0a1c20 msgr2=0x7f09cc0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.307+0000 7f09e73f4640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f09cc0a1c20 0x7f09cc0a40e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f09d8005fd0 tx=0x7f09d8009b00 comp rx=0 tx=0).stop 2024-03-12T06:00:33.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.307+0000 7f09e73f4640 1 -- 172.21.15.76:0/870541373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09e0100260 msgr2=0x7f09e019f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:00:33.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.307+0000 7f09e73f4640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09e0100260 0x7f09e019f670 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00ea60 tx=0x7f09dc00ef20 comp rx=0 tx=0).stop 2024-03-12T06:00:33.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.307+0000 7f09e73f4640 1 -- 172.21.15.76:0/870541373 shutdown_connections 2024-03-12T06:00:33.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.307+0000 7f09e73f4640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f09cc0a1c20 0x7f09cc0a40e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.307+0000 7f09e73f4640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09e01997e0 0x7f09e0199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.307+0000 7f09e73f4640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f09e0105330 0x7f09e019fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.307+0000 7f09e73f4640 1 --2- 172.21.15.76:0/870541373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f09e0100260 0x7f09e019f670 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:00:33.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.307+0000 7f09e73f4640 1 -- 172.21.15.76:0/870541373 >> 172.21.15.76:0/870541373 conn(0x7f09e0077180 msgr2=0x7f09e010bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:00:33.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.307+0000 7f09e73f4640 1 -- 172.21.15.76:0/870541373 shutdown_connections 2024-03-12T06:00:33.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:00:33.307+0000 7f09e73f4640 1 -- 172.21.15.76:0/870541373 wait complete. 2024-03-12T06:00:33.325 INFO:tasks.cephfs.filesystem:scrub status for tag:af40f471-da39-4bae-ad45-1b9f97b9db2e - {'path': '/', 'tag': 'af40f471-da39-4bae-ad45-1b9f97b9db2e', 'options': 'recursive,force'} 2024-03-12T06:00:34.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:34 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T06:00:34.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:34 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:34.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:34 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:00:34.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:34 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:00:34.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:34 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:00:34.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:34 smithi082 ceph-mon[36382]: pgmap v146: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:34.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:34 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:34.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:34 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:34.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:34 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:34 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T06:00:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:34 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:34 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:00:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:34 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:00:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:34 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:00:34.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:34 smithi088 ceph-mon[36585]: pgmap v146: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:34.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:34 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:34.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:34 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:34.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:34 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:34.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:34 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T06:00:34.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:34 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:34.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:34 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:00:34.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:34 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:00:34.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:34 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:00:34.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:34 smithi076 ceph-mon[32095]: pgmap v146: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:34.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:34 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:34.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:34 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:34.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:34 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:00:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:35 smithi082 ceph-mon[36382]: pgmap v147: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:35.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:35 smithi088 ceph-mon[36585]: pgmap v147: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:35.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:35 smithi076 ceph-mon[32095]: pgmap v147: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:38.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:38 smithi082 ceph-mon[36382]: pgmap v148: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:38 smithi088 ceph-mon[36585]: pgmap v148: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:38.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:38 smithi076 ceph-mon[32095]: pgmap v148: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:40.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:40 smithi082 ceph-mon[36382]: pgmap v149: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:00:40.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:40 smithi088 ceph-mon[36585]: pgmap v149: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:00:40.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:40 smithi076 ceph-mon[32095]: pgmap v149: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:00:42.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:42 smithi082 ceph-mon[36382]: pgmap v150: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:00:42.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:42 smithi088 ceph-mon[36585]: pgmap v150: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:00:42.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:42 smithi076 ceph-mon[32095]: pgmap v150: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:00:44.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:44 smithi082 ceph-mon[36382]: pgmap v151: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:00:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:44 smithi088 ceph-mon[36585]: pgmap v151: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:00:44.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:44 smithi076 ceph-mon[32095]: pgmap v151: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:00:46.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:46 smithi082 ceph-mon[36382]: pgmap v152: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:00:46.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:46 smithi088 ceph-mon[36585]: pgmap v152: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:00:46.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:46 smithi076 ceph-mon[32095]: pgmap v152: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:00:48.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:48 smithi082 ceph-mon[36382]: pgmap v153: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:00:48.467 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:48 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:00:48.467 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:48 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:00:48.467 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:48 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:00:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:48 smithi088 ceph-mon[36585]: pgmap v153: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:00:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:48 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:00:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:48 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:00:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:48 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:00:48.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:48 smithi076 ceph-mon[32095]: pgmap v153: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:00:48.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:48 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:00:48.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:48 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:00:48.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:48 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:00:49.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:49 smithi082 ceph-mon[36382]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T06:00:49.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:49 smithi088 ceph-mon[36585]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T06:00:49.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:49 smithi076 ceph-mon[32095]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T06:00:50.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:50 smithi082 ceph-mon[36382]: pgmap v154: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:00:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:50 smithi088 ceph-mon[36585]: pgmap v154: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:00:50.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:50 smithi076 ceph-mon[32095]: pgmap v154: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:00:51.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:51 smithi082 ceph-mon[36382]: pgmap v155: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:51.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:51 smithi088 ceph-mon[36585]: pgmap v155: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:51.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:51 smithi076 ceph-mon[32095]: pgmap v155: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:53 smithi088 ceph-mon[36585]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T06:00:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:53 smithi088 ceph-mon[36585]: pgmap v156: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:53.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:53 smithi076 ceph-mon[32095]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T06:00:53.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:53 smithi076 ceph-mon[32095]: pgmap v156: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:53.695 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:53 smithi082 ceph-mon[36382]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T06:00:53.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:53 smithi082 ceph-mon[36382]: pgmap v156: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:00:56.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:56 smithi082 ceph-mon[36382]: pgmap v157: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:56.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:56 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:00:56.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:56 smithi088 ceph-mon[36585]: pgmap v157: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:56.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:56 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:00:56.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:56 smithi076 ceph-mon[32095]: pgmap v157: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:56.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:56 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:00:57.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:57 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:00:57.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:57 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:00:57.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:57 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:00:57.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:57 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:00:57.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:57 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:00:57.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:57 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:00:57.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:57 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:00:57.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:57 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:00:57.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:57 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:00:58.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:00:58 smithi082 ceph-mon[36382]: pgmap v158: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:58.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:00:58 smithi088 ceph-mon[36585]: pgmap v158: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:00:58.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:00:58 smithi076 ceph-mon[32095]: pgmap v158: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:00.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:00 smithi082 ceph-mon[36382]: pgmap v159: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:00.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:00 smithi088 ceph-mon[36585]: pgmap v159: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:00.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:00 smithi076 ceph-mon[32095]: pgmap v159: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:02.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:02 smithi082 ceph-mon[36382]: pgmap v160: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:02.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:02 smithi088 ceph-mon[36585]: pgmap v160: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:02.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:02 smithi076 ceph-mon[32095]: pgmap v160: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:03.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:01:03.326 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:01:03.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.436+0000 7fc80b5e9640 1 -- 172.21.15.76:0/1708347095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc804100c10 msgr2=0x7fc804104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.436+0000 7fc80b5e9640 1 --2- 172.21.15.76:0/1708347095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc804100c10 0x7fc804104c50 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc800009930 tx=0x7fc80002bc30 comp rx=0 tx=0).stop 2024-03-12T06:01:03.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.436+0000 7fc80b5e9640 1 -- 172.21.15.76:0/1708347095 shutdown_connections 2024-03-12T06:01:03.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.436+0000 7fc80b5e9640 1 --2- 172.21.15.76:0/1708347095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc804105330 0x7fc80410cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.436+0000 7fc80b5e9640 1 --2- 172.21.15.76:0/1708347095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc804100c10 0x7fc804104c50 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.436+0000 7fc80b5e9640 1 --2- 172.21.15.76:0/1708347095 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc804100260 0x7fc804100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.436+0000 7fc80b5e9640 1 -- 172.21.15.76:0/1708347095 >> 172.21.15.76:0/1708347095 conn(0x7fc804077180 msgr2=0x7fc804077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:03.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.436+0000 7fc80b5e9640 1 -- 172.21.15.76:0/1708347095 shutdown_connections 2024-03-12T06:01:03.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.436+0000 7fc80b5e9640 1 -- 172.21.15.76:0/1708347095 wait complete. 2024-03-12T06:01:03.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.436+0000 7fc80b5e9640 1 Processor -- start 2024-03-12T06:01:03.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.436+0000 7fc80b5e9640 1 -- start start 2024-03-12T06:01:03.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc80b5e9640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc804100260 0x7fc804196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc80b5e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc804105330 0x7fc804196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc80b5e9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc80419af50 0x7fc80419b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc80b5e9640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80419b940 con 0x7fc804105330 2024-03-12T06:01:03.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc80b5e9640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80419bab0 con 0x7fc804100260 2024-03-12T06:01:03.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc80b5e9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80419bc20 con 0x7fc80419af50 2024-03-12T06:01:03.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc808b84640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc804105330 0x7fc804196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc808b84640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc804105330 0x7fc804196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38034/0 (socket says 172.21.15.76:38034) 2024-03-12T06:01:03.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc808b84640 1 -- 172.21.15.76:0/1298781158 learned_addr learned my addr 172.21.15.76:0/1298781158 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:03.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc809385640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc804100260 0x7fc804196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc809b86640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc80419af50 0x7fc80419b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc808b84640 1 -- 172.21.15.76:0/1298781158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc80419af50 msgr2=0x7fc80419b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.446 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc808b84640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc80419af50 0x7fc80419b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc808b84640 1 -- 172.21.15.76:0/1298781158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc804100260 msgr2=0x7fc804196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc808b84640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc804100260 0x7fc804196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc808b84640 1 -- 172.21.15.76:0/1298781158 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc800009560 con 0x7fc804105330 2024-03-12T06:01:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc809b86640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc80419af50 0x7fc80419b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc808b84640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc804105330 0x7fc804196bd0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc800009230 tx=0x7fc80002c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc7fa7fc640 1 -- 172.21.15.76:0/1298781158 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc800004350 con 0x7fc804105330 2024-03-12T06:01:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc7fa7fc640 1 -- 172.21.15.76:0/1298781158 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8000044b0 con 0x7fc804105330 2024-03-12T06:01:03.447 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc80b5e9640 1 -- 172.21.15.76:0/1298781158 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8041a3200 con 0x7fc804105330 2024-03-12T06:01:03.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.437+0000 7fc7fa7fc640 1 -- 172.21.15.76:0/1298781158 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc800003da0 con 0x7fc804105330 2024-03-12T06:01:03.448 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.438+0000 7fc80b5e9640 1 -- 172.21.15.76:0/1298781158 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8041a3680 con 0x7fc804105330 2024-03-12T06:01:03.454 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.440+0000 7fc7fa7fc640 1 -- 172.21.15.76:0/1298781158 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc800004df0 con 0x7fc804105330 2024-03-12T06:01:03.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc80e1bf740 1 Processor -- start 2024-03-12T06:01:03.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc80e1bf740 1 -- start start 2024-03-12T06:01:03.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc7fa7fc640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc7f00a1ba0 0x7fc7f00a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc80e1bf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d5b10 0x55dff84d5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc80e1bf740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d6430 0x55dff84e3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc80e1bf740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84e3db0 0x55dff84e61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc80e1bf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff8445280 con 0x55dff84d5b10 2024-03-12T06:01:03.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc80e1bf740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff84427b0 con 0x55dff84d6430 2024-03-12T06:01:03.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc80e1bf740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff843df90 con 0x55dff84e3db0 2024-03-12T06:01:03.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc809385640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d5b10 0x55dff84d5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc809b86640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc7f00a1ba0 0x7fc7f00a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc809385640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d5b10 0x55dff84d5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38050/0 (socket says 172.21.15.76:38050) 2024-03-12T06:01:03.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc809385640 1 -- 172.21.15.76:0/2857940056 learned_addr learned my addr 172.21.15.76:0/2857940056 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:03.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc7fa7fc640 1 -- 172.21.15.76:0/1298781158 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc8000ef860 con 0x7fc804105330 2024-03-12T06:01:03.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.441+0000 7fc808b84640 1 --2- 172.21.15.76:0/2857940056 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d6430 0x55dff84e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.442+0000 7fc809385640 1 -- 172.21.15.76:0/2857940056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84e3db0 msgr2=0x55dff84e61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.442+0000 7fc809385640 1 --2- 172.21.15.76:0/2857940056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84e3db0 0x55dff84e61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.442+0000 7fc809385640 1 -- 172.21.15.76:0/2857940056 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d6430 msgr2=0x55dff84e3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.442+0000 7fc809385640 1 --2- 172.21.15.76:0/2857940056 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d6430 0x55dff84e3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.442+0000 7fc809385640 1 -- 172.21.15.76:0/2857940056 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dff83fc570 con 0x55dff84d5b10 2024-03-12T06:01:03.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.442+0000 7fc808b84640 1 --2- 172.21.15.76:0/2857940056 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d6430 0x55dff84e3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:03.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.442+0000 7fc809b86640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc7f00a1ba0 0x7fc7f00a4060 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc0094d0 tx=0x7fc7fc009ab0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:03.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.442+0000 7fc809385640 1 --2- 172.21.15.76:0/2857940056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d5b10 0x55dff84d5ef0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4009cf0 tx=0x7fc7f4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a176c4aa72fa89f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:03.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.442+0000 7fc7ef7fe640 1 -- 172.21.15.76:0/2857940056 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4009ea0 con 0x55dff84d5b10 2024-03-12T06:01:03.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.442+0000 7fc7ef7fe640 1 -- 172.21.15.76:0/2857940056 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7f4004220 con 0x55dff84d5b10 2024-03-12T06:01:03.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.442+0000 7fc7ef7fe640 1 -- 172.21.15.76:0/2857940056 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f40187d0 con 0x55dff84d5b10 2024-03-12T06:01:03.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.443+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2857940056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d5b10 msgr2=0x55dff84d5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.443+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/2857940056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d5b10 0x55dff84d5ef0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4009cf0 tx=0x7fc7f4011810 comp rx=0 tx=0).stop 2024-03-12T06:01:03.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.443+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2857940056 shutdown_connections 2024-03-12T06:01:03.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.443+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/2857940056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84e3db0 0x55dff84e61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.443+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/2857940056 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d6430 0x55dff84e3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.458 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.443+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/2857940056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d5b10 0x55dff84d5ef0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.443+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2857940056 >> 172.21.15.76:0/2857940056 conn(0x55dff84d0990 msgr2=0x55dff84d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:03.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.443+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2857940056 shutdown_connections 2024-03-12T06:01:03.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.443+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2857940056 wait complete. 2024-03-12T06:01:03.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.443+0000 7fc80e1bf740 1 -- start start 2024-03-12T06:01:03.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.443+0000 7fc80e1bf740 1 Processor -- start 2024-03-12T06:01:03.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc80e1bf740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d5b10 0x55dff84e3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc80e1bf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d6430 0x55dff84e0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.459 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc80e1bf740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84e3db0 0x55dff84e1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc80e1bf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff8111430 con 0x55dff84d6430 2024-03-12T06:01:03.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc80e1bf740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff80cde00 con 0x55dff84d5b10 2024-03-12T06:01:03.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc80e1bf740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff83c2fd0 con 0x55dff84e3db0 2024-03-12T06:01:03.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc80e1bf740 10 client.0 _ll_register_callbacks cb 0x55dff8418fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:03.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc808b84640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d6430 0x55dff84e0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc808b84640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d6430 0x55dff84e0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38066/0 (socket says 172.21.15.76:38066) 2024-03-12T06:01:03.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc808b84640 1 -- 172.21.15.76:0/2683273937 learned_addr learned my addr 172.21.15.76:0/2683273937 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:03.460 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc809385640 1 --2- 172.21.15.76:0/2683273937 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d5b10 0x55dff84e3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc809b86640 1 --2- 172.21.15.76:0/2683273937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84e3db0 0x55dff84e1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc808b84640 1 -- 172.21.15.76:0/2683273937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84e3db0 msgr2=0x55dff84e1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc808b84640 1 --2- 172.21.15.76:0/2683273937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84e3db0 0x55dff84e1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc808b84640 1 -- 172.21.15.76:0/2683273937 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d5b10 msgr2=0x55dff84e3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc808b84640 1 --2- 172.21.15.76:0/2683273937 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d5b10 0x55dff84e3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc808b84640 1 -- 172.21.15.76:0/2683273937 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f4009950 con 0x55dff84d6430 2024-03-12T06:01:03.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc809385640 1 --2- 172.21.15.76:0/2683273937 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d5b10 0x55dff84e3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:03.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc808b84640 1 --2- 172.21.15.76:0/2683273937 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d6430 0x55dff84e0b40 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc8000b8fa0 tx=0x7fc8000bc460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3990b919b2c8827 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:03.461 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc7ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:03.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc7ecff9640 1 -- 172.21.15.76:0/2683273937 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8000bc5d0 con 0x55dff84d6430 2024-03-12T06:01:03.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc7ecff9640 1 -- 172.21.15.76:0/2683273937 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc8000c0720 con 0x55dff84d6430 2024-03-12T06:01:03.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2683273937 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dff813d7e0 con 0x55dff84d6430 2024-03-12T06:01:03.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc7ecff9640 1 -- 172.21.15.76:0/2683273937 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8000d5140 con 0x55dff84d6430 2024-03-12T06:01:03.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc80e1bf740 10 client.16677 fetch_fsmap learned FSMap version 27 2024-03-12T06:01:03.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.444+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2683273937 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55dff843c610 con 0x55dff84d6430 2024-03-12T06:01:03.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.445+0000 7fc7ecff9640 1 -- 172.21.15.76:0/2683273937 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55dff813d7e0 con 0x55dff84d6430 2024-03-12T06:01:03.462 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.445+0000 7fc7ecff9640 1 -- 172.21.15.76:0/2683273937 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 27) v1 ==== 7886+0+0 (secure 0 0 0) 0x7fc80003c070 con 0x55dff84d6430 2024-03-12T06:01:03.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.445+0000 7fc80e1bf740 10 client.16677 fetch_fsmap finished waiting for FSMap version 27 2024-03-12T06:01:03.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.445+0000 7fc80e1bf740 10 client.16677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:03.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.445+0000 7fc80e1bf740 20 client.16677 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:03.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.445+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/2683273937 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55dff84f3b70 0x55dff84f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.445+0000 7fc80e1bf740 4 client.16677 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:01:03.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.445+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2683273937 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dff817a720 con 0x55dff84f3b70 2024-03-12T06:01:03.463 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.445+0000 7fc809385640 1 --2- 172.21.15.76:0/2683273937 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55dff84f3b70 0x55dff84f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.445+0000 7fc7ecff9640 10 client.16677 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:03.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.445+0000 7fc809385640 1 --2- 172.21.15.76:0/2683273937 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55dff84f3b70 0x55dff84f5fa0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c3464d636bfaa8f server_cookie=805958cc52465e2a in_seq=0 out_seq=0 2024-03-12T06:01:03.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.446+0000 7fc7ecff9640 1 -- 172.21.15.76:0/2683273937 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55dff817a720 con 0x55dff84f3b70 2024-03-12T06:01:03.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.446+0000 7fc7ecff9640 10 client.16677 handle_command_reply: tid=0 2024-03-12T06:01:03.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.446+0000 7fc80e1bf740 1 client.16677 shutdown 2024-03-12T06:01:03.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.446+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2683273937 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d6430 msgr2=0x55dff84e0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.446+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/2683273937 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d6430 0x55dff84e0b40 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc8000b8fa0 tx=0x7fc8000bc460 comp rx=0 tx=0).stop 2024-03-12T06:01:03.464 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.446+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2683273937 shutdown_connections 2024-03-12T06:01:03.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.446+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/2683273937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84e3db0 0x55dff84e1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.446+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/2683273937 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55dff84f3b70 0x55dff84f5fa0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.446+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/2683273937 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d6430 0x55dff84e0b40 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.446+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/2683273937 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d5b10 0x55dff84e3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.446+0000 7fc7ecff9640 0 client.16677 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:03.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.446+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2683273937 >> 172.21.15.76:0/2683273937 conn(0x55dff84d0990 msgr2=0x55dff84d2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:03.465 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.446+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2683273937 shutdown_connections 2024-03-12T06:01:03.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.446+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2683273937 wait complete. 2024-03-12T06:01:03.466 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.447+0000 7fc80e1bf740 20 client.16677 trim_cache size 0 max 16384 2024-03-12T06:01:03.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc80e1bf740 1 Processor -- start 2024-03-12T06:01:03.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc80e1bf740 1 -- start start 2024-03-12T06:01:03.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc80e1bf740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d5b10 0x55dff84e3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc80e1bf740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84d6430 0x55dff84eaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc80e1bf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84e3db0 0x55dff84eb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc80e1bf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff84cc0f0 con 0x55dff84e3db0 2024-03-12T06:01:03.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc80e1bf740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff84d19e0 con 0x55dff84d5b10 2024-03-12T06:01:03.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc80e1bf740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff84cc400 con 0x55dff84d6430 2024-03-12T06:01:03.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc809b86640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84e3db0 0x55dff84eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.474 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc809b86640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84e3db0 0x55dff84eb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38082/0 (socket says 172.21.15.76:38082) 2024-03-12T06:01:03.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc809b86640 1 -- 172.21.15.76:0/2514248863 learned_addr learned my addr 172.21.15.76:0/2514248863 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:03.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc809385640 1 --2- 172.21.15.76:0/2514248863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d5b10 0x55dff84e3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc809b86640 1 -- 172.21.15.76:0/2514248863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84d6430 msgr2=0x55dff84eaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc809b86640 1 --2- 172.21.15.76:0/2514248863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84d6430 0x55dff84eaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc809b86640 1 -- 172.21.15.76:0/2514248863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d5b10 msgr2=0x55dff84e3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc809b86640 1 --2- 172.21.15.76:0/2514248863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d5b10 0x55dff84e3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc809b86640 1 -- 172.21.15.76:0/2514248863 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dff83d8ac0 con 0x55dff84e3db0 2024-03-12T06:01:03.475 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.462+0000 7fc809b86640 1 --2- 172.21.15.76:0/2514248863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84e3db0 0x55dff84eb020 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc012560 tx=0x7fc7fc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e88a9456241c9a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:03.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.463+0000 7fc7eeffd640 1 -- 172.21.15.76:0/2514248863 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc027600 con 0x55dff84e3db0 2024-03-12T06:01:03.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.463+0000 7fc7eeffd640 1 -- 172.21.15.76:0/2514248863 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7fc027c40 con 0x55dff84e3db0 2024-03-12T06:01:03.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.463+0000 7fc7eeffd640 1 -- 172.21.15.76:0/2514248863 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc01fd80 con 0x55dff84e3db0 2024-03-12T06:01:03.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.463+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2514248863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84e3db0 msgr2=0x55dff84eb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.463+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/2514248863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84e3db0 0x55dff84eb020 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc012560 tx=0x7fc7fc014000 comp rx=0 tx=0).stop 2024-03-12T06:01:03.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.463+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2514248863 shutdown_connections 2024-03-12T06:01:03.476 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.463+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/2514248863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84e3db0 0x55dff84eb020 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.463+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/2514248863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84d6430 0x55dff84eaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.463+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/2514248863 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84d5b10 0x55dff84e3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.463+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2514248863 >> 172.21.15.76:0/2514248863 conn(0x55dff84d0990 msgr2=0x55dff84e4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:03.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.463+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2514248863 shutdown_connections 2024-03-12T06:01:03.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.463+0000 7fc80e1bf740 1 -- 172.21.15.76:0/2514248863 wait complete. 2024-03-12T06:01:03.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.463+0000 7fc80e1bf740 1 -- start start 2024-03-12T06:01:03.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc80e1bf740 1 Processor -- start 2024-03-12T06:01:03.477 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc80e1bf740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84d5b10 0x55dff84e9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc80e1bf740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d6430 0x55dff84ea120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc80e1bf740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84e3db0 0x55dff854a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc80e1bf740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff84ea820 con 0x55dff84d6430 2024-03-12T06:01:03.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc80e1bf740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff854a5b0 con 0x55dff84e3db0 2024-03-12T06:01:03.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc80e1bf740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff854a720 con 0x55dff84d5b10 2024-03-12T06:01:03.478 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc80e1bf740 10 client.0 _ll_register_callbacks cb 0x7fc7f40097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc808b84640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d6430 0x55dff84ea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc808b84640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d6430 0x55dff84ea120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38086/0 (socket says 172.21.15.76:38086) 2024-03-12T06:01:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc808b84640 1 -- 172.21.15.76:0/1346765263 learned_addr learned my addr 172.21.15.76:0/1346765263 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc809385640 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84d5b10 0x55dff84e9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc809b86640 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84e3db0 0x55dff854a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc808b84640 1 -- 172.21.15.76:0/1346765263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84d5b10 msgr2=0x55dff84e9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.479 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc808b84640 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84d5b10 0x55dff84e9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc808b84640 1 -- 172.21.15.76:0/1346765263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84e3db0 msgr2=0x55dff854a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc808b84640 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84e3db0 0x55dff854a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc808b84640 1 -- 172.21.15.76:0/1346765263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7fc009c60 con 0x55dff84d6430 2024-03-12T06:01:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc809385640 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84d5b10 0x55dff84e9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc809b86640 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84e3db0 0x55dff854a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc808b84640 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d6430 0x55dff84ea120 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fc800005ec0 tx=0x7fc8000c4c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ecf4eb17d31f510 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:03.480 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.464+0000 7fc7ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc7ecff9640 1 -- 172.21.15.76:0/1346765263 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8000d5140 con 0x55dff84d6430 2024-03-12T06:01:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc7ecff9640 1 -- 172.21.15.76:0/1346765263 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc8000c0720 con 0x55dff84d6430 2024-03-12T06:01:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc80e1bf740 1 -- 172.21.15.76:0/1346765263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc7f4004220 con 0x55dff84d6430 2024-03-12T06:01:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc7ecff9640 1 -- 172.21.15.76:0/1346765263 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8000b8870 con 0x55dff84d6430 2024-03-12T06:01:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc80e1bf740 10 client.16683 fetch_fsmap learned FSMap version 27 2024-03-12T06:01:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc80e1bf740 1 -- 172.21.15.76:0/1346765263 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55dff83fe6a0 con 0x55dff84d6430 2024-03-12T06:01:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc7ecff9640 1 -- 172.21.15.76:0/1346765263 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7f4004220 con 0x55dff84d6430 2024-03-12T06:01:03.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc7ecff9640 1 -- 172.21.15.76:0/1346765263 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 27) v1 ==== 7886+0+0 (secure 0 0 0) 0x7fc80003c050 con 0x55dff84d6430 2024-03-12T06:01:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc80e1bf740 10 client.16683 fetch_fsmap finished waiting for FSMap version 27 2024-03-12T06:01:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc80e1bf740 10 client.16683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc80e1bf740 20 client.16683 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55dff854ba00 0x55dff854ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc80e1bf740 4 client.16683 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:01:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc80e1bf740 1 -- 172.21.15.76:0/1346765263 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55dff81c9890 con 0x55dff854ba00 2024-03-12T06:01:03.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.465+0000 7fc809385640 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55dff854ba00 0x55dff854ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc7ecff9640 10 client.16683 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc809385640 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55dff854ba00 0x55dff854ddf0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e309fac03c367685 server_cookie=40b7618c0bee1f73 in_seq=0 out_seq=0 2024-03-12T06:01:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc7ecff9640 1 -- 172.21.15.76:0/1346765263 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55dff81c9890 con 0x55dff854ba00 2024-03-12T06:01:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc7ecff9640 10 client.16683 handle_command_reply: tid=0 2024-03-12T06:01:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc80e1bf740 1 client.16683 shutdown 2024-03-12T06:01:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc80e1bf740 1 -- 172.21.15.76:0/1346765263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d6430 msgr2=0x55dff84ea120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d6430 0x55dff84ea120 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fc800005ec0 tx=0x7fc8000c4c60 comp rx=0 tx=0).stop 2024-03-12T06:01:03.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc80e1bf740 1 -- 172.21.15.76:0/1346765263 shutdown_connections 2024-03-12T06:01:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55dff84e3db0 0x55dff854a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55dff854ba00 0x55dff854ddf0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55dff84d6430 0x55dff84ea120 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc7ecff9640 0 client.16683 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc80e1bf740 1 --2- 172.21.15.76:0/1346765263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dff84d5b10 0x55dff84e9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc80e1bf740 1 -- 172.21.15.76:0/1346765263 >> 172.21.15.76:0/1346765263 conn(0x55dff84d0990 msgr2=0x55dff84e03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc80e1bf740 1 -- 172.21.15.76:0/1346765263 shutdown_connections 2024-03-12T06:01:03.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc80e1bf740 1 -- 172.21.15.76:0/1346765263 wait complete. 2024-03-12T06:01:03.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.466+0000 7fc80e1bf740 20 client.16683 trim_cache size 0 max 16384 2024-03-12T06:01:03.485 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:01:03.485 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T06:01:03.485 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T06:01:03.485 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:01:03.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.467+0000 7fc80b5e9640 1 -- 172.21.15.76:0/1298781158 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc7f00a1ba0 msgr2=0x7fc7f00a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.467+0000 7fc80b5e9640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc7f00a1ba0 0x7fc7f00a4060 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc0094d0 tx=0x7fc7fc009ab0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.467+0000 7fc80b5e9640 1 -- 172.21.15.76:0/1298781158 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc804105330 msgr2=0x7fc804196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.467+0000 7fc80b5e9640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc804105330 0x7fc804196bd0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc800009230 tx=0x7fc80002c680 comp rx=0 tx=0).stop 2024-03-12T06:01:03.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.467+0000 7fc80b5e9640 1 -- 172.21.15.76:0/1298781158 shutdown_connections 2024-03-12T06:01:03.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.467+0000 7fc80b5e9640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc7f00a1ba0 0x7fc7f00a4060 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.467+0000 7fc80b5e9640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc80419af50 0x7fc80419b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.467+0000 7fc80b5e9640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc804105330 0x7fc804196bd0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.467+0000 7fc80b5e9640 1 --2- 172.21.15.76:0/1298781158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc804100260 0x7fc804196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.467+0000 7fc80b5e9640 1 -- 172.21.15.76:0/1298781158 >> 172.21.15.76:0/1298781158 conn(0x7fc804077180 msgr2=0x7fc804106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:03.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.467+0000 7fc80b5e9640 1 -- 172.21.15.76:0/1298781158 shutdown_connections 2024-03-12T06:01:03.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.467+0000 7fc80b5e9640 1 -- 172.21.15.76:0/1298781158 wait complete. 2024-03-12T06:01:03.489 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T06:01:03.489 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T06:01:03.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.587+0000 7f764cf02640 1 -- 172.21.15.76:0/1132593612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7648106460 msgr2=0x7f764810e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.587+0000 7f764cf02640 1 --2- 172.21.15.76:0/1132593612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7648106460 0x7f764810e030 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7640009cd0 tx=0x7f764002c0f0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.587+0000 7f764cf02640 1 -- 172.21.15.76:0/1132593612 shutdown_connections 2024-03-12T06:01:03.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.587+0000 7f764cf02640 1 --2- 172.21.15.76:0/1132593612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7648106460 0x7f764810e030 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.587+0000 7f764cf02640 1 --2- 172.21.15.76:0/1132593612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648101d40 0x7f7648105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.587+0000 7f764cf02640 1 --2- 172.21.15.76:0/1132593612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7648101390 0x7f7648101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.587+0000 7f764cf02640 1 -- 172.21.15.76:0/1132593612 >> 172.21.15.76:0/1132593612 conn(0x7f76480fcc30 msgr2=0x7f76480ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:03.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.587+0000 7f764cf02640 1 -- 172.21.15.76:0/1132593612 shutdown_connections 2024-03-12T06:01:03.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.587+0000 7f764cf02640 1 -- 172.21.15.76:0/1132593612 wait complete. 2024-03-12T06:01:03.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.588+0000 7f764cf02640 1 Processor -- start 2024-03-12T06:01:03.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.588+0000 7f764cf02640 1 -- start start 2024-03-12T06:01:03.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.588+0000 7f764cf02640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7648101390 0x7f764819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.588+0000 7f764cf02640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7648106460 0x7f764819d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.588+0000 7f764cf02640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648197600 0x7f7648197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.588+0000 7f764cf02640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7648197ff0 con 0x7f7648106460 2024-03-12T06:01:03.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.588+0000 7f764cf02640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7648198160 con 0x7f7648101390 2024-03-12T06:01:03.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.588+0000 7f764cf02640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7648198320 con 0x7f7648197600 2024-03-12T06:01:03.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.588+0000 7f7645d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7648106460 0x7f764819d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.588+0000 7f7645d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7648106460 0x7f764819d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38104/0 (socket says 172.21.15.76:38104) 2024-03-12T06:01:03.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.588+0000 7f7645d9b640 1 -- 172.21.15.76:0/3388008755 learned_addr learned my addr 172.21.15.76:0/3388008755 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:03.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.588+0000 7f764659c640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7648101390 0x7f764819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.588+0000 7f7646d9d640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648197600 0x7f7648197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.589+0000 7f7645d9b640 1 -- 172.21.15.76:0/3388008755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648197600 msgr2=0x7f7648197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.589+0000 7f7645d9b640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648197600 0x7f7648197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.589+0000 7f7645d9b640 1 -- 172.21.15.76:0/3388008755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7648101390 msgr2=0x7f764819d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.589+0000 7f7645d9b640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7648101390 0x7f764819d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.589+0000 7f7645d9b640 1 -- 172.21.15.76:0/3388008755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7640009900 con 0x7f7648106460 2024-03-12T06:01:03.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.589+0000 7f7646d9d640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648197600 0x7f7648197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:03.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.589+0000 7f7645d9b640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7648106460 0x7f764819d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f763c00c970 tx=0x7f763c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:03.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.589+0000 7f76377fe640 1 -- 172.21.15.76:0/3388008755 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c007bb0 con 0x7f7648106460 2024-03-12T06:01:03.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.589+0000 7f76377fe640 1 -- 172.21.15.76:0/3388008755 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f763c007d10 con 0x7f7648106460 2024-03-12T06:01:03.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.589+0000 7f76377fe640 1 -- 172.21.15.76:0/3388008755 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c018730 con 0x7f7648106460 2024-03-12T06:01:03.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.589+0000 7f764cf02640 1 -- 172.21.15.76:0/3388008755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76481a4310 con 0x7f7648106460 2024-03-12T06:01:03.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.589+0000 7f764cf02640 1 -- 172.21.15.76:0/3388008755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76481a4820 con 0x7f7648106460 2024-03-12T06:01:03.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.590+0000 7f764cf02640 1 -- 172.21.15.76:0/3388008755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7648068450 con 0x7f7648106460 2024-03-12T06:01:03.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.591+0000 7f76377fe640 1 -- 172.21.15.76:0/3388008755 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f763c00f450 con 0x7f7648106460 2024-03-12T06:01:03.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.591+0000 7f76377fe640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f76300a1c20 0x7f76300a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.592+0000 7f764659c640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f76300a1c20 0x7f76300a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.592+0000 7f76377fe640 1 -- 172.21.15.76:0/3388008755 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f763c0cd830 con 0x7f7648106460 2024-03-12T06:01:03.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.592+0000 7f764659c640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f76300a1c20 0x7f76300a40e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f763800bbe0 tx=0x7f763800b390 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:03.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.596+0000 7f76377fe640 1 -- 172.21.15.76:0/3388008755 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f763c096730 con 0x7f7648106460 2024-03-12T06:01:03.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.806+0000 7f764cf02640 1 -- 172.21.15.76:0/3388008755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7648109720 con 0x7f7648106460 2024-03-12T06:01:03.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.807+0000 7f76377fe640 1 -- 172.21.15.76:0/3388008755 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11329 (secure 0 0 0) 0x7f7648109720 con 0x7f7648106460 2024-03-12T06:01:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:01:03.816 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T06:00:52.159715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T06:01:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 27 2024-03-12T06:01:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.810+0000 7f764cf02640 1 -- 172.21.15.76:0/3388008755 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f76300a1c20 msgr2=0x7f76300a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.810+0000 7f764cf02640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f76300a1c20 0x7f76300a40e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f763800bbe0 tx=0x7f763800b390 comp rx=0 tx=0).stop 2024-03-12T06:01:03.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.810+0000 7f764cf02640 1 -- 172.21.15.76:0/3388008755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7648106460 msgr2=0x7f764819d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.810+0000 7f764cf02640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7648106460 0x7f764819d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f763c00c970 tx=0x7f763c00ce30 comp rx=0 tx=0).stop 2024-03-12T06:01:03.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.810+0000 7f764cf02640 1 -- 172.21.15.76:0/3388008755 shutdown_connections 2024-03-12T06:01:03.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.810+0000 7f764cf02640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f76300a1c20 0x7f76300a40e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.810+0000 7f764cf02640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7648197600 0x7f7648197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.810+0000 7f764cf02640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7648106460 0x7f764819d990 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.810+0000 7f764cf02640 1 --2- 172.21.15.76:0/3388008755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7648101390 0x7f764819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.818 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.810+0000 7f764cf02640 1 -- 172.21.15.76:0/3388008755 >> 172.21.15.76:0/3388008755 conn(0x7f76480fcc30 msgr2=0x7f76480fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:03.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.810+0000 7f764cf02640 1 -- 172.21.15.76:0/3388008755 shutdown_connections 2024-03-12T06:01:03.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.810+0000 7f764cf02640 1 -- 172.21.15.76:0/3388008755 wait complete. 2024-03-12T06:01:03.826 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:01:03.826 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T06:01:03.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.925+0000 7f4a4c8a3640 1 -- 172.21.15.76:0/1213369174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a48105350 msgr2=0x7f4a4810cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.925+0000 7f4a4c8a3640 1 --2- 172.21.15.76:0/1213369174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a48105350 0x7f4a4810cf20 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f4a40009cd0 tx=0x7f4a4002c0f0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.925+0000 7f4a4c8a3640 1 -- 172.21.15.76:0/1213369174 shutdown_connections 2024-03-12T06:01:03.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.925+0000 7f4a4c8a3640 1 --2- 172.21.15.76:0/1213369174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a48105350 0x7f4a4810cf20 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.925+0000 7f4a4c8a3640 1 --2- 172.21.15.76:0/1213369174 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a48100c30 0x7f4a48104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.925+0000 7f4a4c8a3640 1 --2- 172.21.15.76:0/1213369174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a48100280 0x7f4a48100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.925+0000 7f4a4c8a3640 1 -- 172.21.15.76:0/1213369174 >> 172.21.15.76:0/1213369174 conn(0x7f4a480771a0 msgr2=0x7f4a480775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:03.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.925+0000 7f4a4c8a3640 1 -- 172.21.15.76:0/1213369174 shutdown_connections 2024-03-12T06:01:03.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.925+0000 7f4a4c8a3640 1 -- 172.21.15.76:0/1213369174 wait complete. 2024-03-12T06:01:03.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.926+0000 7f4a4c8a3640 1 Processor -- start 2024-03-12T06:01:03.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.926+0000 7f4a4c8a3640 1 -- start start 2024-03-12T06:01:03.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.926+0000 7f4a4c8a3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a48100280 0x7f4a4819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.926+0000 7f4a4c8a3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a48105350 0x7f4a4819d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.926+0000 7f4a4c8a3640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a48197590 0x7f4a48197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.926+0000 7f4a4c8a3640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a48197f80 con 0x7f4a48105350 2024-03-12T06:01:03.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.926+0000 7f4a4c8a3640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a481980f0 con 0x7f4a48197590 2024-03-12T06:01:03.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.926+0000 7f4a4c8a3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a481982b0 con 0x7f4a48100280 2024-03-12T06:01:03.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.926+0000 7f4a3ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a48105350 0x7f4a4819d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.926+0000 7f4a3ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a48105350 0x7f4a4819d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38110/0 (socket says 172.21.15.76:38110) 2024-03-12T06:01:03.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.926+0000 7f4a3ffff640 1 -- 172.21.15.76:0/4045353952 learned_addr learned my addr 172.21.15.76:0/4045353952 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:03.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.926+0000 7f4a4659c640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a48100280 0x7f4a4819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.926+0000 7f4a3ffff640 1 -- 172.21.15.76:0/4045353952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a48100280 msgr2=0x7f4a4819d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.926+0000 7f4a46d9d640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a48197590 0x7f4a48197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.927+0000 7f4a3ffff640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a48100280 0x7f4a4819d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.927+0000 7f4a3ffff640 1 -- 172.21.15.76:0/4045353952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a48197590 msgr2=0x7f4a48197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.927+0000 7f4a3ffff640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a48197590 0x7f4a48197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.927+0000 7f4a3ffff640 1 -- 172.21.15.76:0/4045353952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a40009900 con 0x7f4a48105350 2024-03-12T06:01:03.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.927+0000 7f4a4659c640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a48100280 0x7f4a4819d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:03.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.927+0000 7f4a46d9d640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a48197590 0x7f4a48197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:03.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.927+0000 7f4a3ffff640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a48105350 0x7f4a4819d920 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000e9c0 tx=0x7f4a3000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:03.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.927+0000 7f4a3f7fe640 1 -- 172.21.15.76:0/4045353952 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a30009700 con 0x7f4a48105350 2024-03-12T06:01:03.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.927+0000 7f4a3f7fe640 1 -- 172.21.15.76:0/4045353952 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a3000cd80 con 0x7f4a48105350 2024-03-12T06:01:03.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.927+0000 7f4a4c8a3640 1 -- 172.21.15.76:0/4045353952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a481a42a0 con 0x7f4a48105350 2024-03-12T06:01:03.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.927+0000 7f4a3f7fe640 1 -- 172.21.15.76:0/4045353952 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a30010430 con 0x7f4a48105350 2024-03-12T06:01:03.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.927+0000 7f4a4c8a3640 1 -- 172.21.15.76:0/4045353952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a481a4800 con 0x7f4a48105350 2024-03-12T06:01:03.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.929+0000 7f4a3f7fe640 1 -- 172.21.15.76:0/4045353952 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4a30005000 con 0x7f4a48105350 2024-03-12T06:01:03.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.929+0000 7f4a4f470740 1 Processor -- start 2024-03-12T06:01:03.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.929+0000 7f4a3f7fe640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4a340a1c20 0x7f4a340a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a4f470740 1 -- start start 2024-03-12T06:01:03.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a46d9d640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4a340a1c20 0x7f4a340a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a3f7fe640 1 -- 172.21.15.76:0/4045353952 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4a30014070 con 0x7f4a48105350 2024-03-12T06:01:03.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a4f470740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27a4250 0x555bb27a4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a4f470740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4b70 0x555bb27ad210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a4f470740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27ad750 0x555bb27afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a4f470740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb275f960 con 0x555bb27a4b70 2024-03-12T06:01:03.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a4f470740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb2713310 con 0x555bb27a4250 2024-03-12T06:01:03.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a4f470740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb2710840 con 0x555bb27ad750 2024-03-12T06:01:03.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a3ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4b70 0x555bb27ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a3ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4b70 0x555bb27ad210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38114/0 (socket says 172.21.15.76:38114) 2024-03-12T06:01:03.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a3ffff640 1 -- 172.21.15.76:0/1408830230 learned_addr learned my addr 172.21.15.76:0/1408830230 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:03.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a46d9d640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4a340a1c20 0x7f4a340a40e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4a40006010 tx=0x7f4a40005d80 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:03.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a4659c640 1 --2- 172.21.15.76:0/1408830230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27a4250 0x555bb27a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a3ffff640 1 -- 172.21.15.76:0/1408830230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27ad750 msgr2=0x555bb27afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a3ffff640 1 --2- 172.21.15.76:0/1408830230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27ad750 0x555bb27afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a3ffff640 1 -- 172.21.15.76:0/1408830230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27a4250 msgr2=0x555bb27a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a3ffff640 1 --2- 172.21.15.76:0/1408830230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27a4250 0x555bb27a4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a3ffff640 1 -- 172.21.15.76:0/1408830230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555bb26ce060 con 0x555bb27a4b70 2024-03-12T06:01:03.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.930+0000 7f4a3ffff640 1 --2- 172.21.15.76:0/1408830230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4b70 0x555bb27ad210 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f4a30097270 tx=0x7f4a3009a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84348ede9c4f7af8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:03.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.931+0000 7f4a3cff9640 1 -- 172.21.15.76:0/1408830230 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a300a3040 con 0x555bb27a4b70 2024-03-12T06:01:03.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.931+0000 7f4a3cff9640 1 -- 172.21.15.76:0/1408830230 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a30098ba0 con 0x555bb27a4b70 2024-03-12T06:01:03.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.931+0000 7f4a3cff9640 1 -- 172.21.15.76:0/1408830230 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a30096f10 con 0x555bb27a4b70 2024-03-12T06:01:03.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.931+0000 7f4a4f470740 1 -- 172.21.15.76:0/1408830230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4b70 msgr2=0x555bb27ad210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.931+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1408830230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4b70 0x555bb27ad210 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f4a30097270 tx=0x7f4a3009a7a0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.931+0000 7f4a4f470740 1 -- 172.21.15.76:0/1408830230 shutdown_connections 2024-03-12T06:01:03.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.931+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1408830230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27ad750 0x555bb27afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.931+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1408830230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4b70 0x555bb27ad210 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.931+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1408830230 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27a4250 0x555bb27a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.931+0000 7f4a4f470740 1 -- 172.21.15.76:0/1408830230 >> 172.21.15.76:0/1408830230 conn(0x555bb279e980 msgr2=0x555bb279ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:03.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.931+0000 7f4a4f470740 1 -- 172.21.15.76:0/1408830230 shutdown_connections 2024-03-12T06:01:03.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.931+0000 7f4a4f470740 1 -- 172.21.15.76:0/1408830230 wait complete. 2024-03-12T06:01:03.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a4f470740 1 -- start start 2024-03-12T06:01:03.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a4f470740 1 Processor -- start 2024-03-12T06:01:03.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a4f470740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27a4250 0x555bb27aa7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a4f470740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27a4b70 0x555bb27aacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a4f470740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27ad750 0x555bb27ab230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a4f470740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb270c020 con 0x555bb27ad750 2024-03-12T06:01:03.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a4f470740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb23df430 con 0x555bb27a4b70 2024-03-12T06:01:03.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a4f470740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb239be00 con 0x555bb27a4250 2024-03-12T06:01:03.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a4f470740 10 client.0 _ll_register_callbacks cb 0x555bb26b3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:03.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a46d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27ad750 0x555bb27ab230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a46d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27ad750 0x555bb27ab230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38130/0 (socket says 172.21.15.76:38130) 2024-03-12T06:01:03.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a46d9d640 1 -- 172.21.15.76:0/1267953383 learned_addr learned my addr 172.21.15.76:0/1267953383 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:03.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a46d9d640 1 -- 172.21.15.76:0/1267953383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27a4250 msgr2=0x555bb27aa7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a46d9d640 1 --2- 172.21.15.76:0/1267953383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27a4250 0x555bb27aa7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a46d9d640 1 -- 172.21.15.76:0/1267953383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27a4b70 msgr2=0x555bb27aacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a46d9d640 1 --2- 172.21.15.76:0/1267953383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27a4b70 0x555bb27aacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.932+0000 7f4a46d9d640 1 -- 172.21.15.76:0/1267953383 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a481a4800 con 0x555bb27ad750 2024-03-12T06:01:03.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a46d9d640 1 --2- 172.21.15.76:0/1267953383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27ad750 0x555bb27ab230 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4a40036000 tx=0x7f4a40034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2669dfcd482142f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:03.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:03.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a2dffb640 1 -- 172.21.15.76:0/1267953383 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a40032ac0 con 0x555bb27ad750 2024-03-12T06:01:03.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a2dffb640 1 -- 172.21.15.76:0/1267953383 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4a40045050 con 0x555bb27ad750 2024-03-12T06:01:03.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a4f470740 1 -- 172.21.15.76:0/1267953383 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555bb240b7e0 con 0x555bb27ad750 2024-03-12T06:01:03.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a2dffb640 1 -- 172.21.15.76:0/1267953383 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a40032d70 con 0x555bb27ad750 2024-03-12T06:01:03.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a4f470740 10 client.16701 fetch_fsmap learned FSMap version 27 2024-03-12T06:01:03.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a4f470740 1 -- 172.21.15.76:0/1267953383 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555bb270a6a0 con 0x555bb27ad750 2024-03-12T06:01:03.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a2dffb640 1 -- 172.21.15.76:0/1267953383 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a4005a020 con 0x555bb27ad750 2024-03-12T06:01:03.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a2dffb640 1 -- 172.21.15.76:0/1267953383 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 27) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f4a40056480 con 0x555bb27ad750 2024-03-12T06:01:03.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a4f470740 10 client.16701 fetch_fsmap finished waiting for FSMap version 27 2024-03-12T06:01:03.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a4f470740 10 client.16701 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:01:03.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a4f470740 20 client.16701 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:03.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1267953383 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x555bb27bdbc0 0x555bb27c0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a4f470740 4 client.16701 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:01:03.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.933+0000 7f4a4f470740 1 -- 172.21.15.76:0/1267953383 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555bb2448720 con 0x555bb27bdbc0 2024-03-12T06:01:03.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.934+0000 7f4a4659c640 1 --2- 172.21.15.76:0/1267953383 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x555bb27bdbc0 0x555bb27c0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.934+0000 7f4a2dffb640 10 client.16701 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:01:03.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.934+0000 7f4a4659c640 1 --2- 172.21.15.76:0/1267953383 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x555bb27bdbc0 0x555bb27c0020 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4eba0b4df59185e6 server_cookie=501a9e3b46ab8877 in_seq=0 out_seq=0 2024-03-12T06:01:03.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a2dffb640 1 -- 172.21.15.76:0/1267953383 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555bb2448720 con 0x555bb27bdbc0 2024-03-12T06:01:03.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a2dffb640 10 client.16701 handle_command_reply: tid=0 2024-03-12T06:01:03.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a4f470740 1 client.16701 shutdown 2024-03-12T06:01:03.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a4f470740 1 -- 172.21.15.76:0/1267953383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27ad750 msgr2=0x555bb27ab230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1267953383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27ad750 0x555bb27ab230 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4a40036000 tx=0x7f4a40034cc0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a4f470740 1 -- 172.21.15.76:0/1267953383 shutdown_connections 2024-03-12T06:01:03.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1267953383 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x555bb27bdbc0 0x555bb27c0020 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1267953383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27ad750 0x555bb27ab230 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a2dffb640 0 client.16701 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:01:03.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1267953383 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27a4b70 0x555bb27aacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1267953383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27a4250 0x555bb27aa7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a4f470740 1 -- 172.21.15.76:0/1267953383 >> 172.21.15.76:0/1267953383 conn(0x555bb279e980 msgr2=0x555bb27a06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:03.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a4f470740 1 -- 172.21.15.76:0/1267953383 shutdown_connections 2024-03-12T06:01:03.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a4f470740 1 -- 172.21.15.76:0/1267953383 wait complete. 2024-03-12T06:01:03.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.935+0000 7f4a4f470740 20 client.16701 trim_cache size 0 max 16384 2024-03-12T06:01:03.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.946+0000 7f4a4f470740 1 Processor -- start 2024-03-12T06:01:03.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.946+0000 7f4a4f470740 1 -- start start 2024-03-12T06:01:03.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.946+0000 7f4a4f470740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27a4250 0x555bb27ab050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.946+0000 7f4a4f470740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4b70 0x555bb279da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a4f470740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27ad750 0x555bb27b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a4f470740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb2691130 con 0x555bb27a4b70 2024-03-12T06:01:03.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a4f470740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb27b95b0 con 0x555bb27ad750 2024-03-12T06:01:03.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a4f470740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb27c5270 con 0x555bb27a4250 2024-03-12T06:01:03.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a3ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4b70 0x555bb279da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a3ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4b70 0x555bb279da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38138/0 (socket says 172.21.15.76:38138) 2024-03-12T06:01:03.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a3ffff640 1 -- 172.21.15.76:0/2530463353 learned_addr learned my addr 172.21.15.76:0/2530463353 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:03.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a4659c640 1 --2- 172.21.15.76:0/2530463353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27a4250 0x555bb27ab050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a46d9d640 1 --2- 172.21.15.76:0/2530463353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27ad750 0x555bb27b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a3ffff640 1 -- 172.21.15.76:0/2530463353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27a4250 msgr2=0x555bb27ab050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a3ffff640 1 --2- 172.21.15.76:0/2530463353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27a4250 0x555bb27ab050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a3ffff640 1 -- 172.21.15.76:0/2530463353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27ad750 msgr2=0x555bb27b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a3ffff640 1 --2- 172.21.15.76:0/2530463353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27ad750 0x555bb27b45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a3ffff640 1 -- 172.21.15.76:0/2530463353 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555bb26a6c20 con 0x555bb27a4b70 2024-03-12T06:01:03.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a46d9d640 1 --2- 172.21.15.76:0/2530463353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27ad750 0x555bb27b45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:03.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a3ffff640 1 --2- 172.21.15.76:0/2530463353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4b70 0x555bb279da00 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4a300cdd60 tx=0x7f4a3000efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8faa83abc8d9cfd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:03.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a2ffff640 1 -- 172.21.15.76:0/2530463353 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a300a30c0 con 0x555bb27a4b70 2024-03-12T06:01:03.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a2ffff640 1 -- 172.21.15.76:0/2530463353 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a3009e9f0 con 0x555bb27a4b70 2024-03-12T06:01:03.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.947+0000 7f4a2ffff640 1 -- 172.21.15.76:0/2530463353 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a300052b0 con 0x555bb27a4b70 2024-03-12T06:01:03.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 -- 172.21.15.76:0/2530463353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4b70 msgr2=0x555bb279da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 --2- 172.21.15.76:0/2530463353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4b70 0x555bb279da00 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4a300cdd60 tx=0x7f4a3000efb0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 -- 172.21.15.76:0/2530463353 shutdown_connections 2024-03-12T06:01:03.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 --2- 172.21.15.76:0/2530463353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27ad750 0x555bb27b45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 --2- 172.21.15.76:0/2530463353 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4b70 0x555bb279da00 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 --2- 172.21.15.76:0/2530463353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27a4250 0x555bb27ab050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 -- 172.21.15.76:0/2530463353 >> 172.21.15.76:0/2530463353 conn(0x555bb279e980 msgr2=0x555bb279d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:03.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 -- 172.21.15.76:0/2530463353 shutdown_connections 2024-03-12T06:01:03.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 -- 172.21.15.76:0/2530463353 wait complete. 2024-03-12T06:01:03.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 -- start start 2024-03-12T06:01:03.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 Processor -- start 2024-03-12T06:01:03.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4250 0x555bb27b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27a4b70 0x555bb27b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27ad750 0x555bb2814150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb27b3f60 con 0x555bb27a4250 2024-03-12T06:01:03.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb27b40d0 con 0x555bb27a4b70 2024-03-12T06:01:03.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x555bb2814690 con 0x555bb27ad750 2024-03-12T06:01:03.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4f470740 10 client.0 _ll_register_callbacks cb 0x7f4a3000cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:03.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4250 0x555bb27b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4250 0x555bb27b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38150/0 (socket says 172.21.15.76:38150) 2024-03-12T06:01:03.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.948+0000 7f4a4659c640 1 -- 172.21.15.76:0/1363820608 learned_addr learned my addr 172.21.15.76:0/1363820608 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:03.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a46d9d640 1 --2- 172.21.15.76:0/1363820608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27ad750 0x555bb2814150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a3ffff640 1 --2- 172.21.15.76:0/1363820608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27a4b70 0x555bb27b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a4659c640 1 -- 172.21.15.76:0/1363820608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27ad750 msgr2=0x555bb2814150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a4659c640 1 --2- 172.21.15.76:0/1363820608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27ad750 0x555bb2814150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a4659c640 1 -- 172.21.15.76:0/1363820608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27a4b70 msgr2=0x555bb27b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:03.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a4659c640 1 --2- 172.21.15.76:0/1363820608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27a4b70 0x555bb27b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a4659c640 1 -- 172.21.15.76:0/1363820608 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555bb26ce060 con 0x555bb27a4250 2024-03-12T06:01:03.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a4659c640 1 --2- 172.21.15.76:0/1363820608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4250 0x555bb27b3290 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x555bb27c0940 tx=0x7f4a38017350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4417e03fe6a89d4c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:03.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a2dffb640 1 -- 172.21.15.76:0/1363820608 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a38020670 con 0x555bb27a4250 2024-03-12T06:01:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a4f470740 1 -- 172.21.15.76:0/1363820608 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a30098ba0 con 0x555bb27a4250 2024-03-12T06:01:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a2dffb640 1 -- 172.21.15.76:0/1363820608 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4a38014070 con 0x555bb27a4250 2024-03-12T06:01:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a2dffb640 1 -- 172.21.15.76:0/1363820608 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a380207d0 con 0x555bb27a4250 2024-03-12T06:01:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a4f470740 10 client.16707 fetch_fsmap learned FSMap version 27 2024-03-12T06:01:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a4f470740 1 -- 172.21.15.76:0/1363820608 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555bb26cc8c0 con 0x555bb27a4250 2024-03-12T06:01:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a2dffb640 1 -- 172.21.15.76:0/1363820608 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a30098ba0 con 0x555bb27a4250 2024-03-12T06:01:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a2dffb640 1 -- 172.21.15.76:0/1363820608 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 27) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f4a38029b40 con 0x555bb27a4250 2024-03-12T06:01:03.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a4f470740 10 client.16707 fetch_fsmap finished waiting for FSMap version 27 2024-03-12T06:01:03.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a4f470740 10 client.16707 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:01:03.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a4f470740 20 client.16707 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:03.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.949+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1363820608 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x555bb2815e60 0x555bb2818280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:03.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a4f470740 4 client.16707 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:01:03.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a4f470740 1 -- 172.21.15.76:0/1363820608 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555bb2497890 con 0x555bb2815e60 2024-03-12T06:01:03.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a3ffff640 1 --2- 172.21.15.76:0/1363820608 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x555bb2815e60 0x555bb2818280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:03.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a2dffb640 10 client.16707 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:01:03.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a3ffff640 1 --2- 172.21.15.76:0/1363820608 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x555bb2815e60 0x555bb2818280 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a6515fb937840e44 server_cookie=b35edd046487f5fb in_seq=0 out_seq=0 2024-03-12T06:01:03.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a2dffb640 1 -- 172.21.15.76:0/1363820608 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555bb2497890 con 0x555bb2815e60 2024-03-12T06:01:03.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a2dffb640 10 client.16707 handle_command_reply: tid=0 2024-03-12T06:01:03.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a4f470740 1 client.16707 shutdown 2024-03-12T06:01:03.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a4f470740 1 -- 172.21.15.76:0/1363820608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4250 msgr2=0x555bb27b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1363820608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4250 0x555bb27b3290 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x555bb27c0940 tx=0x7f4a38017350 comp rx=0 tx=0).stop 2024-03-12T06:01:03.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a4f470740 1 -- 172.21.15.76:0/1363820608 shutdown_connections 2024-03-12T06:01:03.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1363820608 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x555bb2815e60 0x555bb2818280 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1363820608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555bb27ad750 0x555bb2814150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a2dffb640 0 client.16707 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:01:03.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1363820608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x555bb27a4b70 0x555bb27b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a4f470740 1 --2- 172.21.15.76:0/1363820608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x555bb27a4250 0x555bb27b3290 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a4f470740 1 -- 172.21.15.76:0/1363820608 >> 172.21.15.76:0/1363820608 conn(0x555bb279e980 msgr2=0x555bb279db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:03.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a4f470740 1 -- 172.21.15.76:0/1363820608 shutdown_connections 2024-03-12T06:01:03.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.950+0000 7f4a4f470740 1 -- 172.21.15.76:0/1363820608 wait complete. 2024-03-12T06:01:03.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.951+0000 7f4a4f470740 20 client.16707 trim_cache size 0 max 16384 2024-03-12T06:01:03.971 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:01:03.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.951+0000 7f4a4c8a3640 1 -- 172.21.15.76:0/4045353952 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4a340a1c20 msgr2=0x7f4a340a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.951+0000 7f4a4c8a3640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4a340a1c20 0x7f4a340a40e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4a40006010 tx=0x7f4a40005d80 comp rx=0 tx=0).stop 2024-03-12T06:01:03.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.951+0000 7f4a4c8a3640 1 -- 172.21.15.76:0/4045353952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a48105350 msgr2=0x7f4a4819d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:03.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.951+0000 7f4a4c8a3640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a48105350 0x7f4a4819d920 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000e9c0 tx=0x7f4a3000ee80 comp rx=0 tx=0).stop 2024-03-12T06:01:03.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.951+0000 7f4a4c8a3640 1 -- 172.21.15.76:0/4045353952 shutdown_connections 2024-03-12T06:01:03.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.951+0000 7f4a4c8a3640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4a340a1c20 0x7f4a340a40e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.951+0000 7f4a4c8a3640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a48197590 0x7f4a48197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.951+0000 7f4a4c8a3640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a48105350 0x7f4a4819d920 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.951+0000 7f4a4c8a3640 1 --2- 172.21.15.76:0/4045353952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a48100280 0x7f4a4819d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:03.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.951+0000 7f4a4c8a3640 1 -- 172.21.15.76:0/4045353952 >> 172.21.15.76:0/4045353952 conn(0x7f4a480771a0 msgr2=0x7f4a480fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:03.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.951+0000 7f4a4c8a3640 1 -- 172.21.15.76:0/4045353952 shutdown_connections 2024-03-12T06:01:03.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:03.951+0000 7f4a4c8a3640 1 -- 172.21.15.76:0/4045353952 wait complete. 2024-03-12T06:01:03.974 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:01:03.974 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T06:01:04.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.073+0000 7fb0d690c640 1 -- 172.21.15.76:0/760902790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0d0100290 msgr2=0x7fb0d0100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.073+0000 7fb0d690c640 1 --2- 172.21.15.76:0/760902790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0d0100290 0x7fb0d0100670 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009930 tx=0x7fb0b802bc30 comp rx=0 tx=0).stop 2024-03-12T06:01:04.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.073+0000 7fb0d690c640 1 -- 172.21.15.76:0/760902790 shutdown_connections 2024-03-12T06:01:04.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.073+0000 7fb0d690c640 1 --2- 172.21.15.76:0/760902790 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0d0105360 0x7fb0d010cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.073+0000 7fb0d690c640 1 --2- 172.21.15.76:0/760902790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0d0100c40 0x7fb0d0104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.073+0000 7fb0d690c640 1 --2- 172.21.15.76:0/760902790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0d0100290 0x7fb0d0100670 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.073+0000 7fb0d690c640 1 -- 172.21.15.76:0/760902790 >> 172.21.15.76:0/760902790 conn(0x7fb0d0077230 msgr2=0x7fb0d0077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:04.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.073+0000 7fb0d690c640 1 -- 172.21.15.76:0/760902790 shutdown_connections 2024-03-12T06:01:04.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.073+0000 7fb0d690c640 1 -- 172.21.15.76:0/760902790 wait complete. 2024-03-12T06:01:04.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.073+0000 7fb0d690c640 1 Processor -- start 2024-03-12T06:01:04.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.073+0000 7fb0d690c640 1 -- start start 2024-03-12T06:01:04.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.073+0000 7fb0d690c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0d0100290 0x7fb0d0196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0d690c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0d0105360 0x7fb0d0196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0d690c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0d019aef0 0x7fb0d019b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0d690c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d019b8e0 con 0x7fb0d0100290 2024-03-12T06:01:04.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0d690c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d019ba50 con 0x7fb0d0105360 2024-03-12T06:01:04.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0d690c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d019bbc0 con 0x7fb0d019aef0 2024-03-12T06:01:04.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0cffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0d0100290 0x7fb0d0196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0cffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0d0100290 0x7fb0d0196680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38158/0 (socket says 172.21.15.76:38158) 2024-03-12T06:01:04.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0cffff640 1 -- 172.21.15.76:0/2236484799 learned_addr learned my addr 172.21.15.76:0/2236484799 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:04.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0cf7fe640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0d0105360 0x7fb0d0196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0d4ea9640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0d019aef0 0x7fb0d019b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0cffff640 1 -- 172.21.15.76:0/2236484799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0d019aef0 msgr2=0x7fb0d019b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0cffff640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0d019aef0 0x7fb0d019b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0cffff640 1 -- 172.21.15.76:0/2236484799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0d0105360 msgr2=0x7fb0d0196bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0cffff640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0d0105360 0x7fb0d0196bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0cffff640 1 -- 172.21.15.76:0/2236484799 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b8009560 con 0x7fb0d0100290 2024-03-12T06:01:04.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0d4ea9640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0d019aef0 0x7fb0d019b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:04.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0cffff640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0d0100290 0x7fb0d0196680 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009230 tx=0x7fb0b8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:04.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0cd7fa640 1 -- 172.21.15.76:0/2236484799 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8004440 con 0x7fb0d0100290 2024-03-12T06:01:04.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0cd7fa640 1 -- 172.21.15.76:0/2236484799 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0b80052d0 con 0x7fb0d0100290 2024-03-12T06:01:04.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0d690c640 1 -- 172.21.15.76:0/2236484799 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d01a31a0 con 0x7fb0d0100290 2024-03-12T06:01:04.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0cd7fa640 1 -- 172.21.15.76:0/2236484799 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8003d00 con 0x7fb0d0100290 2024-03-12T06:01:04.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.074+0000 7fb0d690c640 1 -- 172.21.15.76:0/2236484799 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d01a3620 con 0x7fb0d0100290 2024-03-12T06:01:04.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0cd7fa640 1 -- 172.21.15.76:0/2236484799 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb0b8005440 con 0x7fb0d0100290 2024-03-12T06:01:04.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0d95c2740 1 Processor -- start 2024-03-12T06:01:04.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0d95c2740 1 -- start start 2024-03-12T06:01:04.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0d95c2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3f1f10 0x55e61b3f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0d95c2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0d95c2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3fd960 0x55e61b3ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0d95c2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61b3ad960 con 0x55e61b3f2830 2024-03-12T06:01:04.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0d95c2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61b361310 con 0x55e61b3fd960 2024-03-12T06:01:04.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0d95c2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61b35e840 con 0x55e61b3f1f10 2024-03-12T06:01:04.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0cffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0cffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38160/0 (socket says 172.21.15.76:38160) 2024-03-12T06:01:04.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0cffff640 1 -- 172.21.15.76:0/2819884141 learned_addr learned my addr 172.21.15.76:0/2819884141 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:04.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0cd7fa640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb0bc0a1ba0 0x7fb0bc0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0cf7fe640 1 --2- 172.21.15.76:0/2819884141 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3fd960 0x55e61b3ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.077+0000 7fb0d4ea9640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb0bc0a1ba0 0x7fb0bc0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.078+0000 7fb0d4ea9640 1 --2- 172.21.15.76:0/2819884141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3f1f10 0x55e61b3f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.078+0000 7fb0cd7fa640 1 -- 172.21.15.76:0/2236484799 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb0b80ef8f0 con 0x7fb0d0100290 2024-03-12T06:01:04.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.078+0000 7fb0cffff640 1 -- 172.21.15.76:0/2819884141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3f1f10 msgr2=0x55e61b3f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.078+0000 7fb0cffff640 1 --2- 172.21.15.76:0/2819884141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3f1f10 0x55e61b3f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.078+0000 7fb0cffff640 1 -- 172.21.15.76:0/2819884141 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3fd960 msgr2=0x55e61b3ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.078+0000 7fb0cffff640 1 --2- 172.21.15.76:0/2819884141 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3fd960 0x55e61b3ffd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.078+0000 7fb0cffff640 1 -- 172.21.15.76:0/2819884141 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e61b31c060 con 0x55e61b3f2830 2024-03-12T06:01:04.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.078+0000 7fb0d4ea9640 1 --2- 172.21.15.76:0/2819884141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3f1f10 0x55e61b3f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:04.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.078+0000 7fb0cffff640 1 --2- 172.21.15.76:0/2819884141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fd420 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80b9030 tx=0x7fb0b8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c61f7a5760cfdc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:04.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.078+0000 7fb0d4ea9640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb0bc0a1ba0 0x7fb0bc0a4060 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb0c80092f0 tx=0x7fb0c80095e0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:04.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.078+0000 7fb0c67fc640 1 -- 172.21.15.76:0/2819884141 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80bc1a0 con 0x55e61b3f2830 2024-03-12T06:01:04.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.078+0000 7fb0c67fc640 1 -- 172.21.15.76:0/2819884141 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0b80bbfc0 con 0x55e61b3f2830 2024-03-12T06:01:04.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.078+0000 7fb0c67fc640 1 -- 172.21.15.76:0/2819884141 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80c07b0 con 0x55e61b3f2830 2024-03-12T06:01:04.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.079+0000 7fb0d95c2740 1 -- 172.21.15.76:0/2819884141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 msgr2=0x55e61b3fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.079+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/2819884141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fd420 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80b9030 tx=0x7fb0b8009260 comp rx=0 tx=0).stop 2024-03-12T06:01:04.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.079+0000 7fb0d95c2740 1 -- 172.21.15.76:0/2819884141 shutdown_connections 2024-03-12T06:01:04.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.079+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/2819884141 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3fd960 0x55e61b3ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.079+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/2819884141 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fd420 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.079+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/2819884141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3f1f10 0x55e61b3f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.079+0000 7fb0d95c2740 1 -- 172.21.15.76:0/2819884141 >> 172.21.15.76:0/2819884141 conn(0x55e61b3ec980 msgr2=0x55e61b3ecd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:04.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.079+0000 7fb0d95c2740 1 -- 172.21.15.76:0/2819884141 shutdown_connections 2024-03-12T06:01:04.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.079+0000 7fb0d95c2740 1 -- 172.21.15.76:0/2819884141 wait complete. 2024-03-12T06:01:04.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.079+0000 7fb0d95c2740 1 -- start start 2024-03-12T06:01:04.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.079+0000 7fb0d95c2740 1 Processor -- start 2024-03-12T06:01:04.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0d95c2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3f1f10 0x55e61b3fa920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0d95c2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0d95c2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3fd960 0x55e61b3fb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0d95c2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61b35a020 con 0x55e61b3f2830 2024-03-12T06:01:04.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0d95c2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61b02d430 con 0x55e61b3f1f10 2024-03-12T06:01:04.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0d95c2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61afe9e00 con 0x55e61b3fd960 2024-03-12T06:01:04.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0d95c2740 10 client.0 _ll_register_callbacks cb 0x55e61b301cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:04.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0cf7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0cf7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38168/0 (socket says 172.21.15.76:38168) 2024-03-12T06:01:04.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0cf7fe640 1 -- 172.21.15.76:0/718322539 learned_addr learned my addr 172.21.15.76:0/718322539 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:04.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0cffff640 1 --2- 172.21.15.76:0/718322539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3f1f10 0x55e61b3fa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0d4ea9640 1 --2- 172.21.15.76:0/718322539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3fd960 0x55e61b3fb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0cf7fe640 1 -- 172.21.15.76:0/718322539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3fd960 msgr2=0x55e61b3fb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0cf7fe640 1 --2- 172.21.15.76:0/718322539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3fd960 0x55e61b3fb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0cf7fe640 1 -- 172.21.15.76:0/718322539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3f1f10 msgr2=0x55e61b3fa920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0cf7fe640 1 --2- 172.21.15.76:0/718322539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3f1f10 0x55e61b3fa920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0cf7fe640 1 -- 172.21.15.76:0/718322539 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d01a3620 con 0x55e61b3f2830 2024-03-12T06:01:04.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0d4ea9640 1 --2- 172.21.15.76:0/718322539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3fd960 0x55e61b3fb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:04.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0cf7fe640 1 --2- 172.21.15.76:0/718322539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fae60 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0010870 tx=0x7fb0c0010d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd215517ba30b021 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:04.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:04.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0af7fe640 1 -- 172.21.15.76:0/718322539 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0015070 con 0x55e61b3f2830 2024-03-12T06:01:04.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0af7fe640 1 -- 172.21.15.76:0/718322539 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb0c000fb30 con 0x55e61b3f2830 2024-03-12T06:01:04.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0d95c2740 1 -- 172.21.15.76:0/718322539 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e61b0597e0 con 0x55e61b3f2830 2024-03-12T06:01:04.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0af7fe640 1 -- 172.21.15.76:0/718322539 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0c000fcf0 con 0x55e61b3f2830 2024-03-12T06:01:04.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0d95c2740 10 client.16719 fetch_fsmap learned FSMap version 27 2024-03-12T06:01:04.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.080+0000 7fb0d95c2740 1 -- 172.21.15.76:0/718322539 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e61b3586a0 con 0x55e61b3f2830 2024-03-12T06:01:04.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.081+0000 7fb0af7fe640 1 -- 172.21.15.76:0/718322539 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e61b0597e0 con 0x55e61b3f2830 2024-03-12T06:01:04.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.081+0000 7fb0af7fe640 1 -- 172.21.15.76:0/718322539 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 27) v1 ==== 7886+0+0 (secure 0 0 0) 0x7fb0c00183b0 con 0x55e61b3f2830 2024-03-12T06:01:04.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.081+0000 7fb0d95c2740 10 client.16719 fetch_fsmap finished waiting for FSMap version 27 2024-03-12T06:01:04.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.081+0000 7fb0d95c2740 10 client.16719 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:01:04.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.081+0000 7fb0d95c2740 20 client.16719 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:04.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.081+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/718322539 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e61b40dc80 0x55e61b4100a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.081+0000 7fb0d95c2740 4 client.16719 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:01:04.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.081+0000 7fb0d95c2740 1 -- 172.21.15.76:0/718322539 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e61b096720 con 0x55e61b40dc80 2024-03-12T06:01:04.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.081+0000 7fb0cffff640 1 --2- 172.21.15.76:0/718322539 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e61b40dc80 0x55e61b4100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.081+0000 7fb0af7fe640 10 client.16719 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:01:04.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.081+0000 7fb0cffff640 1 --2- 172.21.15.76:0/718322539 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e61b40dc80 0x55e61b4100a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f19c741ae6bc73a server_cookie=45fa048daa9b07c7 in_seq=0 out_seq=0 2024-03-12T06:01:04.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0af7fe640 1 -- 172.21.15.76:0/718322539 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e61b096720 con 0x55e61b40dc80 2024-03-12T06:01:04.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0af7fe640 10 client.16719 handle_command_reply: tid=0 2024-03-12T06:01:04.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0d95c2740 1 client.16719 shutdown 2024-03-12T06:01:04.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0d95c2740 1 -- 172.21.15.76:0/718322539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 msgr2=0x55e61b3fae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/718322539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fae60 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0010870 tx=0x7fb0c0010d30 comp rx=0 tx=0).stop 2024-03-12T06:01:04.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0d95c2740 1 -- 172.21.15.76:0/718322539 shutdown_connections 2024-03-12T06:01:04.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/718322539 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e61b40dc80 0x55e61b4100a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/718322539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3fd960 0x55e61b3fb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/718322539 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fae60 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0af7fe640 0 client.16719 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:01:04.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/718322539 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3f1f10 0x55e61b3fa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0d95c2740 1 -- 172.21.15.76:0/718322539 >> 172.21.15.76:0/718322539 conn(0x55e61b3ec980 msgr2=0x55e61b3ee6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:04.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0d95c2740 1 -- 172.21.15.76:0/718322539 shutdown_connections 2024-03-12T06:01:04.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0d95c2740 1 -- 172.21.15.76:0/718322539 wait complete. 2024-03-12T06:01:04.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.082+0000 7fb0d95c2740 20 client.16719 trim_cache size 0 max 16384 2024-03-12T06:01:04.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 Processor -- start 2024-03-12T06:01:04.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 -- start start 2024-03-12T06:01:04.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3f1f10 0x55e61b3fd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3fd960 0x55e61b3f7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61b2df130 con 0x55e61b3f2830 2024-03-12T06:01:04.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61b3f2d30 con 0x55e61b3f1f10 2024-03-12T06:01:04.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61b3ec7f0 con 0x55e61b3fd960 2024-03-12T06:01:04.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0cf7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0cf7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38172/0 (socket says 172.21.15.76:38172) 2024-03-12T06:01:04.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0cf7fe640 1 -- 172.21.15.76:0/2120462708 learned_addr learned my addr 172.21.15.76:0/2120462708 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:04.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0cf7fe640 1 -- 172.21.15.76:0/2120462708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3fd960 msgr2=0x55e61b3f7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0cf7fe640 1 --2- 172.21.15.76:0/2120462708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3fd960 0x55e61b3f7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0cf7fe640 1 -- 172.21.15.76:0/2120462708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3f1f10 msgr2=0x55e61b3fd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0cffff640 1 --2- 172.21.15.76:0/2120462708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3f1f10 0x55e61b3fd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0cf7fe640 1 --2- 172.21.15.76:0/2120462708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3f1f10 0x55e61b3fd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0cf7fe640 1 -- 172.21.15.76:0/2120462708 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e61b2f4c20 con 0x55e61b3f2830 2024-03-12T06:01:04.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0cf7fe640 1 --2- 172.21.15.76:0/2120462708 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fb1c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0018810 tx=0x7fb0c0005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd813f56aac947d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:04.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0c5ffb640 1 -- 172.21.15.76:0/2120462708 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0015040 con 0x55e61b3f2830 2024-03-12T06:01:04.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0c5ffb640 1 -- 172.21.15.76:0/2120462708 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0c000a340 con 0x55e61b3f2830 2024-03-12T06:01:04.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0c5ffb640 1 -- 172.21.15.76:0/2120462708 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c00236e0 con 0x55e61b3f2830 2024-03-12T06:01:04.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 -- 172.21.15.76:0/2120462708 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 msgr2=0x55e61b3fb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/2120462708 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fb1c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0018810 tx=0x7fb0c0005670 comp rx=0 tx=0).stop 2024-03-12T06:01:04.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 -- 172.21.15.76:0/2120462708 shutdown_connections 2024-03-12T06:01:04.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/2120462708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3fd960 0x55e61b3f7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/2120462708 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b3fb1c0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/2120462708 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3f1f10 0x55e61b3fd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 -- 172.21.15.76:0/2120462708 >> 172.21.15.76:0/2120462708 conn(0x55e61b3ec980 msgr2=0x55e61b414a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:04.116 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 -- 172.21.15.76:0/2120462708 shutdown_connections 2024-03-12T06:01:04.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.095+0000 7fb0d95c2740 1 -- 172.21.15.76:0/2120462708 wait complete. 2024-03-12T06:01:04.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0d95c2740 1 -- start start 2024-03-12T06:01:04.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0d95c2740 1 Processor -- start 2024-03-12T06:01:04.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0d95c2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3f1f10 0x55e61b45e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0d95c2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b45ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0d95c2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3fd960 0x55e61b403a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0d95c2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61b404090 con 0x55e61b3f2830 2024-03-12T06:01:04.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0d95c2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61b404200 con 0x55e61b3fd960 2024-03-12T06:01:04.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0d95c2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61b4043c0 con 0x55e61b3f1f10 2024-03-12T06:01:04.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0d95c2740 10 client.0 _ll_register_callbacks cb 0x7fb0b80b8730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:04.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0cf7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b45ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0cf7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b45ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38182/0 (socket says 172.21.15.76:38182) 2024-03-12T06:01:04.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0cf7fe640 1 -- 172.21.15.76:0/4263138802 learned_addr learned my addr 172.21.15.76:0/4263138802 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:04.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0d4ea9640 1 --2- 172.21.15.76:0/4263138802 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3fd960 0x55e61b403a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0cf7fe640 1 -- 172.21.15.76:0/4263138802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3f1f10 msgr2=0x55e61b45e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0cf7fe640 1 --2- 172.21.15.76:0/4263138802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3f1f10 0x55e61b45e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0cf7fe640 1 -- 172.21.15.76:0/4263138802 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3fd960 msgr2=0x55e61b403a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0cf7fe640 1 --2- 172.21.15.76:0/4263138802 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3fd960 0x55e61b403a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.096+0000 7fb0cf7fe640 1 -- 172.21.15.76:0/4263138802 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e61b3586a0 con 0x55e61b3f2830 2024-03-12T06:01:04.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0cf7fe640 1 --2- 172.21.15.76:0/4263138802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b45ecb0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0010560 tx=0x7fb0c0010d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d995fa5ea47c6c24 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:04.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:04.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0af7fe640 1 -- 172.21.15.76:0/4263138802 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0015050 con 0x55e61b3f2830 2024-03-12T06:01:04.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0af7fe640 1 -- 172.21.15.76:0/4263138802 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb0c0005ce0 con 0x55e61b3f2830 2024-03-12T06:01:04.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0d95c2740 1 -- 172.21.15.76:0/4263138802 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb0b80bbfc0 con 0x55e61b3f2830 2024-03-12T06:01:04.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0af7fe640 1 -- 172.21.15.76:0/4263138802 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0c002d480 con 0x55e61b3f2830 2024-03-12T06:01:04.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0d95c2740 10 client.16725 fetch_fsmap learned FSMap version 27 2024-03-12T06:01:04.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0d95c2740 1 -- 172.21.15.76:0/4263138802 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e61b31a8c0 con 0x55e61b3f2830 2024-03-12T06:01:04.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0af7fe640 1 -- 172.21.15.76:0/4263138802 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0b80bbfc0 con 0x55e61b3f2830 2024-03-12T06:01:04.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0af7fe640 1 -- 172.21.15.76:0/4263138802 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 27) v1 ==== 7886+0+0 (secure 0 0 0) 0x7fb0c00237b0 con 0x55e61b3f2830 2024-03-12T06:01:04.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0d95c2740 10 client.16725 fetch_fsmap finished waiting for FSMap version 27 2024-03-12T06:01:04.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0d95c2740 10 client.16725 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:01:04.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0d95c2740 20 client.16725 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:04.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/4263138802 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e61b4661c0 0x55e61b4685e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0d95c2740 4 client.16725 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:01:04.122 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.097+0000 7fb0d95c2740 1 -- 172.21.15.76:0/4263138802 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e61b0e5890 con 0x55e61b4661c0 2024-03-12T06:01:04.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0cffff640 1 --2- 172.21.15.76:0/4263138802 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e61b4661c0 0x55e61b4685e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0cffff640 1 --2- 172.21.15.76:0/4263138802 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e61b4661c0 0x55e61b4685e0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1f376e99639ee68b server_cookie=5372b4fdd1c7a79e in_seq=0 out_seq=0 2024-03-12T06:01:04.123 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0af7fe640 10 client.16725 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:01:04.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0af7fe640 1 -- 172.21.15.76:0/4263138802 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e61b0e5890 con 0x55e61b4661c0 2024-03-12T06:01:04.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0af7fe640 10 client.16725 handle_command_reply: tid=0 2024-03-12T06:01:04.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0d95c2740 1 client.16725 shutdown 2024-03-12T06:01:04.124 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0d95c2740 1 -- 172.21.15.76:0/4263138802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 msgr2=0x55e61b45ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/4263138802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b45ecb0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0010560 tx=0x7fb0c0010d30 comp rx=0 tx=0).stop 2024-03-12T06:01:04.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0d95c2740 1 -- 172.21.15.76:0/4263138802 shutdown_connections 2024-03-12T06:01:04.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/4263138802 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e61b4661c0 0x55e61b4685e0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/4263138802 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e61b3fd960 0x55e61b403a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/4263138802 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e61b3f2830 0x55e61b45ecb0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0d95c2740 1 --2- 172.21.15.76:0/4263138802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61b3f1f10 0x55e61b45e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0af7fe640 0 client.16725 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:01:04.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0d95c2740 1 -- 172.21.15.76:0/4263138802 >> 172.21.15.76:0/4263138802 conn(0x55e61b3ec980 msgr2=0x55e61b3edcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:04.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0d95c2740 1 -- 172.21.15.76:0/4263138802 shutdown_connections 2024-03-12T06:01:04.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0d95c2740 1 -- 172.21.15.76:0/4263138802 wait complete. 2024-03-12T06:01:04.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.098+0000 7fb0d95c2740 20 client.16725 trim_cache size 0 max 16384 2024-03-12T06:01:04.128 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:01:04.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.099+0000 7fb0d690c640 1 -- 172.21.15.76:0/2236484799 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb0bc0a1ba0 msgr2=0x7fb0bc0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.099+0000 7fb0d690c640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb0bc0a1ba0 0x7fb0bc0a4060 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb0c80092f0 tx=0x7fb0c80095e0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.099+0000 7fb0d690c640 1 -- 172.21.15.76:0/2236484799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0d0100290 msgr2=0x7fb0d0196680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.099+0000 7fb0d690c640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0d0100290 0x7fb0d0196680 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009230 tx=0x7fb0b8004040 comp rx=0 tx=0).stop 2024-03-12T06:01:04.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.099+0000 7fb0d690c640 1 -- 172.21.15.76:0/2236484799 shutdown_connections 2024-03-12T06:01:04.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.099+0000 7fb0d690c640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb0bc0a1ba0 0x7fb0bc0a4060 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.099+0000 7fb0d690c640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0d019aef0 0x7fb0d019b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.099+0000 7fb0d690c640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb0d0105360 0x7fb0d0196bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.099+0000 7fb0d690c640 1 --2- 172.21.15.76:0/2236484799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0d0100290 0x7fb0d0196680 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.099+0000 7fb0d690c640 1 -- 172.21.15.76:0/2236484799 >> 172.21.15.76:0/2236484799 conn(0x7fb0d0077230 msgr2=0x7fb0d00ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:04.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.099+0000 7fb0d690c640 1 -- 172.21.15.76:0/2236484799 shutdown_connections 2024-03-12T06:01:04.133 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.099+0000 7fb0d690c640 1 -- 172.21.15.76:0/2236484799 wait complete. 2024-03-12T06:01:04.133 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:01:04.134 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T06:01:04.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.235+0000 7f9f7f640640 1 -- 172.21.15.76:0/1572450032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f78100c30 msgr2=0x7f9f78104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.235+0000 7f9f7f640640 1 --2- 172.21.15.76:0/1572450032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f78100c30 0x7f9f78104c70 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009930 tx=0x7f9f6802bc30 comp rx=0 tx=0).stop 2024-03-12T06:01:04.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.235+0000 7f9f7f640640 1 -- 172.21.15.76:0/1572450032 shutdown_connections 2024-03-12T06:01:04.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.235+0000 7f9f7f640640 1 --2- 172.21.15.76:0/1572450032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9f78105350 0x7f9f7810cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.235+0000 7f9f7f640640 1 --2- 172.21.15.76:0/1572450032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f78100c30 0x7f9f78104c70 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.235+0000 7f9f7f640640 1 --2- 172.21.15.76:0/1572450032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f78100280 0x7f9f78100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.235+0000 7f9f7f640640 1 -- 172.21.15.76:0/1572450032 >> 172.21.15.76:0/1572450032 conn(0x7f9f780771a0 msgr2=0x7f9f780775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:04.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.235+0000 7f9f7f640640 1 -- 172.21.15.76:0/1572450032 shutdown_connections 2024-03-12T06:01:04.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.235+0000 7f9f7f640640 1 -- 172.21.15.76:0/1572450032 wait complete. 2024-03-12T06:01:04.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.236+0000 7f9f7f640640 1 Processor -- start 2024-03-12T06:01:04.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.236+0000 7f9f7f640640 1 -- start start 2024-03-12T06:01:04.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.236+0000 7f9f7f640640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f78100280 0x7f9f7819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.236+0000 7f9f7f640640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f78105350 0x7f9f7819d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.236+0000 7f9f7f640640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9f781975f0 0x7f9f78197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.236+0000 7f9f7f640640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f78197fe0 con 0x7f9f78100280 2024-03-12T06:01:04.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.236+0000 7f9f7f640640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f78198150 con 0x7f9f781975f0 2024-03-12T06:01:04.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.236+0000 7f9f7f640640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f78198310 con 0x7f9f78105350 2024-03-12T06:01:04.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.236+0000 7f9f7d3dc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f78100280 0x7f9f7819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.236+0000 7f9f7d3dc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f78100280 0x7f9f7819d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38208/0 (socket says 172.21.15.76:38208) 2024-03-12T06:01:04.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.236+0000 7f9f7d3dc640 1 -- 172.21.15.76:0/3240331820 learned_addr learned my addr 172.21.15.76:0/3240331820 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:04.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.236+0000 7f9f7dbdd640 1 --2- 172.21.15.76:0/3240331820 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9f781975f0 0x7f9f78197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.236+0000 7f9f7cbdb640 1 --2- 172.21.15.76:0/3240331820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f78105350 0x7f9f7819d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.237+0000 7f9f7d3dc640 1 -- 172.21.15.76:0/3240331820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f78105350 msgr2=0x7f9f7819d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.237+0000 7f9f7d3dc640 1 --2- 172.21.15.76:0/3240331820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f78105350 0x7f9f7819d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.237+0000 7f9f7d3dc640 1 -- 172.21.15.76:0/3240331820 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9f781975f0 msgr2=0x7f9f78197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.237+0000 7f9f7d3dc640 1 --2- 172.21.15.76:0/3240331820 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9f781975f0 0x7f9f78197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.237+0000 7f9f7d3dc640 1 -- 172.21.15.76:0/3240331820 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f68009560 con 0x7f9f78100280 2024-03-12T06:01:04.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.237+0000 7f9f7d3dc640 1 --2- 172.21.15.76:0/3240331820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f78100280 0x7f9f7819d440 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f9f7400d770 tx=0x7f9f7400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:04.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.237+0000 7f9f6e7fc640 1 -- 172.21.15.76:0/3240331820 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f74009c30 con 0x7f9f78100280 2024-03-12T06:01:04.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.237+0000 7f9f6e7fc640 1 -- 172.21.15.76:0/3240331820 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f7400de80 con 0x7f9f78100280 2024-03-12T06:01:04.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.237+0000 7f9f6e7fc640 1 -- 172.21.15.76:0/3240331820 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f74004e80 con 0x7f9f78100280 2024-03-12T06:01:04.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.237+0000 7f9f7f640640 1 -- 172.21.15.76:0/3240331820 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f781a4300 con 0x7f9f78100280 2024-03-12T06:01:04.246 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.237+0000 7f9f7f640640 1 -- 172.21.15.76:0/3240331820 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f781a47e0 con 0x7f9f78100280 2024-03-12T06:01:04.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f6e7fc640 1 -- 172.21.15.76:0/3240331820 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9f74005020 con 0x7f9f78100280 2024-03-12T06:01:04.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f8220f740 1 Processor -- start 2024-03-12T06:01:04.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f8220f740 1 -- start start 2024-03-12T06:01:04.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f8220f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638a250 0x563d1638a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f8220f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638ab70 0x563d16393210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f8220f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d16393750 0x563d16395b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f8220f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d16345960 con 0x563d1638a250 2024-03-12T06:01:04.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f8220f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d162f9310 con 0x563d16393750 2024-03-12T06:01:04.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f8220f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d162f6840 con 0x563d1638ab70 2024-03-12T06:01:04.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f6e7fc640 1 --2- 172.21.15.76:0/3240331820 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9f640a1c20 0x7f9f640a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f7dbdd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638a250 0x563d1638a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f7dbdd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638a250 0x563d1638a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38216/0 (socket says 172.21.15.76:38216) 2024-03-12T06:01:04.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f7dbdd640 1 -- 172.21.15.76:0/1149792396 learned_addr learned my addr 172.21.15.76:0/1149792396 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:04.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f7dbdd640 1 --2- 172.21.15.76:0/3240331820 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9f640a1c20 0x7f9f640a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f6e7fc640 1 -- 172.21.15.76:0/3240331820 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9f740cd1e0 con 0x7f9f78100280 2024-03-12T06:01:04.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f7cbdb640 1 --2- 172.21.15.76:0/1149792396 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d16393750 0x563d16395b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f7d3dc640 1 --2- 172.21.15.76:0/1149792396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638ab70 0x563d16393210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.240+0000 7f9f7dbdd640 1 -- 172.21.15.76:0/1149792396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638ab70 msgr2=0x563d16393210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.241+0000 7f9f7dbdd640 1 --2- 172.21.15.76:0/1149792396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638ab70 0x563d16393210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.241+0000 7f9f7dbdd640 1 -- 172.21.15.76:0/1149792396 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d16393750 msgr2=0x563d16395b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.241+0000 7f9f7dbdd640 1 --2- 172.21.15.76:0/1149792396 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d16393750 0x563d16395b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.241+0000 7f9f7dbdd640 1 -- 172.21.15.76:0/1149792396 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d162b4060 con 0x563d1638a250 2024-03-12T06:01:04.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.241+0000 7f9f7d3dc640 1 --2- 172.21.15.76:0/1149792396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638ab70 0x563d16393210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:04.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.241+0000 7f9f7dbdd640 1 --2- 172.21.15.76:0/1149792396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638a250 0x563d1638a630 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f9f70013d90 tx=0x7f9f700117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4c861b62c19253 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:04.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.241+0000 7f9f7dbdd640 1 --2- 172.21.15.76:0/3240331820 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9f640a1c20 0x7f9f640a40e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9f70011cb0 tx=0x7f9f70018000 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:04.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.241+0000 7f9f637fe640 1 -- 172.21.15.76:0/1149792396 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70009e40 con 0x563d1638a250 2024-03-12T06:01:04.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.241+0000 7f9f637fe640 1 -- 172.21.15.76:0/1149792396 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f70003e00 con 0x563d1638a250 2024-03-12T06:01:04.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.241+0000 7f9f637fe640 1 -- 172.21.15.76:0/1149792396 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70005280 con 0x563d1638a250 2024-03-12T06:01:04.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.242+0000 7f9f8220f740 1 -- 172.21.15.76:0/1149792396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638a250 msgr2=0x563d1638a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.242+0000 7f9f8220f740 1 --2- 172.21.15.76:0/1149792396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638a250 0x563d1638a630 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f9f70013d90 tx=0x7f9f700117b0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.242+0000 7f9f8220f740 1 -- 172.21.15.76:0/1149792396 shutdown_connections 2024-03-12T06:01:04.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.242+0000 7f9f8220f740 1 --2- 172.21.15.76:0/1149792396 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d16393750 0x563d16395b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.242+0000 7f9f8220f740 1 --2- 172.21.15.76:0/1149792396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638ab70 0x563d16393210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.242+0000 7f9f8220f740 1 --2- 172.21.15.76:0/1149792396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638a250 0x563d1638a630 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.242+0000 7f9f8220f740 1 -- 172.21.15.76:0/1149792396 >> 172.21.15.76:0/1149792396 conn(0x563d16384980 msgr2=0x563d16384d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:04.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.242+0000 7f9f8220f740 1 -- 172.21.15.76:0/1149792396 shutdown_connections 2024-03-12T06:01:04.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.242+0000 7f9f8220f740 1 -- 172.21.15.76:0/1149792396 wait complete. 2024-03-12T06:01:04.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.242+0000 7f9f8220f740 1 -- start start 2024-03-12T06:01:04.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.242+0000 7f9f8220f740 1 Processor -- start 2024-03-12T06:01:04.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f8220f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d1638a250 0x563d16390770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f8220f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638ab70 0x563d16390cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f8220f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d16393750 0x563d163911f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f8220f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d162f2020 con 0x563d1638ab70 2024-03-12T06:01:04.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f8220f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d15fc5430 con 0x563d1638a250 2024-03-12T06:01:04.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f8220f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d15f81e00 con 0x563d16393750 2024-03-12T06:01:04.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f8220f740 10 client.0 _ll_register_callbacks cb 0x563d16299cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:04.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f7cbdb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638ab70 0x563d16390cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f7cbdb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638ab70 0x563d16390cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38226/0 (socket says 172.21.15.76:38226) 2024-03-12T06:01:04.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f7cbdb640 1 -- 172.21.15.76:0/1079968242 learned_addr learned my addr 172.21.15.76:0/1079968242 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:04.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f7dbdd640 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d16393750 0x563d163911f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f7d3dc640 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d1638a250 0x563d16390770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f7cbdb640 1 -- 172.21.15.76:0/1079968242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d16393750 msgr2=0x563d163911f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f7cbdb640 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d16393750 0x563d163911f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f7cbdb640 1 -- 172.21.15.76:0/1079968242 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d1638a250 msgr2=0x563d16390770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f7cbdb640 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d1638a250 0x563d16390770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f7cbdb640 1 -- 172.21.15.76:0/1079968242 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f7001b040 con 0x563d1638ab70 2024-03-12T06:01:04.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f7d3dc640 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d1638a250 0x563d16390770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:04.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f7dbdd640 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d16393750 0x563d163911f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:04.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f7cbdb640 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638ab70 0x563d16390cb0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f9f78069520 tx=0x7f9f6802ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174ef58824782c85 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:04.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f60ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:04.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f60ff9640 1 -- 172.21.15.76:0/1079968242 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f68004450 con 0x563d1638ab70 2024-03-12T06:01:04.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f60ff9640 1 -- 172.21.15.76:0/1079968242 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9f6802c5d0 con 0x563d1638ab70 2024-03-12T06:01:04.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f8220f740 1 -- 172.21.15.76:0/1079968242 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563d15ff17e0 con 0x563d1638ab70 2024-03-12T06:01:04.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f60ff9640 1 -- 172.21.15.76:0/1079968242 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f68004df0 con 0x563d1638ab70 2024-03-12T06:01:04.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f8220f740 10 client.16737 fetch_fsmap learned FSMap version 27 2024-03-12T06:01:04.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.243+0000 7f9f8220f740 1 -- 172.21.15.76:0/1079968242 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563d162f06a0 con 0x563d1638ab70 2024-03-12T06:01:04.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.244+0000 7f9f60ff9640 1 -- 172.21.15.76:0/1079968242 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563d15ff17e0 con 0x563d1638ab70 2024-03-12T06:01:04.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.244+0000 7f9f60ff9640 1 -- 172.21.15.76:0/1079968242 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 27) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f9f6803c040 con 0x563d1638ab70 2024-03-12T06:01:04.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.244+0000 7f9f8220f740 10 client.16737 fetch_fsmap finished waiting for FSMap version 27 2024-03-12T06:01:04.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.244+0000 7f9f8220f740 10 client.16737 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:04.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.244+0000 7f9f8220f740 20 client.16737 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:04.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.244+0000 7f9f8220f740 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d163a3b00 0x563d163a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.244+0000 7f9f8220f740 4 client.16737 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:01:04.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.244+0000 7f9f8220f740 1 -- 172.21.15.76:0/1079968242 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d1602e720 con 0x563d163a3b00 2024-03-12T06:01:04.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.244+0000 7f9f7d3dc640 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d163a3b00 0x563d163a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.244+0000 7f9f60ff9640 10 client.16737 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.244+0000 7f9f7d3dc640 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d163a3b00 0x563d163a5f20 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a30afb5dbfd38b7 server_cookie=fe658012f9e49593 in_seq=0 out_seq=0 2024-03-12T06:01:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.245+0000 7f9f60ff9640 1 -- 172.21.15.76:0/1079968242 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563d1602e720 con 0x563d163a3b00 2024-03-12T06:01:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.245+0000 7f9f60ff9640 10 client.16737 handle_command_reply: tid=0 2024-03-12T06:01:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.245+0000 7f9f8220f740 1 client.16737 shutdown 2024-03-12T06:01:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.245+0000 7f9f8220f740 1 -- 172.21.15.76:0/1079968242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638ab70 msgr2=0x563d16390cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.245+0000 7f9f8220f740 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638ab70 0x563d16390cb0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f9f78069520 tx=0x7f9f6802ca10 comp rx=0 tx=0).stop 2024-03-12T06:01:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.245+0000 7f9f8220f740 1 -- 172.21.15.76:0/1079968242 shutdown_connections 2024-03-12T06:01:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.245+0000 7f9f8220f740 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d16393750 0x563d163911f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.245+0000 7f9f8220f740 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d163a3b00 0x563d163a5f20 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.245+0000 7f9f8220f740 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638ab70 0x563d16390cb0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.245+0000 7f9f8220f740 1 --2- 172.21.15.76:0/1079968242 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d1638a250 0x563d16390770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.245+0000 7f9f60ff9640 0 client.16737 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.245+0000 7f9f8220f740 1 -- 172.21.15.76:0/1079968242 >> 172.21.15.76:0/1079968242 conn(0x563d16384980 msgr2=0x563d163866d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.245+0000 7f9f8220f740 1 -- 172.21.15.76:0/1079968242 shutdown_connections 2024-03-12T06:01:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.245+0000 7f9f8220f740 1 -- 172.21.15.76:0/1079968242 wait complete. 2024-03-12T06:01:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.246+0000 7f9f8220f740 20 client.16737 trim_cache size 0 max 16384 2024-03-12T06:01:04.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.257+0000 7f9f8220f740 1 Processor -- start 2024-03-12T06:01:04.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.257+0000 7f9f8220f740 1 -- start start 2024-03-12T06:01:04.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.257+0000 7f9f8220f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638a250 0x563d16392ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.257+0000 7f9f8220f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d1638ab70 0x563d16391010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.257+0000 7f9f8220f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d16393750 0x563d1638d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.257+0000 7f9f8220f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d16277130 con 0x563d16393750 2024-03-12T06:01:04.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.257+0000 7f9f8220f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d163857b0 con 0x563d1638ab70 2024-03-12T06:01:04.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.257+0000 7f9f8220f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d16395b40 con 0x563d1638a250 2024-03-12T06:01:04.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.257+0000 7f9f7dbdd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d16393750 0x563d1638d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.257+0000 7f9f7dbdd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d16393750 0x563d1638d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38228/0 (socket says 172.21.15.76:38228) 2024-03-12T06:01:04.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.257+0000 7f9f7dbdd640 1 -- 172.21.15.76:0/2850976600 learned_addr learned my addr 172.21.15.76:0/2850976600 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:04.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.257+0000 7f9f7d3dc640 1 --2- 172.21.15.76:0/2850976600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638a250 0x563d16392ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f7cbdb640 1 --2- 172.21.15.76:0/2850976600 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d1638ab70 0x563d16391010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f7dbdd640 1 -- 172.21.15.76:0/2850976600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638a250 msgr2=0x563d16392ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f7dbdd640 1 --2- 172.21.15.76:0/2850976600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638a250 0x563d16392ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f7dbdd640 1 -- 172.21.15.76:0/2850976600 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d1638ab70 msgr2=0x563d16391010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f7dbdd640 1 --2- 172.21.15.76:0/2850976600 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d1638ab70 0x563d16391010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f7dbdd640 1 -- 172.21.15.76:0/2850976600 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d1628cc20 con 0x563d16393750 2024-03-12T06:01:04.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f7dbdd640 1 --2- 172.21.15.76:0/2850976600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d16393750 0x563d1638d9d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9f70013fa0 tx=0x7f9f70013fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7206779a45145f5e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:04.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f62ffd640 1 -- 172.21.15.76:0/2850976600 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70034070 con 0x563d16393750 2024-03-12T06:01:04.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f62ffd640 1 -- 172.21.15.76:0/2850976600 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f7002c040 con 0x563d16393750 2024-03-12T06:01:04.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f62ffd640 1 -- 172.21.15.76:0/2850976600 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7002b070 con 0x563d16393750 2024-03-12T06:01:04.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f8220f740 1 -- 172.21.15.76:0/2850976600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d16393750 msgr2=0x563d1638d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f8220f740 1 --2- 172.21.15.76:0/2850976600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d16393750 0x563d1638d9d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9f70013fa0 tx=0x7f9f70013fd0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f8220f740 1 -- 172.21.15.76:0/2850976600 shutdown_connections 2024-03-12T06:01:04.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f8220f740 1 --2- 172.21.15.76:0/2850976600 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d16393750 0x563d1638d9d0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f8220f740 1 --2- 172.21.15.76:0/2850976600 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d1638ab70 0x563d16391010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f8220f740 1 --2- 172.21.15.76:0/2850976600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638a250 0x563d16392ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f8220f740 1 -- 172.21.15.76:0/2850976600 >> 172.21.15.76:0/2850976600 conn(0x563d16384980 msgr2=0x563d1638ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:04.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.258+0000 7f9f8220f740 1 -- 172.21.15.76:0/2850976600 shutdown_connections 2024-03-12T06:01:04.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.259+0000 7f9f8220f740 1 -- 172.21.15.76:0/2850976600 wait complete. 2024-03-12T06:01:04.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.259+0000 7f9f8220f740 1 -- start start 2024-03-12T06:01:04.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.259+0000 7f9f8220f740 1 Processor -- start 2024-03-12T06:01:04.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.259+0000 7f9f8220f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638a250 0x563d16399600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.259+0000 7f9f8220f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638ab70 0x563d16399b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.259+0000 7f9f8220f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d16393750 0x563d163fa010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.259+0000 7f9f8220f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d1639a240 con 0x563d1638a250 2024-03-12T06:01:04.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.259+0000 7f9f8220f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d163fa550 con 0x563d16393750 2024-03-12T06:01:04.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.259+0000 7f9f8220f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d163fa690 con 0x563d1638ab70 2024-03-12T06:01:04.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.259+0000 7f9f8220f740 10 client.0 _ll_register_callbacks cb 0x7f9f70013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:04.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.259+0000 7f9f7d3dc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638a250 0x563d16399600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.259+0000 7f9f7d3dc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638a250 0x563d16399600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38238/0 (socket says 172.21.15.76:38238) 2024-03-12T06:01:04.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.259+0000 7f9f7d3dc640 1 -- 172.21.15.76:0/3940688026 learned_addr learned my addr 172.21.15.76:0/3940688026 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:04.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.259+0000 7f9f7cbdb640 1 --2- 172.21.15.76:0/3940688026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638ab70 0x563d16399b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f7d3dc640 1 -- 172.21.15.76:0/3940688026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638ab70 msgr2=0x563d16399b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f7d3dc640 1 --2- 172.21.15.76:0/3940688026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638ab70 0x563d16399b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f7d3dc640 1 -- 172.21.15.76:0/3940688026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d16393750 msgr2=0x563d163fa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:04.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f7dbdd640 1 --2- 172.21.15.76:0/3940688026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d16393750 0x563d163fa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f7d3dc640 1 --2- 172.21.15.76:0/3940688026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d16393750 0x563d163fa010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f7d3dc640 1 -- 172.21.15.76:0/3940688026 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d162b4060 con 0x563d1638a250 2024-03-12T06:01:04.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f7d3dc640 1 --2- 172.21.15.76:0/3940688026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638a250 0x563d16399600 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f9f740caf60 tx=0x7f9f740997a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=822ce1b29879e62b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:04.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f60ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:04.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f60ff9640 1 -- 172.21.15.76:0/3940688026 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f74078ea0 con 0x563d1638a250 2024-03-12T06:01:04.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f8220f740 1 -- 172.21.15.76:0/3940688026 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f70003e00 con 0x563d1638a250 2024-03-12T06:01:04.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f60ff9640 1 -- 172.21.15.76:0/3940688026 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9f740a25e0 con 0x563d1638a250 2024-03-12T06:01:04.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f60ff9640 1 -- 172.21.15.76:0/3940688026 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f7407c070 con 0x563d1638a250 2024-03-12T06:01:04.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f8220f740 10 client.16743 fetch_fsmap learned FSMap version 27 2024-03-12T06:01:04.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f8220f740 1 -- 172.21.15.76:0/3940688026 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563d162b28c0 con 0x563d1638a250 2024-03-12T06:01:04.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f60ff9640 1 -- 172.21.15.76:0/3940688026 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f70003e00 con 0x563d1638a250 2024-03-12T06:01:04.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f60ff9640 1 -- 172.21.15.76:0/3940688026 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 27) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f9f740cabe0 con 0x563d1638a250 2024-03-12T06:01:04.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f8220f740 10 client.16743 fetch_fsmap finished waiting for FSMap version 27 2024-03-12T06:01:04.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f8220f740 10 client.16743 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:04.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f8220f740 20 client.16743 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:04.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f8220f740 1 --2- 172.21.15.76:0/3940688026 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d163fbe20 0x563d163fe240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:04.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f8220f740 4 client.16743 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:01:04.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.260+0000 7f9f8220f740 1 -- 172.21.15.76:0/3940688026 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563d1607d890 con 0x563d163fbe20 2024-03-12T06:01:04.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f7cbdb640 1 --2- 172.21.15.76:0/3940688026 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d163fbe20 0x563d163fe240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:04.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f60ff9640 10 client.16743 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:04.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f7cbdb640 1 --2- 172.21.15.76:0/3940688026 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d163fbe20 0x563d163fe240 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c571d79e544188b0 server_cookie=4c55cb8ef7f8593c in_seq=0 out_seq=0 2024-03-12T06:01:04.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f60ff9640 1 -- 172.21.15.76:0/3940688026 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563d1607d890 con 0x563d163fbe20 2024-03-12T06:01:04.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f60ff9640 10 client.16743 handle_command_reply: tid=0 2024-03-12T06:01:04.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f8220f740 1 client.16743 shutdown 2024-03-12T06:01:04.277 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f8220f740 1 -- 172.21.15.76:0/3940688026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638a250 msgr2=0x563d16399600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f8220f740 1 --2- 172.21.15.76:0/3940688026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638a250 0x563d16399600 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f9f740caf60 tx=0x7f9f740997a0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f8220f740 1 -- 172.21.15.76:0/3940688026 shutdown_connections 2024-03-12T06:01:04.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f8220f740 1 --2- 172.21.15.76:0/3940688026 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563d16393750 0x563d163fa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f8220f740 1 --2- 172.21.15.76:0/3940688026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563d1638ab70 0x563d16399b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f8220f740 1 --2- 172.21.15.76:0/3940688026 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563d163fbe20 0x563d163fe240 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f8220f740 1 --2- 172.21.15.76:0/3940688026 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563d1638a250 0x563d16399600 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f60ff9640 0 client.16743 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:04.278 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f8220f740 1 -- 172.21.15.76:0/3940688026 >> 172.21.15.76:0/3940688026 conn(0x563d16384980 msgr2=0x563d163832f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:04.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f8220f740 1 -- 172.21.15.76:0/3940688026 shutdown_connections 2024-03-12T06:01:04.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f8220f740 1 -- 172.21.15.76:0/3940688026 wait complete. 2024-03-12T06:01:04.279 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.261+0000 7f9f8220f740 20 client.16743 trim_cache size 0 max 16384 2024-03-12T06:01:04.279 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:01:04.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.262+0000 7f9f7f640640 1 -- 172.21.15.76:0/3240331820 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9f640a1c20 msgr2=0x7f9f640a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.262+0000 7f9f7f640640 1 --2- 172.21.15.76:0/3240331820 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9f640a1c20 0x7f9f640a40e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9f70011cb0 tx=0x7f9f70018000 comp rx=0 tx=0).stop 2024-03-12T06:01:04.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.262+0000 7f9f7f640640 1 -- 172.21.15.76:0/3240331820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f78100280 msgr2=0x7f9f7819d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:04.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.262+0000 7f9f7f640640 1 --2- 172.21.15.76:0/3240331820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f78100280 0x7f9f7819d440 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f9f7400d770 tx=0x7f9f7400dc30 comp rx=0 tx=0).stop 2024-03-12T06:01:04.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.262+0000 7f9f7f640640 1 -- 172.21.15.76:0/3240331820 shutdown_connections 2024-03-12T06:01:04.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.262+0000 7f9f7f640640 1 --2- 172.21.15.76:0/3240331820 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9f640a1c20 0x7f9f640a40e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.281 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.262+0000 7f9f7f640640 1 --2- 172.21.15.76:0/3240331820 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9f781975f0 0x7f9f78197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.262+0000 7f9f7f640640 1 --2- 172.21.15.76:0/3240331820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f78105350 0x7f9f7819d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.262+0000 7f9f7f640640 1 --2- 172.21.15.76:0/3240331820 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f78100280 0x7f9f7819d440 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:04.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.262+0000 7f9f7f640640 1 -- 172.21.15.76:0/3240331820 >> 172.21.15.76:0/3240331820 conn(0x7f9f780771a0 msgr2=0x7f9f780fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:04.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.262+0000 7f9f7f640640 1 -- 172.21.15.76:0/3240331820 shutdown_connections 2024-03-12T06:01:04.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:04.262+0000 7f9f7f640640 1 -- 172.21.15.76:0/3240331820 wait complete. 2024-03-12T06:01:04.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:04 smithi082 ceph-mon[36382]: pgmap v161: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:01:04.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:04 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3388008755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:01:04.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:04 smithi088 ceph-mon[36585]: pgmap v161: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:01:04.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:04 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3388008755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:01:04.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:04 smithi076 ceph-mon[32095]: pgmap v161: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:01:04.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:04 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3388008755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:01:05.283 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T06:01:05.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:01:05.284 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T06:01:05.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.391+0000 7fec0bcc0640 1 -- 172.21.15.76:0/912777147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec041023c0 msgr2=0x7fec041027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.391+0000 7fec0bcc0640 1 --2- 172.21.15.76:0/912777147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec041023c0 0x7fec041027a0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fec00009900 tx=0x7fec0002bcd0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.391+0000 7fec0bcc0640 1 -- 172.21.15.76:0/912777147 shutdown_connections 2024-03-12T06:01:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.391+0000 7fec0bcc0640 1 --2- 172.21.15.76:0/912777147 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fec0410b700 0x7fec0410dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.391+0000 7fec0bcc0640 1 --2- 172.21.15.76:0/912777147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec04102ce0 0x7fec0410b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.391+0000 7fec0bcc0640 1 --2- 172.21.15.76:0/912777147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec041023c0 0x7fec041027a0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.392+0000 7fec0bcc0640 1 -- 172.21.15.76:0/912777147 >> 172.21.15.76:0/912777147 conn(0x7fec04077230 msgr2=0x7fec04077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.392+0000 7fec0bcc0640 1 -- 172.21.15.76:0/912777147 shutdown_connections 2024-03-12T06:01:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.392+0000 7fec0bcc0640 1 -- 172.21.15.76:0/912777147 wait complete. 2024-03-12T06:01:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.392+0000 7fec0bcc0640 1 Processor -- start 2024-03-12T06:01:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.392+0000 7fec0bcc0640 1 -- start start 2024-03-12T06:01:05.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.392+0000 7fec0bcc0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec041023c0 0x7fec0419d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.392+0000 7fec0bcc0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fec04102ce0 0x7fec0419db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.392+0000 7fec0bcc0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0410b700 0x7fec041976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.392+0000 7fec0bcc0640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec04197e70 con 0x7fec041023c0 2024-03-12T06:01:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.392+0000 7fec0bcc0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec04197fe0 con 0x7fec04102ce0 2024-03-12T06:01:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.392+0000 7fec0bcc0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec041981a0 con 0x7fec0410b700 2024-03-12T06:01:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.392+0000 7fec09a5c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec041023c0 0x7fec0419d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7fec09a5c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec041023c0 0x7fec0419d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38264/0 (socket says 172.21.15.76:38264) 2024-03-12T06:01:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7fec09a5c640 1 -- 172.21.15.76:0/1969496720 learned_addr learned my addr 172.21.15.76:0/1969496720 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7fec0925b640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fec04102ce0 0x7fec0419db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7fec0a25d640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0410b700 0x7fec041976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7fec09a5c640 1 -- 172.21.15.76:0/1969496720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0410b700 msgr2=0x7fec041976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7fec09a5c640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0410b700 0x7fec041976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7fec09a5c640 1 -- 172.21.15.76:0/1969496720 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fec04102ce0 msgr2=0x7fec0419db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7fec09a5c640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fec04102ce0 0x7fec0419db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7fec09a5c640 1 -- 172.21.15.76:0/1969496720 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec00009560 con 0x7fec041023c0 2024-03-12T06:01:05.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7fec0925b640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fec04102ce0 0x7fec0419db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:05.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7fec0a25d640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0410b700 0x7fec041976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:05.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7fec09a5c640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec041023c0 0x7fec0419d620 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fec00005940 tx=0x7fec000042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:05.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7febfaffd640 1 -- 172.21.15.76:0/1969496720 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec000044e0 con 0x7fec041023c0 2024-03-12T06:01:05.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7febfaffd640 1 -- 172.21.15.76:0/1969496720 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec0002cc60 con 0x7fec041023c0 2024-03-12T06:01:05.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7fec0bcc0640 1 -- 172.21.15.76:0/1969496720 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec04198420 con 0x7fec041023c0 2024-03-12T06:01:05.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7febfaffd640 1 -- 172.21.15.76:0/1969496720 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec00003e20 con 0x7fec041023c0 2024-03-12T06:01:05.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.393+0000 7fec0bcc0640 1 -- 172.21.15.76:0/1969496720 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec041a45a0 con 0x7fec041023c0 2024-03-12T06:01:05.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.395+0000 7febfaffd640 1 -- 172.21.15.76:0/1969496720 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fec000035f0 con 0x7fec041023c0 2024-03-12T06:01:05.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.396+0000 7fec0e969740 1 Processor -- start 2024-03-12T06:01:05.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.396+0000 7febfaffd640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7febf00a1ba0 0x7febf00a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.396+0000 7fec0e969740 1 -- start start 2024-03-12T06:01:05.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.396+0000 7fec0e969740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726380 0x55f160726760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0e969740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726d30 0x55f16072f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0a25d640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7febf00a1ba0 0x7febf00a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0e969740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f16072f8b0 0x55f160731d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0e969740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f160694f90 con 0x55f160726d30 2024-03-12T06:01:05.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0e969740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1606924c0 con 0x55f160726380 2024-03-12T06:01:05.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0e969740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f16068dca0 con 0x55f16072f8b0 2024-03-12T06:01:05.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7febfaffd640 1 -- 172.21.15.76:0/1969496720 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fec000f01c0 con 0x7fec041023c0 2024-03-12T06:01:05.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0925b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726d30 0x55f16072f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0925b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726d30 0x55f16072f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38268/0 (socket says 172.21.15.76:38268) 2024-03-12T06:01:05.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0925b640 1 -- 172.21.15.76:0/248466780 learned_addr learned my addr 172.21.15.76:0/248466780 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:05.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec09a5c640 1 --2- 172.21.15.76:0/248466780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726380 0x55f160726760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0a25d640 1 --2- 172.21.15.76:0/248466780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f16072f8b0 0x55f160731d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0925b640 1 -- 172.21.15.76:0/248466780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f16072f8b0 msgr2=0x55f160731d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0925b640 1 --2- 172.21.15.76:0/248466780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f16072f8b0 0x55f160731d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0925b640 1 -- 172.21.15.76:0/248466780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726380 msgr2=0x55f160726760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0925b640 1 --2- 172.21.15.76:0/248466780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726380 0x55f160726760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0925b640 1 -- 172.21.15.76:0/248466780 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1606500b0 con 0x55f160726d30 2024-03-12T06:01:05.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0a25d640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7febf00a1ba0 0x7febf00a4060 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7febfc007f40 tx=0x7febfc00c730 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:05.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7fec0925b640 1 --2- 172.21.15.76:0/248466780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726d30 0x55f16072f2e0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7febf400a9d0 tx=0x7febf400ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270863ff1399100c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:05.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7febeffff640 1 -- 172.21.15.76:0/248466780 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf400cc60 con 0x55f160726d30 2024-03-12T06:01:05.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7febeffff640 1 -- 172.21.15.76:0/248466780 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7febf400cdc0 con 0x55f160726d30 2024-03-12T06:01:05.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.397+0000 7febeffff640 1 -- 172.21.15.76:0/248466780 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf4010770 con 0x55f160726d30 2024-03-12T06:01:05.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.398+0000 7fec0e969740 1 -- 172.21.15.76:0/248466780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726d30 msgr2=0x55f16072f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.398+0000 7fec0e969740 1 --2- 172.21.15.76:0/248466780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726d30 0x55f16072f2e0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7febf400a9d0 tx=0x7febf400ae90 comp rx=0 tx=0).stop 2024-03-12T06:01:05.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.398+0000 7fec0e969740 1 -- 172.21.15.76:0/248466780 shutdown_connections 2024-03-12T06:01:05.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.398+0000 7fec0e969740 1 --2- 172.21.15.76:0/248466780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f16072f8b0 0x55f160731d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.398+0000 7fec0e969740 1 --2- 172.21.15.76:0/248466780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726d30 0x55f16072f2e0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.398+0000 7fec0e969740 1 --2- 172.21.15.76:0/248466780 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726380 0x55f160726760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.398+0000 7fec0e969740 1 -- 172.21.15.76:0/248466780 >> 172.21.15.76:0/248466780 conn(0x55f160720840 msgr2=0x55f160720c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:05.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.398+0000 7fec0e969740 1 -- 172.21.15.76:0/248466780 shutdown_connections 2024-03-12T06:01:05.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.398+0000 7fec0e969740 1 -- 172.21.15.76:0/248466780 wait complete. 2024-03-12T06:01:05.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0e969740 1 -- start start 2024-03-12T06:01:05.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0e969740 1 Processor -- start 2024-03-12T06:01:05.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0e969740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726380 0x55f16072c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0e969740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f160726d30 0x55f16072cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0e969740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f16072f8b0 0x55f16072d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0e969740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f160632f40 con 0x55f16072f8b0 2024-03-12T06:01:05.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0e969740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1603614b0 con 0x55f160726380 2024-03-12T06:01:05.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0e969740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f160613670 con 0x55f160726d30 2024-03-12T06:01:05.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0e969740 10 client.0 _ll_register_callbacks cb 0x55f160680f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:05.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0a25d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f16072f8b0 0x55f16072d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0a25d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f16072f8b0 0x55f16072d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38274/0 (socket says 172.21.15.76:38274) 2024-03-12T06:01:05.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0a25d640 1 -- 172.21.15.76:0/3818629502 learned_addr learned my addr 172.21.15.76:0/3818629502 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:05.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0925b640 1 --2- 172.21.15.76:0/3818629502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f160726d30 0x55f16072cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec09a5c640 1 --2- 172.21.15.76:0/3818629502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726380 0x55f16072c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0a25d640 1 -- 172.21.15.76:0/3818629502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f160726d30 msgr2=0x55f16072cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0a25d640 1 --2- 172.21.15.76:0/3818629502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f160726d30 0x55f16072cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0a25d640 1 -- 172.21.15.76:0/3818629502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726380 msgr2=0x55f16072c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.399+0000 7fec0a25d640 1 --2- 172.21.15.76:0/3818629502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726380 0x55f16072c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7fec0a25d640 1 -- 172.21.15.76:0/3818629502 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf400a600 con 0x55f16072f8b0 2024-03-12T06:01:05.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7fec09a5c640 1 --2- 172.21.15.76:0/3818629502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726380 0x55f16072c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:05.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7fec0a25d640 1 --2- 172.21.15.76:0/3818629502 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f16072f8b0 0x55f16072d300 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7febfc00aac0 tx=0x7febfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588ccb98a4e386a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:05.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7febed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:05.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7febed7fa640 1 -- 172.21.15.76:0/3818629502 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc00fe80 con 0x55f16072f8b0 2024-03-12T06:01:05.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7febed7fa640 1 -- 172.21.15.76:0/3818629502 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7febfc015040 con 0x55f16072f8b0 2024-03-12T06:01:05.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7fec0e969740 1 -- 172.21.15.76:0/3818629502 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f160667f40 con 0x55f16072f8b0 2024-03-12T06:01:05.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7febed7fa640 1 -- 172.21.15.76:0/3818629502 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7febfc01a430 con 0x55f16072f8b0 2024-03-12T06:01:05.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7fec0e969740 10 client.16755 fetch_fsmap learned FSMap version 27 2024-03-12T06:01:05.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7fec0e969740 1 -- 172.21.15.76:0/3818629502 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f16068c320 con 0x55f16072f8b0 2024-03-12T06:01:05.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7febed7fa640 1 -- 172.21.15.76:0/3818629502 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f160667f40 con 0x55f16072f8b0 2024-03-12T06:01:05.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7febed7fa640 1 -- 172.21.15.76:0/3818629502 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 27) v1 ==== 7886+0+0 (secure 0 0 0) 0x7febfc01b760 con 0x55f16072f8b0 2024-03-12T06:01:05.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7fec0e969740 10 client.16755 fetch_fsmap finished waiting for FSMap version 27 2024-03-12T06:01:05.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7fec0e969740 10 client.16755 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:05.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7fec0e969740 20 client.16755 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:05.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7fec0e969740 1 --2- 172.21.15.76:0/3818629502 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55f16073ff60 0x55f160742380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7fec0e969740 4 client.16755 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:01:05.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.400+0000 7fec0e969740 1 -- 172.21.15.76:0/3818629502 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f160684310 con 0x55f16073ff60 2024-03-12T06:01:05.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.401+0000 7fec09a5c640 1 --2- 172.21.15.76:0/3818629502 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55f16073ff60 0x55f160742380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.401+0000 7febed7fa640 10 client.16755 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:05.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.401+0000 7fec09a5c640 1 --2- 172.21.15.76:0/3818629502 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55f16073ff60 0x55f160742380 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e815bd71363d55dc server_cookie=77d3dc508cfca3ed in_seq=0 out_seq=0 2024-03-12T06:01:05.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7febed7fa640 1 -- 172.21.15.76:0/3818629502 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f160684310 con 0x55f16073ff60 2024-03-12T06:01:05.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7febed7fa640 10 client.16755 handle_command_reply: tid=0 2024-03-12T06:01:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7fec0e969740 1 client.16755 shutdown 2024-03-12T06:01:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7fec0e969740 1 -- 172.21.15.76:0/3818629502 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f16072f8b0 msgr2=0x55f16072d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7fec0e969740 1 --2- 172.21.15.76:0/3818629502 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f16072f8b0 0x55f16072d300 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7febfc00aac0 tx=0x7febfc004a40 comp rx=0 tx=0).stop 2024-03-12T06:01:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7fec0e969740 1 -- 172.21.15.76:0/3818629502 shutdown_connections 2024-03-12T06:01:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7fec0e969740 1 --2- 172.21.15.76:0/3818629502 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55f16073ff60 0x55f160742380 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7fec0e969740 1 --2- 172.21.15.76:0/3818629502 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f16072f8b0 0x55f16072d300 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7fec0e969740 1 --2- 172.21.15.76:0/3818629502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f160726d30 0x55f16072cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7fec0e969740 1 --2- 172.21.15.76:0/3818629502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726380 0x55f16072c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7febed7fa640 0 client.16755 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:05.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7fec0e969740 1 -- 172.21.15.76:0/3818629502 >> 172.21.15.76:0/3818629502 conn(0x55f160720840 msgr2=0x55f1607227a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:05.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7fec0e969740 1 -- 172.21.15.76:0/3818629502 shutdown_connections 2024-03-12T06:01:05.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7fec0e969740 1 -- 172.21.15.76:0/3818629502 wait complete. 2024-03-12T06:01:05.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.402+0000 7fec0e969740 20 client.16755 trim_cache size 0 max 16384 2024-03-12T06:01:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.417+0000 7fec0e969740 1 Processor -- start 2024-03-12T06:01:05.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.417+0000 7fec0e969740 1 -- start start 2024-03-12T06:01:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.417+0000 7fec0e969740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726380 0x55f16072d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.417+0000 7fec0e969740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726d30 0x55f160722590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.417+0000 7fec0e969740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f16072f8b0 0x55f1607368e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.417+0000 7fec0e969740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f160720370 con 0x55f160726380 2024-03-12T06:01:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.417+0000 7fec0e969740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f160728210 con 0x55f160726d30 2024-03-12T06:01:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.417+0000 7fec0e969740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f160684310 con 0x55f16072f8b0 2024-03-12T06:01:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.417+0000 7fec09a5c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726380 0x55f16072d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.427 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.417+0000 7fec09a5c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726380 0x55f16072d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38278/0 (socket says 172.21.15.76:38278) 2024-03-12T06:01:05.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.417+0000 7fec09a5c640 1 -- 172.21.15.76:0/4107990250 learned_addr learned my addr 172.21.15.76:0/4107990250 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:05.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec0925b640 1 --2- 172.21.15.76:0/4107990250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726d30 0x55f160722590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec0a25d640 1 --2- 172.21.15.76:0/4107990250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f16072f8b0 0x55f1607368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec09a5c640 1 -- 172.21.15.76:0/4107990250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f16072f8b0 msgr2=0x55f1607368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec09a5c640 1 --2- 172.21.15.76:0/4107990250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f16072f8b0 0x55f1607368e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec09a5c640 1 -- 172.21.15.76:0/4107990250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726d30 msgr2=0x55f160722590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec09a5c640 1 --2- 172.21.15.76:0/4107990250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726d30 0x55f160722590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec09a5c640 1 -- 172.21.15.76:0/4107990250 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f160628c00 con 0x55f160726380 2024-03-12T06:01:05.428 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec0925b640 1 --2- 172.21.15.76:0/4107990250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726d30 0x55f160722590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec0a25d640 1 --2- 172.21.15.76:0/4107990250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f16072f8b0 0x55f1607368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec09a5c640 1 --2- 172.21.15.76:0/4107990250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726380 0x55f16072d120 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fec000edab0 tx=0x7fec00000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adeb1cb595a9ec8d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7febef7fe640 1 -- 172.21.15.76:0/4107990250 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec000db070 con 0x55f160726380 2024-03-12T06:01:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7febef7fe640 1 -- 172.21.15.76:0/4107990250 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec000d4070 con 0x55f160726380 2024-03-12T06:01:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7febef7fe640 1 -- 172.21.15.76:0/4107990250 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec000b95a0 con 0x55f160726380 2024-03-12T06:01:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec0e969740 1 -- 172.21.15.76:0/4107990250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726380 msgr2=0x55f16072d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec0e969740 1 --2- 172.21.15.76:0/4107990250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726380 0x55f16072d120 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fec000edab0 tx=0x7fec00000bc0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.429 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec0e969740 1 -- 172.21.15.76:0/4107990250 shutdown_connections 2024-03-12T06:01:05.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec0e969740 1 --2- 172.21.15.76:0/4107990250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f16072f8b0 0x55f1607368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec0e969740 1 --2- 172.21.15.76:0/4107990250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f160726d30 0x55f160722590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec0e969740 1 --2- 172.21.15.76:0/4107990250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726380 0x55f16072d120 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.418+0000 7fec0e969740 1 -- 172.21.15.76:0/4107990250 >> 172.21.15.76:0/4107990250 conn(0x55f160720840 msgr2=0x55f160721e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:05.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.419+0000 7fec0e969740 1 -- 172.21.15.76:0/4107990250 shutdown_connections 2024-03-12T06:01:05.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.419+0000 7fec0e969740 1 -- 172.21.15.76:0/4107990250 wait complete. 2024-03-12T06:01:05.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.419+0000 7fec0e969740 1 -- start start 2024-03-12T06:01:05.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.419+0000 7fec0e969740 1 Processor -- start 2024-03-12T06:01:05.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.419+0000 7fec0e969740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f160726380 0x55f160790a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.419+0000 7fec0e969740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726d30 0x55f160790f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.419+0000 7fec0e969740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f16072f8b0 0x55f160735d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.419+0000 7fec0e969740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f160736330 con 0x55f160726d30 2024-03-12T06:01:05.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.419+0000 7fec0e969740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1607364a0 con 0x55f16072f8b0 2024-03-12T06:01:05.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.419+0000 7fec0e969740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f160736660 con 0x55f160726380 2024-03-12T06:01:05.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.419+0000 7fec0e969740 10 client.0 _ll_register_callbacks cb 0x7febf400a1d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:05.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.419+0000 7fec0925b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726d30 0x55f160790f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.419+0000 7fec0925b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726d30 0x55f160790f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38280/0 (socket says 172.21.15.76:38280) 2024-03-12T06:01:05.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.419+0000 7fec0925b640 1 -- 172.21.15.76:0/2391537772 learned_addr learned my addr 172.21.15.76:0/2391537772 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:05.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7fec0a25d640 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f16072f8b0 0x55f160735d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7fec09a5c640 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f160726380 0x55f160790a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7fec0925b640 1 -- 172.21.15.76:0/2391537772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f160726380 msgr2=0x55f160790a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7fec0925b640 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f160726380 0x55f160790a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7fec0925b640 1 -- 172.21.15.76:0/2391537772 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f16072f8b0 msgr2=0x55f160735d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7fec0925b640 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f16072f8b0 0x55f160735d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7fec0925b640 1 -- 172.21.15.76:0/2391537772 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec041a45a0 con 0x55f160726d30 2024-03-12T06:01:05.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7fec09a5c640 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f160726380 0x55f160790a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:05.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7fec0a25d640 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f16072f8b0 0x55f160735d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:05.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7fec0925b640 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726d30 0x55f160790f70 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7febf400a9d0 tx=0x7febf400cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4811f78506dbebf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:05.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7febed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:05.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7febed7fa640 1 -- 172.21.15.76:0/2391537772 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf4020070 con 0x55f160726d30 2024-03-12T06:01:05.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7febed7fa640 1 -- 172.21.15.76:0/2391537772 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7febf40113f0 con 0x55f160726d30 2024-03-12T06:01:05.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7fec0e969740 1 -- 172.21.15.76:0/2391537772 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f1606b3900 con 0x55f160726d30 2024-03-12T06:01:05.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7febed7fa640 1 -- 172.21.15.76:0/2391537772 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7febf40115d0 con 0x55f160726d30 2024-03-12T06:01:05.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7fec0e969740 10 client.16761 fetch_fsmap learned FSMap version 27 2024-03-12T06:01:05.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7fec0e969740 1 -- 172.21.15.76:0/2391537772 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f16064e910 con 0x55f160726d30 2024-03-12T06:01:05.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7febed7fa640 1 -- 172.21.15.76:0/2391537772 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7febf4032020 con 0x55f160726d30 2024-03-12T06:01:05.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7febed7fa640 1 -- 172.21.15.76:0/2391537772 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 27) v1 ==== 7886+0+0 (secure 0 0 0) 0x7febf401a450 con 0x55f160726d30 2024-03-12T06:01:05.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.420+0000 7fec0e969740 10 client.16761 fetch_fsmap finished waiting for FSMap version 27 2024-03-12T06:01:05.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.421+0000 7fec0e969740 10 client.16761 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:05.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.421+0000 7fec0e969740 20 client.16761 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:05.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.421+0000 7fec0e969740 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55f1607987f0 0x55f16079ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.421+0000 7fec0e969740 4 client.16761 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T06:01:05.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.421+0000 7fec0e969740 1 -- 172.21.15.76:0/2391537772 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f16038d860 con 0x55f1607987f0 2024-03-12T06:01:05.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.421+0000 7fec09a5c640 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55f1607987f0 0x55f16079ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.421+0000 7febed7fa640 10 client.16761 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:05.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.421+0000 7fec09a5c640 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55f1607987f0 0x55f16079ac10 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff9c61bfe93d59d4 server_cookie=83cf4f132f65e73d in_seq=0 out_seq=0 2024-03-12T06:01:05.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.422+0000 7febed7fa640 1 -- 172.21.15.76:0/2391537772 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f16038d860 con 0x55f1607987f0 2024-03-12T06:01:05.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.422+0000 7febed7fa640 10 client.16761 handle_command_reply: tid=0 2024-03-12T06:01:05.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.422+0000 7fec0e969740 1 client.16761 shutdown 2024-03-12T06:01:05.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.422+0000 7fec0e969740 1 -- 172.21.15.76:0/2391537772 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726d30 msgr2=0x55f160790f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.422+0000 7fec0e969740 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726d30 0x55f160790f70 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7febf400a9d0 tx=0x7febf400cd00 comp rx=0 tx=0).stop 2024-03-12T06:01:05.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0e969740 1 -- 172.21.15.76:0/2391537772 shutdown_connections 2024-03-12T06:01:05.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0e969740 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55f16072f8b0 0x55f160735d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0e969740 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55f1607987f0 0x55f16079ac10 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0e969740 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55f160726d30 0x55f160790f70 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0e969740 1 --2- 172.21.15.76:0/2391537772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f160726380 0x55f160790a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7febed7fa640 0 client.16761 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:05.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0e969740 1 -- 172.21.15.76:0/2391537772 >> 172.21.15.76:0/2391537772 conn(0x55f160720840 msgr2=0x55f160722710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:05.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0e969740 1 -- 172.21.15.76:0/2391537772 shutdown_connections 2024-03-12T06:01:05.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0e969740 1 -- 172.21.15.76:0/2391537772 wait complete. 2024-03-12T06:01:05.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0e969740 20 client.16761 trim_cache size 0 max 16384 2024-03-12T06:01:05.438 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:01:05.438 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T06:01:05.438 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "e9634bfa-54f8-4262-9f1b-0f662e28cfd1", 2024-03-12T06:01:05.439 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T06:01:05.439 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:01:05.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0bcc0640 1 -- 172.21.15.76:0/1969496720 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7febf00a1ba0 msgr2=0x7febf00a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0bcc0640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7febf00a1ba0 0x7febf00a4060 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7febfc007f40 tx=0x7febfc00c730 comp rx=0 tx=0).stop 2024-03-12T06:01:05.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0bcc0640 1 -- 172.21.15.76:0/1969496720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec041023c0 msgr2=0x7fec0419d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0bcc0640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec041023c0 0x7fec0419d620 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fec00005940 tx=0x7fec000042f0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0bcc0640 1 -- 172.21.15.76:0/1969496720 shutdown_connections 2024-03-12T06:01:05.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0bcc0640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7febf00a1ba0 0x7febf00a4060 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0bcc0640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0410b700 0x7fec041976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0bcc0640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fec04102ce0 0x7fec0419db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0bcc0640 1 --2- 172.21.15.76:0/1969496720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fec041023c0 0x7fec0419d620 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0bcc0640 1 -- 172.21.15.76:0/1969496720 >> 172.21.15.76:0/1969496720 conn(0x7fec04077230 msgr2=0x7fec040fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:05.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0bcc0640 1 -- 172.21.15.76:0/1969496720 shutdown_connections 2024-03-12T06:01:05.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.423+0000 7fec0bcc0640 1 -- 172.21.15.76:0/1969496720 wait complete. 2024-03-12T06:01:05.442 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:01:05.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:01:05.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.560+0000 7fece74d0640 1 -- 172.21.15.76:0/3331285974 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fece0068df0 msgr2=0x7fece010d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.560+0000 7fece74d0640 1 --2- 172.21.15.76:0/3331285974 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fece0068df0 0x7fece010d5e0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fecd0009900 tx=0x7fecd002bc30 comp rx=0 tx=0).stop 2024-03-12T06:01:05.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.560+0000 7fece74d0640 1 -- 172.21.15.76:0/3331285974 shutdown_connections 2024-03-12T06:01:05.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.560+0000 7fece74d0640 1 --2- 172.21.15.76:0/3331285974 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fece010db20 0x7fece010ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.560+0000 7fece74d0640 1 --2- 172.21.15.76:0/3331285974 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fece0068df0 0x7fece010d5e0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.560+0000 7fece74d0640 1 --2- 172.21.15.76:0/3331285974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece00684d0 0x7fece00688b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.560+0000 7fece74d0640 1 -- 172.21.15.76:0/3331285974 >> 172.21.15.76:0/3331285974 conn(0x7fece0077180 msgr2=0x7fece0077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:05.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.560+0000 7fece74d0640 1 -- 172.21.15.76:0/3331285974 shutdown_connections 2024-03-12T06:01:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.560+0000 7fece74d0640 1 -- 172.21.15.76:0/3331285974 wait complete. 2024-03-12T06:01:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece74d0640 1 Processor -- start 2024-03-12T06:01:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece74d0640 1 -- start start 2024-03-12T06:01:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece74d0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fece00684d0 0x7fece019f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece74d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece0068df0 0x7fece019fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece74d0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fece010db20 0x7fece0199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece74d0640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece0199ef0 con 0x7fece00684d0 2024-03-12T06:01:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece74d0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece019a060 con 0x7fece010db20 2024-03-12T06:01:05.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece74d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece019a220 con 0x7fece0068df0 2024-03-12T06:01:05.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece526c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fece00684d0 0x7fece019f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece526c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fece00684d0 0x7fece019f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38300/0 (socket says 172.21.15.76:38300) 2024-03-12T06:01:05.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece526c640 1 -- 172.21.15.76:0/3583112381 learned_addr learned my addr 172.21.15.76:0/3583112381 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:05.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece5a6d640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fece010db20 0x7fece0199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece4a6b640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece0068df0 0x7fece019fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece526c640 1 -- 172.21.15.76:0/3583112381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece0068df0 msgr2=0x7fece019fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece526c640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece0068df0 0x7fece019fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece526c640 1 -- 172.21.15.76:0/3583112381 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fece010db20 msgr2=0x7fece0199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece526c640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fece010db20 0x7fece0199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece526c640 1 -- 172.21.15.76:0/3583112381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd0009560 con 0x7fece00684d0 2024-03-12T06:01:05.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.561+0000 7fece5a6d640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fece010db20 0x7fece0199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:05.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.562+0000 7fece526c640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fece00684d0 0x7fece019f6a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fecdc00e9f0 tx=0x7fecdc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:05.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.562+0000 7fecd67fc640 1 -- 172.21.15.76:0/3583112381 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc009700 con 0x7fece00684d0 2024-03-12T06:01:05.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.562+0000 7fece74d0640 1 -- 172.21.15.76:0/3583112381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece019a500 con 0x7fece00684d0 2024-03-12T06:01:05.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.562+0000 7fecd67fc640 1 -- 172.21.15.76:0/3583112381 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecdc00cd80 con 0x7fece00684d0 2024-03-12T06:01:05.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.562+0000 7fecd67fc640 1 -- 172.21.15.76:0/3583112381 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc018920 con 0x7fece00684d0 2024-03-12T06:01:05.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.562+0000 7fece74d0640 1 -- 172.21.15.76:0/3583112381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece01a6620 con 0x7fece00684d0 2024-03-12T06:01:05.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.565+0000 7fecea177740 1 Processor -- start 2024-03-12T06:01:05.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.565+0000 7fecd67fc640 1 -- 172.21.15.76:0/3583112381 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fecdc018a80 con 0x7fece00684d0 2024-03-12T06:01:05.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.565+0000 7fecea177740 1 -- start start 2024-03-12T06:01:05.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.565+0000 7fecea177740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b96812b0 0x55a8b9681690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.565+0000 7fecea177740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b9681bd0 0x55a8b968a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.565+0000 7fecea177740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b968a7b0 0x55a8b968cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.565+0000 7fecea177740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b95f0280 con 0x55a8b9681bd0 2024-03-12T06:01:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.565+0000 7fecea177740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b95ed7b0 con 0x55a8b96812b0 2024-03-12T06:01:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.565+0000 7fecea177740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b95e8f90 con 0x55a8b968a7b0 2024-03-12T06:01:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.566+0000 7fece5a6d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b96812b0 0x55a8b9681690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.566+0000 7fece526c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b9681bd0 0x55a8b968a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.566+0000 7fece5a6d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b96812b0 0x55a8b9681690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:46506/0 (socket says 172.21.15.76:46506) 2024-03-12T06:01:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.566+0000 7fece5a6d640 1 -- 172.21.15.76:0/1714316059 learned_addr learned my addr 172.21.15.76:0/1714316059 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.566+0000 7fece4a6b640 1 --2- 172.21.15.76:0/1714316059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b968a7b0 0x55a8b968cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.566+0000 7fecd67fc640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7feccc0a1c20 0x7feccc0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.566+0000 7fece4a6b640 1 -- 172.21.15.76:0/1714316059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b96812b0 msgr2=0x55a8b9681690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.566+0000 7fece4a6b640 1 --2- 172.21.15.76:0/1714316059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b96812b0 0x55a8b9681690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.566+0000 7fece4a6b640 1 -- 172.21.15.76:0/1714316059 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b9681bd0 msgr2=0x55a8b968a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.566+0000 7fece4a6b640 1 --2- 172.21.15.76:0/1714316059 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b9681bd0 0x55a8b968a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.566+0000 7fece4a6b640 1 -- 172.21.15.76:0/1714316059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8b95a7570 con 0x55a8b968a7b0 2024-03-12T06:01:05.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.566+0000 7fecd67fc640 1 -- 172.21.15.76:0/3583112381 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fecdc014070 con 0x7fece00684d0 2024-03-12T06:01:05.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.566+0000 7fece5a6d640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7feccc0a1c20 0x7feccc0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.567+0000 7fece4a6b640 1 --2- 172.21.15.76:0/1714316059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b968a7b0 0x55a8b968cba0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fecd0000c00 tx=0x7fecd002c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=212c36dab6da4ffc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:05.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.567+0000 7feccb7fe640 1 -- 172.21.15.76:0/1714316059 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd00047c0 con 0x55a8b968a7b0 2024-03-12T06:01:05.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.567+0000 7feccb7fe640 1 -- 172.21.15.76:0/1714316059 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecd0004920 con 0x55a8b968a7b0 2024-03-12T06:01:05.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.567+0000 7feccb7fe640 1 -- 172.21.15.76:0/1714316059 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd00050e0 con 0x55a8b968a7b0 2024-03-12T06:01:05.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.568+0000 7fecea177740 1 -- 172.21.15.76:0/1714316059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b968a7b0 msgr2=0x55a8b968cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.568+0000 7fece5a6d640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7feccc0a1c20 0x7feccc0a40e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fecd8013620 tx=0x7fecd8004a40 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:05.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.568+0000 7fecea177740 1 --2- 172.21.15.76:0/1714316059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b968a7b0 0x55a8b968cba0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fecd0000c00 tx=0x7fecd002c6a0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.568+0000 7fecea177740 1 -- 172.21.15.76:0/1714316059 shutdown_connections 2024-03-12T06:01:05.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.568+0000 7fecea177740 1 --2- 172.21.15.76:0/1714316059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b968a7b0 0x55a8b968cba0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.568+0000 7fecea177740 1 --2- 172.21.15.76:0/1714316059 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b9681bd0 0x55a8b968a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.568+0000 7fecea177740 1 --2- 172.21.15.76:0/1714316059 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b96812b0 0x55a8b9681690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.568+0000 7fecea177740 1 -- 172.21.15.76:0/1714316059 >> 172.21.15.76:0/1714316059 conn(0x55a8b967b990 msgr2=0x55a8b967bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:05.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.568+0000 7fecea177740 1 -- 172.21.15.76:0/1714316059 shutdown_connections 2024-03-12T06:01:05.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.568+0000 7fecea177740 1 -- 172.21.15.76:0/1714316059 wait complete. 2024-03-12T06:01:05.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.568+0000 7fecea177740 1 -- start start 2024-03-12T06:01:05.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.568+0000 7fecea177740 1 Processor -- start 2024-03-12T06:01:05.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fecea177740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 0x55a8b968a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fecea177740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b9681bd0 0x55a8b9687500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fecea177740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b968a7b0 0x55a8b9687b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fecea177740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b92bc430 con 0x55a8b9681bd0 2024-03-12T06:01:05.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fecea177740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b9278e00 con 0x55a8b968a7b0 2024-03-12T06:01:05.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fecea177740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b956dfd0 con 0x55a8b96812b0 2024-03-12T06:01:05.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fece4a6b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b9681bd0 0x55a8b9687500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fece526c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 0x55a8b968a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fece4a6b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b9681bd0 0x55a8b9687500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38312/0 (socket says 172.21.15.76:38312) 2024-03-12T06:01:05.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fece4a6b640 1 -- 172.21.15.76:0/3422238432 learned_addr learned my addr 172.21.15.76:0/3422238432 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:05.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fece5a6d640 1 --2- 172.21.15.76:0/3422238432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b968a7b0 0x55a8b9687b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fecea177740 10 client.0 _ll_register_callbacks cb 0x55a8b95c3fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:05.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fece526c640 1 -- 172.21.15.76:0/3422238432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b968a7b0 msgr2=0x55a8b9687b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fece526c640 1 --2- 172.21.15.76:0/3422238432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b968a7b0 0x55a8b9687b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fece526c640 1 -- 172.21.15.76:0/3422238432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b9681bd0 msgr2=0x55a8b9687500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fece526c640 1 --2- 172.21.15.76:0/3422238432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b9681bd0 0x55a8b9687500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.569+0000 7fece526c640 1 -- 172.21.15.76:0/3422238432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece0110a00 con 0x55a8b96812b0 2024-03-12T06:01:05.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.570+0000 7fece526c640 1 --2- 172.21.15.76:0/3422238432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 0x55a8b968a170 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fecdc0cb300 tx=0x7fecdc096b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fa880d4c0a459e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:05.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.570+0000 7fecc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-03-12T06:01:05.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.570+0000 7fecc8ff9640 1 -- 172.21.15.76:0/3422238432 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc09aee0 con 0x55a8b96812b0 2024-03-12T06:01:05.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.570+0000 7fecea177740 1 -- 172.21.15.76:0/3422238432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a8b92e87e0 con 0x55a8b96812b0 2024-03-12T06:01:05.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.570+0000 7fecc8ff9640 1 -- 172.21.15.76:0/3422238432 <== mon.1 v2:172.21.15.88:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fecdc09a7e0 con 0x55a8b96812b0 2024-03-12T06:01:05.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.570+0000 7fecc8ff9640 1 -- 172.21.15.76:0/3422238432 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecdc09e930 con 0x55a8b96812b0 2024-03-12T06:01:05.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.570+0000 7fecea177740 10 client.25927 fetch_fsmap learned FSMap version 27 2024-03-12T06:01:05.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.570+0000 7fecea177740 1 -- 172.21.15.76:0/3422238432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a8b95e7610 con 0x55a8b96812b0 2024-03-12T06:01:05.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.570+0000 7fecc8ff9640 1 -- 172.21.15.76:0/3422238432 <== mon.1 v2:172.21.15.88:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a8b92e87e0 con 0x55a8b96812b0 2024-03-12T06:01:05.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.571+0000 7fecc8ff9640 1 -- 172.21.15.76:0/3422238432 <== mon.1 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7886+0+0 (secure 0 0 0) 0x7fecdc096530 con 0x55a8b96812b0 2024-03-12T06:01:05.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.571+0000 7fecea177740 10 client.25927 fetch_fsmap finished waiting for FSMap version 27 2024-03-12T06:01:05.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.571+0000 7fecea177740 10 client.25927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:05.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.571+0000 7fecea177740 20 client.25927 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:05.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.571+0000 7fecea177740 1 --2- 172.21.15.76:0/3422238432 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a8b969a8a0 0x55a8b969cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.571+0000 7fecea177740 4 client.25927 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:01:05.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.571+0000 7fecea177740 1 -- 172.21.15.76:0/3422238432 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a8b9325720 con 0x55a8b969a8a0 2024-03-12T06:01:05.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.571+0000 7fece4a6b640 1 --2- 172.21.15.76:0/3422238432 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a8b969a8a0 0x55a8b969cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.571+0000 7fece4a6b640 1 --2- 172.21.15.76:0/3422238432 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a8b969a8a0 0x55a8b969cc90 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63aa393eea81360a server_cookie=a08b47c02dfcaf5e in_seq=0 out_seq=0 2024-03-12T06:01:05.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.571+0000 7fecc8ff9640 10 client.25927 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:05.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.572+0000 7fecc8ff9640 1 -- 172.21.15.76:0/3422238432 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a8b9325720 con 0x55a8b969a8a0 2024-03-12T06:01:05.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.572+0000 7fecc8ff9640 10 client.25927 handle_command_reply: tid=0 2024-03-12T06:01:05.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.572+0000 7fecea177740 1 client.25927 shutdown 2024-03-12T06:01:05.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.572+0000 7fecea177740 1 -- 172.21.15.76:0/3422238432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 msgr2=0x55a8b968a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.572+0000 7fecea177740 1 --2- 172.21.15.76:0/3422238432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 0x55a8b968a170 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fecdc0cb300 tx=0x7fecdc096b40 comp rx=0 tx=0).stop 2024-03-12T06:01:05.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.572+0000 7fecea177740 1 -- 172.21.15.76:0/3422238432 shutdown_connections 2024-03-12T06:01:05.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.572+0000 7fecea177740 1 --2- 172.21.15.76:0/3422238432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b968a7b0 0x55a8b9687b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.572+0000 7fecea177740 1 --2- 172.21.15.76:0/3422238432 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a8b969a8a0 0x55a8b969cc90 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.572+0000 7fecea177740 1 --2- 172.21.15.76:0/3422238432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b9681bd0 0x55a8b9687500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.572+0000 7fecea177740 1 --2- 172.21.15.76:0/3422238432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 0x55a8b968a170 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.572+0000 7fecea177740 1 -- 172.21.15.76:0/3422238432 >> 172.21.15.76:0/3422238432 conn(0x55a8b967b990 msgr2=0x55a8b967d730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:05.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.573+0000 7fecc8ff9640 0 client.25927 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:05.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.573+0000 7fecea177740 1 -- 172.21.15.76:0/3422238432 shutdown_connections 2024-03-12T06:01:05.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.573+0000 7fecea177740 1 -- 172.21.15.76:0/3422238432 wait complete. 2024-03-12T06:01:05.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.573+0000 7fecea177740 20 client.25927 trim_cache size 0 max 16384 2024-03-12T06:01:05.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.593+0000 7fecea177740 1 Processor -- start 2024-03-12T06:01:05.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.593+0000 7fecea177740 1 -- start start 2024-03-12T06:01:05.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.593+0000 7fecea177740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 0x55a8b9689c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.593+0000 7fecea177740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b9681bd0 0x55a8b968a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.593+0000 7fecea177740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b968a7b0 0x55a8b9682810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fecea177740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b9675e20 con 0x55a8b968a7b0 2024-03-12T06:01:05.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fecea177740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b9685a50 con 0x55a8b9681bd0 2024-03-12T06:01:05.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fecea177740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b96819c0 con 0x55a8b96812b0 2024-03-12T06:01:05.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fece5a6d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b968a7b0 0x55a8b9682810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fece4a6b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b9681bd0 0x55a8b968a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fece5a6d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b968a7b0 0x55a8b9682810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38320/0 (socket says 172.21.15.76:38320) 2024-03-12T06:01:05.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fece4a6b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b9681bd0 0x55a8b968a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:46522/0 (socket says 172.21.15.76:46522) 2024-03-12T06:01:05.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fece5a6d640 1 -- 172.21.15.76:0/4187378710 learned_addr learned my addr 172.21.15.76:0/4187378710 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:05.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fece526c640 1 --2- 172.21.15.76:0/4187378710 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 0x55a8b9689c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fece5a6d640 1 -- 172.21.15.76:0/4187378710 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 msgr2=0x55a8b9689c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fece5a6d640 1 --2- 172.21.15.76:0/4187378710 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 0x55a8b9689c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fece5a6d640 1 -- 172.21.15.76:0/4187378710 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b9681bd0 msgr2=0x55a8b968a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fece5a6d640 1 --2- 172.21.15.76:0/4187378710 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b9681bd0 0x55a8b968a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fece5a6d640 1 -- 172.21.15.76:0/4187378710 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8b9583ac0 con 0x55a8b968a7b0 2024-03-12T06:01:05.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7fece5a6d640 1 --2- 172.21.15.76:0/4187378710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b968a7b0 0x55a8b9682810 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fecd8013410 tx=0x7fecd8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6618b078259be86 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:05.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.594+0000 7feccaffd640 1 -- 172.21.15.76:0/4187378710 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8023040 con 0x55a8b968a7b0 2024-03-12T06:01:05.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.595+0000 7feccaffd640 1 -- 172.21.15.76:0/4187378710 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecd8010db0 con 0x55a8b968a7b0 2024-03-12T06:01:05.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.595+0000 7feccaffd640 1 -- 172.21.15.76:0/4187378710 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd80118e0 con 0x55a8b968a7b0 2024-03-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.595+0000 7fecea177740 1 -- 172.21.15.76:0/4187378710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b968a7b0 msgr2=0x55a8b9682810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.595+0000 7fecea177740 1 --2- 172.21.15.76:0/4187378710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b968a7b0 0x55a8b9682810 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fecd8013410 tx=0x7fecd8018000 comp rx=0 tx=0).stop 2024-03-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.595+0000 7fecea177740 1 -- 172.21.15.76:0/4187378710 shutdown_connections 2024-03-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.595+0000 7fecea177740 1 --2- 172.21.15.76:0/4187378710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b968a7b0 0x55a8b9682810 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.595+0000 7fecea177740 1 --2- 172.21.15.76:0/4187378710 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b9681bd0 0x55a8b968a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.595+0000 7fecea177740 1 --2- 172.21.15.76:0/4187378710 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 0x55a8b9689c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.595+0000 7fecea177740 1 -- 172.21.15.76:0/4187378710 >> 172.21.15.76:0/4187378710 conn(0x55a8b967b990 msgr2=0x55a8b96a1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:05.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.595+0000 7fecea177740 1 -- 172.21.15.76:0/4187378710 shutdown_connections 2024-03-12T06:01:05.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.595+0000 7fecea177740 1 -- 172.21.15.76:0/4187378710 wait complete. 2024-03-12T06:01:05.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.596+0000 7fecea177740 1 -- start start 2024-03-12T06:01:05.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.596+0000 7fecea177740 1 Processor -- start 2024-03-12T06:01:05.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.596+0000 7fecea177740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 0x55a8b968e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.596+0000 7fecea177740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b9681bd0 0x55a8b968e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.596+0000 7fecea177740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b968a7b0 0x55a8b968ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.596+0000 7fecea177740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b96f0e80 con 0x55a8b968a7b0 2024-03-12T06:01:05.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.596+0000 7fecea177740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b96f0ff0 con 0x55a8b9681bd0 2024-03-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.596+0000 7fecea177740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8b96f11b0 con 0x55a8b96812b0 2024-03-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fecea177740 10 client.0 _ll_register_callbacks cb 0x7fecdc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fece4a6b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b9681bd0 0x55a8b968e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fece526c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 0x55a8b968e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fece4a6b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b9681bd0 0x55a8b968e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:46536/0 (socket says 172.21.15.76:46536) 2024-03-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fece4a6b640 1 -- 172.21.15.76:0/2864151191 learned_addr learned my addr 172.21.15.76:0/2864151191 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fece5a6d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b968a7b0 0x55a8b968ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fece4a6b640 1 -- 172.21.15.76:0/2864151191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 msgr2=0x55a8b968e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.608 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fece4a6b640 1 --2- 172.21.15.76:0/2864151191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 0x55a8b968e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fece4a6b640 1 -- 172.21.15.76:0/2864151191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b968a7b0 msgr2=0x55a8b968ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:05.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fece4a6b640 1 --2- 172.21.15.76:0/2864151191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b968a7b0 0x55a8b968ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fece4a6b640 1 -- 172.21.15.76:0/2864151191 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd80130f0 con 0x55a8b9681bd0 2024-03-12T06:01:05.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fece5a6d640 1 --2- 172.21.15.76:0/2864151191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b968a7b0 0x55a8b968ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:01:05.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fece4a6b640 1 --2- 172.21.15.76:0/2864151191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b9681bd0 0x55a8b968e770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fecd0009230 tx=0x7fecd0004570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81df589154e41dac server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:05.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fecc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T06:01:05.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fecc8ff9640 1 -- 172.21.15.76:0/2864151191 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd0051de0 con 0x55a8b9681bd0 2024-03-12T06:01:05.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fecea177740 1 -- 172.21.15.76:0/2864151191 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a8b9687330 con 0x55a8b9681bd0 2024-03-12T06:01:05.609 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.597+0000 7fecc8ff9640 1 -- 172.21.15.76:0/2864151191 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fecd004e040 con 0x55a8b9681bd0 2024-03-12T06:01:05.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.598+0000 7fecc8ff9640 1 -- 172.21.15.76:0/2864151191 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecd002c8e0 con 0x55a8b9681bd0 2024-03-12T06:01:05.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.598+0000 7fecea177740 10 client.26024 fetch_fsmap learned FSMap version 27 2024-03-12T06:01:05.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.598+0000 7fecea177740 1 -- 172.21.15.76:0/2864151191 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a8b95a96a0 con 0x55a8b9681bd0 2024-03-12T06:01:05.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.598+0000 7fecc8ff9640 1 -- 172.21.15.76:0/2864151191 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a8b9687330 con 0x55a8b9681bd0 2024-03-12T06:01:05.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.598+0000 7fecc8ff9640 1 -- 172.21.15.76:0/2864151191 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 27) v1 ==== 7886+0+0 (secure 0 0 0) 0x7fecd003c050 con 0x55a8b9681bd0 2024-03-12T06:01:05.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.598+0000 7fecea177740 10 client.26024 fetch_fsmap finished waiting for FSMap version 27 2024-03-12T06:01:05.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.598+0000 7fecea177740 10 client.26024 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:05.610 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.598+0000 7fecea177740 20 client.26024 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:05.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.598+0000 7fecea177740 1 --2- 172.21.15.76:0/2864151191 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a8b96f2620 0x55a8b96f4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:05.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.598+0000 7fecea177740 4 client.26024 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:01:05.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.598+0000 7fecea177740 1 -- 172.21.15.76:0/2864151191 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fecd0004920 con 0x55a8b96f2620 2024-03-12T06:01:05.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.598+0000 7fece526c640 1 --2- 172.21.15.76:0/2864151191 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a8b96f2620 0x55a8b96f4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:05.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fece526c640 1 --2- 172.21.15.76:0/2864151191 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a8b96f2620 0x55a8b96f4a40 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fa349d9b3ff4fb7 server_cookie=30b25ae25810e239 in_seq=0 out_seq=0 2024-03-12T06:01:05.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecc8ff9640 10 client.26024 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:05.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecc8ff9640 1 -- 172.21.15.76:0/2864151191 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7fecd0004920 con 0x55a8b96f2620 2024-03-12T06:01:05.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecc8ff9640 10 client.26024 handle_command_reply: tid=0 2024-03-12T06:01:05.611 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecea177740 1 client.26024 shutdown 2024-03-12T06:01:05.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecea177740 1 -- 172.21.15.76:0/2864151191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b9681bd0 msgr2=0x55a8b968e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecea177740 1 --2- 172.21.15.76:0/2864151191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b9681bd0 0x55a8b968e770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fecd0009230 tx=0x7fecd0004570 comp rx=0 tx=0).stop 2024-03-12T06:01:05.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecea177740 1 -- 172.21.15.76:0/2864151191 shutdown_connections 2024-03-12T06:01:05.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecea177740 1 --2- 172.21.15.76:0/2864151191 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a8b96f2620 0x55a8b96f4a40 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecea177740 1 --2- 172.21.15.76:0/2864151191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a8b968a7b0 0x55a8b968ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecea177740 1 --2- 172.21.15.76:0/2864151191 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a8b9681bd0 0x55a8b968e770 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecea177740 1 --2- 172.21.15.76:0/2864151191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a8b96812b0 0x55a8b968e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecea177740 1 -- 172.21.15.76:0/2864151191 >> 172.21.15.76:0/2864151191 conn(0x55a8b967b990 msgr2=0x55a8b9682810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:05.612 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecc8ff9640 0 client.26024 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:05.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecea177740 1 -- 172.21.15.76:0/2864151191 shutdown_connections 2024-03-12T06:01:05.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecea177740 1 -- 172.21.15.76:0/2864151191 wait complete. 2024-03-12T06:01:05.613 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.599+0000 7fecea177740 20 client.26024 trim_cache size 0 max 16384 2024-03-12T06:01:05.613 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:01:05.613 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T06:01:05.613 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T06:01:05.614 INFO:teuthology.orchestra.run.smithi076.stdout: "e9634bfa-54f8-4262-9f1b-0f662e28cfd1": { 2024-03-12T06:01:05.614 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T06:01:05.614 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "e9634bfa-54f8-4262-9f1b-0f662e28cfd1", 2024-03-12T06:01:05.614 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T06:01:05.614 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:01:05.614 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:01:05.614 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:01:05.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.600+0000 7fece74d0640 1 -- 172.21.15.76:0/3583112381 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7feccc0a1c20 msgr2=0x7feccc0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.600+0000 7fece74d0640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7feccc0a1c20 0x7feccc0a40e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fecd8013620 tx=0x7fecd8004a40 comp rx=0 tx=0).stop 2024-03-12T06:01:05.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.600+0000 7fece74d0640 1 -- 172.21.15.76:0/3583112381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fece00684d0 msgr2=0x7fece019f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:05.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.600+0000 7fece74d0640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fece00684d0 0x7fece019f6a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fecdc00e9f0 tx=0x7fecdc00eeb0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.600+0000 7fece74d0640 1 -- 172.21.15.76:0/3583112381 shutdown_connections 2024-03-12T06:01:05.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.600+0000 7fece74d0640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7feccc0a1c20 0x7feccc0a40e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.600+0000 7fece74d0640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fece010db20 0x7fece0199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.600+0000 7fece74d0640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece0068df0 0x7fece019fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.600+0000 7fece74d0640 1 --2- 172.21.15.76:0/3583112381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fece00684d0 0x7fece019f6a0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:05.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.600+0000 7fece74d0640 1 -- 172.21.15.76:0/3583112381 >> 172.21.15.76:0/3583112381 conn(0x7fece0077180 msgr2=0x7fece010ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:05.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.600+0000 7fece74d0640 1 -- 172.21.15.76:0/3583112381 shutdown_connections 2024-03-12T06:01:05.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:05.600+0000 7fece74d0640 1 -- 172.21.15.76:0/3583112381 wait complete. 2024-03-12T06:01:05.617 INFO:tasks.cephfs.filesystem:scrub status for tag:e9634bfa-54f8-4262-9f1b-0f662e28cfd1 - {'path': '/', 'tag': 'e9634bfa-54f8-4262-9f1b-0f662e28cfd1', 'options': 'recursive,force'} 2024-03-12T06:01:06.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:06 smithi082 ceph-mon[36382]: pgmap v162: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:01:06.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:06 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:01:06.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:06 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:01:06.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:06 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:06.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:06 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:06.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:06 smithi088 ceph-mon[36585]: pgmap v162: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:01:06.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:06 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:01:06.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:06 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:01:06.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:06 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:06.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:06 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:06.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:06 smithi076 ceph-mon[32095]: pgmap v162: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:01:06.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:06 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:01:06.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:06 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:01:06.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:06 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:06.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:06 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:07.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:07 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T06:01:07.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:07 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:07.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:07 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:01:07.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:07 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:07.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:07 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T06:01:07.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:07 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:07.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:07 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:01:07.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:07 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:07.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:07 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T06:01:07.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:07 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:07.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:07 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:01:07.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:07 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:08.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:08 smithi082 ceph-mon[36382]: pgmap v163: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:01:08.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:08 smithi088 ceph-mon[36585]: pgmap v163: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:01:08.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:08 smithi076 ceph-mon[32095]: pgmap v163: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:01:09.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:09 smithi082 ceph-mon[36382]: pgmap v164: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:01:09.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:09 smithi088 ceph-mon[36585]: pgmap v164: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:01:09.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:09 smithi076 ceph-mon[32095]: pgmap v164: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:01:12.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:12 smithi082 ceph-mon[36382]: pgmap v165: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:01:12.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:12 smithi088 ceph-mon[36585]: pgmap v165: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:01:12.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:12 smithi076 ceph-mon[32095]: pgmap v165: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:01:14.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:14 smithi082 ceph-mon[36382]: pgmap v166: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:01:14.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:14 smithi088 ceph-mon[36585]: pgmap v166: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:01:14.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:14 smithi076 ceph-mon[32095]: pgmap v166: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:01:16.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:16 smithi082 ceph-mon[36382]: pgmap v167: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:01:16.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:16 smithi088 ceph-mon[36585]: pgmap v167: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:01:16.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:16 smithi076 ceph-mon[32095]: pgmap v167: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:01:18.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:18 smithi082 ceph-mon[36382]: pgmap v168: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:01:18.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:18 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:01:18.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:18 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:01:18.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:18 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:01:18.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:18 smithi088 ceph-mon[36585]: pgmap v168: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:01:18.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:18 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:01:18.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:18 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:01:18.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:18 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:01:18.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:18 smithi076 ceph-mon[32095]: pgmap v168: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:01:18.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:18 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:01:18.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:18 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:01:18.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:18 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:01:20.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:20 smithi082 ceph-mon[36382]: pgmap v169: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-12T06:01:20.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:20 smithi088 ceph-mon[36585]: pgmap v169: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-12T06:01:20.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:20 smithi076 ceph-mon[32095]: pgmap v169: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-12T06:01:22.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:22 smithi082 ceph-mon[36382]: pgmap v170: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:22.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:22 smithi088 ceph-mon[36585]: pgmap v170: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:22.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:22 smithi076 ceph-mon[32095]: pgmap v170: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:24.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:24 smithi082 ceph-mon[36382]: pgmap v171: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:24.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:24 smithi088 ceph-mon[36585]: pgmap v171: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:24.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:24 smithi076 ceph-mon[32095]: pgmap v171: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:25 smithi088 ceph-mon[36585]: pgmap v172: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:25.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:25 smithi076 ceph-mon[32095]: pgmap v172: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:25.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:25 smithi082 ceph-mon[36382]: pgmap v172: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:28.399 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:28 smithi082 ceph-mon[36382]: pgmap v173: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:28.458 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:28 smithi088 ceph-mon[36585]: pgmap v173: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:28.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:28 smithi076 ceph-mon[32095]: pgmap v173: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:30.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:30 smithi082 ceph-mon[36382]: pgmap v174: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:30 smithi088 ceph-mon[36585]: pgmap v174: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:30.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:30 smithi076 ceph-mon[32095]: pgmap v174: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:32.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:32 smithi082 ceph-mon[36382]: pgmap v175: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:32.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:32 smithi088 ceph-mon[36585]: pgmap v175: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:32.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:32 smithi076 ceph-mon[32095]: pgmap v175: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:34.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:34 smithi082 ceph-mon[36382]: pgmap v176: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:34 smithi088 ceph-mon[36585]: pgmap v176: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:34.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:34 smithi076 ceph-mon[32095]: pgmap v176: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:35.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:35 smithi082 ceph-mon[36382]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T06:01:35.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:35 smithi088 ceph-mon[36585]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T06:01:35.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:35 smithi076 ceph-mon[32095]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 3 up:standby-replay 6 up:standby 2024-03-12T06:01:35.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:01:35.619 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:01:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.727+0000 7f4536fd7640 1 -- 172.21.15.76:0/111829812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4530100280 msgr2=0x7f4530100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.727+0000 7f4536fd7640 1 --2- 172.21.15.76:0/111829812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4530100280 0x7f4530100660 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4520009930 tx=0x7f452002bc30 comp rx=0 tx=0).stop 2024-03-12T06:01:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.727+0000 7f4536fd7640 1 -- 172.21.15.76:0/111829812 shutdown_connections 2024-03-12T06:01:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.727+0000 7f4536fd7640 1 --2- 172.21.15.76:0/111829812 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4530105340 0x7f453010cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.727+0000 7f4536fd7640 1 --2- 172.21.15.76:0/111829812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4530100c30 0x7f4530104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.727+0000 7f4536fd7640 1 --2- 172.21.15.76:0/111829812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4530100280 0x7f4530100660 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.727+0000 7f4536fd7640 1 -- 172.21.15.76:0/111829812 >> 172.21.15.76:0/111829812 conn(0x7f4530077120 msgr2=0x7f4530077530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.727+0000 7f4536fd7640 1 -- 172.21.15.76:0/111829812 shutdown_connections 2024-03-12T06:01:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.727+0000 7f4536fd7640 1 -- 172.21.15.76:0/111829812 wait complete. 2024-03-12T06:01:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f4536fd7640 1 Processor -- start 2024-03-12T06:01:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f4536fd7640 1 -- start start 2024-03-12T06:01:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f4536fd7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4530100280 0x7f453019d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f4536fd7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4530105340 0x7f453019db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f4536fd7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4530197680 0x7f4530197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f4536fd7640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4530198150 con 0x7f4530105340 2024-03-12T06:01:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f4536fd7640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45301982c0 con 0x7f4530197680 2024-03-12T06:01:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f4536fd7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4530198480 con 0x7f4530100280 2024-03-12T06:01:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f452ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4530105340 0x7f453019db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f452ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4530105340 0x7f453019db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38864/0 (socket says 172.21.15.76:38864) 2024-03-12T06:01:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f452ffff640 1 -- 172.21.15.76:0/653695757 learned_addr learned my addr 172.21.15.76:0/653695757 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f4534d73640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4530100280 0x7f453019d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f4535574640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4530197680 0x7f4530197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f452ffff640 1 -- 172.21.15.76:0/653695757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4530100280 msgr2=0x7f453019d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f452ffff640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4530100280 0x7f453019d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f452ffff640 1 -- 172.21.15.76:0/653695757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4530197680 msgr2=0x7f4530197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f452ffff640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4530197680 0x7f4530197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f452ffff640 1 -- 172.21.15.76:0/653695757 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4520009560 con 0x7f4530105340 2024-03-12T06:01:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.728+0000 7f4535574640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4530197680 0x7f4530197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.729+0000 7f452ffff640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4530105340 0x7f453019db00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f4524005fd0 tx=0x7f452400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.729+0000 7f452dffb640 1 -- 172.21.15.76:0/653695757 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4524009c30 con 0x7f4530105340 2024-03-12T06:01:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.729+0000 7f452dffb640 1 -- 172.21.15.76:0/653695757 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4524009d90 con 0x7f4530105340 2024-03-12T06:01:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.729+0000 7f4536fd7640 1 -- 172.21.15.76:0/653695757 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45301a4420 con 0x7f4530105340 2024-03-12T06:01:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.729+0000 7f452dffb640 1 -- 172.21.15.76:0/653695757 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4524010460 con 0x7f4530105340 2024-03-12T06:01:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.729+0000 7f4536fd7640 1 -- 172.21.15.76:0/653695757 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45301a4900 con 0x7f4530105340 2024-03-12T06:01:35.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.731+0000 7f452dffb640 1 -- 172.21.15.76:0/653695757 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4524003f20 con 0x7f4530105340 2024-03-12T06:01:35.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.731+0000 7f452dffb640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f45180a1ba0 0x7f45180a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.731+0000 7f4539baa740 1 Processor -- start 2024-03-12T06:01:35.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.731+0000 7f4539baa740 1 -- start start 2024-03-12T06:01:35.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.731+0000 7f4534d73640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f45180a1ba0 0x7f45180a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f452dffb640 1 -- 172.21.15.76:0/653695757 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f45240cd450 con 0x7f4530105340 2024-03-12T06:01:35.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4539baa740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac172b0 0x55fe7ac17690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4539baa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac20270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4539baa740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac207b0 0x55fe7ac22ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4539baa740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7ab86280 con 0x55fe7ac172b0 2024-03-12T06:01:35.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4539baa740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7ab837b0 con 0x55fe7ac207b0 2024-03-12T06:01:35.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4539baa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7ab7ef90 con 0x55fe7ac17bd0 2024-03-12T06:01:35.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4534d73640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac172b0 0x55fe7ac17690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4534d73640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac172b0 0x55fe7ac17690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38872/0 (socket says 172.21.15.76:38872) 2024-03-12T06:01:35.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4534d73640 1 -- 172.21.15.76:0/4266244922 learned_addr learned my addr 172.21.15.76:0/4266244922 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:35.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f452ffff640 1 --2- 172.21.15.76:0/4266244922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac20270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4535574640 1 --2- 172.21.15.76:0/4266244922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac207b0 0x55fe7ac22ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4534d73640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f45180a1ba0 0x7f45180a4060 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4520036040 tx=0x7f4520004a80 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:35.749 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4534d73640 1 -- 172.21.15.76:0/4266244922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac17bd0 msgr2=0x55fe7ac20270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:35.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4534d73640 1 --2- 172.21.15.76:0/4266244922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac20270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4534d73640 1 -- 172.21.15.76:0/4266244922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac207b0 msgr2=0x55fe7ac22ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:35.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4534d73640 1 --2- 172.21.15.76:0/4266244922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac207b0 0x55fe7ac22ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4534d73640 1 -- 172.21.15.76:0/4266244922 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe7ab3d570 con 0x55fe7ac172b0 2024-03-12T06:01:35.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4534d73640 1 --2- 172.21.15.76:0/4266244922 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac172b0 0x55fe7ac17690 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4520035000 tx=0x7f452004b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8508ab25e417fd57 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:35.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.732+0000 7f4516ffd640 1 -- 172.21.15.76:0/4266244922 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4520004180 con 0x55fe7ac172b0 2024-03-12T06:01:35.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.733+0000 7f4516ffd640 1 -- 172.21.15.76:0/4266244922 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45200042e0 con 0x55fe7ac172b0 2024-03-12T06:01:35.750 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.733+0000 7f4516ffd640 1 -- 172.21.15.76:0/4266244922 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4520003990 con 0x55fe7ac172b0 2024-03-12T06:01:35.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.733+0000 7f4539baa740 1 -- 172.21.15.76:0/4266244922 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac172b0 msgr2=0x55fe7ac17690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:35.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.733+0000 7f4539baa740 1 --2- 172.21.15.76:0/4266244922 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac172b0 0x55fe7ac17690 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4520035000 tx=0x7f452004b000 comp rx=0 tx=0).stop 2024-03-12T06:01:35.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.733+0000 7f4539baa740 1 -- 172.21.15.76:0/4266244922 shutdown_connections 2024-03-12T06:01:35.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.733+0000 7f4539baa740 1 --2- 172.21.15.76:0/4266244922 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac207b0 0x55fe7ac22ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.733+0000 7f4539baa740 1 --2- 172.21.15.76:0/4266244922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac20270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.733+0000 7f4539baa740 1 --2- 172.21.15.76:0/4266244922 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac172b0 0x55fe7ac17690 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.733+0000 7f4539baa740 1 -- 172.21.15.76:0/4266244922 >> 172.21.15.76:0/4266244922 conn(0x55fe7ac11990 msgr2=0x55fe7ac11d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:35.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.733+0000 7f4539baa740 1 -- 172.21.15.76:0/4266244922 shutdown_connections 2024-03-12T06:01:35.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.733+0000 7f4539baa740 1 -- 172.21.15.76:0/4266244922 wait complete. 2024-03-12T06:01:35.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4539baa740 1 -- start start 2024-03-12T06:01:35.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4539baa740 1 Processor -- start 2024-03-12T06:01:35.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4539baa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac172b0 0x55fe7ac20230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4539baa740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac1d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4539baa740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac207b0 0x55fe7ac1dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4539baa740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7a852430 con 0x55fe7ac207b0 2024-03-12T06:01:35.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4539baa740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7a80ee00 con 0x55fe7ac17bd0 2024-03-12T06:01:35.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4539baa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7ab03fd0 con 0x55fe7ac172b0 2024-03-12T06:01:35.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4539baa740 10 client.0 _ll_register_callbacks cb 0x55fe7ab59fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:35.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4535574640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac207b0 0x55fe7ac1dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4535574640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac207b0 0x55fe7ac1dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38888/0 (socket says 172.21.15.76:38888) 2024-03-12T06:01:35.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4535574640 1 -- 172.21.15.76:0/1682490171 learned_addr learned my addr 172.21.15.76:0/1682490171 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:35.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4534d73640 1 --2- 172.21.15.76:0/1682490171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac172b0 0x55fe7ac20230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f452ffff640 1 --2- 172.21.15.76:0/1682490171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac1d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4535574640 1 -- 172.21.15.76:0/1682490171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac172b0 msgr2=0x55fe7ac20230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:35.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4535574640 1 --2- 172.21.15.76:0/1682490171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac172b0 0x55fe7ac20230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4535574640 1 -- 172.21.15.76:0/1682490171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac17bd0 msgr2=0x55fe7ac1d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:35.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4535574640 1 --2- 172.21.15.76:0/1682490171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac1d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.734+0000 7f4535574640 1 -- 172.21.15.76:0/1682490171 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4520049050 con 0x55fe7ac207b0 2024-03-12T06:01:35.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f4535574640 1 --2- 172.21.15.76:0/1682490171 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac207b0 0x55fe7ac1dd00 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f4528011d10 tx=0x7f45280107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf3af0587033612 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:35.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f450ffff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:35.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f450ffff640 1 -- 172.21.15.76:0/1682490171 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4528016070 con 0x55fe7ac207b0 2024-03-12T06:01:35.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f450ffff640 1 -- 172.21.15.76:0/1682490171 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4528009e30 con 0x55fe7ac207b0 2024-03-12T06:01:35.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f4539baa740 1 -- 172.21.15.76:0/1682490171 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe7a87e7e0 con 0x55fe7ac207b0 2024-03-12T06:01:35.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f450ffff640 1 -- 172.21.15.76:0/1682490171 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4528004b20 con 0x55fe7ac207b0 2024-03-12T06:01:35.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f4539baa740 10 client.16791 fetch_fsmap learned FSMap version 28 2024-03-12T06:01:35.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f4539baa740 1 -- 172.21.15.76:0/1682490171 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fe7ab7d610 con 0x55fe7ac207b0 2024-03-12T06:01:35.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f450ffff640 1 -- 172.21.15.76:0/1682490171 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55fe7a87e7e0 con 0x55fe7ac207b0 2024-03-12T06:01:35.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f450ffff640 1 -- 172.21.15.76:0/1682490171 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f452800e5d0 con 0x55fe7ac207b0 2024-03-12T06:01:35.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f4539baa740 10 client.16791 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:01:35.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f4539baa740 10 client.16791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:35.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f4539baa740 20 client.16791 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:35.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f4539baa740 1 --2- 172.21.15.76:0/1682490171 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fe7ac30980 0x55fe7ac32da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f4539baa740 4 client.16791 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:01:35.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.735+0000 7f4539baa740 1 -- 172.21.15.76:0/1682490171 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe7a8bb720 con 0x55fe7ac30980 2024-03-12T06:01:35.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.736+0000 7f4534d73640 1 --2- 172.21.15.76:0/1682490171 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fe7ac30980 0x55fe7ac32da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.736+0000 7f4534d73640 1 --2- 172.21.15.76:0/1682490171 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fe7ac30980 0x55fe7ac32da0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec9da73c922a11d5 server_cookie=d16467a86814977b in_seq=0 out_seq=0 2024-03-12T06:01:35.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.736+0000 7f450ffff640 10 client.16791 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:35.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f450ffff640 1 -- 172.21.15.76:0/1682490171 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fe7a8bb720 con 0x55fe7ac30980 2024-03-12T06:01:35.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f450ffff640 10 client.16791 handle_command_reply: tid=0 2024-03-12T06:01:35.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f4539baa740 1 client.16791 shutdown 2024-03-12T06:01:35.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f4539baa740 1 -- 172.21.15.76:0/1682490171 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac207b0 msgr2=0x55fe7ac1dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:35.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f4539baa740 1 --2- 172.21.15.76:0/1682490171 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac207b0 0x55fe7ac1dd00 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f4528011d10 tx=0x7f45280107a0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f4539baa740 1 -- 172.21.15.76:0/1682490171 shutdown_connections 2024-03-12T06:01:35.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f4539baa740 1 --2- 172.21.15.76:0/1682490171 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fe7ac30980 0x55fe7ac32da0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f4539baa740 1 --2- 172.21.15.76:0/1682490171 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac207b0 0x55fe7ac1dd00 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f4539baa740 1 --2- 172.21.15.76:0/1682490171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac1d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f4539baa740 1 --2- 172.21.15.76:0/1682490171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac172b0 0x55fe7ac20230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f450ffff640 0 client.16791 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:35.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f4539baa740 1 -- 172.21.15.76:0/1682490171 >> 172.21.15.76:0/1682490171 conn(0x55fe7ac11990 msgr2=0x55fe7ac13730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:35.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f4539baa740 1 -- 172.21.15.76:0/1682490171 shutdown_connections 2024-03-12T06:01:35.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f4539baa740 1 -- 172.21.15.76:0/1682490171 wait complete. 2024-03-12T06:01:35.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.737+0000 7f4539baa740 20 client.16791 trim_cache size 0 max 16384 2024-03-12T06:01:35.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.752+0000 7f4539baa740 1 Processor -- start 2024-03-12T06:01:35.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.752+0000 7f4539baa740 1 -- start start 2024-03-12T06:01:35.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.752+0000 7f4539baa740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac172b0 0x55fe7ac1fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.752+0000 7f4539baa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac20230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.752+0000 7f4539baa740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac207b0 0x55fe7ac275a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.752+0000 7f4539baa740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7ac22fa0 con 0x55fe7ac172b0 2024-03-12T06:01:35.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.752+0000 7f4539baa740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7ac16cf0 con 0x55fe7ac207b0 2024-03-12T06:01:35.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.752+0000 7f4539baa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7ac129e0 con 0x55fe7ac17bd0 2024-03-12T06:01:35.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.752+0000 7f4534d73640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac172b0 0x55fe7ac1fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.752+0000 7f4534d73640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac172b0 0x55fe7ac1fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38894/0 (socket says 172.21.15.76:38894) 2024-03-12T06:01:35.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.752+0000 7f4534d73640 1 -- 172.21.15.76:0/2512463613 learned_addr learned my addr 172.21.15.76:0/2512463613 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:35.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.753+0000 7f4534d73640 1 -- 172.21.15.76:0/2512463613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac17bd0 msgr2=0x55fe7ac20230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:35.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.753+0000 7f4534d73640 1 --2- 172.21.15.76:0/2512463613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac20230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.753+0000 7f4534d73640 1 -- 172.21.15.76:0/2512463613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac207b0 msgr2=0x55fe7ac275a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:35.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.753+0000 7f4534d73640 1 --2- 172.21.15.76:0/2512463613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac207b0 0x55fe7ac275a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.753+0000 7f4534d73640 1 -- 172.21.15.76:0/2512463613 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe7ab19ac0 con 0x55fe7ac172b0 2024-03-12T06:01:35.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.753+0000 7f4534d73640 1 --2- 172.21.15.76:0/2512463613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac172b0 0x55fe7ac1fcf0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f45200056a0 tx=0x7f45200056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f407f39a9eaba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:35.768 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.753+0000 7f45167fc640 1 -- 172.21.15.76:0/2512463613 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4520004180 con 0x55fe7ac172b0 2024-03-12T06:01:35.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.753+0000 7f45167fc640 1 -- 172.21.15.76:0/2512463613 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4520004450 con 0x55fe7ac172b0 2024-03-12T06:01:35.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.753+0000 7f45167fc640 1 -- 172.21.15.76:0/2512463613 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4520003d90 con 0x55fe7ac172b0 2024-03-12T06:01:35.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 -- 172.21.15.76:0/2512463613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac172b0 msgr2=0x55fe7ac1fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:35.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 --2- 172.21.15.76:0/2512463613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac172b0 0x55fe7ac1fcf0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f45200056a0 tx=0x7f45200056d0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 -- 172.21.15.76:0/2512463613 shutdown_connections 2024-03-12T06:01:35.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 --2- 172.21.15.76:0/2512463613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac207b0 0x55fe7ac275a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 --2- 172.21.15.76:0/2512463613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac20230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 --2- 172.21.15.76:0/2512463613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac172b0 0x55fe7ac1fcf0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 -- 172.21.15.76:0/2512463613 >> 172.21.15.76:0/2512463613 conn(0x55fe7ac11990 msgr2=0x55fe7ac188b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:35.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 -- 172.21.15.76:0/2512463613 shutdown_connections 2024-03-12T06:01:35.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 -- 172.21.15.76:0/2512463613 wait complete. 2024-03-12T06:01:35.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 -- start start 2024-03-12T06:01:35.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 Processor -- start 2024-03-12T06:01:35.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac172b0 0x55fe7ac277e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac25b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac207b0 0x55fe7ac260c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7ac26680 con 0x55fe7ac207b0 2024-03-12T06:01:35.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7ac267f0 con 0x55fe7ac17bd0 2024-03-12T06:01:35.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe7ac269b0 con 0x55fe7ac172b0 2024-03-12T06:01:35.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4539baa740 10 client.0 _ll_register_callbacks cb 0x7f4520009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:35.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4535574640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac207b0 0x55fe7ac260c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4535574640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac207b0 0x55fe7ac260c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38904/0 (socket says 172.21.15.76:38904) 2024-03-12T06:01:35.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.754+0000 7f4535574640 1 -- 172.21.15.76:0/1678775993 learned_addr learned my addr 172.21.15.76:0/1678775993 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:35.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f452ffff640 1 --2- 172.21.15.76:0/1678775993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac25b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f4535574640 1 -- 172.21.15.76:0/1678775993 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac172b0 msgr2=0x55fe7ac277e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:35.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f4535574640 1 --2- 172.21.15.76:0/1678775993 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac172b0 0x55fe7ac277e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f4535574640 1 -- 172.21.15.76:0/1678775993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac17bd0 msgr2=0x55fe7ac25b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:35.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f4535574640 1 --2- 172.21.15.76:0/1678775993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac25b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f4535574640 1 -- 172.21.15.76:0/1678775993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe7ab3d570 con 0x55fe7ac207b0 2024-03-12T06:01:35.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f4535574640 1 --2- 172.21.15.76:0/1678775993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac207b0 0x55fe7ac260c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f4528006010 tx=0x7f4528010ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66cf356040373f73 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:35.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f450ffff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:35.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f450ffff640 1 -- 172.21.15.76:0/1678775993 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4528016070 con 0x55fe7ac207b0 2024-03-12T06:01:35.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f450ffff640 1 -- 172.21.15.76:0/1678775993 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f452800eba0 con 0x55fe7ac207b0 2024-03-12T06:01:35.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f4539baa740 1 -- 172.21.15.76:0/1678775993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45200042e0 con 0x55fe7ac207b0 2024-03-12T06:01:35.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f450ffff640 1 -- 172.21.15.76:0/1678775993 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4528009a70 con 0x55fe7ac207b0 2024-03-12T06:01:35.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f4539baa740 10 client.16797 fetch_fsmap learned FSMap version 28 2024-03-12T06:01:35.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f4539baa740 1 -- 172.21.15.76:0/1678775993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fe7ab3f6a0 con 0x55fe7ac207b0 2024-03-12T06:01:35.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f450ffff640 1 -- 172.21.15.76:0/1678775993 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45200042e0 con 0x55fe7ac207b0 2024-03-12T06:01:35.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.755+0000 7f450ffff640 1 -- 172.21.15.76:0/1678775993 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f45280050c0 con 0x55fe7ac207b0 2024-03-12T06:01:35.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 10 client.16797 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:01:35.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 10 client.16797 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:35.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 20 client.16797 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:35.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 1 --2- 172.21.15.76:0/1678775993 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fe7ac8c840 0x55fe7ac8ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 4 client.16797 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:01:35.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 1 -- 172.21.15.76:0/1678775993 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fe7a90a890 con 0x55fe7ac8c840 2024-03-12T06:01:35.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4534d73640 1 --2- 172.21.15.76:0/1678775993 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fe7ac8c840 0x55fe7ac8ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f450ffff640 10 client.16797 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:35.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4534d73640 1 --2- 172.21.15.76:0/1678775993 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fe7ac8c840 0x55fe7ac8ec60 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75549e6137836d48 server_cookie=38ddd85075d27666 in_seq=0 out_seq=0 2024-03-12T06:01:35.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f450ffff640 1 -- 172.21.15.76:0/1678775993 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55fe7a90a890 con 0x55fe7ac8c840 2024-03-12T06:01:35.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f450ffff640 10 client.16797 handle_command_reply: tid=0 2024-03-12T06:01:35.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 1 client.16797 shutdown 2024-03-12T06:01:35.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 1 -- 172.21.15.76:0/1678775993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac207b0 msgr2=0x55fe7ac260c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:35.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 1 --2- 172.21.15.76:0/1678775993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac207b0 0x55fe7ac260c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f4528006010 tx=0x7f4528010ef0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 1 -- 172.21.15.76:0/1678775993 shutdown_connections 2024-03-12T06:01:35.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 1 --2- 172.21.15.76:0/1678775993 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fe7ac8c840 0x55fe7ac8ec60 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 1 --2- 172.21.15.76:0/1678775993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fe7ac207b0 0x55fe7ac260c0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 1 --2- 172.21.15.76:0/1678775993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe7ac17bd0 0x55fe7ac25b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f450ffff640 0 client.16797 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:35.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 1 --2- 172.21.15.76:0/1678775993 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe7ac172b0 0x55fe7ac277e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.756+0000 7f4539baa740 1 -- 172.21.15.76:0/1678775993 >> 172.21.15.76:0/1678775993 conn(0x55fe7ac11990 msgr2=0x55fe7ac101c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:35.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4539baa740 1 -- 172.21.15.76:0/1678775993 shutdown_connections 2024-03-12T06:01:35.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4539baa740 1 -- 172.21.15.76:0/1678775993 wait complete. 2024-03-12T06:01:35.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4539baa740 20 client.16797 trim_cache size 0 max 16384 2024-03-12T06:01:35.776 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:01:35.777 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T06:01:35.777 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T06:01:35.777 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:01:35.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4536fd7640 1 -- 172.21.15.76:0/653695757 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f45180a1ba0 msgr2=0x7f45180a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:35.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4536fd7640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f45180a1ba0 0x7f45180a4060 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4520036040 tx=0x7f4520004a80 comp rx=0 tx=0).stop 2024-03-12T06:01:35.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4536fd7640 1 -- 172.21.15.76:0/653695757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4530105340 msgr2=0x7f453019db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:35.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4536fd7640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4530105340 0x7f453019db00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f4524005fd0 tx=0x7f452400dde0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4536fd7640 1 -- 172.21.15.76:0/653695757 shutdown_connections 2024-03-12T06:01:35.779 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4536fd7640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f45180a1ba0 0x7f45180a4060 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4536fd7640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4530197680 0x7f4530197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4536fd7640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4530105340 0x7f453019db00 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4536fd7640 1 --2- 172.21.15.76:0/653695757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4530100280 0x7f453019d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4536fd7640 1 -- 172.21.15.76:0/653695757 >> 172.21.15.76:0/653695757 conn(0x7f4530077120 msgr2=0x7f4530069b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:35.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4536fd7640 1 -- 172.21.15.76:0/653695757 shutdown_connections 2024-03-12T06:01:35.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.757+0000 7f4536fd7640 1 -- 172.21.15.76:0/653695757 wait complete. 2024-03-12T06:01:35.780 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T06:01:35.781 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T06:01:35.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.878+0000 7f912d839640 1 -- 172.21.15.76:0/2515522826 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9128101d50 msgr2=0x7f9128105d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:35.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.878+0000 7f912d839640 1 --2- 172.21.15.76:0/2515522826 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9128101d50 0x7f9128105d90 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f911c009930 tx=0x7f911c02bc30 comp rx=0 tx=0).stop 2024-03-12T06:01:35.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.878+0000 7f912d839640 1 -- 172.21.15.76:0/2515522826 shutdown_connections 2024-03-12T06:01:35.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.878+0000 7f912d839640 1 --2- 172.21.15.76:0/2515522826 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9128106470 0x7f912810e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.878+0000 7f912d839640 1 --2- 172.21.15.76:0/2515522826 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9128101d50 0x7f9128105d90 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.878+0000 7f912d839640 1 --2- 172.21.15.76:0/2515522826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91281013a0 0x7f9128101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.878+0000 7f912d839640 1 -- 172.21.15.76:0/2515522826 >> 172.21.15.76:0/2515522826 conn(0x7f91280fcc40 msgr2=0x7f91280ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:35.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.879+0000 7f912d839640 1 -- 172.21.15.76:0/2515522826 shutdown_connections 2024-03-12T06:01:35.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.879+0000 7f912d839640 1 -- 172.21.15.76:0/2515522826 wait complete. 2024-03-12T06:01:35.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.879+0000 7f912d839640 1 Processor -- start 2024-03-12T06:01:35.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.879+0000 7f912d839640 1 -- start start 2024-03-12T06:01:35.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.879+0000 7f912d839640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91281013a0 0x7f912819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.879+0000 7f912d839640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9128106470 0x7f912819dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.879+0000 7f912d839640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9128197630 0x7f9128197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.879+0000 7f912d839640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9128198100 con 0x7f9128197630 2024-03-12T06:01:35.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.879+0000 7f912d839640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9128198270 con 0x7f9128106470 2024-03-12T06:01:35.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.879+0000 7f912d839640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9128198430 con 0x7f91281013a0 2024-03-12T06:01:35.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f91277fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9128197630 0x7f9128197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f91277fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9128197630 0x7f9128197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38918/0 (socket says 172.21.15.76:38918) 2024-03-12T06:01:35.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f91277fe640 1 -- 172.21.15.76:0/4069607309 learned_addr learned my addr 172.21.15.76:0/4069607309 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:35.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f91267fc640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9128106470 0x7f912819dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f9126ffd640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91281013a0 0x7f912819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f91277fe640 1 -- 172.21.15.76:0/4069607309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91281013a0 msgr2=0x7f912819d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:35.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f91277fe640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91281013a0 0x7f912819d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f91277fe640 1 -- 172.21.15.76:0/4069607309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9128106470 msgr2=0x7f912819dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:35.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f91277fe640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9128106470 0x7f912819dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:35.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f91277fe640 1 -- 172.21.15.76:0/4069607309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c009560 con 0x7f9128197630 2024-03-12T06:01:35.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f9126ffd640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91281013a0 0x7f912819d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:35.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f91277fe640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9128197630 0x7f9128197ab0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f912000d750 tx=0x7f912000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:35.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f912c837640 1 -- 172.21.15.76:0/4069607309 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120009bd0 con 0x7f9128197630 2024-03-12T06:01:35.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f912c837640 1 -- 172.21.15.76:0/4069607309 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f912000de60 con 0x7f9128197630 2024-03-12T06:01:35.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f912c837640 1 -- 172.21.15.76:0/4069607309 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120004e70 con 0x7f9128197630 2024-03-12T06:01:35.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f912d839640 1 -- 172.21.15.76:0/4069607309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91281a43d0 con 0x7f9128197630 2024-03-12T06:01:35.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.880+0000 7f912d839640 1 -- 172.21.15.76:0/4069607309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91281a4860 con 0x7f9128197630 2024-03-12T06:01:35.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.881+0000 7f912d839640 1 -- 172.21.15.76:0/4069607309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9128102870 con 0x7f9128197630 2024-03-12T06:01:35.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.882+0000 7f912c837640 1 -- 172.21.15.76:0/4069607309 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9120005000 con 0x7f9128197630 2024-03-12T06:01:35.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.883+0000 7f912c837640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f91100a1ba0 0x7f91100a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:35.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.883+0000 7f9126ffd640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f91100a1ba0 0x7f91100a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:35.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.883+0000 7f912c837640 1 -- 172.21.15.76:0/4069607309 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f91200cd530 con 0x7f9128197630 2024-03-12T06:01:35.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.883+0000 7f9126ffd640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f91100a1ba0 0x7f91100a4060 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9118005fd0 tx=0x7f9118009520 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:35.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:35.887+0000 7f912c837640 1 -- 172.21.15.76:0/4069607309 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9120099f60 con 0x7f9128197630 2024-03-12T06:01:36.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.096+0000 7f912d839640 1 -- 172.21.15.76:0/4069607309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91281096a0 con 0x7f9128197630 2024-03-12T06:01:36.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.097+0000 7f912c837640 1 -- 172.21.15.76:0/4069607309 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11328 (secure 0 0 0) 0x7f9120017020 con 0x7f9128197630 2024-03-12T06:01:36.103 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:01:36.103 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T06:01:33.842001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T06:01:36.104 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 28 2024-03-12T06:01:36.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.100+0000 7f912d839640 1 -- 172.21.15.76:0/4069607309 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f91100a1ba0 msgr2=0x7f91100a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.100+0000 7f912d839640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f91100a1ba0 0x7f91100a4060 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9118005fd0 tx=0x7f9118009520 comp rx=0 tx=0).stop 2024-03-12T06:01:36.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.100+0000 7f912d839640 1 -- 172.21.15.76:0/4069607309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9128197630 msgr2=0x7f9128197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.100+0000 7f912d839640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9128197630 0x7f9128197ab0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f912000d750 tx=0x7f912000dc10 comp rx=0 tx=0).stop 2024-03-12T06:01:36.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.100+0000 7f912d839640 1 -- 172.21.15.76:0/4069607309 shutdown_connections 2024-03-12T06:01:36.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.100+0000 7f912d839640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f91100a1ba0 0x7f91100a4060 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.100+0000 7f912d839640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9128197630 0x7f9128197ab0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.100+0000 7f912d839640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9128106470 0x7f912819dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.100+0000 7f912d839640 1 --2- 172.21.15.76:0/4069607309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91281013a0 0x7f912819d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.100+0000 7f912d839640 1 -- 172.21.15.76:0/4069607309 >> 172.21.15.76:0/4069607309 conn(0x7f91280fcc40 msgr2=0x7f91280fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.100+0000 7f912d839640 1 -- 172.21.15.76:0/4069607309 shutdown_connections 2024-03-12T06:01:36.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.100+0000 7f912d839640 1 -- 172.21.15.76:0/4069607309 wait complete. 2024-03-12T06:01:36.116 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:01:36.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T06:01:36.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.217+0000 7fda9dd48640 1 -- 172.21.15.76:0/961742644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda98100010 msgr2=0x7fda9810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.217+0000 7fda9dd48640 1 --2- 172.21.15.76:0/961742644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda98100010 0x7fda9810dfe0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fda90009cd0 tx=0x7fda9002c0f0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.217+0000 7fda9dd48640 1 -- 172.21.15.76:0/961742644 shutdown_connections 2024-03-12T06:01:36.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.217+0000 7fda9dd48640 1 --2- 172.21.15.76:0/961742644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda98100010 0x7fda9810dfe0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.217+0000 7fda9dd48640 1 --2- 172.21.15.76:0/961742644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda980ff650 0x7fda980ffad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.217+0000 7fda9dd48640 1 --2- 172.21.15.76:0/961742644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda98106150 0x7fda98106530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.217+0000 7fda9dd48640 1 -- 172.21.15.76:0/961742644 >> 172.21.15.76:0/961742644 conn(0x7fda98076580 msgr2=0x7fda980fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.217+0000 7fda9dd48640 1 -- 172.21.15.76:0/961742644 shutdown_connections 2024-03-12T06:01:36.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.217+0000 7fda9dd48640 1 -- 172.21.15.76:0/961742644 wait complete. 2024-03-12T06:01:36.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.217+0000 7fda9dd48640 1 Processor -- start 2024-03-12T06:01:36.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda9dd48640 1 -- start start 2024-03-12T06:01:36.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda9dd48640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda980ff650 0x7fda9819d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda9dd48640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda98100010 0x7fda9819d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda9dd48640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda98106150 0x7fda98197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda9dd48640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda98197cc0 con 0x7fda980ff650 2024-03-12T06:01:36.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda9dd48640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda98197e30 con 0x7fda98106150 2024-03-12T06:01:36.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda9dd48640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda98197ff0 con 0x7fda98100010 2024-03-12T06:01:36.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda977fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda980ff650 0x7fda9819d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda977fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda980ff650 0x7fda9819d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38936/0 (socket says 172.21.15.76:38936) 2024-03-12T06:01:36.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda977fe640 1 -- 172.21.15.76:0/3177891129 learned_addr learned my addr 172.21.15.76:0/3177891129 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda96ffd640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda98100010 0x7fda9819d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda97fff640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda98106150 0x7fda98197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda977fe640 1 -- 172.21.15.76:0/3177891129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda98100010 msgr2=0x7fda9819d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda977fe640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda98100010 0x7fda9819d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda977fe640 1 -- 172.21.15.76:0/3177891129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda98106150 msgr2=0x7fda98197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda977fe640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda98106150 0x7fda98197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda977fe640 1 -- 172.21.15.76:0/3177891129 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda90009900 con 0x7fda980ff650 2024-03-12T06:01:36.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda97fff640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda98106150 0x7fda98197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:36.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.218+0000 7fda977fe640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda980ff650 0x7fda9819d470 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fda8800d770 tx=0x7fda8800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.219+0000 7fda94ff9640 1 -- 172.21.15.76:0/3177891129 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda88009c30 con 0x7fda980ff650 2024-03-12T06:01:36.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.219+0000 7fda94ff9640 1 -- 172.21.15.76:0/3177891129 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda8800de80 con 0x7fda980ff650 2024-03-12T06:01:36.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.219+0000 7fda9dd48640 1 -- 172.21.15.76:0/3177891129 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda981982d0 con 0x7fda980ff650 2024-03-12T06:01:36.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.219+0000 7fda94ff9640 1 -- 172.21.15.76:0/3177891129 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda88004e80 con 0x7fda980ff650 2024-03-12T06:01:36.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.219+0000 7fda9dd48640 1 -- 172.21.15.76:0/3177891129 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda981a4510 con 0x7fda980ff650 2024-03-12T06:01:36.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.221+0000 7fda94ff9640 1 -- 172.21.15.76:0/3177891129 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fda88005020 con 0x7fda980ff650 2024-03-12T06:01:36.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.221+0000 7fdaa09fa740 1 Processor -- start 2024-03-12T06:01:36.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.221+0000 7fdaa09fa740 1 -- start start 2024-03-12T06:01:36.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.221+0000 7fdaa09fa740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794250 0x55b128794630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.221+0000 7fdaa09fa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b128794b70 0x55b12879d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.221+0000 7fdaa09fa740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b12879d750 0x55b12879fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.221+0000 7fdaa09fa740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b12874f960 con 0x55b12879d750 2024-03-12T06:01:36.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.221+0000 7fdaa09fa740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b128703310 con 0x55b128794250 2024-03-12T06:01:36.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.221+0000 7fdaa09fa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b128700840 con 0x55b128794b70 2024-03-12T06:01:36.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda94ff9640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fda800a1c20 0x7fda800a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda96ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b12879d750 0x55b12879fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda96ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b12879d750 0x55b12879fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38942/0 (socket says 172.21.15.76:38942) 2024-03-12T06:01:36.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda96ffd640 1 -- 172.21.15.76:0/3038882832 learned_addr learned my addr 172.21.15.76:0/3038882832 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda97fff640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fda800a1c20 0x7fda800a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda97fff640 1 --2- 172.21.15.76:0/3038882832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794250 0x55b128794630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda977fe640 1 --2- 172.21.15.76:0/3038882832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b128794b70 0x55b12879d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda94ff9640 1 -- 172.21.15.76:0/3177891129 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fda880cd1e0 con 0x7fda980ff650 2024-03-12T06:01:36.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda96ffd640 1 -- 172.21.15.76:0/3038882832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b128794b70 msgr2=0x55b12879d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda96ffd640 1 --2- 172.21.15.76:0/3038882832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b128794b70 0x55b12879d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda96ffd640 1 -- 172.21.15.76:0/3038882832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794250 msgr2=0x55b128794630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda96ffd640 1 --2- 172.21.15.76:0/3038882832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794250 0x55b128794630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda96ffd640 1 -- 172.21.15.76:0/3038882832 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b1286be060 con 0x55b12879d750 2024-03-12T06:01:36.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda977fe640 1 --2- 172.21.15.76:0/3038882832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b128794b70 0x55b12879d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:36.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda97fff640 1 --2- 172.21.15.76:0/3038882832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794250 0x55b128794630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:36.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda96ffd640 1 --2- 172.21.15.76:0/3038882832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b12879d750 0x55b12879fb40 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00bc70 tx=0x7fda8c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb85eb885c52020 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda97fff640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fda800a1c20 0x7fda800a40e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fda90032640 tx=0x7fda90004a40 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda7dffb640 1 -- 172.21.15.76:0/3038882832 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c005220 con 0x55b12879d750 2024-03-12T06:01:36.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda7dffb640 1 -- 172.21.15.76:0/3038882832 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda8c015070 con 0x55b12879d750 2024-03-12T06:01:36.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.222+0000 7fda7dffb640 1 -- 172.21.15.76:0/3038882832 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c0116a0 con 0x55b12879d750 2024-03-12T06:01:36.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.223+0000 7fdaa09fa740 1 -- 172.21.15.76:0/3038882832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b12879d750 msgr2=0x55b12879fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.223+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/3038882832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b12879d750 0x55b12879fb40 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00bc70 tx=0x7fda8c0107a0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.223+0000 7fdaa09fa740 1 -- 172.21.15.76:0/3038882832 shutdown_connections 2024-03-12T06:01:36.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.223+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/3038882832 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b12879d750 0x55b12879fb40 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.223+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/3038882832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b128794b70 0x55b12879d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.223+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/3038882832 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794250 0x55b128794630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.223+0000 7fdaa09fa740 1 -- 172.21.15.76:0/3038882832 >> 172.21.15.76:0/3038882832 conn(0x55b12878e980 msgr2=0x55b12878ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.223+0000 7fdaa09fa740 1 -- 172.21.15.76:0/3038882832 shutdown_connections 2024-03-12T06:01:36.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.223+0000 7fdaa09fa740 1 -- 172.21.15.76:0/3038882832 wait complete. 2024-03-12T06:01:36.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.223+0000 7fdaa09fa740 1 -- start start 2024-03-12T06:01:36.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fdaa09fa740 1 Processor -- start 2024-03-12T06:01:36.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fdaa09fa740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b128794250 0x55b12879a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fdaa09fa740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794b70 0x55b12879ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fdaa09fa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b12879d750 0x55b12879b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fdaa09fa740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1286fc020 con 0x55b128794250 2024-03-12T06:01:36.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fdaa09fa740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1283cf430 con 0x55b128794b70 2024-03-12T06:01:36.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fdaa09fa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b12838be00 con 0x55b12879d750 2024-03-12T06:01:36.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fdaa09fa740 10 client.0 _ll_register_callbacks cb 0x55b1286a3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:36.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda977fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b128794250 0x55b12879a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda977fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b128794250 0x55b12879a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38944/0 (socket says 172.21.15.76:38944) 2024-03-12T06:01:36.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda977fe640 1 -- 172.21.15.76:0/733306521 learned_addr learned my addr 172.21.15.76:0/733306521 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda97fff640 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b12879d750 0x55b12879b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda96ffd640 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794b70 0x55b12879ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda977fe640 1 -- 172.21.15.76:0/733306521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b12879d750 msgr2=0x55b12879b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda977fe640 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b12879d750 0x55b12879b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda977fe640 1 -- 172.21.15.76:0/733306521 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794b70 msgr2=0x55b12879ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda977fe640 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794b70 0x55b12879ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda977fe640 1 -- 172.21.15.76:0/733306521 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c00b8a0 con 0x55b128794250 2024-03-12T06:01:36.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda97fff640 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b12879d750 0x55b12879b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:36.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda96ffd640 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794b70 0x55b12879ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:36.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda977fe640 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b128794250 0x55b12879a6f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fda880caa10 tx=0x7fda88010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd0afb3d43c0edf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:36.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda76ffd640 1 -- 172.21.15.76:0/733306521 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda88099e40 con 0x55b128794250 2024-03-12T06:01:36.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fda76ffd640 1 -- 172.21.15.76:0/733306521 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fda8809df90 con 0x55b128794250 2024-03-12T06:01:36.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.224+0000 7fdaa09fa740 1 -- 172.21.15.76:0/733306521 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b1283fb7e0 con 0x55b128794250 2024-03-12T06:01:36.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.225+0000 7fda76ffd640 1 -- 172.21.15.76:0/733306521 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda880a25e0 con 0x55b128794250 2024-03-12T06:01:36.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.225+0000 7fdaa09fa740 10 client.16815 fetch_fsmap learned FSMap version 28 2024-03-12T06:01:36.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.225+0000 7fdaa09fa740 1 -- 172.21.15.76:0/733306521 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b1286fa6a0 con 0x55b128794250 2024-03-12T06:01:36.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.225+0000 7fda76ffd640 1 -- 172.21.15.76:0/733306521 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b1283fb7e0 con 0x55b128794250 2024-03-12T06:01:36.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.225+0000 7fda76ffd640 1 -- 172.21.15.76:0/733306521 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fda88099290 con 0x55b128794250 2024-03-12T06:01:36.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.225+0000 7fdaa09fa740 10 client.16815 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:01:36.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.225+0000 7fdaa09fa740 10 client.16815 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:01:36.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.225+0000 7fdaa09fa740 20 client.16815 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:36.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.225+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b1287adad0 0x55b1287aff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.225+0000 7fdaa09fa740 4 client.16815 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:01:36.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.225+0000 7fdaa09fa740 1 -- 172.21.15.76:0/733306521 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b128438720 con 0x55b1287adad0 2024-03-12T06:01:36.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.225+0000 7fda96ffd640 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b1287adad0 0x55b1287aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.226+0000 7fda76ffd640 10 client.16815 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:01:36.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.226+0000 7fda96ffd640 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b1287adad0 0x55b1287aff00 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8937784f8c9bcdc6 server_cookie=965de2b20227033c in_seq=0 out_seq=0 2024-03-12T06:01:36.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.226+0000 7fda76ffd640 1 -- 172.21.15.76:0/733306521 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b128438720 con 0x55b1287adad0 2024-03-12T06:01:36.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.226+0000 7fda76ffd640 10 client.16815 handle_command_reply: tid=0 2024-03-12T06:01:36.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.227+0000 7fdaa09fa740 1 client.16815 shutdown 2024-03-12T06:01:36.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.227+0000 7fdaa09fa740 1 -- 172.21.15.76:0/733306521 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b128794250 msgr2=0x55b12879a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.227+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b128794250 0x55b12879a6f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fda880caa10 tx=0x7fda88010000 comp rx=0 tx=0).stop 2024-03-12T06:01:36.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.227+0000 7fdaa09fa740 1 -- 172.21.15.76:0/733306521 shutdown_connections 2024-03-12T06:01:36.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.227+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b1287adad0 0x55b1287aff00 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.227+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b12879d750 0x55b12879b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.227+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794b70 0x55b12879ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.227+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/733306521 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b128794250 0x55b12879a6f0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.227+0000 7fdaa09fa740 1 -- 172.21.15.76:0/733306521 >> 172.21.15.76:0/733306521 conn(0x55b12878e980 msgr2=0x55b1287906d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.227+0000 7fda76ffd640 0 client.16815 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:01:36.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.227+0000 7fdaa09fa740 1 -- 172.21.15.76:0/733306521 shutdown_connections 2024-03-12T06:01:36.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.227+0000 7fdaa09fa740 1 -- 172.21.15.76:0/733306521 wait complete. 2024-03-12T06:01:36.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.227+0000 7fdaa09fa740 20 client.16815 trim_cache size 0 max 16384 2024-03-12T06:01:36.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fdaa09fa740 1 Processor -- start 2024-03-12T06:01:36.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fdaa09fa740 1 -- start start 2024-03-12T06:01:36.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fdaa09fa740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794250 0x55b12879ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fdaa09fa740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b128794b70 0x55b12879af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fdaa09fa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b12879d750 0x55b128798fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fdaa09fa740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b128681130 con 0x55b128794b70 2024-03-12T06:01:36.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fdaa09fa740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b12878a5c0 con 0x55b128794250 2024-03-12T06:01:36.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fdaa09fa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1287aa5c0 con 0x55b12879d750 2024-03-12T06:01:36.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fda96ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b128794b70 0x55b12879af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fda96ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b128794b70 0x55b12879af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38946/0 (socket says 172.21.15.76:38946) 2024-03-12T06:01:36.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fda96ffd640 1 -- 172.21.15.76:0/406734229 learned_addr learned my addr 172.21.15.76:0/406734229 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fda977fe640 1 --2- 172.21.15.76:0/406734229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794250 0x55b12879ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fda96ffd640 1 -- 172.21.15.76:0/406734229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b12879d750 msgr2=0x55b128798fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fda96ffd640 1 --2- 172.21.15.76:0/406734229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b12879d750 0x55b128798fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fda96ffd640 1 -- 172.21.15.76:0/406734229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794250 msgr2=0x55b12879ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fda96ffd640 1 --2- 172.21.15.76:0/406734229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794250 0x55b12879ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fda96ffd640 1 -- 172.21.15.76:0/406734229 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b128696c20 con 0x55b128794b70 2024-03-12T06:01:36.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fda96ffd640 1 --2- 172.21.15.76:0/406734229 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b128794b70 0x55b12879af90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00bb90 tx=0x7fda8c013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ca2a456c1594cb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fda7d7fa640 1 -- 172.21.15.76:0/406734229 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c022040 con 0x55b128794b70 2024-03-12T06:01:36.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fda7d7fa640 1 -- 172.21.15.76:0/406734229 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda8c00bea0 con 0x55b128794b70 2024-03-12T06:01:36.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.239+0000 7fda7d7fa640 1 -- 172.21.15.76:0/406734229 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c003850 con 0x55b128794b70 2024-03-12T06:01:36.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.240+0000 7fdaa09fa740 1 -- 172.21.15.76:0/406734229 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b128794b70 msgr2=0x55b12879af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.240+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/406734229 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b128794b70 0x55b12879af90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fda8c00bb90 tx=0x7fda8c013040 comp rx=0 tx=0).stop 2024-03-12T06:01:36.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.240+0000 7fdaa09fa740 1 -- 172.21.15.76:0/406734229 shutdown_connections 2024-03-12T06:01:36.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.240+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/406734229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b12879d750 0x55b128798fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.240+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/406734229 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b128794b70 0x55b12879af90 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.240+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/406734229 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794250 0x55b12879ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.240+0000 7fdaa09fa740 1 -- 172.21.15.76:0/406734229 >> 172.21.15.76:0/406734229 conn(0x55b12878e980 msgr2=0x55b1287959f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.240+0000 7fdaa09fa740 1 -- 172.21.15.76:0/406734229 shutdown_connections 2024-03-12T06:01:36.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.240+0000 7fdaa09fa740 1 -- 172.21.15.76:0/406734229 wait complete. 2024-03-12T06:01:36.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.240+0000 7fdaa09fa740 1 -- start start 2024-03-12T06:01:36.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fdaa09fa740 1 Processor -- start 2024-03-12T06:01:36.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fdaa09fa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b128794250 0x55b1287a27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fdaa09fa740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794b70 0x55b1287a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fdaa09fa740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b12879d750 0x55b128803020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fdaa09fa740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1287a3490 con 0x55b12879d750 2024-03-12T06:01:36.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fdaa09fa740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b128803590 con 0x55b128794b70 2024-03-12T06:01:36.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fdaa09fa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b128803750 con 0x55b128794250 2024-03-12T06:01:36.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fdaa09fa740 10 client.0 _ll_register_callbacks cb 0x7fda8c00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:36.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fda97fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b12879d750 0x55b128803020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fda977fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b128794250 0x55b1287a27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fda97fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b12879d750 0x55b128803020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38952/0 (socket says 172.21.15.76:38952) 2024-03-12T06:01:36.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fda977fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b128794250 0x55b1287a27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.76:60802/0 (socket says 172.21.15.76:60802) 2024-03-12T06:01:36.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fda97fff640 1 -- 172.21.15.76:0/1374567315 learned_addr learned my addr 172.21.15.76:0/1374567315 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fda96ffd640 1 --2- 172.21.15.76:0/1374567315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794b70 0x55b1287a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fda97fff640 1 -- 172.21.15.76:0/1374567315 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b128794250 msgr2=0x55b1287a27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fda97fff640 1 --2- 172.21.15.76:0/1374567315 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b128794250 0x55b1287a27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fda97fff640 1 -- 172.21.15.76:0/1374567315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794b70 msgr2=0x55b1287a2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fda97fff640 1 --2- 172.21.15.76:0/1374567315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794b70 0x55b1287a2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fda97fff640 1 -- 172.21.15.76:0/1374567315 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b1286be060 con 0x55b12879d750 2024-03-12T06:01:36.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fda97fff640 1 --2- 172.21.15.76:0/1374567315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b12879d750 0x55b128803020 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fda90009cd0 tx=0x7fda90032ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6524a3e2fde6ba server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.241+0000 7fda76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:36.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fda76ffd640 1 -- 172.21.15.76:0/1374567315 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90003d80 con 0x55b12879d750 2024-03-12T06:01:36.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fda76ffd640 1 -- 172.21.15.76:0/1374567315 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fda90047040 con 0x55b12879d750 2024-03-12T06:01:36.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fdaa09fa740 1 -- 172.21.15.76:0/1374567315 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda8c015070 con 0x55b12879d750 2024-03-12T06:01:36.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fda76ffd640 1 -- 172.21.15.76:0/1374567315 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda90032460 con 0x55b12879d750 2024-03-12T06:01:36.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fdaa09fa740 10 client.16821 fetch_fsmap learned FSMap version 28 2024-03-12T06:01:36.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fdaa09fa740 1 -- 172.21.15.76:0/1374567315 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b1286bc8c0 con 0x55b12879d750 2024-03-12T06:01:36.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fda76ffd640 1 -- 172.21.15.76:0/1374567315 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda8c015070 con 0x55b12879d750 2024-03-12T06:01:36.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fda76ffd640 1 -- 172.21.15.76:0/1374567315 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fda90042650 con 0x55b12879d750 2024-03-12T06:01:36.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fdaa09fa740 10 client.16821 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:01:36.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fdaa09fa740 10 client.16821 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:01:36.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fdaa09fa740 20 client.16821 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:36.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/1374567315 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b128804e60 0x55b128807280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fdaa09fa740 4 client.16821 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:01:36.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fdaa09fa740 1 -- 172.21.15.76:0/1374567315 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b128487890 con 0x55b128804e60 2024-03-12T06:01:36.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.242+0000 7fda977fe640 1 --2- 172.21.15.76:0/1374567315 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b128804e60 0x55b128807280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fda977fe640 1 --2- 172.21.15.76:0/1374567315 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b128804e60 0x55b128807280 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=39854e53579e1cf0 server_cookie=1b43cd289395a679 in_seq=0 out_seq=0 2024-03-12T06:01:36.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fda76ffd640 10 client.16821 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:01:36.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fda76ffd640 1 -- 172.21.15.76:0/1374567315 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b128487890 con 0x55b128804e60 2024-03-12T06:01:36.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fda76ffd640 10 client.16821 handle_command_reply: tid=0 2024-03-12T06:01:36.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fdaa09fa740 1 client.16821 shutdown 2024-03-12T06:01:36.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fdaa09fa740 1 -- 172.21.15.76:0/1374567315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b12879d750 msgr2=0x55b128803020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/1374567315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b12879d750 0x55b128803020 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fda90009cd0 tx=0x7fda90032ed0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fdaa09fa740 1 -- 172.21.15.76:0/1374567315 shutdown_connections 2024-03-12T06:01:36.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/1374567315 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b128804e60 0x55b128807280 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/1374567315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b12879d750 0x55b128803020 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/1374567315 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b128794b70 0x55b1287a2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fda76ffd640 0 client.16821 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:01:36.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fdaa09fa740 1 --2- 172.21.15.76:0/1374567315 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b128794250 0x55b1287a27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fdaa09fa740 1 -- 172.21.15.76:0/1374567315 >> 172.21.15.76:0/1374567315 conn(0x55b12878e980 msgr2=0x55b128798f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fdaa09fa740 1 -- 172.21.15.76:0/1374567315 shutdown_connections 2024-03-12T06:01:36.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fdaa09fa740 1 -- 172.21.15.76:0/1374567315 wait complete. 2024-03-12T06:01:36.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.243+0000 7fdaa09fa740 20 client.16821 trim_cache size 0 max 16384 2024-03-12T06:01:36.260 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:01:36.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.244+0000 7fda9dd48640 1 -- 172.21.15.76:0/3177891129 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fda800a1c20 msgr2=0x7fda800a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.244+0000 7fda9dd48640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fda800a1c20 0x7fda800a40e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fda90032640 tx=0x7fda90004a40 comp rx=0 tx=0).stop 2024-03-12T06:01:36.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.244+0000 7fda9dd48640 1 -- 172.21.15.76:0/3177891129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda980ff650 msgr2=0x7fda9819d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.244+0000 7fda9dd48640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda980ff650 0x7fda9819d470 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fda8800d770 tx=0x7fda8800dc30 comp rx=0 tx=0).stop 2024-03-12T06:01:36.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.244+0000 7fda9dd48640 1 -- 172.21.15.76:0/3177891129 shutdown_connections 2024-03-12T06:01:36.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.244+0000 7fda9dd48640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fda800a1c20 0x7fda800a40e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.244+0000 7fda9dd48640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fda98106150 0x7fda98197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.244+0000 7fda9dd48640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda98100010 0x7fda9819d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.244+0000 7fda9dd48640 1 --2- 172.21.15.76:0/3177891129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda980ff650 0x7fda9819d470 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.244+0000 7fda9dd48640 1 -- 172.21.15.76:0/3177891129 >> 172.21.15.76:0/3177891129 conn(0x7fda98076580 msgr2=0x7fda981041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.244+0000 7fda9dd48640 1 -- 172.21.15.76:0/3177891129 shutdown_connections 2024-03-12T06:01:36.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.244+0000 7fda9dd48640 1 -- 172.21.15.76:0/3177891129 wait complete. 2024-03-12T06:01:36.263 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:01:36.263 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T06:01:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.362+0000 7f703ba17640 1 -- 172.21.15.76:0/2391307047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7034100c40 msgr2=0x7f7034104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.362+0000 7f703ba17640 1 --2- 172.21.15.76:0/2391307047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7034100c40 0x7f7034104c80 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7028009cd0 tx=0x7f702802c0f0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.362+0000 7f703ba17640 1 -- 172.21.15.76:0/2391307047 shutdown_connections 2024-03-12T06:01:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.362+0000 7f703ba17640 1 --2- 172.21.15.76:0/2391307047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7034105360 0x7f703410cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.362+0000 7f703ba17640 1 --2- 172.21.15.76:0/2391307047 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7034100c40 0x7f7034104c80 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.362+0000 7f703ba17640 1 --2- 172.21.15.76:0/2391307047 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7034100290 0x7f7034100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.362+0000 7f703ba17640 1 -- 172.21.15.76:0/2391307047 >> 172.21.15.76:0/2391307047 conn(0x7f7034077230 msgr2=0x7f7034077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.363+0000 7f703ba17640 1 -- 172.21.15.76:0/2391307047 shutdown_connections 2024-03-12T06:01:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.363+0000 7f703ba17640 1 -- 172.21.15.76:0/2391307047 wait complete. 2024-03-12T06:01:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.363+0000 7f703ba17640 1 Processor -- start 2024-03-12T06:01:36.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.363+0000 7f703ba17640 1 -- start start 2024-03-12T06:01:36.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.363+0000 7f703ba17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7034100290 0x7f70341985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.363+0000 7f703ba17640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7034105360 0x7f7034198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.363+0000 7f703ba17640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7034199100 0x7f70341a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.363+0000 7f703ba17640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70341a3d50 con 0x7f7034199100 2024-03-12T06:01:36.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.363+0000 7f703ba17640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70341a3ec0 con 0x7f7034105360 2024-03-12T06:01:36.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.363+0000 7f703ba17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70341a4080 con 0x7f7034100290 2024-03-12T06:01:36.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.363+0000 7f7039fb4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7034199100 0x7f70341a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.363+0000 7f7039fb4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7034199100 0x7f70341a3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38968/0 (socket says 172.21.15.76:38968) 2024-03-12T06:01:36.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.363+0000 7f7039fb4640 1 -- 172.21.15.76:0/362726804 learned_addr learned my addr 172.21.15.76:0/362726804 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.364+0000 7f70397b3640 1 --2- 172.21.15.76:0/362726804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7034100290 0x7f70341985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.364+0000 7f7038fb2640 1 --2- 172.21.15.76:0/362726804 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7034105360 0x7f7034198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.364+0000 7f7039fb4640 1 -- 172.21.15.76:0/362726804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7034100290 msgr2=0x7f70341985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.364+0000 7f7039fb4640 1 --2- 172.21.15.76:0/362726804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7034100290 0x7f70341985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.364+0000 7f7039fb4640 1 -- 172.21.15.76:0/362726804 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7034105360 msgr2=0x7f7034198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.364+0000 7f7039fb4640 1 --2- 172.21.15.76:0/362726804 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7034105360 0x7f7034198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.364+0000 7f7039fb4640 1 -- 172.21.15.76:0/362726804 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7028009900 con 0x7f7034199100 2024-03-12T06:01:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.364+0000 7f7039fb4640 1 --2- 172.21.15.76:0/362726804 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7034199100 0x7f70341a3660 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f702400d680 tx=0x7f702400db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.364+0000 7f70327fc640 1 -- 172.21.15.76:0/362726804 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7024005220 con 0x7f7034199100 2024-03-12T06:01:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.364+0000 7f70327fc640 1 -- 172.21.15.76:0/362726804 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7024009d00 con 0x7f7034199100 2024-03-12T06:01:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.364+0000 7f703ba17640 1 -- 172.21.15.76:0/362726804 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70341a4360 con 0x7f7034199100 2024-03-12T06:01:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.364+0000 7f70327fc640 1 -- 172.21.15.76:0/362726804 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7024004dd0 con 0x7f7034199100 2024-03-12T06:01:36.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.364+0000 7f703ba17640 1 -- 172.21.15.76:0/362726804 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70341a4870 con 0x7f7034199100 2024-03-12T06:01:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.366+0000 7f70327fc640 1 -- 172.21.15.76:0/362726804 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f7024009820 con 0x7f7034199100 2024-03-12T06:01:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f703e5df740 1 Processor -- start 2024-03-12T06:01:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f703e5df740 1 -- start start 2024-03-12T06:01:36.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f703e5df740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31250 0x55cc25c31630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f703e5df740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31b70 0x55cc25c3a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f703e5df740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 0x55cc25c3cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f703e5df740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc25bec960 con 0x55cc25c31b70 2024-03-12T06:01:36.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f703e5df740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc25ba0310 con 0x55cc25c3a750 2024-03-12T06:01:36.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f703e5df740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc25b9d840 con 0x55cc25c31250 2024-03-12T06:01:36.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f70327fc640 1 --2- 172.21.15.76:0/362726804 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f701c0a1ba0 0x7f701c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f70397b3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31b70 0x55cc25c3a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f70397b3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31b70 0x55cc25c3a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38974/0 (socket says 172.21.15.76:38974) 2024-03-12T06:01:36.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f70397b3640 1 -- 172.21.15.76:0/3989083652 learned_addr learned my addr 172.21.15.76:0/3989083652 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f7038fb2640 1 --2- 172.21.15.76:0/3989083652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31250 0x55cc25c31630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f7039fb4640 1 --2- 172.21.15.76:0/362726804 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f701c0a1ba0 0x7f701c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f70327fc640 1 -- 172.21.15.76:0/362726804 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f70240cd0d0 con 0x7f7034199100 2024-03-12T06:01:36.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f7038fb2640 1 --2- 172.21.15.76:0/3989083652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 0x55cc25c3cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f70397b3640 1 -- 172.21.15.76:0/3989083652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31250 msgr2=0x55cc25c31630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f70397b3640 1 --2- 172.21.15.76:0/3989083652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31250 0x55cc25c31630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f70397b3640 1 -- 172.21.15.76:0/3989083652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 msgr2=0x55cc25c3cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f70397b3640 1 --2- 172.21.15.76:0/3989083652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 0x55cc25c3cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f70397b3640 1 -- 172.21.15.76:0/3989083652 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc25b5b060 con 0x55cc25c31b70 2024-03-12T06:01:36.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.367+0000 7f7038fb2640 1 --2- 172.21.15.76:0/3989083652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 0x55cc25c3cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:01:36.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.368+0000 7f7039fb4640 1 --2- 172.21.15.76:0/362726804 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f701c0a1ba0 0x7f701c0a4060 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f70240ca900 tx=0x7f702408f400 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.368+0000 7f70397b3640 1 --2- 172.21.15.76:0/3989083652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31b70 0x55cc25c3a210 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f702c00bcd0 tx=0x7f702c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28bc1b4b88583368 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.368+0000 7f701b7fe640 1 -- 172.21.15.76:0/3989083652 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c005220 con 0x55cc25c31b70 2024-03-12T06:01:36.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.368+0000 7f701b7fe640 1 -- 172.21.15.76:0/3989083652 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f702c015070 con 0x55cc25c31b70 2024-03-12T06:01:36.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.368+0000 7f701b7fe640 1 -- 172.21.15.76:0/3989083652 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c011640 con 0x55cc25c31b70 2024-03-12T06:01:36.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.368+0000 7f703e5df740 1 -- 172.21.15.76:0/3989083652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31b70 msgr2=0x55cc25c3a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.368+0000 7f703e5df740 1 --2- 172.21.15.76:0/3989083652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31b70 0x55cc25c3a210 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f702c00bcd0 tx=0x7f702c0107a0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.368+0000 7f703e5df740 1 -- 172.21.15.76:0/3989083652 shutdown_connections 2024-03-12T06:01:36.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.368+0000 7f703e5df740 1 --2- 172.21.15.76:0/3989083652 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 0x55cc25c3cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.368+0000 7f703e5df740 1 --2- 172.21.15.76:0/3989083652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31b70 0x55cc25c3a210 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.368+0000 7f703e5df740 1 --2- 172.21.15.76:0/3989083652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31250 0x55cc25c31630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.368+0000 7f703e5df740 1 -- 172.21.15.76:0/3989083652 >> 172.21.15.76:0/3989083652 conn(0x55cc25c2b980 msgr2=0x55cc25c2bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.369+0000 7f703e5df740 1 -- 172.21.15.76:0/3989083652 shutdown_connections 2024-03-12T06:01:36.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.369+0000 7f703e5df740 1 -- 172.21.15.76:0/3989083652 wait complete. 2024-03-12T06:01:36.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.369+0000 7f703e5df740 1 -- start start 2024-03-12T06:01:36.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.369+0000 7f703e5df740 1 Processor -- start 2024-03-12T06:01:36.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.369+0000 7f703e5df740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c31250 0x55cc25c377e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.369+0000 7f703e5df740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 0x55cc25c37d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.369+0000 7f703e5df740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c3a750 0x55cc25c38260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.369+0000 7f703e5df740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc25b99020 con 0x55cc25c3a750 2024-03-12T06:01:36.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.369+0000 7f703e5df740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc2586c430 con 0x55cc25c31250 2024-03-12T06:01:36.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.369+0000 7f703e5df740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc25828e00 con 0x55cc25c31b70 2024-03-12T06:01:36.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.369+0000 7f703e5df740 10 client.0 _ll_register_callbacks cb 0x55cc25b40cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:36.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.369+0000 7f7038fb2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c3a750 0x55cc25c38260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.369+0000 7f7038fb2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c3a750 0x55cc25c38260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38984/0 (socket says 172.21.15.76:38984) 2024-03-12T06:01:36.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.369+0000 7f7038fb2640 1 -- 172.21.15.76:0/574757402 learned_addr learned my addr 172.21.15.76:0/574757402 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f70397b3640 1 --2- 172.21.15.76:0/574757402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 0x55cc25c37d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f7038fb2640 1 --2- 172.21.15.76:0/574757402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c31250 0x55cc25c377e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f7038fb2640 1 -- 172.21.15.76:0/574757402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 msgr2=0x55cc25c37d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f7038fb2640 1 --2- 172.21.15.76:0/574757402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 0x55cc25c37d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f7038fb2640 1 -- 172.21.15.76:0/574757402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c31250 msgr2=0x55cc25c377e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f7038fb2640 1 --2- 172.21.15.76:0/574757402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c31250 0x55cc25c377e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f7038fb2640 1 -- 172.21.15.76:0/574757402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702c00b900 con 0x55cc25c3a750 2024-03-12T06:01:36.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f7038fb2640 1 --2- 172.21.15.76:0/574757402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c3a750 0x55cc25c38260 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f702802c700 tx=0x7f70280095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6449d1d543b855af server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f7018ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:36.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f7018ff9640 1 -- 172.21.15.76:0/574757402 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7028035e70 con 0x55cc25c3a750 2024-03-12T06:01:36.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f7018ff9640 1 -- 172.21.15.76:0/574757402 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f70280326b0 con 0x55cc25c3a750 2024-03-12T06:01:36.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f703e5df740 1 -- 172.21.15.76:0/574757402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cc258987e0 con 0x55cc25c3a750 2024-03-12T06:01:36.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f7018ff9640 1 -- 172.21.15.76:0/574757402 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7028032870 con 0x55cc25c3a750 2024-03-12T06:01:36.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f703e5df740 10 client.16833 fetch_fsmap learned FSMap version 28 2024-03-12T06:01:36.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f703e5df740 1 -- 172.21.15.76:0/574757402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55cc25b976a0 con 0x55cc25c3a750 2024-03-12T06:01:36.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f7018ff9640 1 -- 172.21.15.76:0/574757402 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55cc258987e0 con 0x55cc25c3a750 2024-03-12T06:01:36.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.370+0000 7f7018ff9640 1 -- 172.21.15.76:0/574757402 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7028004380 con 0x55cc25c3a750 2024-03-12T06:01:36.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.371+0000 7f703e5df740 10 client.16833 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:01:36.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.371+0000 7f703e5df740 10 client.16833 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:01:36.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.371+0000 7f703e5df740 20 client.16833 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:36.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.371+0000 7f703e5df740 1 --2- 172.21.15.76:0/574757402 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cc25c4ab60 0x55cc25c4cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.371+0000 7f703e5df740 4 client.16833 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:01:36.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.371+0000 7f703e5df740 1 -- 172.21.15.76:0/574757402 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc258d5720 con 0x55cc25c4ab60 2024-03-12T06:01:36.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.371+0000 7f70397b3640 1 --2- 172.21.15.76:0/574757402 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cc25c4ab60 0x55cc25c4cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.371+0000 7f7018ff9640 10 client.16833 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:01:36.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.371+0000 7f70397b3640 1 --2- 172.21.15.76:0/574757402 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cc25c4ab60 0x55cc25c4cf80 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ceacd2d62dde68e server_cookie=afe53369a1929a9c in_seq=0 out_seq=0 2024-03-12T06:01:36.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.372+0000 7f7018ff9640 1 -- 172.21.15.76:0/574757402 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cc258d5720 con 0x55cc25c4ab60 2024-03-12T06:01:36.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.372+0000 7f7018ff9640 10 client.16833 handle_command_reply: tid=0 2024-03-12T06:01:36.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.372+0000 7f703e5df740 1 client.16833 shutdown 2024-03-12T06:01:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.372+0000 7f703e5df740 1 -- 172.21.15.76:0/574757402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c3a750 msgr2=0x55cc25c38260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.372+0000 7f703e5df740 1 --2- 172.21.15.76:0/574757402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c3a750 0x55cc25c38260 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f702802c700 tx=0x7f70280095c0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.372+0000 7f703e5df740 1 -- 172.21.15.76:0/574757402 shutdown_connections 2024-03-12T06:01:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.372+0000 7f703e5df740 1 --2- 172.21.15.76:0/574757402 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cc25c4ab60 0x55cc25c4cf80 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.372+0000 7f703e5df740 1 --2- 172.21.15.76:0/574757402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c3a750 0x55cc25c38260 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.372+0000 7f7018ff9640 0 client.16833 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:01:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.372+0000 7f703e5df740 1 --2- 172.21.15.76:0/574757402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 0x55cc25c37d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.372+0000 7f703e5df740 1 --2- 172.21.15.76:0/574757402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c31250 0x55cc25c377e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.372+0000 7f703e5df740 1 -- 172.21.15.76:0/574757402 >> 172.21.15.76:0/574757402 conn(0x55cc25c2b980 msgr2=0x55cc25c2d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.372+0000 7f703e5df740 1 -- 172.21.15.76:0/574757402 shutdown_connections 2024-03-12T06:01:36.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.372+0000 7f703e5df740 1 -- 172.21.15.76:0/574757402 wait complete. 2024-03-12T06:01:36.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.373+0000 7f703e5df740 20 client.16833 trim_cache size 0 max 16384 2024-03-12T06:01:36.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.384+0000 7f703e5df740 1 Processor -- start 2024-03-12T06:01:36.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.384+0000 7f703e5df740 1 -- start start 2024-03-12T06:01:36.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.384+0000 7f703e5df740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31250 0x55cc25c39f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.384+0000 7f703e5df740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 0x55cc25c34a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.384+0000 7f703e5df740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 0x55cc25c34f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.384+0000 7f703e5df740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc25b1e130 con 0x55cc25c31250 2024-03-12T06:01:36.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.384+0000 7f703e5df740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc25c476c0 con 0x55cc25c3a750 2024-03-12T06:01:36.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.384+0000 7f703e5df740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc25c4a960 con 0x55cc25c31b70 2024-03-12T06:01:36.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.384+0000 7f7038fb2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31250 0x55cc25c39f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.384+0000 7f7038fb2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31250 0x55cc25c39f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:38996/0 (socket says 172.21.15.76:38996) 2024-03-12T06:01:36.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.384+0000 7f7038fb2640 1 -- 172.21.15.76:0/686682590 learned_addr learned my addr 172.21.15.76:0/686682590 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.385+0000 7f70397b3640 1 --2- 172.21.15.76:0/686682590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 0x55cc25c34a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.385+0000 7f7038fb2640 1 -- 172.21.15.76:0/686682590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 msgr2=0x55cc25c34a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.385+0000 7f7038fb2640 1 --2- 172.21.15.76:0/686682590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 0x55cc25c34a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.385+0000 7f7038fb2640 1 -- 172.21.15.76:0/686682590 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 msgr2=0x55cc25c34f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.385+0000 7f7038fb2640 1 --2- 172.21.15.76:0/686682590 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 0x55cc25c34f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.385+0000 7f7038fb2640 1 -- 172.21.15.76:0/686682590 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc25b33c20 con 0x55cc25c31250 2024-03-12T06:01:36.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.385+0000 7f70397b3640 1 --2- 172.21.15.76:0/686682590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 0x55cc25c34a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:36.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.385+0000 7f7038fb2640 1 --2- 172.21.15.76:0/686682590 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31250 0x55cc25c39f10 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f7028002230 tx=0x7f7028034820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495ff82151b7ee66 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.385+0000 7f701affd640 1 -- 172.21.15.76:0/686682590 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702805b070 con 0x55cc25c31250 2024-03-12T06:01:36.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.385+0000 7f701affd640 1 -- 172.21.15.76:0/686682590 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f702802c770 con 0x55cc25c31250 2024-03-12T06:01:36.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.385+0000 7f701affd640 1 -- 172.21.15.76:0/686682590 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7028032950 con 0x55cc25c31250 2024-03-12T06:01:36.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.385+0000 7f703e5df740 1 -- 172.21.15.76:0/686682590 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31250 msgr2=0x55cc25c39f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.385+0000 7f703e5df740 1 --2- 172.21.15.76:0/686682590 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31250 0x55cc25c39f10 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f7028002230 tx=0x7f7028034820 comp rx=0 tx=0).stop 2024-03-12T06:01:36.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 -- 172.21.15.76:0/686682590 shutdown_connections 2024-03-12T06:01:36.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 --2- 172.21.15.76:0/686682590 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 0x55cc25c34f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 --2- 172.21.15.76:0/686682590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 0x55cc25c34a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 --2- 172.21.15.76:0/686682590 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31250 0x55cc25c39f10 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 -- 172.21.15.76:0/686682590 >> 172.21.15.76:0/686682590 conn(0x55cc25c2b980 msgr2=0x55cc25c32ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 -- 172.21.15.76:0/686682590 shutdown_connections 2024-03-12T06:01:36.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 -- 172.21.15.76:0/686682590 wait complete. 2024-03-12T06:01:36.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 -- start start 2024-03-12T06:01:36.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 Processor -- start 2024-03-12T06:01:36.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31250 0x55cc25c40660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 0x55cc25c40ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 0x55cc25ca1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc25c412a0 con 0x55cc25c31250 2024-03-12T06:01:36.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc25ca15b0 con 0x55cc25c3a750 2024-03-12T06:01:36.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc25ca16f0 con 0x55cc25c31b70 2024-03-12T06:01:36.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f703e5df740 10 client.0 _ll_register_callbacks cb 0x7f702c00b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:36.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f7038fb2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31250 0x55cc25c40660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f7038fb2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31250 0x55cc25c40660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39004/0 (socket says 172.21.15.76:39004) 2024-03-12T06:01:36.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.386+0000 7f7038fb2640 1 -- 172.21.15.76:0/2472739939 learned_addr learned my addr 172.21.15.76:0/2472739939 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f7038fb2640 1 --2- 172.21.15.76:0/2472739939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 0x55cc25ca1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f70397b3640 1 --2- 172.21.15.76:0/2472739939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 0x55cc25c40ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f7038fb2640 1 -- 172.21.15.76:0/2472739939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 msgr2=0x55cc25c40ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f7038fb2640 1 --2- 172.21.15.76:0/2472739939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 0x55cc25c40ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f7038fb2640 1 -- 172.21.15.76:0/2472739939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 msgr2=0x55cc25ca1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f7038fb2640 1 --2- 172.21.15.76:0/2472739939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 0x55cc25ca1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f7038fb2640 1 -- 172.21.15.76:0/2472739939 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc25b976a0 con 0x55cc25c31250 2024-03-12T06:01:36.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f70397b3640 1 --2- 172.21.15.76:0/2472739939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 0x55cc25c40ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:36.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f7038fb2640 1 --2- 172.21.15.76:0/2472739939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31250 0x55cc25c40660 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f7028009430 tx=0x7f7028032750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581033db3f39b0e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f7018ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:36.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f7018ff9640 1 -- 172.21.15.76:0/2472739939 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702805b070 con 0x55cc25c31250 2024-03-12T06:01:36.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f7018ff9640 1 -- 172.21.15.76:0/2472739939 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7028051020 con 0x55cc25c31250 2024-03-12T06:01:36.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f703e5df740 1 -- 172.21.15.76:0/2472739939 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f702c015070 con 0x55cc25c31250 2024-03-12T06:01:36.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f7018ff9640 1 -- 172.21.15.76:0/2472739939 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70280327f0 con 0x55cc25c31250 2024-03-12T06:01:36.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f703e5df740 10 client.16839 fetch_fsmap learned FSMap version 28 2024-03-12T06:01:36.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f703e5df740 1 -- 172.21.15.76:0/2472739939 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55cc25b598c0 con 0x55cc25c31250 2024-03-12T06:01:36.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f7018ff9640 1 -- 172.21.15.76:0/2472739939 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f702806e020 con 0x55cc25c31250 2024-03-12T06:01:36.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f7018ff9640 1 -- 172.21.15.76:0/2472739939 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f702806a430 con 0x55cc25c31250 2024-03-12T06:01:36.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f703e5df740 10 client.16839 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:01:36.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.387+0000 7f703e5df740 10 client.16839 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:01:36.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f703e5df740 20 client.16839 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:36.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f703e5df740 1 --2- 172.21.15.76:0/2472739939 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cc25c4ab60 0x55cc25ca4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f703e5df740 4 client.16839 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:01:36.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f703e5df740 1 -- 172.21.15.76:0/2472739939 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cc25924890 con 0x55cc25c4ab60 2024-03-12T06:01:36.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f70397b3640 1 --2- 172.21.15.76:0/2472739939 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cc25c4ab60 0x55cc25ca4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f7018ff9640 10 client.16839 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:01:36.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f70397b3640 1 --2- 172.21.15.76:0/2472739939 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cc25c4ab60 0x55cc25ca4940 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2a8777021d0f1b80 server_cookie=f004f792ec2391d4 in_seq=0 out_seq=0 2024-03-12T06:01:36.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f7018ff9640 1 -- 172.21.15.76:0/2472739939 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cc25924890 con 0x55cc25c4ab60 2024-03-12T06:01:36.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f7018ff9640 10 client.16839 handle_command_reply: tid=0 2024-03-12T06:01:36.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f703e5df740 1 client.16839 shutdown 2024-03-12T06:01:36.405 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f703e5df740 1 -- 172.21.15.76:0/2472739939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31250 msgr2=0x55cc25c40660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f703e5df740 1 --2- 172.21.15.76:0/2472739939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31250 0x55cc25c40660 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f7028009430 tx=0x7f7028032750 comp rx=0 tx=0).stop 2024-03-12T06:01:36.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f703e5df740 1 -- 172.21.15.76:0/2472739939 shutdown_connections 2024-03-12T06:01:36.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f703e5df740 1 --2- 172.21.15.76:0/2472739939 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55cc25c4ab60 0x55cc25ca4940 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f703e5df740 1 --2- 172.21.15.76:0/2472739939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55cc25c3a750 0x55cc25ca1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f7018ff9640 0 client.16839 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:01:36.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f703e5df740 1 --2- 172.21.15.76:0/2472739939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cc25c31b70 0x55cc25c40ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f703e5df740 1 --2- 172.21.15.76:0/2472739939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55cc25c31250 0x55cc25c40660 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.388+0000 7f703e5df740 1 -- 172.21.15.76:0/2472739939 >> 172.21.15.76:0/2472739939 conn(0x55cc25c2b980 msgr2=0x55cc25c2a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703e5df740 1 -- 172.21.15.76:0/2472739939 shutdown_connections 2024-03-12T06:01:36.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703e5df740 1 -- 172.21.15.76:0/2472739939 wait complete. 2024-03-12T06:01:36.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703e5df740 20 client.16839 trim_cache size 0 max 16384 2024-03-12T06:01:36.407 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:01:36.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703ba17640 1 -- 172.21.15.76:0/362726804 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f701c0a1ba0 msgr2=0x7f701c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703ba17640 1 --2- 172.21.15.76:0/362726804 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f701c0a1ba0 0x7f701c0a4060 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f70240ca900 tx=0x7f702408f400 comp rx=0 tx=0).stop 2024-03-12T06:01:36.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703ba17640 1 -- 172.21.15.76:0/362726804 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7034199100 msgr2=0x7f70341a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703ba17640 1 --2- 172.21.15.76:0/362726804 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7034199100 0x7f70341a3660 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f702400d680 tx=0x7f702400db40 comp rx=0 tx=0).stop 2024-03-12T06:01:36.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703ba17640 1 -- 172.21.15.76:0/362726804 shutdown_connections 2024-03-12T06:01:36.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703ba17640 1 --2- 172.21.15.76:0/362726804 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f701c0a1ba0 0x7f701c0a4060 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703ba17640 1 --2- 172.21.15.76:0/362726804 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7034199100 0x7f70341a3660 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703ba17640 1 --2- 172.21.15.76:0/362726804 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7034105360 0x7f7034198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703ba17640 1 --2- 172.21.15.76:0/362726804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7034100290 0x7f70341985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703ba17640 1 -- 172.21.15.76:0/362726804 >> 172.21.15.76:0/362726804 conn(0x7f7034077230 msgr2=0x7f70340fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703ba17640 1 -- 172.21.15.76:0/362726804 shutdown_connections 2024-03-12T06:01:36.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.389+0000 7f703ba17640 1 -- 172.21.15.76:0/362726804 wait complete. 2024-03-12T06:01:36.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:36 smithi076 ceph-mon[32095]: pgmap v177: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:36.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:36 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4069607309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:01:36.411 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:01:36.411 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T06:01:36.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:36 smithi082 ceph-mon[36382]: pgmap v177: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:36.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:36 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4069607309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:01:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:36 smithi088 ceph-mon[36585]: pgmap v177: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:36 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4069607309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:01:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.511+0000 7f4bb4e4e640 1 -- 172.21.15.76:0/1474461786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bb01007c0 msgr2=0x7f4bb0100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.511+0000 7f4bb4e4e640 1 --2- 172.21.15.76:0/1474461786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bb01007c0 0x7f4bb0100c20 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009900 tx=0x7f4ba402bc30 comp rx=0 tx=0).stop 2024-03-12T06:01:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.511+0000 7f4bb4e4e640 1 -- 172.21.15.76:0/1474461786 shutdown_connections 2024-03-12T06:01:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.511+0000 7f4bb4e4e640 1 --2- 172.21.15.76:0/1474461786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb0101160 0x7f4bb010cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.511+0000 7f4bb4e4e640 1 --2- 172.21.15.76:0/1474461786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bb01007c0 0x7f4bb0100c20 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.511+0000 7f4bb4e4e640 1 --2- 172.21.15.76:0/1474461786 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4bb0101ff0 0x7f4bb01023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.511+0000 7f4bb4e4e640 1 -- 172.21.15.76:0/1474461786 >> 172.21.15.76:0/1474461786 conn(0x7f4bb0076570 msgr2=0x7f4bb00fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.511+0000 7f4bb4e4e640 1 -- 172.21.15.76:0/1474461786 shutdown_connections 2024-03-12T06:01:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.511+0000 7f4bb4e4e640 1 -- 172.21.15.76:0/1474461786 wait complete. 2024-03-12T06:01:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bb4e4e640 1 Processor -- start 2024-03-12T06:01:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bb4e4e640 1 -- start start 2024-03-12T06:01:36.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bb4e4e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bb01007c0 0x7f4bb0198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bb4e4e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4bb0101160 0x7f4bb0198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bb4e4e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb0101ff0 0x7f4bb01a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bb4e4e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb01994d0 con 0x7f4bb01007c0 2024-03-12T06:01:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bb4e4e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb0199640 con 0x7f4bb0101160 2024-03-12T06:01:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bb4e4e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb01a3dc0 con 0x7f4bb0101ff0 2024-03-12T06:01:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bae59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bb01007c0 0x7f4bb0198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bae59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bb01007c0 0x7f4bb0198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39026/0 (socket says 172.21.15.76:39026) 2024-03-12T06:01:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bae59c640 1 -- 172.21.15.76:0/430419343 learned_addr learned my addr 172.21.15.76:0/430419343 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4baed9d640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb0101ff0 0x7f4bb01a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4badd9b640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4bb0101160 0x7f4bb0198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bae59c640 1 -- 172.21.15.76:0/430419343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb0101ff0 msgr2=0x7f4bb01a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bae59c640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb0101ff0 0x7f4bb01a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bae59c640 1 -- 172.21.15.76:0/430419343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4bb0101160 msgr2=0x7f4bb0198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bae59c640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4bb0101160 0x7f4bb0198cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4bae59c640 1 -- 172.21.15.76:0/430419343 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba4009560 con 0x7f4bb01007c0 2024-03-12T06:01:36.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.512+0000 7f4badd9b640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4bb0101160 0x7f4bb0198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:36.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.513+0000 7f4bae59c640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bb01007c0 0x7f4bb0198780 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000e9f0 tx=0x7f4ba000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.513+0000 7f4b9f7fe640 1 -- 172.21.15.76:0/430419343 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0009700 con 0x7f4bb01007c0 2024-03-12T06:01:36.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.513+0000 7f4bb4e4e640 1 -- 172.21.15.76:0/430419343 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb01a40a0 con 0x7f4bb01007c0 2024-03-12T06:01:36.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.513+0000 7f4b9f7fe640 1 -- 172.21.15.76:0/430419343 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ba000cd80 con 0x7f4bb01007c0 2024-03-12T06:01:36.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.513+0000 7f4b9f7fe640 1 -- 172.21.15.76:0/430419343 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0018920 con 0x7f4bb01007c0 2024-03-12T06:01:36.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.513+0000 7f4bb4e4e640 1 -- 172.21.15.76:0/430419343 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb01a45b0 con 0x7f4bb01007c0 2024-03-12T06:01:36.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.515+0000 7f4b9f7fe640 1 -- 172.21.15.76:0/430419343 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4ba0018a80 con 0x7f4bb01007c0 2024-03-12T06:01:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.515+0000 7f4bb7a27740 1 Processor -- start 2024-03-12T06:01:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.515+0000 7f4bb7a27740 1 -- start start 2024-03-12T06:01:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4bb7a27740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4d250 0x558055b4d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4bb7a27740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4db70 0x558055b56210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4bb7a27740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b58b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4bb7a27740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558055b08960 con 0x558055b56750 2024-03-12T06:01:36.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4bb7a27740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558055abc310 con 0x558055b4d250 2024-03-12T06:01:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4bb7a27740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558055ab9840 con 0x558055b4db70 2024-03-12T06:01:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4b9f7fe640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4b980a1ba0 0x7f4b980a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4badd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b58b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4bae59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4db70 0x558055b56210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4baed9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4d250 0x558055b4d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4badd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b58b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39042/0 (socket says 172.21.15.76:39042) 2024-03-12T06:01:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4baed9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4d250 0x558055b4d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:55430/0 (socket says 172.21.15.76:55430) 2024-03-12T06:01:36.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4badd9b640 1 -- 172.21.15.76:0/1231281585 learned_addr learned my addr 172.21.15.76:0/1231281585 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4baed9d640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4b980a1ba0 0x7f4b980a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4b9f7fe640 1 -- 172.21.15.76:0/430419343 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4ba0014070 con 0x7f4bb01007c0 2024-03-12T06:01:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4badd9b640 1 -- 172.21.15.76:0/1231281585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4db70 msgr2=0x558055b56210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4badd9b640 1 --2- 172.21.15.76:0/1231281585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4db70 0x558055b56210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4badd9b640 1 -- 172.21.15.76:0/1231281585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4d250 msgr2=0x558055b4d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4badd9b640 1 --2- 172.21.15.76:0/1231281585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4d250 0x558055b4d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4badd9b640 1 -- 172.21.15.76:0/1231281585 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558055a77060 con 0x558055b56750 2024-03-12T06:01:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4baed9d640 1 --2- 172.21.15.76:0/1231281585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4d250 0x558055b4d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:01:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4baed9d640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4b980a1ba0 0x7f4b980a4060 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8009fd0 tx=0x7f4ba8009600 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.516+0000 7f4badd9b640 1 --2- 172.21.15.76:0/1231281585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b58b40 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4006010 tx=0x7f4ba402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a0449efb748a10 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.517+0000 7f4b9cff9640 1 -- 172.21.15.76:0/1231281585 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba402cb50 con 0x558055b56750 2024-03-12T06:01:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.517+0000 7f4b9cff9640 1 -- 172.21.15.76:0/1231281585 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ba4004030 con 0x558055b56750 2024-03-12T06:01:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.517+0000 7f4b9cff9640 1 -- 172.21.15.76:0/1231281585 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba40051d0 con 0x558055b56750 2024-03-12T06:01:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.517+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1231281585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 msgr2=0x558055b58b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.517+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/1231281585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b58b40 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4006010 tx=0x7f4ba402c650 comp rx=0 tx=0).stop 2024-03-12T06:01:36.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.517+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1231281585 shutdown_connections 2024-03-12T06:01:36.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.517+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/1231281585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b58b40 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.517+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/1231281585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4db70 0x558055b56210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.517+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/1231281585 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4d250 0x558055b4d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.517+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1231281585 >> 172.21.15.76:0/1231281585 conn(0x558055b47980 msgr2=0x558055b47d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.517+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1231281585 shutdown_connections 2024-03-12T06:01:36.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1231281585 wait complete. 2024-03-12T06:01:36.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bb7a27740 1 -- start start 2024-03-12T06:01:36.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bb7a27740 1 Processor -- start 2024-03-12T06:01:36.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bb7a27740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b4d250 0x558055b536a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bb7a27740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4db70 0x558055b53be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bb7a27740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b56750 0x558055b54120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bb7a27740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558055ab5020 con 0x558055b4d250 2024-03-12T06:01:36.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bb7a27740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558055788430 con 0x558055b4db70 2024-03-12T06:01:36.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bb7a27740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558055744e00 con 0x558055b56750 2024-03-12T06:01:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bb7a27740 10 client.0 _ll_register_callbacks cb 0x558055a5ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bae59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b4d250 0x558055b536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bae59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b4d250 0x558055b536a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39044/0 (socket says 172.21.15.76:39044) 2024-03-12T06:01:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bae59c640 1 -- 172.21.15.76:0/2605121567 learned_addr learned my addr 172.21.15.76:0/2605121567 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4badd9b640 1 --2- 172.21.15.76:0/2605121567 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4db70 0x558055b53be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bae59c640 1 -- 172.21.15.76:0/2605121567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b56750 msgr2=0x558055b54120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bae59c640 1 --2- 172.21.15.76:0/2605121567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b56750 0x558055b54120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bae59c640 1 -- 172.21.15.76:0/2605121567 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4db70 msgr2=0x558055b53be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bae59c640 1 --2- 172.21.15.76:0/2605121567 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4db70 0x558055b53be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4bae59c640 1 -- 172.21.15.76:0/2605121567 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb010d8e0 con 0x558055b4d250 2024-03-12T06:01:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.518+0000 7f4badd9b640 1 --2- 172.21.15.76:0/2605121567 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4db70 0x558055b53be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4bae59c640 1 --2- 172.21.15.76:0/2605121567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b4d250 0x558055b536a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4ba009a740 tx=0x7f4ba006a910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbac4b040a341c90 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4b95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4b95ffb640 1 -- 172.21.15.76:0/2605121567 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba009e900 con 0x558055b4d250 2024-03-12T06:01:36.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4b95ffb640 1 -- 172.21.15.76:0/2605121567 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4ba00a2f50 con 0x558055b4d250 2024-03-12T06:01:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4bb7a27740 1 -- 172.21.15.76:0/2605121567 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5580557b47e0 con 0x558055b4d250 2024-03-12T06:01:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4b95ffb640 1 -- 172.21.15.76:0/2605121567 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ba0096e20 con 0x558055b4d250 2024-03-12T06:01:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4bb7a27740 10 client.16851 fetch_fsmap learned FSMap version 28 2024-03-12T06:01:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4bb7a27740 1 -- 172.21.15.76:0/2605121567 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558055ab36a0 con 0x558055b4d250 2024-03-12T06:01:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4b95ffb640 1 -- 172.21.15.76:0/2605121567 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5580557b47e0 con 0x558055b4d250 2024-03-12T06:01:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4b95ffb640 1 -- 172.21.15.76:0/2605121567 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4ba00ba070 con 0x558055b4d250 2024-03-12T06:01:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4bb7a27740 10 client.16851 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:01:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4bb7a27740 10 client.16851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:36.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4bb7a27740 20 client.16851 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:36.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/2605121567 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558055b66ac0 0x558055b68ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4bb7a27740 4 client.16851 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:01:36.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4bb7a27740 1 -- 172.21.15.76:0/2605121567 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5580557f1720 con 0x558055b66ac0 2024-03-12T06:01:36.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.519+0000 7f4badd9b640 1 --2- 172.21.15.76:0/2605121567 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558055b66ac0 0x558055b68ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.520+0000 7f4b95ffb640 10 client.16851 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:36.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.520+0000 7f4badd9b640 1 --2- 172.21.15.76:0/2605121567 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558055b66ac0 0x558055b68ef0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72826b16568135b5 server_cookie=ae701802c8b339fd in_seq=0 out_seq=0 2024-03-12T06:01:36.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.520+0000 7f4b95ffb640 1 -- 172.21.15.76:0/2605121567 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5580557f1720 con 0x558055b66ac0 2024-03-12T06:01:36.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.521+0000 7f4b95ffb640 10 client.16851 handle_command_reply: tid=0 2024-03-12T06:01:36.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.521+0000 7f4bb7a27740 1 client.16851 shutdown 2024-03-12T06:01:36.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.521+0000 7f4bb7a27740 1 -- 172.21.15.76:0/2605121567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b4d250 msgr2=0x558055b536a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.521+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/2605121567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b4d250 0x558055b536a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4ba009a740 tx=0x7f4ba006a910 comp rx=0 tx=0).stop 2024-03-12T06:01:36.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.521+0000 7f4bb7a27740 1 -- 172.21.15.76:0/2605121567 shutdown_connections 2024-03-12T06:01:36.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.521+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/2605121567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b56750 0x558055b54120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.521+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/2605121567 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4db70 0x558055b53be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.521+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/2605121567 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558055b66ac0 0x558055b68ef0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.521+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/2605121567 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b4d250 0x558055b536a0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.521+0000 7f4b95ffb640 0 client.16851 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:36.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.521+0000 7f4bb7a27740 1 -- 172.21.15.76:0/2605121567 >> 172.21.15.76:0/2605121567 conn(0x558055b47980 msgr2=0x558055b496d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.521+0000 7f4bb7a27740 1 -- 172.21.15.76:0/2605121567 shutdown_connections 2024-03-12T06:01:36.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.521+0000 7f4bb7a27740 1 -- 172.21.15.76:0/2605121567 wait complete. 2024-03-12T06:01:36.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.521+0000 7f4bb7a27740 20 client.16851 trim_cache size 0 max 16384 2024-03-12T06:01:36.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.532+0000 7f4bb7a27740 1 Processor -- start 2024-03-12T06:01:36.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.532+0000 7f4bb7a27740 1 -- start start 2024-03-12T06:01:36.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.532+0000 7f4bb7a27740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4d250 0x558055b55dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.532+0000 7f4bb7a27740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4db70 0x558055b53f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.532+0000 7f4bb7a27740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b50940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.532+0000 7f4bb7a27740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558055a3a130 con 0x558055b56750 2024-03-12T06:01:36.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.532+0000 7f4bb7a27740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558055b6e140 con 0x558055b4db70 2024-03-12T06:01:36.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.532+0000 7f4bb7a27740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558055b4c570 con 0x558055b4d250 2024-03-12T06:01:36.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4baed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b50940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4baed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b50940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39048/0 (socket says 172.21.15.76:39048) 2024-03-12T06:01:36.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4baed9d640 1 -- 172.21.15.76:0/1101684236 learned_addr learned my addr 172.21.15.76:0/1101684236 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4bae59c640 1 --2- 172.21.15.76:0/1101684236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4d250 0x558055b55dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4baed9d640 1 -- 172.21.15.76:0/1101684236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4d250 msgr2=0x558055b55dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4baed9d640 1 --2- 172.21.15.76:0/1101684236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4d250 0x558055b55dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4badd9b640 1 --2- 172.21.15.76:0/1101684236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4db70 0x558055b53f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4baed9d640 1 -- 172.21.15.76:0/1101684236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4db70 msgr2=0x558055b53f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4baed9d640 1 --2- 172.21.15.76:0/1101684236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4db70 0x558055b53f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4baed9d640 1 -- 172.21.15.76:0/1101684236 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558055a4fc20 con 0x558055b56750 2024-03-12T06:01:36.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4badd9b640 1 --2- 172.21.15.76:0/1101684236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4db70 0x558055b53f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:36.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4baed9d640 1 --2- 172.21.15.76:0/1101684236 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b50940 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8014bf0 tx=0x7f4ba8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b296e2dc55b72d9e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4b97fff640 1 -- 172.21.15.76:0/1101684236 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba800acd0 con 0x558055b56750 2024-03-12T06:01:36.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4b97fff640 1 -- 172.21.15.76:0/1101684236 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ba8027070 con 0x558055b56750 2024-03-12T06:01:36.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.533+0000 7f4b97fff640 1 -- 172.21.15.76:0/1101684236 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba802c600 con 0x558055b56750 2024-03-12T06:01:36.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1101684236 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 msgr2=0x558055b50940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/1101684236 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b50940 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8014bf0 tx=0x7f4ba8004ab0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1101684236 shutdown_connections 2024-03-12T06:01:36.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/1101684236 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b50940 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/1101684236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4db70 0x558055b53f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/1101684236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4d250 0x558055b55dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1101684236 >> 172.21.15.76:0/1101684236 conn(0x558055b47980 msgr2=0x558055b42a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1101684236 shutdown_connections 2024-03-12T06:01:36.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1101684236 wait complete. 2024-03-12T06:01:36.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 -- start start 2024-03-12T06:01:36.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 Processor -- start 2024-03-12T06:01:36.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4d250 0x558055b5d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4db70 0x558055b5da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b5ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558055b5c180 con 0x558055b56750 2024-03-12T06:01:36.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558055b5c2f0 con 0x558055b4d250 2024-03-12T06:01:36.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558055b5c4b0 con 0x558055b4db70 2024-03-12T06:01:36.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bb7a27740 10 client.0 _ll_register_callbacks cb 0x7f4ba0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:36.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4baed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b5ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4baed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b5ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39064/0 (socket says 172.21.15.76:39064) 2024-03-12T06:01:36.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4baed9d640 1 -- 172.21.15.76:0/1119900952 learned_addr learned my addr 172.21.15.76:0/1119900952 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:36.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.534+0000 7f4bae59c640 1 --2- 172.21.15.76:0/1119900952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4d250 0x558055b5d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4badd9b640 1 --2- 172.21.15.76:0/1119900952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4db70 0x558055b5da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4baed9d640 1 -- 172.21.15.76:0/1119900952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4db70 msgr2=0x558055b5da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4baed9d640 1 --2- 172.21.15.76:0/1119900952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4db70 0x558055b5da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4baed9d640 1 -- 172.21.15.76:0/1119900952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4d250 msgr2=0x558055b5d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:36.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4baed9d640 1 --2- 172.21.15.76:0/1119900952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4d250 0x558055b5d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4baed9d640 1 -- 172.21.15.76:0/1119900952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba801c040 con 0x558055b56750 2024-03-12T06:01:36.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4badd9b640 1 --2- 172.21.15.76:0/1119900952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4db70 0x558055b5da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:01:36.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4baed9d640 1 --2- 172.21.15.76:0/1119900952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b5ba80 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800ae40 tx=0x7f4ba8009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96be6f8d0623cbed server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:36.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4b95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:36.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4b95ffb640 1 -- 172.21.15.76:0/1119900952 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8023650 con 0x558055b56750 2024-03-12T06:01:36.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1119900952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ba4004030 con 0x558055b56750 2024-03-12T06:01:36.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4b95ffb640 1 -- 172.21.15.76:0/1119900952 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4ba80237b0 con 0x558055b56750 2024-03-12T06:01:36.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4b95ffb640 1 -- 172.21.15.76:0/1119900952 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ba8023970 con 0x558055b56750 2024-03-12T06:01:36.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4bb7a27740 10 client.16857 fetch_fsmap learned FSMap version 28 2024-03-12T06:01:36.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1119900952 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558055a758c0 con 0x558055b56750 2024-03-12T06:01:36.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4b95ffb640 1 -- 172.21.15.76:0/1119900952 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ba4004030 con 0x558055b56750 2024-03-12T06:01:36.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4b95ffb640 1 -- 172.21.15.76:0/1119900952 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4ba802b070 con 0x558055b56750 2024-03-12T06:01:36.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4bb7a27740 10 client.16857 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:01:36.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4bb7a27740 10 client.16857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:36.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4bb7a27740 20 client.16857 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:36.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/1119900952 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558055bbe1f0 0x558055bc0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:36.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4bb7a27740 4 client.16857 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:01:36.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.535+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1119900952 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558055840890 con 0x558055bbe1f0 2024-03-12T06:01:36.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4bae59c640 1 --2- 172.21.15.76:0/1119900952 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558055bbe1f0 0x558055bc0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:36.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4b95ffb640 10 client.16857 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:36.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4bae59c640 1 --2- 172.21.15.76:0/1119900952 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558055bbe1f0 0x558055bc0610 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14ee1a53e44ebb0 server_cookie=85f8250cb4b2702 in_seq=0 out_seq=0 2024-03-12T06:01:36.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4b95ffb640 1 -- 172.21.15.76:0/1119900952 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558055840890 con 0x558055bbe1f0 2024-03-12T06:01:36.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4b95ffb640 10 client.16857 handle_command_reply: tid=0 2024-03-12T06:01:36.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4bb7a27740 1 client.16857 shutdown 2024-03-12T06:01:36.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1119900952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 msgr2=0x558055b5ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/1119900952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b5ba80 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800ae40 tx=0x7f4ba8009cc0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1119900952 shutdown_connections 2024-03-12T06:01:36.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/1119900952 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558055bbe1f0 0x558055bc0610 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/1119900952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558055b56750 0x558055b5ba80 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4b95ffb640 0 client.16857 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:36.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/1119900952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558055b4db70 0x558055b5da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4bb7a27740 1 --2- 172.21.15.76:0/1119900952 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558055b4d250 0x558055b5d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1119900952 >> 172.21.15.76:0/1119900952 conn(0x558055b47980 msgr2=0x558055b51f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1119900952 shutdown_connections 2024-03-12T06:01:36.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4bb7a27740 1 -- 172.21.15.76:0/1119900952 wait complete. 2024-03-12T06:01:36.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.536+0000 7f4bb7a27740 20 client.16857 trim_cache size 0 max 16384 2024-03-12T06:01:36.564 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:01:36.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.537+0000 7f4bb4e4e640 1 -- 172.21.15.76:0/430419343 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4b980a1ba0 msgr2=0x7f4b980a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.537+0000 7f4bb4e4e640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4b980a1ba0 0x7f4b980a4060 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8009fd0 tx=0x7f4ba8009600 comp rx=0 tx=0).stop 2024-03-12T06:01:36.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.537+0000 7f4bb4e4e640 1 -- 172.21.15.76:0/430419343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bb01007c0 msgr2=0x7f4bb0198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:36.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.537+0000 7f4bb4e4e640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bb01007c0 0x7f4bb0198780 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000e9f0 tx=0x7f4ba000eeb0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.537+0000 7f4bb4e4e640 1 -- 172.21.15.76:0/430419343 shutdown_connections 2024-03-12T06:01:36.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.537+0000 7f4bb4e4e640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4b980a1ba0 0x7f4b980a4060 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.537+0000 7f4bb4e4e640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bb0101ff0 0x7f4bb01a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.537+0000 7f4bb4e4e640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4bb0101160 0x7f4bb0198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.537+0000 7f4bb4e4e640 1 --2- 172.21.15.76:0/430419343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bb01007c0 0x7f4bb0198780 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:36.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.537+0000 7f4bb4e4e640 1 -- 172.21.15.76:0/430419343 >> 172.21.15.76:0/430419343 conn(0x7f4bb0076570 msgr2=0x7f4bb0104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:36.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.537+0000 7f4bb4e4e640 1 -- 172.21.15.76:0/430419343 shutdown_connections 2024-03-12T06:01:36.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:36.537+0000 7f4bb4e4e640 1 -- 172.21.15.76:0/430419343 wait complete. 2024-03-12T06:01:37.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:37 smithi082 ceph-mon[36382]: pgmap v178: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:37.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:37 smithi088 ceph-mon[36585]: pgmap v178: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:37.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:37 smithi076 ceph-mon[32095]: pgmap v178: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:37.568 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T06:01:37.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:01:37.568 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T06:01:37.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.680+0000 7f0849d4b640 1 -- 172.21.15.76:0/2491163498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0844106120 msgr2=0x7f0844106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.680+0000 7f0849d4b640 1 --2- 172.21.15.76:0/2491163498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0844106120 0x7f0844106500 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f0834009930 tx=0x7f083402bc30 comp rx=0 tx=0).stop 2024-03-12T06:01:37.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.680+0000 7f0849d4b640 1 -- 172.21.15.76:0/2491163498 shutdown_connections 2024-03-12T06:01:37.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.680+0000 7f0849d4b640 1 --2- 172.21.15.76:0/2491163498 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0844100070 0x7f084410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.680+0000 7f0849d4b640 1 --2- 172.21.15.76:0/2491163498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08440ff6b0 0x7f08440ffb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.680+0000 7f0849d4b640 1 --2- 172.21.15.76:0/2491163498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0844106120 0x7f0844106500 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.680+0000 7f0849d4b640 1 -- 172.21.15.76:0/2491163498 >> 172.21.15.76:0/2491163498 conn(0x7f0844076580 msgr2=0x7f08440fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:37.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.681+0000 7f0849d4b640 1 -- 172.21.15.76:0/2491163498 shutdown_connections 2024-03-12T06:01:37.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.681+0000 7f0849d4b640 1 -- 172.21.15.76:0/2491163498 wait complete. 2024-03-12T06:01:37.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.681+0000 7f0849d4b640 1 Processor -- start 2024-03-12T06:01:37.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.681+0000 7f0849d4b640 1 -- start start 2024-03-12T06:01:37.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.681+0000 7f0849d4b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08440ff6b0 0x7f084419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f0849d4b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0844100070 0x7f084419db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f0849d4b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0844106120 0x7f08441976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f0849d4b640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0844197e40 con 0x7f08440ff6b0 2024-03-12T06:01:37.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f0849d4b640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0844197fb0 con 0x7f0844106120 2024-03-12T06:01:37.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f0849d4b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0844198170 con 0x7f0844100070 2024-03-12T06:01:37.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f08437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08440ff6b0 0x7f084419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f08437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08440ff6b0 0x7f084419d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39092/0 (socket says 172.21.15.76:39092) 2024-03-12T06:01:37.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f08437fe640 1 -- 172.21.15.76:0/601252581 learned_addr learned my addr 172.21.15.76:0/601252581 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:37.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f0843fff640 1 --2- 172.21.15.76:0/601252581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0844106120 0x7f08441976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f0842ffd640 1 --2- 172.21.15.76:0/601252581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0844100070 0x7f084419db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f08437fe640 1 -- 172.21.15.76:0/601252581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0844100070 msgr2=0x7f084419db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f08437fe640 1 --2- 172.21.15.76:0/601252581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0844100070 0x7f084419db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f08437fe640 1 -- 172.21.15.76:0/601252581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0844106120 msgr2=0x7f08441976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f08437fe640 1 --2- 172.21.15.76:0/601252581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0844106120 0x7f08441976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f08437fe640 1 -- 172.21.15.76:0/601252581 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0834009560 con 0x7f08440ff6b0 2024-03-12T06:01:37.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f08437fe640 1 --2- 172.21.15.76:0/601252581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08440ff6b0 0x7f084419d5f0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f0834009230 tx=0x7f0834004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:37.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f0840ff9640 1 -- 172.21.15.76:0/601252581 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834004440 con 0x7f08440ff6b0 2024-03-12T06:01:37.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f0849d4b640 1 -- 172.21.15.76:0/601252581 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08441983f0 con 0x7f08440ff6b0 2024-03-12T06:01:37.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f0840ff9640 1 -- 172.21.15.76:0/601252581 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08340052d0 con 0x7f08440ff6b0 2024-03-12T06:01:37.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f0840ff9640 1 -- 172.21.15.76:0/601252581 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834003d70 con 0x7f08440ff6b0 2024-03-12T06:01:37.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.682+0000 7f0849d4b640 1 -- 172.21.15.76:0/601252581 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08441a4570 con 0x7f08440ff6b0 2024-03-12T06:01:37.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.684+0000 7f0840ff9640 1 -- 172.21.15.76:0/601252581 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f0834005440 con 0x7f08440ff6b0 2024-03-12T06:01:37.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f084c9f3740 1 Processor -- start 2024-03-12T06:01:37.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f084c9f3740 1 -- start start 2024-03-12T06:01:37.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f084c9f3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba01380 0x55861ba01760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f0840ff9640 1 --2- 172.21.15.76:0/601252581 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f082c0a1ba0 0x7f082c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f084c9f3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01d30 0x55861ba0a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f084c9f3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba0a8b0 0x55861ba0cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f084c9f3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861b96ff90 con 0x55861ba01d30 2024-03-12T06:01:37.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f084c9f3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861b96d4c0 con 0x55861ba0a8b0 2024-03-12T06:01:37.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f084c9f3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861b968ca0 con 0x55861ba01380 2024-03-12T06:01:37.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f0842ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01d30 0x55861ba0a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f0843fff640 1 --2- 172.21.15.76:0/601252581 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f082c0a1ba0 0x7f082c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f0842ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01d30 0x55861ba0a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39104/0 (socket says 172.21.15.76:39104) 2024-03-12T06:01:37.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f0842ffd640 1 -- 172.21.15.76:0/2259181110 learned_addr learned my addr 172.21.15.76:0/2259181110 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:37.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f0840ff9640 1 -- 172.21.15.76:0/601252581 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f08340ef980 con 0x7f08440ff6b0 2024-03-12T06:01:37.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.685+0000 7f08437fe640 1 --2- 172.21.15.76:0/2259181110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba01380 0x55861ba01760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.686+0000 7f0843fff640 1 --2- 172.21.15.76:0/2259181110 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba0a8b0 0x55861ba0cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.686+0000 7f0842ffd640 1 -- 172.21.15.76:0/2259181110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba01380 msgr2=0x55861ba01760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.686+0000 7f0842ffd640 1 --2- 172.21.15.76:0/2259181110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba01380 0x55861ba01760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.686+0000 7f0842ffd640 1 -- 172.21.15.76:0/2259181110 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba0a8b0 msgr2=0x55861ba0cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.686+0000 7f0842ffd640 1 --2- 172.21.15.76:0/2259181110 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba0a8b0 0x55861ba0cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.686+0000 7f0842ffd640 1 -- 172.21.15.76:0/2259181110 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55861b92b0b0 con 0x55861ba01d30 2024-03-12T06:01:37.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.686+0000 7f0843fff640 1 --2- 172.21.15.76:0/601252581 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f082c0a1ba0 0x7f082c0a4060 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f083c00a780 tx=0x7f083c013ae0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:37.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.686+0000 7f0842ffd640 1 --2- 172.21.15.76:0/2259181110 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01d30 0x55861ba0a2e0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f083800bc90 tx=0x7f083800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a7cbaf07bf3fce server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:37.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.686+0000 7f0829ffb640 1 -- 172.21.15.76:0/2259181110 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083801a460 con 0x55861ba01d30 2024-03-12T06:01:37.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.686+0000 7f0829ffb640 1 -- 172.21.15.76:0/2259181110 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f083801aaa0 con 0x55861ba01d30 2024-03-12T06:01:37.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.686+0000 7f0829ffb640 1 -- 172.21.15.76:0/2259181110 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0838011a70 con 0x55861ba01d30 2024-03-12T06:01:37.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.687+0000 7f084c9f3740 1 -- 172.21.15.76:0/2259181110 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01d30 msgr2=0x55861ba0a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.687+0000 7f084c9f3740 1 --2- 172.21.15.76:0/2259181110 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01d30 0x55861ba0a2e0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f083800bc90 tx=0x7f083800f730 comp rx=0 tx=0).stop 2024-03-12T06:01:37.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.687+0000 7f084c9f3740 1 -- 172.21.15.76:0/2259181110 shutdown_connections 2024-03-12T06:01:37.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.687+0000 7f084c9f3740 1 --2- 172.21.15.76:0/2259181110 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba0a8b0 0x55861ba0cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.687+0000 7f084c9f3740 1 --2- 172.21.15.76:0/2259181110 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01d30 0x55861ba0a2e0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.687+0000 7f084c9f3740 1 --2- 172.21.15.76:0/2259181110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba01380 0x55861ba01760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.687+0000 7f084c9f3740 1 -- 172.21.15.76:0/2259181110 >> 172.21.15.76:0/2259181110 conn(0x55861b9fb840 msgr2=0x55861b9fbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:37.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.687+0000 7f084c9f3740 1 -- 172.21.15.76:0/2259181110 shutdown_connections 2024-03-12T06:01:37.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.687+0000 7f084c9f3740 1 -- 172.21.15.76:0/2259181110 wait complete. 2024-03-12T06:01:37.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.687+0000 7f084c9f3740 1 -- start start 2024-03-12T06:01:37.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.687+0000 7f084c9f3740 1 Processor -- start 2024-03-12T06:01:37.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f084c9f3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba01380 0x55861ba07820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f084c9f3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba01d30 0x55861ba07d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f084c9f3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba0a8b0 0x55861ba082a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f084c9f3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861b90df40 con 0x55861ba0a8b0 2024-03-12T06:01:37.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f084c9f3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861b63c4b0 con 0x55861ba01d30 2024-03-12T06:01:37.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f084c9f3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861b8ee670 con 0x55861ba01380 2024-03-12T06:01:37.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f084c9f3740 10 client.0 _ll_register_callbacks cb 0x55861b95bf70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:37.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f0843fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba0a8b0 0x55861ba082a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f0843fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba0a8b0 0x55861ba082a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39110/0 (socket says 172.21.15.76:39110) 2024-03-12T06:01:37.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f0843fff640 1 -- 172.21.15.76:0/540450426 learned_addr learned my addr 172.21.15.76:0/540450426 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:37.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f08437fe640 1 --2- 172.21.15.76:0/540450426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba01380 0x55861ba07820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f0842ffd640 1 --2- 172.21.15.76:0/540450426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba01d30 0x55861ba07d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f0843fff640 1 -- 172.21.15.76:0/540450426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba01380 msgr2=0x55861ba07820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f0843fff640 1 --2- 172.21.15.76:0/540450426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba01380 0x55861ba07820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f0843fff640 1 -- 172.21.15.76:0/540450426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba01d30 msgr2=0x55861ba07d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f0843fff640 1 --2- 172.21.15.76:0/540450426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba01d30 0x55861ba07d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f0843fff640 1 -- 172.21.15.76:0/540450426 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083800b8c0 con 0x55861ba0a8b0 2024-03-12T06:01:37.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f0843fff640 1 --2- 172.21.15.76:0/540450426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba0a8b0 0x55861ba082a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f083c011f70 tx=0x7f083c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94857aff3cc2a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:37.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f0822ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:37.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f0822ffd640 1 -- 172.21.15.76:0/540450426 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c003970 con 0x55861ba0a8b0 2024-03-12T06:01:37.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f0822ffd640 1 -- 172.21.15.76:0/540450426 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f083c023070 con 0x55861ba0a8b0 2024-03-12T06:01:37.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.688+0000 7f084c9f3740 1 -- 172.21.15.76:0/540450426 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55861b942f40 con 0x55861ba0a8b0 2024-03-12T06:01:37.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.689+0000 7f0822ffd640 1 -- 172.21.15.76:0/540450426 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f083c004b20 con 0x55861ba0a8b0 2024-03-12T06:01:37.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.689+0000 7f084c9f3740 10 client.16869 fetch_fsmap learned FSMap version 28 2024-03-12T06:01:37.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.689+0000 7f084c9f3740 1 -- 172.21.15.76:0/540450426 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55861b967320 con 0x55861ba0a8b0 2024-03-12T06:01:37.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.689+0000 7f0822ffd640 1 -- 172.21.15.76:0/540450426 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55861b942f40 con 0x55861ba0a8b0 2024-03-12T06:01:37.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.689+0000 7f0822ffd640 1 -- 172.21.15.76:0/540450426 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f083c02f730 con 0x55861ba0a8b0 2024-03-12T06:01:37.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.689+0000 7f084c9f3740 10 client.16869 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:01:37.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.689+0000 7f084c9f3740 10 client.16869 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:37.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.689+0000 7f084c9f3740 20 client.16869 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:37.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.689+0000 7f084c9f3740 1 --2- 172.21.15.76:0/540450426 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55861ba1adb0 0x55861ba1d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.689+0000 7f084c9f3740 4 client.16869 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:01:37.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.689+0000 7f084c9f3740 1 -- 172.21.15.76:0/540450426 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55861b95f310 con 0x55861ba1adb0 2024-03-12T06:01:37.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.689+0000 7f08437fe640 1 --2- 172.21.15.76:0/540450426 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55861ba1adb0 0x55861ba1d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.690+0000 7f0822ffd640 10 client.16869 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:37.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.690+0000 7f08437fe640 1 --2- 172.21.15.76:0/540450426 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55861ba1adb0 0x55861ba1d1d0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b85c94a9aad8c171 server_cookie=12be883feacc7761 in_seq=0 out_seq=0 2024-03-12T06:01:37.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.690+0000 7f0822ffd640 1 -- 172.21.15.76:0/540450426 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55861b95f310 con 0x55861ba1adb0 2024-03-12T06:01:37.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.690+0000 7f0822ffd640 10 client.16869 handle_command_reply: tid=0 2024-03-12T06:01:37.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.690+0000 7f084c9f3740 1 client.16869 shutdown 2024-03-12T06:01:37.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.690+0000 7f084c9f3740 1 -- 172.21.15.76:0/540450426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba0a8b0 msgr2=0x55861ba082a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.690+0000 7f084c9f3740 1 --2- 172.21.15.76:0/540450426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba0a8b0 0x55861ba082a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f083c011f70 tx=0x7f083c004ab0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.690+0000 7f084c9f3740 1 -- 172.21.15.76:0/540450426 shutdown_connections 2024-03-12T06:01:37.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.690+0000 7f084c9f3740 1 --2- 172.21.15.76:0/540450426 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55861ba1adb0 0x55861ba1d1d0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.690+0000 7f084c9f3740 1 --2- 172.21.15.76:0/540450426 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba0a8b0 0x55861ba082a0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.690+0000 7f084c9f3740 1 --2- 172.21.15.76:0/540450426 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba01d30 0x55861ba07d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.690+0000 7f084c9f3740 1 --2- 172.21.15.76:0/540450426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba01380 0x55861ba07820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.690+0000 7f0822ffd640 0 client.16869 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:37.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.690+0000 7f084c9f3740 1 -- 172.21.15.76:0/540450426 >> 172.21.15.76:0/540450426 conn(0x55861b9fb840 msgr2=0x55861b9fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:37.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.691+0000 7f084c9f3740 1 -- 172.21.15.76:0/540450426 shutdown_connections 2024-03-12T06:01:37.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.691+0000 7f084c9f3740 1 -- 172.21.15.76:0/540450426 wait complete. 2024-03-12T06:01:37.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.691+0000 7f084c9f3740 20 client.16869 trim_cache size 0 max 16384 2024-03-12T06:01:37.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.705+0000 7f084c9f3740 1 Processor -- start 2024-03-12T06:01:37.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f084c9f3740 1 -- start start 2024-03-12T06:01:37.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f084c9f3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01380 0x55861ba080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f084c9f3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba01d30 0x55861ba11940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f084c9f3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba0a8b0 0x55861ba11e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f084c9f3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861ba00e80 con 0x55861ba01380 2024-03-12T06:01:37.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f084c9f3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861ba08600 con 0x55861ba01d30 2024-03-12T06:01:37.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f084c9f3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861ba08770 con 0x55861ba0a8b0 2024-03-12T06:01:37.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f08437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01380 0x55861ba080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f08437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01380 0x55861ba080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39120/0 (socket says 172.21.15.76:39120) 2024-03-12T06:01:37.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f08437fe640 1 -- 172.21.15.76:0/4201582722 learned_addr learned my addr 172.21.15.76:0/4201582722 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:37.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f08437fe640 1 -- 172.21.15.76:0/4201582722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba0a8b0 msgr2=0x55861ba11e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f08437fe640 1 --2- 172.21.15.76:0/4201582722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba0a8b0 0x55861ba11e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f08437fe640 1 -- 172.21.15.76:0/4201582722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba01d30 msgr2=0x55861ba11940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f08437fe640 1 --2- 172.21.15.76:0/4201582722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba01d30 0x55861ba11940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f08437fe640 1 -- 172.21.15.76:0/4201582722 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55861b903c00 con 0x55861ba01380 2024-03-12T06:01:37.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f08437fe640 1 --2- 172.21.15.76:0/4201582722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01380 0x55861ba080c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f08340ed210 tx=0x7f08340dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b6273e6abc6cea5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:37.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f08297fa640 1 -- 172.21.15.76:0/4201582722 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08340c07e0 con 0x55861ba01380 2024-03-12T06:01:37.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f08297fa640 1 -- 172.21.15.76:0/4201582722 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08340c4e30 con 0x55861ba01380 2024-03-12T06:01:37.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.706+0000 7f08297fa640 1 -- 172.21.15.76:0/4201582722 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08340b8d00 con 0x55861ba01380 2024-03-12T06:01:37.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.707+0000 7f084c9f3740 1 -- 172.21.15.76:0/4201582722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01380 msgr2=0x55861ba080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.707+0000 7f084c9f3740 1 --2- 172.21.15.76:0/4201582722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01380 0x55861ba080c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f08340ed210 tx=0x7f08340dc040 comp rx=0 tx=0).stop 2024-03-12T06:01:37.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.707+0000 7f084c9f3740 1 -- 172.21.15.76:0/4201582722 shutdown_connections 2024-03-12T06:01:37.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.707+0000 7f084c9f3740 1 --2- 172.21.15.76:0/4201582722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba0a8b0 0x55861ba11e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.707+0000 7f084c9f3740 1 --2- 172.21.15.76:0/4201582722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba01d30 0x55861ba11940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.707+0000 7f084c9f3740 1 --2- 172.21.15.76:0/4201582722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01380 0x55861ba080c0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.707+0000 7f084c9f3740 1 -- 172.21.15.76:0/4201582722 >> 172.21.15.76:0/4201582722 conn(0x55861b9fb840 msgr2=0x55861b9fa050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:37.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.707+0000 7f084c9f3740 1 -- 172.21.15.76:0/4201582722 shutdown_connections 2024-03-12T06:01:37.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.707+0000 7f084c9f3740 1 -- 172.21.15.76:0/4201582722 wait complete. 2024-03-12T06:01:37.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.707+0000 7f084c9f3740 1 -- start start 2024-03-12T06:01:37.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.707+0000 7f084c9f3740 1 Processor -- start 2024-03-12T06:01:37.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f084c9f3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba01380 0x55861ba6bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f084c9f3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01d30 0x55861ba6c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f084c9f3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba0a8b0 0x55861ba11160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f084c9f3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861ba11720 con 0x55861ba01d30 2024-03-12T06:01:37.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f084c9f3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861ba11890 con 0x55861ba01380 2024-03-12T06:01:37.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f084c9f3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861ba11a50 con 0x55861ba0a8b0 2024-03-12T06:01:37.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f084c9f3740 10 client.0 _ll_register_callbacks cb 0x7f083800b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:37.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f0842ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01d30 0x55861ba6c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f0842ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01d30 0x55861ba6c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39136/0 (socket says 172.21.15.76:39136) 2024-03-12T06:01:37.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f0842ffd640 1 -- 172.21.15.76:0/2403203337 learned_addr learned my addr 172.21.15.76:0/2403203337 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:37.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f08437fe640 1 --2- 172.21.15.76:0/2403203337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba01380 0x55861ba6bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f0843fff640 1 --2- 172.21.15.76:0/2403203337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba0a8b0 0x55861ba11160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f0842ffd640 1 -- 172.21.15.76:0/2403203337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba0a8b0 msgr2=0x55861ba11160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f0842ffd640 1 --2- 172.21.15.76:0/2403203337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba0a8b0 0x55861ba11160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f0842ffd640 1 -- 172.21.15.76:0/2403203337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba01380 msgr2=0x55861ba6bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f0842ffd640 1 --2- 172.21.15.76:0/2403203337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba01380 0x55861ba6bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f0842ffd640 1 -- 172.21.15.76:0/2403203337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08441a4570 con 0x55861ba01d30 2024-03-12T06:01:37.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f0842ffd640 1 --2- 172.21.15.76:0/2403203337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01d30 0x55861ba6c180 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f083800bc90 tx=0x7f0838005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11d1478f3e54d60 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:37.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f0822ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:01:37.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f0822ffd640 1 -- 172.21.15.76:0/2403203337 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083801e040 con 0x55861ba01d30 2024-03-12T06:01:37.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f084c9f3740 1 -- 172.21.15.76:0/2403203337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55861b98e900 con 0x55861ba01d30 2024-03-12T06:01:37.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.708+0000 7f0822ffd640 1 -- 172.21.15.76:0/2403203337 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0838004680 con 0x55861ba01d30 2024-03-12T06:01:37.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.709+0000 7f0822ffd640 1 -- 172.21.15.76:0/2403203337 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0838005680 con 0x55861ba01d30 2024-03-12T06:01:37.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.709+0000 7f084c9f3740 10 client.16875 fetch_fsmap learned FSMap version 28 2024-03-12T06:01:37.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.709+0000 7f084c9f3740 1 -- 172.21.15.76:0/2403203337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55861b929910 con 0x55861ba01d30 2024-03-12T06:01:37.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.709+0000 7f0822ffd640 1 -- 172.21.15.76:0/2403203337 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0838032020 con 0x55861ba01d30 2024-03-12T06:01:37.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.709+0000 7f0822ffd640 1 -- 172.21.15.76:0/2403203337 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f083802ead0 con 0x55861ba01d30 2024-03-12T06:01:37.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.709+0000 7f084c9f3740 10 client.16875 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:01:37.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.709+0000 7f084c9f3740 10 client.16875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:37.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.709+0000 7f084c9f3740 20 client.16875 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:37.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.709+0000 7f084c9f3740 1 --2- 172.21.15.76:0/2403203337 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55861ba73ae0 0x55861ba75f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.709+0000 7f084c9f3740 4 client.16875 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T06:01:37.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.709+0000 7f084c9f3740 1 -- 172.21.15.76:0/2403203337 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55861b668860 con 0x55861ba73ae0 2024-03-12T06:01:37.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.709+0000 7f08437fe640 1 --2- 172.21.15.76:0/2403203337 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55861ba73ae0 0x55861ba75f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.710+0000 7f08437fe640 1 --2- 172.21.15.76:0/2403203337 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55861ba73ae0 0x55861ba75f00 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29a0d30554793c71 server_cookie=b5220934c68f0c1 in_seq=0 out_seq=0 2024-03-12T06:01:37.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.710+0000 7f0822ffd640 10 client.16875 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:37.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f0822ffd640 1 -- 172.21.15.76:0/2403203337 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55861b668860 con 0x55861ba73ae0 2024-03-12T06:01:37.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f0822ffd640 10 client.16875 handle_command_reply: tid=0 2024-03-12T06:01:37.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f084c9f3740 1 client.16875 shutdown 2024-03-12T06:01:37.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f084c9f3740 1 -- 172.21.15.76:0/2403203337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01d30 msgr2=0x55861ba6c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f084c9f3740 1 --2- 172.21.15.76:0/2403203337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01d30 0x55861ba6c180 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f083800bc90 tx=0x7f0838005040 comp rx=0 tx=0).stop 2024-03-12T06:01:37.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f084c9f3740 1 -- 172.21.15.76:0/2403203337 shutdown_connections 2024-03-12T06:01:37.727 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f084c9f3740 1 --2- 172.21.15.76:0/2403203337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55861ba0a8b0 0x55861ba11160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f084c9f3740 1 --2- 172.21.15.76:0/2403203337 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55861ba73ae0 0x55861ba75f00 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f084c9f3740 1 --2- 172.21.15.76:0/2403203337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55861ba01d30 0x55861ba6c180 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f084c9f3740 1 --2- 172.21.15.76:0/2403203337 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55861ba01380 0x55861ba6bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f084c9f3740 1 -- 172.21.15.76:0/2403203337 >> 172.21.15.76:0/2403203337 conn(0x55861b9fb840 msgr2=0x55861b9fa8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:37.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f0822ffd640 0 client.16875 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:37.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f084c9f3740 1 -- 172.21.15.76:0/2403203337 shutdown_connections 2024-03-12T06:01:37.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f084c9f3740 1 -- 172.21.15.76:0/2403203337 wait complete. 2024-03-12T06:01:37.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f084c9f3740 20 client.16875 trim_cache size 0 max 16384 2024-03-12T06:01:37.729 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:01:37.729 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T06:01:37.729 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "a7ff6bc6-5dd2-4789-9bdd-153d88fdd086", 2024-03-12T06:01:37.729 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T06:01:37.729 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:01:37.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f0849d4b640 1 -- 172.21.15.76:0/601252581 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f082c0a1ba0 msgr2=0x7f082c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f0849d4b640 1 --2- 172.21.15.76:0/601252581 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f082c0a1ba0 0x7f082c0a4060 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f083c00a780 tx=0x7f083c013ae0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.711+0000 7f0849d4b640 1 -- 172.21.15.76:0/601252581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08440ff6b0 msgr2=0x7f084419d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.712+0000 7f0849d4b640 1 --2- 172.21.15.76:0/601252581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08440ff6b0 0x7f084419d5f0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f0834009230 tx=0x7f0834004040 comp rx=0 tx=0).stop 2024-03-12T06:01:37.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.712+0000 7f0849d4b640 1 -- 172.21.15.76:0/601252581 shutdown_connections 2024-03-12T06:01:37.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.712+0000 7f0849d4b640 1 --2- 172.21.15.76:0/601252581 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f082c0a1ba0 0x7f082c0a4060 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.712+0000 7f0849d4b640 1 --2- 172.21.15.76:0/601252581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0844106120 0x7f08441976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.712+0000 7f0849d4b640 1 --2- 172.21.15.76:0/601252581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0844100070 0x7f084419db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.712+0000 7f0849d4b640 1 --2- 172.21.15.76:0/601252581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08440ff6b0 0x7f084419d5f0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.712+0000 7f0849d4b640 1 -- 172.21.15.76:0/601252581 >> 172.21.15.76:0/601252581 conn(0x7f0844076580 msgr2=0x7f0844104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:37.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.712+0000 7f0849d4b640 1 -- 172.21.15.76:0/601252581 shutdown_connections 2024-03-12T06:01:37.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.712+0000 7f0849d4b640 1 -- 172.21.15.76:0/601252581 wait complete. 2024-03-12T06:01:37.732 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:01:37.733 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:01:37.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.851+0000 7fd81b32c640 1 -- 172.21.15.76:0/2531821726 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd814100260 msgr2=0x7fd814100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.851+0000 7fd81b32c640 1 --2- 172.21.15.76:0/2531821726 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd814100260 0x7fd814100640 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd8100098f0 tx=0x7fd81002dd90 comp rx=0 tx=0).stop 2024-03-12T06:01:37.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.851+0000 7fd81b32c640 1 -- 172.21.15.76:0/2531821726 shutdown_connections 2024-03-12T06:01:37.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.851+0000 7fd81b32c640 1 --2- 172.21.15.76:0/2531821726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd814105330 0x7fd81410cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.851+0000 7fd81b32c640 1 --2- 172.21.15.76:0/2531821726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd814100c10 0x7fd814104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.851+0000 7fd81b32c640 1 --2- 172.21.15.76:0/2531821726 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd814100260 0x7fd814100640 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.851+0000 7fd81b32c640 1 -- 172.21.15.76:0/2531821726 >> 172.21.15.76:0/2531821726 conn(0x7fd814077180 msgr2=0x7fd814077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:37.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.851+0000 7fd81b32c640 1 -- 172.21.15.76:0/2531821726 shutdown_connections 2024-03-12T06:01:37.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.852+0000 7fd81b32c640 1 -- 172.21.15.76:0/2531821726 wait complete. 2024-03-12T06:01:37.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.852+0000 7fd81b32c640 1 Processor -- start 2024-03-12T06:01:37.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.852+0000 7fd81b32c640 1 -- start start 2024-03-12T06:01:37.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd81b32c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd814100260 0x7fd814195570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd81b32c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd814105330 0x7fd814195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd81b32c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd814199e30 0x7fd81419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd81b32c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81419a820 con 0x7fd814199e30 2024-03-12T06:01:37.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd81b32c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81419a990 con 0x7fd814100260 2024-03-12T06:01:37.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd81b32c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81419ab00 con 0x7fd814105330 2024-03-12T06:01:37.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd8190c8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd814100260 0x7fd814195570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd8188c7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd814105330 0x7fd814195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd8190c8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd814100260 0x7fd814195570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:55544/0 (socket says 172.21.15.76:55544) 2024-03-12T06:01:37.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd8198c9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd814199e30 0x7fd81419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd8190c8640 1 -- 172.21.15.76:0/1032903572 learned_addr learned my addr 172.21.15.76:0/1032903572 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:37.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd8188c7640 1 -- 172.21.15.76:0/1032903572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd814100260 msgr2=0x7fd814195570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd8188c7640 1 --2- 172.21.15.76:0/1032903572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd814100260 0x7fd814195570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd8188c7640 1 -- 172.21.15.76:0/1032903572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd814199e30 msgr2=0x7fd81419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd8188c7640 1 --2- 172.21.15.76:0/1032903572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd814199e30 0x7fd81419a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd8188c7640 1 -- 172.21.15.76:0/1032903572 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd810009490 con 0x7fd814105330 2024-03-12T06:01:37.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd8198c9640 1 --2- 172.21.15.76:0/1032903572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd814199e30 0x7fd81419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:01:37.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.853+0000 7fd8190c8640 1 --2- 172.21.15.76:0/1032903572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd814100260 0x7fd814195570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:01:37.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.854+0000 7fd8188c7640 1 --2- 172.21.15.76:0/1032903572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd814105330 0x7fd814195ab0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd80400e9c0 tx=0x7fd80400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:37.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.854+0000 7fd80a7fc640 1 -- 172.21.15.76:0/1032903572 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80400ccc0 con 0x7fd814105330 2024-03-12T06:01:37.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.854+0000 7fd80a7fc640 1 -- 172.21.15.76:0/1032903572 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd80400ce20 con 0x7fd814105330 2024-03-12T06:01:37.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.854+0000 7fd80a7fc640 1 -- 172.21.15.76:0/1032903572 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd804018880 con 0x7fd814105330 2024-03-12T06:01:37.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.854+0000 7fd81b32c640 1 -- 172.21.15.76:0/1032903572 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8141a20e0 con 0x7fd814105330 2024-03-12T06:01:37.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.854+0000 7fd81b32c640 1 -- 172.21.15.76:0/1032903572 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8141a2540 con 0x7fd814105330 2024-03-12T06:01:37.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.857+0000 7fd81dfde740 1 Processor -- start 2024-03-12T06:01:37.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.857+0000 7fd80a7fc640 1 -- 172.21.15.76:0/1032903572 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd8040189e0 con 0x7fd814105330 2024-03-12T06:01:37.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd81dfde740 1 -- start start 2024-03-12T06:01:37.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd81dfde740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082992b0 0x562708299690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd81dfde740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd81dfde740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082a27b0 0x5627082a4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd81dfde740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562708208280 con 0x562708299bd0 2024-03-12T06:01:37.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd81dfde740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627082057b0 con 0x5627082992b0 2024-03-12T06:01:37.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd81dfde740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562708200f90 con 0x5627082a27b0 2024-03-12T06:01:37.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd8198c9640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082992b0 0x562708299690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd8190c8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd8198c9640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082992b0 0x562708299690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:55560/0 (socket says 172.21.15.76:55560) 2024-03-12T06:01:37.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd8198c9640 1 -- 172.21.15.76:0/126614739 learned_addr learned my addr 172.21.15.76:0/126614739 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:37.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd80a7fc640 1 --2- 172.21.15.76:0/1032903572 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8000a1ca0 0x7fd8000a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd8188c7640 1 --2- 172.21.15.76:0/126614739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082a27b0 0x5627082a4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd8198c9640 1 -- 172.21.15.76:0/126614739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082a27b0 msgr2=0x5627082a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd8198c9640 1 --2- 172.21.15.76:0/126614739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082a27b0 0x5627082a4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.858+0000 7fd8198c9640 1 -- 172.21.15.76:0/126614739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 msgr2=0x5627082a2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.859+0000 7fd8198c9640 1 --2- 172.21.15.76:0/126614739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.859+0000 7fd8198c9640 1 -- 172.21.15.76:0/126614739 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627081bf570 con 0x5627082992b0 2024-03-12T06:01:37.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.859+0000 7fd80a7fc640 1 -- 172.21.15.76:0/1032903572 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd804014070 con 0x7fd814105330 2024-03-12T06:01:37.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.859+0000 7fd8198c9640 1 --2- 172.21.15.76:0/1032903572 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8000a1ca0 0x7fd8000a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.859+0000 7fd8198c9640 1 --2- 172.21.15.76:0/126614739 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082992b0 0x562708299690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00e940 tx=0x7fd80c00ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=898e5f5c9df63ee1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:37.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.859+0000 7fd7ff7fe640 1 -- 172.21.15.76:0/126614739 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c003b70 con 0x5627082992b0 2024-03-12T06:01:37.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.859+0000 7fd7ff7fe640 1 -- 172.21.15.76:0/126614739 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd80c003d50 con 0x5627082992b0 2024-03-12T06:01:37.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.859+0000 7fd7ff7fe640 1 -- 172.21.15.76:0/126614739 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c01ab30 con 0x5627082992b0 2024-03-12T06:01:37.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.860+0000 7fd81dfde740 1 -- 172.21.15.76:0/126614739 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082992b0 msgr2=0x562708299690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.860+0000 7fd81dfde740 1 --2- 172.21.15.76:0/126614739 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082992b0 0x562708299690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00e940 tx=0x7fd80c00ee00 comp rx=0 tx=0).stop 2024-03-12T06:01:37.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.860+0000 7fd8198c9640 1 --2- 172.21.15.76:0/1032903572 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8000a1ca0 0x7fd8000a4160 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd80c020000 tx=0x7fd80c0197a0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:37.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.860+0000 7fd81dfde740 1 -- 172.21.15.76:0/126614739 shutdown_connections 2024-03-12T06:01:37.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.860+0000 7fd81dfde740 1 --2- 172.21.15.76:0/126614739 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082a27b0 0x5627082a4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.860+0000 7fd81dfde740 1 --2- 172.21.15.76:0/126614739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.860+0000 7fd81dfde740 1 --2- 172.21.15.76:0/126614739 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082992b0 0x562708299690 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.860+0000 7fd81dfde740 1 -- 172.21.15.76:0/126614739 >> 172.21.15.76:0/126614739 conn(0x562708293990 msgr2=0x562708293d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:37.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.860+0000 7fd81dfde740 1 -- 172.21.15.76:0/126614739 shutdown_connections 2024-03-12T06:01:37.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.860+0000 7fd81dfde740 1 -- 172.21.15.76:0/126614739 wait complete. 2024-03-12T06:01:37.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.860+0000 7fd81dfde740 1 -- start start 2024-03-12T06:01:37.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.860+0000 7fd81dfde740 1 Processor -- start 2024-03-12T06:01:37.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd81dfde740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082992b0 0x5627082a2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd81dfde740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562708299bd0 0x56270829f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd81dfde740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627082a27b0 0x56270829fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd81dfde740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562707ed4430 con 0x5627082a27b0 2024-03-12T06:01:37.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd81dfde740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562707e90e00 con 0x562708299bd0 2024-03-12T06:01:37.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd81dfde740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562708185fd0 con 0x5627082992b0 2024-03-12T06:01:37.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd81dfde740 10 client.0 _ll_register_callbacks cb 0x5627081dbfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:37.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd8190c8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082992b0 0x5627082a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd8190c8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082992b0 0x5627082a2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.76:32808/0 (socket says 172.21.15.76:32808) 2024-03-12T06:01:37.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd8190c8640 1 -- 172.21.15.76:0/826825159 learned_addr learned my addr 172.21.15.76:0/826825159 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:37.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd8198c9640 1 --2- 172.21.15.76:0/826825159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627082a27b0 0x56270829fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd8188c7640 1 --2- 172.21.15.76:0/826825159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562708299bd0 0x56270829f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd8188c7640 1 -- 172.21.15.76:0/826825159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082992b0 msgr2=0x5627082a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd8188c7640 1 --2- 172.21.15.76:0/826825159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082992b0 0x5627082a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd8188c7640 1 -- 172.21.15.76:0/826825159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627082a27b0 msgr2=0x56270829fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd8188c7640 1 --2- 172.21.15.76:0/826825159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627082a27b0 0x56270829fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd8188c7640 1 -- 172.21.15.76:0/826825159 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd80c00e570 con 0x562708299bd0 2024-03-12T06:01:37.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.861+0000 7fd8190c8640 1 --2- 172.21.15.76:0/826825159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082992b0 0x5627082a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:01:37.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.862+0000 7fd8188c7640 1 --2- 172.21.15.76:0/826825159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562708299bd0 0x56270829f5f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd8040cb210 tx=0x7fd80409a3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a722268f499d0af6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:37.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.862+0000 7fd7fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T06:01:37.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.862+0000 7fd7fcff9640 1 -- 172.21.15.76:0/826825159 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80409a7a0 con 0x562708299bd0 2024-03-12T06:01:37.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.862+0000 7fd7fcff9640 1 -- 172.21.15.76:0/826825159 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd8040ba0e0 con 0x562708299bd0 2024-03-12T06:01:37.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.862+0000 7fd81dfde740 1 -- 172.21.15.76:0/826825159 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562707f007e0 con 0x562708299bd0 2024-03-12T06:01:37.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.862+0000 7fd7fcff9640 1 -- 172.21.15.76:0/826825159 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8040ba2a0 con 0x562708299bd0 2024-03-12T06:01:37.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.862+0000 7fd81dfde740 10 client.26102 fetch_fsmap learned FSMap version 28 2024-03-12T06:01:37.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.862+0000 7fd81dfde740 1 -- 172.21.15.76:0/826825159 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5627081ff610 con 0x562708299bd0 2024-03-12T06:01:37.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.862+0000 7fd7fcff9640 1 -- 172.21.15.76:0/826825159 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8040b3140 con 0x562708299bd0 2024-03-12T06:01:37.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.863+0000 7fd7fcff9640 1 -- 172.21.15.76:0/826825159 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd8040964f0 con 0x562708299bd0 2024-03-12T06:01:37.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.863+0000 7fd81dfde740 10 client.26102 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:01:37.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.863+0000 7fd81dfde740 10 client.26102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:37.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.863+0000 7fd81dfde740 20 client.26102 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:37.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.863+0000 7fd81dfde740 1 --2- 172.21.15.76:0/826825159 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5627082b2990 0x5627082b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.863+0000 7fd81dfde740 4 client.26102 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:01:37.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.863+0000 7fd81dfde740 1 -- 172.21.15.76:0/826825159 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562707f007e0 con 0x5627082b2990 2024-03-12T06:01:37.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.863+0000 7fd8190c8640 1 --2- 172.21.15.76:0/826825159 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5627082b2990 0x5627082b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.863+0000 7fd8190c8640 1 --2- 172.21.15.76:0/826825159 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5627082b2990 0x5627082b4df0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cb60d955595c998 server_cookie=e5418259caa737aa in_seq=0 out_seq=0 2024-03-12T06:01:37.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.863+0000 7fd7fcff9640 10 client.26102 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:37.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.864+0000 7fd7fcff9640 1 -- 172.21.15.76:0/826825159 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562707f007e0 con 0x5627082b2990 2024-03-12T06:01:37.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.864+0000 7fd7fcff9640 10 client.26102 handle_command_reply: tid=0 2024-03-12T06:01:37.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.864+0000 7fd81dfde740 1 client.26102 shutdown 2024-03-12T06:01:37.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.864+0000 7fd81dfde740 1 -- 172.21.15.76:0/826825159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562708299bd0 msgr2=0x56270829f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.864+0000 7fd81dfde740 1 --2- 172.21.15.76:0/826825159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562708299bd0 0x56270829f5f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd8040cb210 tx=0x7fd80409a3a0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.864+0000 7fd81dfde740 1 -- 172.21.15.76:0/826825159 shutdown_connections 2024-03-12T06:01:37.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.864+0000 7fd81dfde740 1 --2- 172.21.15.76:0/826825159 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5627082b2990 0x5627082b4df0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.864+0000 7fd81dfde740 1 --2- 172.21.15.76:0/826825159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5627082a27b0 0x56270829fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.864+0000 7fd81dfde740 1 --2- 172.21.15.76:0/826825159 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562708299bd0 0x56270829f5f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.864+0000 7fd7fcff9640 0 client.26102 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:37.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.864+0000 7fd81dfde740 1 --2- 172.21.15.76:0/826825159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082992b0 0x5627082a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.864+0000 7fd81dfde740 1 -- 172.21.15.76:0/826825159 >> 172.21.15.76:0/826825159 conn(0x562708293990 msgr2=0x562708295730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:37.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.864+0000 7fd81dfde740 1 -- 172.21.15.76:0/826825159 shutdown_connections 2024-03-12T06:01:37.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.864+0000 7fd81dfde740 1 -- 172.21.15.76:0/826825159 wait complete. 2024-03-12T06:01:37.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.865+0000 7fd81dfde740 20 client.26102 trim_cache size 0 max 16384 2024-03-12T06:01:37.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.885+0000 7fd81dfde740 1 Processor -- start 2024-03-12T06:01:37.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.885+0000 7fd81dfde740 1 -- start start 2024-03-12T06:01:37.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.885+0000 7fd81dfde740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082992b0 0x5627082a9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.885+0000 7fd81dfde740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.885+0000 7fd81dfde740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082a27b0 0x5627082a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.885+0000 7fd81dfde740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562708294c40 con 0x562708299bd0 2024-03-12T06:01:37.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.885+0000 7fd81dfde740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562708298f10 con 0x5627082992b0 2024-03-12T06:01:37.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.885+0000 7fd81dfde740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562708294a10 con 0x5627082a27b0 2024-03-12T06:01:37.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.885+0000 7fd8188c7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.885+0000 7fd8190c8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082992b0 0x5627082a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.885+0000 7fd8190c8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082992b0 0x5627082a9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:55588/0 (socket says 172.21.15.76:55588) 2024-03-12T06:01:37.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.885+0000 7fd8190c8640 1 -- 172.21.15.76:0/2478075731 learned_addr learned my addr 172.21.15.76:0/2478075731 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:37.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.885+0000 7fd8198c9640 1 --2- 172.21.15.76:0/2478075731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082a27b0 0x5627082a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.886+0000 7fd8190c8640 1 -- 172.21.15.76:0/2478075731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082a27b0 msgr2=0x5627082a1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.886+0000 7fd8190c8640 1 --2- 172.21.15.76:0/2478075731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082a27b0 0x5627082a1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.886+0000 7fd8190c8640 1 -- 172.21.15.76:0/2478075731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 msgr2=0x5627082a9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.886+0000 7fd8190c8640 1 --2- 172.21.15.76:0/2478075731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.886+0000 7fd8190c8640 1 -- 172.21.15.76:0/2478075731 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56270819bac0 con 0x5627082992b0 2024-03-12T06:01:37.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.886+0000 7fd8188c7640 1 --2- 172.21.15.76:0/2478075731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:01:37.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.886+0000 7fd8190c8640 1 --2- 172.21.15.76:0/2478075731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082992b0 0x5627082a9660 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd8100098c0 tx=0x7fd810030040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=727ef89e91eeb701 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:37.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.886+0000 7fd7feffd640 1 -- 172.21.15.76:0/2478075731 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd810045040 con 0x5627082992b0 2024-03-12T06:01:37.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.886+0000 7fd7feffd640 1 -- 172.21.15.76:0/2478075731 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd81002edc0 con 0x5627082992b0 2024-03-12T06:01:37.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.886+0000 7fd7feffd640 1 -- 172.21.15.76:0/2478075731 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd810040cd0 con 0x5627082992b0 2024-03-12T06:01:37.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.887+0000 7fd81dfde740 1 -- 172.21.15.76:0/2478075731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082992b0 msgr2=0x5627082a9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.887+0000 7fd81dfde740 1 --2- 172.21.15.76:0/2478075731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082992b0 0x5627082a9660 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd8100098c0 tx=0x7fd810030040 comp rx=0 tx=0).stop 2024-03-12T06:01:37.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.887+0000 7fd81dfde740 1 -- 172.21.15.76:0/2478075731 shutdown_connections 2024-03-12T06:01:37.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.887+0000 7fd81dfde740 1 --2- 172.21.15.76:0/2478075731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082a27b0 0x5627082a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.887+0000 7fd81dfde740 1 --2- 172.21.15.76:0/2478075731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.887+0000 7fd81dfde740 1 --2- 172.21.15.76:0/2478075731 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082992b0 0x5627082a9660 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.887+0000 7fd81dfde740 1 -- 172.21.15.76:0/2478075731 >> 172.21.15.76:0/2478075731 conn(0x562708293990 msgr2=0x562708293d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:37.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.887+0000 7fd81dfde740 1 -- 172.21.15.76:0/2478075731 shutdown_connections 2024-03-12T06:01:37.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.887+0000 7fd81dfde740 1 -- 172.21.15.76:0/2478075731 wait complete. 2024-03-12T06:01:37.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.888+0000 7fd81dfde740 1 -- start start 2024-03-12T06:01:37.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.888+0000 7fd81dfde740 1 Processor -- start 2024-03-12T06:01:37.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.888+0000 7fd81dfde740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082992b0 0x5627082a8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.888+0000 7fd81dfde740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.888+0000 7fd81dfde740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082a27b0 0x562708308f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.888+0000 7fd81dfde740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627082a92a0 con 0x562708299bd0 2024-03-12T06:01:37.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.888+0000 7fd81dfde740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562708309460 con 0x5627082a27b0 2024-03-12T06:01:37.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.888+0000 7fd81dfde740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627083095a0 con 0x5627082992b0 2024-03-12T06:01:37.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.888+0000 7fd8188c7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd81dfde740 10 client.0 _ll_register_callbacks cb 0x7fd80c00e450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:01:37.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd8190c8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082992b0 0x5627082a8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd8188c7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39182/0 (socket says 172.21.15.76:39182) 2024-03-12T06:01:37.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd8188c7640 1 -- 172.21.15.76:0/228510233 learned_addr learned my addr 172.21.15.76:0/228510233 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:01:37.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd8198c9640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082a27b0 0x562708308f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd8198c9640 1 -- 172.21.15.76:0/228510233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082992b0 msgr2=0x5627082a8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd8198c9640 1 --2- 172.21.15.76:0/228510233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082992b0 0x5627082a8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd8198c9640 1 -- 172.21.15.76:0/228510233 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 msgr2=0x5627082a89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:01:37.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd8198c9640 1 --2- 172.21.15.76:0/228510233 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd8198c9640 1 -- 172.21.15.76:0/228510233 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627081ff610 con 0x5627082a27b0 2024-03-12T06:01:37.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd8188c7640 1 --2- 172.21.15.76:0/228510233 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:01:37.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd8198c9640 1 --2- 172.21.15.76:0/228510233 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082a27b0 0x562708308f20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00d7b0 tx=0x7fd80c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c88c2247ae29409 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:01:37.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd7fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T06:01:37.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd7fcff9640 1 -- 172.21.15.76:0/228510233 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c013030 con 0x5627082a27b0 2024-03-12T06:01:37.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd7fcff9640 1 -- 172.21.15.76:0/228510233 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd80c028070 con 0x5627082a27b0 2024-03-12T06:01:37.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd81dfde740 1 -- 172.21.15.76:0/228510233 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd80c003d50 con 0x5627082a27b0 2024-03-12T06:01:37.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.889+0000 7fd7fcff9640 1 -- 172.21.15.76:0/228510233 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd80c005eb0 con 0x5627082a27b0 2024-03-12T06:01:37.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.890+0000 7fd81dfde740 10 client.26108 fetch_fsmap learned FSMap version 28 2024-03-12T06:01:37.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.890+0000 7fd81dfde740 1 -- 172.21.15.76:0/228510233 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fd81410d710 con 0x5627082a27b0 2024-03-12T06:01:37.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.890+0000 7fd7fcff9640 1 -- 172.21.15.76:0/228510233 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd80c003d50 con 0x5627082a27b0 2024-03-12T06:01:37.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.890+0000 7fd7fcff9640 1 -- 172.21.15.76:0/228510233 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd80c043450 con 0x5627082a27b0 2024-03-12T06:01:37.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.890+0000 7fd81dfde740 10 client.26108 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:01:37.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.890+0000 7fd81dfde740 10 client.26108 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:01:37.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.890+0000 7fd81dfde740 20 client.26108 populate_metadata read hostname 'smithi076' 2024-03-12T06:01:37.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.890+0000 7fd81dfde740 1 --2- 172.21.15.76:0/228510233 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56270830ad80 0x56270830d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:01:37.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.890+0000 7fd81dfde740 4 client.26108 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:01:37.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.890+0000 7fd81dfde740 1 -- 172.21.15.76:0/228510233 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562707f3d720 con 0x56270830ad80 2024-03-12T06:01:37.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.890+0000 7fd8190c8640 1 --2- 172.21.15.76:0/228510233 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56270830ad80 0x56270830d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:01:37.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd7fcff9640 10 client.26108 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:37.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd8190c8640 1 --2- 172.21.15.76:0/228510233 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56270830ad80 0x56270830d1a0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bc33fd170190530 server_cookie=29c8f6406a1ade4e in_seq=0 out_seq=0 2024-03-12T06:01:37.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd7fcff9640 1 -- 172.21.15.76:0/228510233 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x562707f3d720 con 0x56270830ad80 2024-03-12T06:01:37.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd7fcff9640 10 client.26108 handle_command_reply: tid=0 2024-03-12T06:01:37.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd81dfde740 1 client.26108 shutdown 2024-03-12T06:01:37.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd81dfde740 1 -- 172.21.15.76:0/228510233 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082a27b0 msgr2=0x562708308f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd81dfde740 1 --2- 172.21.15.76:0/228510233 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082a27b0 0x562708308f20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00d7b0 tx=0x7fd80c017000 comp rx=0 tx=0).stop 2024-03-12T06:01:37.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd81dfde740 1 -- 172.21.15.76:0/228510233 shutdown_connections 2024-03-12T06:01:37.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd81dfde740 1 --2- 172.21.15.76:0/228510233 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5627082a27b0 0x562708308f20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd81dfde740 1 --2- 172.21.15.76:0/228510233 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56270830ad80 0x56270830d1a0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd81dfde740 1 --2- 172.21.15.76:0/228510233 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562708299bd0 0x5627082a89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd81dfde740 1 --2- 172.21.15.76:0/228510233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5627082992b0 0x5627082a8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd7fcff9640 0 client.26108 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:01:37.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd81dfde740 1 -- 172.21.15.76:0/228510233 >> 172.21.15.76:0/228510233 conn(0x562708293990 msgr2=0x56270829a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:37.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd81dfde740 1 -- 172.21.15.76:0/228510233 shutdown_connections 2024-03-12T06:01:37.907 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.891+0000 7fd81dfde740 1 -- 172.21.15.76:0/228510233 wait complete. 2024-03-12T06:01:37.908 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.892+0000 7fd81dfde740 20 client.26108 trim_cache size 0 max 16384 2024-03-12T06:01:37.908 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:01:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T06:01:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T06:01:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: "a7ff6bc6-5dd2-4789-9bdd-153d88fdd086": { 2024-03-12T06:01:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T06:01:37.909 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "a7ff6bc6-5dd2-4789-9bdd-153d88fdd086", 2024-03-12T06:01:37.910 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T06:01:37.910 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:01:37.910 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:01:37.910 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:01:37.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.892+0000 7fd81b32c640 1 -- 172.21.15.76:0/1032903572 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8000a1ca0 msgr2=0x7fd8000a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.892+0000 7fd81b32c640 1 --2- 172.21.15.76:0/1032903572 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8000a1ca0 0x7fd8000a4160 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd80c020000 tx=0x7fd80c0197a0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.892+0000 7fd81b32c640 1 -- 172.21.15.76:0/1032903572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd814105330 msgr2=0x7fd814195ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:01:37.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.892+0000 7fd81b32c640 1 --2- 172.21.15.76:0/1032903572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd814105330 0x7fd814195ab0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd80400e9c0 tx=0x7fd80400ee80 comp rx=0 tx=0).stop 2024-03-12T06:01:37.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.892+0000 7fd81b32c640 1 -- 172.21.15.76:0/1032903572 shutdown_connections 2024-03-12T06:01:37.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.892+0000 7fd81b32c640 1 --2- 172.21.15.76:0/1032903572 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd8000a1ca0 0x7fd8000a4160 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.892+0000 7fd81b32c640 1 --2- 172.21.15.76:0/1032903572 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd814199e30 0x7fd81419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.892+0000 7fd81b32c640 1 --2- 172.21.15.76:0/1032903572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd814105330 0x7fd814195ab0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.892+0000 7fd81b32c640 1 --2- 172.21.15.76:0/1032903572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd814100260 0x7fd814195570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:01:37.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.892+0000 7fd81b32c640 1 -- 172.21.15.76:0/1032903572 >> 172.21.15.76:0/1032903572 conn(0x7fd814077180 msgr2=0x7fd814106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:01:37.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.892+0000 7fd81b32c640 1 -- 172.21.15.76:0/1032903572 shutdown_connections 2024-03-12T06:01:37.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:01:37.892+0000 7fd81b32c640 1 -- 172.21.15.76:0/1032903572 wait complete. 2024-03-12T06:01:37.914 INFO:tasks.cephfs.filesystem:scrub status for tag:a7ff6bc6-5dd2-4789-9bdd-153d88fdd086 - {'path': '/', 'tag': 'a7ff6bc6-5dd2-4789-9bdd-153d88fdd086', 'options': 'recursive,force'} 2024-03-12T06:01:38.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:38 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:01:38.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:38 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:01:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:38 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:01:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:38 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:01:38.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:38 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:01:38.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:38 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:01:39.445 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:39 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T06:01:39.445 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:39 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:39.445 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:39 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:01:39.445 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:39 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:39.446 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:39 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:39.446 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:39 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:39.446 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:39 smithi088 ceph-mon[36585]: pgmap v179: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:01:39.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:39 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T06:01:39.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:39 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:39.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:39 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:01:39.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:39 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:39.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:39 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:39.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:39 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:39.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:39 smithi082 ceph-mon[36382]: pgmap v179: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:01:39.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:39 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T06:01:39.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:39 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:39.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:39 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:01:39.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:39 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:39.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:39 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:39.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:39 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:01:39.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:39 smithi076 ceph-mon[32095]: pgmap v179: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:01:42.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:42 smithi082 ceph-mon[36382]: pgmap v180: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:01:42.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:42 smithi088 ceph-mon[36585]: pgmap v180: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:01:42.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:42 smithi076 ceph-mon[32095]: pgmap v180: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:01:44.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:44 smithi082 ceph-mon[36382]: pgmap v181: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:01:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:44 smithi088 ceph-mon[36585]: pgmap v181: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:01:44.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:44 smithi076 ceph-mon[32095]: pgmap v181: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:01:46.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:46 smithi082 ceph-mon[36382]: pgmap v182: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:01:46.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:46 smithi088 ceph-mon[36585]: pgmap v182: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:01:46.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:46 smithi076 ceph-mon[32095]: pgmap v182: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:01:48.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:48 smithi082 ceph-mon[36382]: pgmap v183: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:01:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:48 smithi088 ceph-mon[36585]: pgmap v183: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:01:48.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:48 smithi076 ceph-mon[32095]: pgmap v183: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:01:50.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:50 smithi082 ceph-mon[36382]: pgmap v184: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:01:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:50 smithi088 ceph-mon[36585]: pgmap v184: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:01:50.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:50 smithi076 ceph-mon[32095]: pgmap v184: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:01:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:52 smithi082 ceph-mon[36382]: pgmap v185: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:52.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:52 smithi088 ceph-mon[36585]: pgmap v185: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:52.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:52 smithi076 ceph-mon[32095]: pgmap v185: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:53.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:53 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:01:53.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:53 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:01:53.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:53 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:01:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:53 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:01:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:53 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:01:53.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:53 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:01:53.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:53 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:01:53.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:53 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:01:53.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:53 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:01:54.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:54 smithi082 ceph-mon[36382]: pgmap v186: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:54.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:54 smithi088 ceph-mon[36585]: pgmap v186: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:54.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:54 smithi076 ceph-mon[32095]: pgmap v186: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:01:55.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:55 smithi082 ceph-mon[36382]: pgmap v187: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:55.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:55 smithi088 ceph-mon[36585]: pgmap v187: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:55.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:55 smithi076 ceph-mon[32095]: pgmap v187: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:56.989 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:56 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:01:57.195 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:56 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:01:57.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:56 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:01:58.097 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:57 smithi082 ceph-mon[36382]: pgmap v188: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:58.097 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:57 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:01:58.097 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:57 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:01:58.097 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:01:57 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:01:58.136 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:57 smithi088 ceph-mon[36585]: pgmap v188: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:58.137 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:57 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:01:58.137 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:57 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:01:58.137 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:01:57 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:01:58.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:57 smithi076 ceph-mon[32095]: pgmap v188: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:01:58.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:57 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:01:58.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:57 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:01:58.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:01:57 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:02:00.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:00 smithi082 ceph-mon[36382]: pgmap v189: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:02:00.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:00 smithi088 ceph-mon[36585]: pgmap v189: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:02:00.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:00 smithi076 ceph-mon[32095]: pgmap v189: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:02:02.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:02 smithi082 ceph-mon[36382]: pgmap v190: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:02.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:02 smithi088 ceph-mon[36585]: pgmap v190: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:02.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:02 smithi076 ceph-mon[32095]: pgmap v190: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:04.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:04 smithi082 ceph-mon[36382]: pgmap v191: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-12T06:02:04.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:04 smithi088 ceph-mon[36585]: pgmap v191: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-12T06:02:04.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:04 smithi076 ceph-mon[32095]: pgmap v191: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-12T06:02:06.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:06 smithi082 ceph-mon[36382]: pgmap v192: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:02:06.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:06 smithi088 ceph-mon[36585]: pgmap v192: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:02:06.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:06 smithi076 ceph-mon[32095]: pgmap v192: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:02:07.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:02:07.916 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:02:08.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.025+0000 7f9e13cce640 1 -- 172.21.15.76:0/1293726274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e0c069e60 msgr2=0x7f9e0c06a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.025+0000 7f9e13cce640 1 --2- 172.21.15.76:0/1293726274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e0c069e60 0x7f9e0c06a2c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc009930 tx=0x7f9dfc02bc30 comp rx=0 tx=0).stop 2024-03-12T06:02:08.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.025+0000 7f9e13cce640 1 -- 172.21.15.76:0/1293726274 shutdown_connections 2024-03-12T06:02:08.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.025+0000 7f9e13cce640 1 --2- 172.21.15.76:0/1293726274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e0c103c40 0x7f9e0c1040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.025+0000 7f9e13cce640 1 --2- 172.21.15.76:0/1293726274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e0c069e60 0x7f9e0c06a2c0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.025+0000 7f9e13cce640 1 --2- 172.21.15.76:0/1293726274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e0c1060d0 0x7f9e0c069920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.025+0000 7f9e13cce640 1 -- 172.21.15.76:0/1293726274 >> 172.21.15.76:0/1293726274 conn(0x7f9e0c077120 msgr2=0x7f9e0c077530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.025+0000 7f9e13cce640 1 -- 172.21.15.76:0/1293726274 shutdown_connections 2024-03-12T06:02:08.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.025+0000 7f9e13cce640 1 -- 172.21.15.76:0/1293726274 wait complete. 2024-03-12T06:02:08.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.025+0000 7f9e13cce640 1 Processor -- start 2024-03-12T06:02:08.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.025+0000 7f9e13cce640 1 -- start start 2024-03-12T06:02:08.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e13cce640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e0c069e60 0x7f9e0c1a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e13cce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e0c103c40 0x7f9e0c1a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e13cce640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e0c1060d0 0x7f9e0c19dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e13cce640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c19e130 con 0x7f9e0c1060d0 2024-03-12T06:02:08.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e13cce640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c19e2a0 con 0x7f9e0c069e60 2024-03-12T06:02:08.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e13cce640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c19e460 con 0x7f9e0c103c40 2024-03-12T06:02:08.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e1226b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e0c1060d0 0x7f9e0c19dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e1226b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e0c1060d0 0x7f9e0c19dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49522/0 (socket says 172.21.15.76:49522) 2024-03-12T06:02:08.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e1226b640 1 -- 172.21.15.76:0/1211863476 learned_addr learned my addr 172.21.15.76:0/1211863476 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e11269640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e0c103c40 0x7f9e0c1a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e11a6a640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e0c069e60 0x7f9e0c1a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e1226b640 1 -- 172.21.15.76:0/1211863476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e0c103c40 msgr2=0x7f9e0c1a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e1226b640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e0c103c40 0x7f9e0c1a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e1226b640 1 -- 172.21.15.76:0/1211863476 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e0c069e60 msgr2=0x7f9e0c1a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e1226b640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e0c069e60 0x7f9e0c1a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e1226b640 1 -- 172.21.15.76:0/1211863476 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc009560 con 0x7f9e0c1060d0 2024-03-12T06:02:08.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e11a6a640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e0c069e60 0x7f9e0c1a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:08.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e1226b640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e0c1060d0 0x7f9e0c19dae0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400eaa0 tx=0x7f9e0400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e02ffd640 1 -- 172.21.15.76:0/1211863476 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0400cdb0 con 0x7f9e0c1060d0 2024-03-12T06:02:08.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e13cce640 1 -- 172.21.15.76:0/1211863476 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0c19e740 con 0x7f9e0c1060d0 2024-03-12T06:02:08.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e02ffd640 1 -- 172.21.15.76:0/1211863476 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e04005020 con 0x7f9e0c1060d0 2024-03-12T06:02:08.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e02ffd640 1 -- 172.21.15.76:0/1211863476 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e040116d0 con 0x7f9e0c1060d0 2024-03-12T06:02:08.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.026+0000 7f9e13cce640 1 -- 172.21.15.76:0/1211863476 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0c1aa9d0 con 0x7f9e0c1060d0 2024-03-12T06:02:08.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.028+0000 7f9e02ffd640 1 -- 172.21.15.76:0/1211863476 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9e04005490 con 0x7f9e0c1060d0 2024-03-12T06:02:08.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.029+0000 7f9e16978740 1 Processor -- start 2024-03-12T06:02:08.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.029+0000 7f9e16978740 1 -- start start 2024-03-12T06:02:08.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.029+0000 7f9e02ffd640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9df80a1ba0 0x7f9df80a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.029+0000 7f9e16978740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40c2b0 0x5629fd40c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.029+0000 7f9e16978740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd40cbd0 0x5629fd415270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.029+0000 7f9e16978740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd4157b0 0x5629fd417ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.029+0000 7f9e16978740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fd37b280 con 0x5629fd40cbd0 2024-03-12T06:02:08.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.029+0000 7f9e16978740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fd3787b0 con 0x5629fd4157b0 2024-03-12T06:02:08.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.029+0000 7f9e16978740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fd373f90 con 0x5629fd40c2b0 2024-03-12T06:02:08.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e11269640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9df80a1ba0 0x7f9df80a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e02ffd640 1 -- 172.21.15.76:0/1211863476 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9e040cd9b0 con 0x7f9e0c1060d0 2024-03-12T06:02:08.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e11269640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd40cbd0 0x5629fd415270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e1226b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd4157b0 0x5629fd417ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e11269640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd40cbd0 0x5629fd415270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49530/0 (socket says 172.21.15.76:49530) 2024-03-12T06:02:08.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e11269640 1 -- 172.21.15.76:0/4116891918 learned_addr learned my addr 172.21.15.76:0/4116891918 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e11a6a640 1 --2- 172.21.15.76:0/4116891918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40c2b0 0x5629fd40c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e11269640 1 -- 172.21.15.76:0/4116891918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40c2b0 msgr2=0x5629fd40c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e11269640 1 --2- 172.21.15.76:0/4116891918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40c2b0 0x5629fd40c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e11269640 1 -- 172.21.15.76:0/4116891918 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd4157b0 msgr2=0x5629fd417ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e11269640 1 --2- 172.21.15.76:0/4116891918 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd4157b0 0x5629fd417ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e11269640 1 -- 172.21.15.76:0/4116891918 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629fd332570 con 0x5629fd40cbd0 2024-03-12T06:02:08.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e11a6a640 1 --2- 172.21.15.76:0/4116891918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40c2b0 0x5629fd40c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:02:08.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e1226b640 1 --2- 172.21.15.76:0/4116891918 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd4157b0 0x5629fd417ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:02:08.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e11269640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9df80a1ba0 0x7f9df80a4060 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc004c50 tx=0x7f9dfc005fd0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9e11269640 1 --2- 172.21.15.76:0/4116891918 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd40cbd0 0x5629fd415270 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc036040 tx=0x7f9dfc048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9337c9cc1e453f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9df7fff640 1 -- 172.21.15.76:0/4116891918 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc048070 con 0x5629fd40cbd0 2024-03-12T06:02:08.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9df7fff640 1 -- 172.21.15.76:0/4116891918 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9dfc004410 con 0x5629fd40cbd0 2024-03-12T06:02:08.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.030+0000 7f9df7fff640 1 -- 172.21.15.76:0/4116891918 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc0037f0 con 0x5629fd40cbd0 2024-03-12T06:02:08.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.031+0000 7f9e16978740 1 -- 172.21.15.76:0/4116891918 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd40cbd0 msgr2=0x5629fd415270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.031+0000 7f9e16978740 1 --2- 172.21.15.76:0/4116891918 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd40cbd0 0x5629fd415270 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc036040 tx=0x7f9dfc048000 comp rx=0 tx=0).stop 2024-03-12T06:02:08.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.031+0000 7f9e16978740 1 -- 172.21.15.76:0/4116891918 shutdown_connections 2024-03-12T06:02:08.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.031+0000 7f9e16978740 1 --2- 172.21.15.76:0/4116891918 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd4157b0 0x5629fd417ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.031+0000 7f9e16978740 1 --2- 172.21.15.76:0/4116891918 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd40cbd0 0x5629fd415270 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.031+0000 7f9e16978740 1 --2- 172.21.15.76:0/4116891918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40c2b0 0x5629fd40c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.031+0000 7f9e16978740 1 -- 172.21.15.76:0/4116891918 >> 172.21.15.76:0/4116891918 conn(0x5629fd406990 msgr2=0x5629fd406d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.031+0000 7f9e16978740 1 -- 172.21.15.76:0/4116891918 shutdown_connections 2024-03-12T06:02:08.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.031+0000 7f9e16978740 1 -- 172.21.15.76:0/4116891918 wait complete. 2024-03-12T06:02:08.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.032+0000 7f9e16978740 1 -- start start 2024-03-12T06:02:08.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.032+0000 7f9e16978740 1 Processor -- start 2024-03-12T06:02:08.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.032+0000 7f9e16978740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40c2b0 0x5629fd415200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.032+0000 7f9e16978740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd40cbd0 0x5629fd412590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.032+0000 7f9e16978740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd4157b0 0x5629fd412cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.032+0000 7f9e16978740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fd047430 con 0x5629fd40cbd0 2024-03-12T06:02:08.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.032+0000 7f9e16978740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fd003e00 con 0x5629fd4157b0 2024-03-12T06:02:08.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.032+0000 7f9e16978740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fd2f8fd0 con 0x5629fd40c2b0 2024-03-12T06:02:08.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.032+0000 7f9e16978740 10 client.0 _ll_register_callbacks cb 0x5629fd34efd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:08.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.032+0000 7f9e11269640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd40cbd0 0x5629fd412590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.032+0000 7f9e11269640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd40cbd0 0x5629fd412590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49538/0 (socket says 172.21.15.76:49538) 2024-03-12T06:02:08.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.032+0000 7f9e11269640 1 -- 172.21.15.76:0/1880556474 learned_addr learned my addr 172.21.15.76:0/1880556474 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.032+0000 7f9e11a6a640 1 --2- 172.21.15.76:0/1880556474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40c2b0 0x5629fd415200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.032+0000 7f9e11269640 1 -- 172.21.15.76:0/1880556474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40c2b0 msgr2=0x5629fd415200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.033+0000 7f9e11269640 1 --2- 172.21.15.76:0/1880556474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40c2b0 0x5629fd415200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.033+0000 7f9e1226b640 1 --2- 172.21.15.76:0/1880556474 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd4157b0 0x5629fd412cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.033+0000 7f9e11269640 1 -- 172.21.15.76:0/1880556474 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd4157b0 msgr2=0x5629fd412cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.033+0000 7f9e11269640 1 --2- 172.21.15.76:0/1880556474 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd4157b0 0x5629fd412cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.033+0000 7f9e11269640 1 -- 172.21.15.76:0/1880556474 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0c114f10 con 0x5629fd40cbd0 2024-03-12T06:02:08.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.033+0000 7f9e11269640 1 --2- 172.21.15.76:0/1880556474 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd40cbd0 0x5629fd412590 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc02cbe0 tx=0x7f9dfc005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a447f89254c56f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.033+0000 7f9df57fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:08.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.033+0000 7f9df57fa640 1 -- 172.21.15.76:0/1880556474 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc003c80 con 0x5629fd40cbd0 2024-03-12T06:02:08.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.033+0000 7f9df57fa640 1 -- 172.21.15.76:0/1880556474 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9dfc003de0 con 0x5629fd40cbd0 2024-03-12T06:02:08.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.033+0000 7f9e16978740 1 -- 172.21.15.76:0/1880556474 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5629fd0737e0 con 0x5629fd40cbd0 2024-03-12T06:02:08.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.033+0000 7f9df57fa640 1 -- 172.21.15.76:0/1880556474 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9dfc004df0 con 0x5629fd40cbd0 2024-03-12T06:02:08.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.033+0000 7f9e16978740 10 client.16905 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:08.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.033+0000 7f9e16978740 1 -- 172.21.15.76:0/1880556474 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5629fd372610 con 0x5629fd40cbd0 2024-03-12T06:02:08.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.033+0000 7f9df57fa640 1 -- 172.21.15.76:0/1880556474 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5629fd0737e0 con 0x5629fd40cbd0 2024-03-12T06:02:08.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.034+0000 7f9df57fa640 1 -- 172.21.15.76:0/1880556474 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9dfc055070 con 0x5629fd40cbd0 2024-03-12T06:02:08.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.034+0000 7f9e16978740 10 client.16905 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:08.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.034+0000 7f9e16978740 10 client.16905 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:08.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.034+0000 7f9e16978740 20 client.16905 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:08.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.034+0000 7f9e16978740 1 --2- 172.21.15.76:0/1880556474 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5629fd4255a0 0x5629fd4279c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.034+0000 7f9e16978740 4 client.16905 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:02:08.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.034+0000 7f9e16978740 1 -- 172.21.15.76:0/1880556474 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5629fd0b0720 con 0x5629fd4255a0 2024-03-12T06:02:08.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.034+0000 7f9e11a6a640 1 --2- 172.21.15.76:0/1880556474 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5629fd4255a0 0x5629fd4279c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.034+0000 7f9e11a6a640 1 --2- 172.21.15.76:0/1880556474 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5629fd4255a0 0x5629fd4279c0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2992b9169920b1f2 server_cookie=6931edf075328d2b in_seq=0 out_seq=0 2024-03-12T06:02:08.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.034+0000 7f9df57fa640 10 client.16905 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:08.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9df57fa640 1 -- 172.21.15.76:0/1880556474 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5629fd0b0720 con 0x5629fd4255a0 2024-03-12T06:02:08.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9df57fa640 10 client.16905 handle_command_reply: tid=0 2024-03-12T06:02:08.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9e16978740 1 client.16905 shutdown 2024-03-12T06:02:08.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9e16978740 1 -- 172.21.15.76:0/1880556474 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd40cbd0 msgr2=0x5629fd412590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9e16978740 1 --2- 172.21.15.76:0/1880556474 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd40cbd0 0x5629fd412590 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc02cbe0 tx=0x7f9dfc005c90 comp rx=0 tx=0).stop 2024-03-12T06:02:08.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9e16978740 1 -- 172.21.15.76:0/1880556474 shutdown_connections 2024-03-12T06:02:08.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9e16978740 1 --2- 172.21.15.76:0/1880556474 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd4157b0 0x5629fd412cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9e16978740 1 --2- 172.21.15.76:0/1880556474 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5629fd4255a0 0x5629fd4279c0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9e16978740 1 --2- 172.21.15.76:0/1880556474 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd40cbd0 0x5629fd412590 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9df57fa640 0 client.16905 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:08.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9e16978740 1 --2- 172.21.15.76:0/1880556474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40c2b0 0x5629fd415200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9e16978740 1 -- 172.21.15.76:0/1880556474 >> 172.21.15.76:0/1880556474 conn(0x5629fd406990 msgr2=0x5629fd408730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9e16978740 1 -- 172.21.15.76:0/1880556474 shutdown_connections 2024-03-12T06:02:08.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9e16978740 1 -- 172.21.15.76:0/1880556474 wait complete. 2024-03-12T06:02:08.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.035+0000 7f9e16978740 20 client.16905 trim_cache size 0 max 16384 2024-03-12T06:02:08.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.050+0000 7f9e16978740 1 Processor -- start 2024-03-12T06:02:08.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.050+0000 7f9e16978740 1 -- start start 2024-03-12T06:02:08.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e16978740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd40c2b0 0x5629fd414cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e16978740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40cbd0 0x5629fd415200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e16978740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd4157b0 0x5629fd410e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e16978740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fd40bf10 con 0x5629fd4157b0 2024-03-12T06:02:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e16978740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fd4079e0 con 0x5629fd40c2b0 2024-03-12T06:02:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e16978740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fd413570 con 0x5629fd40cbd0 2024-03-12T06:02:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e1226b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd4157b0 0x5629fd410e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e1226b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd4157b0 0x5629fd410e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49540/0 (socket says 172.21.15.76:49540) 2024-03-12T06:02:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e1226b640 1 -- 172.21.15.76:0/1952263854 learned_addr learned my addr 172.21.15.76:0/1952263854 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e11a6a640 1 --2- 172.21.15.76:0/1952263854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd40c2b0 0x5629fd414cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e11269640 1 --2- 172.21.15.76:0/1952263854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40cbd0 0x5629fd415200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e1226b640 1 -- 172.21.15.76:0/1952263854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40cbd0 msgr2=0x5629fd415200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e1226b640 1 --2- 172.21.15.76:0/1952263854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40cbd0 0x5629fd415200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e1226b640 1 -- 172.21.15.76:0/1952263854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd40c2b0 msgr2=0x5629fd414cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e1226b640 1 --2- 172.21.15.76:0/1952263854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd40c2b0 0x5629fd414cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e1226b640 1 -- 172.21.15.76:0/1952263854 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629fd30eac0 con 0x5629fd4157b0 2024-03-12T06:02:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9e1226b640 1 --2- 172.21.15.76:0/1952263854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd4157b0 0x5629fd410e80 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f9e040cb730 tx=0x7f9e0409a280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28927f96b1a3d3a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9df77fe640 1 -- 172.21.15.76:0/1952263854 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0409a610 con 0x5629fd4157b0 2024-03-12T06:02:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9df77fe640 1 -- 172.21.15.76:0/1952263854 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e0409e760 con 0x5629fd4157b0 2024-03-12T06:02:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.051+0000 7f9df77fe640 1 -- 172.21.15.76:0/1952263854 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e040cb520 con 0x5629fd4157b0 2024-03-12T06:02:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.052+0000 7f9e16978740 1 -- 172.21.15.76:0/1952263854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd4157b0 msgr2=0x5629fd410e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.052+0000 7f9e16978740 1 --2- 172.21.15.76:0/1952263854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd4157b0 0x5629fd410e80 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f9e040cb730 tx=0x7f9e0409a280 comp rx=0 tx=0).stop 2024-03-12T06:02:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.052+0000 7f9e16978740 1 -- 172.21.15.76:0/1952263854 shutdown_connections 2024-03-12T06:02:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.052+0000 7f9e16978740 1 --2- 172.21.15.76:0/1952263854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd4157b0 0x5629fd410e80 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.052+0000 7f9e16978740 1 --2- 172.21.15.76:0/1952263854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40cbd0 0x5629fd415200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.052+0000 7f9e16978740 1 --2- 172.21.15.76:0/1952263854 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd40c2b0 0x5629fd414cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.052+0000 7f9e16978740 1 -- 172.21.15.76:0/1952263854 >> 172.21.15.76:0/1952263854 conn(0x5629fd406990 msgr2=0x5629fd40d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.052+0000 7f9e16978740 1 -- 172.21.15.76:0/1952263854 shutdown_connections 2024-03-12T06:02:08.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.052+0000 7f9e16978740 1 -- 172.21.15.76:0/1952263854 wait complete. 2024-03-12T06:02:08.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.052+0000 7f9e16978740 1 -- start start 2024-03-12T06:02:08.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.052+0000 7f9e16978740 1 Processor -- start 2024-03-12T06:02:08.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e16978740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd40c2b0 0x5629fd40f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e16978740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40cbd0 0x5629fd40f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e16978740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd4157b0 0x5629fd40fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e16978740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fd47bae0 con 0x5629fd4157b0 2024-03-12T06:02:08.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e16978740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fd47bc20 con 0x5629fd40c2b0 2024-03-12T06:02:08.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e16978740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629fd47bd60 con 0x5629fd40cbd0 2024-03-12T06:02:08.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e16978740 10 client.0 _ll_register_callbacks cb 0x7f9dfc02c860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:08.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e1226b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd4157b0 0x5629fd40fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e1226b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd4157b0 0x5629fd40fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49546/0 (socket says 172.21.15.76:49546) 2024-03-12T06:02:08.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e1226b640 1 -- 172.21.15.76:0/166016543 learned_addr learned my addr 172.21.15.76:0/166016543 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e11a6a640 1 --2- 172.21.15.76:0/166016543 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd40c2b0 0x5629fd40f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e11269640 1 --2- 172.21.15.76:0/166016543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40cbd0 0x5629fd40f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e1226b640 1 -- 172.21.15.76:0/166016543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40cbd0 msgr2=0x5629fd40f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e1226b640 1 --2- 172.21.15.76:0/166016543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40cbd0 0x5629fd40f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e1226b640 1 -- 172.21.15.76:0/166016543 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd40c2b0 msgr2=0x5629fd40f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e1226b640 1 --2- 172.21.15.76:0/166016543 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd40c2b0 0x5629fd40f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e1226b640 1 -- 172.21.15.76:0/166016543 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0c19e740 con 0x5629fd4157b0 2024-03-12T06:02:08.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9e1226b640 1 --2- 172.21.15.76:0/166016543 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd4157b0 0x5629fd40fd40 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f9e040cb1e0 tx=0x7f9e04005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b659f798bfcfc07 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9df57fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:08.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.053+0000 7f9df57fa640 1 -- 172.21.15.76:0/166016543 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0409a610 con 0x5629fd4157b0 2024-03-12T06:02:08.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.054+0000 7f9df57fa640 1 -- 172.21.15.76:0/166016543 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9e040a2db0 con 0x5629fd4157b0 2024-03-12T06:02:08.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.054+0000 7f9e16978740 1 -- 172.21.15.76:0/166016543 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9dfc004410 con 0x5629fd4157b0 2024-03-12T06:02:08.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.054+0000 7f9df57fa640 1 -- 172.21.15.76:0/166016543 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e04096c80 con 0x5629fd4157b0 2024-03-12T06:02:08.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.054+0000 7f9e16978740 10 client.16911 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:08.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.054+0000 7f9e16978740 1 -- 172.21.15.76:0/166016543 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5629fd3346a0 con 0x5629fd4157b0 2024-03-12T06:02:08.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.054+0000 7f9df57fa640 1 -- 172.21.15.76:0/166016543 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9dfc004410 con 0x5629fd4157b0 2024-03-12T06:02:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.054+0000 7f9df57fa640 1 -- 172.21.15.76:0/166016543 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9e04018030 con 0x5629fd4157b0 2024-03-12T06:02:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.054+0000 7f9e16978740 10 client.16911 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.054+0000 7f9e16978740 10 client.16911 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.054+0000 7f9e16978740 20 client.16911 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.054+0000 7f9e16978740 1 --2- 172.21.15.76:0/166016543 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5629fd47d150 0x5629fd47f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.054+0000 7f9e16978740 4 client.16911 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:02:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.054+0000 7f9e16978740 1 -- 172.21.15.76:0/166016543 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5629fd0ff890 con 0x5629fd47d150 2024-03-12T06:02:08.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.054+0000 7f9e11a6a640 1 --2- 172.21.15.76:0/166016543 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5629fd47d150 0x5629fd47f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9e11a6a640 1 --2- 172.21.15.76:0/166016543 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5629fd47d150 0x5629fd47f540 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1de5efc7ce2b0822 server_cookie=6c01ecb9b4b31127 in_seq=0 out_seq=0 2024-03-12T06:02:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9df57fa640 10 client.16911 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9df57fa640 1 -- 172.21.15.76:0/166016543 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5629fd0ff890 con 0x5629fd47d150 2024-03-12T06:02:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9df57fa640 10 client.16911 handle_command_reply: tid=0 2024-03-12T06:02:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9e16978740 1 client.16911 shutdown 2024-03-12T06:02:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9e16978740 1 -- 172.21.15.76:0/166016543 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd4157b0 msgr2=0x5629fd40fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9e16978740 1 --2- 172.21.15.76:0/166016543 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd4157b0 0x5629fd40fd40 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f9e040cb1e0 tx=0x7f9e04005ea0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9e16978740 1 -- 172.21.15.76:0/166016543 shutdown_connections 2024-03-12T06:02:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9e16978740 1 --2- 172.21.15.76:0/166016543 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5629fd47d150 0x5629fd47f540 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9e16978740 1 --2- 172.21.15.76:0/166016543 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5629fd4157b0 0x5629fd40fd40 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9df57fa640 0 client.16911 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9e16978740 1 --2- 172.21.15.76:0/166016543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629fd40cbd0 0x5629fd40f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9e16978740 1 --2- 172.21.15.76:0/166016543 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5629fd40c2b0 0x5629fd40f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9e16978740 1 -- 172.21.15.76:0/166016543 >> 172.21.15.76:0/166016543 conn(0x5629fd406990 msgr2=0x5629fd411e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9e16978740 1 -- 172.21.15.76:0/166016543 shutdown_connections 2024-03-12T06:02:08.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9e16978740 1 -- 172.21.15.76:0/166016543 wait complete. 2024-03-12T06:02:08.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.055+0000 7f9e16978740 20 client.16911 trim_cache size 0 max 16384 2024-03-12T06:02:08.073 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:02:08.073 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T06:02:08.073 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T06:02:08.073 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:02:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.056+0000 7f9e13cce640 1 -- 172.21.15.76:0/1211863476 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9df80a1ba0 msgr2=0x7f9df80a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.056+0000 7f9e13cce640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9df80a1ba0 0x7f9df80a4060 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc004c50 tx=0x7f9dfc005fd0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.056+0000 7f9e13cce640 1 -- 172.21.15.76:0/1211863476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e0c1060d0 msgr2=0x7f9e0c19dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.056+0000 7f9e13cce640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e0c1060d0 0x7f9e0c19dae0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400eaa0 tx=0x7f9e0400ef60 comp rx=0 tx=0).stop 2024-03-12T06:02:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.056+0000 7f9e13cce640 1 -- 172.21.15.76:0/1211863476 shutdown_connections 2024-03-12T06:02:08.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.056+0000 7f9e13cce640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9df80a1ba0 0x7f9df80a4060 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.056+0000 7f9e13cce640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e0c1060d0 0x7f9e0c19dae0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.056+0000 7f9e13cce640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e0c103c40 0x7f9e0c1a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.056+0000 7f9e13cce640 1 --2- 172.21.15.76:0/1211863476 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9e0c069e60 0x7f9e0c1a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.056+0000 7f9e13cce640 1 -- 172.21.15.76:0/1211863476 >> 172.21.15.76:0/1211863476 conn(0x7f9e0c077120 msgr2=0x7f9e0c101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.056+0000 7f9e13cce640 1 -- 172.21.15.76:0/1211863476 shutdown_connections 2024-03-12T06:02:08.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.056+0000 7f9e13cce640 1 -- 172.21.15.76:0/1211863476 wait complete. 2024-03-12T06:02:08.076 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T06:02:08.077 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T06:02:08.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 -- 172.21.15.76:0/2338846165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c8c106320 msgr2=0x7f2c8c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 --2- 172.21.15.76:0/2338846165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c8c106320 0x7f2c8c106700 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c009930 tx=0x7f2c7c02bc30 comp rx=0 tx=0).stop 2024-03-12T06:02:08.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 -- 172.21.15.76:0/2338846165 shutdown_connections 2024-03-12T06:02:08.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 --2- 172.21.15.76:0/2338846165 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c8c1014b0 0x7f2c8c10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 --2- 172.21.15.76:0/2338846165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c8c100af0 0x7f2c8c100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 --2- 172.21.15.76:0/2338846165 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c8c106320 0x7f2c8c106700 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 -- 172.21.15.76:0/2338846165 >> 172.21.15.76:0/2338846165 conn(0x7f2c8c075390 msgr2=0x7f2c8c0757a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 -- 172.21.15.76:0/2338846165 shutdown_connections 2024-03-12T06:02:08.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 -- 172.21.15.76:0/2338846165 wait complete. 2024-03-12T06:02:08.186 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 Processor -- start 2024-03-12T06:02:08.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 -- start start 2024-03-12T06:02:08.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c8c100af0 0x7f2c8c19d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c8c1014b0 0x7f2c8c19db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c8c106320 0x7f2c8c197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c197eb0 con 0x7f2c8c100af0 2024-03-12T06:02:08.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c198020 con 0x7f2c8c1014b0 2024-03-12T06:02:08.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c1981e0 con 0x7f2c8c106320 2024-03-12T06:02:08.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c8a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c8c100af0 0x7f2c8c19d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.187 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c8a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c8c100af0 0x7f2c8c19d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49568/0 (socket says 172.21.15.76:49568) 2024-03-12T06:02:08.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c8a59c640 1 -- 172.21.15.76:0/1723628104 learned_addr learned my addr 172.21.15.76:0/1723628104 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c83fff640 1 --2- 172.21.15.76:0/1723628104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c8c1014b0 0x7f2c8c19db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c8ad9d640 1 --2- 172.21.15.76:0/1723628104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c8c106320 0x7f2c8c197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c8a59c640 1 -- 172.21.15.76:0/1723628104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c8c106320 msgr2=0x7f2c8c197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c8a59c640 1 --2- 172.21.15.76:0/1723628104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c8c106320 0x7f2c8c197720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c8a59c640 1 -- 172.21.15.76:0/1723628104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c8c1014b0 msgr2=0x7f2c8c19db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c8a59c640 1 --2- 172.21.15.76:0/1723628104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c8c1014b0 0x7f2c8c19db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c8a59c640 1 -- 172.21.15.76:0/1723628104 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c009560 con 0x7f2c8c100af0 2024-03-12T06:02:08.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c8a59c640 1 --2- 172.21.15.76:0/1723628104 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c8c100af0 0x7f2c8c19d610 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c009230 tx=0x7f2c7c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c837fe640 1 -- 172.21.15.76:0/1723628104 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c004440 con 0x7f2c8c100af0 2024-03-12T06:02:08.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c837fe640 1 -- 172.21.15.76:0/1723628104 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c7c0052d0 con 0x7f2c8c100af0 2024-03-12T06:02:08.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 -- 172.21.15.76:0/1723628104 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8c198460 con 0x7f2c8c100af0 2024-03-12T06:02:08.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c837fe640 1 -- 172.21.15.76:0/1723628104 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c003d70 con 0x7f2c8c100af0 2024-03-12T06:02:08.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.179+0000 7f2c90ce1640 1 -- 172.21.15.76:0/1723628104 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8c1a4590 con 0x7f2c8c100af0 2024-03-12T06:02:08.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.181+0000 7f2c90ce1640 1 -- 172.21.15.76:0/1723628104 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c8c0694d0 con 0x7f2c8c100af0 2024-03-12T06:02:08.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.182+0000 7f2c837fe640 1 -- 172.21.15.76:0/1723628104 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f2c7c005440 con 0x7f2c8c100af0 2024-03-12T06:02:08.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.183+0000 7f2c837fe640 1 --2- 172.21.15.76:0/1723628104 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f2c780a1ba0 0x7f2c780a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.183+0000 7f2c83fff640 1 --2- 172.21.15.76:0/1723628104 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f2c780a1ba0 0x7f2c780a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.183+0000 7f2c837fe640 1 -- 172.21.15.76:0/1723628104 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f2c7c0ef960 con 0x7f2c8c100af0 2024-03-12T06:02:08.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.183+0000 7f2c83fff640 1 --2- 172.21.15.76:0/1723628104 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f2c780a1ba0 0x7f2c780a4060 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c198f20 tx=0x7f2c7400b390 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.186+0000 7f2c837fe640 1 -- 172.21.15.76:0/1723628104 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2c7c0b8920 con 0x7f2c8c100af0 2024-03-12T06:02:08.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:08 smithi076 ceph-mon[32095]: pgmap v193: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:02:08.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.404+0000 7f2c90ce1640 1 -- 172.21.15.76:0/1723628104 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c8c198650 con 0x7f2c8c100af0 2024-03-12T06:02:08.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.404+0000 7f2c837fe640 1 -- 172.21.15.76:0/1723628104 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11328 (secure 0 0 0) 0x7f2c7c0bbfe0 con 0x7f2c8c100af0 2024-03-12T06:02:08.409 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:02:08.409 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T06:01:33.842001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T06:02:08.410 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 28 2024-03-12T06:02:08.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.406+0000 7f2c90ce1640 1 -- 172.21.15.76:0/1723628104 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f2c780a1ba0 msgr2=0x7f2c780a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.406+0000 7f2c90ce1640 1 --2- 172.21.15.76:0/1723628104 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f2c780a1ba0 0x7f2c780a4060 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c198f20 tx=0x7f2c7400b390 comp rx=0 tx=0).stop 2024-03-12T06:02:08.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.406+0000 7f2c90ce1640 1 -- 172.21.15.76:0/1723628104 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c8c100af0 msgr2=0x7f2c8c19d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.406+0000 7f2c90ce1640 1 --2- 172.21.15.76:0/1723628104 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c8c100af0 0x7f2c8c19d610 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c009230 tx=0x7f2c7c004040 comp rx=0 tx=0).stop 2024-03-12T06:02:08.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.406+0000 7f2c90ce1640 1 -- 172.21.15.76:0/1723628104 shutdown_connections 2024-03-12T06:02:08.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.406+0000 7f2c90ce1640 1 --2- 172.21.15.76:0/1723628104 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f2c780a1ba0 0x7f2c780a4060 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.406+0000 7f2c90ce1640 1 --2- 172.21.15.76:0/1723628104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c8c106320 0x7f2c8c197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.406+0000 7f2c90ce1640 1 --2- 172.21.15.76:0/1723628104 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2c8c1014b0 0x7f2c8c19db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.406+0000 7f2c90ce1640 1 --2- 172.21.15.76:0/1723628104 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2c8c100af0 0x7f2c8c19d610 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.406+0000 7f2c90ce1640 1 -- 172.21.15.76:0/1723628104 >> 172.21.15.76:0/1723628104 conn(0x7f2c8c075390 msgr2=0x7f2c8c0fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.406+0000 7f2c90ce1640 1 -- 172.21.15.76:0/1723628104 shutdown_connections 2024-03-12T06:02:08.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.406+0000 7f2c90ce1640 1 -- 172.21.15.76:0/1723628104 wait complete. 2024-03-12T06:02:08.422 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:02:08.422 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T06:02:08.445 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:08 smithi082 ceph-mon[36382]: pgmap v193: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:02:08.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:08 smithi088 ceph-mon[36585]: pgmap v193: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:02:08.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 -- 172.21.15.76:0/2872758260 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8760100c40 msgr2=0x7f8760104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 --2- 172.21.15.76:0/2872758260 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8760100c40 0x7f8760104c80 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f874c009930 tx=0x7f874c02bc30 comp rx=0 tx=0).stop 2024-03-12T06:02:08.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 -- 172.21.15.76:0/2872758260 shutdown_connections 2024-03-12T06:02:08.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 --2- 172.21.15.76:0/2872758260 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8760105360 0x7f876010cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 --2- 172.21.15.76:0/2872758260 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8760100c40 0x7f8760104c80 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 --2- 172.21.15.76:0/2872758260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8760100290 0x7f8760100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 -- 172.21.15.76:0/2872758260 >> 172.21.15.76:0/2872758260 conn(0x7f8760077230 msgr2=0x7f8760077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 -- 172.21.15.76:0/2872758260 shutdown_connections 2024-03-12T06:02:08.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 -- 172.21.15.76:0/2872758260 wait complete. 2024-03-12T06:02:08.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 Processor -- start 2024-03-12T06:02:08.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 -- start start 2024-03-12T06:02:08.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8760100290 0x7f87601986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8760105360 0x7f8760198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87601991b0 0x7f87601a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87601a3e30 con 0x7f87601991b0 2024-03-12T06:02:08.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87601a3fa0 con 0x7f8760100290 2024-03-12T06:02:08.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.522+0000 7f876702e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87601a4160 con 0x7f8760105360 2024-03-12T06:02:08.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f87655cb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87601991b0 0x7f87601a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f87655cb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87601991b0 0x7f87601a3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49588/0 (socket says 172.21.15.76:49588) 2024-03-12T06:02:08.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f87655cb640 1 -- 172.21.15.76:0/2939748507 learned_addr learned my addr 172.21.15.76:0/2939748507 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f8764dca640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8760100290 0x7f87601986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f875ffff640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8760105360 0x7f8760198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f87655cb640 1 -- 172.21.15.76:0/2939748507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8760105360 msgr2=0x7f8760198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f87655cb640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8760105360 0x7f8760198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f87655cb640 1 -- 172.21.15.76:0/2939748507 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8760100290 msgr2=0x7f87601986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f87655cb640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8760100290 0x7f87601986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f87655cb640 1 -- 172.21.15.76:0/2939748507 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874c009560 con 0x7f87601991b0 2024-03-12T06:02:08.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f8764dca640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8760100290 0x7f87601986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:02:08.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f875ffff640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8760105360 0x7f8760198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:08.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f87655cb640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87601991b0 0x7f87601a3760 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f875800cab0 tx=0x7f875800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f875dffb640 1 -- 172.21.15.76:0/2939748507 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875800adc0 con 0x7f87601991b0 2024-03-12T06:02:08.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f875dffb640 1 -- 172.21.15.76:0/2939748507 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f875800f930 con 0x7f87601991b0 2024-03-12T06:02:08.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f876702e640 1 -- 172.21.15.76:0/2939748507 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87601a4440 con 0x7f87601991b0 2024-03-12T06:02:08.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f875dffb640 1 -- 172.21.15.76:0/2939748507 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87580189a0 con 0x7f87601991b0 2024-03-12T06:02:08.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.523+0000 7f876702e640 1 -- 172.21.15.76:0/2939748507 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87601a4900 con 0x7f87601991b0 2024-03-12T06:02:08.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.525+0000 7f875dffb640 1 -- 172.21.15.76:0/2939748507 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f875800faa0 con 0x7f87601991b0 2024-03-12T06:02:08.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8769c03740 1 Processor -- start 2024-03-12T06:02:08.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f875dffb640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f87500a1ba0 0x7f87500a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8769c03740 1 -- start start 2024-03-12T06:02:08.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f875ffff640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f87500a1ba0 0x7f87500a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8769c03740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b6311250 0x5642b6311630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f875dffb640 1 -- 172.21.15.76:0/2939748507 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f87580cdc60 con 0x7f87601991b0 2024-03-12T06:02:08.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8769c03740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311b70 0x5642b631a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8769c03740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b631a750 0x5642b631cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8769c03740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642b62cc960 con 0x5642b6311250 2024-03-12T06:02:08.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8769c03740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642b6280310 con 0x5642b631a750 2024-03-12T06:02:08.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8769c03740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642b627d840 con 0x5642b6311b70 2024-03-12T06:02:08.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8764dca640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b6311250 0x5642b6311630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8764dca640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b6311250 0x5642b6311630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49596/0 (socket says 172.21.15.76:49596) 2024-03-12T06:02:08.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8764dca640 1 -- 172.21.15.76:0/3762099428 learned_addr learned my addr 172.21.15.76:0/3762099428 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f875ffff640 1 --2- 172.21.15.76:0/3762099428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311b70 0x5642b631a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8764dca640 1 -- 172.21.15.76:0/3762099428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311b70 msgr2=0x5642b631a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8764dca640 1 --2- 172.21.15.76:0/3762099428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311b70 0x5642b631a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f87655cb640 1 --2- 172.21.15.76:0/3762099428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b631a750 0x5642b631cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8764dca640 1 -- 172.21.15.76:0/3762099428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b631a750 msgr2=0x5642b631cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f875ffff640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f87500a1ba0 0x7f87500a4060 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f874c006010 tx=0x7f874c004840 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8764dca640 1 --2- 172.21.15.76:0/3762099428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b631a750 0x5642b631cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.526+0000 7f8764dca640 1 -- 172.21.15.76:0/3762099428 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5642b623b060 con 0x5642b6311250 2024-03-12T06:02:08.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.527+0000 7f8764dca640 1 --2- 172.21.15.76:0/3762099428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b6311250 0x5642b6311630 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f8754007850 tx=0x7f8754007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c709924f71d696 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.527+0000 7f874affd640 1 -- 172.21.15.76:0/3762099428 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875400d950 con 0x5642b6311250 2024-03-12T06:02:08.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.527+0000 7f874affd640 1 -- 172.21.15.76:0/3762099428 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87540074c0 con 0x5642b6311250 2024-03-12T06:02:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.527+0000 7f874affd640 1 -- 172.21.15.76:0/3762099428 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87540123e0 con 0x5642b6311250 2024-03-12T06:02:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.527+0000 7f8769c03740 1 -- 172.21.15.76:0/3762099428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b6311250 msgr2=0x5642b6311630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.527+0000 7f8769c03740 1 --2- 172.21.15.76:0/3762099428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b6311250 0x5642b6311630 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f8754007850 tx=0x7f8754007d10 comp rx=0 tx=0).stop 2024-03-12T06:02:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.527+0000 7f8769c03740 1 -- 172.21.15.76:0/3762099428 shutdown_connections 2024-03-12T06:02:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.527+0000 7f8769c03740 1 --2- 172.21.15.76:0/3762099428 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b631a750 0x5642b631cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.527+0000 7f8769c03740 1 --2- 172.21.15.76:0/3762099428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311b70 0x5642b631a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.527+0000 7f8769c03740 1 --2- 172.21.15.76:0/3762099428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b6311250 0x5642b6311630 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.527+0000 7f8769c03740 1 -- 172.21.15.76:0/3762099428 >> 172.21.15.76:0/3762099428 conn(0x5642b630b980 msgr2=0x5642b630bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.527+0000 7f8769c03740 1 -- 172.21.15.76:0/3762099428 shutdown_connections 2024-03-12T06:02:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.527+0000 7f8769c03740 1 -- 172.21.15.76:0/3762099428 wait complete. 2024-03-12T06:02:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.528+0000 7f8769c03740 1 -- start start 2024-03-12T06:02:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.528+0000 7f8769c03740 1 Processor -- start 2024-03-12T06:02:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.528+0000 7f8769c03740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311250 0x5642b6317730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.528+0000 7f8769c03740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b6311b70 0x5642b6317c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.528+0000 7f8769c03740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b631a750 0x5642b63181b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.528+0000 7f8769c03740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642b6279020 con 0x5642b631a750 2024-03-12T06:02:08.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.528+0000 7f8769c03740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642b5f4c430 con 0x5642b6311b70 2024-03-12T06:02:08.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.528+0000 7f8769c03740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642b5f08e00 con 0x5642b6311250 2024-03-12T06:02:08.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.528+0000 7f8769c03740 10 client.0 _ll_register_callbacks cb 0x5642b6220cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:08.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.528+0000 7f87655cb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b631a750 0x5642b63181b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.528+0000 7f87655cb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b631a750 0x5642b63181b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49608/0 (socket says 172.21.15.76:49608) 2024-03-12T06:02:08.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.528+0000 7f87655cb640 1 -- 172.21.15.76:0/336884878 learned_addr learned my addr 172.21.15.76:0/336884878 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.528+0000 7f8764dca640 1 --2- 172.21.15.76:0/336884878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311250 0x5642b6317730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.528+0000 7f875ffff640 1 --2- 172.21.15.76:0/336884878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b6311b70 0x5642b6317c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f87655cb640 1 -- 172.21.15.76:0/336884878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311250 msgr2=0x5642b6317730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f87655cb640 1 --2- 172.21.15.76:0/336884878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311250 0x5642b6317730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f87655cb640 1 -- 172.21.15.76:0/336884878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b6311b70 msgr2=0x5642b6317c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f87655cb640 1 --2- 172.21.15.76:0/336884878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b6311b70 0x5642b6317c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f87655cb640 1 -- 172.21.15.76:0/336884878 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f875400e040 con 0x5642b631a750 2024-03-12T06:02:08.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f87655cb640 1 --2- 172.21.15.76:0/336884878 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b631a750 0x5642b63181b0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f8758096cd0 tx=0x7f875809af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ebdfc13faf098c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f873ffff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:08.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f873ffff640 1 -- 172.21.15.76:0/336884878 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87580a3010 con 0x5642b631a750 2024-03-12T06:02:08.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f873ffff640 1 -- 172.21.15.76:0/336884878 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f875800fe10 con 0x5642b631a750 2024-03-12T06:02:08.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f8769c03740 1 -- 172.21.15.76:0/336884878 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5642b5fb5720 con 0x5642b631a750 2024-03-12T06:02:08.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f873ffff640 1 -- 172.21.15.76:0/336884878 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87580b3200 con 0x5642b631a750 2024-03-12T06:02:08.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f8769c03740 10 client.16929 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:08.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f8769c03740 1 -- 172.21.15.76:0/336884878 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5642b62776a0 con 0x5642b631a750 2024-03-12T06:02:08.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f873ffff640 1 -- 172.21.15.76:0/336884878 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5642b5fb5720 con 0x5642b631a750 2024-03-12T06:02:08.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f873ffff640 1 -- 172.21.15.76:0/336884878 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8758016020 con 0x5642b631a750 2024-03-12T06:02:08.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f8769c03740 10 client.16929 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:08.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f8769c03740 10 client.16929 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:02:08.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f8769c03740 20 client.16929 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:08.550 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f8769c03740 1 --2- 172.21.15.76:0/336884878 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5642b632a9a0 0x5642b632cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f8769c03740 4 client.16929 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:02:08.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.529+0000 7f8769c03740 1 -- 172.21.15.76:0/336884878 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5642b6004890 con 0x5642b632a9a0 2024-03-12T06:02:08.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.530+0000 7f8764dca640 1 --2- 172.21.15.76:0/336884878 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5642b632a9a0 0x5642b632cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.551 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.530+0000 7f873ffff640 10 client.16929 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:02:08.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.530+0000 7f8764dca640 1 --2- 172.21.15.76:0/336884878 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5642b632a9a0 0x5642b632cd90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=72ba4323fa349255 server_cookie=b79543c7f2d7b950 in_seq=0 out_seq=0 2024-03-12T06:02:08.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f873ffff640 1 -- 172.21.15.76:0/336884878 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5642b6004890 con 0x5642b632a9a0 2024-03-12T06:02:08.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f873ffff640 10 client.16929 handle_command_reply: tid=0 2024-03-12T06:02:08.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f8769c03740 1 client.16929 shutdown 2024-03-12T06:02:08.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f8769c03740 1 -- 172.21.15.76:0/336884878 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b631a750 msgr2=0x5642b63181b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.552 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f8769c03740 1 --2- 172.21.15.76:0/336884878 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b631a750 0x5642b63181b0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f8758096cd0 tx=0x7f875809af70 comp rx=0 tx=0).stop 2024-03-12T06:02:08.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f8769c03740 1 -- 172.21.15.76:0/336884878 shutdown_connections 2024-03-12T06:02:08.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f8769c03740 1 --2- 172.21.15.76:0/336884878 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5642b632a9a0 0x5642b632cd90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f8769c03740 1 --2- 172.21.15.76:0/336884878 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b631a750 0x5642b63181b0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f873ffff640 0 client.16929 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:02:08.553 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f8769c03740 1 --2- 172.21.15.76:0/336884878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b6311b70 0x5642b6317c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f8769c03740 1 --2- 172.21.15.76:0/336884878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311250 0x5642b6317730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f8769c03740 1 -- 172.21.15.76:0/336884878 >> 172.21.15.76:0/336884878 conn(0x5642b630b980 msgr2=0x5642b630d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f8769c03740 1 -- 172.21.15.76:0/336884878 shutdown_connections 2024-03-12T06:02:08.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f8769c03740 1 -- 172.21.15.76:0/336884878 wait complete. 2024-03-12T06:02:08.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.531+0000 7f8769c03740 20 client.16929 trim_cache size 0 max 16384 2024-03-12T06:02:08.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.546+0000 7f8769c03740 1 Processor -- start 2024-03-12T06:02:08.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.546+0000 7f8769c03740 1 -- start start 2024-03-12T06:02:08.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.546+0000 7f8769c03740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311250 0x5642b6319e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.546+0000 7f8769c03740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b6311b70 0x5642b6317fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.546+0000 7f8769c03740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b631a750 0x5642b63149b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.546+0000 7f8769c03740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642b61fe130 con 0x5642b631a750 2024-03-12T06:02:08.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.546+0000 7f8769c03740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642b6327610 con 0x5642b6311b70 2024-03-12T06:02:08.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.546+0000 7f8769c03740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642b630cbe0 con 0x5642b6311250 2024-03-12T06:02:08.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.546+0000 7f87655cb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b631a750 0x5642b63149b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.546+0000 7f87655cb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b631a750 0x5642b63149b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49624/0 (socket says 172.21.15.76:49624) 2024-03-12T06:02:08.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.546+0000 7f87655cb640 1 -- 172.21.15.76:0/4258654493 learned_addr learned my addr 172.21.15.76:0/4258654493 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.546+0000 7f875ffff640 1 --2- 172.21.15.76:0/4258654493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b6311b70 0x5642b6317fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.547+0000 7f8764dca640 1 --2- 172.21.15.76:0/4258654493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311250 0x5642b6319e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.547+0000 7f87655cb640 1 -- 172.21.15.76:0/4258654493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311250 msgr2=0x5642b6319e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.547+0000 7f87655cb640 1 --2- 172.21.15.76:0/4258654493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311250 0x5642b6319e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.547+0000 7f87655cb640 1 -- 172.21.15.76:0/4258654493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b6311b70 msgr2=0x5642b6317fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.547+0000 7f87655cb640 1 --2- 172.21.15.76:0/4258654493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b6311b70 0x5642b6317fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.547+0000 7f87655cb640 1 -- 172.21.15.76:0/4258654493 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5642b6213c20 con 0x5642b631a750 2024-03-12T06:02:08.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.547+0000 7f875ffff640 1 --2- 172.21.15.76:0/4258654493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b6311b70 0x5642b6317fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:02:08.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.547+0000 7f87655cb640 1 --2- 172.21.15.76:0/4258654493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b631a750 0x5642b63149b0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f875800a6e0 tx=0x7f8758098d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff2bb4e1205d037 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.547+0000 7f874a7fc640 1 -- 172.21.15.76:0/4258654493 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875809f8e0 con 0x5642b631a750 2024-03-12T06:02:08.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.547+0000 7f874a7fc640 1 -- 172.21.15.76:0/4258654493 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87580a3010 con 0x5642b631a750 2024-03-12T06:02:08.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.547+0000 7f874a7fc640 1 -- 172.21.15.76:0/4258654493 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87580a2b80 con 0x5642b631a750 2024-03-12T06:02:08.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.548+0000 7f8769c03740 1 -- 172.21.15.76:0/4258654493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b631a750 msgr2=0x5642b63149b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.548+0000 7f8769c03740 1 --2- 172.21.15.76:0/4258654493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b631a750 0x5642b63149b0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f875800a6e0 tx=0x7f8758098d80 comp rx=0 tx=0).stop 2024-03-12T06:02:08.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.548+0000 7f8769c03740 1 -- 172.21.15.76:0/4258654493 shutdown_connections 2024-03-12T06:02:08.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.548+0000 7f8769c03740 1 --2- 172.21.15.76:0/4258654493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b631a750 0x5642b63149b0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.548+0000 7f8769c03740 1 --2- 172.21.15.76:0/4258654493 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b6311b70 0x5642b6317fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.548+0000 7f8769c03740 1 --2- 172.21.15.76:0/4258654493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311250 0x5642b6319e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.548+0000 7f8769c03740 1 -- 172.21.15.76:0/4258654493 >> 172.21.15.76:0/4258654493 conn(0x5642b630b980 msgr2=0x5642b6312a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.548+0000 7f8769c03740 1 -- 172.21.15.76:0/4258654493 shutdown_connections 2024-03-12T06:02:08.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.548+0000 7f8769c03740 1 -- 172.21.15.76:0/4258654493 wait complete. 2024-03-12T06:02:08.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.548+0000 7f8769c03740 1 -- start start 2024-03-12T06:02:08.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.548+0000 7f8769c03740 1 Processor -- start 2024-03-12T06:02:08.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.548+0000 7f8769c03740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b6311250 0x5642b63217d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.548+0000 7f8769c03740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311b70 0x5642b631fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.548+0000 7f8769c03740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b631a750 0x5642b63200e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8769c03740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642b6320650 con 0x5642b6311250 2024-03-12T06:02:08.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8769c03740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642b63207c0 con 0x5642b631a750 2024-03-12T06:02:08.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8769c03740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642b6320980 con 0x5642b6311b70 2024-03-12T06:02:08.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8769c03740 10 client.0 _ll_register_callbacks cb 0x7f87540072a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:08.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8764dca640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b6311250 0x5642b63217d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8764dca640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b6311250 0x5642b63217d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49630/0 (socket says 172.21.15.76:49630) 2024-03-12T06:02:08.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8764dca640 1 -- 172.21.15.76:0/452916599 learned_addr learned my addr 172.21.15.76:0/452916599 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f875ffff640 1 --2- 172.21.15.76:0/452916599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311b70 0x5642b631fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8764dca640 1 -- 172.21.15.76:0/452916599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311b70 msgr2=0x5642b631fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8764dca640 1 --2- 172.21.15.76:0/452916599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311b70 0x5642b631fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8764dca640 1 -- 172.21.15.76:0/452916599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b631a750 msgr2=0x5642b63200e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8764dca640 1 --2- 172.21.15.76:0/452916599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b631a750 0x5642b63200e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8764dca640 1 -- 172.21.15.76:0/452916599 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5642b62776a0 con 0x5642b6311250 2024-03-12T06:02:08.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f875ffff640 1 --2- 172.21.15.76:0/452916599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311b70 0x5642b631fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:02:08.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8764dca640 1 --2- 172.21.15.76:0/452916599 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b6311250 0x5642b63217d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8754013d70 tx=0x7f8754006ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af7de9fd2924db55 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f873ffff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:08.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f873ffff640 1 -- 172.21.15.76:0/452916599 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875400d560 con 0x5642b6311250 2024-03-12T06:02:08.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f873ffff640 1 -- 172.21.15.76:0/452916599 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f8754023040 con 0x5642b6311250 2024-03-12T06:02:08.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8769c03740 1 -- 172.21.15.76:0/452916599 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5642b5f787e0 con 0x5642b6311250 2024-03-12T06:02:08.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f873ffff640 1 -- 172.21.15.76:0/452916599 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8754026df0 con 0x5642b6311250 2024-03-12T06:02:08.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8769c03740 10 client.16935 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:08.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.549+0000 7f8769c03740 1 -- 172.21.15.76:0/452916599 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5642b62398c0 con 0x5642b6311250 2024-03-12T06:02:08.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.550+0000 7f873ffff640 1 -- 172.21.15.76:0/452916599 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5642b5f787e0 con 0x5642b6311250 2024-03-12T06:02:08.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.550+0000 7f873ffff640 1 -- 172.21.15.76:0/452916599 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f87540264b0 con 0x5642b6311250 2024-03-12T06:02:08.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.550+0000 7f8769c03740 10 client.16935 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:08.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.550+0000 7f8769c03740 10 client.16935 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:02:08.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.550+0000 7f8769c03740 20 client.16935 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:08.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.550+0000 7f8769c03740 1 --2- 172.21.15.76:0/452916599 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5642b6382400 0x5642b6384820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.571 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.550+0000 7f8769c03740 4 client.16935 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:02:08.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.550+0000 7f8769c03740 1 -- 172.21.15.76:0/452916599 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5642b5f24700 con 0x5642b6382400 2024-03-12T06:02:08.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.550+0000 7f875ffff640 1 --2- 172.21.15.76:0/452916599 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5642b6382400 0x5642b6384820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.550+0000 7f873ffff640 10 client.16935 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:02:08.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.550+0000 7f875ffff640 1 --2- 172.21.15.76:0/452916599 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5642b6382400 0x5642b6384820 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d9a163cefebd368f server_cookie=e89eea0d8fb85126 in_seq=0 out_seq=0 2024-03-12T06:02:08.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.550+0000 7f873ffff640 1 -- 172.21.15.76:0/452916599 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5642b5f24700 con 0x5642b6382400 2024-03-12T06:02:08.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.550+0000 7f873ffff640 10 client.16935 handle_command_reply: tid=0 2024-03-12T06:02:08.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.550+0000 7f8769c03740 1 client.16935 shutdown 2024-03-12T06:02:08.572 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f8769c03740 1 -- 172.21.15.76:0/452916599 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b6311250 msgr2=0x5642b63217d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f8769c03740 1 --2- 172.21.15.76:0/452916599 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b6311250 0x5642b63217d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8754013d70 tx=0x7f8754006ba0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f8769c03740 1 -- 172.21.15.76:0/452916599 shutdown_connections 2024-03-12T06:02:08.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f8769c03740 1 --2- 172.21.15.76:0/452916599 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x5642b6382400 0x5642b6384820 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f8769c03740 1 --2- 172.21.15.76:0/452916599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5642b631a750 0x5642b63200e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f873ffff640 0 client.16935 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:02:08.573 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f8769c03740 1 --2- 172.21.15.76:0/452916599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5642b6311b70 0x5642b631fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f8769c03740 1 --2- 172.21.15.76:0/452916599 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5642b6311250 0x5642b63217d0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f8769c03740 1 -- 172.21.15.76:0/452916599 >> 172.21.15.76:0/452916599 conn(0x5642b630b980 msgr2=0x5642b630a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f8769c03740 1 -- 172.21.15.76:0/452916599 shutdown_connections 2024-03-12T06:02:08.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f8769c03740 1 -- 172.21.15.76:0/452916599 wait complete. 2024-03-12T06:02:08.574 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f8769c03740 20 client.16935 trim_cache size 0 max 16384 2024-03-12T06:02:08.575 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:02:08.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f876702e640 1 -- 172.21.15.76:0/2939748507 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f87500a1ba0 msgr2=0x7f87500a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f876702e640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f87500a1ba0 0x7f87500a4060 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f874c006010 tx=0x7f874c004840 comp rx=0 tx=0).stop 2024-03-12T06:02:08.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f876702e640 1 -- 172.21.15.76:0/2939748507 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87601991b0 msgr2=0x7f87601a3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f876702e640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87601991b0 0x7f87601a3760 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f875800cab0 tx=0x7f875800cf70 comp rx=0 tx=0).stop 2024-03-12T06:02:08.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f876702e640 1 -- 172.21.15.76:0/2939748507 shutdown_connections 2024-03-12T06:02:08.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f876702e640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f87500a1ba0 0x7f87500a4060 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f876702e640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87601991b0 0x7f87601a3760 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f876702e640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8760105360 0x7f8760198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f876702e640 1 --2- 172.21.15.76:0/2939748507 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8760100290 0x7f87601986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f876702e640 1 -- 172.21.15.76:0/2939748507 >> 172.21.15.76:0/2939748507 conn(0x7f8760077230 msgr2=0x7f87600fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.551+0000 7f876702e640 1 -- 172.21.15.76:0/2939748507 shutdown_connections 2024-03-12T06:02:08.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.552+0000 7f876702e640 1 -- 172.21.15.76:0/2939748507 wait complete. 2024-03-12T06:02:08.579 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:02:08.579 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T06:02:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.679+0000 7f3670d4f640 1 -- 172.21.15.76:0/1421652551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366c100290 msgr2=0x7f366c100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.679+0000 7f3670d4f640 1 --2- 172.21.15.76:0/1421652551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366c100290 0x7f366c100670 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f365c009930 tx=0x7f365c02bc30 comp rx=0 tx=0).stop 2024-03-12T06:02:08.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.679+0000 7f3670d4f640 1 -- 172.21.15.76:0/1421652551 shutdown_connections 2024-03-12T06:02:08.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.679+0000 7f3670d4f640 1 --2- 172.21.15.76:0/1421652551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f366c105360 0x7f366c10cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.679+0000 7f3670d4f640 1 --2- 172.21.15.76:0/1421652551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f366c100c40 0x7f366c104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.679+0000 7f3670d4f640 1 --2- 172.21.15.76:0/1421652551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366c100290 0x7f366c100670 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.679+0000 7f3670d4f640 1 -- 172.21.15.76:0/1421652551 >> 172.21.15.76:0/1421652551 conn(0x7f366c077200 msgr2=0x7f366c077610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.679+0000 7f3670d4f640 1 -- 172.21.15.76:0/1421652551 shutdown_connections 2024-03-12T06:02:08.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.679+0000 7f3670d4f640 1 -- 172.21.15.76:0/1421652551 wait complete. 2024-03-12T06:02:08.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.680+0000 7f3670d4f640 1 Processor -- start 2024-03-12T06:02:08.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.680+0000 7f3670d4f640 1 -- start start 2024-03-12T06:02:08.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.680+0000 7f3670d4f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f366c100290 0x7f366c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.680+0000 7f3670d4f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f366c105360 0x7f366c198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.680+0000 7f3670d4f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366c199070 0x7f366c1a3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.680+0000 7f3670d4f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c1a3cc0 con 0x7f366c199070 2024-03-12T06:02:08.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.680+0000 7f3670d4f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c1a3e30 con 0x7f366c105360 2024-03-12T06:02:08.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.680+0000 7f3670d4f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366c1a3ff0 con 0x7f366c100290 2024-03-12T06:02:08.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.680+0000 7f366ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366c199070 0x7f366c1a3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.680+0000 7f366ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366c199070 0x7f366c1a3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49646/0 (socket says 172.21.15.76:49646) 2024-03-12T06:02:08.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.680+0000 7f366ad9d640 1 -- 172.21.15.76:0/3926804081 learned_addr learned my addr 172.21.15.76:0/3926804081 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.680+0000 7f3663fff640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f366c105360 0x7f366c198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.680+0000 7f366a59c640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f366c100290 0x7f366c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.680+0000 7f366ad9d640 1 -- 172.21.15.76:0/3926804081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f366c100290 msgr2=0x7f366c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.681+0000 7f366ad9d640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f366c100290 0x7f366c1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.681+0000 7f366ad9d640 1 -- 172.21.15.76:0/3926804081 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f366c105360 msgr2=0x7f366c198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.681+0000 7f366ad9d640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f366c105360 0x7f366c198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.681+0000 7f366ad9d640 1 -- 172.21.15.76:0/3926804081 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c009560 con 0x7f366c199070 2024-03-12T06:02:08.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.681+0000 7f3663fff640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f366c105360 0x7f366c198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:08.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.681+0000 7f366a59c640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f366c100290 0x7f366c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:08.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.681+0000 7f366ad9d640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366c199070 0x7f366c1a3620 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f366400da40 tx=0x7f366400df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.681+0000 7f36637fe640 1 -- 172.21.15.76:0/3926804081 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366400bdd0 con 0x7f366c199070 2024-03-12T06:02:08.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.681+0000 7f36637fe640 1 -- 172.21.15.76:0/3926804081 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3664009890 con 0x7f366c199070 2024-03-12T06:02:08.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.681+0000 7f36637fe640 1 -- 172.21.15.76:0/3926804081 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640106c0 con 0x7f366c199070 2024-03-12T06:02:08.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.681+0000 7f3670d4f640 1 -- 172.21.15.76:0/3926804081 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366c1a42d0 con 0x7f366c199070 2024-03-12T06:02:08.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.681+0000 7f3670d4f640 1 -- 172.21.15.76:0/3926804081 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c1a4790 con 0x7f366c199070 2024-03-12T06:02:08.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.683+0000 7f36637fe640 1 -- 172.21.15.76:0/3926804081 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f36640041b0 con 0x7f366c199070 2024-03-12T06:02:08.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.683+0000 7f3673a00740 1 Processor -- start 2024-03-12T06:02:08.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f3673a00740 1 -- start start 2024-03-12T06:02:08.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f36637fe640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f36580a1ba0 0x7f36580a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f3673a00740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232250 0x55e66d232630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f3673a00740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d232b70 0x55e66d23b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f3673a00740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d23b750 0x55e66d23db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f3673a00740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d1ed960 con 0x55e66d23b750 2024-03-12T06:02:08.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f3673a00740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d1a1310 con 0x55e66d232250 2024-03-12T06:02:08.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f36637fe640 1 -- 172.21.15.76:0/3926804081 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f36640cd530 con 0x7f366c199070 2024-03-12T06:02:08.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f3673a00740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d19e840 con 0x55e66d232b70 2024-03-12T06:02:08.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f3663fff640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f36580a1ba0 0x7f36580a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f366ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d23b750 0x55e66d23db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f366a59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232250 0x55e66d232630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f366ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d23b750 0x55e66d23db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49652/0 (socket says 172.21.15.76:49652) 2024-03-12T06:02:08.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f366ad9d640 1 -- 172.21.15.76:0/3209518111 learned_addr learned my addr 172.21.15.76:0/3209518111 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f3663fff640 1 --2- 172.21.15.76:0/3209518111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d232b70 0x55e66d23b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f366ad9d640 1 -- 172.21.15.76:0/3209518111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d232b70 msgr2=0x55e66d23b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f366ad9d640 1 --2- 172.21.15.76:0/3209518111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d232b70 0x55e66d23b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f366ad9d640 1 -- 172.21.15.76:0/3209518111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232250 msgr2=0x55e66d232630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f366ad9d640 1 --2- 172.21.15.76:0/3209518111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232250 0x55e66d232630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f366ad9d640 1 -- 172.21.15.76:0/3209518111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e66d15c060 con 0x55e66d23b750 2024-03-12T06:02:08.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f3663fff640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f36580a1ba0 0x7f36580a4060 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f366c06a3d0 tx=0x7f36540117b0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.684+0000 7f366ad9d640 1 --2- 172.21.15.76:0/3209518111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d23b750 0x55e66d23db40 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f36640a23b0 tx=0x7f3664099d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f717e47ba6d74f70 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.685+0000 7f3660ff9640 1 -- 172.21.15.76:0/3209518111 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366409e290 con 0x55e66d23b750 2024-03-12T06:02:08.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.685+0000 7f3660ff9640 1 -- 172.21.15.76:0/3209518111 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36640a28e0 con 0x55e66d23b750 2024-03-12T06:02:08.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.685+0000 7f3660ff9640 1 -- 172.21.15.76:0/3209518111 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640997f0 con 0x55e66d23b750 2024-03-12T06:02:08.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.685+0000 7f3673a00740 1 -- 172.21.15.76:0/3209518111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d23b750 msgr2=0x55e66d23db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.685+0000 7f3673a00740 1 --2- 172.21.15.76:0/3209518111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d23b750 0x55e66d23db40 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f36640a23b0 tx=0x7f3664099d40 comp rx=0 tx=0).stop 2024-03-12T06:02:08.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.685+0000 7f3673a00740 1 -- 172.21.15.76:0/3209518111 shutdown_connections 2024-03-12T06:02:08.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.685+0000 7f3673a00740 1 --2- 172.21.15.76:0/3209518111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d23b750 0x55e66d23db40 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.685+0000 7f3673a00740 1 --2- 172.21.15.76:0/3209518111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d232b70 0x55e66d23b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.685+0000 7f3673a00740 1 --2- 172.21.15.76:0/3209518111 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232250 0x55e66d232630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.685+0000 7f3673a00740 1 -- 172.21.15.76:0/3209518111 >> 172.21.15.76:0/3209518111 conn(0x55e66d22c980 msgr2=0x55e66d22cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f3673a00740 1 -- 172.21.15.76:0/3209518111 shutdown_connections 2024-03-12T06:02:08.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f3673a00740 1 -- 172.21.15.76:0/3209518111 wait complete. 2024-03-12T06:02:08.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f3673a00740 1 -- start start 2024-03-12T06:02:08.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f3673a00740 1 Processor -- start 2024-03-12T06:02:08.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f3673a00740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d232250 0x55e66d238770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f3673a00740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232b70 0x55e66d238cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f3673a00740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d23b750 0x55e66d2391f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f3673a00740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d19a020 con 0x55e66d23b750 2024-03-12T06:02:08.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f3673a00740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66ce6d430 con 0x55e66d232b70 2024-03-12T06:02:08.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f3673a00740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66ce29e00 con 0x55e66d232250 2024-03-12T06:02:08.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f3673a00740 10 client.0 _ll_register_callbacks cb 0x55e66d141cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:08.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f366ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d23b750 0x55e66d2391f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f366ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d23b750 0x55e66d2391f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49662/0 (socket says 172.21.15.76:49662) 2024-03-12T06:02:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f366ad9d640 1 -- 172.21.15.76:0/1711321561 learned_addr learned my addr 172.21.15.76:0/1711321561 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f3663fff640 1 --2- 172.21.15.76:0/1711321561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232b70 0x55e66d238cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f366ad9d640 1 -- 172.21.15.76:0/1711321561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d232250 msgr2=0x55e66d238770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f366ad9d640 1 --2- 172.21.15.76:0/1711321561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d232250 0x55e66d238770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f366ad9d640 1 -- 172.21.15.76:0/1711321561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232b70 msgr2=0x55e66d238cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.686+0000 7f366ad9d640 1 --2- 172.21.15.76:0/1711321561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232b70 0x55e66d238cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f366ad9d640 1 -- 172.21.15.76:0/1711321561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366c1a4790 con 0x55e66d23b750 2024-03-12T06:02:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3663fff640 1 --2- 172.21.15.76:0/1711321561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232b70 0x55e66d238cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f366ad9d640 1 --2- 172.21.15.76:0/1711321561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d23b750 0x55e66d2391f0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f36640cb260 tx=0x7f36640cb110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1152f78d76a94ad3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3651ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3651ffb640 1 -- 172.21.15.76:0/1711321561 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664096900 con 0x55e66d23b750 2024-03-12T06:02:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3651ffb640 1 -- 172.21.15.76:0/1711321561 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f36640af070 con 0x55e66d23b750 2024-03-12T06:02:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3673a00740 1 -- 172.21.15.76:0/1711321561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e66ce997e0 con 0x55e66d23b750 2024-03-12T06:02:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3651ffb640 1 -- 172.21.15.76:0/1711321561 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f366409e290 con 0x55e66d23b750 2024-03-12T06:02:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3673a00740 10 client.16947 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3673a00740 1 -- 172.21.15.76:0/1711321561 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e66d1986a0 con 0x55e66d23b750 2024-03-12T06:02:08.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3651ffb640 1 -- 172.21.15.76:0/1711321561 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e66ce997e0 con 0x55e66d23b750 2024-03-12T06:02:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3651ffb640 1 -- 172.21.15.76:0/1711321561 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f36640b1070 con 0x55e66d23b750 2024-03-12T06:02:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3673a00740 10 client.16947 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3673a00740 10 client.16947 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:02:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3673a00740 20 client.16947 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3673a00740 1 --2- 172.21.15.76:0/1711321561 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e66d24b690 0x55e66d24daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3673a00740 4 client.16947 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:02:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.687+0000 7f3673a00740 1 -- 172.21.15.76:0/1711321561 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e66ced6720 con 0x55e66d24b690 2024-03-12T06:02:08.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.688+0000 7f366a59c640 1 --2- 172.21.15.76:0/1711321561 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e66d24b690 0x55e66d24daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.688+0000 7f366a59c640 1 --2- 172.21.15.76:0/1711321561 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e66d24b690 0x55e66d24daa0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1d77b1ff44af923c server_cookie=95941c78076407a9 in_seq=0 out_seq=0 2024-03-12T06:02:08.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.688+0000 7f3651ffb640 10 client.16947 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:02:08.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.688+0000 7f3651ffb640 1 -- 172.21.15.76:0/1711321561 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e66ced6720 con 0x55e66d24b690 2024-03-12T06:02:08.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.688+0000 7f3651ffb640 10 client.16947 handle_command_reply: tid=0 2024-03-12T06:02:08.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.688+0000 7f3673a00740 1 client.16947 shutdown 2024-03-12T06:02:08.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.689+0000 7f3673a00740 1 -- 172.21.15.76:0/1711321561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d23b750 msgr2=0x55e66d2391f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.689+0000 7f3673a00740 1 --2- 172.21.15.76:0/1711321561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d23b750 0x55e66d2391f0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f36640cb260 tx=0x7f36640cb110 comp rx=0 tx=0).stop 2024-03-12T06:02:08.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.689+0000 7f3673a00740 1 -- 172.21.15.76:0/1711321561 shutdown_connections 2024-03-12T06:02:08.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.689+0000 7f3673a00740 1 --2- 172.21.15.76:0/1711321561 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e66d24b690 0x55e66d24daa0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.689+0000 7f3673a00740 1 --2- 172.21.15.76:0/1711321561 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d23b750 0x55e66d2391f0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.689+0000 7f3651ffb640 0 client.16947 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:02:08.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.689+0000 7f3673a00740 1 --2- 172.21.15.76:0/1711321561 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232b70 0x55e66d238cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.689+0000 7f3673a00740 1 --2- 172.21.15.76:0/1711321561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d232250 0x55e66d238770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.689+0000 7f3673a00740 1 -- 172.21.15.76:0/1711321561 >> 172.21.15.76:0/1711321561 conn(0x55e66d22c980 msgr2=0x55e66d22e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.689+0000 7f3673a00740 1 -- 172.21.15.76:0/1711321561 shutdown_connections 2024-03-12T06:02:08.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.689+0000 7f3673a00740 1 -- 172.21.15.76:0/1711321561 wait complete. 2024-03-12T06:02:08.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.689+0000 7f3673a00740 20 client.16947 trim_cache size 0 max 16384 2024-03-12T06:02:08.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.700+0000 7f3673a00740 1 Processor -- start 2024-03-12T06:02:08.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.700+0000 7f3673a00740 1 -- start start 2024-03-12T06:02:08.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.700+0000 7f3673a00740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d232250 0x55e66d23aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.700+0000 7f3673a00740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d232b70 0x55e66d239010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.700+0000 7f3673a00740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d23b750 0x55e66d2359d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.700+0000 7f3673a00740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d11f130 con 0x55e66d232b70 2024-03-12T06:02:08.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.700+0000 7f3673a00740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66ce997e0 con 0x55e66d23b750 2024-03-12T06:02:08.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.700+0000 7f3673a00740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d248580 con 0x55e66d232250 2024-03-12T06:02:08.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f3663fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d232b70 0x55e66d239010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f3663fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d232b70 0x55e66d239010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49664/0 (socket says 172.21.15.76:49664) 2024-03-12T06:02:08.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f3663fff640 1 -- 172.21.15.76:0/1933245737 learned_addr learned my addr 172.21.15.76:0/1933245737 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f366a59c640 1 --2- 172.21.15.76:0/1933245737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d232250 0x55e66d23aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f3663fff640 1 -- 172.21.15.76:0/1933245737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d232250 msgr2=0x55e66d23aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f3663fff640 1 --2- 172.21.15.76:0/1933245737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d232250 0x55e66d23aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f366ad9d640 1 --2- 172.21.15.76:0/1933245737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d23b750 0x55e66d2359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f3663fff640 1 -- 172.21.15.76:0/1933245737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d23b750 msgr2=0x55e66d2359d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f3663fff640 1 --2- 172.21.15.76:0/1933245737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d23b750 0x55e66d2359d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f3663fff640 1 -- 172.21.15.76:0/1933245737 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e66d134c20 con 0x55e66d232b70 2024-03-12T06:02:08.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f366ad9d640 1 --2- 172.21.15.76:0/1933245737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d23b750 0x55e66d2359d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:08.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f3663fff640 1 --2- 172.21.15.76:0/1933245737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d232b70 0x55e66d239010 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3654008fd0 tx=0x7f3654014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f7e283de92c55b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f3653fff640 1 -- 172.21.15.76:0/1933245737 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3654005220 con 0x55e66d232b70 2024-03-12T06:02:08.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f3653fff640 1 -- 172.21.15.76:0/1933245737 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3654004c80 con 0x55e66d232b70 2024-03-12T06:02:08.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.701+0000 7f3653fff640 1 -- 172.21.15.76:0/1933245737 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365401fb90 con 0x55e66d232b70 2024-03-12T06:02:08.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 -- 172.21.15.76:0/1933245737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d232b70 msgr2=0x55e66d239010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 --2- 172.21.15.76:0/1933245737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d232b70 0x55e66d239010 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3654008fd0 tx=0x7f3654014000 comp rx=0 tx=0).stop 2024-03-12T06:02:08.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 -- 172.21.15.76:0/1933245737 shutdown_connections 2024-03-12T06:02:08.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 --2- 172.21.15.76:0/1933245737 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d23b750 0x55e66d2359d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 --2- 172.21.15.76:0/1933245737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d232b70 0x55e66d239010 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 --2- 172.21.15.76:0/1933245737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d232250 0x55e66d23aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 -- 172.21.15.76:0/1933245737 >> 172.21.15.76:0/1933245737 conn(0x55e66d22c980 msgr2=0x55e66d233a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 -- 172.21.15.76:0/1933245737 shutdown_connections 2024-03-12T06:02:08.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 -- 172.21.15.76:0/1933245737 wait complete. 2024-03-12T06:02:08.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 -- start start 2024-03-12T06:02:08.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 Processor -- start 2024-03-12T06:02:08.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d232250 0x55e66d241140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232b70 0x55e66d241680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d23b750 0x55e66d241bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d2a1c90 con 0x55e66d232250 2024-03-12T06:02:08.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d2a1e00 con 0x55e66d232b70 2024-03-12T06:02:08.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e66d2a1fc0 con 0x55e66d23b750 2024-03-12T06:02:08.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f3673a00740 10 client.0 _ll_register_callbacks cb 0x7f3664004020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:08.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f366a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d232250 0x55e66d241140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f366a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d232250 0x55e66d241140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49668/0 (socket says 172.21.15.76:49668) 2024-03-12T06:02:08.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.702+0000 7f366a59c640 1 -- 172.21.15.76:0/3167127330 learned_addr learned my addr 172.21.15.76:0/3167127330 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f366ad9d640 1 --2- 172.21.15.76:0/3167127330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d23b750 0x55e66d241bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f3663fff640 1 --2- 172.21.15.76:0/3167127330 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232b70 0x55e66d241680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f366a59c640 1 -- 172.21.15.76:0/3167127330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d23b750 msgr2=0x55e66d241bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f366a59c640 1 --2- 172.21.15.76:0/3167127330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d23b750 0x55e66d241bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f366a59c640 1 -- 172.21.15.76:0/3167127330 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232b70 msgr2=0x55e66d241680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f366a59c640 1 --2- 172.21.15.76:0/3167127330 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232b70 0x55e66d241680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f366a59c640 1 -- 172.21.15.76:0/3167127330 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3654008d00 con 0x55e66d232250 2024-03-12T06:02:08.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f3663fff640 1 --2- 172.21.15.76:0/3167127330 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232b70 0x55e66d241680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:08.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f366a59c640 1 --2- 172.21.15.76:0/3167127330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d232250 0x55e66d241140 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f365c005ab0 tx=0x7f365c004ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58ad2e3640edc34 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f3651ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:08.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f3651ffb640 1 -- 172.21.15.76:0/3167127330 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365c02c7e0 con 0x55e66d232250 2024-03-12T06:02:08.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f3651ffb640 1 -- 172.21.15.76:0/3167127330 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f365c02c940 con 0x55e66d232250 2024-03-12T06:02:08.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f3673a00740 1 -- 172.21.15.76:0/3167127330 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36640a28e0 con 0x55e66d232250 2024-03-12T06:02:08.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f3651ffb640 1 -- 172.21.15.76:0/3167127330 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f365c02cb00 con 0x55e66d232250 2024-03-12T06:02:08.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f3673a00740 10 client.16953 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:08.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f3673a00740 1 -- 172.21.15.76:0/3167127330 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e66d15a8c0 con 0x55e66d232250 2024-03-12T06:02:08.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f3651ffb640 1 -- 172.21.15.76:0/3167127330 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f365c057080 con 0x55e66d232250 2024-03-12T06:02:08.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f3651ffb640 1 -- 172.21.15.76:0/3167127330 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f365c03c040 con 0x55e66d232250 2024-03-12T06:02:08.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f3673a00740 10 client.16953 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:08.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f3673a00740 10 client.16953 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:02:08.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.703+0000 7f3673a00740 20 client.16953 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:08.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3673a00740 1 --2- 172.21.15.76:0/3167127330 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e66d2a3760 0x55e66d2a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3673a00740 4 client.16953 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:02:08.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3673a00740 1 -- 172.21.15.76:0/3167127330 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e66cf25890 con 0x55e66d2a3760 2024-03-12T06:02:08.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3663fff640 1 --2- 172.21.15.76:0/3167127330 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e66d2a3760 0x55e66d2a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3651ffb640 10 client.16953 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:02:08.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3663fff640 1 --2- 172.21.15.76:0/3167127330 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e66d2a3760 0x55e66d2a5b80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cf1734e13cc7d0f3 server_cookie=a6e70778779d8ab9 in_seq=0 out_seq=0 2024-03-12T06:02:08.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3651ffb640 1 -- 172.21.15.76:0/3167127330 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e66cf25890 con 0x55e66d2a3760 2024-03-12T06:02:08.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3651ffb640 10 client.16953 handle_command_reply: tid=0 2024-03-12T06:02:08.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3673a00740 1 client.16953 shutdown 2024-03-12T06:02:08.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3673a00740 1 -- 172.21.15.76:0/3167127330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d232250 msgr2=0x55e66d241140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3673a00740 1 --2- 172.21.15.76:0/3167127330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d232250 0x55e66d241140 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f365c005ab0 tx=0x7f365c004ba0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3673a00740 1 -- 172.21.15.76:0/3167127330 shutdown_connections 2024-03-12T06:02:08.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3673a00740 1 --2- 172.21.15.76:0/3167127330 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55e66d2a3760 0x55e66d2a5b80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3673a00740 1 --2- 172.21.15.76:0/3167127330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e66d23b750 0x55e66d241bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3673a00740 1 --2- 172.21.15.76:0/3167127330 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e66d232b70 0x55e66d241680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3673a00740 1 --2- 172.21.15.76:0/3167127330 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e66d232250 0x55e66d241140 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3673a00740 1 -- 172.21.15.76:0/3167127330 >> 172.21.15.76:0/3167127330 conn(0x55e66d22c980 msgr2=0x55e66d22b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3651ffb640 0 client.16953 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:02:08.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3673a00740 1 -- 172.21.15.76:0/3167127330 shutdown_connections 2024-03-12T06:02:08.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.704+0000 7f3673a00740 1 -- 172.21.15.76:0/3167127330 wait complete. 2024-03-12T06:02:08.726 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.705+0000 7f3673a00740 20 client.16953 trim_cache size 0 max 16384 2024-03-12T06:02:08.727 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:02:08.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.705+0000 7f3670d4f640 1 -- 172.21.15.76:0/3926804081 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f36580a1ba0 msgr2=0x7f36580a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.705+0000 7f3670d4f640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f36580a1ba0 0x7f36580a4060 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f366c06a3d0 tx=0x7f36540117b0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.705+0000 7f3670d4f640 1 -- 172.21.15.76:0/3926804081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366c199070 msgr2=0x7f366c1a3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.705+0000 7f3670d4f640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366c199070 0x7f366c1a3620 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f366400da40 tx=0x7f366400df00 comp rx=0 tx=0).stop 2024-03-12T06:02:08.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.705+0000 7f3670d4f640 1 -- 172.21.15.76:0/3926804081 shutdown_connections 2024-03-12T06:02:08.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.705+0000 7f3670d4f640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f36580a1ba0 0x7f36580a4060 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.705+0000 7f3670d4f640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366c199070 0x7f366c1a3620 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.705+0000 7f3670d4f640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f366c105360 0x7f366c198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.705+0000 7f3670d4f640 1 --2- 172.21.15.76:0/3926804081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f366c100290 0x7f366c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.705+0000 7f3670d4f640 1 -- 172.21.15.76:0/3926804081 >> 172.21.15.76:0/3926804081 conn(0x7f366c077200 msgr2=0x7f366c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.705+0000 7f3670d4f640 1 -- 172.21.15.76:0/3926804081 shutdown_connections 2024-03-12T06:02:08.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.705+0000 7f3670d4f640 1 -- 172.21.15.76:0/3926804081 wait complete. 2024-03-12T06:02:08.730 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:02:08.731 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T06:02:08.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.832+0000 7fc513a1c640 1 -- 172.21.15.76:0/1075321581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc50c106150 msgr2=0x7fc50c106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.832+0000 7fc513a1c640 1 --2- 172.21.15.76:0/1075321581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc50c106150 0x7fc50c106530 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fc504009cd0 tx=0x7fc50402c0f0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.832+0000 7fc513a1c640 1 -- 172.21.15.76:0/1075321581 shutdown_connections 2024-03-12T06:02:08.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.832+0000 7fc513a1c640 1 --2- 172.21.15.76:0/1075321581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc50c0fffb0 0x7fc50c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.832+0000 7fc513a1c640 1 --2- 172.21.15.76:0/1075321581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc50c0ff5f0 0x7fc50c0ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.832+0000 7fc513a1c640 1 --2- 172.21.15.76:0/1075321581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc50c106150 0x7fc50c106530 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.832+0000 7fc513a1c640 1 -- 172.21.15.76:0/1075321581 >> 172.21.15.76:0/1075321581 conn(0x7fc50c076580 msgr2=0x7fc50c0fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.832+0000 7fc513a1c640 1 -- 172.21.15.76:0/1075321581 shutdown_connections 2024-03-12T06:02:08.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.832+0000 7fc513a1c640 1 -- 172.21.15.76:0/1075321581 wait complete. 2024-03-12T06:02:08.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc513a1c640 1 Processor -- start 2024-03-12T06:02:08.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc513a1c640 1 -- start start 2024-03-12T06:02:08.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc513a1c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc50c0ff5f0 0x7fc50c19d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc513a1c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc50c0fffb0 0x7fc50c19dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc513a1c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc50c106150 0x7fc50c197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc513a1c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50c197dc0 con 0x7fc50c0ff5f0 2024-03-12T06:02:08.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc513a1c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50c197f30 con 0x7fc50c0fffb0 2024-03-12T06:02:08.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc513a1c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50c1980f0 con 0x7fc50c106150 2024-03-12T06:02:08.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc5117b8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc50c0ff5f0 0x7fc50c19d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc5117b8640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc50c0ff5f0 0x7fc50c19d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49692/0 (socket says 172.21.15.76:49692) 2024-03-12T06:02:08.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc5117b8640 1 -- 172.21.15.76:0/779792074 learned_addr learned my addr 172.21.15.76:0/779792074 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc510fb7640 1 --2- 172.21.15.76:0/779792074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc50c0fffb0 0x7fc50c19dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc5117b8640 1 -- 172.21.15.76:0/779792074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc50c106150 msgr2=0x7fc50c197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc5117b8640 1 --2- 172.21.15.76:0/779792074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc50c106150 0x7fc50c197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc5117b8640 1 -- 172.21.15.76:0/779792074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc50c0fffb0 msgr2=0x7fc50c19dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc5117b8640 1 --2- 172.21.15.76:0/779792074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc50c0fffb0 0x7fc50c19dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc5117b8640 1 -- 172.21.15.76:0/779792074 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc504009900 con 0x7fc50c0ff5f0 2024-03-12T06:02:08.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc510fb7640 1 --2- 172.21.15.76:0/779792074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc50c0fffb0 0x7fc50c19dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:02:08.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.833+0000 7fc5117b8640 1 --2- 172.21.15.76:0/779792074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc50c0ff5f0 0x7fc50c19d570 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fc504004af0 tx=0x7fc504035cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.834+0000 7fc50a7fc640 1 -- 172.21.15.76:0/779792074 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc504032ab0 con 0x7fc50c0ff5f0 2024-03-12T06:02:08.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.834+0000 7fc50a7fc640 1 -- 172.21.15.76:0/779792074 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc504003e50 con 0x7fc50c0ff5f0 2024-03-12T06:02:08.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.834+0000 7fc50a7fc640 1 -- 172.21.15.76:0/779792074 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50403f440 con 0x7fc50c0ff5f0 2024-03-12T06:02:08.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.834+0000 7fc513a1c640 1 -- 172.21.15.76:0/779792074 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc50c198370 con 0x7fc50c0ff5f0 2024-03-12T06:02:08.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.834+0000 7fc513a1c640 1 -- 172.21.15.76:0/779792074 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc50c1a44f0 con 0x7fc50c0ff5f0 2024-03-12T06:02:08.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.836+0000 7fc50a7fc640 1 -- 172.21.15.76:0/779792074 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc50403b040 con 0x7fc50c0ff5f0 2024-03-12T06:02:08.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.836+0000 7fc5165f3740 1 Processor -- start 2024-03-12T06:02:08.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.836+0000 7fc5165f3740 1 -- start start 2024-03-12T06:02:08.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.836+0000 7fc5165f3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782250 0x55ed26782630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.836+0000 7fc5165f3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782b70 0x55ed2678b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.836+0000 7fc5165f3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed2678db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.836+0000 7fc5165f3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed2673d960 con 0x55ed2678b750 2024-03-12T06:02:08.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.836+0000 7fc5165f3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed266f1310 con 0x55ed26782250 2024-03-12T06:02:08.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.836+0000 7fc5165f3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed266ee840 con 0x55ed26782b70 2024-03-12T06:02:08.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.836+0000 7fc50a7fc640 1 --2- 172.21.15.76:0/779792074 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc4f40a1c20 0x7fc4f40a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc510fb7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed2678db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc510fb7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed2678db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49700/0 (socket says 172.21.15.76:49700) 2024-03-12T06:02:08.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc510fb7640 1 -- 172.21.15.76:0/3584335758 learned_addr learned my addr 172.21.15.76:0/3584335758 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc511fb9640 1 --2- 172.21.15.76:0/779792074 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc4f40a1c20 0x7fc4f40a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc50a7fc640 1 -- 172.21.15.76:0/779792074 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fc5040f00f0 con 0x7fc50c0ff5f0 2024-03-12T06:02:08.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc5117b8640 1 --2- 172.21.15.76:0/3584335758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782b70 0x55ed2678b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc511fb9640 1 --2- 172.21.15.76:0/3584335758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782250 0x55ed26782630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc510fb7640 1 -- 172.21.15.76:0/3584335758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782b70 msgr2=0x55ed2678b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc510fb7640 1 --2- 172.21.15.76:0/3584335758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782b70 0x55ed2678b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc510fb7640 1 -- 172.21.15.76:0/3584335758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782250 msgr2=0x55ed26782630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc510fb7640 1 --2- 172.21.15.76:0/3584335758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782250 0x55ed26782630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc510fb7640 1 -- 172.21.15.76:0/3584335758 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed266ac060 con 0x55ed2678b750 2024-03-12T06:02:08.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc511fb9640 1 --2- 172.21.15.76:0/3584335758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782250 0x55ed26782630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:08.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc5117b8640 1 --2- 172.21.15.76:0/3584335758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782b70 0x55ed2678b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:02:08.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc510fb7640 1 --2- 172.21.15.76:0/3584335758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed2678db40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc50000cc50 tx=0x7fc50000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7ee75599124ae1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc511fb9640 1 --2- 172.21.15.76:0/779792074 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc4f40a1c20 0x7fc4f40a40e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc005d00 tx=0x7fc4fc005c50 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc4f37fe640 1 -- 172.21.15.76:0/3584335758 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc500019660 con 0x55ed2678b750 2024-03-12T06:02:08.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc4f37fe640 1 -- 172.21.15.76:0/3584335758 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc500019ca0 con 0x55ed2678b750 2024-03-12T06:02:08.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.837+0000 7fc4f37fe640 1 -- 172.21.15.76:0/3584335758 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc500010cd0 con 0x55ed2678b750 2024-03-12T06:02:08.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.838+0000 7fc5165f3740 1 -- 172.21.15.76:0/3584335758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 msgr2=0x55ed2678db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.838+0000 7fc5165f3740 1 --2- 172.21.15.76:0/3584335758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed2678db40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc50000cc50 tx=0x7fc50000e720 comp rx=0 tx=0).stop 2024-03-12T06:02:08.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.838+0000 7fc5165f3740 1 -- 172.21.15.76:0/3584335758 shutdown_connections 2024-03-12T06:02:08.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.838+0000 7fc5165f3740 1 --2- 172.21.15.76:0/3584335758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed2678db40 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.838+0000 7fc5165f3740 1 --2- 172.21.15.76:0/3584335758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782b70 0x55ed2678b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.838+0000 7fc5165f3740 1 --2- 172.21.15.76:0/3584335758 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782250 0x55ed26782630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.838+0000 7fc5165f3740 1 -- 172.21.15.76:0/3584335758 >> 172.21.15.76:0/3584335758 conn(0x55ed2677c980 msgr2=0x55ed2677cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.838+0000 7fc5165f3740 1 -- 172.21.15.76:0/3584335758 shutdown_connections 2024-03-12T06:02:08.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.838+0000 7fc5165f3740 1 -- 172.21.15.76:0/3584335758 wait complete. 2024-03-12T06:02:08.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.838+0000 7fc5165f3740 1 -- start start 2024-03-12T06:02:08.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.838+0000 7fc5165f3740 1 Processor -- start 2024-03-12T06:02:08.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc5165f3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782250 0x55ed26788770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc5165f3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed26782b70 0x55ed26788cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc5165f3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed2678b750 0x55ed267891f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc5165f3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed266ea020 con 0x55ed26782b70 2024-03-12T06:02:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc5165f3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed263bd430 con 0x55ed2678b750 2024-03-12T06:02:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc5165f3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed26379e00 con 0x55ed26782250 2024-03-12T06:02:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc5165f3740 10 client.0 _ll_register_callbacks cb 0x55ed26691cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc510fb7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed26782b70 0x55ed26788cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc510fb7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed26782b70 0x55ed26788cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49710/0 (socket says 172.21.15.76:49710) 2024-03-12T06:02:08.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc510fb7640 1 -- 172.21.15.76:0/2553537358 learned_addr learned my addr 172.21.15.76:0/2553537358 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc511fb9640 1 --2- 172.21.15.76:0/2553537358 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed2678b750 0x55ed267891f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc510fb7640 1 -- 172.21.15.76:0/2553537358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782250 msgr2=0x55ed26788770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc510fb7640 1 --2- 172.21.15.76:0/2553537358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782250 0x55ed26788770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc510fb7640 1 -- 172.21.15.76:0/2553537358 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed2678b750 msgr2=0x55ed267891f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc510fb7640 1 --2- 172.21.15.76:0/2553537358 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed2678b750 0x55ed267891f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc510fb7640 1 -- 172.21.15.76:0/2553537358 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc50000c8b0 con 0x55ed26782b70 2024-03-12T06:02:08.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc510fb7640 1 --2- 172.21.15.76:0/2553537358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed26782b70 0x55ed26788cb0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc500010950 tx=0x7fc500010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c13fd8b58860b47 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc4f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:08.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc4f0ff9640 1 -- 172.21.15.76:0/2553537358 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc500019870 con 0x55ed26782b70 2024-03-12T06:02:08.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc4f0ff9640 1 -- 172.21.15.76:0/2553537358 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fc5000199d0 con 0x55ed26782b70 2024-03-12T06:02:08.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.839+0000 7fc5165f3740 1 -- 172.21.15.76:0/2553537358 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ed263e97e0 con 0x55ed26782b70 2024-03-12T06:02:08.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.840+0000 7fc4f0ff9640 1 -- 172.21.15.76:0/2553537358 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc500010e50 con 0x55ed26782b70 2024-03-12T06:02:08.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.840+0000 7fc5165f3740 10 client.16965 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:08.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.840+0000 7fc5165f3740 1 -- 172.21.15.76:0/2553537358 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ed266e86a0 con 0x55ed26782b70 2024-03-12T06:02:08.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.840+0000 7fc4f0ff9640 1 -- 172.21.15.76:0/2553537358 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc500030080 con 0x55ed26782b70 2024-03-12T06:02:08.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.840+0000 7fc4f0ff9640 1 -- 172.21.15.76:0/2553537358 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc500022070 con 0x55ed26782b70 2024-03-12T06:02:08.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.840+0000 7fc5165f3740 10 client.16965 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:08.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.840+0000 7fc5165f3740 10 client.16965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:08.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.840+0000 7fc5165f3740 20 client.16965 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:08.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.840+0000 7fc5165f3740 1 --2- 172.21.15.76:0/2553537358 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ed2679ba90 0x55ed2679dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.840+0000 7fc5165f3740 4 client.16965 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:02:08.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.840+0000 7fc5165f3740 1 -- 172.21.15.76:0/2553537358 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed26426720 con 0x55ed2679ba90 2024-03-12T06:02:08.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.840+0000 7fc5117b8640 1 --2- 172.21.15.76:0/2553537358 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ed2679ba90 0x55ed2679dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.841+0000 7fc4f0ff9640 10 client.16965 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:08.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.841+0000 7fc5117b8640 1 --2- 172.21.15.76:0/2553537358 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ed2679ba90 0x55ed2679dea0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9cf13ff8d1a41ba server_cookie=64b06980e0fcf02b in_seq=0 out_seq=0 2024-03-12T06:02:08.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.841+0000 7fc4f0ff9640 1 -- 172.21.15.76:0/2553537358 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ed26426720 con 0x55ed2679ba90 2024-03-12T06:02:08.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.841+0000 7fc4f0ff9640 10 client.16965 handle_command_reply: tid=0 2024-03-12T06:02:08.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.841+0000 7fc5165f3740 1 client.16965 shutdown 2024-03-12T06:02:08.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.841+0000 7fc5165f3740 1 -- 172.21.15.76:0/2553537358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed26782b70 msgr2=0x55ed26788cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.841+0000 7fc5165f3740 1 --2- 172.21.15.76:0/2553537358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed26782b70 0x55ed26788cb0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc500010950 tx=0x7fc500010ce0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.841+0000 7fc5165f3740 1 -- 172.21.15.76:0/2553537358 shutdown_connections 2024-03-12T06:02:08.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.841+0000 7fc5165f3740 1 --2- 172.21.15.76:0/2553537358 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed2678b750 0x55ed267891f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.841+0000 7fc5165f3740 1 --2- 172.21.15.76:0/2553537358 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ed2679ba90 0x55ed2679dea0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.841+0000 7fc5165f3740 1 --2- 172.21.15.76:0/2553537358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed26782b70 0x55ed26788cb0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.841+0000 7fc5165f3740 1 --2- 172.21.15.76:0/2553537358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782250 0x55ed26788770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.841+0000 7fc4f0ff9640 0 client.16965 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:08.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.841+0000 7fc5165f3740 1 -- 172.21.15.76:0/2553537358 >> 172.21.15.76:0/2553537358 conn(0x55ed2677c980 msgr2=0x55ed2677e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.842+0000 7fc5165f3740 1 -- 172.21.15.76:0/2553537358 shutdown_connections 2024-03-12T06:02:08.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.842+0000 7fc5165f3740 1 -- 172.21.15.76:0/2553537358 wait complete. 2024-03-12T06:02:08.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.842+0000 7fc5165f3740 20 client.16965 trim_cache size 0 max 16384 2024-03-12T06:02:08.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.853+0000 7fc5165f3740 1 Processor -- start 2024-03-12T06:02:08.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.853+0000 7fc5165f3740 1 -- start start 2024-03-12T06:02:08.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc5165f3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782250 0x55ed2678aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc5165f3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782b70 0x55ed267929e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc5165f3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed26789010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc5165f3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed2666f130 con 0x55ed2678b750 2024-03-12T06:02:08.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc5165f3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed26792f20 con 0x55ed26782b70 2024-03-12T06:02:08.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc5165f3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed26786d10 con 0x55ed26782250 2024-03-12T06:02:08.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc511fb9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed26789010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc511fb9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed26789010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49718/0 (socket says 172.21.15.76:49718) 2024-03-12T06:02:08.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc511fb9640 1 -- 172.21.15.76:0/1277778499 learned_addr learned my addr 172.21.15.76:0/1277778499 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc5117b8640 1 --2- 172.21.15.76:0/1277778499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782250 0x55ed2678aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc510fb7640 1 --2- 172.21.15.76:0/1277778499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782b70 0x55ed267929e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc511fb9640 1 -- 172.21.15.76:0/1277778499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782250 msgr2=0x55ed2678aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc511fb9640 1 --2- 172.21.15.76:0/1277778499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782250 0x55ed2678aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc511fb9640 1 -- 172.21.15.76:0/1277778499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782b70 msgr2=0x55ed267929e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc511fb9640 1 --2- 172.21.15.76:0/1277778499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782b70 0x55ed267929e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc511fb9640 1 -- 172.21.15.76:0/1277778499 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed26684c20 con 0x55ed2678b750 2024-03-12T06:02:08.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc511fb9640 1 --2- 172.21.15.76:0/1277778499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed26789010 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0098e0 tx=0x7fc4fc00b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7102bb0e908439db server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc4f2ffd640 1 -- 172.21.15.76:0/1277778499 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc003d70 con 0x55ed2678b750 2024-03-12T06:02:08.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc4f2ffd640 1 -- 172.21.15.76:0/1277778499 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4fc0044c0 con 0x55ed2678b750 2024-03-12T06:02:08.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.854+0000 7fc4f2ffd640 1 -- 172.21.15.76:0/1277778499 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc020810 con 0x55ed2678b750 2024-03-12T06:02:08.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.855+0000 7fc5165f3740 1 -- 172.21.15.76:0/1277778499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 msgr2=0x55ed26789010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.855+0000 7fc5165f3740 1 --2- 172.21.15.76:0/1277778499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed26789010 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0098e0 tx=0x7fc4fc00b000 comp rx=0 tx=0).stop 2024-03-12T06:02:08.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.855+0000 7fc5165f3740 1 -- 172.21.15.76:0/1277778499 shutdown_connections 2024-03-12T06:02:08.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.855+0000 7fc5165f3740 1 --2- 172.21.15.76:0/1277778499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed26789010 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.855+0000 7fc5165f3740 1 --2- 172.21.15.76:0/1277778499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782b70 0x55ed267929e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.855+0000 7fc5165f3740 1 --2- 172.21.15.76:0/1277778499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782250 0x55ed2678aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.855+0000 7fc5165f3740 1 -- 172.21.15.76:0/1277778499 >> 172.21.15.76:0/1277778499 conn(0x55ed2677c980 msgr2=0x55ed26783a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.855+0000 7fc5165f3740 1 -- 172.21.15.76:0/1277778499 shutdown_connections 2024-03-12T06:02:08.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.855+0000 7fc5165f3740 1 -- 172.21.15.76:0/1277778499 wait complete. 2024-03-12T06:02:08.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.855+0000 7fc5165f3740 1 -- start start 2024-03-12T06:02:08.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.855+0000 7fc5165f3740 1 Processor -- start 2024-03-12T06:02:08.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc5165f3740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782250 0x55ed267f08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc5165f3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782b70 0x55ed267f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc5165f3740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed267eace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc5165f3740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed267eb2a0 con 0x55ed2678b750 2024-03-12T06:02:08.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc5165f3740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed267eb410 con 0x55ed26782250 2024-03-12T06:02:08.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc5165f3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed267eb5d0 con 0x55ed26782b70 2024-03-12T06:02:08.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc5165f3740 10 client.0 _ll_register_callbacks cb 0x7fc50000c410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:08.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc511fb9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed267eace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc511fb9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed267eace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49734/0 (socket says 172.21.15.76:49734) 2024-03-12T06:02:08.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc511fb9640 1 -- 172.21.15.76:0/1830314138 learned_addr learned my addr 172.21.15.76:0/1830314138 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:08.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc510fb7640 1 --2- 172.21.15.76:0/1830314138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782b70 0x55ed267f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc5117b8640 1 --2- 172.21.15.76:0/1830314138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782250 0x55ed267f08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc511fb9640 1 -- 172.21.15.76:0/1830314138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782b70 msgr2=0x55ed267f0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc511fb9640 1 --2- 172.21.15.76:0/1830314138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782b70 0x55ed267f0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc511fb9640 1 -- 172.21.15.76:0/1830314138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782250 msgr2=0x55ed267f08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:08.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc511fb9640 1 --2- 172.21.15.76:0/1830314138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782250 0x55ed267f08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc511fb9640 1 -- 172.21.15.76:0/1830314138 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4fc009ce0 con 0x55ed2678b750 2024-03-12T06:02:08.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc5117b8640 1 --2- 172.21.15.76:0/1830314138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782250 0x55ed267f08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:08.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc511fb9640 1 --2- 172.21.15.76:0/1830314138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed267eace0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc004700 tx=0x7fc4fc0058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49345788f2184bf server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:08.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc4f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:08.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc4f0ff9640 1 -- 172.21.15.76:0/1830314138 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc011b10 con 0x55ed2678b750 2024-03-12T06:02:08.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc4f0ff9640 1 -- 172.21.15.76:0/1830314138 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fc4fc011c70 con 0x55ed2678b750 2024-03-12T06:02:08.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc5165f3740 1 -- 172.21.15.76:0/1830314138 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc500019ca0 con 0x55ed2678b750 2024-03-12T06:02:08.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc4f0ff9640 1 -- 172.21.15.76:0/1830314138 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4fc005020 con 0x55ed2678b750 2024-03-12T06:02:08.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc5165f3740 10 client.16971 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:08.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.856+0000 7fc5165f3740 1 -- 172.21.15.76:0/1830314138 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ed266aa8c0 con 0x55ed2678b750 2024-03-12T06:02:08.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.857+0000 7fc4f0ff9640 1 -- 172.21.15.76:0/1830314138 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc500019ca0 con 0x55ed2678b750 2024-03-12T06:02:08.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.857+0000 7fc4f0ff9640 1 -- 172.21.15.76:0/1830314138 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc4fc032410 con 0x55ed2678b750 2024-03-12T06:02:08.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.857+0000 7fc5165f3740 10 client.16971 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:08.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.857+0000 7fc5165f3740 10 client.16971 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:08.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.857+0000 7fc5165f3740 20 client.16971 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:08.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.857+0000 7fc5165f3740 1 --2- 172.21.15.76:0/1830314138 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ed267f87a0 0x55ed267fabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:08.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.857+0000 7fc5165f3740 4 client.16971 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:02:08.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.857+0000 7fc5165f3740 1 -- 172.21.15.76:0/1830314138 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ed26475890 con 0x55ed267f87a0 2024-03-12T06:02:08.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.857+0000 7fc5117b8640 1 --2- 172.21.15.76:0/1830314138 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ed267f87a0 0x55ed267fabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:08.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.857+0000 7fc4f0ff9640 10 client.16971 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:08.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.857+0000 7fc5117b8640 1 --2- 172.21.15.76:0/1830314138 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ed267f87a0 0x55ed267fabc0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70c9cb545d62bfdd server_cookie=f9273f551ac49f46 in_seq=0 out_seq=0 2024-03-12T06:02:08.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc4f0ff9640 1 -- 172.21.15.76:0/1830314138 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ed26475890 con 0x55ed267f87a0 2024-03-12T06:02:08.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc4f0ff9640 10 client.16971 handle_command_reply: tid=0 2024-03-12T06:02:08.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc5165f3740 1 client.16971 shutdown 2024-03-12T06:02:08.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc5165f3740 1 -- 172.21.15.76:0/1830314138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 msgr2=0x55ed267eace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc5165f3740 1 --2- 172.21.15.76:0/1830314138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed267eace0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc004700 tx=0x7fc4fc0058e0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc5165f3740 1 -- 172.21.15.76:0/1830314138 shutdown_connections 2024-03-12T06:02:08.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc5165f3740 1 --2- 172.21.15.76:0/1830314138 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ed267f87a0 0x55ed267fabc0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc5165f3740 1 --2- 172.21.15.76:0/1830314138 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ed2678b750 0x55ed267eace0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc5165f3740 1 --2- 172.21.15.76:0/1830314138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed26782b70 0x55ed267f0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc5165f3740 1 --2- 172.21.15.76:0/1830314138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ed26782250 0x55ed267f08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc4f0ff9640 0 client.16971 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:08.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc5165f3740 1 -- 172.21.15.76:0/1830314138 >> 172.21.15.76:0/1830314138 conn(0x55ed2677c980 msgr2=0x55ed2677b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc5165f3740 1 -- 172.21.15.76:0/1830314138 shutdown_connections 2024-03-12T06:02:08.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc5165f3740 1 -- 172.21.15.76:0/1830314138 wait complete. 2024-03-12T06:02:08.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc5165f3740 20 client.16971 trim_cache size 0 max 16384 2024-03-12T06:02:08.881 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:02:08.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc513a1c640 1 -- 172.21.15.76:0/779792074 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc4f40a1c20 msgr2=0x7fc4f40a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc513a1c640 1 --2- 172.21.15.76:0/779792074 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc4f40a1c20 0x7fc4f40a40e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc005d00 tx=0x7fc4fc005c50 comp rx=0 tx=0).stop 2024-03-12T06:02:08.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc513a1c640 1 -- 172.21.15.76:0/779792074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc50c0ff5f0 msgr2=0x7fc50c19d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:08.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc513a1c640 1 --2- 172.21.15.76:0/779792074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc50c0ff5f0 0x7fc50c19d570 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fc504004af0 tx=0x7fc504035cd0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc513a1c640 1 -- 172.21.15.76:0/779792074 shutdown_connections 2024-03-12T06:02:08.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc513a1c640 1 --2- 172.21.15.76:0/779792074 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc4f40a1c20 0x7fc4f40a40e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc513a1c640 1 --2- 172.21.15.76:0/779792074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc50c106150 0x7fc50c197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc513a1c640 1 --2- 172.21.15.76:0/779792074 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc50c0fffb0 0x7fc50c19dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc513a1c640 1 --2- 172.21.15.76:0/779792074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc50c0ff5f0 0x7fc50c19d570 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:08.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc513a1c640 1 -- 172.21.15.76:0/779792074 >> 172.21.15.76:0/779792074 conn(0x7fc50c076580 msgr2=0x7fc50c104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:08.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.858+0000 7fc513a1c640 1 -- 172.21.15.76:0/779792074 shutdown_connections 2024-03-12T06:02:08.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:08.859+0000 7fc513a1c640 1 -- 172.21.15.76:0/779792074 wait complete. 2024-03-12T06:02:09.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:09 smithi082 ceph-mon[36382]: osdmap e90: 12 total, 12 up, 12 in 2024-03-12T06:02:09.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:09 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1723628104' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:02:09.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:09 smithi082 ceph-mon[36382]: pgmap v195: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:02:09.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:09 smithi088 ceph-mon[36585]: osdmap e90: 12 total, 12 up, 12 in 2024-03-12T06:02:09.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:09 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1723628104' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:02:09.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:09 smithi088 ceph-mon[36585]: pgmap v195: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:02:09.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:09 smithi076 ceph-mon[32095]: osdmap e90: 12 total, 12 up, 12 in 2024-03-12T06:02:09.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:09 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1723628104' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:02:09.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:09 smithi076 ceph-mon[32095]: pgmap v195: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:02:09.885 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T06:02:09.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:02:09.886 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T06:02:10.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.002+0000 7f077202e640 1 -- 172.21.15.76:0/3182087091 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f076c102d10 msgr2=0x7f076c10b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.002+0000 7f077202e640 1 --2- 172.21.15.76:0/3182087091 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f076c102d10 0x7f076c10b1f0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0760009900 tx=0x7f076002bcd0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.002+0000 7f077202e640 1 -- 172.21.15.76:0/3182087091 shutdown_connections 2024-03-12T06:02:10.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.002+0000 7f077202e640 1 --2- 172.21.15.76:0/3182087091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f076c10b860 0x7f076c10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.002+0000 7f077202e640 1 --2- 172.21.15.76:0/3182087091 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f076c102d10 0x7f076c10b1f0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.002+0000 7f077202e640 1 --2- 172.21.15.76:0/3182087091 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f076c1023f0 0x7f076c1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.002+0000 7f077202e640 1 -- 172.21.15.76:0/3182087091 >> 172.21.15.76:0/3182087091 conn(0x7f076c0771f0 msgr2=0x7f076c077600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:10.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.002+0000 7f077202e640 1 -- 172.21.15.76:0/3182087091 shutdown_connections 2024-03-12T06:02:10.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.002+0000 7f077202e640 1 -- 172.21.15.76:0/3182087091 wait complete. 2024-03-12T06:02:10.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.003+0000 7f077202e640 1 Processor -- start 2024-03-12T06:02:10.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.003+0000 7f077202e640 1 -- start start 2024-03-12T06:02:10.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.003+0000 7f077202e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f076c1023f0 0x7f076c100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.003+0000 7f077202e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f076c102d10 0x7f076c100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.003+0000 7f077202e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f076c10b860 0x7f076c100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.003+0000 7f077202e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c1a5ca0 con 0x7f076c1023f0 2024-03-12T06:02:10.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.003+0000 7f077202e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c1a5e10 con 0x7f076c102d10 2024-03-12T06:02:10.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.003+0000 7f077202e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c1a5fd0 con 0x7f076c10b860 2024-03-12T06:02:10.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.003+0000 7f076b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f076c1023f0 0x7f076c100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.003+0000 7f076b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f076c1023f0 0x7f076c100040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49754/0 (socket says 172.21.15.76:49754) 2024-03-12T06:02:10.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.003+0000 7f076b7fe640 1 -- 172.21.15.76:0/3490283127 learned_addr learned my addr 172.21.15.76:0/3490283127 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:10.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.004+0000 7f076affd640 1 --2- 172.21.15.76:0/3490283127 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f076c102d10 0x7f076c100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.004+0000 7f076bfff640 1 --2- 172.21.15.76:0/3490283127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f076c10b860 0x7f076c100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.004+0000 7f076b7fe640 1 -- 172.21.15.76:0/3490283127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f076c10b860 msgr2=0x7f076c100ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.004+0000 7f076b7fe640 1 --2- 172.21.15.76:0/3490283127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f076c10b860 0x7f076c100ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.004+0000 7f076b7fe640 1 -- 172.21.15.76:0/3490283127 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f076c102d10 msgr2=0x7f076c100580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.004+0000 7f076b7fe640 1 --2- 172.21.15.76:0/3490283127 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f076c102d10 0x7f076c100580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.004+0000 7f076b7fe640 1 -- 172.21.15.76:0/3490283127 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0760009560 con 0x7f076c1023f0 2024-03-12T06:02:10.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.004+0000 7f076b7fe640 1 --2- 172.21.15.76:0/3490283127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f076c1023f0 0x7f076c100040 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f075c00d950 tx=0x7f075c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:10.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.004+0000 7f0768ff9640 1 -- 172.21.15.76:0/3490283127 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c009c30 con 0x7f076c1023f0 2024-03-12T06:02:10.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.004+0000 7f0768ff9640 1 -- 172.21.15.76:0/3490283127 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f075c009d90 con 0x7f076c1023f0 2024-03-12T06:02:10.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.004+0000 7f077202e640 1 -- 172.21.15.76:0/3490283127 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076c1a62b0 con 0x7f076c1023f0 2024-03-12T06:02:10.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.004+0000 7f0768ff9640 1 -- 172.21.15.76:0/3490283127 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c00f7e0 con 0x7f076c1023f0 2024-03-12T06:02:10.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.004+0000 7f077202e640 1 -- 172.21.15.76:0/3490283127 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076c1a6790 con 0x7f076c1023f0 2024-03-12T06:02:10.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.006+0000 7f0768ff9640 1 -- 172.21.15.76:0/3490283127 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f075c010ad0 con 0x7f076c1023f0 2024-03-12T06:02:10.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f0774c00740 1 Processor -- start 2024-03-12T06:02:10.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f0774c00740 1 -- start start 2024-03-12T06:02:10.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f0768ff9640 1 --2- 172.21.15.76:0/3490283127 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f07540a1ba0 0x7f07540a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f0774c00740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d07380 0x562645d07760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f0774c00740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d07d30 0x562645d102e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f076bfff640 1 --2- 172.21.15.76:0/3490283127 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f07540a1ba0 0x7f07540a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f0768ff9640 1 -- 172.21.15.76:0/3490283127 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f075c0cd4c0 con 0x7f076c1023f0 2024-03-12T06:02:10.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f0774c00740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d108b0 0x562645d12d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f0774c00740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562645c75f90 con 0x562645d07d30 2024-03-12T06:02:10.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f0774c00740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562645c734c0 con 0x562645d108b0 2024-03-12T06:02:10.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f0774c00740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562645c6eca0 con 0x562645d07380 2024-03-12T06:02:10.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f076affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d07d30 0x562645d102e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f076affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d07d30 0x562645d102e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49762/0 (socket says 172.21.15.76:49762) 2024-03-12T06:02:10.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f076affd640 1 -- 172.21.15.76:0/580537552 learned_addr learned my addr 172.21.15.76:0/580537552 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:10.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f076b7fe640 1 --2- 172.21.15.76:0/580537552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d07380 0x562645d07760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f076bfff640 1 --2- 172.21.15.76:0/580537552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d108b0 0x562645d12d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f076affd640 1 -- 172.21.15.76:0/580537552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d07380 msgr2=0x562645d07760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f076affd640 1 --2- 172.21.15.76:0/580537552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d07380 0x562645d07760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f076affd640 1 -- 172.21.15.76:0/580537552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d108b0 msgr2=0x562645d12d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f076affd640 1 --2- 172.21.15.76:0/580537552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d108b0 0x562645d12d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f076affd640 1 -- 172.21.15.76:0/580537552 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562645c310b0 con 0x562645d07d30 2024-03-12T06:02:10.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.007+0000 7f076bfff640 1 --2- 172.21.15.76:0/580537552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d108b0 0x562645d12d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:10.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.008+0000 7f076bfff640 1 --2- 172.21.15.76:0/3490283127 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f07540a1ba0 0x7f07540a4060 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0764005cc0 tx=0x7f076400a810 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:10.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.008+0000 7f076affd640 1 --2- 172.21.15.76:0/580537552 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d07d30 0x562645d102e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f0760009900 tx=0x7f076002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780e8444a1e34e57 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:10.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.008+0000 7f0751ffb640 1 -- 172.21.15.76:0/580537552 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0760004980 con 0x562645d07d30 2024-03-12T06:02:10.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.008+0000 7f0751ffb640 1 -- 172.21.15.76:0/580537552 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0760004ae0 con 0x562645d07d30 2024-03-12T06:02:10.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.008+0000 7f0751ffb640 1 -- 172.21.15.76:0/580537552 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07600052f0 con 0x562645d07d30 2024-03-12T06:02:10.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.008+0000 7f0774c00740 1 -- 172.21.15.76:0/580537552 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d07d30 msgr2=0x562645d102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.008+0000 7f0774c00740 1 --2- 172.21.15.76:0/580537552 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d07d30 0x562645d102e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f0760009900 tx=0x7f076002c7e0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.008+0000 7f0774c00740 1 -- 172.21.15.76:0/580537552 shutdown_connections 2024-03-12T06:02:10.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.008+0000 7f0774c00740 1 --2- 172.21.15.76:0/580537552 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d108b0 0x562645d12d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.008+0000 7f0774c00740 1 --2- 172.21.15.76:0/580537552 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d07d30 0x562645d102e0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.008+0000 7f0774c00740 1 --2- 172.21.15.76:0/580537552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d07380 0x562645d07760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.008+0000 7f0774c00740 1 -- 172.21.15.76:0/580537552 >> 172.21.15.76:0/580537552 conn(0x562645d01840 msgr2=0x562645d01c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:10.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.009+0000 7f0774c00740 1 -- 172.21.15.76:0/580537552 shutdown_connections 2024-03-12T06:02:10.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.009+0000 7f0774c00740 1 -- 172.21.15.76:0/580537552 wait complete. 2024-03-12T06:02:10.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.009+0000 7f0774c00740 1 -- start start 2024-03-12T06:02:10.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.009+0000 7f0774c00740 1 Processor -- start 2024-03-12T06:02:10.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.009+0000 7f0774c00740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07380 0x562645d0d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.009+0000 7f0774c00740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d07d30 0x562645d0dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.009+0000 7f0774c00740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d108b0 0x562645d0e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.009+0000 7f0774c00740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562645c13f40 con 0x562645d108b0 2024-03-12T06:02:10.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.009+0000 7f0774c00740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626459424b0 con 0x562645d07380 2024-03-12T06:02:10.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.009+0000 7f0774c00740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562645bf4670 con 0x562645d07d30 2024-03-12T06:02:10.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.009+0000 7f0774c00740 10 client.0 _ll_register_callbacks cb 0x562645c61f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:10.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f076bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d108b0 0x562645d0e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f076bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d108b0 0x562645d0e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49768/0 (socket says 172.21.15.76:49768) 2024-03-12T06:02:10.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f076bfff640 1 -- 172.21.15.76:0/2582148663 learned_addr learned my addr 172.21.15.76:0/2582148663 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:10.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f076b7fe640 1 --2- 172.21.15.76:0/2582148663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07380 0x562645d0d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f076affd640 1 --2- 172.21.15.76:0/2582148663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d07d30 0x562645d0dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f076bfff640 1 -- 172.21.15.76:0/2582148663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d07d30 msgr2=0x562645d0dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f076bfff640 1 --2- 172.21.15.76:0/2582148663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d07d30 0x562645d0dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f076bfff640 1 -- 172.21.15.76:0/2582148663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07380 msgr2=0x562645d0d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f076bfff640 1 --2- 172.21.15.76:0/2582148663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07380 0x562645d0d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f076bfff640 1 -- 172.21.15.76:0/2582148663 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076c10e950 con 0x562645d108b0 2024-03-12T06:02:10.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f076b7fe640 1 --2- 172.21.15.76:0/2582148663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07380 0x562645d0d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:02:10.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f076bfff640 1 --2- 172.21.15.76:0/2582148663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d108b0 0x562645d0e220 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f0764012d40 tx=0x7f0764004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d695a9cca0b842 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:10.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f074affd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:10.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f074affd640 1 -- 172.21.15.76:0/2582148663 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076401f450 con 0x562645d108b0 2024-03-12T06:02:10.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f074affd640 1 -- 172.21.15.76:0/2582148663 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f076401f5b0 con 0x562645d108b0 2024-03-12T06:02:10.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f0774c00740 1 -- 172.21.15.76:0/2582148663 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562645c48f40 con 0x562645d108b0 2024-03-12T06:02:10.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f074affd640 1 -- 172.21.15.76:0/2582148663 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f076401f770 con 0x562645d108b0 2024-03-12T06:02:10.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f0774c00740 10 client.16983 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:10.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.010+0000 7f0774c00740 1 -- 172.21.15.76:0/2582148663 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562645c6d320 con 0x562645d108b0 2024-03-12T06:02:10.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.011+0000 7f074affd640 1 -- 172.21.15.76:0/2582148663 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x562645c48f40 con 0x562645d108b0 2024-03-12T06:02:10.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.011+0000 7f074affd640 1 -- 172.21.15.76:0/2582148663 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0764027980 con 0x562645d108b0 2024-03-12T06:02:10.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.011+0000 7f0774c00740 10 client.16983 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:10.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.011+0000 7f0774c00740 10 client.16983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:10.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.011+0000 7f0774c00740 20 client.16983 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:10.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.011+0000 7f0774c00740 1 --2- 172.21.15.76:0/2582148663 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562645d20bd0 0x562645d22ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.011+0000 7f0774c00740 4 client.16983 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:02:10.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.011+0000 7f0774c00740 1 -- 172.21.15.76:0/2582148663 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562645c65310 con 0x562645d20bd0 2024-03-12T06:02:10.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.011+0000 7f076b7fe640 1 --2- 172.21.15.76:0/2582148663 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562645d20bd0 0x562645d22ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.011+0000 7f074affd640 10 client.16983 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:10.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.011+0000 7f076b7fe640 1 --2- 172.21.15.76:0/2582148663 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562645d20bd0 0x562645d22ff0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b6105b6aff9c0a4 server_cookie=c6c9f7c5502eca57 in_seq=0 out_seq=0 2024-03-12T06:02:10.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.012+0000 7f074affd640 1 -- 172.21.15.76:0/2582148663 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562645c65310 con 0x562645d20bd0 2024-03-12T06:02:10.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.012+0000 7f074affd640 10 client.16983 handle_command_reply: tid=0 2024-03-12T06:02:10.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.012+0000 7f0774c00740 1 client.16983 shutdown 2024-03-12T06:02:10.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.012+0000 7f0774c00740 1 -- 172.21.15.76:0/2582148663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d108b0 msgr2=0x562645d0e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.012+0000 7f0774c00740 1 --2- 172.21.15.76:0/2582148663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d108b0 0x562645d0e220 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f0764012d40 tx=0x7f0764004ab0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.012+0000 7f0774c00740 1 -- 172.21.15.76:0/2582148663 shutdown_connections 2024-03-12T06:02:10.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.012+0000 7f0774c00740 1 --2- 172.21.15.76:0/2582148663 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562645d20bd0 0x562645d22ff0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.012+0000 7f0774c00740 1 --2- 172.21.15.76:0/2582148663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d108b0 0x562645d0e220 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.012+0000 7f0774c00740 1 --2- 172.21.15.76:0/2582148663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d07d30 0x562645d0dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.012+0000 7f0774c00740 1 --2- 172.21.15.76:0/2582148663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07380 0x562645d0d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.012+0000 7f074affd640 0 client.16983 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:10.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.012+0000 7f0774c00740 1 -- 172.21.15.76:0/2582148663 >> 172.21.15.76:0/2582148663 conn(0x562645d01840 msgr2=0x562645d037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:10.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.012+0000 7f0774c00740 1 -- 172.21.15.76:0/2582148663 shutdown_connections 2024-03-12T06:02:10.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.012+0000 7f0774c00740 1 -- 172.21.15.76:0/2582148663 wait complete. 2024-03-12T06:02:10.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.013+0000 7f0774c00740 20 client.16983 trim_cache size 0 max 16384 2024-03-12T06:02:10.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f0774c00740 1 Processor -- start 2024-03-12T06:02:10.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f0774c00740 1 -- start start 2024-03-12T06:02:10.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f0774c00740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d07380 0x562645d0fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f0774c00740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07d30 0x562645d0e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f0774c00740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d108b0 0x562645d0aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f0774c00740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562645d20950 con 0x562645d07380 2024-03-12T06:02:10.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f0774c00740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562645d01670 con 0x562645d07d30 2024-03-12T06:02:10.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f0774c00740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562645d085c0 con 0x562645d108b0 2024-03-12T06:02:10.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f076b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d07380 0x562645d0fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f076b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d07380 0x562645d0fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49782/0 (socket says 172.21.15.76:49782) 2024-03-12T06:02:10.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f076b7fe640 1 -- 172.21.15.76:0/2517606800 learned_addr learned my addr 172.21.15.76:0/2517606800 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:10.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f076bfff640 1 --2- 172.21.15.76:0/2517606800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d108b0 0x562645d0aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f076affd640 1 --2- 172.21.15.76:0/2517606800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07d30 0x562645d0e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f076b7fe640 1 -- 172.21.15.76:0/2517606800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d108b0 msgr2=0x562645d0aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f076b7fe640 1 --2- 172.21.15.76:0/2517606800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d108b0 0x562645d0aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f076b7fe640 1 -- 172.21.15.76:0/2517606800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07d30 msgr2=0x562645d0e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f076b7fe640 1 --2- 172.21.15.76:0/2517606800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07d30 0x562645d0e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f076b7fe640 1 -- 172.21.15.76:0/2517606800 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562645c09c00 con 0x562645d07380 2024-03-12T06:02:10.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f076bfff640 1 --2- 172.21.15.76:0/2517606800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d108b0 0x562645d0aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:02:10.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.028+0000 7f076b7fe640 1 --2- 172.21.15.76:0/2517606800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d07380 0x562645d0fed0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f075c096530 tx=0x7f075c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d982588c9546cb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:10.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.029+0000 7f07517fa640 1 -- 172.21.15.76:0/2517606800 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c07ea90 con 0x562645d07380 2024-03-12T06:02:10.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.029+0000 7f07517fa640 1 -- 172.21.15.76:0/2517606800 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f075c0a2870 con 0x562645d07380 2024-03-12T06:02:10.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.029+0000 7f07517fa640 1 -- 172.21.15.76:0/2517606800 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c099780 con 0x562645d07380 2024-03-12T06:02:10.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.029+0000 7f0774c00740 1 -- 172.21.15.76:0/2517606800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d07380 msgr2=0x562645d0fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.029+0000 7f0774c00740 1 --2- 172.21.15.76:0/2517606800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d07380 0x562645d0fed0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f075c096530 tx=0x7f075c000bc0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.029+0000 7f0774c00740 1 -- 172.21.15.76:0/2517606800 shutdown_connections 2024-03-12T06:02:10.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.029+0000 7f0774c00740 1 --2- 172.21.15.76:0/2517606800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d108b0 0x562645d0aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.029+0000 7f0774c00740 1 --2- 172.21.15.76:0/2517606800 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07d30 0x562645d0e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.029+0000 7f0774c00740 1 --2- 172.21.15.76:0/2517606800 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d07380 0x562645d0fed0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.029+0000 7f0774c00740 1 -- 172.21.15.76:0/2517606800 >> 172.21.15.76:0/2517606800 conn(0x562645d01840 msgr2=0x562645d0c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:10.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.029+0000 7f0774c00740 1 -- 172.21.15.76:0/2517606800 shutdown_connections 2024-03-12T06:02:10.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.029+0000 7f0774c00740 1 -- 172.21.15.76:0/2517606800 wait complete. 2024-03-12T06:02:10.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.029+0000 7f0774c00740 1 -- start start 2024-03-12T06:02:10.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f0774c00740 1 Processor -- start 2024-03-12T06:02:10.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f0774c00740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07380 0x562645d16480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f0774c00740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d07d30 0x562645d169c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f0774c00740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d108b0 0x562645d16f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f0774c00740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562645d772e0 con 0x562645d108b0 2024-03-12T06:02:10.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f0774c00740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562645d77450 con 0x562645d07380 2024-03-12T06:02:10.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f0774c00740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562645d77610 con 0x562645d07d30 2024-03-12T06:02:10.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f0774c00740 10 client.0 _ll_register_callbacks cb 0x7f0760005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:10.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f076bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d108b0 0x562645d16f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f076bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d108b0 0x562645d16f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49786/0 (socket says 172.21.15.76:49786) 2024-03-12T06:02:10.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f076bfff640 1 -- 172.21.15.76:0/1494502745 learned_addr learned my addr 172.21.15.76:0/1494502745 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:10.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f076b7fe640 1 --2- 172.21.15.76:0/1494502745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07380 0x562645d16480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f076bfff640 1 -- 172.21.15.76:0/1494502745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d07d30 msgr2=0x562645d169c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f076bfff640 1 --2- 172.21.15.76:0/1494502745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d07d30 0x562645d169c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f076bfff640 1 -- 172.21.15.76:0/1494502745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07380 msgr2=0x562645d16480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f076bfff640 1 --2- 172.21.15.76:0/1494502745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07380 0x562645d16480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f076bfff640 1 -- 172.21.15.76:0/1494502745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076c1a6790 con 0x562645d108b0 2024-03-12T06:02:10.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f076b7fe640 1 --2- 172.21.15.76:0/1494502745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07380 0x562645d16480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:10.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f076bfff640 1 --2- 172.21.15.76:0/1494502745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d108b0 0x562645d16f40 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f076400a3d0 tx=0x7f0764011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685928a6c994a6e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:10.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.030+0000 7f074affd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:10.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f074affd640 1 -- 172.21.15.76:0/1494502745 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764004d20 con 0x562645d108b0 2024-03-12T06:02:10.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f074affd640 1 -- 172.21.15.76:0/1494502745 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f07640273b0 con 0x562645d108b0 2024-03-12T06:02:10.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f0774c00740 1 -- 172.21.15.76:0/1494502745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562645c94900 con 0x562645d108b0 2024-03-12T06:02:10.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f074affd640 1 -- 172.21.15.76:0/1494502745 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0764004e80 con 0x562645d108b0 2024-03-12T06:02:10.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f0774c00740 10 client.16989 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:10.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f0774c00740 1 -- 172.21.15.76:0/1494502745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562645c2f910 con 0x562645d108b0 2024-03-12T06:02:10.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f074affd640 1 -- 172.21.15.76:0/1494502745 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x562645c94900 con 0x562645d108b0 2024-03-12T06:02:10.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f074affd640 1 -- 172.21.15.76:0/1494502745 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f076401ebc0 con 0x562645d108b0 2024-03-12T06:02:10.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f0774c00740 10 client.16989 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:10.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f0774c00740 10 client.16989 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:10.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f0774c00740 20 client.16989 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:10.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f0774c00740 1 --2- 172.21.15.76:0/1494502745 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562645d78cb0 0x562645d7b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f0774c00740 4 client.16989 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T06:02:10.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f0774c00740 1 -- 172.21.15.76:0/1494502745 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56264596e860 con 0x562645d78cb0 2024-03-12T06:02:10.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.031+0000 7f076b7fe640 1 --2- 172.21.15.76:0/1494502745 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562645d78cb0 0x562645d7b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.032+0000 7f076b7fe640 1 --2- 172.21.15.76:0/1494502745 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562645d78cb0 0x562645d7b0f0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad1556b3738ba91e server_cookie=2b497d421e57aac7 in_seq=0 out_seq=0 2024-03-12T06:02:10.052 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.032+0000 7f074affd640 10 client.16989 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:10.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.033+0000 7f074affd640 1 -- 172.21.15.76:0/1494502745 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56264596e860 con 0x562645d78cb0 2024-03-12T06:02:10.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.033+0000 7f074affd640 10 client.16989 handle_command_reply: tid=0 2024-03-12T06:02:10.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.033+0000 7f0774c00740 1 client.16989 shutdown 2024-03-12T06:02:10.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.033+0000 7f0774c00740 1 -- 172.21.15.76:0/1494502745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d108b0 msgr2=0x562645d16f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.033+0000 7f0774c00740 1 --2- 172.21.15.76:0/1494502745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d108b0 0x562645d16f40 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f076400a3d0 tx=0x7f0764011cb0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.033+0000 7f0774c00740 1 -- 172.21.15.76:0/1494502745 shutdown_connections 2024-03-12T06:02:10.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.033+0000 7f0774c00740 1 --2- 172.21.15.76:0/1494502745 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x562645d78cb0 0x562645d7b0f0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.033+0000 7f0774c00740 1 --2- 172.21.15.76:0/1494502745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562645d108b0 0x562645d16f40 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.033+0000 7f0774c00740 1 --2- 172.21.15.76:0/1494502745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562645d07d30 0x562645d169c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.033+0000 7f0774c00740 1 --2- 172.21.15.76:0/1494502745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562645d07380 0x562645d16480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.033+0000 7f0774c00740 1 -- 172.21.15.76:0/1494502745 >> 172.21.15.76:0/1494502745 conn(0x562645d01840 msgr2=0x562645d02d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:10.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.033+0000 7f074affd640 0 client.16989 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:10.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f0774c00740 1 -- 172.21.15.76:0/1494502745 shutdown_connections 2024-03-12T06:02:10.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f0774c00740 1 -- 172.21.15.76:0/1494502745 wait complete. 2024-03-12T06:02:10.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f0774c00740 20 client.16989 trim_cache size 0 max 16384 2024-03-12T06:02:10.056 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:02:10.056 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T06:02:10.056 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "1b10910a-35cd-4e6b-9069-6ed287e6abf9", 2024-03-12T06:02:10.057 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T06:02:10.057 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:02:10.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f077202e640 1 -- 172.21.15.76:0/3490283127 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f07540a1ba0 msgr2=0x7f07540a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f077202e640 1 --2- 172.21.15.76:0/3490283127 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f07540a1ba0 0x7f07540a4060 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0764005cc0 tx=0x7f076400a810 comp rx=0 tx=0).stop 2024-03-12T06:02:10.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f077202e640 1 -- 172.21.15.76:0/3490283127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f076c1023f0 msgr2=0x7f076c100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f077202e640 1 --2- 172.21.15.76:0/3490283127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f076c1023f0 0x7f076c100040 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f075c00d950 tx=0x7f075c00de10 comp rx=0 tx=0).stop 2024-03-12T06:02:10.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f077202e640 1 -- 172.21.15.76:0/3490283127 shutdown_connections 2024-03-12T06:02:10.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f077202e640 1 --2- 172.21.15.76:0/3490283127 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f07540a1ba0 0x7f07540a4060 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f077202e640 1 --2- 172.21.15.76:0/3490283127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f076c10b860 0x7f076c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f077202e640 1 --2- 172.21.15.76:0/3490283127 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f076c102d10 0x7f076c100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f077202e640 1 --2- 172.21.15.76:0/3490283127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f076c1023f0 0x7f076c100040 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f077202e640 1 -- 172.21.15.76:0/3490283127 >> 172.21.15.76:0/3490283127 conn(0x7f076c0771f0 msgr2=0x7f076c0fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:10.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f077202e640 1 -- 172.21.15.76:0/3490283127 shutdown_connections 2024-03-12T06:02:10.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.034+0000 7f077202e640 1 -- 172.21.15.76:0/3490283127 wait complete. 2024-03-12T06:02:10.060 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:02:10.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:02:10.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.184+0000 7f8e34ca3640 1 -- 172.21.15.76:0/810453928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e30100260 msgr2=0x7f8e30100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.184+0000 7f8e34ca3640 1 --2- 172.21.15.76:0/810453928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e30100260 0x7f8e30100640 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8e2000d990 tx=0x7f8e2002b8d0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.184+0000 7f8e34ca3640 1 -- 172.21.15.76:0/810453928 shutdown_connections 2024-03-12T06:02:10.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.184+0000 7f8e34ca3640 1 --2- 172.21.15.76:0/810453928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e30105330 0x7f8e3010cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.184+0000 7f8e34ca3640 1 --2- 172.21.15.76:0/810453928 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e30100c10 0x7f8e30104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.184+0000 7f8e34ca3640 1 --2- 172.21.15.76:0/810453928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e30100260 0x7f8e30100640 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.184+0000 7f8e34ca3640 1 -- 172.21.15.76:0/810453928 >> 172.21.15.76:0/810453928 conn(0x7f8e30077180 msgr2=0x7f8e30077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:10.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.184+0000 7f8e34ca3640 1 -- 172.21.15.76:0/810453928 shutdown_connections 2024-03-12T06:02:10.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.184+0000 7f8e34ca3640 1 -- 172.21.15.76:0/810453928 wait complete. 2024-03-12T06:02:10.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e34ca3640 1 Processor -- start 2024-03-12T06:02:10.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e34ca3640 1 -- start start 2024-03-12T06:02:10.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e34ca3640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e30100260 0x7f8e30195570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e34ca3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e30105330 0x7f8e30195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e34ca3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e30199e30 0x7f8e3019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e34ca3640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3019a820 con 0x7f8e30105330 2024-03-12T06:02:10.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e34ca3640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3019a990 con 0x7f8e30100260 2024-03-12T06:02:10.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e34ca3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3019ab00 con 0x7f8e30199e30 2024-03-12T06:02:10.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e2dd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e30105330 0x7f8e30195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e2dd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e30105330 0x7f8e30195ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49796/0 (socket says 172.21.15.76:49796) 2024-03-12T06:02:10.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e2dd9b640 1 -- 172.21.15.76:0/1744177102 learned_addr learned my addr 172.21.15.76:0/1744177102 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:10.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e2e59c640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e30100260 0x7f8e30195570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e2ed9d640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e30199e30 0x7f8e3019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e2dd9b640 1 -- 172.21.15.76:0/1744177102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e30199e30 msgr2=0x7f8e3019a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e30199e30 0x7f8e3019a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e2dd9b640 1 -- 172.21.15.76:0/1744177102 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e30100260 msgr2=0x7f8e30195570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e30100260 0x7f8e30195570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e2dd9b640 1 -- 172.21.15.76:0/1744177102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2000d590 con 0x7f8e30105330 2024-03-12T06:02:10.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.185+0000 7f8e2ed9d640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e30199e30 0x7f8e3019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:02:10.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.186+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e30105330 0x7f8e30195ab0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f8e2400e9c0 tx=0x7f8e2400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:10.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.186+0000 7f8e1f7fe640 1 -- 172.21.15.76:0/1744177102 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e24009700 con 0x7f8e30105330 2024-03-12T06:02:10.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.186+0000 7f8e1f7fe640 1 -- 172.21.15.76:0/1744177102 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e2400cd80 con 0x7f8e30105330 2024-03-12T06:02:10.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.186+0000 7f8e34ca3640 1 -- 172.21.15.76:0/1744177102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e301a20e0 con 0x7f8e30105330 2024-03-12T06:02:10.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.186+0000 7f8e1f7fe640 1 -- 172.21.15.76:0/1744177102 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e24010430 con 0x7f8e30105330 2024-03-12T06:02:10.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.186+0000 7f8e34ca3640 1 -- 172.21.15.76:0/1744177102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e301a2590 con 0x7f8e30105330 2024-03-12T06:02:10.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.189+0000 7f8e1f7fe640 1 -- 172.21.15.76:0/1744177102 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8e24005000 con 0x7f8e30105330 2024-03-12T06:02:10.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.189+0000 7f8e37870740 1 Processor -- start 2024-03-12T06:02:10.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e37870740 1 -- start start 2024-03-12T06:02:10.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e37870740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af640690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e37870740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af649270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e37870740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 0x5621af64bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e37870740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621af5af280 con 0x5621af6402b0 2024-03-12T06:02:10.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e37870740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621af5ac7b0 con 0x5621af640bd0 2024-03-12T06:02:10.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e37870740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621af5a7f90 con 0x5621af6497b0 2024-03-12T06:02:10.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e2e59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af649270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e2ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af640690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e2e59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af649270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:46178/0 (socket says 172.21.15.76:46178) 2024-03-12T06:02:10.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e2e59c640 1 -- 172.21.15.76:0/1564486351 learned_addr learned my addr 172.21.15.76:0/1564486351 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:10.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e1f7fe640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8e180a1ba0 0x7f8e180a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1564486351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 0x5621af64bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e2ed9d640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8e180a1ba0 0x7f8e180a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e1f7fe640 1 -- 172.21.15.76:0/1744177102 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f8e24014070 con 0x7f8e30105330 2024-03-12T06:02:10.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.190+0000 7f8e2dd9b640 1 -- 172.21.15.76:0/1564486351 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 msgr2=0x5621af649270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.191+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1564486351 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af649270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.191+0000 7f8e2dd9b640 1 -- 172.21.15.76:0/1564486351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 msgr2=0x5621af640690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.191+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1564486351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af640690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.191+0000 7f8e2dd9b640 1 -- 172.21.15.76:0/1564486351 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621af566570 con 0x5621af6497b0 2024-03-12T06:02:10.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.191+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1564486351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 0x5621af64bba0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8e240cb3e0 tx=0x7f8e24005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8e6be2ee60a519 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:10.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.191+0000 7f8e2ed9d640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8e180a1ba0 0x7f8e180a4060 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8e28005fd0 tx=0x7f8e28004ab0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:10.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.191+0000 7f8e1cff9640 1 -- 172.21.15.76:0/1564486351 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e240d4040 con 0x5621af6497b0 2024-03-12T06:02:10.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.192+0000 7f8e1cff9640 1 -- 172.21.15.76:0/1564486351 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e240a3060 con 0x5621af6497b0 2024-03-12T06:02:10.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.192+0000 7f8e1cff9640 1 -- 172.21.15.76:0/1564486351 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e240052b0 con 0x5621af6497b0 2024-03-12T06:02:10.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 -- 172.21.15.76:0/1564486351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 msgr2=0x5621af64bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 --2- 172.21.15.76:0/1564486351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 0x5621af64bba0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8e240cb3e0 tx=0x7f8e24005880 comp rx=0 tx=0).stop 2024-03-12T06:02:10.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 -- 172.21.15.76:0/1564486351 shutdown_connections 2024-03-12T06:02:10.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 --2- 172.21.15.76:0/1564486351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 0x5621af64bba0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 --2- 172.21.15.76:0/1564486351 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af649270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 --2- 172.21.15.76:0/1564486351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af640690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 -- 172.21.15.76:0/1564486351 >> 172.21.15.76:0/1564486351 conn(0x5621af63a990 msgr2=0x5621af63ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:10.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 -- 172.21.15.76:0/1564486351 shutdown_connections 2024-03-12T06:02:10.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 -- 172.21.15.76:0/1564486351 wait complete. 2024-03-12T06:02:10.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 -- start start 2024-03-12T06:02:10.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 Processor -- start 2024-03-12T06:02:10.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6402b0 0x5621af646620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af646b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6497b0 0x5621af6470a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621af27b430 con 0x5621af6497b0 2024-03-12T06:02:10.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621af237e00 con 0x5621af640bd0 2024-03-12T06:02:10.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621af52cfd0 con 0x5621af6402b0 2024-03-12T06:02:10.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.193+0000 7f8e37870740 10 client.0 _ll_register_callbacks cb 0x5621af582fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:10.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e2e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6402b0 0x5621af646620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e2e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6402b0 0x5621af646620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.76:47436/0 (socket says 172.21.15.76:47436) 2024-03-12T06:02:10.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e2e59c640 1 -- 172.21.15.76:0/1730605967 learned_addr learned my addr 172.21.15.76:0/1730605967 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:10.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1730605967 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af646b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e2ed9d640 1 --2- 172.21.15.76:0/1730605967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6497b0 0x5621af6470a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e2e59c640 1 -- 172.21.15.76:0/1730605967 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 msgr2=0x5621af646b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e2e59c640 1 --2- 172.21.15.76:0/1730605967 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af646b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e2e59c640 1 -- 172.21.15.76:0/1730605967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6497b0 msgr2=0x5621af6470a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e2e59c640 1 --2- 172.21.15.76:0/1730605967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6497b0 0x5621af6470a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e2e59c640 1 -- 172.21.15.76:0/1730605967 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e301a2590 con 0x5621af6402b0 2024-03-12T06:02:10.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1730605967 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af646b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:02:10.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e2e59c640 1 --2- 172.21.15.76:0/1730605967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6402b0 0x5621af646620 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8e20005e00 tx=0x7f8e200306f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20525e947e35b73b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:10.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-03-12T06:02:10.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e15ffb640 1 -- 172.21.15.76:0/1730605967 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e20041040 con 0x5621af6402b0 2024-03-12T06:02:10.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.194+0000 7f8e15ffb640 1 -- 172.21.15.76:0/1730605967 <== mon.1 v2:172.21.15.88:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f8e20004450 con 0x5621af6402b0 2024-03-12T06:02:10.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.195+0000 7f8e37870740 1 -- 172.21.15.76:0/1730605967 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5621af2a77e0 con 0x5621af6402b0 2024-03-12T06:02:10.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.195+0000 7f8e15ffb640 1 -- 172.21.15.76:0/1730605967 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e200307e0 con 0x5621af6402b0 2024-03-12T06:02:10.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.195+0000 7f8e37870740 10 client.26086 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:10.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.195+0000 7f8e37870740 1 -- 172.21.15.76:0/1730605967 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5621af5a6610 con 0x5621af6402b0 2024-03-12T06:02:10.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.195+0000 7f8e15ffb640 1 -- 172.21.15.76:0/1730605967 <== mon.1 v2:172.21.15.88:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5621af2a77e0 con 0x5621af6402b0 2024-03-12T06:02:10.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.195+0000 7f8e15ffb640 1 -- 172.21.15.76:0/1730605967 <== mon.1 v2:172.21.15.88:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8e2004d070 con 0x5621af6402b0 2024-03-12T06:02:10.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.195+0000 7f8e37870740 10 client.26086 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:10.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.195+0000 7f8e37870740 10 client.26086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:10.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.195+0000 7f8e37870740 20 client.26086 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:10.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.195+0000 7f8e37870740 1 --2- 172.21.15.76:0/1730605967 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621af659990 0x5621af65bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.195+0000 7f8e37870740 4 client.26086 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:02:10.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.195+0000 7f8e37870740 1 -- 172.21.15.76:0/1730605967 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5621af2e4720 con 0x5621af659990 2024-03-12T06:02:10.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.195+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1730605967 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621af659990 0x5621af65bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.196+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1730605967 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621af659990 0x5621af65bd80 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32c8bd0c66c4c281 server_cookie=5d0e6d3fc814ba8d in_seq=0 out_seq=0 2024-03-12T06:02:10.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.196+0000 7f8e15ffb640 10 client.26086 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:10.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e15ffb640 1 -- 172.21.15.76:0/1730605967 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5621af2e4720 con 0x5621af659990 2024-03-12T06:02:10.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e15ffb640 10 client.26086 handle_command_reply: tid=0 2024-03-12T06:02:10.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e37870740 1 client.26086 shutdown 2024-03-12T06:02:10.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e37870740 1 -- 172.21.15.76:0/1730605967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6402b0 msgr2=0x5621af646620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e37870740 1 --2- 172.21.15.76:0/1730605967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6402b0 0x5621af646620 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8e20005e00 tx=0x7f8e200306f0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e37870740 1 -- 172.21.15.76:0/1730605967 shutdown_connections 2024-03-12T06:02:10.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e37870740 1 --2- 172.21.15.76:0/1730605967 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621af659990 0x5621af65bd80 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e37870740 1 --2- 172.21.15.76:0/1730605967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6497b0 0x5621af6470a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e37870740 1 --2- 172.21.15.76:0/1730605967 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af646b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e15ffb640 0 client.26086 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:10.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e37870740 1 --2- 172.21.15.76:0/1730605967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6402b0 0x5621af646620 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e37870740 1 -- 172.21.15.76:0/1730605967 >> 172.21.15.76:0/1730605967 conn(0x5621af63a990 msgr2=0x5621af63c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:10.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e37870740 1 -- 172.21.15.76:0/1730605967 shutdown_connections 2024-03-12T06:02:10.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e37870740 1 -- 172.21.15.76:0/1730605967 wait complete. 2024-03-12T06:02:10.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.197+0000 7f8e37870740 20 client.26086 trim_cache size 0 max 16384 2024-03-12T06:02:10.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.217+0000 7f8e37870740 1 Processor -- start 2024-03-12T06:02:10.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e37870740 1 -- start start 2024-03-12T06:02:10.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e37870740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af648d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e37870740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af646ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e37870740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 0x5621af643880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e37870740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621af63fcf0 con 0x5621af6402b0 2024-03-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e37870740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621af655510 con 0x5621af640bd0 2024-03-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e37870740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621af63bc40 con 0x5621af6497b0 2024-03-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e2e59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af648d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e2e59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af648d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49818/0 (socket says 172.21.15.76:49818) 2024-03-12T06:02:10.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e2e59c640 1 -- 172.21.15.76:0/2144143864 learned_addr learned my addr 172.21.15.76:0/2144143864 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:10.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e2ed9d640 1 --2- 172.21.15.76:0/2144143864 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 0x5621af643880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/2144143864 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af646ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e2e59c640 1 -- 172.21.15.76:0/2144143864 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 msgr2=0x5621af643880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e2e59c640 1 --2- 172.21.15.76:0/2144143864 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 0x5621af643880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e2e59c640 1 -- 172.21.15.76:0/2144143864 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 msgr2=0x5621af646ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e2e59c640 1 --2- 172.21.15.76:0/2144143864 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af646ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.218+0000 7f8e2e59c640 1 -- 172.21.15.76:0/2144143864 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621af542ac0 con 0x5621af6402b0 2024-03-12T06:02:10.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.219+0000 7f8e2e59c640 1 --2- 172.21.15.76:0/2144143864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af648d50 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8e20005940 tx=0x7f8e2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76474241a782aea6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:10.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.219+0000 7f8e17fff640 1 -- 172.21.15.76:0/2144143864 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2002bfd0 con 0x5621af6402b0 2024-03-12T06:02:10.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.219+0000 7f8e17fff640 1 -- 172.21.15.76:0/2144143864 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e20003730 con 0x5621af6402b0 2024-03-12T06:02:10.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.219+0000 7f8e17fff640 1 -- 172.21.15.76:0/2144143864 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e20030da0 con 0x5621af6402b0 2024-03-12T06:02:10.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.220+0000 7f8e37870740 1 -- 172.21.15.76:0/2144143864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 msgr2=0x5621af648d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.220+0000 7f8e37870740 1 --2- 172.21.15.76:0/2144143864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af648d50 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f8e20005940 tx=0x7f8e2002b8d0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.220+0000 7f8e37870740 1 -- 172.21.15.76:0/2144143864 shutdown_connections 2024-03-12T06:02:10.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.220+0000 7f8e37870740 1 --2- 172.21.15.76:0/2144143864 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 0x5621af643880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.220+0000 7f8e37870740 1 --2- 172.21.15.76:0/2144143864 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af646ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.220+0000 7f8e37870740 1 --2- 172.21.15.76:0/2144143864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af648d50 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.220+0000 7f8e37870740 1 -- 172.21.15.76:0/2144143864 >> 172.21.15.76:0/2144143864 conn(0x5621af63a990 msgr2=0x5621af641910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:10.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.220+0000 7f8e37870740 1 -- 172.21.15.76:0/2144143864 shutdown_connections 2024-03-12T06:02:10.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.220+0000 7f8e37870740 1 -- 172.21.15.76:0/2144143864 wait complete. 2024-03-12T06:02:10.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.220+0000 7f8e37870740 1 -- start start 2024-03-12T06:02:10.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.220+0000 7f8e37870740 1 Processor -- start 2024-03-12T06:02:10.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.220+0000 7f8e37870740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af6507d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e37870740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af64eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e37870740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 0x5621af64f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e37870740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621af64f620 con 0x5621af6402b0 2024-03-12T06:02:10.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e37870740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621af64f790 con 0x5621af640bd0 2024-03-12T06:02:10.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e37870740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621af64f950 con 0x5621af6497b0 2024-03-12T06:02:10.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e37870740 10 client.0 _ll_register_callbacks cb 0x7f8e2409a690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:10.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e2e59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af6507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e2e59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af6507d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49824/0 (socket says 172.21.15.76:49824) 2024-03-12T06:02:10.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e2e59c640 1 -- 172.21.15.76:0/1838554538 learned_addr learned my addr 172.21.15.76:0/1838554538 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:10.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1838554538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af64eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e2ed9d640 1 --2- 172.21.15.76:0/1838554538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 0x5621af64f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e2e59c640 1 -- 172.21.15.76:0/1838554538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 msgr2=0x5621af64f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e2e59c640 1 --2- 172.21.15.76:0/1838554538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 0x5621af64f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e2e59c640 1 -- 172.21.15.76:0/1838554538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 msgr2=0x5621af64eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:10.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e2e59c640 1 --2- 172.21.15.76:0/1838554538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af64eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e2e59c640 1 -- 172.21.15.76:0/1838554538 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5621af5a6610 con 0x5621af6402b0 2024-03-12T06:02:10.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.221+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1838554538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af64eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:02:10.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.222+0000 7f8e2e59c640 1 --2- 172.21.15.76:0/1838554538 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af6507d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8e20004650 tx=0x7f8e20004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a67cf7be12f1ca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:10.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.222+0000 7f8e15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:10.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.223+0000 7f8e15ffb640 1 -- 172.21.15.76:0/1838554538 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e20041070 con 0x5621af6402b0 2024-03-12T06:02:10.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.223+0000 7f8e15ffb640 1 -- 172.21.15.76:0/1838554538 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f8e20000e40 con 0x5621af6402b0 2024-03-12T06:02:10.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.223+0000 7f8e37870740 1 -- 172.21.15.76:0/1838554538 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e240a3060 con 0x5621af6402b0 2024-03-12T06:02:10.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.223+0000 7f8e15ffb640 1 -- 172.21.15.76:0/1838554538 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e20003d40 con 0x5621af6402b0 2024-03-12T06:02:10.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.223+0000 7f8e37870740 10 client.17007 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:10.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.223+0000 7f8e37870740 1 -- 172.21.15.76:0/1838554538 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5621af5686a0 con 0x5621af6402b0 2024-03-12T06:02:10.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.223+0000 7f8e15ffb640 1 -- 172.21.15.76:0/1838554538 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e2003f070 con 0x5621af6402b0 2024-03-12T06:02:10.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.223+0000 7f8e15ffb640 1 -- 172.21.15.76:0/1838554538 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8e2005d070 con 0x5621af6402b0 2024-03-12T06:02:10.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.224+0000 7f8e37870740 10 client.17007 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:10.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.224+0000 7f8e37870740 10 client.17007 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:10.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.224+0000 7f8e37870740 20 client.17007 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:10.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.224+0000 7f8e37870740 1 --2- 172.21.15.76:0/1838554538 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621af6b97d0 0x5621af6bbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:10.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.224+0000 7f8e37870740 4 client.17007 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:02:10.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.224+0000 7f8e37870740 1 -- 172.21.15.76:0/1838554538 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f8e240a3060 con 0x5621af6b97d0 2024-03-12T06:02:10.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.224+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1838554538 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621af6b97d0 0x5621af6bbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:10.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.224+0000 7f8e15ffb640 10 client.17007 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:10.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.224+0000 7f8e2dd9b640 1 --2- 172.21.15.76:0/1838554538 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621af6b97d0 0x5621af6bbbc0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d140d316a8c4ef22 server_cookie=b3214f27d3251e7f in_seq=0 out_seq=0 2024-03-12T06:02:10.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e15ffb640 1 -- 172.21.15.76:0/1838554538 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7f8e240a3060 con 0x5621af6b97d0 2024-03-12T06:02:10.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e15ffb640 10 client.17007 handle_command_reply: tid=0 2024-03-12T06:02:10.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e37870740 1 client.17007 shutdown 2024-03-12T06:02:10.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e37870740 1 -- 172.21.15.76:0/1838554538 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 msgr2=0x5621af6507d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e37870740 1 --2- 172.21.15.76:0/1838554538 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af6507d0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8e20004650 tx=0x7f8e20004590 comp rx=0 tx=0).stop 2024-03-12T06:02:10.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e37870740 1 -- 172.21.15.76:0/1838554538 shutdown_connections 2024-03-12T06:02:10.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e37870740 1 --2- 172.21.15.76:0/1838554538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621af6497b0 0x5621af64f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e37870740 1 --2- 172.21.15.76:0/1838554538 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5621af640bd0 0x5621af64eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e37870740 1 --2- 172.21.15.76:0/1838554538 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5621af6b97d0 0x5621af6bbbc0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e37870740 1 --2- 172.21.15.76:0/1838554538 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5621af6402b0 0x5621af6507d0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e15ffb640 0 client.17007 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:10.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e37870740 1 -- 172.21.15.76:0/1838554538 >> 172.21.15.76:0/1838554538 conn(0x5621af63a990 msgr2=0x5621af63be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:10.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e37870740 1 -- 172.21.15.76:0/1838554538 shutdown_connections 2024-03-12T06:02:10.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e37870740 1 -- 172.21.15.76:0/1838554538 wait complete. 2024-03-12T06:02:10.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.225+0000 7f8e37870740 20 client.17007 trim_cache size 0 max 16384 2024-03-12T06:02:10.240 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:02:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T06:02:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T06:02:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "1b10910a-35cd-4e6b-9069-6ed287e6abf9": { 2024-03-12T06:02:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T06:02:10.240 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "1b10910a-35cd-4e6b-9069-6ed287e6abf9", 2024-03-12T06:02:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T06:02:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:02:10.241 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:02:10.241 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:02:10.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.226+0000 7f8e34ca3640 1 -- 172.21.15.76:0/1744177102 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8e180a1ba0 msgr2=0x7f8e180a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.226+0000 7f8e34ca3640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8e180a1ba0 0x7f8e180a4060 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8e28005fd0 tx=0x7f8e28004ab0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.226+0000 7f8e34ca3640 1 -- 172.21.15.76:0/1744177102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e30105330 msgr2=0x7f8e30195ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:10.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.226+0000 7f8e34ca3640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e30105330 0x7f8e30195ab0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f8e2400e9c0 tx=0x7f8e2400ee80 comp rx=0 tx=0).stop 2024-03-12T06:02:10.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.226+0000 7f8e34ca3640 1 -- 172.21.15.76:0/1744177102 shutdown_connections 2024-03-12T06:02:10.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.226+0000 7f8e34ca3640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8e180a1ba0 0x7f8e180a4060 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.226+0000 7f8e34ca3640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8e30199e30 0x7f8e3019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.226+0000 7f8e34ca3640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e30105330 0x7f8e30195ab0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.226+0000 7f8e34ca3640 1 --2- 172.21.15.76:0/1744177102 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e30100260 0x7f8e30195570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:10.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.226+0000 7f8e34ca3640 1 -- 172.21.15.76:0/1744177102 >> 172.21.15.76:0/1744177102 conn(0x7f8e30077180 msgr2=0x7f8e300fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:10.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.226+0000 7f8e34ca3640 1 -- 172.21.15.76:0/1744177102 shutdown_connections 2024-03-12T06:02:10.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:10.226+0000 7f8e34ca3640 1 -- 172.21.15.76:0/1744177102 wait complete. 2024-03-12T06:02:10.244 INFO:tasks.cephfs.filesystem:scrub status for tag:1b10910a-35cd-4e6b-9069-6ed287e6abf9 - {'path': '/', 'tag': '1b10910a-35cd-4e6b-9069-6ed287e6abf9', 'options': 'recursive,force'} 2024-03-12T06:02:10.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:10 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T06:02:10.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:10 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:10.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:10 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:02:10.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:10 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:10.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:10 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T06:02:10.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:10 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:10.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:10 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:02:10.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:10 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:10.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:10 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T06:02:10.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:10 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:10.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:10 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:02:10.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:10 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:11.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:11 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:02:11.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:11 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:02:11.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:11 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:11.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:11 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:11.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:11 smithi082 ceph-mon[36382]: pgmap v196: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:02:11.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:11 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:02:11.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:11 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:02:11.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:11 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:11.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:11 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:11.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:11 smithi088 ceph-mon[36585]: pgmap v196: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:02:11.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:11 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:02:11.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:11 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:02:11.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:11 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:11.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:11 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:11.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:11 smithi076 ceph-mon[32095]: pgmap v196: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:02:14.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:14 smithi082 ceph-mon[36382]: pgmap v197: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-12T06:02:14.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:14 smithi088 ceph-mon[36585]: pgmap v197: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-12T06:02:14.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:14 smithi076 ceph-mon[32095]: pgmap v197: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-12T06:02:16.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:16 smithi082 ceph-mon[36382]: pgmap v198: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-12T06:02:16.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:16 smithi088 ceph-mon[36585]: pgmap v198: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-12T06:02:16.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:16 smithi076 ceph-mon[32095]: pgmap v198: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-12T06:02:17.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:17 smithi088 ceph-mon[36585]: osdmap e91: 12 total, 12 up, 12 in 2024-03-12T06:02:17.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:17 smithi088 ceph-mon[36585]: pgmap v200: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 114 B/s wr, 3 op/s 2024-03-12T06:02:17.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:17 smithi076 ceph-mon[32095]: osdmap e91: 12 total, 12 up, 12 in 2024-03-12T06:02:17.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:17 smithi076 ceph-mon[32095]: pgmap v200: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 114 B/s wr, 3 op/s 2024-03-12T06:02:17.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:17 smithi082 ceph-mon[36382]: osdmap e91: 12 total, 12 up, 12 in 2024-03-12T06:02:17.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:17 smithi082 ceph-mon[36382]: pgmap v200: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 114 B/s wr, 3 op/s 2024-03-12T06:02:20.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:20 smithi082 ceph-mon[36382]: pgmap v201: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:02:20.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:20 smithi088 ceph-mon[36585]: pgmap v201: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:02:20.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:20 smithi076 ceph-mon[32095]: pgmap v201: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:02:22.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:22 smithi082 ceph-mon[36382]: pgmap v202: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:02:22.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:22 smithi088 ceph-mon[36585]: pgmap v202: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:02:22.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:22 smithi076 ceph-mon[32095]: pgmap v202: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:02:23.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:23 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:02:23.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:23 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:02:23.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:23 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:02:23.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:23 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:02:23.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:23 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:02:23.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:23 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:02:23.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:23 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:02:23.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:23 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:02:23.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:23 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:02:24.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:24 smithi082 ceph-mon[36382]: pgmap v203: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:02:24.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:24 smithi088 ceph-mon[36585]: pgmap v203: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:02:24.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:24 smithi076 ceph-mon[32095]: pgmap v203: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:02:25.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:25 smithi082 ceph-mon[36382]: pgmap v204: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:02:25.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:25 smithi088 ceph-mon[36585]: pgmap v204: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:02:25.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:25 smithi076 ceph-mon[32095]: pgmap v204: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:02:28.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:28 smithi082 ceph-mon[36382]: pgmap v205: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-12T06:02:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:28 smithi088 ceph-mon[36585]: pgmap v205: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-12T06:02:28.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:28 smithi076 ceph-mon[32095]: pgmap v205: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-12T06:02:30.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:30 smithi082 ceph-mon[36382]: pgmap v206: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:02:30.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:30 smithi088 ceph-mon[36585]: pgmap v206: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:02:30.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:30 smithi076 ceph-mon[32095]: pgmap v206: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:02:32.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:32 smithi082 ceph-mon[36382]: pgmap v207: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:32.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:32 smithi088 ceph-mon[36585]: pgmap v207: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:32.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:32 smithi076 ceph-mon[32095]: pgmap v207: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:34.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:34 smithi082 ceph-mon[36382]: pgmap v208: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:02:34.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:34 smithi088 ceph-mon[36585]: pgmap v208: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:02:34.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:34 smithi076 ceph-mon[32095]: pgmap v208: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:02:35.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:35 smithi082 ceph-mon[36382]: pgmap v209: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:35.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:35 smithi088 ceph-mon[36585]: pgmap v209: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:35.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:35 smithi076 ceph-mon[32095]: pgmap v209: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:38.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:38 smithi082 ceph-mon[36382]: pgmap v210: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:38.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:38 smithi088 ceph-mon[36585]: pgmap v210: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:38.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:38 smithi076 ceph-mon[32095]: pgmap v210: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:40.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:02:40.245 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:02:40.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.355+0000 7f7781e83640 1 -- 172.21.15.76:0/1569657082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c10db20 msgr2=0x7f777c10ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.355+0000 7f7781e83640 1 --2- 172.21.15.76:0/1569657082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c10db20 0x7f777c10ff10 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7774009ca0 tx=0x7f777402c0f0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.355+0000 7f7781e83640 1 -- 172.21.15.76:0/1569657082 shutdown_connections 2024-03-12T06:02:40.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.355+0000 7f7781e83640 1 --2- 172.21.15.76:0/1569657082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c10db20 0x7f777c10ff10 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.355+0000 7f7781e83640 1 --2- 172.21.15.76:0/1569657082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c068df0 0x7f777c10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.355+0000 7f7781e83640 1 --2- 172.21.15.76:0/1569657082 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f777c0684d0 0x7f777c0688b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.355+0000 7f7781e83640 1 -- 172.21.15.76:0/1569657082 >> 172.21.15.76:0/1569657082 conn(0x7f777c077180 msgr2=0x7f777c077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.355+0000 7f7781e83640 1 -- 172.21.15.76:0/1569657082 shutdown_connections 2024-03-12T06:02:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.355+0000 7f7781e83640 1 -- 172.21.15.76:0/1569657082 wait complete. 2024-03-12T06:02:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.355+0000 7f7781e83640 1 Processor -- start 2024-03-12T06:02:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f7781e83640 1 -- start start 2024-03-12T06:02:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f7781e83640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f777c0684d0 0x7f777c19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f7781e83640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c068df0 0x7f777c19fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f7781e83640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c10db20 0x7f777c1996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f7781e83640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c199e50 con 0x7f777c10db20 2024-03-12T06:02:40.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f7781e83640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c199fc0 con 0x7f777c0684d0 2024-03-12T06:02:40.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f7781e83640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777c19a180 con 0x7f777c068df0 2024-03-12T06:02:40.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f777bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c10db20 0x7f777c1996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f777bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c10db20 0x7f777c1996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52004/0 (socket says 172.21.15.76:52004) 2024-03-12T06:02:40.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f777bfff640 1 -- 172.21.15.76:0/3584397281 learned_addr learned my addr 172.21.15.76:0/3584397281 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:40.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f777b7fe640 1 --2- 172.21.15.76:0/3584397281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f777c0684d0 0x7f777c19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f777affd640 1 --2- 172.21.15.76:0/3584397281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c068df0 0x7f777c19fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f777bfff640 1 -- 172.21.15.76:0/3584397281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c068df0 msgr2=0x7f777c19fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f777bfff640 1 --2- 172.21.15.76:0/3584397281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c068df0 0x7f777c19fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f777bfff640 1 -- 172.21.15.76:0/3584397281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f777c0684d0 msgr2=0x7f777c19f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f777bfff640 1 --2- 172.21.15.76:0/3584397281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f777c0684d0 0x7f777c19f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.356+0000 7f777bfff640 1 -- 172.21.15.76:0/3584397281 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7774009900 con 0x7f777c10db20 2024-03-12T06:02:40.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.357+0000 7f777bfff640 1 --2- 172.21.15.76:0/3584397281 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c10db20 0x7f777c1996c0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f7774004af0 tx=0x7f7774034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:40.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.357+0000 7f7778ff9640 1 -- 172.21.15.76:0/3584397281 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77740328c0 con 0x7f777c10db20 2024-03-12T06:02:40.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.357+0000 7f7781e83640 1 -- 172.21.15.76:0/3584397281 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f777c19a400 con 0x7f777c10db20 2024-03-12T06:02:40.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.357+0000 7f7778ff9640 1 -- 172.21.15.76:0/3584397281 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7774003e50 con 0x7f777c10db20 2024-03-12T06:02:40.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.357+0000 7f7778ff9640 1 -- 172.21.15.76:0/3584397281 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77740047f0 con 0x7f777c10db20 2024-03-12T06:02:40.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.357+0000 7f7781e83640 1 -- 172.21.15.76:0/3584397281 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777c1a6580 con 0x7f777c10db20 2024-03-12T06:02:40.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.359+0000 7f7778ff9640 1 -- 172.21.15.76:0/3584397281 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f777400a040 con 0x7f777c10db20 2024-03-12T06:02:40.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.359+0000 7f7778ff9640 1 --2- 172.21.15.76:0/3584397281 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f77640a1ba0 0x7f77640a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.359+0000 7f7784a4b740 1 Processor -- start 2024-03-12T06:02:40.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.359+0000 7f7784a4b740 1 -- start start 2024-03-12T06:02:40.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f777b7fe640 1 --2- 172.21.15.76:0/3584397281 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f77640a1ba0 0x7f77640a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f7778ff9640 1 -- 172.21.15.76:0/3584397281 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f77740f02e0 con 0x7f777c10db20 2024-03-12T06:02:40.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f7784a4b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a32b0 0x55e7380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f7784a4b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380a3bd0 0x55e7380ac270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f7784a4b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380ac7b0 0x55e7380aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f7784a4b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e738012280 con 0x55e7380a32b0 2024-03-12T06:02:40.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f7784a4b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73800f7b0 con 0x55e7380a3bd0 2024-03-12T06:02:40.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f7784a4b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73800af90 con 0x55e7380ac7b0 2024-03-12T06:02:40.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f777b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a32b0 0x55e7380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f777b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a32b0 0x55e7380a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52008/0 (socket says 172.21.15.76:52008) 2024-03-12T06:02:40.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f777b7fe640 1 -- 172.21.15.76:0/2334633554 learned_addr learned my addr 172.21.15.76:0/2334633554 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:40.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f777affd640 1 --2- 172.21.15.76:0/2334633554 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380a3bd0 0x55e7380ac270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f777bfff640 1 --2- 172.21.15.76:0/2334633554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380ac7b0 0x55e7380aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f777b7fe640 1 --2- 172.21.15.76:0/3584397281 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f77640a1ba0 0x7f77640a4060 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f776c013520 tx=0x7f776c013920 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:40.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f777b7fe640 1 -- 172.21.15.76:0/2334633554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380ac7b0 msgr2=0x55e7380aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f777b7fe640 1 --2- 172.21.15.76:0/2334633554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380ac7b0 0x55e7380aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f777b7fe640 1 -- 172.21.15.76:0/2334633554 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380a3bd0 msgr2=0x55e7380ac270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f777b7fe640 1 --2- 172.21.15.76:0/2334633554 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380a3bd0 0x55e7380ac270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f777b7fe640 1 -- 172.21.15.76:0/2334633554 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e737fc9570 con 0x55e7380a32b0 2024-03-12T06:02:40.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f777affd640 1 --2- 172.21.15.76:0/2334633554 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380a3bd0 0x55e7380ac270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:02:40.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f777b7fe640 1 --2- 172.21.15.76:0/2334633554 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a32b0 0x55e7380a3690 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f776c009b70 tx=0x7f776c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826c183783a5b7de server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:40.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.360+0000 7f7761ffb640 1 -- 172.21.15.76:0/2334633554 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c00fe30 con 0x55e7380a32b0 2024-03-12T06:02:40.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.361+0000 7f7761ffb640 1 -- 172.21.15.76:0/2334633554 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f776c020930 con 0x55e7380a32b0 2024-03-12T06:02:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.361+0000 7f7761ffb640 1 -- 172.21.15.76:0/2334633554 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776c0052f0 con 0x55e7380a32b0 2024-03-12T06:02:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.361+0000 7f7784a4b740 1 -- 172.21.15.76:0/2334633554 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a32b0 msgr2=0x55e7380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.361+0000 7f7784a4b740 1 --2- 172.21.15.76:0/2334633554 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a32b0 0x55e7380a3690 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f776c009b70 tx=0x7f776c020000 comp rx=0 tx=0).stop 2024-03-12T06:02:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.361+0000 7f7784a4b740 1 -- 172.21.15.76:0/2334633554 shutdown_connections 2024-03-12T06:02:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.361+0000 7f7784a4b740 1 --2- 172.21.15.76:0/2334633554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380ac7b0 0x55e7380aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.361+0000 7f7784a4b740 1 --2- 172.21.15.76:0/2334633554 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380a3bd0 0x55e7380ac270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.361+0000 7f7784a4b740 1 --2- 172.21.15.76:0/2334633554 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a32b0 0x55e7380a3690 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.361+0000 7f7784a4b740 1 -- 172.21.15.76:0/2334633554 >> 172.21.15.76:0/2334633554 conn(0x55e73809d990 msgr2=0x55e73809dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:40.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.361+0000 7f7784a4b740 1 -- 172.21.15.76:0/2334633554 shutdown_connections 2024-03-12T06:02:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.361+0000 7f7784a4b740 1 -- 172.21.15.76:0/2334633554 wait complete. 2024-03-12T06:02:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.361+0000 7f7784a4b740 1 -- start start 2024-03-12T06:02:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f7784a4b740 1 Processor -- start 2024-03-12T06:02:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f7784a4b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380a32b0 0x55e7380ac260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f7784a4b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380a95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f7784a4b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380ac7b0 0x55e7380a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f7784a4b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e737cde430 con 0x55e7380a3bd0 2024-03-12T06:02:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f7784a4b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e737c9ae00 con 0x55e7380a32b0 2024-03-12T06:02:40.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f7784a4b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e737f8ffd0 con 0x55e7380ac7b0 2024-03-12T06:02:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f7784a4b740 10 client.0 _ll_register_callbacks cb 0x55e737fe5fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f777affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380a95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f777affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380a95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52014/0 (socket says 172.21.15.76:52014) 2024-03-12T06:02:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f777affd640 1 -- 172.21.15.76:0/427415810 learned_addr learned my addr 172.21.15.76:0/427415810 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f777bfff640 1 --2- 172.21.15.76:0/427415810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380ac7b0 0x55e7380a9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f777b7fe640 1 --2- 172.21.15.76:0/427415810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380a32b0 0x55e7380ac260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f777affd640 1 -- 172.21.15.76:0/427415810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380ac7b0 msgr2=0x55e7380a9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f777affd640 1 --2- 172.21.15.76:0/427415810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380ac7b0 0x55e7380a9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f777affd640 1 -- 172.21.15.76:0/427415810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380a32b0 msgr2=0x55e7380ac260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f777affd640 1 --2- 172.21.15.76:0/427415810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380a32b0 0x55e7380ac260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.362+0000 7f777affd640 1 -- 172.21.15.76:0/427415810 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776c00f680 con 0x55e7380a3bd0 2024-03-12T06:02:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f777affd640 1 --2- 172.21.15.76:0/427415810 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380a95f0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f7770010840 tx=0x7f7770010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa69e3ed92423e26 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f775affd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f775affd640 1 -- 172.21.15.76:0/427415810 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770015070 con 0x55e7380a3bd0 2024-03-12T06:02:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f775affd640 1 -- 172.21.15.76:0/427415810 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f777000fb90 con 0x55e7380a3bd0 2024-03-12T06:02:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f7784a4b740 1 -- 172.21.15.76:0/427415810 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e737d0a7e0 con 0x55e7380a3bd0 2024-03-12T06:02:40.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f775affd640 1 -- 172.21.15.76:0/427415810 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f777000fd50 con 0x55e7380a3bd0 2024-03-12T06:02:40.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f7784a4b740 10 client.17019 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:40.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f7784a4b740 1 -- 172.21.15.76:0/427415810 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e738009610 con 0x55e7380a3bd0 2024-03-12T06:02:40.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f775affd640 1 -- 172.21.15.76:0/427415810 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7770026020 con 0x55e7380a3bd0 2024-03-12T06:02:40.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f775affd640 1 -- 172.21.15.76:0/427415810 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f77700194a0 con 0x55e7380a3bd0 2024-03-12T06:02:40.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f7784a4b740 10 client.17019 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:40.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f7784a4b740 10 client.17019 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:40.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f7784a4b740 20 client.17019 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:40.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f7784a4b740 1 --2- 172.21.15.76:0/427415810 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e7380bca10 0x55e7380bee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f7784a4b740 4 client.17019 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:02:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.363+0000 7f7784a4b740 1 -- 172.21.15.76:0/427415810 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e737d47720 con 0x55e7380bca10 2024-03-12T06:02:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.364+0000 7f777b7fe640 1 --2- 172.21.15.76:0/427415810 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e7380bca10 0x55e7380bee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.364+0000 7f775affd640 10 client.17019 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.364+0000 7f777b7fe640 1 --2- 172.21.15.76:0/427415810 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e7380bca10 0x55e7380bee30 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a166d8da48c86e server_cookie=ba4e8c050c5dd4bd in_seq=0 out_seq=0 2024-03-12T06:02:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f775affd640 1 -- 172.21.15.76:0/427415810 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e737d47720 con 0x55e7380bca10 2024-03-12T06:02:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f775affd640 10 client.17019 handle_command_reply: tid=0 2024-03-12T06:02:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f7784a4b740 1 client.17019 shutdown 2024-03-12T06:02:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f7784a4b740 1 -- 172.21.15.76:0/427415810 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 msgr2=0x55e7380a95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f7784a4b740 1 --2- 172.21.15.76:0/427415810 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380a95f0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f7770010840 tx=0x7f7770010d00 comp rx=0 tx=0).stop 2024-03-12T06:02:40.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f7784a4b740 1 -- 172.21.15.76:0/427415810 shutdown_connections 2024-03-12T06:02:40.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f7784a4b740 1 --2- 172.21.15.76:0/427415810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380ac7b0 0x55e7380a9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f7784a4b740 1 --2- 172.21.15.76:0/427415810 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e7380bca10 0x55e7380bee30 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f7784a4b740 1 --2- 172.21.15.76:0/427415810 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380a95f0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f7784a4b740 1 --2- 172.21.15.76:0/427415810 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380a32b0 0x55e7380ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f775affd640 0 client.17019 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:40.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f7784a4b740 1 -- 172.21.15.76:0/427415810 >> 172.21.15.76:0/427415810 conn(0x55e73809d990 msgr2=0x55e73809f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:40.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f7784a4b740 1 -- 172.21.15.76:0/427415810 shutdown_connections 2024-03-12T06:02:40.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f7784a4b740 1 -- 172.21.15.76:0/427415810 wait complete. 2024-03-12T06:02:40.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.365+0000 7f7784a4b740 20 client.17019 trim_cache size 0 max 16384 2024-03-12T06:02:40.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f7784a4b740 1 Processor -- start 2024-03-12T06:02:40.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f7784a4b740 1 -- start start 2024-03-12T06:02:40.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f7784a4b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380a32b0 0x55e7380abd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f7784a4b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380ac260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f7784a4b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380ac7b0 0x55e7380a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f7784a4b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7380a2cf0 con 0x55e7380a3bd0 2024-03-12T06:02:40.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f7784a4b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7380992b0 con 0x55e7380ac7b0 2024-03-12T06:02:40.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f7784a4b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e73809e770 con 0x55e7380a32b0 2024-03-12T06:02:40.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f777affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380ac260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f777affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380ac260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52018/0 (socket says 172.21.15.76:52018) 2024-03-12T06:02:40.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f777affd640 1 -- 172.21.15.76:0/628502273 learned_addr learned my addr 172.21.15.76:0/628502273 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:40.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f777affd640 1 -- 172.21.15.76:0/628502273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380a32b0 msgr2=0x55e7380abd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f777affd640 1 --2- 172.21.15.76:0/628502273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380a32b0 0x55e7380abd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f777affd640 1 -- 172.21.15.76:0/628502273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380ac7b0 msgr2=0x55e7380a8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f777bfff640 1 --2- 172.21.15.76:0/628502273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380ac7b0 0x55e7380a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f777affd640 1 --2- 172.21.15.76:0/628502273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380ac7b0 0x55e7380a8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f777affd640 1 -- 172.21.15.76:0/628502273 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e737fa5ac0 con 0x55e7380a3bd0 2024-03-12T06:02:40.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.380+0000 7f777bfff640 1 --2- 172.21.15.76:0/628502273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380ac7b0 0x55e7380a8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:40.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.381+0000 7f777affd640 1 --2- 172.21.15.76:0/628502273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380ac260 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f7770025040 tx=0x7f77700104b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715d96aca9298fe4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:40.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.381+0000 7f77617fa640 1 -- 172.21.15.76:0/628502273 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770015040 con 0x55e7380a3bd0 2024-03-12T06:02:40.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.381+0000 7f77617fa640 1 -- 172.21.15.76:0/628502273 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7770026040 con 0x55e7380a3bd0 2024-03-12T06:02:40.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.381+0000 7f77617fa640 1 -- 172.21.15.76:0/628502273 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777002b6e0 con 0x55e7380a3bd0 2024-03-12T06:02:40.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.381+0000 7f7784a4b740 1 -- 172.21.15.76:0/628502273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 msgr2=0x55e7380ac260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.381+0000 7f7784a4b740 1 --2- 172.21.15.76:0/628502273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380ac260 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f7770025040 tx=0x7f77700104b0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.381+0000 7f7784a4b740 1 -- 172.21.15.76:0/628502273 shutdown_connections 2024-03-12T06:02:40.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.381+0000 7f7784a4b740 1 --2- 172.21.15.76:0/628502273 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380ac7b0 0x55e7380a8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.381+0000 7f7784a4b740 1 --2- 172.21.15.76:0/628502273 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380ac260 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.381+0000 7f7784a4b740 1 --2- 172.21.15.76:0/628502273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380a32b0 0x55e7380abd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.381+0000 7f7784a4b740 1 -- 172.21.15.76:0/628502273 >> 172.21.15.76:0/628502273 conn(0x55e73809d990 msgr2=0x55e7380a7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:40.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.381+0000 7f7784a4b740 1 -- 172.21.15.76:0/628502273 shutdown_connections 2024-03-12T06:02:40.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.381+0000 7f7784a4b740 1 -- 172.21.15.76:0/628502273 wait complete. 2024-03-12T06:02:40.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f7784a4b740 1 -- start start 2024-03-12T06:02:40.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f7784a4b740 1 Processor -- start 2024-03-12T06:02:40.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f7784a4b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380a32b0 0x55e7380b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f7784a4b740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f7784a4b740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380ac7b0 0x55e7380b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f7784a4b740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7380b1610 con 0x55e7380a3bd0 2024-03-12T06:02:40.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f7784a4b740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7380b1780 con 0x55e7380ac7b0 2024-03-12T06:02:40.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f7784a4b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7380b1940 con 0x55e7380a32b0 2024-03-12T06:02:40.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f7784a4b740 10 client.0 _ll_register_callbacks cb 0x7f776c009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:40.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f777affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f777affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52020/0 (socket says 172.21.15.76:52020) 2024-03-12T06:02:40.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f777affd640 1 -- 172.21.15.76:0/1433674146 learned_addr learned my addr 172.21.15.76:0/1433674146 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:40.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f777affd640 1 -- 172.21.15.76:0/1433674146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380a32b0 msgr2=0x55e7380b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f777affd640 1 --2- 172.21.15.76:0/1433674146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380a32b0 0x55e7380b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f777affd640 1 -- 172.21.15.76:0/1433674146 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380ac7b0 msgr2=0x55e7380b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f777affd640 1 --2- 172.21.15.76:0/1433674146 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380ac7b0 0x55e7380b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.382+0000 7f777affd640 1 -- 172.21.15.76:0/1433674146 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e738009610 con 0x55e7380a3bd0 2024-03-12T06:02:40.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f777affd640 1 --2- 172.21.15.76:0/1433674146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380b0b90 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f7770025040 tx=0x7f7770010520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b49d5a12be270 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:40.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f775affd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:40.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f775affd640 1 -- 172.21.15.76:0/1433674146 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770019c20 con 0x55e7380a3bd0 2024-03-12T06:02:40.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f775affd640 1 -- 172.21.15.76:0/1433674146 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7770027080 con 0x55e7380a3bd0 2024-03-12T06:02:40.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f7784a4b740 1 -- 172.21.15.76:0/1433674146 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f776c020930 con 0x55e7380a3bd0 2024-03-12T06:02:40.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f775affd640 1 -- 172.21.15.76:0/1433674146 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7770019620 con 0x55e7380a3bd0 2024-03-12T06:02:40.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f7784a4b740 10 client.17025 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:40.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f7784a4b740 1 -- 172.21.15.76:0/1433674146 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e737fcb6a0 con 0x55e7380a3bd0 2024-03-12T06:02:40.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f775affd640 1 -- 172.21.15.76:0/1433674146 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7770045020 con 0x55e7380a3bd0 2024-03-12T06:02:40.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f775affd640 1 -- 172.21.15.76:0/1433674146 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7770022460 con 0x55e7380a3bd0 2024-03-12T06:02:40.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f7784a4b740 10 client.17025 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:40.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f7784a4b740 10 client.17025 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:40.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f7784a4b740 20 client.17025 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:40.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f7784a4b740 1 --2- 172.21.15.76:0/1433674146 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e738115130 0x55e738117550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f7784a4b740 4 client.17025 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:02:40.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.383+0000 7f7784a4b740 1 -- 172.21.15.76:0/1433674146 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e737d96890 con 0x55e738115130 2024-03-12T06:02:40.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f777b7fe640 1 --2- 172.21.15.76:0/1433674146 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e738115130 0x55e738117550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f775affd640 10 client.17025 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:40.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f777b7fe640 1 --2- 172.21.15.76:0/1433674146 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e738115130 0x55e738117550 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5305cd5f77c697dd server_cookie=f9e00082db070150 in_seq=0 out_seq=0 2024-03-12T06:02:40.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f775affd640 1 -- 172.21.15.76:0/1433674146 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e737d96890 con 0x55e738115130 2024-03-12T06:02:40.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f775affd640 10 client.17025 handle_command_reply: tid=0 2024-03-12T06:02:40.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f7784a4b740 1 client.17025 shutdown 2024-03-12T06:02:40.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f7784a4b740 1 -- 172.21.15.76:0/1433674146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 msgr2=0x55e7380b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f7784a4b740 1 --2- 172.21.15.76:0/1433674146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380b0b90 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f7770025040 tx=0x7f7770010520 comp rx=0 tx=0).stop 2024-03-12T06:02:40.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f7784a4b740 1 -- 172.21.15.76:0/1433674146 shutdown_connections 2024-03-12T06:02:40.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f7784a4b740 1 --2- 172.21.15.76:0/1433674146 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55e7380ac7b0 0x55e7380b10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f7784a4b740 1 --2- 172.21.15.76:0/1433674146 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55e738115130 0x55e738117550 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f7784a4b740 1 --2- 172.21.15.76:0/1433674146 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55e7380a3bd0 0x55e7380b0b90 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f775affd640 0 client.17025 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:40.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f7784a4b740 1 --2- 172.21.15.76:0/1433674146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7380a32b0 0x55e7380b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f7784a4b740 1 -- 172.21.15.76:0/1433674146 >> 172.21.15.76:0/1433674146 conn(0x55e73809d990 msgr2=0x55e7380a8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:40.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f7784a4b740 1 -- 172.21.15.76:0/1433674146 shutdown_connections 2024-03-12T06:02:40.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.384+0000 7f7784a4b740 1 -- 172.21.15.76:0/1433674146 wait complete. 2024-03-12T06:02:40.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.385+0000 7f7784a4b740 20 client.17025 trim_cache size 0 max 16384 2024-03-12T06:02:40.400 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:02:40.400 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T06:02:40.400 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T06:02:40.400 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:02:40.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.385+0000 7f7781e83640 1 -- 172.21.15.76:0/3584397281 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f77640a1ba0 msgr2=0x7f77640a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.385+0000 7f7781e83640 1 --2- 172.21.15.76:0/3584397281 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f77640a1ba0 0x7f77640a4060 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f776c013520 tx=0x7f776c013920 comp rx=0 tx=0).stop 2024-03-12T06:02:40.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.385+0000 7f7781e83640 1 -- 172.21.15.76:0/3584397281 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c10db20 msgr2=0x7f777c1996c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.385+0000 7f7781e83640 1 --2- 172.21.15.76:0/3584397281 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c10db20 0x7f777c1996c0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f7774004af0 tx=0x7f7774034cc0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.385+0000 7f7781e83640 1 -- 172.21.15.76:0/3584397281 shutdown_connections 2024-03-12T06:02:40.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.385+0000 7f7781e83640 1 --2- 172.21.15.76:0/3584397281 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f77640a1ba0 0x7f77640a4060 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.385+0000 7f7781e83640 1 --2- 172.21.15.76:0/3584397281 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f777c10db20 0x7f777c1996c0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.385+0000 7f7781e83640 1 --2- 172.21.15.76:0/3584397281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c068df0 0x7f777c19fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.385+0000 7f7781e83640 1 --2- 172.21.15.76:0/3584397281 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f777c0684d0 0x7f777c19f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.385+0000 7f7781e83640 1 -- 172.21.15.76:0/3584397281 >> 172.21.15.76:0/3584397281 conn(0x7f777c077180 msgr2=0x7f777c10ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:40.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.385+0000 7f7781e83640 1 -- 172.21.15.76:0/3584397281 shutdown_connections 2024-03-12T06:02:40.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.385+0000 7f7781e83640 1 -- 172.21.15.76:0/3584397281 wait complete. 2024-03-12T06:02:40.403 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T06:02:40.403 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T06:02:40.427 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:40 smithi076 ceph-mon[32095]: pgmap v211: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:02:40.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:40 smithi082 ceph-mon[36382]: pgmap v211: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:02:40.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:40 smithi088 ceph-mon[36585]: pgmap v211: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:02:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.502+0000 7fae88e17640 1 -- 172.21.15.76:0/416188402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae841002a0 msgr2=0x7fae84100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.502+0000 7fae88e17640 1 --2- 172.21.15.76:0/416188402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae841002a0 0x7fae84100680 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fae7c009930 tx=0x7fae7c02bc30 comp rx=0 tx=0).stop 2024-03-12T06:02:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.502+0000 7fae88e17640 1 -- 172.21.15.76:0/416188402 shutdown_connections 2024-03-12T06:02:40.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.502+0000 7fae88e17640 1 --2- 172.21.15.76:0/416188402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae84105370 0x7fae8410cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.502+0000 7fae88e17640 1 --2- 172.21.15.76:0/416188402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae84100c50 0x7fae84104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.502+0000 7fae88e17640 1 --2- 172.21.15.76:0/416188402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae841002a0 0x7fae84100680 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.502+0000 7fae88e17640 1 -- 172.21.15.76:0/416188402 >> 172.21.15.76:0/416188402 conn(0x7fae84077200 msgr2=0x7fae84077610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.502+0000 7fae88e17640 1 -- 172.21.15.76:0/416188402 shutdown_connections 2024-03-12T06:02:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.502+0000 7fae88e17640 1 -- 172.21.15.76:0/416188402 wait complete. 2024-03-12T06:02:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.502+0000 7fae88e17640 1 Processor -- start 2024-03-12T06:02:40.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.502+0000 7fae88e17640 1 -- start start 2024-03-12T06:02:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae88e17640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae841002a0 0x7fae84101d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae88e17640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae84105370 0x7fae84102270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae88e17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae841027b0 0x7fae841a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae88e17640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae841a5f90 con 0x7fae841002a0 2024-03-12T06:02:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae88e17640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae841a6100 con 0x7fae84105370 2024-03-12T06:02:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae88e17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae841a62c0 con 0x7fae841027b0 2024-03-12T06:02:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae8259c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae841002a0 0x7fae84101d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae8259c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae841002a0 0x7fae84101d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52040/0 (socket says 172.21.15.76:52040) 2024-03-12T06:02:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae8259c640 1 -- 172.21.15.76:0/1689206591 learned_addr learned my addr 172.21.15.76:0/1689206591 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:40.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae82d9d640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae841027b0 0x7fae841a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae81d9b640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae84105370 0x7fae84102270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae8259c640 1 -- 172.21.15.76:0/1689206591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae841027b0 msgr2=0x7fae841a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae8259c640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae841027b0 0x7fae841a5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae8259c640 1 -- 172.21.15.76:0/1689206591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae84105370 msgr2=0x7fae84102270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae8259c640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae84105370 0x7fae84102270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae8259c640 1 -- 172.21.15.76:0/1689206591 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c009560 con 0x7fae841002a0 2024-03-12T06:02:40.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae82d9d640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae841027b0 0x7fae841a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:02:40.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae81d9b640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae84105370 0x7fae84102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:40.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae8259c640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae841002a0 0x7fae84101d30 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fae7c009230 tx=0x7fae7c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:40.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae737fe640 1 -- 172.21.15.76:0/1689206591 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c004440 con 0x7fae841002a0 2024-03-12T06:02:40.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae88e17640 1 -- 172.21.15.76:0/1689206591 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae841a6540 con 0x7fae841002a0 2024-03-12T06:02:40.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae737fe640 1 -- 172.21.15.76:0/1689206591 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae7c0052d0 con 0x7fae841002a0 2024-03-12T06:02:40.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae737fe640 1 -- 172.21.15.76:0/1689206591 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c003d70 con 0x7fae841002a0 2024-03-12T06:02:40.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.503+0000 7fae88e17640 1 -- 172.21.15.76:0/1689206591 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae841a6a50 con 0x7fae841002a0 2024-03-12T06:02:40.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.504+0000 7fae88e17640 1 -- 172.21.15.76:0/1689206591 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae84068000 con 0x7fae841002a0 2024-03-12T06:02:40.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.505+0000 7fae737fe640 1 -- 172.21.15.76:0/1689206591 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fae7c005440 con 0x7fae841002a0 2024-03-12T06:02:40.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.506+0000 7fae737fe640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fae6c0a1ba0 0x7fae6c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.506+0000 7fae81d9b640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fae6c0a1ba0 0x7fae6c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.506+0000 7fae737fe640 1 -- 172.21.15.76:0/1689206591 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fae7c0efa20 con 0x7fae841002a0 2024-03-12T06:02:40.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.506+0000 7fae81d9b640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fae6c0a1ba0 0x7fae6c0a4060 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fae84103770 tx=0x7fae780093e0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:40.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.510+0000 7fae737fe640 1 -- 172.21.15.76:0/1689206591 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fae7c0b8a00 con 0x7fae841002a0 2024-03-12T06:02:40.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.720+0000 7fae88e17640 1 -- 172.21.15.76:0/1689206591 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae84103210 con 0x7fae841002a0 2024-03-12T06:02:40.725 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.721+0000 7fae737fe640 1 -- 172.21.15.76:0/1689206591 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11328 (secure 0 0 0) 0x7fae7c0bc0c0 con 0x7fae841002a0 2024-03-12T06:02:40.726 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:02:40.726 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T06:01:33.842001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T06:02:40.726 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 28 2024-03-12T06:02:40.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.723+0000 7fae88e17640 1 -- 172.21.15.76:0/1689206591 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fae6c0a1ba0 msgr2=0x7fae6c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.723+0000 7fae88e17640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fae6c0a1ba0 0x7fae6c0a4060 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fae84103770 tx=0x7fae780093e0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.723+0000 7fae88e17640 1 -- 172.21.15.76:0/1689206591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae841002a0 msgr2=0x7fae84101d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.723+0000 7fae88e17640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae841002a0 0x7fae84101d30 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fae7c009230 tx=0x7fae7c004040 comp rx=0 tx=0).stop 2024-03-12T06:02:40.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.723+0000 7fae88e17640 1 -- 172.21.15.76:0/1689206591 shutdown_connections 2024-03-12T06:02:40.728 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.723+0000 7fae88e17640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fae6c0a1ba0 0x7fae6c0a4060 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.723+0000 7fae88e17640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae841027b0 0x7fae841a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.723+0000 7fae88e17640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae84105370 0x7fae84102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.723+0000 7fae88e17640 1 --2- 172.21.15.76:0/1689206591 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae841002a0 0x7fae84101d30 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.723+0000 7fae88e17640 1 -- 172.21.15.76:0/1689206591 >> 172.21.15.76:0/1689206591 conn(0x7fae84077200 msgr2=0x7fae840687d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:40.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.723+0000 7fae88e17640 1 -- 172.21.15.76:0/1689206591 shutdown_connections 2024-03-12T06:02:40.729 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.723+0000 7fae88e17640 1 -- 172.21.15.76:0/1689206591 wait complete. 2024-03-12T06:02:40.739 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:02:40.739 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T06:02:40.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.878+0000 7f8afd4d4640 1 -- 172.21.15.76:0/1323573651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8af8105360 msgr2=0x7f8af810cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.878+0000 7f8afd4d4640 1 --2- 172.21.15.76:0/1323573651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8af8105360 0x7f8af810cf30 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f8af0009cd0 tx=0x7f8af002c0f0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.878+0000 7f8afd4d4640 1 -- 172.21.15.76:0/1323573651 shutdown_connections 2024-03-12T06:02:40.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.878+0000 7f8afd4d4640 1 --2- 172.21.15.76:0/1323573651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8af8105360 0x7f8af810cf30 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.878+0000 7f8afd4d4640 1 --2- 172.21.15.76:0/1323573651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8af8100c40 0x7f8af8104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.878+0000 7f8afd4d4640 1 --2- 172.21.15.76:0/1323573651 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af8100290 0x7f8af8100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.878+0000 7f8afd4d4640 1 -- 172.21.15.76:0/1323573651 >> 172.21.15.76:0/1323573651 conn(0x7f8af8077230 msgr2=0x7f8af8077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:40.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.878+0000 7f8afd4d4640 1 -- 172.21.15.76:0/1323573651 shutdown_connections 2024-03-12T06:02:40.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.878+0000 7f8afd4d4640 1 -- 172.21.15.76:0/1323573651 wait complete. 2024-03-12T06:02:40.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.879+0000 7f8afd4d4640 1 Processor -- start 2024-03-12T06:02:40.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.879+0000 7f8afd4d4640 1 -- start start 2024-03-12T06:02:40.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.879+0000 7f8afd4d4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8af8100290 0x7f8af819d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.879+0000 7f8afd4d4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af8105360 0x7f8af819d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.879+0000 7f8afd4d4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8af81975a0 0x7f8af8197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.879+0000 7f8afd4d4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af8197f90 con 0x7f8af81975a0 2024-03-12T06:02:40.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.879+0000 7f8afd4d4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af8198100 con 0x7f8af8100290 2024-03-12T06:02:40.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.879+0000 7f8afd4d4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af81982c0 con 0x7f8af8105360 2024-03-12T06:02:40.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.879+0000 7f8af77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8af81975a0 0x7f8af8197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.879+0000 7f8af77fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8af81975a0 0x7f8af8197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52050/0 (socket says 172.21.15.76:52050) 2024-03-12T06:02:40.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.879+0000 7f8af77fe640 1 -- 172.21.15.76:0/3652983332 learned_addr learned my addr 172.21.15.76:0/3652983332 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:40.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.880+0000 7f8af6ffd640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8af8100290 0x7f8af819d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.880+0000 7f8af67fc640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af8105360 0x7f8af819d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.880+0000 7f8af77fe640 1 -- 172.21.15.76:0/3652983332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af8105360 msgr2=0x7f8af819d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.880+0000 7f8af77fe640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af8105360 0x7f8af819d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.880+0000 7f8af77fe640 1 -- 172.21.15.76:0/3652983332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8af8100290 msgr2=0x7f8af819d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.880+0000 7f8af77fe640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8af8100290 0x7f8af819d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.880+0000 7f8af77fe640 1 -- 172.21.15.76:0/3652983332 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af0009900 con 0x7f8af81975a0 2024-03-12T06:02:40.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.880+0000 7f8af67fc640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af8105360 0x7f8af819d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:40.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.880+0000 7f8af77fe640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8af81975a0 0x7f8af8197a50 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8af0004af0 tx=0x7f8af0034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:40.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.880+0000 7f8adffff640 1 -- 172.21.15.76:0/3652983332 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af00328c0 con 0x7f8af81975a0 2024-03-12T06:02:40.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.880+0000 7f8adffff640 1 -- 172.21.15.76:0/3652983332 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8af0003e50 con 0x7f8af81975a0 2024-03-12T06:02:40.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.880+0000 7f8afd4d4640 1 -- 172.21.15.76:0/3652983332 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af81a42b0 con 0x7f8af81975a0 2024-03-12T06:02:40.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.880+0000 7f8adffff640 1 -- 172.21.15.76:0/3652983332 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af0004860 con 0x7f8af81975a0 2024-03-12T06:02:40.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.880+0000 7f8afd4d4640 1 -- 172.21.15.76:0/3652983332 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af81a4730 con 0x7f8af81975a0 2024-03-12T06:02:40.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.882+0000 7f8adffff640 1 -- 172.21.15.76:0/3652983332 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f8af000a040 con 0x7f8af81975a0 2024-03-12T06:02:40.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.882+0000 7f8b00187740 1 Processor -- start 2024-03-12T06:02:40.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.882+0000 7f8b00187740 1 -- start start 2024-03-12T06:02:40.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8b00187740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b184580250 0x55b184580630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8b00187740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580b70 0x55b184589210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8b00187740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184589750 0x55b18458bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8b00187740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b18453b960 con 0x55b184580b70 2024-03-12T06:02:40.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8b00187740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1844ef310 con 0x55b184589750 2024-03-12T06:02:40.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8b00187740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1844ec840 con 0x55b184580250 2024-03-12T06:02:40.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8af6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580b70 0x55b184589210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8af6ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580b70 0x55b184589210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52056/0 (socket says 172.21.15.76:52056) 2024-03-12T06:02:40.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8af6ffd640 1 -- 172.21.15.76:0/1871659233 learned_addr learned my addr 172.21.15.76:0/1871659233 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:40.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8af67fc640 1 --2- 172.21.15.76:0/1871659233 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184589750 0x55b18458bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8adffff640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8ae00a1ba0 0x7f8ae00a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8af67fc640 1 --2- 172.21.15.76:0/1871659233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b184580250 0x55b184580630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8af77fe640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8ae00a1ba0 0x7f8ae00a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8af6ffd640 1 -- 172.21.15.76:0/1871659233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b184580250 msgr2=0x55b184580630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8af6ffd640 1 --2- 172.21.15.76:0/1871659233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b184580250 0x55b184580630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8af6ffd640 1 -- 172.21.15.76:0/1871659233 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184589750 msgr2=0x55b18458bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8af6ffd640 1 --2- 172.21.15.76:0/1871659233 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184589750 0x55b18458bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8af6ffd640 1 -- 172.21.15.76:0/1871659233 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b1844aa060 con 0x55b184580b70 2024-03-12T06:02:40.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8adffff640 1 -- 172.21.15.76:0/3652983332 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f8af00f0570 con 0x7f8af81975a0 2024-03-12T06:02:40.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.883+0000 7f8af6ffd640 1 --2- 172.21.15.76:0/1871659233 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580b70 0x55b184589210 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8009bc0 tx=0x7f8ae800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4bb1a66fc9b1d59 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:40.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.884+0000 7f8af77fe640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8ae00a1ba0 0x7f8ae00a4060 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8af00b9c20 tx=0x7f8af008f450 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:40.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.884+0000 7f8add7fa640 1 -- 172.21.15.76:0/1871659233 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae800ee70 con 0x55b184580b70 2024-03-12T06:02:40.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.884+0000 7f8add7fa640 1 -- 172.21.15.76:0/1871659233 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ae8011950 con 0x55b184580b70 2024-03-12T06:02:40.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.884+0000 7f8add7fa640 1 -- 172.21.15.76:0/1871659233 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae80109c0 con 0x55b184580b70 2024-03-12T06:02:40.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.884+0000 7f8b00187740 1 -- 172.21.15.76:0/1871659233 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580b70 msgr2=0x55b184589210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.884+0000 7f8b00187740 1 --2- 172.21.15.76:0/1871659233 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580b70 0x55b184589210 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8009bc0 tx=0x7f8ae800e6c0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.884+0000 7f8b00187740 1 -- 172.21.15.76:0/1871659233 shutdown_connections 2024-03-12T06:02:40.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.884+0000 7f8b00187740 1 --2- 172.21.15.76:0/1871659233 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184589750 0x55b18458bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.884+0000 7f8b00187740 1 --2- 172.21.15.76:0/1871659233 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580b70 0x55b184589210 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.884+0000 7f8b00187740 1 --2- 172.21.15.76:0/1871659233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b184580250 0x55b184580630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.884+0000 7f8b00187740 1 -- 172.21.15.76:0/1871659233 >> 172.21.15.76:0/1871659233 conn(0x55b18457a980 msgr2=0x55b18457ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:40.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.884+0000 7f8b00187740 1 -- 172.21.15.76:0/1871659233 shutdown_connections 2024-03-12T06:02:40.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.884+0000 7f8b00187740 1 -- 172.21.15.76:0/1871659233 wait complete. 2024-03-12T06:02:40.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8b00187740 1 -- start start 2024-03-12T06:02:40.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8b00187740 1 Processor -- start 2024-03-12T06:02:40.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8b00187740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b184580250 0x55b184586760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8b00187740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184580b70 0x55b184586ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8b00187740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184589750 0x55b1845871e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8b00187740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1844e8020 con 0x55b184589750 2024-03-12T06:02:40.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8b00187740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1841bb430 con 0x55b184580b70 2024-03-12T06:02:40.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8b00187740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b184177e00 con 0x55b184580250 2024-03-12T06:02:40.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8b00187740 10 client.0 _ll_register_callbacks cb 0x55b18448fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:40.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8af67fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184589750 0x55b1845871e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8af67fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184589750 0x55b1845871e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52058/0 (socket says 172.21.15.76:52058) 2024-03-12T06:02:40.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8af67fc640 1 -- 172.21.15.76:0/3175302018 learned_addr learned my addr 172.21.15.76:0/3175302018 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:40.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8af6ffd640 1 --2- 172.21.15.76:0/3175302018 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184580b70 0x55b184586ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8af67fc640 1 --2- 172.21.15.76:0/3175302018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b184580250 0x55b184586760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8af67fc640 1 -- 172.21.15.76:0/3175302018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b184580250 msgr2=0x55b184586760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8af67fc640 1 --2- 172.21.15.76:0/3175302018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b184580250 0x55b184586760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8af67fc640 1 -- 172.21.15.76:0/3175302018 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184580b70 msgr2=0x55b184586ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8af67fc640 1 --2- 172.21.15.76:0/3175302018 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184580b70 0x55b184586ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.885+0000 7f8af67fc640 1 -- 172.21.15.76:0/3175302018 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae8009820 con 0x55b184589750 2024-03-12T06:02:40.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8af67fc640 1 --2- 172.21.15.76:0/3175302018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184589750 0x55b1845871e0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f8af81986e0 tx=0x7f8aec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2801ed468cfa5637 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:40.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8adaff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:40.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8adaff5640 1 -- 172.21.15.76:0/3175302018 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec004020 con 0x55b184589750 2024-03-12T06:02:40.902 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8adaff5640 1 -- 172.21.15.76:0/3175302018 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f8aec004180 con 0x55b184589750 2024-03-12T06:02:40.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8b00187740 1 -- 172.21.15.76:0/3175302018 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b1841e77e0 con 0x55b184589750 2024-03-12T06:02:40.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8adaff5640 1 -- 172.21.15.76:0/3175302018 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8aec004340 con 0x55b184589750 2024-03-12T06:02:40.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8b00187740 10 client.17043 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:40.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8b00187740 1 -- 172.21.15.76:0/3175302018 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b1844e66a0 con 0x55b184589750 2024-03-12T06:02:40.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8adaff5640 1 -- 172.21.15.76:0/3175302018 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b1841e77e0 con 0x55b184589750 2024-03-12T06:02:40.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8adaff5640 1 -- 172.21.15.76:0/3175302018 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8aec004c80 con 0x55b184589750 2024-03-12T06:02:40.903 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8b00187740 10 client.17043 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:40.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8b00187740 10 client.17043 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:02:40.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8b00187740 20 client.17043 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:40.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8b00187740 1 --2- 172.21.15.76:0/3175302018 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b184599be0 0x55b18459c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8b00187740 4 client.17043 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:02:40.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.886+0000 7f8b00187740 1 -- 172.21.15.76:0/3175302018 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b184224720 con 0x55b184599be0 2024-03-12T06:02:40.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.887+0000 7f8af6ffd640 1 --2- 172.21.15.76:0/3175302018 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b184599be0 0x55b18459c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.887+0000 7f8adaff5640 10 client.17043 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:02:40.904 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.887+0000 7f8af6ffd640 1 --2- 172.21.15.76:0/3175302018 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b184599be0 0x55b18459c000 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b365f40c875db652 server_cookie=4f9839536757a4e1 in_seq=0 out_seq=0 2024-03-12T06:02:40.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8adaff5640 1 -- 172.21.15.76:0/3175302018 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b184224720 con 0x55b184599be0 2024-03-12T06:02:40.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8adaff5640 10 client.17043 handle_command_reply: tid=0 2024-03-12T06:02:40.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8b00187740 1 client.17043 shutdown 2024-03-12T06:02:40.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8b00187740 1 -- 172.21.15.76:0/3175302018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184589750 msgr2=0x55b1845871e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8b00187740 1 --2- 172.21.15.76:0/3175302018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184589750 0x55b1845871e0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f8af81986e0 tx=0x7f8aec0046c0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8b00187740 1 -- 172.21.15.76:0/3175302018 shutdown_connections 2024-03-12T06:02:40.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8b00187740 1 --2- 172.21.15.76:0/3175302018 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b184599be0 0x55b18459c000 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.905 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8b00187740 1 --2- 172.21.15.76:0/3175302018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184589750 0x55b1845871e0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8b00187740 1 --2- 172.21.15.76:0/3175302018 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184580b70 0x55b184586ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8adaff5640 0 client.17043 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:02:40.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8b00187740 1 --2- 172.21.15.76:0/3175302018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b184580250 0x55b184586760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8b00187740 1 -- 172.21.15.76:0/3175302018 >> 172.21.15.76:0/3175302018 conn(0x55b18457a980 msgr2=0x55b18457c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:40.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8b00187740 1 -- 172.21.15.76:0/3175302018 shutdown_connections 2024-03-12T06:02:40.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8b00187740 1 -- 172.21.15.76:0/3175302018 wait complete. 2024-03-12T06:02:40.906 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.888+0000 7f8b00187740 20 client.17043 trim_cache size 0 max 16384 2024-03-12T06:02:40.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8b00187740 1 Processor -- start 2024-03-12T06:02:40.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8b00187740 1 -- start start 2024-03-12T06:02:40.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8b00187740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580250 0x55b184588e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.912 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8b00187740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b184580b70 0x55b184581a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8b00187740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184589750 0x55b184581fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8b00187740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b18446d130 con 0x55b184580250 2024-03-12T06:02:40.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8b00187740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b184224720 con 0x55b184589750 2024-03-12T06:02:40.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8b00187740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1841e77e0 con 0x55b184580b70 2024-03-12T06:02:40.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8af67fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580250 0x55b184588e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8af67fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580250 0x55b184588e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52074/0 (socket says 172.21.15.76:52074) 2024-03-12T06:02:40.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8af67fc640 1 -- 172.21.15.76:0/1754511590 learned_addr learned my addr 172.21.15.76:0/1754511590 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:40.913 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8af67fc640 1 -- 172.21.15.76:0/1754511590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b184580b70 msgr2=0x55b184581a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8af67fc640 1 --2- 172.21.15.76:0/1754511590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b184580b70 0x55b184581a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8af67fc640 1 -- 172.21.15.76:0/1754511590 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184589750 msgr2=0x55b184581fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8af67fc640 1 --2- 172.21.15.76:0/1754511590 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184589750 0x55b184581fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8af67fc640 1 -- 172.21.15.76:0/1754511590 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b184482c20 con 0x55b184580250 2024-03-12T06:02:40.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8af67fc640 1 --2- 172.21.15.76:0/1754511590 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580250 0x55b184588e90 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f8aec001eb0 tx=0x7f8aec0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfffd336090b1574 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:40.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8adcff9640 1 -- 172.21.15.76:0/1754511590 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec013ac0 con 0x55b184580250 2024-03-12T06:02:40.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8adcff9640 1 -- 172.21.15.76:0/1754511590 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8aec032040 con 0x55b184580250 2024-03-12T06:02:40.914 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.900+0000 7f8adcff9640 1 -- 172.21.15.76:0/1754511590 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec005020 con 0x55b184580250 2024-03-12T06:02:40.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.901+0000 7f8b00187740 1 -- 172.21.15.76:0/1754511590 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580250 msgr2=0x55b184588e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.901+0000 7f8b00187740 1 --2- 172.21.15.76:0/1754511590 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580250 0x55b184588e90 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f8aec001eb0 tx=0x7f8aec0035f0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.901+0000 7f8b00187740 1 -- 172.21.15.76:0/1754511590 shutdown_connections 2024-03-12T06:02:40.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.901+0000 7f8b00187740 1 --2- 172.21.15.76:0/1754511590 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184589750 0x55b184581fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.901+0000 7f8b00187740 1 --2- 172.21.15.76:0/1754511590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b184580b70 0x55b184581a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.901+0000 7f8b00187740 1 --2- 172.21.15.76:0/1754511590 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580250 0x55b184588e90 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.901+0000 7f8b00187740 1 -- 172.21.15.76:0/1754511590 >> 172.21.15.76:0/1754511590 conn(0x55b18457a980 msgr2=0x55b184575a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:40.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.901+0000 7f8b00187740 1 -- 172.21.15.76:0/1754511590 shutdown_connections 2024-03-12T06:02:40.915 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.901+0000 7f8b00187740 1 -- 172.21.15.76:0/1754511590 wait complete. 2024-03-12T06:02:40.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.901+0000 7f8b00187740 1 -- start start 2024-03-12T06:02:40.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.901+0000 7f8b00187740 1 Processor -- start 2024-03-12T06:02:40.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.901+0000 7f8b00187740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580b70 0x55b18458f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8b00187740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184589750 0x55b18458ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8b00187740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b1845e9090 0x55b1845e9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8b00187740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1845e9a50 con 0x55b184580b70 2024-03-12T06:02:40.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8b00187740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1845e9bc0 con 0x55b184589750 2024-03-12T06:02:40.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8b00187740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1845e9d80 con 0x55b1845e9090 2024-03-12T06:02:40.916 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8b00187740 10 client.0 _ll_register_callbacks cb 0x7f8ae8009380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:40.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8af67fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580b70 0x55b18458f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8af67fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580b70 0x55b18458f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52082/0 (socket says 172.21.15.76:52082) 2024-03-12T06:02:40.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8af67fc640 1 -- 172.21.15.76:0/4273608581 learned_addr learned my addr 172.21.15.76:0/4273608581 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:40.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8af6ffd640 1 --2- 172.21.15.76:0/4273608581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184589750 0x55b18458ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8af67fc640 1 --2- 172.21.15.76:0/4273608581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b1845e9090 0x55b1845e9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8af67fc640 1 -- 172.21.15.76:0/4273608581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b1845e9090 msgr2=0x55b1845e9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8af67fc640 1 --2- 172.21.15.76:0/4273608581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b1845e9090 0x55b1845e9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8af67fc640 1 -- 172.21.15.76:0/4273608581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184589750 msgr2=0x55b18458ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:40.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8af67fc640 1 --2- 172.21.15.76:0/4273608581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184589750 0x55b18458ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8af67fc640 1 -- 172.21.15.76:0/4273608581 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b1844e66a0 con 0x55b184580b70 2024-03-12T06:02:40.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8af67fc640 1 --2- 172.21.15.76:0/4273608581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580b70 0x55b18458f9e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f8aec011800 tx=0x7f8aec011700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eaa5dceab407c5b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:40.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8adaff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:40.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8adaff5640 1 -- 172.21.15.76:0/4273608581 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec03d040 con 0x55b184580b70 2024-03-12T06:02:40.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8adaff5640 1 -- 172.21.15.76:0/4273608581 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f8aec000e40 con 0x55b184580b70 2024-03-12T06:02:40.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8b00187740 1 -- 172.21.15.76:0/4273608581 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b184586590 con 0x55b184580b70 2024-03-12T06:02:40.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8adaff5640 1 -- 172.21.15.76:0/4273608581 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8aec013a30 con 0x55b184580b70 2024-03-12T06:02:40.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.902+0000 7f8b00187740 10 client.17049 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:40.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.903+0000 7f8b00187740 1 -- 172.21.15.76:0/4273608581 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b1844a88c0 con 0x55b184580b70 2024-03-12T06:02:40.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.903+0000 7f8adaff5640 1 -- 172.21.15.76:0/4273608581 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b184586590 con 0x55b184580b70 2024-03-12T06:02:40.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.903+0000 7f8adaff5640 1 -- 172.21.15.76:0/4273608581 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8aec004680 con 0x55b184580b70 2024-03-12T06:02:40.919 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.903+0000 7f8b00187740 10 client.17049 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:40.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.903+0000 7f8b00187740 10 client.17049 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:02:40.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.903+0000 7f8b00187740 20 client.17049 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:40.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.903+0000 7f8b00187740 1 --2- 172.21.15.76:0/4273608581 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b1845f21f0 0x55b1845f45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:40.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.903+0000 7f8b00187740 4 client.17049 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:02:40.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.903+0000 7f8b00187740 1 -- 172.21.15.76:0/4273608581 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f8ae8011950 con 0x55b1845f21f0 2024-03-12T06:02:40.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.903+0000 7f8af6ffd640 1 --2- 172.21.15.76:0/4273608581 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b1845f21f0 0x55b1845f45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:40.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.903+0000 7f8adaff5640 10 client.17049 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:02:40.922 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.903+0000 7f8af6ffd640 1 --2- 172.21.15.76:0/4273608581 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b1845f21f0 0x55b1845f45e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac48bf3d86e67e30 server_cookie=7ba7e495a8829c72 in_seq=0 out_seq=0 2024-03-12T06:02:40.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8adaff5640 1 -- 172.21.15.76:0/4273608581 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f8ae8011950 con 0x55b1845f21f0 2024-03-12T06:02:40.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8adaff5640 10 client.17049 handle_command_reply: tid=0 2024-03-12T06:02:40.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8b00187740 1 client.17049 shutdown 2024-03-12T06:02:40.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8b00187740 1 -- 172.21.15.76:0/4273608581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580b70 msgr2=0x55b18458f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8b00187740 1 --2- 172.21.15.76:0/4273608581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580b70 0x55b18458f9e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f8aec011800 tx=0x7f8aec011700 comp rx=0 tx=0).stop 2024-03-12T06:02:40.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8b00187740 1 -- 172.21.15.76:0/4273608581 shutdown_connections 2024-03-12T06:02:40.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8b00187740 1 --2- 172.21.15.76:0/4273608581 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55b1845f21f0 0x55b1845f45e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8b00187740 1 --2- 172.21.15.76:0/4273608581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b1845e9090 0x55b1845e9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.923 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8adaff5640 0 client.17049 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:02:40.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8b00187740 1 --2- 172.21.15.76:0/4273608581 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b184589750 0x55b18458ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8b00187740 1 --2- 172.21.15.76:0/4273608581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b184580b70 0x55b18458f9e0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8b00187740 1 -- 172.21.15.76:0/4273608581 >> 172.21.15.76:0/4273608581 conn(0x55b18457a980 msgr2=0x55b184584c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:40.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8b00187740 1 -- 172.21.15.76:0/4273608581 shutdown_connections 2024-03-12T06:02:40.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8b00187740 1 -- 172.21.15.76:0/4273608581 wait complete. 2024-03-12T06:02:40.924 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8b00187740 20 client.17049 trim_cache size 0 max 16384 2024-03-12T06:02:40.925 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:02:40.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8afd4d4640 1 -- 172.21.15.76:0/3652983332 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8ae00a1ba0 msgr2=0x7f8ae00a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8afd4d4640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8ae00a1ba0 0x7f8ae00a4060 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8af00b9c20 tx=0x7f8af008f450 comp rx=0 tx=0).stop 2024-03-12T06:02:40.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8afd4d4640 1 -- 172.21.15.76:0/3652983332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8af81975a0 msgr2=0x7f8af8197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:40.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8afd4d4640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8af81975a0 0x7f8af8197a50 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8af0004af0 tx=0x7f8af0034cc0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8afd4d4640 1 -- 172.21.15.76:0/3652983332 shutdown_connections 2024-03-12T06:02:40.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8afd4d4640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f8ae00a1ba0 0x7f8ae00a4060 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8afd4d4640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8af81975a0 0x7f8af8197a50 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8afd4d4640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af8105360 0x7f8af819d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8afd4d4640 1 --2- 172.21.15.76:0/3652983332 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8af8100290 0x7f8af819d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:40.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.904+0000 7f8afd4d4640 1 -- 172.21.15.76:0/3652983332 >> 172.21.15.76:0/3652983332 conn(0x7f8af8077230 msgr2=0x7f8af80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:40.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.905+0000 7f8afd4d4640 1 -- 172.21.15.76:0/3652983332 shutdown_connections 2024-03-12T06:02:40.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:40.905+0000 7f8afd4d4640 1 -- 172.21.15.76:0/3652983332 wait complete. 2024-03-12T06:02:40.927 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:02:40.928 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T06:02:41.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.027+0000 7fe58c2e1640 1 -- 172.21.15.76:0/761574088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe584100390 msgr2=0x7fe584100770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.027+0000 7fe58c2e1640 1 --2- 172.21.15.76:0/761574088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe584100390 0x7fe584100770 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe580009930 tx=0x7fe58002bc30 comp rx=0 tx=0).stop 2024-03-12T06:02:41.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.027+0000 7fe58c2e1640 1 -- 172.21.15.76:0/761574088 shutdown_connections 2024-03-12T06:02:41.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.027+0000 7fe58c2e1640 1 --2- 172.21.15.76:0/761574088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe584102610 0x7fe58410d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.027+0000 7fe58c2e1640 1 --2- 172.21.15.76:0/761574088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe584101c70 0x7fe5841020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.027+0000 7fe58c2e1640 1 --2- 172.21.15.76:0/761574088 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe584100390 0x7fe584100770 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.027+0000 7fe58c2e1640 1 -- 172.21.15.76:0/761574088 >> 172.21.15.76:0/761574088 conn(0x7fe584077260 msgr2=0x7fe584077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:41.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.027+0000 7fe58c2e1640 1 -- 172.21.15.76:0/761574088 shutdown_connections 2024-03-12T06:02:41.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.027+0000 7fe58c2e1640 1 -- 172.21.15.76:0/761574088 wait complete. 2024-03-12T06:02:41.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58c2e1640 1 Processor -- start 2024-03-12T06:02:41.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58c2e1640 1 -- start start 2024-03-12T06:02:41.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58c2e1640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe584100390 0x7fe5841986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58c2e1640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe584101c70 0x7fe584198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58c2e1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe584102610 0x7fe5841a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58c2e1640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe584199410 con 0x7fe584101c70 2024-03-12T06:02:41.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58c2e1640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe584199580 con 0x7fe584100390 2024-03-12T06:02:41.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58c2e1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5841a3cf0 con 0x7fe584102610 2024-03-12T06:02:41.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58987c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe584101c70 0x7fe584198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58987c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe584101c70 0x7fe584198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52106/0 (socket says 172.21.15.76:52106) 2024-03-12T06:02:41.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58987c640 1 -- 172.21.15.76:0/1299220438 learned_addr learned my addr 172.21.15.76:0/1299220438 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:41.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58a07d640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe584100390 0x7fe5841986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58987c640 1 -- 172.21.15.76:0/1299220438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe584102610 msgr2=0x7fe5841a3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58a87e640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe584102610 0x7fe5841a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58987c640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe584102610 0x7fe5841a3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58987c640 1 -- 172.21.15.76:0/1299220438 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe584100390 msgr2=0x7fe5841986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58987c640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe584100390 0x7fe5841986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58987c640 1 -- 172.21.15.76:0/1299220438 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe580009560 con 0x7fe584101c70 2024-03-12T06:02:41.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.028+0000 7fe58a87e640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe584102610 0x7fe5841a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:41.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.029+0000 7fe58987c640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe584101c70 0x7fe584198c20 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe57400d770 tx=0x7fe57400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:41.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.029+0000 7fe57b7fe640 1 -- 172.21.15.76:0/1299220438 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe574009c30 con 0x7fe584101c70 2024-03-12T06:02:41.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.029+0000 7fe57b7fe640 1 -- 172.21.15.76:0/1299220438 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe57400de80 con 0x7fe584101c70 2024-03-12T06:02:41.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.029+0000 7fe57b7fe640 1 -- 172.21.15.76:0/1299220438 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe574004e80 con 0x7fe584101c70 2024-03-12T06:02:41.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.029+0000 7fe58c2e1640 1 -- 172.21.15.76:0/1299220438 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5841a3fd0 con 0x7fe584101c70 2024-03-12T06:02:41.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.029+0000 7fe58c2e1640 1 -- 172.21.15.76:0/1299220438 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5841a4450 con 0x7fe584101c70 2024-03-12T06:02:41.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.032+0000 7fe58ef8a740 1 Processor -- start 2024-03-12T06:02:41.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.032+0000 7fe57b7fe640 1 -- 172.21.15.76:0/1299220438 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe574005020 con 0x7fe584101c70 2024-03-12T06:02:41.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.032+0000 7fe58ef8a740 1 -- start start 2024-03-12T06:02:41.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.032+0000 7fe58ef8a740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f291630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.032+0000 7fe58ef8a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f29a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.032+0000 7fe58ef8a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f29a750 0x562b9f29cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.032+0000 7fe58ef8a740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b9f24c960 con 0x562b9f291250 2024-03-12T06:02:41.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.032+0000 7fe58ef8a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b9f200310 con 0x562b9f29a750 2024-03-12T06:02:41.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.032+0000 7fe58ef8a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b9f1fd840 con 0x562b9f291b70 2024-03-12T06:02:41.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.032+0000 7fe58a87e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f291630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.032+0000 7fe58a87e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f291630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52110/0 (socket says 172.21.15.76:52110) 2024-03-12T06:02:41.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.032+0000 7fe58a87e640 1 -- 172.21.15.76:0/3942688198 learned_addr learned my addr 172.21.15.76:0/3942688198 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:41.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe57b7fe640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe5700a1c20 0x7fe5700a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe58a07d640 1 --2- 172.21.15.76:0/3942688198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f29a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe58987c640 1 --2- 172.21.15.76:0/3942688198 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f29a750 0x562b9f29cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe58a87e640 1 -- 172.21.15.76:0/3942688198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 msgr2=0x562b9f29a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe58a87e640 1 --2- 172.21.15.76:0/3942688198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f29a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe58a87e640 1 -- 172.21.15.76:0/3942688198 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f29a750 msgr2=0x562b9f29cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe58a87e640 1 --2- 172.21.15.76:0/3942688198 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f29a750 0x562b9f29cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe57b7fe640 1 -- 172.21.15.76:0/1299220438 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fe5740cd210 con 0x7fe584101c70 2024-03-12T06:02:41.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe58a87e640 1 -- 172.21.15.76:0/3942688198 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b9f1bb060 con 0x562b9f291250 2024-03-12T06:02:41.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe58a07d640 1 --2- 172.21.15.76:0/3942688198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f29a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:41.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe58987c640 1 --2- 172.21.15.76:0/3942688198 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f29a750 0x562b9f29cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:41.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe58a87e640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe5700a1c20 0x7fe5700a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe58a87e640 1 --2- 172.21.15.76:0/3942688198 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f291630 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00f360 tx=0x7fe57c00f820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e661a96b91f0ba06 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:41.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe578bfe640 1 -- 172.21.15.76:0/3942688198 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c020030 con 0x562b9f291250 2024-03-12T06:02:41.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe578bfe640 1 -- 172.21.15.76:0/3942688198 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe57c01abb0 con 0x562b9f291250 2024-03-12T06:02:41.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.033+0000 7fe578bfe640 1 -- 172.21.15.76:0/3942688198 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c00d9e0 con 0x562b9f291250 2024-03-12T06:02:41.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.034+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3942688198 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 msgr2=0x562b9f291630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.034+0000 7fe58a87e640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe5700a1c20 0x7fe5700a40e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe57c020000 tx=0x7fe57c01d000 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:41.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.034+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3942688198 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f291630 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00f360 tx=0x7fe57c00f820 comp rx=0 tx=0).stop 2024-03-12T06:02:41.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.034+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3942688198 shutdown_connections 2024-03-12T06:02:41.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.034+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3942688198 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f29a750 0x562b9f29cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.034+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3942688198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f29a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.034+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3942688198 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f291630 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.034+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3942688198 >> 172.21.15.76:0/3942688198 conn(0x562b9f28b980 msgr2=0x562b9f28bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:41.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.034+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3942688198 shutdown_connections 2024-03-12T06:02:41.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.034+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3942688198 wait complete. 2024-03-12T06:02:41.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.034+0000 7fe58ef8a740 1 -- start start 2024-03-12T06:02:41.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.034+0000 7fe58ef8a740 1 Processor -- start 2024-03-12T06:02:41.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58ef8a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f291250 0x562b9f297670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58ef8a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f297bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58ef8a740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f29a750 0x562b9f2980f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58ef8a740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b9f1f9020 con 0x562b9f29a750 2024-03-12T06:02:41.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58ef8a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b9eecc430 con 0x562b9f291250 2024-03-12T06:02:41.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58ef8a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b9ee88e00 con 0x562b9f291b70 2024-03-12T06:02:41.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58ef8a740 10 client.0 _ll_register_callbacks cb 0x562b9f1a0cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:41.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58a87e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f29a750 0x562b9f2980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58a87e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f29a750 0x562b9f2980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52114/0 (socket says 172.21.15.76:52114) 2024-03-12T06:02:41.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58a87e640 1 -- 172.21.15.76:0/3453780221 learned_addr learned my addr 172.21.15.76:0/3453780221 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:41.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58987c640 1 --2- 172.21.15.76:0/3453780221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f297bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58a07d640 1 --2- 172.21.15.76:0/3453780221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f291250 0x562b9f297670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58a87e640 1 -- 172.21.15.76:0/3453780221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 msgr2=0x562b9f297bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58a87e640 1 --2- 172.21.15.76:0/3453780221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f297bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58a87e640 1 -- 172.21.15.76:0/3453780221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f291250 msgr2=0x562b9f297670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58a87e640 1 --2- 172.21.15.76:0/3453780221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f291250 0x562b9f297670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58a87e640 1 -- 172.21.15.76:0/3453780221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57c009b20 con 0x562b9f29a750 2024-03-12T06:02:41.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58a07d640 1 --2- 172.21.15.76:0/3453780221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f291250 0x562b9f297670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:02:41.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58a87e640 1 --2- 172.21.15.76:0/3453780221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f29a750 0x562b9f2980f0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fe57c010010 tx=0x7fe57c009470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6855f1a92383091e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:41.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe56dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:41.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe56dffb640 1 -- 172.21.15.76:0/3453780221 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c00de00 con 0x562b9f29a750 2024-03-12T06:02:41.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe56dffb640 1 -- 172.21.15.76:0/3453780221 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fe57c02a040 con 0x562b9f29a750 2024-03-12T06:02:41.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.035+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3453780221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562b9eef87e0 con 0x562b9f29a750 2024-03-12T06:02:41.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.036+0000 7fe56dffb640 1 -- 172.21.15.76:0/3453780221 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe57c01a4d0 con 0x562b9f29a750 2024-03-12T06:02:41.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.036+0000 7fe58ef8a740 10 client.17061 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:41.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.036+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3453780221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562b9f1f76a0 con 0x562b9f29a750 2024-03-12T06:02:41.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.036+0000 7fe56dffb640 1 -- 172.21.15.76:0/3453780221 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x562b9eef87e0 con 0x562b9f29a750 2024-03-12T06:02:41.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.036+0000 7fe56dffb640 1 -- 172.21.15.76:0/3453780221 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe57c02eba0 con 0x562b9f29a750 2024-03-12T06:02:41.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.036+0000 7fe58ef8a740 10 client.17061 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:41.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.036+0000 7fe58ef8a740 10 client.17061 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:02:41.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.036+0000 7fe58ef8a740 20 client.17061 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:41.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.036+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3453780221 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x562b9f2aa9b0 0x562b9f2acda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.036+0000 7fe58ef8a740 4 client.17061 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:02:41.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.036+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3453780221 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b9ef35720 con 0x562b9f2aa9b0 2024-03-12T06:02:41.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.036+0000 7fe58a07d640 1 --2- 172.21.15.76:0/3453780221 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x562b9f2aa9b0 0x562b9f2acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe56dffb640 10 client.17061 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:02:41.053 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe58a07d640 1 --2- 172.21.15.76:0/3453780221 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x562b9f2aa9b0 0x562b9f2acda0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2b64771873fca664 server_cookie=7af943d119c39572 in_seq=0 out_seq=0 2024-03-12T06:02:41.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe56dffb640 1 -- 172.21.15.76:0/3453780221 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562b9ef35720 con 0x562b9f2aa9b0 2024-03-12T06:02:41.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe56dffb640 10 client.17061 handle_command_reply: tid=0 2024-03-12T06:02:41.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe58ef8a740 1 client.17061 shutdown 2024-03-12T06:02:41.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3453780221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f29a750 msgr2=0x562b9f2980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3453780221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f29a750 0x562b9f2980f0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fe57c010010 tx=0x7fe57c009470 comp rx=0 tx=0).stop 2024-03-12T06:02:41.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3453780221 shutdown_connections 2024-03-12T06:02:41.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3453780221 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x562b9f2aa9b0 0x562b9f2acda0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3453780221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f29a750 0x562b9f2980f0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3453780221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f297bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe56dffb640 0 client.17061 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:02:41.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3453780221 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f291250 0x562b9f297670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3453780221 >> 172.21.15.76:0/3453780221 conn(0x562b9f28b980 msgr2=0x562b9f28d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:41.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.037+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3453780221 shutdown_connections 2024-03-12T06:02:41.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.038+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3453780221 wait complete. 2024-03-12T06:02:41.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.038+0000 7fe58ef8a740 20 client.17061 trim_cache size 0 max 16384 2024-03-12T06:02:41.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.049+0000 7fe58ef8a740 1 Processor -- start 2024-03-12T06:02:41.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.049+0000 7fe58ef8a740 1 -- start start 2024-03-12T06:02:41.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.049+0000 7fe58ef8a740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f299da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.049+0000 7fe58ef8a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f2a1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.049+0000 7fe58ef8a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f29a750 0x562b9f2a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.049+0000 7fe58ef8a740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b9f17e130 con 0x562b9f291250 2024-03-12T06:02:41.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.049+0000 7fe58ef8a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b9f28c980 con 0x562b9f29a750 2024-03-12T06:02:41.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.049+0000 7fe58ef8a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b9f2985c0 con 0x562b9f291b70 2024-03-12T06:02:41.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.049+0000 7fe58a07d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f299da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.049+0000 7fe58a07d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f299da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52124/0 (socket says 172.21.15.76:52124) 2024-03-12T06:02:41.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.049+0000 7fe58a07d640 1 -- 172.21.15.76:0/1266699480 learned_addr learned my addr 172.21.15.76:0/1266699480 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:41.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe58a07d640 1 -- 172.21.15.76:0/1266699480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 msgr2=0x562b9f2a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe58a07d640 1 --2- 172.21.15.76:0/1266699480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f2a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe58a07d640 1 -- 172.21.15.76:0/1266699480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f29a750 msgr2=0x562b9f2a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe58a07d640 1 --2- 172.21.15.76:0/1266699480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f29a750 0x562b9f2a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe58a07d640 1 -- 172.21.15.76:0/1266699480 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b9f193c20 con 0x562b9f291250 2024-03-12T06:02:41.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe58a07d640 1 --2- 172.21.15.76:0/1266699480 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f299da0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fe580005ab0 tx=0x7fe580009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12020fd942b5357f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:41.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe56ffff640 1 -- 172.21.15.76:0/1266699480 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe580048b80 con 0x562b9f291250 2024-03-12T06:02:41.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe56ffff640 1 -- 172.21.15.76:0/1266699480 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe580048ce0 con 0x562b9f291250 2024-03-12T06:02:41.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe56ffff640 1 -- 172.21.15.76:0/1266699480 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5800484f0 con 0x562b9f291250 2024-03-12T06:02:41.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe58ef8a740 1 -- 172.21.15.76:0/1266699480 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 msgr2=0x562b9f299da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/1266699480 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f299da0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fe580005ab0 tx=0x7fe580009260 comp rx=0 tx=0).stop 2024-03-12T06:02:41.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe58ef8a740 1 -- 172.21.15.76:0/1266699480 shutdown_connections 2024-03-12T06:02:41.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/1266699480 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f29a750 0x562b9f2a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/1266699480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f2a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/1266699480 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f299da0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe58ef8a740 1 -- 172.21.15.76:0/1266699480 >> 172.21.15.76:0/1266699480 conn(0x562b9f28b980 msgr2=0x562b9f286a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:41.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.050+0000 7fe58ef8a740 1 -- 172.21.15.76:0/1266699480 shutdown_connections 2024-03-12T06:02:41.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58ef8a740 1 -- 172.21.15.76:0/1266699480 wait complete. 2024-03-12T06:02:41.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58ef8a740 1 -- start start 2024-03-12T06:02:41.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58ef8a740 1 Processor -- start 2024-03-12T06:02:41.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58ef8a740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f2fa900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58ef8a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f2fae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58ef8a740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f29a750 0x562b9f2f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58ef8a740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b9f2f53b0 con 0x562b9f291250 2024-03-12T06:02:41.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58ef8a740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b9f2f5520 con 0x562b9f29a750 2024-03-12T06:02:41.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58ef8a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b9f2f56e0 con 0x562b9f291b70 2024-03-12T06:02:41.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58ef8a740 10 client.0 _ll_register_callbacks cb 0x7fe584069680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:41.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58a07d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f2fa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58a07d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f2fa900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52140/0 (socket says 172.21.15.76:52140) 2024-03-12T06:02:41.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58a07d640 1 -- 172.21.15.76:0/3098529177 learned_addr learned my addr 172.21.15.76:0/3098529177 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:41.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58a07d640 1 -- 172.21.15.76:0/3098529177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 msgr2=0x562b9f2fae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58a07d640 1 --2- 172.21.15.76:0/3098529177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f2fae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58a07d640 1 -- 172.21.15.76:0/3098529177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f29a750 msgr2=0x562b9f2f4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58a07d640 1 --2- 172.21.15.76:0/3098529177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f29a750 0x562b9f2f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.051+0000 7fe58a07d640 1 -- 172.21.15.76:0/3098529177 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58410d910 con 0x562b9f291250 2024-03-12T06:02:41.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe58a07d640 1 --2- 172.21.15.76:0/3098529177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f2fa900 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fe580009230 tx=0x7fe580005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2970a54ca0971d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:41.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe56dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:41.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe56dffb640 1 -- 172.21.15.76:0/3098529177 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe580048490 con 0x562b9f291250 2024-03-12T06:02:41.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe56dffb640 1 -- 172.21.15.76:0/3098529177 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fe5800485f0 con 0x562b9f291250 2024-03-12T06:02:41.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3098529177 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe57c01abb0 con 0x562b9f291250 2024-03-12T06:02:41.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe56dffb640 1 -- 172.21.15.76:0/3098529177 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5800487b0 con 0x562b9f291250 2024-03-12T06:02:41.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe58ef8a740 10 client.17067 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:41.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3098529177 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562b9f1b98c0 con 0x562b9f291250 2024-03-12T06:02:41.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe56dffb640 1 -- 172.21.15.76:0/3098529177 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe580067080 con 0x562b9f291250 2024-03-12T06:02:41.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe56dffb640 1 -- 172.21.15.76:0/3098529177 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe580061030 con 0x562b9f291250 2024-03-12T06:02:41.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe58ef8a740 10 client.17067 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:41.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe58ef8a740 10 client.17067 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:02:41.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe58ef8a740 20 client.17067 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:41.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3098529177 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x562b9f302590 0x562b9f3049b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe58ef8a740 4 client.17067 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:02:41.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3098529177 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562b9ef84890 con 0x562b9f302590 2024-03-12T06:02:41.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.052+0000 7fe58987c640 1 --2- 172.21.15.76:0/3098529177 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x562b9f302590 0x562b9f3049b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe56dffb640 10 client.17067 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:02:41.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe58987c640 1 --2- 172.21.15.76:0/3098529177 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x562b9f302590 0x562b9f3049b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4a2ead3ed4266241 server_cookie=10545801616a8756 in_seq=0 out_seq=0 2024-03-12T06:02:41.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe56dffb640 1 -- 172.21.15.76:0/3098529177 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562b9ef84890 con 0x562b9f302590 2024-03-12T06:02:41.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe56dffb640 10 client.17067 handle_command_reply: tid=0 2024-03-12T06:02:41.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe58ef8a740 1 client.17067 shutdown 2024-03-12T06:02:41.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3098529177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 msgr2=0x562b9f2fa900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3098529177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f2fa900 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fe580009230 tx=0x7fe580005c00 comp rx=0 tx=0).stop 2024-03-12T06:02:41.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3098529177 shutdown_connections 2024-03-12T06:02:41.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3098529177 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x562b9f302590 0x562b9f3049b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3098529177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x562b9f29a750 0x562b9f2f4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3098529177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562b9f291b70 0x562b9f2fae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe56dffb640 0 client.17067 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:02:41.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe58ef8a740 1 --2- 172.21.15.76:0/3098529177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x562b9f291250 0x562b9f2fa900 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3098529177 >> 172.21.15.76:0/3098529177 conn(0x562b9f28b980 msgr2=0x562b9f28a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:41.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3098529177 shutdown_connections 2024-03-12T06:02:41.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe58ef8a740 1 -- 172.21.15.76:0/3098529177 wait complete. 2024-03-12T06:02:41.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.053+0000 7fe58ef8a740 20 client.17067 trim_cache size 0 max 16384 2024-03-12T06:02:41.070 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:02:41.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.054+0000 7fe58c2e1640 1 -- 172.21.15.76:0/1299220438 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe5700a1c20 msgr2=0x7fe5700a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.054+0000 7fe58c2e1640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe5700a1c20 0x7fe5700a40e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe57c020000 tx=0x7fe57c01d000 comp rx=0 tx=0).stop 2024-03-12T06:02:41.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.054+0000 7fe58c2e1640 1 -- 172.21.15.76:0/1299220438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe584101c70 msgr2=0x7fe584198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.054+0000 7fe58c2e1640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe584101c70 0x7fe584198c20 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe57400d770 tx=0x7fe57400dc30 comp rx=0 tx=0).stop 2024-03-12T06:02:41.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.054+0000 7fe58c2e1640 1 -- 172.21.15.76:0/1299220438 shutdown_connections 2024-03-12T06:02:41.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.054+0000 7fe58c2e1640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe5700a1c20 0x7fe5700a40e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.054+0000 7fe58c2e1640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe584102610 0x7fe5841a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.054+0000 7fe58c2e1640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe584101c70 0x7fe584198c20 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.054+0000 7fe58c2e1640 1 --2- 172.21.15.76:0/1299220438 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe584100390 0x7fe5841986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.054+0000 7fe58c2e1640 1 -- 172.21.15.76:0/1299220438 >> 172.21.15.76:0/1299220438 conn(0x7fe584077260 msgr2=0x7fe584103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:41.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.054+0000 7fe58c2e1640 1 -- 172.21.15.76:0/1299220438 shutdown_connections 2024-03-12T06:02:41.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.054+0000 7fe58c2e1640 1 -- 172.21.15.76:0/1299220438 wait complete. 2024-03-12T06:02:41.073 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:02:41.073 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T06:02:41.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.173+0000 7f46e3b9a640 1 -- 172.21.15.76:0/3126024745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46dc101d30 msgr2=0x7f46dc105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.173+0000 7f46e3b9a640 1 --2- 172.21.15.76:0/3126024745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46dc101d30 0x7f46dc105d70 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009930 tx=0x7f46cc02bc30 comp rx=0 tx=0).stop 2024-03-12T06:02:41.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.173+0000 7f46e3b9a640 1 -- 172.21.15.76:0/3126024745 shutdown_connections 2024-03-12T06:02:41.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.173+0000 7f46e3b9a640 1 --2- 172.21.15.76:0/3126024745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46dc106450 0x7f46dc10e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.173+0000 7f46e3b9a640 1 --2- 172.21.15.76:0/3126024745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46dc101d30 0x7f46dc105d70 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.173+0000 7f46e3b9a640 1 --2- 172.21.15.76:0/3126024745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46dc101380 0x7f46dc101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.173+0000 7f46e3b9a640 1 -- 172.21.15.76:0/3126024745 >> 172.21.15.76:0/3126024745 conn(0x7f46dc0fcc20 msgr2=0x7f46dc0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:41.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.173+0000 7f46e3b9a640 1 -- 172.21.15.76:0/3126024745 shutdown_connections 2024-03-12T06:02:41.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.173+0000 7f46e3b9a640 1 -- 172.21.15.76:0/3126024745 wait complete. 2024-03-12T06:02:41.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.174+0000 7f46e3b9a640 1 Processor -- start 2024-03-12T06:02:41.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.174+0000 7f46e3b9a640 1 -- start start 2024-03-12T06:02:41.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.174+0000 7f46e3b9a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46dc101380 0x7f46dc19d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.174+0000 7f46e3b9a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46dc106450 0x7f46dc19dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.174+0000 7f46e3b9a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46dc197630 0x7f46dc197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.174+0000 7f46e3b9a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc198100 con 0x7f46dc101380 2024-03-12T06:02:41.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.174+0000 7f46e3b9a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc198270 con 0x7f46dc106450 2024-03-12T06:02:41.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.174+0000 7f46e3b9a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc198430 con 0x7f46dc197630 2024-03-12T06:02:41.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.174+0000 7f46e1936640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46dc101380 0x7f46dc19d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.174+0000 7f46e1936640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46dc101380 0x7f46dc19d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52154/0 (socket says 172.21.15.76:52154) 2024-03-12T06:02:41.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.174+0000 7f46e1936640 1 -- 172.21.15.76:0/4031748167 learned_addr learned my addr 172.21.15.76:0/4031748167 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:41.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.174+0000 7f46e1135640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46dc106450 0x7f46dc19dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.175+0000 7f46e2137640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46dc197630 0x7f46dc197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.175+0000 7f46e1936640 1 -- 172.21.15.76:0/4031748167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46dc197630 msgr2=0x7f46dc197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.175+0000 7f46e1936640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46dc197630 0x7f46dc197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.175+0000 7f46e1936640 1 -- 172.21.15.76:0/4031748167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46dc106450 msgr2=0x7f46dc19dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.175+0000 7f46e1936640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46dc106450 0x7f46dc19dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.175+0000 7f46e1936640 1 -- 172.21.15.76:0/4031748167 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc009560 con 0x7f46dc101380 2024-03-12T06:02:41.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.175+0000 7f46e1135640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46dc106450 0x7f46dc19dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:02:41.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.175+0000 7f46e1936640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46dc101380 0x7f46dc19d570 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f46d800d950 tx=0x7f46d800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:41.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.175+0000 7f46d2ffd640 1 -- 172.21.15.76:0/4031748167 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8009a30 con 0x7f46dc101380 2024-03-12T06:02:41.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.175+0000 7f46d2ffd640 1 -- 172.21.15.76:0/4031748167 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46d8009b90 con 0x7f46dc101380 2024-03-12T06:02:41.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.175+0000 7f46e3b9a640 1 -- 172.21.15.76:0/4031748167 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46dc1a43d0 con 0x7f46dc101380 2024-03-12T06:02:41.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.175+0000 7f46d2ffd640 1 -- 172.21.15.76:0/4031748167 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8004a00 con 0x7f46dc101380 2024-03-12T06:02:41.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.175+0000 7f46e3b9a640 1 -- 172.21.15.76:0/4031748167 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46dc1a48b0 con 0x7f46dc101380 2024-03-12T06:02:41.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.177+0000 7f46d2ffd640 1 -- 172.21.15.76:0/4031748167 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f46d8009d00 con 0x7f46dc101380 2024-03-12T06:02:41.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.177+0000 7f46e6771740 1 Processor -- start 2024-03-12T06:02:41.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e6771740 1 -- start start 2024-03-12T06:02:41.190 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e6771740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f245f00 0x55c89f2462e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e6771740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f246820 0x55c89f253620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e6771740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f253b60 0x55c89f255f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e6771740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c89f201960 con 0x55c89f246820 2024-03-12T06:02:41.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e6771740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c89f1b5310 con 0x55c89f253b60 2024-03-12T06:02:41.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e6771740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c89f1b2840 con 0x55c89f245f00 2024-03-12T06:02:41.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e1936640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f246820 0x55c89f253620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e1936640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f246820 0x55c89f253620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52166/0 (socket says 172.21.15.76:52166) 2024-03-12T06:02:41.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e1936640 1 -- 172.21.15.76:0/2546617951 learned_addr learned my addr 172.21.15.76:0/2546617951 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:41.191 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e1135640 1 --2- 172.21.15.76:0/2546617951 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f253b60 0x55c89f255f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e2137640 1 --2- 172.21.15.76:0/2546617951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f245f00 0x55c89f2462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46d2ffd640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46c80a1ba0 0x7f46c80a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e2137640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46c80a1ba0 0x7f46c80a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e1936640 1 -- 172.21.15.76:0/2546617951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f245f00 msgr2=0x55c89f2462e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e1936640 1 --2- 172.21.15.76:0/2546617951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f245f00 0x55c89f2462e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e1936640 1 -- 172.21.15.76:0/2546617951 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f253b60 msgr2=0x55c89f255f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e1936640 1 --2- 172.21.15.76:0/2546617951 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f253b60 0x55c89f255f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46d2ffd640 1 -- 172.21.15.76:0/4031748167 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f46d80cd2a0 con 0x7f46dc101380 2024-03-12T06:02:41.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e1936640 1 -- 172.21.15.76:0/2546617951 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c89f170060 con 0x55c89f246820 2024-03-12T06:02:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.178+0000 7f46e1135640 1 --2- 172.21.15.76:0/2546617951 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f253b60 0x55c89f255f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:02:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.179+0000 7f46e1936640 1 --2- 172.21.15.76:0/2546617951 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f246820 0x55c89f253620 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f46d80968a0 tx=0x7f46d8005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c542329be50b8e46 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.179+0000 7f46e2137640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46c80a1ba0 0x7f46c80a4060 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f46d4005c40 tx=0x7f46d4004ab0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.179+0000 7f46c7fff640 1 -- 172.21.15.76:0/2546617951 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8099ca0 con 0x55c89f246820 2024-03-12T06:02:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.179+0000 7f46c7fff640 1 -- 172.21.15.76:0/2546617951 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46d8099a10 con 0x55c89f246820 2024-03-12T06:02:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.179+0000 7f46c7fff640 1 -- 172.21.15.76:0/2546617951 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8099ed0 con 0x55c89f246820 2024-03-12T06:02:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.179+0000 7f46e6771740 1 -- 172.21.15.76:0/2546617951 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f246820 msgr2=0x55c89f253620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 --2- 172.21.15.76:0/2546617951 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f246820 0x55c89f253620 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f46d80968a0 tx=0x7f46d8005340 comp rx=0 tx=0).stop 2024-03-12T06:02:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 -- 172.21.15.76:0/2546617951 shutdown_connections 2024-03-12T06:02:41.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 --2- 172.21.15.76:0/2546617951 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f253b60 0x55c89f255f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 --2- 172.21.15.76:0/2546617951 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f246820 0x55c89f253620 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 --2- 172.21.15.76:0/2546617951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f245f00 0x55c89f2462e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 -- 172.21.15.76:0/2546617951 >> 172.21.15.76:0/2546617951 conn(0x55c89f240980 msgr2=0x55c89f240d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 -- 172.21.15.76:0/2546617951 shutdown_connections 2024-03-12T06:02:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 -- 172.21.15.76:0/2546617951 wait complete. 2024-03-12T06:02:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 -- start start 2024-03-12T06:02:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 Processor -- start 2024-03-12T06:02:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 0x55c89f250ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 0x55c89f251000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f251540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c89f1ae020 con 0x55c89f253b60 2024-03-12T06:02:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c89ee81430 con 0x55c89f245f00 2024-03-12T06:02:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c89ee3de00 con 0x55c89f246820 2024-03-12T06:02:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.180+0000 7f46e6771740 10 client.0 _ll_register_callbacks cb 0x55c89f155cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e2137640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f251540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e2137640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f251540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52170/0 (socket says 172.21.15.76:52170) 2024-03-12T06:02:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e2137640 1 -- 172.21.15.76:0/1850700762 learned_addr learned my addr 172.21.15.76:0/1850700762 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:41.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e1135640 1 --2- 172.21.15.76:0/1850700762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 0x55c89f251000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e1936640 1 --2- 172.21.15.76:0/1850700762 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 0x55c89f250ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e2137640 1 -- 172.21.15.76:0/1850700762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 msgr2=0x55c89f251000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e2137640 1 --2- 172.21.15.76:0/1850700762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 0x55c89f251000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e2137640 1 -- 172.21.15.76:0/1850700762 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 msgr2=0x55c89f250ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e2137640 1 --2- 172.21.15.76:0/1850700762 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 0x55c89f250ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e2137640 1 -- 172.21.15.76:0/1850700762 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46dc1a48b0 con 0x55c89f253b60 2024-03-12T06:02:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e1135640 1 --2- 172.21.15.76:0/1850700762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 0x55c89f251000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:02:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e2137640 1 --2- 172.21.15.76:0/1850700762 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f251540 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f46d4013300 tx=0x7f46d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a91b1f7442f22e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:41.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46c57fa640 1 -- 172.21.15.76:0/1850700762 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d4013330 con 0x55c89f253b60 2024-03-12T06:02:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46c57fa640 1 -- 172.21.15.76:0/1850700762 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f46d4004030 con 0x55c89f253b60 2024-03-12T06:02:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e6771740 1 -- 172.21.15.76:0/1850700762 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c89eead7e0 con 0x55c89f253b60 2024-03-12T06:02:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46c57fa640 1 -- 172.21.15.76:0/1850700762 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46d40041f0 con 0x55c89f253b60 2024-03-12T06:02:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e6771740 10 client.17079 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46e6771740 1 -- 172.21.15.76:0/1850700762 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c89f1ac6a0 con 0x55c89f253b60 2024-03-12T06:02:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.181+0000 7f46c57fa640 1 -- 172.21.15.76:0/1850700762 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c89eead7e0 con 0x55c89f253b60 2024-03-12T06:02:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.182+0000 7f46c57fa640 1 -- 172.21.15.76:0/1850700762 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f46d4020a80 con 0x55c89f253b60 2024-03-12T06:02:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.182+0000 7f46e6771740 10 client.17079 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.182+0000 7f46e6771740 10 client.17079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:41.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.182+0000 7f46e6771740 20 client.17079 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.182+0000 7f46e6771740 1 --2- 172.21.15.76:0/1850700762 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c89f264010 0x55c89f266430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.182+0000 7f46e6771740 4 client.17079 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:02:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.182+0000 7f46e6771740 1 -- 172.21.15.76:0/1850700762 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c89eeea720 con 0x55c89f264010 2024-03-12T06:02:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.182+0000 7f46e1936640 1 --2- 172.21.15.76:0/1850700762 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c89f264010 0x55c89f266430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.182+0000 7f46e1936640 1 --2- 172.21.15.76:0/1850700762 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c89f264010 0x55c89f266430 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4e8f502e0bad729 server_cookie=5d9e39c7074eef26 in_seq=0 out_seq=0 2024-03-12T06:02:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.182+0000 7f46c57fa640 10 client.17079 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46c57fa640 1 -- 172.21.15.76:0/1850700762 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c89eeea720 con 0x55c89f264010 2024-03-12T06:02:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46c57fa640 10 client.17079 handle_command_reply: tid=0 2024-03-12T06:02:41.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46e6771740 1 client.17079 shutdown 2024-03-12T06:02:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46e6771740 1 -- 172.21.15.76:0/1850700762 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 msgr2=0x55c89f251540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46e6771740 1 --2- 172.21.15.76:0/1850700762 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f251540 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f46d4013300 tx=0x7f46d4004a40 comp rx=0 tx=0).stop 2024-03-12T06:02:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46e6771740 1 -- 172.21.15.76:0/1850700762 shutdown_connections 2024-03-12T06:02:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46e6771740 1 --2- 172.21.15.76:0/1850700762 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c89f264010 0x55c89f266430 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46e6771740 1 --2- 172.21.15.76:0/1850700762 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f251540 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46c57fa640 0 client.17079 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46e6771740 1 --2- 172.21.15.76:0/1850700762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 0x55c89f251000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46e6771740 1 --2- 172.21.15.76:0/1850700762 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 0x55c89f250ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46e6771740 1 -- 172.21.15.76:0/1850700762 >> 172.21.15.76:0/1850700762 conn(0x55c89f240980 msgr2=0x55c89f2426d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:41.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46e6771740 1 -- 172.21.15.76:0/1850700762 shutdown_connections 2024-03-12T06:02:41.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46e6771740 1 -- 172.21.15.76:0/1850700762 wait complete. 2024-03-12T06:02:41.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.183+0000 7f46e6771740 20 client.17079 trim_cache size 0 max 16384 2024-03-12T06:02:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e6771740 1 Processor -- start 2024-03-12T06:02:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e6771740 1 -- start start 2024-03-12T06:02:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e6771740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 0x55c89f2531f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e6771740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 0x55c89f251360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e6771740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f24bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e6771740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c89f133130 con 0x55c89f253b60 2024-03-12T06:02:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e6771740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c89f255f50 con 0x55c89f245f00 2024-03-12T06:02:41.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e6771740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c89f25f950 con 0x55c89f246820 2024-03-12T06:02:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e2137640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f24bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e2137640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f24bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52174/0 (socket says 172.21.15.76:52174) 2024-03-12T06:02:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e2137640 1 -- 172.21.15.76:0/3329224 learned_addr learned my addr 172.21.15.76:0/3329224 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e1936640 1 --2- 172.21.15.76:0/3329224 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 0x55c89f2531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e1135640 1 --2- 172.21.15.76:0/3329224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 0x55c89f251360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e2137640 1 -- 172.21.15.76:0/3329224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 msgr2=0x55c89f251360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e2137640 1 --2- 172.21.15.76:0/3329224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 0x55c89f251360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e2137640 1 -- 172.21.15.76:0/3329224 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 msgr2=0x55c89f2531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e2137640 1 --2- 172.21.15.76:0/3329224 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 0x55c89f2531f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e2137640 1 -- 172.21.15.76:0/3329224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c89f148c20 con 0x55c89f253b60 2024-03-12T06:02:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.195+0000 7f46e2137640 1 --2- 172.21.15.76:0/3329224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f24bda0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f46d4006010 tx=0x7f46d4013740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e62941e824aa8349 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.196+0000 7f46c77fe640 1 -- 172.21.15.76:0/3329224 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d4013e30 con 0x55c89f253b60 2024-03-12T06:02:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.196+0000 7f46c77fe640 1 -- 172.21.15.76:0/3329224 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46d4037070 con 0x55c89f253b60 2024-03-12T06:02:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.196+0000 7f46c77fe640 1 -- 172.21.15.76:0/3329224 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d4020400 con 0x55c89f253b60 2024-03-12T06:02:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.196+0000 7f46e6771740 1 -- 172.21.15.76:0/3329224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 msgr2=0x55c89f24bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.196+0000 7f46e6771740 1 --2- 172.21.15.76:0/3329224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f24bda0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f46d4006010 tx=0x7f46d4013740 comp rx=0 tx=0).stop 2024-03-12T06:02:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.196+0000 7f46e6771740 1 -- 172.21.15.76:0/3329224 shutdown_connections 2024-03-12T06:02:41.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.196+0000 7f46e6771740 1 --2- 172.21.15.76:0/3329224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f24bda0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.196+0000 7f46e6771740 1 --2- 172.21.15.76:0/3329224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 0x55c89f251360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.196+0000 7f46e6771740 1 --2- 172.21.15.76:0/3329224 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 0x55c89f2531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.196+0000 7f46e6771740 1 -- 172.21.15.76:0/3329224 >> 172.21.15.76:0/3329224 conn(0x55c89f240980 msgr2=0x55c89f23ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.196+0000 7f46e6771740 1 -- 172.21.15.76:0/3329224 shutdown_connections 2024-03-12T06:02:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.196+0000 7f46e6771740 1 -- 172.21.15.76:0/3329224 wait complete. 2024-03-12T06:02:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.196+0000 7f46e6771740 1 -- start start 2024-03-12T06:02:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e6771740 1 Processor -- start 2024-03-12T06:02:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e6771740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 0x55c89f2593c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e6771740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 0x55c89f259900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e6771740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f259e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e6771740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c89f25b360 con 0x55c89f253b60 2024-03-12T06:02:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e6771740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c89f2ba540 con 0x55c89f245f00 2024-03-12T06:02:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e6771740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c89f2ba680 con 0x55c89f246820 2024-03-12T06:02:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e6771740 10 client.0 _ll_register_callbacks cb 0x7f46d80d0c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e2137640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f259e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e2137640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f259e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52180/0 (socket says 172.21.15.76:52180) 2024-03-12T06:02:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e2137640 1 -- 172.21.15.76:0/31063284 learned_addr learned my addr 172.21.15.76:0/31063284 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:41.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e1135640 1 --2- 172.21.15.76:0/31063284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 0x55c89f259900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e1936640 1 --2- 172.21.15.76:0/31063284 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 0x55c89f2593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e2137640 1 -- 172.21.15.76:0/31063284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 msgr2=0x55c89f259900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e2137640 1 --2- 172.21.15.76:0/31063284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 0x55c89f259900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e2137640 1 -- 172.21.15.76:0/31063284 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 msgr2=0x55c89f2593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e2137640 1 --2- 172.21.15.76:0/31063284 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 0x55c89f2593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e2137640 1 -- 172.21.15.76:0/31063284 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c89f1ac6a0 con 0x55c89f253b60 2024-03-12T06:02:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.197+0000 7f46e2137640 1 --2- 172.21.15.76:0/31063284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f259e40 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f46d4000f80 tx=0x7f46d4003e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a561a4a4c592cb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:41.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46c57fa640 1 -- 172.21.15.76:0/31063284 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d4004d20 con 0x55c89f253b60 2024-03-12T06:02:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46c57fa640 1 -- 172.21.15.76:0/31063284 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f46d400f4f0 con 0x55c89f253b60 2024-03-12T06:02:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46e6771740 1 -- 172.21.15.76:0/31063284 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f46d8099a10 con 0x55c89f253b60 2024-03-12T06:02:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46c57fa640 1 -- 172.21.15.76:0/31063284 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46d4004e80 con 0x55c89f253b60 2024-03-12T06:02:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46e6771740 10 client.17085 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46e6771740 1 -- 172.21.15.76:0/31063284 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c89f16e8c0 con 0x55c89f253b60 2024-03-12T06:02:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46c57fa640 1 -- 172.21.15.76:0/31063284 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46d8099a10 con 0x55c89f253b60 2024-03-12T06:02:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46c57fa640 1 -- 172.21.15.76:0/31063284 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f46d4020970 con 0x55c89f253b60 2024-03-12T06:02:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46e6771740 10 client.17085 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46e6771740 10 client.17085 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46e6771740 20 client.17085 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46e6771740 1 --2- 172.21.15.76:0/31063284 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c89f2bbb70 0x55c89f2bdf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46e6771740 4 client.17085 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:02:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46e6771740 1 -- 172.21.15.76:0/31063284 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c89ef39890 con 0x55c89f2bbb70 2024-03-12T06:02:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.198+0000 7f46e1936640 1 --2- 172.21.15.76:0/31063284 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c89f2bbb70 0x55c89f2bdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46c57fa640 10 client.17085 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46e1936640 1 --2- 172.21.15.76:0/31063284 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c89f2bbb70 0x55c89f2bdf60 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b4e8532f2ddf42e server_cookie=9f5e09552ff7e23b in_seq=0 out_seq=0 2024-03-12T06:02:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46c57fa640 1 -- 172.21.15.76:0/31063284 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c89ef39890 con 0x55c89f2bbb70 2024-03-12T06:02:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46c57fa640 10 client.17085 handle_command_reply: tid=0 2024-03-12T06:02:41.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46e6771740 1 client.17085 shutdown 2024-03-12T06:02:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46e6771740 1 -- 172.21.15.76:0/31063284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 msgr2=0x55c89f259e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46e6771740 1 --2- 172.21.15.76:0/31063284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f259e40 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f46d4000f80 tx=0x7f46d4003e90 comp rx=0 tx=0).stop 2024-03-12T06:02:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46e6771740 1 -- 172.21.15.76:0/31063284 shutdown_connections 2024-03-12T06:02:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46e6771740 1 --2- 172.21.15.76:0/31063284 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c89f2bbb70 0x55c89f2bdf60 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46e6771740 1 --2- 172.21.15.76:0/31063284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c89f253b60 0x55c89f259e40 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46e6771740 1 --2- 172.21.15.76:0/31063284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c89f246820 0x55c89f259900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46e6771740 1 --2- 172.21.15.76:0/31063284 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c89f245f00 0x55c89f2593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46c57fa640 0 client.17085 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:41.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46e6771740 1 -- 172.21.15.76:0/31063284 >> 172.21.15.76:0/31063284 conn(0x55c89f240980 msgr2=0x55c89f24bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:41.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46e6771740 1 -- 172.21.15.76:0/31063284 shutdown_connections 2024-03-12T06:02:41.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46e6771740 1 -- 172.21.15.76:0/31063284 wait complete. 2024-03-12T06:02:41.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.199+0000 7f46e6771740 20 client.17085 trim_cache size 0 max 16384 2024-03-12T06:02:41.215 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:02:41.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.200+0000 7f46e3b9a640 1 -- 172.21.15.76:0/4031748167 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46c80a1ba0 msgr2=0x7f46c80a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.200+0000 7f46e3b9a640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46c80a1ba0 0x7f46c80a4060 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f46d4005c40 tx=0x7f46d4004ab0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.200+0000 7f46e3b9a640 1 -- 172.21.15.76:0/4031748167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46dc101380 msgr2=0x7f46dc19d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.200+0000 7f46e3b9a640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46dc101380 0x7f46dc19d570 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f46d800d950 tx=0x7f46d800de10 comp rx=0 tx=0).stop 2024-03-12T06:02:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.200+0000 7f46e3b9a640 1 -- 172.21.15.76:0/4031748167 shutdown_connections 2024-03-12T06:02:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.200+0000 7f46e3b9a640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f46c80a1ba0 0x7f46c80a4060 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.200+0000 7f46e3b9a640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46dc197630 0x7f46dc197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.200+0000 7f46e3b9a640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f46dc106450 0x7f46dc19dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.200+0000 7f46e3b9a640 1 --2- 172.21.15.76:0/4031748167 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46dc101380 0x7f46dc19d570 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.200+0000 7f46e3b9a640 1 -- 172.21.15.76:0/4031748167 >> 172.21.15.76:0/4031748167 conn(0x7f46dc0fcc20 msgr2=0x7f46dc0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.200+0000 7f46e3b9a640 1 -- 172.21.15.76:0/4031748167 shutdown_connections 2024-03-12T06:02:41.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:41.200+0000 7f46e3b9a640 1 -- 172.21.15.76:0/4031748167 wait complete. 2024-03-12T06:02:41.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:41 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1689206591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:02:41.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:41 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1689206591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:02:41.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:41 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1689206591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:02:42.219 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T06:02:42.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:02:42.219 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T06:02:42.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 -- 172.21.15.76:0/79109034 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fccbc0735b0 msgr2=0x7fccbc073990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 --2- 172.21.15.76:0/79109034 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fccbc0735b0 0x7fccbc073990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009900 tx=0x7fccb802bcd0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.341 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 -- 172.21.15.76:0/79109034 shutdown_connections 2024-03-12T06:02:42.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 --2- 172.21.15.76:0/79109034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccbc10d950 0x7fccbc10fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 --2- 172.21.15.76:0/79109034 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fccbc073ed0 0x7fccbc10d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 --2- 172.21.15.76:0/79109034 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fccbc0735b0 0x7fccbc073990 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 -- 172.21.15.76:0/79109034 >> 172.21.15.76:0/79109034 conn(0x7fccbc0fcc20 msgr2=0x7fccbc0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:42.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 -- 172.21.15.76:0/79109034 shutdown_connections 2024-03-12T06:02:42.342 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 -- 172.21.15.76:0/79109034 wait complete. 2024-03-12T06:02:42.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 Processor -- start 2024-03-12T06:02:42.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 -- start start 2024-03-12T06:02:42.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fccbc0735b0 0x7fccbc19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccbc073ed0 0x7fccbc19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fccbc10d950 0x7fccbc199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc199f30 con 0x7fccbc10d950 2024-03-12T06:02:42.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc19a0a0 con 0x7fccbc0735b0 2024-03-12T06:02:42.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.334+0000 7fccc333a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccbc19a260 con 0x7fccbc073ed0 2024-03-12T06:02:42.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccc18d7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fccbc10d950 0x7fccbc199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccc18d7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fccbc10d950 0x7fccbc199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33570/0 (socket says 172.21.15.76:33570) 2024-03-12T06:02:42.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccc18d7640 1 -- 172.21.15.76:0/2842538079 learned_addr learned my addr 172.21.15.76:0/2842538079 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:42.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccc10d6640 1 --2- 172.21.15.76:0/2842538079 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fccbc0735b0 0x7fccbc19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccc08d5640 1 --2- 172.21.15.76:0/2842538079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccbc073ed0 0x7fccbc19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccc18d7640 1 -- 172.21.15.76:0/2842538079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccbc073ed0 msgr2=0x7fccbc19fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccc18d7640 1 --2- 172.21.15.76:0/2842538079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccbc073ed0 0x7fccbc19fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccc18d7640 1 -- 172.21.15.76:0/2842538079 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fccbc0735b0 msgr2=0x7fccbc19f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccc18d7640 1 --2- 172.21.15.76:0/2842538079 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fccbc0735b0 0x7fccbc19f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccc18d7640 1 -- 172.21.15.76:0/2842538079 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb8009560 con 0x7fccbc10d950 2024-03-12T06:02:42.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccc18d7640 1 --2- 172.21.15.76:0/2842538079 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fccbc10d950 0x7fccbc199890 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fccb400b970 tx=0x7fccb400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:42.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccb27fc640 1 -- 172.21.15.76:0/2842538079 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb400cb40 con 0x7fccbc10d950 2024-03-12T06:02:42.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccc333a640 1 -- 172.21.15.76:0/2842538079 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccbc19a540 con 0x7fccbc10d950 2024-03-12T06:02:42.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccb27fc640 1 -- 172.21.15.76:0/2842538079 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccb400cca0 con 0x7fccbc10d950 2024-03-12T06:02:42.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccb27fc640 1 -- 172.21.15.76:0/2842538079 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb40077d0 con 0x7fccbc10d950 2024-03-12T06:02:42.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.335+0000 7fccc333a640 1 -- 172.21.15.76:0/2842538079 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccbc1a6700 con 0x7fccbc10d950 2024-03-12T06:02:42.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.337+0000 7fccb27fc640 1 -- 172.21.15.76:0/2842538079 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fccb4014440 con 0x7fccbc10d950 2024-03-12T06:02:42.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccc5fe1740 1 Processor -- start 2024-03-12T06:02:42.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccc5fe1740 1 -- start start 2024-03-12T06:02:42.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccb27fc640 1 --2- 172.21.15.76:0/2842538079 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcca80a1ba0 0x7fcca80a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccc5fe1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bd380 0x55c98a7bd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccc5fe1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7bdd30 0x55c98a7c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccc5fe1740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7c68b0 0x55c98a7c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccc5fe1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c98a72bf90 con 0x55c98a7bd380 2024-03-12T06:02:42.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccc5fe1740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c98a7294c0 con 0x55c98a7c68b0 2024-03-12T06:02:42.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccc5fe1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c98a724ca0 con 0x55c98a7bdd30 2024-03-12T06:02:42.353 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccc08d5640 1 --2- 172.21.15.76:0/2842538079 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcca80a1ba0 0x7fcca80a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccc10d6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bd380 0x55c98a7bd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccb27fc640 1 -- 172.21.15.76:0/2842538079 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fccb40ccf30 con 0x7fccbc10d950 2024-03-12T06:02:42.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccc10d6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bd380 0x55c98a7bd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33572/0 (socket says 172.21.15.76:33572) 2024-03-12T06:02:42.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccc10d6640 1 -- 172.21.15.76:0/4023441761 learned_addr learned my addr 172.21.15.76:0/4023441761 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:42.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccc08d5640 1 --2- 172.21.15.76:0/4023441761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7bdd30 0x55c98a7c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.338+0000 7fccc18d7640 1 --2- 172.21.15.76:0/4023441761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7c68b0 0x55c98a7c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.339+0000 7fccc10d6640 1 -- 172.21.15.76:0/4023441761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7bdd30 msgr2=0x55c98a7c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.339+0000 7fccc10d6640 1 --2- 172.21.15.76:0/4023441761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7bdd30 0x55c98a7c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.339+0000 7fccc10d6640 1 -- 172.21.15.76:0/4023441761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7c68b0 msgr2=0x55c98a7c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.339+0000 7fccc10d6640 1 --2- 172.21.15.76:0/4023441761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7c68b0 0x55c98a7c8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.339+0000 7fccc10d6640 1 -- 172.21.15.76:0/4023441761 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c98a6e70b0 con 0x55c98a7bd380 2024-03-12T06:02:42.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.339+0000 7fccc08d5640 1 --2- 172.21.15.76:0/2842538079 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcca80a1ba0 0x7fcca80a4060 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fccbc0ff2b0 tx=0x7fccac0138e0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:42.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.339+0000 7fccc10d6640 1 --2- 172.21.15.76:0/4023441761 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bd380 0x55c98a7bd760 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009900 tx=0x7fccb802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b90a2c3c319689fc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:42.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.339+0000 7fcca77fe640 1 -- 172.21.15.76:0/4023441761 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8004980 con 0x55c98a7bd380 2024-03-12T06:02:42.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.339+0000 7fcca77fe640 1 -- 172.21.15.76:0/4023441761 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccb8004ae0 con 0x55c98a7bd380 2024-03-12T06:02:42.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.339+0000 7fcca77fe640 1 -- 172.21.15.76:0/4023441761 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8005280 con 0x55c98a7bd380 2024-03-12T06:02:42.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.340+0000 7fccc5fe1740 1 -- 172.21.15.76:0/4023441761 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bd380 msgr2=0x55c98a7bd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.340+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/4023441761 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bd380 0x55c98a7bd760 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009900 tx=0x7fccb802c7e0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.340+0000 7fccc5fe1740 1 -- 172.21.15.76:0/4023441761 shutdown_connections 2024-03-12T06:02:42.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.340+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/4023441761 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7c68b0 0x55c98a7c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.340+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/4023441761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7bdd30 0x55c98a7c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.340+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/4023441761 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bd380 0x55c98a7bd760 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.340+0000 7fccc5fe1740 1 -- 172.21.15.76:0/4023441761 >> 172.21.15.76:0/4023441761 conn(0x55c98a7b7840 msgr2=0x55c98a7b7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:42.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.340+0000 7fccc5fe1740 1 -- 172.21.15.76:0/4023441761 shutdown_connections 2024-03-12T06:02:42.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.340+0000 7fccc5fe1740 1 -- 172.21.15.76:0/4023441761 wait complete. 2024-03-12T06:02:42.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.340+0000 7fccc5fe1740 1 -- start start 2024-03-12T06:02:42.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.340+0000 7fccc5fe1740 1 Processor -- start 2024-03-12T06:02:42.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc5fe1740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7bd380 0x55c98a7c3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc5fe1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7bdd30 0x55c98a7c3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc5fe1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7c68b0 0x55c98a7c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc5fe1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c98a6c9f40 con 0x55c98a7c68b0 2024-03-12T06:02:42.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc5fe1740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c98a3f84b0 con 0x55c98a7bd380 2024-03-12T06:02:42.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc5fe1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c98a6aa670 con 0x55c98a7bdd30 2024-03-12T06:02:42.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc5fe1740 10 client.0 _ll_register_callbacks cb 0x55c98a717f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:42.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc18d7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7c68b0 0x55c98a7c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc18d7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7c68b0 0x55c98a7c41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33588/0 (socket says 172.21.15.76:33588) 2024-03-12T06:02:42.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc18d7640 1 -- 172.21.15.76:0/2428665688 learned_addr learned my addr 172.21.15.76:0/2428665688 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:42.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc10d6640 1 --2- 172.21.15.76:0/2428665688 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7bd380 0x55c98a7c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc18d7640 1 -- 172.21.15.76:0/2428665688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7bdd30 msgr2=0x55c98a7c3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc18d7640 1 --2- 172.21.15.76:0/2428665688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7bdd30 0x55c98a7c3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc18d7640 1 -- 172.21.15.76:0/2428665688 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7bd380 msgr2=0x55c98a7c3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc18d7640 1 --2- 172.21.15.76:0/2428665688 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7bd380 0x55c98a7c3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc18d7640 1 -- 172.21.15.76:0/2428665688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccbc110a70 con 0x55c98a7c68b0 2024-03-12T06:02:42.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc18d7640 1 --2- 172.21.15.76:0/2428665688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7c68b0 0x55c98a7c41d0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fccb4096180 tx=0x7fccb4099740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bcae518fbe3bb0a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:42.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fcca4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:42.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fcca4ff9640 1 -- 172.21.15.76:0/2428665688 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb40b1070 con 0x55c98a7c68b0 2024-03-12T06:02:42.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fcca4ff9640 1 -- 172.21.15.76:0/2428665688 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fccb409ddc0 con 0x55c98a7c68b0 2024-03-12T06:02:42.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc5fe1740 1 -- 172.21.15.76:0/2428665688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c98a6fef40 con 0x55c98a7c68b0 2024-03-12T06:02:42.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fcca4ff9640 1 -- 172.21.15.76:0/2428665688 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccb40b81a0 con 0x55c98a7c68b0 2024-03-12T06:02:42.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc5fe1740 10 client.17097 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:42.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.341+0000 7fccc5fe1740 1 -- 172.21.15.76:0/2428665688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c98a723320 con 0x55c98a7c68b0 2024-03-12T06:02:42.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.342+0000 7fcca4ff9640 1 -- 172.21.15.76:0/2428665688 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c98a6fef40 con 0x55c98a7c68b0 2024-03-12T06:02:42.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.342+0000 7fcca4ff9640 1 -- 172.21.15.76:0/2428665688 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fccb40491f0 con 0x55c98a7c68b0 2024-03-12T06:02:42.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.342+0000 7fccc5fe1740 10 client.17097 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:42.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.342+0000 7fccc5fe1740 10 client.17097 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:42.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.342+0000 7fccc5fe1740 20 client.17097 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:42.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.342+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/2428665688 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c98a7d6c10 0x55c98a7d9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.342+0000 7fccc5fe1740 4 client.17097 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:02:42.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.342+0000 7fccc5fe1740 1 -- 172.21.15.76:0/2428665688 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c98a71b310 con 0x55c98a7d6c10 2024-03-12T06:02:42.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.342+0000 7fccc10d6640 1 --2- 172.21.15.76:0/2428665688 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c98a7d6c10 0x55c98a7d9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.342+0000 7fcca4ff9640 10 client.17097 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:42.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.342+0000 7fccc10d6640 1 --2- 172.21.15.76:0/2428665688 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c98a7d6c10 0x55c98a7d9030 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1f69cd923902c0e server_cookie=c07ef1a09a5b982 in_seq=0 out_seq=0 2024-03-12T06:02:42.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.343+0000 7fcca4ff9640 1 -- 172.21.15.76:0/2428665688 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c98a71b310 con 0x55c98a7d6c10 2024-03-12T06:02:42.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.343+0000 7fcca4ff9640 10 client.17097 handle_command_reply: tid=0 2024-03-12T06:02:42.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.343+0000 7fccc5fe1740 1 client.17097 shutdown 2024-03-12T06:02:42.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.343+0000 7fccc5fe1740 1 -- 172.21.15.76:0/2428665688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7c68b0 msgr2=0x55c98a7c41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.343+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/2428665688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7c68b0 0x55c98a7c41d0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fccb4096180 tx=0x7fccb4099740 comp rx=0 tx=0).stop 2024-03-12T06:02:42.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.343+0000 7fccc5fe1740 1 -- 172.21.15.76:0/2428665688 shutdown_connections 2024-03-12T06:02:42.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.343+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/2428665688 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c98a7d6c10 0x55c98a7d9030 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.343+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/2428665688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7c68b0 0x55c98a7c41d0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.343+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/2428665688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7bdd30 0x55c98a7c3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.343+0000 7fcca4ff9640 0 client.17097 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:42.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.343+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/2428665688 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7bd380 0x55c98a7c3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.343+0000 7fccc5fe1740 1 -- 172.21.15.76:0/2428665688 >> 172.21.15.76:0/2428665688 conn(0x55c98a7b7840 msgr2=0x55c98a7b97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:42.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.343+0000 7fccc5fe1740 1 -- 172.21.15.76:0/2428665688 shutdown_connections 2024-03-12T06:02:42.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.343+0000 7fccc5fe1740 1 -- 172.21.15.76:0/2428665688 wait complete. 2024-03-12T06:02:42.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.344+0000 7fccc5fe1740 20 client.17097 trim_cache size 0 max 16384 2024-03-12T06:02:42.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.358+0000 7fccc5fe1740 1 Processor -- start 2024-03-12T06:02:42.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.358+0000 7fccc5fe1740 1 -- start start 2024-03-12T06:02:42.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc5fe1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7bd380 0x55c98a7c28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc5fe1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bdd30 0x55c98a7c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc5fe1740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7c68b0 0x55c98a7b6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc5fe1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c98a7b31b0 con 0x55c98a7bdd30 2024-03-12T06:02:42.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc5fe1740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c98a7b8650 con 0x55c98a7c68b0 2024-03-12T06:02:42.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc5fe1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c98a7c2e70 con 0x55c98a7bd380 2024-03-12T06:02:42.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc08d5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bdd30 0x55c98a7c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc08d5640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bdd30 0x55c98a7c3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33590/0 (socket says 172.21.15.76:33590) 2024-03-12T06:02:42.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc08d5640 1 -- 172.21.15.76:0/754158066 learned_addr learned my addr 172.21.15.76:0/754158066 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:42.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc18d7640 1 --2- 172.21.15.76:0/754158066 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7c68b0 0x55c98a7b6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc10d6640 1 --2- 172.21.15.76:0/754158066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7bd380 0x55c98a7c28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc08d5640 1 -- 172.21.15.76:0/754158066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7bd380 msgr2=0x55c98a7c28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc08d5640 1 --2- 172.21.15.76:0/754158066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7bd380 0x55c98a7c28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc08d5640 1 -- 172.21.15.76:0/754158066 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7c68b0 msgr2=0x55c98a7b6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc08d5640 1 --2- 172.21.15.76:0/754158066 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7c68b0 0x55c98a7b6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc08d5640 1 -- 172.21.15.76:0/754158066 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c98a6bfc00 con 0x55c98a7bdd30 2024-03-12T06:02:42.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fccc08d5640 1 --2- 172.21.15.76:0/754158066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bdd30 0x55c98a7c3ff0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fccbc19ab90 tx=0x7fccac019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3d11599f897634 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:42.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fcca6ffd640 1 -- 172.21.15.76:0/754158066 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac020620 con 0x55c98a7bdd30 2024-03-12T06:02:42.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fcca6ffd640 1 -- 172.21.15.76:0/754158066 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccac019070 con 0x55c98a7bdd30 2024-03-12T06:02:42.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.359+0000 7fcca6ffd640 1 -- 172.21.15.76:0/754158066 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac01fa20 con 0x55c98a7bdd30 2024-03-12T06:02:42.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.360+0000 7fccc5fe1740 1 -- 172.21.15.76:0/754158066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bdd30 msgr2=0x55c98a7c3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.360+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/754158066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bdd30 0x55c98a7c3ff0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fccbc19ab90 tx=0x7fccac019000 comp rx=0 tx=0).stop 2024-03-12T06:02:42.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.360+0000 7fccc5fe1740 1 -- 172.21.15.76:0/754158066 shutdown_connections 2024-03-12T06:02:42.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.360+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/754158066 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7c68b0 0x55c98a7b6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.360+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/754158066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bdd30 0x55c98a7c3ff0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.360+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/754158066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7bd380 0x55c98a7c28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.360+0000 7fccc5fe1740 1 -- 172.21.15.76:0/754158066 >> 172.21.15.76:0/754158066 conn(0x55c98a7b7840 msgr2=0x55c98a7b6050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:42.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.360+0000 7fccc5fe1740 1 -- 172.21.15.76:0/754158066 shutdown_connections 2024-03-12T06:02:42.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.360+0000 7fccc5fe1740 1 -- 172.21.15.76:0/754158066 wait complete. 2024-03-12T06:02:42.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.360+0000 7fccc5fe1740 1 -- start start 2024-03-12T06:02:42.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.360+0000 7fccc5fe1740 1 Processor -- start 2024-03-12T06:02:42.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc5fe1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bd380 0x55c98a7ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc5fe1740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7bdd30 0x55c98a7cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc5fe1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7c68b0 0x55c98a8271a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc5fe1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c98a8276e0 con 0x55c98a7bd380 2024-03-12T06:02:42.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc5fe1740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c98a827850 con 0x55c98a7bdd30 2024-03-12T06:02:42.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc5fe1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c98a827a10 con 0x55c98a7c68b0 2024-03-12T06:02:42.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc5fe1740 10 client.0 _ll_register_callbacks cb 0x7fccb8005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:42.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc10d6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bd380 0x55c98a7ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc10d6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bd380 0x55c98a7ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33606/0 (socket says 172.21.15.76:33606) 2024-03-12T06:02:42.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc10d6640 1 -- 172.21.15.76:0/1337976697 learned_addr learned my addr 172.21.15.76:0/1337976697 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:42.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc18d7640 1 --2- 172.21.15.76:0/1337976697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7c68b0 0x55c98a8271a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc08d5640 1 --2- 172.21.15.76:0/1337976697 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7bdd30 0x55c98a7cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc10d6640 1 -- 172.21.15.76:0/1337976697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7c68b0 msgr2=0x55c98a8271a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc10d6640 1 --2- 172.21.15.76:0/1337976697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7c68b0 0x55c98a8271a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc10d6640 1 -- 172.21.15.76:0/1337976697 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7bdd30 msgr2=0x55c98a7cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc10d6640 1 --2- 172.21.15.76:0/1337976697 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7bdd30 0x55c98a7cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc10d6640 1 -- 172.21.15.76:0/1337976697 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c98a6e5910 con 0x55c98a7bd380 2024-03-12T06:02:42.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc10d6640 1 --2- 172.21.15.76:0/1337976697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bd380 0x55c98a7ccae0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fccb802c210 tx=0x7fccb8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=271ee4601dcf665 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:42.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fcca4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:42.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fcca4ff9640 1 -- 172.21.15.76:0/1337976697 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8056070 con 0x55c98a7bd380 2024-03-12T06:02:42.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fcca4ff9640 1 -- 172.21.15.76:0/1337976697 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fccb804f040 con 0x55c98a7bd380 2024-03-12T06:02:42.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.361+0000 7fccc5fe1740 1 -- 172.21.15.76:0/1337976697 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c98a74a900 con 0x55c98a7bd380 2024-03-12T06:02:42.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.362+0000 7fcca4ff9640 1 -- 172.21.15.76:0/1337976697 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccb8004310 con 0x55c98a7bd380 2024-03-12T06:02:42.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.362+0000 7fccc5fe1740 10 client.17103 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:42.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.362+0000 7fccc5fe1740 1 -- 172.21.15.76:0/1337976697 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c98a6cc0a0 con 0x55c98a7bd380 2024-03-12T06:02:42.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.362+0000 7fcca4ff9640 1 -- 172.21.15.76:0/1337976697 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c98a74a900 con 0x55c98a7bd380 2024-03-12T06:02:42.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.362+0000 7fcca4ff9640 1 -- 172.21.15.76:0/1337976697 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fccb8059020 con 0x55c98a7bd380 2024-03-12T06:02:42.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.362+0000 7fccc5fe1740 10 client.17103 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:42.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.362+0000 7fccc5fe1740 10 client.17103 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:42.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.362+0000 7fccc5fe1740 20 client.17103 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:42.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.362+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/1337976697 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c98a82eec0 0x55c98a8312e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.362+0000 7fccc5fe1740 4 client.17103 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T06:02:42.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.362+0000 7fccc5fe1740 1 -- 172.21.15.76:0/1337976697 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c98a424860 con 0x55c98a82eec0 2024-03-12T06:02:42.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.362+0000 7fccc08d5640 1 --2- 172.21.15.76:0/1337976697 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c98a82eec0 0x55c98a8312e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.363+0000 7fcca4ff9640 10 client.17103 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:42.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.363+0000 7fccc08d5640 1 --2- 172.21.15.76:0/1337976697 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c98a82eec0 0x55c98a8312e0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4a02674e9fc6769 server_cookie=ec68325a27e660a5 in_seq=0 out_seq=0 2024-03-12T06:02:42.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fcca4ff9640 1 -- 172.21.15.76:0/1337976697 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c98a424860 con 0x55c98a82eec0 2024-03-12T06:02:42.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fcca4ff9640 10 client.17103 handle_command_reply: tid=0 2024-03-12T06:02:42.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fccc5fe1740 1 client.17103 shutdown 2024-03-12T06:02:42.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fccc5fe1740 1 -- 172.21.15.76:0/1337976697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bd380 msgr2=0x55c98a7ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/1337976697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bd380 0x55c98a7ccae0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fccb802c210 tx=0x7fccb8000bc0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fccc5fe1740 1 -- 172.21.15.76:0/1337976697 shutdown_connections 2024-03-12T06:02:42.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/1337976697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c98a7c68b0 0x55c98a8271a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/1337976697 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c98a7bdd30 0x55c98a7cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/1337976697 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c98a82eec0 0x55c98a8312e0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fccc5fe1740 1 --2- 172.21.15.76:0/1337976697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c98a7bd380 0x55c98a7ccae0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fcca4ff9640 0 client.17103 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:42.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fccc5fe1740 1 -- 172.21.15.76:0/1337976697 >> 172.21.15.76:0/1337976697 conn(0x55c98a7b7840 msgr2=0x55c98a7b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:42.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fccc5fe1740 1 -- 172.21.15.76:0/1337976697 shutdown_connections 2024-03-12T06:02:42.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fccc5fe1740 1 -- 172.21.15.76:0/1337976697 wait complete. 2024-03-12T06:02:42.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.364+0000 7fccc5fe1740 20 client.17103 trim_cache size 0 max 16384 2024-03-12T06:02:42.382 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:02:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T06:02:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "4a64affa-5c88-475e-92ec-33d7ad2c6b05", 2024-03-12T06:02:42.383 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T06:02:42.383 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:02:42.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.365+0000 7fccc333a640 1 -- 172.21.15.76:0/2842538079 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcca80a1ba0 msgr2=0x7fcca80a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.365+0000 7fccc333a640 1 --2- 172.21.15.76:0/2842538079 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcca80a1ba0 0x7fcca80a4060 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fccbc0ff2b0 tx=0x7fccac0138e0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.365+0000 7fccc333a640 1 -- 172.21.15.76:0/2842538079 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fccbc10d950 msgr2=0x7fccbc199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.365+0000 7fccc333a640 1 --2- 172.21.15.76:0/2842538079 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fccbc10d950 0x7fccbc199890 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fccb400b970 tx=0x7fccb400be30 comp rx=0 tx=0).stop 2024-03-12T06:02:42.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.365+0000 7fccc333a640 1 -- 172.21.15.76:0/2842538079 shutdown_connections 2024-03-12T06:02:42.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.365+0000 7fccc333a640 1 --2- 172.21.15.76:0/2842538079 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcca80a1ba0 0x7fcca80a4060 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.365+0000 7fccc333a640 1 --2- 172.21.15.76:0/2842538079 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fccbc10d950 0x7fccbc199890 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.365+0000 7fccc333a640 1 --2- 172.21.15.76:0/2842538079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccbc073ed0 0x7fccbc19fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.365+0000 7fccc333a640 1 --2- 172.21.15.76:0/2842538079 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fccbc0735b0 0x7fccbc19f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.365+0000 7fccc333a640 1 -- 172.21.15.76:0/2842538079 >> 172.21.15.76:0/2842538079 conn(0x7fccbc0fcc20 msgr2=0x7fccbc0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:42.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.365+0000 7fccc333a640 1 -- 172.21.15.76:0/2842538079 shutdown_connections 2024-03-12T06:02:42.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.365+0000 7fccc333a640 1 -- 172.21.15.76:0/2842538079 wait complete. 2024-03-12T06:02:42.386 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:02:42.386 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:02:42.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:42 smithi082 ceph-mon[36382]: pgmap v212: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:42.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:42 smithi088 ceph-mon[36585]: pgmap v212: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:42.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:42 smithi076 ceph-mon[32095]: pgmap v212: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.505+0000 7f673258a640 1 -- 172.21.15.76:0/2816441224 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f672c105340 msgr2=0x7f672c10cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.505+0000 7f673258a640 1 --2- 172.21.15.76:0/2816441224 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f672c105340 0x7f672c10cf10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6724009c10 tx=0x7f672402bee0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.505+0000 7f673258a640 1 -- 172.21.15.76:0/2816441224 shutdown_connections 2024-03-12T06:02:42.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.505+0000 7f673258a640 1 --2- 172.21.15.76:0/2816441224 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f672c105340 0x7f672c10cf10 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.505+0000 7f673258a640 1 --2- 172.21.15.76:0/2816441224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f672c100c30 0x7f672c104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.505+0000 7f673258a640 1 --2- 172.21.15.76:0/2816441224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f672c100280 0x7f672c100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.505+0000 7f673258a640 1 -- 172.21.15.76:0/2816441224 >> 172.21.15.76:0/2816441224 conn(0x7f672c077120 msgr2=0x7f672c077530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:42.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.505+0000 7f673258a640 1 -- 172.21.15.76:0/2816441224 shutdown_connections 2024-03-12T06:02:42.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.505+0000 7f673258a640 1 -- 172.21.15.76:0/2816441224 wait complete. 2024-03-12T06:02:42.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.506+0000 7f673258a640 1 Processor -- start 2024-03-12T06:02:42.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.506+0000 7f673258a640 1 -- start start 2024-03-12T06:02:42.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.506+0000 7f673258a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f672c100280 0x7f672c198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.506+0000 7f673258a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f672c105340 0x7f672c198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.506+0000 7f673258a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f672c199040 0x7f672c1a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.506+0000 7f673258a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c1a3c90 con 0x7f672c105340 2024-03-12T06:02:42.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.506+0000 7f673258a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c1a3e00 con 0x7f672c100280 2024-03-12T06:02:42.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.506+0000 7f673258a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c1a3fc0 con 0x7f672c199040 2024-03-12T06:02:42.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.506+0000 7f672bfff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f672c100280 0x7f672c198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.506+0000 7f672b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f672c105340 0x7f672c198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.506+0000 7f672bfff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f672c100280 0x7f672c198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:53390/0 (socket says 172.21.15.76:53390) 2024-03-12T06:02:42.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.506+0000 7f672bfff640 1 -- 172.21.15.76:0/4279254388 learned_addr learned my addr 172.21.15.76:0/4279254388 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:42.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.506+0000 7f6730b27640 1 --2- 172.21.15.76:0/4279254388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f672c199040 0x7f672c1a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.506+0000 7f672bfff640 1 -- 172.21.15.76:0/4279254388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f672c199040 msgr2=0x7f672c1a35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.507+0000 7f672bfff640 1 --2- 172.21.15.76:0/4279254388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f672c199040 0x7f672c1a35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.507+0000 7f672bfff640 1 -- 172.21.15.76:0/4279254388 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f672c105340 msgr2=0x7f672c198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.507+0000 7f672bfff640 1 --2- 172.21.15.76:0/4279254388 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f672c105340 0x7f672c198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.507+0000 7f672bfff640 1 -- 172.21.15.76:0/4279254388 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6724009840 con 0x7f672c100280 2024-03-12T06:02:42.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.507+0000 7f672bfff640 1 --2- 172.21.15.76:0/4279254388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f672c100280 0x7f672c198520 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f671c00e9c0 tx=0x7f671c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:42.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.507+0000 7f67297fa640 1 -- 172.21.15.76:0/4279254388 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c00ccc0 con 0x7f672c100280 2024-03-12T06:02:42.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.507+0000 7f67297fa640 1 -- 172.21.15.76:0/4279254388 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f671c00ce20 con 0x7f672c100280 2024-03-12T06:02:42.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.507+0000 7f673258a640 1 -- 172.21.15.76:0/4279254388 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672c1a42a0 con 0x7f672c100280 2024-03-12T06:02:42.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.507+0000 7f67297fa640 1 -- 172.21.15.76:0/4279254388 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c010430 con 0x7f672c100280 2024-03-12T06:02:42.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.507+0000 7f673258a640 1 -- 172.21.15.76:0/4279254388 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f672c1a47b0 con 0x7f672c100280 2024-03-12T06:02:42.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.510+0000 7f67297fa640 1 -- 172.21.15.76:0/4279254388 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f671c004f80 con 0x7f672c100280 2024-03-12T06:02:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.510+0000 7f6735233740 1 Processor -- start 2024-03-12T06:02:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f67297fa640 1 --2- 172.21.15.76:0/4279254388 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f67140a1ba0 0x7f67140a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f6735233740 1 -- start start 2024-03-12T06:02:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f6730b27640 1 --2- 172.21.15.76:0/4279254388 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f67140a1ba0 0x7f67140a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f6735233740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911e2b0 0x55fb9911e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f67297fa640 1 -- 172.21.15.76:0/4279254388 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f671c014070 con 0x7f672c100280 2024-03-12T06:02:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f6735233740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911ebd0 0x55fb99127270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f672bfff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911e2b0 0x55fb9911e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f6735233740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 0x55fb99129ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f672bfff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911e2b0 0x55fb9911e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:53406/0 (socket says 172.21.15.76:53406) 2024-03-12T06:02:42.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f6735233740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb9908d280 con 0x55fb991277b0 2024-03-12T06:02:42.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f672bfff640 1 -- 172.21.15.76:0/376898108 learned_addr learned my addr 172.21.15.76:0/376898108 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:42.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f6735233740 1 -- 172.21.15.76:0/376898108 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb9908a7b0 con 0x55fb9911e2b0 2024-03-12T06:02:42.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f6735233740 1 -- 172.21.15.76:0/376898108 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb99085f90 con 0x55fb9911ebd0 2024-03-12T06:02:42.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f672b7fe640 1 --2- 172.21.15.76:0/376898108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911ebd0 0x55fb99127270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f672bfff640 1 -- 172.21.15.76:0/376898108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911ebd0 msgr2=0x55fb99127270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f672bfff640 1 --2- 172.21.15.76:0/376898108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911ebd0 0x55fb99127270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f672bfff640 1 -- 172.21.15.76:0/376898108 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 msgr2=0x55fb99129ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-12T06:02:42.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f672bfff640 1 --2- 172.21.15.76:0/376898108 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 0x55fb99129ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f672bfff640 1 -- 172.21.15.76:0/376898108 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb99044570 con 0x55fb9911e2b0 2024-03-12T06:02:42.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f6730b27640 1 --2- 172.21.15.76:0/4279254388 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f67140a1ba0 0x7f67140a4060 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6724036040 tx=0x7f6724005fd0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:42.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.511+0000 7f672bfff640 1 --2- 172.21.15.76:0/376898108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911e2b0 0x55fb9911e690 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f671c096b90 tx=0x7f671c09a090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ca5f18eb2945f60 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:42.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.512+0000 7f67127fc640 1 -- 172.21.15.76:0/376898108 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c0b8ea0 con 0x55fb9911e2b0 2024-03-12T06:02:42.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.512+0000 7f67127fc640 1 -- 172.21.15.76:0/376898108 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f671c09e880 con 0x55fb9911e2b0 2024-03-12T06:02:42.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.512+0000 7f67127fc640 1 -- 172.21.15.76:0/376898108 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c0cb620 con 0x55fb9911e2b0 2024-03-12T06:02:42.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.513+0000 7f6735233740 1 -- 172.21.15.76:0/376898108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911e2b0 msgr2=0x55fb9911e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.513+0000 7f6735233740 1 --2- 172.21.15.76:0/376898108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911e2b0 0x55fb9911e690 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f671c096b90 tx=0x7f671c09a090 comp rx=0 tx=0).stop 2024-03-12T06:02:42.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.513+0000 7f6735233740 1 -- 172.21.15.76:0/376898108 shutdown_connections 2024-03-12T06:02:42.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.513+0000 7f6735233740 1 --2- 172.21.15.76:0/376898108 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 0x55fb99129ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.513+0000 7f6735233740 1 --2- 172.21.15.76:0/376898108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911ebd0 0x55fb99127270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.513+0000 7f6735233740 1 --2- 172.21.15.76:0/376898108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911e2b0 0x55fb9911e690 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.513+0000 7f6735233740 1 -- 172.21.15.76:0/376898108 >> 172.21.15.76:0/376898108 conn(0x55fb99118990 msgr2=0x55fb99118d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:42.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.513+0000 7f6735233740 1 -- 172.21.15.76:0/376898108 shutdown_connections 2024-03-12T06:02:42.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.513+0000 7f6735233740 1 -- 172.21.15.76:0/376898108 wait complete. 2024-03-12T06:02:42.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.513+0000 7f6735233740 1 -- start start 2024-03-12T06:02:42.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.513+0000 7f6735233740 1 Processor -- start 2024-03-12T06:02:42.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.514+0000 7f6735233740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911e2b0 0x55fb99127270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.514+0000 7f6735233740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911ebd0 0x55fb99124600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.514+0000 7f6735233740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 0x55fb99124d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.514+0000 7f672bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911e2b0 0x55fb99127270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.514+0000 7f672bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911e2b0 0x55fb99127270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.76:56952/0 (socket says 172.21.15.76:56952) 2024-03-12T06:02:42.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.514+0000 7f672bfff640 1 -- 172.21.15.76:0/3411127402 learned_addr learned my addr 172.21.15.76:0/3411127402 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:42.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.514+0000 7f672b7fe640 1 --2- 172.21.15.76:0/3411127402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911ebd0 0x55fb99124600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.514+0000 7f6735233740 1 -- 172.21.15.76:0/3411127402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb98d59430 con 0x55fb991277b0 2024-03-12T06:02:42.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.514+0000 7f6735233740 1 -- 172.21.15.76:0/3411127402 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb98d15e00 con 0x55fb9911ebd0 2024-03-12T06:02:42.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.514+0000 7f6730b27640 1 --2- 172.21.15.76:0/3411127402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 0x55fb99124d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.514+0000 7f6735233740 1 -- 172.21.15.76:0/3411127402 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb9900afd0 con 0x55fb9911e2b0 2024-03-12T06:02:42.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.514+0000 7f6735233740 10 client.0 _ll_register_callbacks cb 0x55fb99060fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:42.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.515+0000 7f672b7fe640 1 -- 172.21.15.76:0/3411127402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911e2b0 msgr2=0x55fb99127270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.515+0000 7f672b7fe640 1 --2- 172.21.15.76:0/3411127402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911e2b0 0x55fb99127270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.515+0000 7f672b7fe640 1 -- 172.21.15.76:0/3411127402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 msgr2=0x55fb99124d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.515+0000 7f672b7fe640 1 --2- 172.21.15.76:0/3411127402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 0x55fb99124d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.515+0000 7f672b7fe640 1 -- 172.21.15.76:0/3411127402 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f672c1a47b0 con 0x55fb9911ebd0 2024-03-12T06:02:42.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.515+0000 7f672b7fe640 1 --2- 172.21.15.76:0/3411127402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911ebd0 0x55fb99124600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f672c06a3f0 tx=0x7f6720012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4008ec9b13a8fe16 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:42.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.515+0000 7f670b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T06:02:42.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.515+0000 7f670b7fe640 1 -- 172.21.15.76:0/3411127402 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720017070 con 0x55fb9911ebd0 2024-03-12T06:02:42.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.515+0000 7f6735233740 1 -- 172.21.15.76:0/3411127402 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fb98d857e0 con 0x55fb9911ebd0 2024-03-12T06:02:42.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.515+0000 7f670b7fe640 1 -- 172.21.15.76:0/3411127402 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f672000f8f0 con 0x55fb9911ebd0 2024-03-12T06:02:42.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.515+0000 7f670b7fe640 1 -- 172.21.15.76:0/3411127402 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f672000fab0 con 0x55fb9911ebd0 2024-03-12T06:02:42.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.516+0000 7f6735233740 10 client.26252 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:42.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.516+0000 7f6735233740 1 -- 172.21.15.76:0/3411127402 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fb99084610 con 0x55fb9911ebd0 2024-03-12T06:02:42.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.516+0000 7f670b7fe640 1 -- 172.21.15.76:0/3411127402 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6720004c80 con 0x55fb9911ebd0 2024-03-12T06:02:42.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.516+0000 7f670b7fe640 1 -- 172.21.15.76:0/3411127402 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6720005420 con 0x55fb9911ebd0 2024-03-12T06:02:42.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.516+0000 7f6735233740 10 client.26252 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:42.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.516+0000 7f6735233740 10 client.26252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:42.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.516+0000 7f6735233740 20 client.26252 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:42.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.516+0000 7f6735233740 1 --2- 172.21.15.76:0/3411127402 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fb9913be20 0x55fb9913e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.516+0000 7f6735233740 4 client.26252 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:02:42.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.516+0000 7f6735233740 1 -- 172.21.15.76:0/3411127402 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fb98d857e0 con 0x55fb9913be20 2024-03-12T06:02:42.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.516+0000 7f672bfff640 1 --2- 172.21.15.76:0/3411127402 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fb9913be20 0x55fb9913e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.517+0000 7f672bfff640 1 --2- 172.21.15.76:0/3411127402 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fb9913be20 0x55fb9913e280 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ff3f949151814c5 server_cookie=ab10beec048cb0e1 in_seq=0 out_seq=0 2024-03-12T06:02:42.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.517+0000 7f670b7fe640 10 client.26252 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:42.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.517+0000 7f670b7fe640 1 -- 172.21.15.76:0/3411127402 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fb98d857e0 con 0x55fb9913be20 2024-03-12T06:02:42.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.517+0000 7f670b7fe640 10 client.26252 handle_command_reply: tid=0 2024-03-12T06:02:42.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.517+0000 7f6735233740 1 client.26252 shutdown 2024-03-12T06:02:42.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.518+0000 7f6735233740 1 -- 172.21.15.76:0/3411127402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911ebd0 msgr2=0x55fb99124600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.518+0000 7f6735233740 1 --2- 172.21.15.76:0/3411127402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911ebd0 0x55fb99124600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f672c06a3f0 tx=0x7f6720012b60 comp rx=0 tx=0).stop 2024-03-12T06:02:42.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.518+0000 7f6735233740 1 -- 172.21.15.76:0/3411127402 shutdown_connections 2024-03-12T06:02:42.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.518+0000 7f6735233740 1 --2- 172.21.15.76:0/3411127402 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fb9913be20 0x55fb9913e280 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.518+0000 7f6735233740 1 --2- 172.21.15.76:0/3411127402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 0x55fb99124d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.518+0000 7f6735233740 1 --2- 172.21.15.76:0/3411127402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911ebd0 0x55fb99124600 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.518+0000 7f6735233740 1 --2- 172.21.15.76:0/3411127402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911e2b0 0x55fb99127270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.518+0000 7f6735233740 1 -- 172.21.15.76:0/3411127402 >> 172.21.15.76:0/3411127402 conn(0x55fb99118990 msgr2=0x55fb9911a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:42.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.518+0000 7f670b7fe640 0 client.26252 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:42.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.518+0000 7f6735233740 1 -- 172.21.15.76:0/3411127402 shutdown_connections 2024-03-12T06:02:42.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.518+0000 7f6735233740 1 -- 172.21.15.76:0/3411127402 wait complete. 2024-03-12T06:02:42.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.518+0000 7f6735233740 20 client.26252 trim_cache size 0 max 16384 2024-03-12T06:02:42.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f6735233740 1 Processor -- start 2024-03-12T06:02:42.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f6735233740 1 -- start start 2024-03-12T06:02:42.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f6735233740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911e2b0 0x55fb99126d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f6735233740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911ebd0 0x55fb99127270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f6735233740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 0x55fb99132a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f6735233740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb991199e0 con 0x55fb991277b0 2024-03-12T06:02:42.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f6735233740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb9911dcf0 con 0x55fb9911ebd0 2024-03-12T06:02:42.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f6735233740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb9911df40 con 0x55fb9911e2b0 2024-03-12T06:02:42.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f6730b27640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 0x55fb99132a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f672b7fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911ebd0 0x55fb99127270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f6730b27640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 0x55fb99132a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33630/0 (socket says 172.21.15.76:33630) 2024-03-12T06:02:42.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f6730b27640 1 -- 172.21.15.76:0/2796961736 learned_addr learned my addr 172.21.15.76:0/2796961736 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:42.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f672bfff640 1 --2- 172.21.15.76:0/2796961736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911e2b0 0x55fb99126d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f672b7fe640 1 -- 172.21.15.76:0/2796961736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911e2b0 msgr2=0x55fb99126d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f672b7fe640 1 --2- 172.21.15.76:0/2796961736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911e2b0 0x55fb99126d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f672b7fe640 1 -- 172.21.15.76:0/2796961736 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 msgr2=0x55fb99132a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f672b7fe640 1 --2- 172.21.15.76:0/2796961736 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 0x55fb99132a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.538+0000 7f672b7fe640 1 -- 172.21.15.76:0/2796961736 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb99020ac0 con 0x55fb9911ebd0 2024-03-12T06:02:42.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.539+0000 7f672b7fe640 1 --2- 172.21.15.76:0/2796961736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911ebd0 0x55fb99127270 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6720005850 tx=0x7f6720005690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2629ee477d0e219d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:42.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.539+0000 7f6711ffb640 1 -- 172.21.15.76:0/2796961736 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720017040 con 0x55fb9911ebd0 2024-03-12T06:02:42.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.539+0000 7f6711ffb640 1 -- 172.21.15.76:0/2796961736 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f672000f8f0 con 0x55fb9911ebd0 2024-03-12T06:02:42.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.539+0000 7f6711ffb640 1 -- 172.21.15.76:0/2796961736 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720010af0 con 0x55fb9911ebd0 2024-03-12T06:02:42.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.540+0000 7f6735233740 1 -- 172.21.15.76:0/2796961736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911ebd0 msgr2=0x55fb99127270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.540+0000 7f6735233740 1 --2- 172.21.15.76:0/2796961736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911ebd0 0x55fb99127270 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6720005850 tx=0x7f6720005690 comp rx=0 tx=0).stop 2024-03-12T06:02:42.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.540+0000 7f6735233740 1 -- 172.21.15.76:0/2796961736 shutdown_connections 2024-03-12T06:02:42.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.540+0000 7f6735233740 1 --2- 172.21.15.76:0/2796961736 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb991277b0 0x55fb99132a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.540+0000 7f6735233740 1 --2- 172.21.15.76:0/2796961736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911ebd0 0x55fb99127270 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.540+0000 7f6735233740 1 --2- 172.21.15.76:0/2796961736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb9911e2b0 0x55fb99126d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.540+0000 7f6735233740 1 -- 172.21.15.76:0/2796961736 >> 172.21.15.76:0/2796961736 conn(0x55fb99118990 msgr2=0x55fb99122ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:42.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.540+0000 7f6735233740 1 -- 172.21.15.76:0/2796961736 shutdown_connections 2024-03-12T06:02:42.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.540+0000 7f6735233740 1 -- 172.21.15.76:0/2796961736 wait complete. 2024-03-12T06:02:42.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.540+0000 7f6735233740 1 -- start start 2024-03-12T06:02:42.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f6735233740 1 Processor -- start 2024-03-12T06:02:42.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f6735233740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911e2b0 0x55fb99131880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.549 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f6735233740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb9911ebd0 0x55fb99131dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f6735233740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb991277b0 0x55fb991923b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f6735233740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb991325d0 con 0x55fb9911ebd0 2024-03-12T06:02:42.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f6735233740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb99132740 con 0x55fb9911e2b0 2024-03-12T06:02:42.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f6735233740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb99192970 con 0x55fb991277b0 2024-03-12T06:02:42.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f6735233740 10 client.0 _ll_register_callbacks cb 0x7f671c096310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:02:42.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f672b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb9911ebd0 0x55fb99131dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f672bfff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911e2b0 0x55fb99131880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f672b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb9911ebd0 0x55fb99131dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:33640/0 (socket says 172.21.15.76:33640) 2024-03-12T06:02:42.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f672b7fe640 1 -- 172.21.15.76:0/255518247 learned_addr learned my addr 172.21.15.76:0/255518247 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:02:42.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f6730b27640 1 --2- 172.21.15.76:0/255518247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb991277b0 0x55fb991923b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f672b7fe640 1 -- 172.21.15.76:0/255518247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb991277b0 msgr2=0x55fb991923b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f672b7fe640 1 --2- 172.21.15.76:0/255518247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb991277b0 0x55fb991923b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f672b7fe640 1 -- 172.21.15.76:0/255518247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911e2b0 msgr2=0x55fb99131880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:02:42.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f672b7fe640 1 --2- 172.21.15.76:0/255518247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911e2b0 0x55fb99131880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f672b7fe640 1 -- 172.21.15.76:0/255518247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fb99084610 con 0x55fb9911ebd0 2024-03-12T06:02:42.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.541+0000 7f672bfff640 1 --2- 172.21.15.76:0/255518247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911e2b0 0x55fb99131880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:02:42.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f672b7fe640 1 --2- 172.21.15.76:0/255518247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb9911ebd0 0x55fb99131dc0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f67200058c0 tx=0x7f6720005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ba1cf35c9e5351 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:02:42.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f670b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:02:42.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f670b7fe640 1 -- 172.21.15.76:0/255518247 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720017070 con 0x55fb9911ebd0 2024-03-12T06:02:42.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f670b7fe640 1 -- 172.21.15.76:0/255518247 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f6720000e40 con 0x55fb9911ebd0 2024-03-12T06:02:42.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f6735233740 1 -- 172.21.15.76:0/255518247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f671c09e880 con 0x55fb9911ebd0 2024-03-12T06:02:42.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f670b7fe640 1 -- 172.21.15.76:0/255518247 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f672000fc60 con 0x55fb9911ebd0 2024-03-12T06:02:42.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f6735233740 10 client.17118 fetch_fsmap learned FSMap version 28 2024-03-12T06:02:42.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f6735233740 1 -- 172.21.15.76:0/255518247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fb990466a0 con 0x55fb9911ebd0 2024-03-12T06:02:42.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f670b7fe640 1 -- 172.21.15.76:0/255518247 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f671c09e880 con 0x55fb9911ebd0 2024-03-12T06:02:42.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f670b7fe640 1 -- 172.21.15.76:0/255518247 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f672002d030 con 0x55fb9911ebd0 2024-03-12T06:02:42.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f6735233740 10 client.17118 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:02:42.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f6735233740 10 client.17118 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:02:42.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f6735233740 20 client.17118 populate_metadata read hostname 'smithi076' 2024-03-12T06:02:42.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f6735233740 1 --2- 172.21.15.76:0/255518247 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fb99193d60 0x55fb99196180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:02:42.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f6735233740 4 client.17118 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:02:42.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.542+0000 7f6735233740 1 -- 172.21.15.76:0/255518247 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fb98dc2720 con 0x55fb99193d60 2024-03-12T06:02:42.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.543+0000 7f672bfff640 1 --2- 172.21.15.76:0/255518247 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fb99193d60 0x55fb99196180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:02:42.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.543+0000 7f670b7fe640 10 client.17118 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:42.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.543+0000 7f672bfff640 1 --2- 172.21.15.76:0/255518247 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fb99193d60 0x55fb99196180 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e877d35383ca913a server_cookie=4d21960e7bb952 in_seq=0 out_seq=0 2024-03-12T06:02:42.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.543+0000 7f670b7fe640 1 -- 172.21.15.76:0/255518247 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55fb98dc2720 con 0x55fb99193d60 2024-03-12T06:02:42.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.543+0000 7f670b7fe640 10 client.17118 handle_command_reply: tid=0 2024-03-12T06:02:42.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.544+0000 7f6735233740 1 client.17118 shutdown 2024-03-12T06:02:42.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.544+0000 7f6735233740 1 -- 172.21.15.76:0/255518247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb9911ebd0 msgr2=0x55fb99131dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.544+0000 7f6735233740 1 --2- 172.21.15.76:0/255518247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb9911ebd0 0x55fb99131dc0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f67200058c0 tx=0x7f6720005f10 comp rx=0 tx=0).stop 2024-03-12T06:02:42.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.544+0000 7f6735233740 1 -- 172.21.15.76:0/255518247 shutdown_connections 2024-03-12T06:02:42.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.544+0000 7f6735233740 1 --2- 172.21.15.76:0/255518247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb991277b0 0x55fb991923b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.544+0000 7f6735233740 1 --2- 172.21.15.76:0/255518247 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55fb99193d60 0x55fb99196180 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.544+0000 7f670b7fe640 0 client.17118 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:02:42.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.544+0000 7f6735233740 1 --2- 172.21.15.76:0/255518247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55fb9911ebd0 0x55fb99131dc0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.544+0000 7f6735233740 1 --2- 172.21.15.76:0/255518247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fb9911e2b0 0x55fb99131880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.544+0000 7f6735233740 1 -- 172.21.15.76:0/255518247 >> 172.21.15.76:0/255518247 conn(0x55fb99118990 msgr2=0x55fb99122ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:42.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.544+0000 7f6735233740 1 -- 172.21.15.76:0/255518247 shutdown_connections 2024-03-12T06:02:42.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.544+0000 7f6735233740 1 -- 172.21.15.76:0/255518247 wait complete. 2024-03-12T06:02:42.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.544+0000 7f6735233740 20 client.17118 trim_cache size 0 max 16384 2024-03-12T06:02:42.562 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:02:42.562 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T06:02:42.562 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T06:02:42.562 INFO:teuthology.orchestra.run.smithi076.stdout: "4a64affa-5c88-475e-92ec-33d7ad2c6b05": { 2024-03-12T06:02:42.562 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T06:02:42.562 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "4a64affa-5c88-475e-92ec-33d7ad2c6b05", 2024-03-12T06:02:42.562 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T06:02:42.563 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:02:42.563 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:02:42.563 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:02:42.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.545+0000 7f673258a640 1 -- 172.21.15.76:0/4279254388 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f67140a1ba0 msgr2=0x7f67140a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.545+0000 7f673258a640 1 --2- 172.21.15.76:0/4279254388 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f67140a1ba0 0x7f67140a4060 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6724036040 tx=0x7f6724005fd0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.545+0000 7f673258a640 1 -- 172.21.15.76:0/4279254388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f672c100280 msgr2=0x7f672c198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:02:42.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.545+0000 7f673258a640 1 --2- 172.21.15.76:0/4279254388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f672c100280 0x7f672c198520 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f671c00e9c0 tx=0x7f671c00ee80 comp rx=0 tx=0).stop 2024-03-12T06:02:42.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.545+0000 7f673258a640 1 -- 172.21.15.76:0/4279254388 shutdown_connections 2024-03-12T06:02:42.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.545+0000 7f673258a640 1 --2- 172.21.15.76:0/4279254388 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f67140a1ba0 0x7f67140a4060 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.545+0000 7f673258a640 1 --2- 172.21.15.76:0/4279254388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f672c199040 0x7f672c1a35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.545+0000 7f673258a640 1 --2- 172.21.15.76:0/4279254388 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f672c105340 0x7f672c198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.545+0000 7f673258a640 1 --2- 172.21.15.76:0/4279254388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f672c100280 0x7f672c198520 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:02:42.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.545+0000 7f673258a640 1 -- 172.21.15.76:0/4279254388 >> 172.21.15.76:0/4279254388 conn(0x7f672c077120 msgr2=0x7f672c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:02:42.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.545+0000 7f673258a640 1 -- 172.21.15.76:0/4279254388 shutdown_connections 2024-03-12T06:02:42.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:02:42.545+0000 7f673258a640 1 -- 172.21.15.76:0/4279254388 wait complete. 2024-03-12T06:02:42.566 INFO:tasks.cephfs.filesystem:scrub status for tag:4a64affa-5c88-475e-92ec-33d7ad2c6b05 - {'path': '/', 'tag': '4a64affa-5c88-475e-92ec-33d7ad2c6b05', 'options': 'recursive,force'} 2024-03-12T06:02:43.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:43 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:02:43.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:43 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:02:43.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:43 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:43.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:43 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:43.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:43 smithi082 ceph-mon[36382]: pgmap v213: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:02:43.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:43 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:02:43.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:43 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:02:43.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:43 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:43.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:43 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:43.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:43 smithi088 ceph-mon[36585]: pgmap v213: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:02:43.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:43 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:02:43.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:43 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:02:43.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:43 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:43.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:43 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:43.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:43 smithi076 ceph-mon[32095]: pgmap v213: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:02:44.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:44 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T06:02:44.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:44 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:44.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:44 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:02:44.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:44 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:44 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T06:02:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:44 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:44.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:44 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:02:44.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:44 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:44.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:44 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T06:02:44.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:44 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:44.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:44 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:02:44.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:44 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:02:45.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:45 smithi082 ceph-mon[36382]: pgmap v214: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:02:45.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:45 smithi088 ceph-mon[36585]: pgmap v214: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:02:45.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:45 smithi076 ceph-mon[32095]: pgmap v214: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:02:48.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:48 smithi082 ceph-mon[36382]: pgmap v215: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:02:48.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:48 smithi088 ceph-mon[36585]: pgmap v215: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:02:48.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:48 smithi076 ceph-mon[32095]: pgmap v215: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:02:50.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:50 smithi082 ceph-mon[36382]: pgmap v216: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:02:50.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:50 smithi088 ceph-mon[36585]: pgmap v216: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:02:50.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:50 smithi076 ceph-mon[32095]: pgmap v216: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:02:52.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:52 smithi082 ceph-mon[36382]: pgmap v217: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:02:52.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:52 smithi088 ceph-mon[36585]: pgmap v217: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:02:52.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:52 smithi076 ceph-mon[32095]: pgmap v217: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:02:53.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:53 smithi082 ceph-mon[36382]: pgmap v218: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:02:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:53 smithi088 ceph-mon[36585]: pgmap v218: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:02:53.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:53 smithi076 ceph-mon[32095]: pgmap v218: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:02:56.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:56 smithi082 ceph-mon[36382]: pgmap v219: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:56.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:56 smithi088 ceph-mon[36585]: pgmap v219: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:56.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:56 smithi076 ceph-mon[32095]: pgmap v219: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:58.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:58 smithi082 ceph-mon[36382]: pgmap v220: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:58.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:58 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:02:58.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:58 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:02:58.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:58 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:02:58.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:58 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:02:58.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:58 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:02:58.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:58 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:02:58.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:58 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:02:58.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:58 smithi088 ceph-mon[36585]: pgmap v220: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:58.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:58 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:02:58.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:58 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:02:58.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:58 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:02:58.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:58 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:02:58.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:58 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:02:58.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:58 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:02:58.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:58 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:02:58.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:58 smithi076 ceph-mon[32095]: pgmap v220: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:02:58.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:58 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:02:58.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:58 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:02:58.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:58 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:02:58.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:58 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:02:58.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:58 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:02:58.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:58 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:02:58.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:58 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:02:59.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:02:59 smithi088 ceph-mon[36585]: pgmap v221: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:02:59.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:02:59 smithi076 ceph-mon[32095]: pgmap v221: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:02:59.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:02:59 smithi082 ceph-mon[36382]: pgmap v221: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:03:02.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:02 smithi082 ceph-mon[36382]: pgmap v222: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:02.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:02 smithi088 ceph-mon[36585]: pgmap v222: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:02.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:02 smithi076 ceph-mon[32095]: pgmap v222: 97 pgs: 97 active+clean; 82 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:04.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:04 smithi082 ceph-mon[36382]: pgmap v223: 97 pgs: 97 active+clean; 82 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-12T06:03:04.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:04 smithi088 ceph-mon[36585]: pgmap v223: 97 pgs: 97 active+clean; 82 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-12T06:03:04.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:04 smithi076 ceph-mon[32095]: pgmap v223: 97 pgs: 97 active+clean; 82 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-12T06:03:05.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:05 smithi088 ceph-mon[36585]: pgmap v224: 97 pgs: 97 active+clean; 82 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:03:05.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:05 smithi076 ceph-mon[32095]: pgmap v224: 97 pgs: 97 active+clean; 82 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:03:05.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:05 smithi082 ceph-mon[36382]: pgmap v224: 97 pgs: 97 active+clean; 82 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:03:08.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:08 smithi082 ceph-mon[36382]: pgmap v225: 97 pgs: 97 active+clean; 82 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:03:08.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:08 smithi082 ceph-mon[36382]: osdmap e92: 12 total, 12 up, 12 in 2024-03-12T06:03:08.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:08 smithi088 ceph-mon[36585]: pgmap v225: 97 pgs: 97 active+clean; 82 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:03:08.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:08 smithi088 ceph-mon[36585]: osdmap e92: 12 total, 12 up, 12 in 2024-03-12T06:03:08.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:08 smithi076 ceph-mon[32095]: pgmap v225: 97 pgs: 97 active+clean; 82 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:03:08.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:08 smithi076 ceph-mon[32095]: osdmap e92: 12 total, 12 up, 12 in 2024-03-12T06:03:10.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:10 smithi082 ceph-mon[36382]: pgmap v227: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:03:10.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:10 smithi088 ceph-mon[36585]: pgmap v227: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:03:10.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:10 smithi076 ceph-mon[32095]: pgmap v227: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:03:11.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:11 smithi088 ceph-mon[36585]: pgmap v228: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:03:11.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:11 smithi076 ceph-mon[32095]: pgmap v228: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:03:11.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:11 smithi082 ceph-mon[36382]: pgmap v228: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:03:12.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:03:12.567 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:03:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.676+0000 7f9a15e76640 1 -- 172.21.15.76:0/3850391903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a10100c10 msgr2=0x7f9a10104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:12.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.676+0000 7f9a15e76640 1 --2- 172.21.15.76:0/3850391903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a10100c10 0x7f9a10104c50 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009930 tx=0x7f9a0402bc30 comp rx=0 tx=0).stop 2024-03-12T06:03:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.676+0000 7f9a15e76640 1 -- 172.21.15.76:0/3850391903 shutdown_connections 2024-03-12T06:03:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.676+0000 7f9a15e76640 1 --2- 172.21.15.76:0/3850391903 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a10105330 0x7f9a1010cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.676+0000 7f9a15e76640 1 --2- 172.21.15.76:0/3850391903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a10100c10 0x7f9a10104c50 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.676+0000 7f9a15e76640 1 --2- 172.21.15.76:0/3850391903 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a10100260 0x7f9a10100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.676+0000 7f9a15e76640 1 -- 172.21.15.76:0/3850391903 >> 172.21.15.76:0/3850391903 conn(0x7f9a10077180 msgr2=0x7f9a10077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.676+0000 7f9a15e76640 1 -- 172.21.15.76:0/3850391903 shutdown_connections 2024-03-12T06:03:12.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.676+0000 7f9a15e76640 1 -- 172.21.15.76:0/3850391903 wait complete. 2024-03-12T06:03:12.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.677+0000 7f9a15e76640 1 Processor -- start 2024-03-12T06:03:12.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.677+0000 7f9a15e76640 1 -- start start 2024-03-12T06:03:12.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.677+0000 7f9a15e76640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a10100260 0x7f9a101966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.677+0000 7f9a15e76640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a10105330 0x7f9a10196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.677+0000 7f9a15e76640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a1019af70 0x7f9a1019b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.677+0000 7f9a15e76640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1019b960 con 0x7f9a10105330 2024-03-12T06:03:12.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.677+0000 7f9a15e76640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1019bad0 con 0x7f9a10100260 2024-03-12T06:03:12.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.677+0000 7f9a15e76640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1019bc40 con 0x7f9a1019af70 2024-03-12T06:03:12.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.677+0000 7f9a0effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a10105330 0x7f9a10196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.677+0000 7f9a0effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a10105330 0x7f9a10196bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49274/0 (socket says 172.21.15.76:49274) 2024-03-12T06:03:12.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.677+0000 7f9a0effd640 1 -- 172.21.15.76:0/904178324 learned_addr learned my addr 172.21.15.76:0/904178324 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:12.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a0ffff640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a1019af70 0x7f9a1019b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a0f7fe640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a10100260 0x7f9a101966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a0effd640 1 -- 172.21.15.76:0/904178324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a1019af70 msgr2=0x7f9a1019b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:12.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a0effd640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a1019af70 0x7f9a1019b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a0effd640 1 -- 172.21.15.76:0/904178324 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a10100260 msgr2=0x7f9a101966b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:12.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a0effd640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a10100260 0x7f9a101966b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a0effd640 1 -- 172.21.15.76:0/904178324 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a04009560 con 0x7f9a10105330 2024-03-12T06:03:12.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a0ffff640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a1019af70 0x7f9a1019b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:12.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a0f7fe640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a10100260 0x7f9a101966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:12.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a0effd640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a10105330 0x7f9a10196bf0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009230 tx=0x7f9a04004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:12.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a0cff9640 1 -- 172.21.15.76:0/904178324 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0402cbc0 con 0x7f9a10105330 2024-03-12T06:03:12.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a15e76640 1 -- 172.21.15.76:0/904178324 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a101a3220 con 0x7f9a10105330 2024-03-12T06:03:12.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a0cff9640 1 -- 172.21.15.76:0/904178324 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a040052d0 con 0x7f9a10105330 2024-03-12T06:03:12.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a0cff9640 1 -- 172.21.15.76:0/904178324 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a04003e50 con 0x7f9a10105330 2024-03-12T06:03:12.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.678+0000 7f9a15e76640 1 -- 172.21.15.76:0/904178324 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a101a36d0 con 0x7f9a10105330 2024-03-12T06:03:12.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.680+0000 7f9a0cff9640 1 -- 172.21.15.76:0/904178324 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9a040035f0 con 0x7f9a10105330 2024-03-12T06:03:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a18a46740 1 Processor -- start 2024-03-12T06:03:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a18a46740 1 -- start start 2024-03-12T06:03:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a0cff9640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f99f80a1ba0 0x7f99f80a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a18a46740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb695f70 0x5626cb696350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a18a46740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb696890 0x5626cb6a1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a18a46740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb6a19c0 0x5626cb6a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a18a46740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb605280 con 0x5626cb6a19c0 2024-03-12T06:03:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a18a46740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb6027b0 con 0x5626cb695f70 2024-03-12T06:03:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a18a46740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb5fdf90 con 0x5626cb696890 2024-03-12T06:03:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a0ffff640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f99f80a1ba0 0x7f99f80a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a0cff9640 1 -- 172.21.15.76:0/904178324 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9a040efa40 con 0x7f9a10105330 2024-03-12T06:03:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a0ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb6a19c0 0x5626cb6a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a0ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb6a19c0 0x5626cb6a3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49278/0 (socket says 172.21.15.76:49278) 2024-03-12T06:03:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a0ffff640 1 -- 172.21.15.76:0/1595145856 learned_addr learned my addr 172.21.15.76:0/1595145856 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a0f7fe640 1 --2- 172.21.15.76:0/1595145856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb695f70 0x5626cb696350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a0effd640 1 --2- 172.21.15.76:0/1595145856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb696890 0x5626cb6a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a0ffff640 1 -- 172.21.15.76:0/1595145856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb696890 msgr2=0x5626cb6a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a0ffff640 1 --2- 172.21.15.76:0/1595145856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb696890 0x5626cb6a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a0ffff640 1 -- 172.21.15.76:0/1595145856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb695f70 msgr2=0x5626cb696350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a0ffff640 1 --2- 172.21.15.76:0/1595145856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb695f70 0x5626cb696350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a0ffff640 1 -- 172.21.15.76:0/1595145856 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626cb5bc570 con 0x5626cb6a19c0 2024-03-12T06:03:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.681+0000 7f9a0f7fe640 1 --2- 172.21.15.76:0/1595145856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb695f70 0x5626cb696350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.682+0000 7f9a0ffff640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f99f80a1ba0 0x7f99f80a4060 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9a08012be0 tx=0x7f9a08009db0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:12.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.682+0000 7f9a0ffff640 1 --2- 172.21.15.76:0/1595145856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb6a19c0 0x5626cb6a3db0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f9a08005b30 tx=0x7f9a0801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444ba61273dac580 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.682+0000 7f99f5ffb640 1 -- 172.21.15.76:0/1595145856 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0800c7a0 con 0x5626cb6a19c0 2024-03-12T06:03:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.682+0000 7f99f5ffb640 1 -- 172.21.15.76:0/1595145856 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a0800cde0 con 0x5626cb6a19c0 2024-03-12T06:03:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.682+0000 7f99f5ffb640 1 -- 172.21.15.76:0/1595145856 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a08020e00 con 0x5626cb6a19c0 2024-03-12T06:03:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.682+0000 7f9a18a46740 1 -- 172.21.15.76:0/1595145856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb6a19c0 msgr2=0x5626cb6a3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.682+0000 7f9a18a46740 1 --2- 172.21.15.76:0/1595145856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb6a19c0 0x5626cb6a3db0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f9a08005b30 tx=0x7f9a0801c000 comp rx=0 tx=0).stop 2024-03-12T06:03:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.682+0000 7f9a18a46740 1 -- 172.21.15.76:0/1595145856 shutdown_connections 2024-03-12T06:03:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 1 --2- 172.21.15.76:0/1595145856 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb6a19c0 0x5626cb6a3db0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 1 --2- 172.21.15.76:0/1595145856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb696890 0x5626cb6a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 1 --2- 172.21.15.76:0/1595145856 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb695f70 0x5626cb696350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 1 -- 172.21.15.76:0/1595145856 >> 172.21.15.76:0/1595145856 conn(0x5626cb690990 msgr2=0x5626cb690d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 1 -- 172.21.15.76:0/1595145856 shutdown_connections 2024-03-12T06:03:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 1 -- 172.21.15.76:0/1595145856 wait complete. 2024-03-12T06:03:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 1 -- start start 2024-03-12T06:03:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 1 Processor -- start 2024-03-12T06:03:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb695f70 0x5626cb6a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb696890 0x5626cb69e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb6a19c0 0x5626cb69ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb2d1430 con 0x5626cb695f70 2024-03-12T06:03:12.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb28de00 con 0x5626cb696890 2024-03-12T06:03:12.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb582fd0 con 0x5626cb6a19c0 2024-03-12T06:03:12.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a18a46740 10 client.0 _ll_register_callbacks cb 0x5626cb5d8fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:12.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a0f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb695f70 0x5626cb6a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a0f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb695f70 0x5626cb6a13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49280/0 (socket says 172.21.15.76:49280) 2024-03-12T06:03:12.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.683+0000 7f9a0f7fe640 1 -- 172.21.15.76:0/3587380417 learned_addr learned my addr 172.21.15.76:0/3587380417 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:12.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f9a0ffff640 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb6a19c0 0x5626cb69ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f9a0f7fe640 1 -- 172.21.15.76:0/3587380417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb6a19c0 msgr2=0x5626cb69ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:12.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f9a0f7fe640 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb6a19c0 0x5626cb69ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f9a0effd640 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb696890 0x5626cb69e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f9a0f7fe640 1 -- 172.21.15.76:0/3587380417 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb696890 msgr2=0x5626cb69e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:12.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f9a0f7fe640 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb696890 0x5626cb69e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f9a0f7fe640 1 -- 172.21.15.76:0/3587380417 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a0801a040 con 0x5626cb695f70 2024-03-12T06:03:12.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f9a0ffff640 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb6a19c0 0x5626cb69ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:12.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f9a0effd640 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb696890 0x5626cb69e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:12.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f9a0f7fe640 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb695f70 0x5626cb6a13e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f9a00010a70 tx=0x7f9a00010f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d3631f10d4d128 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:12.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f99eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:12.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f99eeffd640 1 -- 172.21.15.76:0/3587380417 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0000eda0 con 0x5626cb695f70 2024-03-12T06:03:12.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f99eeffd640 1 -- 172.21.15.76:0/3587380417 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9a00017070 con 0x5626cb695f70 2024-03-12T06:03:12.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f9a18a46740 1 -- 172.21.15.76:0/3587380417 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5626cb2fd7e0 con 0x5626cb695f70 2024-03-12T06:03:12.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f99eeffd640 1 -- 172.21.15.76:0/3587380417 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a0001c480 con 0x5626cb695f70 2024-03-12T06:03:12.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f9a18a46740 10 client.17130 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:12.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f9a18a46740 1 -- 172.21.15.76:0/3587380417 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5626cb5fc610 con 0x5626cb695f70 2024-03-12T06:03:12.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.684+0000 7f99eeffd640 1 -- 172.21.15.76:0/3587380417 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5626cb2fd7e0 con 0x5626cb695f70 2024-03-12T06:03:12.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.685+0000 7f99eeffd640 1 -- 172.21.15.76:0/3587380417 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9a00025860 con 0x5626cb695f70 2024-03-12T06:03:12.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.685+0000 7f9a18a46740 10 client.17130 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:12.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.685+0000 7f9a18a46740 10 client.17130 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:12.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.685+0000 7f9a18a46740 20 client.17130 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:12.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.685+0000 7f9a18a46740 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5626cb6b1b40 0x5626cb6b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.685+0000 7f9a18a46740 4 client.17130 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:03:12.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.685+0000 7f9a18a46740 1 -- 172.21.15.76:0/3587380417 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626cb33a720 con 0x5626cb6b1b40 2024-03-12T06:03:12.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.685+0000 7f9a0effd640 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5626cb6b1b40 0x5626cb6b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.685+0000 7f9a0effd640 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5626cb6b1b40 0x5626cb6b3f60 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=826ee3dcf2aa0492 server_cookie=32253bb90c53a536 in_seq=0 out_seq=0 2024-03-12T06:03:12.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.685+0000 7f99eeffd640 10 client.17130 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:12.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f99eeffd640 1 -- 172.21.15.76:0/3587380417 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5626cb33a720 con 0x5626cb6b1b40 2024-03-12T06:03:12.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f99eeffd640 10 client.17130 handle_command_reply: tid=0 2024-03-12T06:03:12.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f9a18a46740 1 client.17130 shutdown 2024-03-12T06:03:12.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f9a18a46740 1 -- 172.21.15.76:0/3587380417 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb695f70 msgr2=0x5626cb6a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:12.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f9a18a46740 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb695f70 0x5626cb6a13e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f9a00010a70 tx=0x7f9a00010f30 comp rx=0 tx=0).stop 2024-03-12T06:03:12.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f9a18a46740 1 -- 172.21.15.76:0/3587380417 shutdown_connections 2024-03-12T06:03:12.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f9a18a46740 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb6a19c0 0x5626cb69ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f9a18a46740 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb696890 0x5626cb69e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f9a18a46740 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5626cb6b1b40 0x5626cb6b3f60 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f9a18a46740 1 --2- 172.21.15.76:0/3587380417 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb695f70 0x5626cb6a13e0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f99eeffd640 0 client.17130 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:12.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f9a18a46740 1 -- 172.21.15.76:0/3587380417 >> 172.21.15.76:0/3587380417 conn(0x5626cb690990 msgr2=0x5626cb692730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:12.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f9a18a46740 1 -- 172.21.15.76:0/3587380417 shutdown_connections 2024-03-12T06:03:12.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f9a18a46740 1 -- 172.21.15.76:0/3587380417 wait complete. 2024-03-12T06:03:12.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.686+0000 7f9a18a46740 20 client.17130 trim_cache size 0 max 16384 2024-03-12T06:03:12.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a18a46740 1 Processor -- start 2024-03-12T06:03:12.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a18a46740 1 -- start start 2024-03-12T06:03:12.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a18a46740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb695f70 0x5626cb6a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a18a46740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb696890 0x5626cb6a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a18a46740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb6a19c0 0x5626cb69e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a18a46740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb6ae050 con 0x5626cb696890 2024-03-12T06:03:12.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a18a46740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb696d90 con 0x5626cb6a19c0 2024-03-12T06:03:12.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a18a46740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb68c0f0 con 0x5626cb695f70 2024-03-12T06:03:12.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a0effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb696890 0x5626cb6a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a0effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb696890 0x5626cb6a13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49282/0 (socket says 172.21.15.76:49282) 2024-03-12T06:03:12.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a0effd640 1 -- 172.21.15.76:0/3723205150 learned_addr learned my addr 172.21.15.76:0/3723205150 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:12.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a0effd640 1 -- 172.21.15.76:0/3723205150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb695f70 msgr2=0x5626cb6a0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:12.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a0effd640 1 --2- 172.21.15.76:0/3723205150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb695f70 0x5626cb6a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a0effd640 1 -- 172.21.15.76:0/3723205150 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb6a19c0 msgr2=0x5626cb69e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:12.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a0effd640 1 --2- 172.21.15.76:0/3723205150 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb6a19c0 0x5626cb69e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.701+0000 7f9a0effd640 1 -- 172.21.15.76:0/3723205150 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626cb598ac0 con 0x5626cb696890 2024-03-12T06:03:12.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.702+0000 7f9a0effd640 1 --2- 172.21.15.76:0/3723205150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb696890 0x5626cb6a13e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f9a040bc710 tx=0x7f9a040dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff4ddebca58af1f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:12.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.702+0000 7f99f57fa640 1 -- 172.21.15.76:0/3723205150 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a040bce80 con 0x5626cb696890 2024-03-12T06:03:12.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.702+0000 7f99f57fa640 1 -- 172.21.15.76:0/3723205150 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a040bc780 con 0x5626cb696890 2024-03-12T06:03:12.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.702+0000 7f99f57fa640 1 -- 172.21.15.76:0/3723205150 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a040bab50 con 0x5626cb696890 2024-03-12T06:03:12.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.702+0000 7f9a18a46740 1 -- 172.21.15.76:0/3723205150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb696890 msgr2=0x5626cb6a13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:12.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.702+0000 7f9a18a46740 1 --2- 172.21.15.76:0/3723205150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb696890 0x5626cb6a13e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f9a040bc710 tx=0x7f9a040dc040 comp rx=0 tx=0).stop 2024-03-12T06:03:12.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.702+0000 7f9a18a46740 1 -- 172.21.15.76:0/3723205150 shutdown_connections 2024-03-12T06:03:12.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.702+0000 7f9a18a46740 1 --2- 172.21.15.76:0/3723205150 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb6a19c0 0x5626cb69e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.702+0000 7f9a18a46740 1 --2- 172.21.15.76:0/3723205150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb696890 0x5626cb6a13e0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.702+0000 7f9a18a46740 1 --2- 172.21.15.76:0/3723205150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb695f70 0x5626cb6a0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.702+0000 7f9a18a46740 1 -- 172.21.15.76:0/3723205150 >> 172.21.15.76:0/3723205150 conn(0x5626cb690990 msgr2=0x5626cb69d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:12.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.702+0000 7f9a18a46740 1 -- 172.21.15.76:0/3723205150 shutdown_connections 2024-03-12T06:03:12.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.702+0000 7f9a18a46740 1 -- 172.21.15.76:0/3723205150 wait complete. 2024-03-12T06:03:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a18a46740 1 -- start start 2024-03-12T06:03:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a18a46740 1 Processor -- start 2024-03-12T06:03:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a18a46740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb695f70 0x5626cb6a8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a18a46740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb696890 0x5626cb6a8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a18a46740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb6a19c0 0x5626cb6a5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a18a46740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb6a64a0 con 0x5626cb6a19c0 2024-03-12T06:03:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a18a46740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb6a6610 con 0x5626cb695f70 2024-03-12T06:03:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a18a46740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb6a67d0 con 0x5626cb696890 2024-03-12T06:03:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a18a46740 10 client.0 _ll_register_callbacks cb 0x7f9a080126f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a0ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb6a19c0 0x5626cb6a5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a0ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb6a19c0 0x5626cb6a5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49284/0 (socket says 172.21.15.76:49284) 2024-03-12T06:03:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a0ffff640 1 -- 172.21.15.76:0/1400800307 learned_addr learned my addr 172.21.15.76:0/1400800307 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a0f7fe640 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb695f70 0x5626cb6a8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a0ffff640 1 -- 172.21.15.76:0/1400800307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb696890 msgr2=0x5626cb6a8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:12.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a0effd640 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb696890 0x5626cb6a8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.703+0000 7f9a0ffff640 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb696890 0x5626cb6a8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a0ffff640 1 -- 172.21.15.76:0/1400800307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb695f70 msgr2=0x5626cb6a8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a0ffff640 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb695f70 0x5626cb6a8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a0ffff640 1 -- 172.21.15.76:0/1400800307 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a040a7040 con 0x5626cb6a19c0 2024-03-12T06:03:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a0f7fe640 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb695f70 0x5626cb6a8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a0effd640 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb696890 0x5626cb6a8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a0ffff640 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb6a19c0 0x5626cb6a5dd0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f9a08012460 tx=0x7f9a08004c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f6699a4547600d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f99eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:12.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f99eeffd640 1 -- 172.21.15.76:0/1400800307 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a08009b50 con 0x5626cb6a19c0 2024-03-12T06:03:12.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f99eeffd640 1 -- 172.21.15.76:0/1400800307 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9a0800c7a0 con 0x5626cb6a19c0 2024-03-12T06:03:12.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a18a46740 1 -- 172.21.15.76:0/1400800307 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a0800cde0 con 0x5626cb6a19c0 2024-03-12T06:03:12.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f99eeffd640 1 -- 172.21.15.76:0/1400800307 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a0800c960 con 0x5626cb6a19c0 2024-03-12T06:03:12.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a18a46740 10 client.17136 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:12.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a18a46740 1 -- 172.21.15.76:0/1400800307 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5626cb5be6a0 con 0x5626cb6a19c0 2024-03-12T06:03:12.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f99eeffd640 1 -- 172.21.15.76:0/1400800307 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a0800cde0 con 0x5626cb6a19c0 2024-03-12T06:03:12.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f99eeffd640 1 -- 172.21.15.76:0/1400800307 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9a08030020 con 0x5626cb6a19c0 2024-03-12T06:03:12.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a18a46740 10 client.17136 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:12.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a18a46740 10 client.17136 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:12.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a18a46740 20 client.17136 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:12.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a18a46740 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5626cb70a250 0x5626cb70c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a18a46740 4 client.17136 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:03:12.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.704+0000 7f9a18a46740 1 -- 172.21.15.76:0/1400800307 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5626cb389890 con 0x5626cb70a250 2024-03-12T06:03:12.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f9a0f7fe640 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5626cb70a250 0x5626cb70c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f99eeffd640 10 client.17136 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:12.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f9a0f7fe640 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5626cb70a250 0x5626cb70c670 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e6180760cede1b3 server_cookie=c5eb8c4dbee59268 in_seq=0 out_seq=0 2024-03-12T06:03:12.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f99eeffd640 1 -- 172.21.15.76:0/1400800307 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5626cb389890 con 0x5626cb70a250 2024-03-12T06:03:12.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f99eeffd640 10 client.17136 handle_command_reply: tid=0 2024-03-12T06:03:12.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f9a18a46740 1 client.17136 shutdown 2024-03-12T06:03:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f9a18a46740 1 -- 172.21.15.76:0/1400800307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb6a19c0 msgr2=0x5626cb6a5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f9a18a46740 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb6a19c0 0x5626cb6a5dd0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f9a08012460 tx=0x7f9a08004c30 comp rx=0 tx=0).stop 2024-03-12T06:03:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f9a18a46740 1 -- 172.21.15.76:0/1400800307 shutdown_connections 2024-03-12T06:03:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f9a18a46740 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5626cb70a250 0x5626cb70c670 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f9a18a46740 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5626cb6a19c0 0x5626cb6a5dd0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f99eeffd640 0 client.17136 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f9a18a46740 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cb696890 0x5626cb6a8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f9a18a46740 1 --2- 172.21.15.76:0/1400800307 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5626cb695f70 0x5626cb6a8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f9a18a46740 1 -- 172.21.15.76:0/1400800307 >> 172.21.15.76:0/1400800307 conn(0x5626cb690990 msgr2=0x5626cb69cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:12.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f9a18a46740 1 -- 172.21.15.76:0/1400800307 shutdown_connections 2024-03-12T06:03:12.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.705+0000 7f9a18a46740 1 -- 172.21.15.76:0/1400800307 wait complete. 2024-03-12T06:03:12.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.706+0000 7f9a18a46740 20 client.17136 trim_cache size 0 max 16384 2024-03-12T06:03:12.721 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:03:12.721 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T06:03:12.721 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T06:03:12.722 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:03:12.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.706+0000 7f9a15e76640 1 -- 172.21.15.76:0/904178324 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f99f80a1ba0 msgr2=0x7f99f80a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:12.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.706+0000 7f9a15e76640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f99f80a1ba0 0x7f99f80a4060 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9a08012be0 tx=0x7f9a08009db0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.706+0000 7f9a15e76640 1 -- 172.21.15.76:0/904178324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a10105330 msgr2=0x7f9a10196bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:12.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.706+0000 7f9a15e76640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a10105330 0x7f9a10196bf0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009230 tx=0x7f9a04004270 comp rx=0 tx=0).stop 2024-03-12T06:03:12.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.706+0000 7f9a15e76640 1 -- 172.21.15.76:0/904178324 shutdown_connections 2024-03-12T06:03:12.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.706+0000 7f9a15e76640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f99f80a1ba0 0x7f99f80a4060 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.706+0000 7f9a15e76640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a1019af70 0x7f9a1019b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.706+0000 7f9a15e76640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a10105330 0x7f9a10196bf0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.706+0000 7f9a15e76640 1 --2- 172.21.15.76:0/904178324 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a10100260 0x7f9a101966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.706+0000 7f9a15e76640 1 -- 172.21.15.76:0/904178324 >> 172.21.15.76:0/904178324 conn(0x7f9a10077180 msgr2=0x7f9a100ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:12.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.706+0000 7f9a15e76640 1 -- 172.21.15.76:0/904178324 shutdown_connections 2024-03-12T06:03:12.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.706+0000 7f9a15e76640 1 -- 172.21.15.76:0/904178324 wait complete. 2024-03-12T06:03:12.724 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T06:03:12.725 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T06:03:12.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:12 smithi088 ceph-mon[36585]: osdmap e93: 12 total, 12 up, 12 in 2024-03-12T06:03:12.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:12 smithi076 ceph-mon[32095]: osdmap e93: 12 total, 12 up, 12 in 2024-03-12T06:03:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.822+0000 7fd188f8a640 1 -- 172.21.15.76:0/2915622362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd184101d40 msgr2=0x7fd184105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.822+0000 7fd188f8a640 1 --2- 172.21.15.76:0/2915622362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd184101d40 0x7fd184105d80 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fd174009930 tx=0x7fd17402bc30 comp rx=0 tx=0).stop 2024-03-12T06:03:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.822+0000 7fd188f8a640 1 -- 172.21.15.76:0/2915622362 shutdown_connections 2024-03-12T06:03:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.822+0000 7fd188f8a640 1 --2- 172.21.15.76:0/2915622362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd184106460 0x7fd18410e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.822+0000 7fd188f8a640 1 --2- 172.21.15.76:0/2915622362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd184101d40 0x7fd184105d80 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.822+0000 7fd188f8a640 1 --2- 172.21.15.76:0/2915622362 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd184101390 0x7fd184101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.829 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.822+0000 7fd188f8a640 1 -- 172.21.15.76:0/2915622362 >> 172.21.15.76:0/2915622362 conn(0x7fd1840fcc30 msgr2=0x7fd1840ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.822+0000 7fd188f8a640 1 -- 172.21.15.76:0/2915622362 shutdown_connections 2024-03-12T06:03:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.822+0000 7fd188f8a640 1 -- 172.21.15.76:0/2915622362 wait complete. 2024-03-12T06:03:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd188f8a640 1 Processor -- start 2024-03-12T06:03:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd188f8a640 1 -- start start 2024-03-12T06:03:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd188f8a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd184101390 0x7fd18419d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd188f8a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd184106460 0x7fd18419dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd188f8a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd184197640 0x7fd184197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd188f8a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd184198110 con 0x7fd184197640 2024-03-12T06:03:12.830 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd188f8a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd184198280 con 0x7fd184101390 2024-03-12T06:03:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd188f8a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd184198440 con 0x7fd184106460 2024-03-12T06:03:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd182d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd184197640 0x7fd184197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd182d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd184197640 0x7fd184197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49310/0 (socket says 172.21.15.76:49310) 2024-03-12T06:03:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd182d9d640 1 -- 172.21.15.76:0/1560883811 learned_addr learned my addr 172.21.15.76:0/1560883811 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd18259c640 1 --2- 172.21.15.76:0/1560883811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd184101390 0x7fd18419d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd181d9b640 1 --2- 172.21.15.76:0/1560883811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd184106460 0x7fd18419dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd182d9d640 1 -- 172.21.15.76:0/1560883811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd184106460 msgr2=0x7fd18419dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:12.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd182d9d640 1 --2- 172.21.15.76:0/1560883811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd184106460 0x7fd18419dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd182d9d640 1 -- 172.21.15.76:0/1560883811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd184101390 msgr2=0x7fd18419d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd182d9d640 1 --2- 172.21.15.76:0/1560883811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd184101390 0x7fd18419d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.823+0000 7fd182d9d640 1 -- 172.21.15.76:0/1560883811 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd174009560 con 0x7fd184197640 2024-03-12T06:03:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.824+0000 7fd182d9d640 1 --2- 172.21.15.76:0/1560883811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd184197640 0x7fd184197ac0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd17c00eaa0 tx=0x7fd17c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.824+0000 7fd1737fe640 1 -- 172.21.15.76:0/1560883811 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c00cdb0 con 0x7fd184197640 2024-03-12T06:03:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.824+0000 7fd188f8a640 1 -- 172.21.15.76:0/1560883811 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1841a43e0 con 0x7fd184197640 2024-03-12T06:03:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.824+0000 7fd1737fe640 1 -- 172.21.15.76:0/1560883811 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd17c005020 con 0x7fd184197640 2024-03-12T06:03:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.824+0000 7fd1737fe640 1 -- 172.21.15.76:0/1560883811 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c0116b0 con 0x7fd184197640 2024-03-12T06:03:12.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.824+0000 7fd188f8a640 1 -- 172.21.15.76:0/1560883811 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1841a48c0 con 0x7fd184197640 2024-03-12T06:03:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.825+0000 7fd188f8a640 1 -- 172.21.15.76:0/1560883811 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd184102860 con 0x7fd184197640 2024-03-12T06:03:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.826+0000 7fd1737fe640 1 -- 172.21.15.76:0/1560883811 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd17c005520 con 0x7fd184197640 2024-03-12T06:03:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.826+0000 7fd1737fe640 1 --2- 172.21.15.76:0/1560883811 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd16c0a1ba0 0x7fd16c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.826+0000 7fd1737fe640 1 -- 172.21.15.76:0/1560883811 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fd17c0cdd60 con 0x7fd184197640 2024-03-12T06:03:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.826+0000 7fd18259c640 1 --2- 172.21.15.76:0/1560883811 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd16c0a1ba0 0x7fd16c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:12.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.827+0000 7fd18259c640 1 --2- 172.21.15.76:0/1560883811 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd16c0a1ba0 0x7fd16c0a4060 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd1780058c0 tx=0x7fd17800a2d0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:12.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:12.830+0000 7fd1737fe640 1 -- 172.21.15.76:0/1560883811 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd17c096c30 con 0x7fd184197640 2024-03-12T06:03:12.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:12 smithi082 ceph-mon[36382]: osdmap e93: 12 total, 12 up, 12 in 2024-03-12T06:03:13.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.037+0000 7fd188f8a640 1 -- 172.21.15.76:0/1560883811 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd184068450 con 0x7fd184197640 2024-03-12T06:03:13.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.038+0000 7fd1737fe640 1 -- 172.21.15.76:0/1560883811 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11328 (secure 0 0 0) 0x7fd17c01f020 con 0x7fd184197640 2024-03-12T06:03:13.043 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:03:13.043 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T06:01:33.842001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T06:03:13.043 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 28 2024-03-12T06:03:13.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.040+0000 7fd188f8a640 1 -- 172.21.15.76:0/1560883811 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd16c0a1ba0 msgr2=0x7fd16c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.040+0000 7fd188f8a640 1 --2- 172.21.15.76:0/1560883811 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd16c0a1ba0 0x7fd16c0a4060 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd1780058c0 tx=0x7fd17800a2d0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.040+0000 7fd188f8a640 1 -- 172.21.15.76:0/1560883811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd184197640 msgr2=0x7fd184197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.040+0000 7fd188f8a640 1 --2- 172.21.15.76:0/1560883811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd184197640 0x7fd184197ac0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd17c00eaa0 tx=0x7fd17c00ef60 comp rx=0 tx=0).stop 2024-03-12T06:03:13.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.040+0000 7fd188f8a640 1 -- 172.21.15.76:0/1560883811 shutdown_connections 2024-03-12T06:03:13.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.040+0000 7fd188f8a640 1 --2- 172.21.15.76:0/1560883811 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd16c0a1ba0 0x7fd16c0a4060 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.040+0000 7fd188f8a640 1 --2- 172.21.15.76:0/1560883811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd184197640 0x7fd184197ac0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.040+0000 7fd188f8a640 1 --2- 172.21.15.76:0/1560883811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd184106460 0x7fd18419dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.040+0000 7fd188f8a640 1 --2- 172.21.15.76:0/1560883811 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd184101390 0x7fd18419d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.040+0000 7fd188f8a640 1 -- 172.21.15.76:0/1560883811 >> 172.21.15.76:0/1560883811 conn(0x7fd1840fcc30 msgr2=0x7fd1840fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.040+0000 7fd188f8a640 1 -- 172.21.15.76:0/1560883811 shutdown_connections 2024-03-12T06:03:13.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.040+0000 7fd188f8a640 1 -- 172.21.15.76:0/1560883811 wait complete. 2024-03-12T06:03:13.056 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:03:13.056 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T06:03:13.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.195+0000 7fb945d48640 1 -- 172.21.15.76:0/1600798646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9401023c0 msgr2=0x7fb9401027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.195+0000 7fb945d48640 1 --2- 172.21.15.76:0/1600798646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9401023c0 0x7fb9401027a0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fb930009900 tx=0x7fb93002bcd0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.195+0000 7fb945d48640 1 -- 172.21.15.76:0/1600798646 shutdown_connections 2024-03-12T06:03:13.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.195+0000 7fb945d48640 1 --2- 172.21.15.76:0/1600798646 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb94010b700 0x7fb94010dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.195+0000 7fb945d48640 1 --2- 172.21.15.76:0/1600798646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb940102ce0 0x7fb94010b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.195+0000 7fb945d48640 1 --2- 172.21.15.76:0/1600798646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb9401023c0 0x7fb9401027a0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.196+0000 7fb945d48640 1 -- 172.21.15.76:0/1600798646 >> 172.21.15.76:0/1600798646 conn(0x7fb940077230 msgr2=0x7fb940077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.196+0000 7fb945d48640 1 -- 172.21.15.76:0/1600798646 shutdown_connections 2024-03-12T06:03:13.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.196+0000 7fb945d48640 1 -- 172.21.15.76:0/1600798646 wait complete. 2024-03-12T06:03:13.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.196+0000 7fb945d48640 1 Processor -- start 2024-03-12T06:03:13.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.196+0000 7fb945d48640 1 -- start start 2024-03-12T06:03:13.203 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.196+0000 7fb945d48640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb9401023c0 0x7fb940100010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.196+0000 7fb945d48640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb940102ce0 0x7fb940100550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.196+0000 7fb945d48640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb94010b700 0x7fb940100a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.196+0000 7fb945d48640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9401a5cf0 con 0x7fb940102ce0 2024-03-12T06:03:13.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.196+0000 7fb945d48640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9401a5e60 con 0x7fb9401023c0 2024-03-12T06:03:13.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.196+0000 7fb945d48640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9401a6020 con 0x7fb94010b700 2024-03-12T06:03:13.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb93effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb940102ce0 0x7fb940100550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb93effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb940102ce0 0x7fb940100550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49324/0 (socket says 172.21.15.76:49324) 2024-03-12T06:03:13.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb93effd640 1 -- 172.21.15.76:0/314758845 learned_addr learned my addr 172.21.15.76:0/314758845 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.204 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb93f7fe640 1 --2- 172.21.15.76:0/314758845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb9401023c0 0x7fb940100010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb93effd640 1 -- 172.21.15.76:0/314758845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb94010b700 msgr2=0x7fb940100a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb93effd640 1 --2- 172.21.15.76:0/314758845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb94010b700 0x7fb940100a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb93effd640 1 -- 172.21.15.76:0/314758845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb9401023c0 msgr2=0x7fb940100010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb93effd640 1 --2- 172.21.15.76:0/314758845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb9401023c0 0x7fb940100010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb93effd640 1 -- 172.21.15.76:0/314758845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb930009560 con 0x7fb940102ce0 2024-03-12T06:03:13.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb93effd640 1 --2- 172.21.15.76:0/314758845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb940102ce0 0x7fb940100550 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fb93400b4a0 tx=0x7fb93400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb93cff9640 1 -- 172.21.15.76:0/314758845 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb934010610 con 0x7fb940102ce0 2024-03-12T06:03:13.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb93cff9640 1 -- 172.21.15.76:0/314758845 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb934010c50 con 0x7fb940102ce0 2024-03-12T06:03:13.205 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb93cff9640 1 -- 172.21.15.76:0/314758845 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb934017e20 con 0x7fb940102ce0 2024-03-12T06:03:13.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb945d48640 1 -- 172.21.15.76:0/314758845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9401a6300 con 0x7fb940102ce0 2024-03-12T06:03:13.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.197+0000 7fb945d48640 1 -- 172.21.15.76:0/314758845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9401a67c0 con 0x7fb940102ce0 2024-03-12T06:03:13.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.199+0000 7fb93cff9640 1 -- 172.21.15.76:0/314758845 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb934017480 con 0x7fb940102ce0 2024-03-12T06:03:13.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.199+0000 7fb9489f2740 1 Processor -- start 2024-03-12T06:03:13.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb9489f2740 1 -- start start 2024-03-12T06:03:13.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb9489f2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de01844250 0x55de01844630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb9489f2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de0184d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb9489f2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de0184d750 0x55de0184fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb9489f2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de017ff960 con 0x55de0184d750 2024-03-12T06:03:13.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb9489f2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de017b3310 con 0x55de01844b70 2024-03-12T06:03:13.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb9489f2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de017b0840 con 0x55de01844250 2024-03-12T06:03:13.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de0184d750 0x55de0184fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de0184d750 0x55de0184fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49340/0 (socket says 172.21.15.76:49340) 2024-03-12T06:03:13.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93effd640 1 -- 172.21.15.76:0/3690940131 learned_addr learned my addr 172.21.15.76:0/3690940131 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93f7fe640 1 --2- 172.21.15.76:0/3690940131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de0184d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93cff9640 1 --2- 172.21.15.76:0/314758845 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb9280a1ba0 0x7fb9280a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93ffff640 1 --2- 172.21.15.76:0/3690940131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de01844250 0x55de01844630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93ffff640 1 --2- 172.21.15.76:0/314758845 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb9280a1ba0 0x7fb9280a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93cff9640 1 -- 172.21.15.76:0/314758845 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb9340cce00 con 0x7fb940102ce0 2024-03-12T06:03:13.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93effd640 1 -- 172.21.15.76:0/3690940131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de01844250 msgr2=0x55de01844630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93effd640 1 --2- 172.21.15.76:0/3690940131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de01844250 0x55de01844630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93effd640 1 -- 172.21.15.76:0/3690940131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 msgr2=0x55de0184d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93effd640 1 --2- 172.21.15.76:0/3690940131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de0184d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93effd640 1 -- 172.21.15.76:0/3690940131 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de0176e060 con 0x55de0184d750 2024-03-12T06:03:13.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93f7fe640 1 --2- 172.21.15.76:0/3690940131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de0184d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:03:13.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.200+0000 7fb93ffff640 1 --2- 172.21.15.76:0/3690940131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de01844250 0x55de01844630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:13.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.201+0000 7fb93effd640 1 --2- 172.21.15.76:0/3690940131 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de0184d750 0x55de0184fb40 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb934099840 tx=0x7fb934000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7a9c735782c304 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.201+0000 7fb93ffff640 1 --2- 172.21.15.76:0/314758845 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb9280a1ba0 0x7fb9280a4060 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb938009f10 tx=0x7fb938006d80 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.201+0000 7fb925ffb640 1 -- 172.21.15.76:0/3690940131 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9340d0420 con 0x55de0184d750 2024-03-12T06:03:13.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.201+0000 7fb925ffb640 1 -- 172.21.15.76:0/3690940131 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb9340994a0 con 0x55de0184d750 2024-03-12T06:03:13.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.201+0000 7fb925ffb640 1 -- 172.21.15.76:0/3690940131 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9340a2360 con 0x55de0184d750 2024-03-12T06:03:13.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.201+0000 7fb9489f2740 1 -- 172.21.15.76:0/3690940131 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de0184d750 msgr2=0x55de0184fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.201+0000 7fb9489f2740 1 --2- 172.21.15.76:0/3690940131 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de0184d750 0x55de0184fb40 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb934099840 tx=0x7fb934000bc0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.201+0000 7fb9489f2740 1 -- 172.21.15.76:0/3690940131 shutdown_connections 2024-03-12T06:03:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.201+0000 7fb9489f2740 1 --2- 172.21.15.76:0/3690940131 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de0184d750 0x55de0184fb40 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.201+0000 7fb9489f2740 1 --2- 172.21.15.76:0/3690940131 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de0184d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.201+0000 7fb9489f2740 1 --2- 172.21.15.76:0/3690940131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de01844250 0x55de01844630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.201+0000 7fb9489f2740 1 -- 172.21.15.76:0/3690940131 >> 172.21.15.76:0/3690940131 conn(0x55de0183e980 msgr2=0x55de0183ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.201+0000 7fb9489f2740 1 -- 172.21.15.76:0/3690940131 shutdown_connections 2024-03-12T06:03:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.201+0000 7fb9489f2740 1 -- 172.21.15.76:0/3690940131 wait complete. 2024-03-12T06:03:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.202+0000 7fb9489f2740 1 -- start start 2024-03-12T06:03:13.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.202+0000 7fb9489f2740 1 Processor -- start 2024-03-12T06:03:13.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.202+0000 7fb9489f2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de01844250 0x55de0184a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.202+0000 7fb9489f2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de0184ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.202+0000 7fb9489f2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0184d750 0x55de0184b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.202+0000 7fb9489f2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de017ac020 con 0x55de01844250 2024-03-12T06:03:13.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.202+0000 7fb9489f2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de0147f430 con 0x55de01844b70 2024-03-12T06:03:13.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.202+0000 7fb9489f2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de0143be00 con 0x55de0184d750 2024-03-12T06:03:13.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.202+0000 7fb9489f2740 10 client.0 _ll_register_callbacks cb 0x55de01753cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:13.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.202+0000 7fb93f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de01844250 0x55de0184a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.202+0000 7fb93f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de01844250 0x55de0184a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49348/0 (socket says 172.21.15.76:49348) 2024-03-12T06:03:13.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.202+0000 7fb93f7fe640 1 -- 172.21.15.76:0/2954126845 learned_addr learned my addr 172.21.15.76:0/2954126845 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb93effd640 1 --2- 172.21.15.76:0/2954126845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de0184ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb93ffff640 1 --2- 172.21.15.76:0/2954126845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0184d750 0x55de0184b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb93f7fe640 1 -- 172.21.15.76:0/2954126845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0184d750 msgr2=0x55de0184b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb93f7fe640 1 --2- 172.21.15.76:0/2954126845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0184d750 0x55de0184b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb93f7fe640 1 -- 172.21.15.76:0/2954126845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 msgr2=0x55de0184ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb93f7fe640 1 --2- 172.21.15.76:0/2954126845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de0184ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb93f7fe640 1 -- 172.21.15.76:0/2954126845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb93409ae70 con 0x55de01844250 2024-03-12T06:03:13.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb93ffff640 1 --2- 172.21.15.76:0/2954126845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0184d750 0x55de0184b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:13.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb93f7fe640 1 --2- 172.21.15.76:0/2954126845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de01844250 0x55de0184a740 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fb930000c00 tx=0x7fb93002cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68d406664c91bf41 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb91effd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:13.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb91effd640 1 -- 172.21.15.76:0/2954126845 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb930004510 con 0x55de01844250 2024-03-12T06:03:13.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb91effd640 1 -- 172.21.15.76:0/2954126845 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb930045070 con 0x55de01844250 2024-03-12T06:03:13.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb9489f2740 1 -- 172.21.15.76:0/2954126845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de014ab7e0 con 0x55de01844250 2024-03-12T06:03:13.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb91effd640 1 -- 172.21.15.76:0/2954126845 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb930004df0 con 0x55de01844250 2024-03-12T06:03:13.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb9489f2740 10 client.17154 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:13.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb9489f2740 1 -- 172.21.15.76:0/2954126845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55de017aa6a0 con 0x55de01844250 2024-03-12T06:03:13.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb91effd640 1 -- 172.21.15.76:0/2954126845 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55de014ab7e0 con 0x55de01844250 2024-03-12T06:03:13.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb91effd640 1 -- 172.21.15.76:0/2954126845 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb930050020 con 0x55de01844250 2024-03-12T06:03:13.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb9489f2740 10 client.17154 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:13.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb9489f2740 10 client.17154 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:03:13.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.203+0000 7fb9489f2740 20 client.17154 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:13.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.204+0000 7fb9489f2740 1 --2- 172.21.15.76:0/2954126845 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55de0185dab0 0x55de0185fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.204+0000 7fb9489f2740 4 client.17154 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:03:13.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.204+0000 7fb9489f2740 1 -- 172.21.15.76:0/2954126845 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de014e8720 con 0x55de0185dab0 2024-03-12T06:03:13.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.204+0000 7fb93effd640 1 --2- 172.21.15.76:0/2954126845 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55de0185dab0 0x55de0185fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.204+0000 7fb93effd640 1 --2- 172.21.15.76:0/2954126845 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55de0185dab0 0x55de0185fea0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5da9a208e0f768d server_cookie=955a17b6c62fe4a4 in_seq=0 out_seq=0 2024-03-12T06:03:13.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.204+0000 7fb91effd640 10 client.17154 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:03:13.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb91effd640 1 -- 172.21.15.76:0/2954126845 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55de014e8720 con 0x55de0185dab0 2024-03-12T06:03:13.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb91effd640 10 client.17154 handle_command_reply: tid=0 2024-03-12T06:03:13.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb9489f2740 1 client.17154 shutdown 2024-03-12T06:03:13.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb9489f2740 1 -- 172.21.15.76:0/2954126845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de01844250 msgr2=0x55de0184a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb9489f2740 1 --2- 172.21.15.76:0/2954126845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de01844250 0x55de0184a740 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fb930000c00 tx=0x7fb93002cbd0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb9489f2740 1 -- 172.21.15.76:0/2954126845 shutdown_connections 2024-03-12T06:03:13.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb9489f2740 1 --2- 172.21.15.76:0/2954126845 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55de0185dab0 0x55de0185fea0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb9489f2740 1 --2- 172.21.15.76:0/2954126845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0184d750 0x55de0184b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb91effd640 0 client.17154 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:03:13.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb9489f2740 1 --2- 172.21.15.76:0/2954126845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de0184ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb9489f2740 1 --2- 172.21.15.76:0/2954126845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de01844250 0x55de0184a740 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb9489f2740 1 -- 172.21.15.76:0/2954126845 >> 172.21.15.76:0/2954126845 conn(0x55de0183e980 msgr2=0x55de018406d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb9489f2740 1 -- 172.21.15.76:0/2954126845 shutdown_connections 2024-03-12T06:03:13.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb9489f2740 1 -- 172.21.15.76:0/2954126845 wait complete. 2024-03-12T06:03:13.223 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.205+0000 7fb9489f2740 20 client.17154 trim_cache size 0 max 16384 2024-03-12T06:03:13.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.216+0000 7fb9489f2740 1 Processor -- start 2024-03-12T06:03:13.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.216+0000 7fb9489f2740 1 -- start start 2024-03-12T06:03:13.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb9489f2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de01844250 0x55de0184ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb9489f2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de0184afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb9489f2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de0184d750 0x55de018479a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb9489f2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de01731130 con 0x55de0184d750 2024-03-12T06:03:13.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb9489f2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de01846340 con 0x55de01844b70 2024-03-12T06:03:13.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb9489f2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de0183e7b0 con 0x55de01844250 2024-03-12T06:03:13.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb93ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de0184d750 0x55de018479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb93ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de0184d750 0x55de018479a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49360/0 (socket says 172.21.15.76:49360) 2024-03-12T06:03:13.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb93ffff640 1 -- 172.21.15.76:0/3074201941 learned_addr learned my addr 172.21.15.76:0/3074201941 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb93f7fe640 1 --2- 172.21.15.76:0/3074201941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de01844250 0x55de0184ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb93effd640 1 --2- 172.21.15.76:0/3074201941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de0184afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb93ffff640 1 -- 172.21.15.76:0/3074201941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de01844250 msgr2=0x55de0184ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb93ffff640 1 --2- 172.21.15.76:0/3074201941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de01844250 0x55de0184ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb93ffff640 1 -- 172.21.15.76:0/3074201941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 msgr2=0x55de0184afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb93ffff640 1 --2- 172.21.15.76:0/3074201941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de0184afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb93ffff640 1 -- 172.21.15.76:0/3074201941 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de01746c20 con 0x55de0184d750 2024-03-12T06:03:13.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb93ffff640 1 --2- 172.21.15.76:0/3074201941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de0184d750 0x55de018479a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fb9380154e0 tx=0x7fb938013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe00aa9bc38236e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb9257fa640 1 -- 172.21.15.76:0/3074201941 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9380064e0 con 0x55de0184d750 2024-03-12T06:03:13.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb9257fa640 1 -- 172.21.15.76:0/3074201941 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb93801b050 con 0x55de0184d750 2024-03-12T06:03:13.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.217+0000 7fb9257fa640 1 -- 172.21.15.76:0/3074201941 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938020c60 con 0x55de0184d750 2024-03-12T06:03:13.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.218+0000 7fb9489f2740 1 -- 172.21.15.76:0/3074201941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de0184d750 msgr2=0x55de018479a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.218+0000 7fb9489f2740 1 --2- 172.21.15.76:0/3074201941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de0184d750 0x55de018479a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fb9380154e0 tx=0x7fb938013000 comp rx=0 tx=0).stop 2024-03-12T06:03:13.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.218+0000 7fb9489f2740 1 -- 172.21.15.76:0/3074201941 shutdown_connections 2024-03-12T06:03:13.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.218+0000 7fb9489f2740 1 --2- 172.21.15.76:0/3074201941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de0184d750 0x55de018479a0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.218+0000 7fb9489f2740 1 --2- 172.21.15.76:0/3074201941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de0184afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.218+0000 7fb9489f2740 1 --2- 172.21.15.76:0/3074201941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de01844250 0x55de0184ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.218+0000 7fb9489f2740 1 -- 172.21.15.76:0/3074201941 >> 172.21.15.76:0/3074201941 conn(0x55de0183e980 msgr2=0x55de01864840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.218+0000 7fb9489f2740 1 -- 172.21.15.76:0/3074201941 shutdown_connections 2024-03-12T06:03:13.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.218+0000 7fb9489f2740 1 -- 172.21.15.76:0/3074201941 wait complete. 2024-03-12T06:03:13.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.218+0000 7fb9489f2740 1 -- start start 2024-03-12T06:03:13.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb9489f2740 1 Processor -- start 2024-03-12T06:03:13.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb9489f2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de01844250 0x55de01853860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb9489f2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de01853da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb9489f2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0184d750 0x55de018b6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb9489f2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de018b6d60 con 0x55de01844250 2024-03-12T06:03:13.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb9489f2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de018b6ea0 con 0x55de01844b70 2024-03-12T06:03:13.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb9489f2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de018b7060 con 0x55de0184d750 2024-03-12T06:03:13.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb9489f2740 10 client.0 _ll_register_callbacks cb 0x7fb934095e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:13.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb93f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de01844250 0x55de01853860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb93f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de01844250 0x55de01853860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49376/0 (socket says 172.21.15.76:49376) 2024-03-12T06:03:13.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb93f7fe640 1 -- 172.21.15.76:0/3810718499 learned_addr learned my addr 172.21.15.76:0/3810718499 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb93ffff640 1 --2- 172.21.15.76:0/3810718499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0184d750 0x55de018b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb93effd640 1 --2- 172.21.15.76:0/3810718499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de01853da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb93f7fe640 1 -- 172.21.15.76:0/3810718499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0184d750 msgr2=0x55de018b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb93f7fe640 1 --2- 172.21.15.76:0/3810718499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0184d750 0x55de018b6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb93f7fe640 1 -- 172.21.15.76:0/3810718499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 msgr2=0x55de01853da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb93f7fe640 1 --2- 172.21.15.76:0/3810718499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de01853da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.219+0000 7fb93f7fe640 1 -- 172.21.15.76:0/3810718499 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb938011d60 con 0x55de01844250 2024-03-12T06:03:13.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb93f7fe640 1 --2- 172.21.15.76:0/3810718499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de01844250 0x55de01853860 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb930034510 tx=0x7fb930004bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5411fa5e805217a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb91effd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:13.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb91effd640 1 -- 172.21.15.76:0/3810718499 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb930003a00 con 0x55de01844250 2024-03-12T06:03:13.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb91effd640 1 -- 172.21.15.76:0/3810718499 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb930045040 con 0x55de01844250 2024-03-12T06:03:13.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb9489f2740 1 -- 172.21.15.76:0/3810718499 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de0183f300 con 0x55de01844250 2024-03-12T06:03:13.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb91effd640 1 -- 172.21.15.76:0/3810718499 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb93002c670 con 0x55de01844250 2024-03-12T06:03:13.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb9489f2740 10 client.17160 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:13.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb9489f2740 1 -- 172.21.15.76:0/3810718499 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55de0176c8c0 con 0x55de01844250 2024-03-12T06:03:13.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb91effd640 1 -- 172.21.15.76:0/3810718499 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb93005f020 con 0x55de01844250 2024-03-12T06:03:13.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb91effd640 1 -- 172.21.15.76:0/3810718499 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb930050070 con 0x55de01844250 2024-03-12T06:03:13.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb9489f2740 10 client.17160 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:13.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb9489f2740 10 client.17160 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:03:13.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb9489f2740 20 client.17160 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb9489f2740 1 --2- 172.21.15.76:0/3810718499 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55de018be500 0x55de018c0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb9489f2740 4 client.17160 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:03:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.220+0000 7fb9489f2740 1 -- 172.21.15.76:0/3810718499 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fb9340994a0 con 0x55de018be500 2024-03-12T06:03:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.221+0000 7fb93effd640 1 --2- 172.21.15.76:0/3810718499 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55de018be500 0x55de018c0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.221+0000 7fb93effd640 1 --2- 172.21.15.76:0/3810718499 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55de018be500 0x55de018c0920 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e7de2bfab6bda5e2 server_cookie=787acd5baa97346c in_seq=0 out_seq=0 2024-03-12T06:03:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.221+0000 7fb91effd640 10 client.17160 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:03:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb91effd640 1 -- 172.21.15.76:0/3810718499 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fb9340994a0 con 0x55de018be500 2024-03-12T06:03:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb91effd640 10 client.17160 handle_command_reply: tid=0 2024-03-12T06:03:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb9489f2740 1 client.17160 shutdown 2024-03-12T06:03:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb9489f2740 1 -- 172.21.15.76:0/3810718499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de01844250 msgr2=0x55de01853860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb9489f2740 1 --2- 172.21.15.76:0/3810718499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de01844250 0x55de01853860 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb930034510 tx=0x7fb930004bd0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb9489f2740 1 -- 172.21.15.76:0/3810718499 shutdown_connections 2024-03-12T06:03:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb9489f2740 1 --2- 172.21.15.76:0/3810718499 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55de018be500 0x55de018c0920 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb9489f2740 1 --2- 172.21.15.76:0/3810718499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0184d750 0x55de018b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb9489f2740 1 --2- 172.21.15.76:0/3810718499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55de01844b70 0x55de01853da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb9489f2740 1 --2- 172.21.15.76:0/3810718499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55de01844250 0x55de01853860 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb91effd640 0 client.17160 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:03:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb9489f2740 1 -- 172.21.15.76:0/3810718499 >> 172.21.15.76:0/3810718499 conn(0x55de0183e980 msgr2=0x55de0183d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb9489f2740 1 -- 172.21.15.76:0/3810718499 shutdown_connections 2024-03-12T06:03:13.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb9489f2740 1 -- 172.21.15.76:0/3810718499 wait complete. 2024-03-12T06:03:13.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.222+0000 7fb9489f2740 20 client.17160 trim_cache size 0 max 16384 2024-03-12T06:03:13.238 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:03:13.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.223+0000 7fb945d48640 1 -- 172.21.15.76:0/314758845 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb9280a1ba0 msgr2=0x7fb9280a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.223+0000 7fb945d48640 1 --2- 172.21.15.76:0/314758845 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb9280a1ba0 0x7fb9280a4060 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb938009f10 tx=0x7fb938006d80 comp rx=0 tx=0).stop 2024-03-12T06:03:13.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.223+0000 7fb945d48640 1 -- 172.21.15.76:0/314758845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb940102ce0 msgr2=0x7fb940100550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.223+0000 7fb945d48640 1 --2- 172.21.15.76:0/314758845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb940102ce0 0x7fb940100550 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fb93400b4a0 tx=0x7fb93400b960 comp rx=0 tx=0).stop 2024-03-12T06:03:13.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.223+0000 7fb945d48640 1 -- 172.21.15.76:0/314758845 shutdown_connections 2024-03-12T06:03:13.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.223+0000 7fb945d48640 1 --2- 172.21.15.76:0/314758845 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb9280a1ba0 0x7fb9280a4060 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.223+0000 7fb945d48640 1 --2- 172.21.15.76:0/314758845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb94010b700 0x7fb940100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.223+0000 7fb945d48640 1 --2- 172.21.15.76:0/314758845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb940102ce0 0x7fb940100550 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.223+0000 7fb945d48640 1 --2- 172.21.15.76:0/314758845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb9401023c0 0x7fb940100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.223+0000 7fb945d48640 1 -- 172.21.15.76:0/314758845 >> 172.21.15.76:0/314758845 conn(0x7fb940077230 msgr2=0x7fb9400fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.223+0000 7fb945d48640 1 -- 172.21.15.76:0/314758845 shutdown_connections 2024-03-12T06:03:13.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.223+0000 7fb945d48640 1 -- 172.21.15.76:0/314758845 wait complete. 2024-03-12T06:03:13.241 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:03:13.241 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T06:03:13.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.340+0000 7f856273a640 1 -- 172.21.15.76:0/2488610647 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1004b0 msgr2=0x7f855c10c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.340+0000 7f856273a640 1 --2- 172.21.15.76:0/2488610647 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1004b0 0x7f855c10c690 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f855400ade0 tx=0x7f855402cfd0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.340+0000 7f856273a640 1 -- 172.21.15.76:0/2488610647 shutdown_connections 2024-03-12T06:03:13.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.340+0000 7f856273a640 1 --2- 172.21.15.76:0/2488610647 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1004b0 0x7f855c10c690 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.340+0000 7f856273a640 1 --2- 172.21.15.76:0/2488610647 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f855c0ffaf0 0x7f855c0fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.340+0000 7f856273a640 1 --2- 172.21.15.76:0/2488610647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f855c1058d0 0x7f855c105cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.340+0000 7f856273a640 1 -- 172.21.15.76:0/2488610647 >> 172.21.15.76:0/2488610647 conn(0x7f855c075360 msgr2=0x7f855c075770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.340+0000 7f856273a640 1 -- 172.21.15.76:0/2488610647 shutdown_connections 2024-03-12T06:03:13.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.340+0000 7f856273a640 1 -- 172.21.15.76:0/2488610647 wait complete. 2024-03-12T06:03:13.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f856273a640 1 Processor -- start 2024-03-12T06:03:13.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f856273a640 1 -- start start 2024-03-12T06:03:13.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f856273a640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f855c0ffaf0 0x7f855c197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f856273a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f855c1004b0 0x7f855c197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f856273a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1058d0 0x7f855c19c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f856273a640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855c19c610 con 0x7f855c1058d0 2024-03-12T06:03:13.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f856273a640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855c19c780 con 0x7f855c0ffaf0 2024-03-12T06:03:13.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f856273a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855c19c940 con 0x7f855c1004b0 2024-03-12T06:03:13.348 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f8560cd7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1058d0 0x7f855c19c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f8560cd7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1058d0 0x7f855c19c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49388/0 (socket says 172.21.15.76:49388) 2024-03-12T06:03:13.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f8560cd7640 1 -- 172.21.15.76:0/3076842759 learned_addr learned my addr 172.21.15.76:0/3076842759 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f855b7fe640 1 --2- 172.21.15.76:0/3076842759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f855c1004b0 0x7f855c197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f855bfff640 1 --2- 172.21.15.76:0/3076842759 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f855c0ffaf0 0x7f855c197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f8560cd7640 1 -- 172.21.15.76:0/3076842759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f855c1004b0 msgr2=0x7f855c197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f8560cd7640 1 --2- 172.21.15.76:0/3076842759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f855c1004b0 0x7f855c197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f8560cd7640 1 -- 172.21.15.76:0/3076842759 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f855c0ffaf0 msgr2=0x7f855c197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f8560cd7640 1 --2- 172.21.15.76:0/3076842759 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f855c0ffaf0 0x7f855c197750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.349 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.341+0000 7f8560cd7640 1 -- 172.21.15.76:0/3076842759 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855400aa10 con 0x7f855c1058d0 2024-03-12T06:03:13.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.342+0000 7f8560cd7640 1 --2- 172.21.15.76:0/3076842759 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1058d0 0x7f855c19c010 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8554004af0 tx=0x7f8554035670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.342+0000 7f85597fa640 1 -- 172.21.15.76:0/3076842759 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8554033e50 con 0x7f855c1058d0 2024-03-12T06:03:13.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.342+0000 7f85597fa640 1 -- 172.21.15.76:0/3076842759 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8554003970 con 0x7f855c1058d0 2024-03-12T06:03:13.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.342+0000 7f856273a640 1 -- 172.21.15.76:0/3076842759 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f855c19cbc0 con 0x7f855c1058d0 2024-03-12T06:03:13.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.342+0000 7f85597fa640 1 -- 172.21.15.76:0/3076842759 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85540044a0 con 0x7f855c1058d0 2024-03-12T06:03:13.350 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.342+0000 7f856273a640 1 -- 172.21.15.76:0/3076842759 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f855c19ce90 con 0x7f855c1058d0 2024-03-12T06:03:13.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.344+0000 7f85597fa640 1 -- 172.21.15.76:0/3076842759 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f855403c040 con 0x7f855c1058d0 2024-03-12T06:03:13.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.344+0000 7f85653e9740 1 Processor -- start 2024-03-12T06:03:13.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.344+0000 7f85653e9740 1 -- start start 2024-03-12T06:03:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.344+0000 7f85597fa640 1 --2- 172.21.15.76:0/3076842759 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f85440a1ba0 0x7f85440a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f85653e9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebcf40 0x565419ebd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f85653e9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 0x565419eca620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f855b7fe640 1 --2- 172.21.15.76:0/3076842759 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f85440a1ba0 0x7f85440a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f85597fa640 1 -- 172.21.15.76:0/3076842759 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f85540ef950 con 0x7f855c1058d0 2024-03-12T06:03:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f85653e9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ecab60 0x565419eccf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f85653e9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x565419e78960 con 0x565419ebcf40 2024-03-12T06:03:13.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f85653e9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x565419e2c310 con 0x565419ebd860 2024-03-12T06:03:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f85653e9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x565419e29840 con 0x565419ecab60 2024-03-12T06:03:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f855bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebcf40 0x565419ebd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f855bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebcf40 0x565419ebd320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49404/0 (socket says 172.21.15.76:49404) 2024-03-12T06:03:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f855bfff640 1 -- 172.21.15.76:0/1114676502 learned_addr learned my addr 172.21.15.76:0/1114676502 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f8560cd7640 1 --2- 172.21.15.76:0/1114676502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ecab60 0x565419eccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f855bfff640 1 -- 172.21.15.76:0/1114676502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ecab60 msgr2=0x565419eccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f855bfff640 1 --2- 172.21.15.76:0/1114676502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ecab60 0x565419eccf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f855bfff640 1 -- 172.21.15.76:0/1114676502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 msgr2=0x565419eca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f855b7fe640 1 --2- 172.21.15.76:0/1114676502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 0x565419eca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f855bfff640 1 --2- 172.21.15.76:0/1114676502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 0x565419eca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f855bfff640 1 -- 172.21.15.76:0/1114676502 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565419de7060 con 0x565419ebcf40 2024-03-12T06:03:13.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f855b7fe640 1 --2- 172.21.15.76:0/1114676502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 0x565419eca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:13.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f855b7fe640 1 --2- 172.21.15.76:0/3076842759 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f85440a1ba0 0x7f85440a4060 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f85500058a0 tx=0x7f85500103d0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f855bfff640 1 --2- 172.21.15.76:0/1114676502 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebcf40 0x565419ebd320 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f854c00ac10 tx=0x7f854c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b01eb582a18501 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f85427fc640 1 -- 172.21.15.76:0/1114676502 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c00a6e0 con 0x565419ebcf40 2024-03-12T06:03:13.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f85427fc640 1 -- 172.21.15.76:0/1114676502 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f854c018b20 con 0x565419ebcf40 2024-03-12T06:03:13.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.345+0000 7f85427fc640 1 -- 172.21.15.76:0/1114676502 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c010c20 con 0x565419ebcf40 2024-03-12T06:03:13.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.346+0000 7f85653e9740 1 -- 172.21.15.76:0/1114676502 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebcf40 msgr2=0x565419ebd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.346+0000 7f85653e9740 1 --2- 172.21.15.76:0/1114676502 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebcf40 0x565419ebd320 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f854c00ac10 tx=0x7f854c00f790 comp rx=0 tx=0).stop 2024-03-12T06:03:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.346+0000 7f85653e9740 1 -- 172.21.15.76:0/1114676502 shutdown_connections 2024-03-12T06:03:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.346+0000 7f85653e9740 1 --2- 172.21.15.76:0/1114676502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ecab60 0x565419eccf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.346+0000 7f85653e9740 1 --2- 172.21.15.76:0/1114676502 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 0x565419eca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.346+0000 7f85653e9740 1 --2- 172.21.15.76:0/1114676502 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebcf40 0x565419ebd320 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.346+0000 7f85653e9740 1 -- 172.21.15.76:0/1114676502 >> 172.21.15.76:0/1114676502 conn(0x565419eb7980 msgr2=0x565419eb7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.346+0000 7f85653e9740 1 -- 172.21.15.76:0/1114676502 shutdown_connections 2024-03-12T06:03:13.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.346+0000 7f85653e9740 1 -- 172.21.15.76:0/1114676502 wait complete. 2024-03-12T06:03:13.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.346+0000 7f85653e9740 1 -- start start 2024-03-12T06:03:13.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f85653e9740 1 Processor -- start 2024-03-12T06:03:13.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f85653e9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebcf40 0x565419ec7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f85653e9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 0x565419ec7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f85653e9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ecab60 0x565419ec8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f85653e9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x565419e25020 con 0x565419ebcf40 2024-03-12T06:03:13.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f85653e9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x565419af8430 con 0x565419ebd860 2024-03-12T06:03:13.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f85653e9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x565419ab4e00 con 0x565419ecab60 2024-03-12T06:03:13.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f855bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebcf40 0x565419ec7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f855bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebcf40 0x565419ec7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49410/0 (socket says 172.21.15.76:49410) 2024-03-12T06:03:13.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f85653e9740 10 client.0 _ll_register_callbacks cb 0x565419dcccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:13.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f855bfff640 1 -- 172.21.15.76:0/4039179188 learned_addr learned my addr 172.21.15.76:0/4039179188 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f855b7fe640 1 --2- 172.21.15.76:0/4039179188 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 0x565419ec7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f8560cd7640 1 --2- 172.21.15.76:0/4039179188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ecab60 0x565419ec8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f855bfff640 1 -- 172.21.15.76:0/4039179188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ecab60 msgr2=0x565419ec8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f855bfff640 1 --2- 172.21.15.76:0/4039179188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ecab60 0x565419ec8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f855bfff640 1 -- 172.21.15.76:0/4039179188 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 msgr2=0x565419ec7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f855bfff640 1 --2- 172.21.15.76:0/4039179188 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 0x565419ec7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f855bfff640 1 -- 172.21.15.76:0/4039179188 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c00a840 con 0x565419ebcf40 2024-03-12T06:03:13.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.347+0000 7f855b7fe640 1 --2- 172.21.15.76:0/4039179188 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 0x565419ec7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:13.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f855bfff640 1 --2- 172.21.15.76:0/4039179188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebcf40 0x565419ec7a80 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f854c0107a0 tx=0x7f854c010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67dacfb0fdc4087 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f853b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:13.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f853b7fe640 1 -- 172.21.15.76:0/4039179188 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c00a6e0 con 0x565419ebcf40 2024-03-12T06:03:13.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f853b7fe640 1 -- 172.21.15.76:0/4039179188 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f854c010e40 con 0x565419ebcf40 2024-03-12T06:03:13.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f85653e9740 1 -- 172.21.15.76:0/4039179188 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565419b247e0 con 0x565419ebcf40 2024-03-12T06:03:13.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f853b7fe640 1 -- 172.21.15.76:0/4039179188 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f854c018850 con 0x565419ebcf40 2024-03-12T06:03:13.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f85653e9740 10 client.17172 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:13.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f85653e9740 1 -- 172.21.15.76:0/4039179188 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x565419e236a0 con 0x565419ebcf40 2024-03-12T06:03:13.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f853b7fe640 1 -- 172.21.15.76:0/4039179188 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x565419b247e0 con 0x565419ebcf40 2024-03-12T06:03:13.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f853b7fe640 1 -- 172.21.15.76:0/4039179188 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f854c004b40 con 0x565419ebcf40 2024-03-12T06:03:13.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f85653e9740 10 client.17172 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:13.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f85653e9740 10 client.17172 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:03:13.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f85653e9740 20 client.17172 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:13.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f85653e9740 1 --2- 172.21.15.76:0/4039179188 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x565419edad90 0x565419edd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f85653e9740 4 client.17172 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:03:13.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.348+0000 7f85653e9740 1 -- 172.21.15.76:0/4039179188 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565419b61720 con 0x565419edad90 2024-03-12T06:03:13.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.349+0000 7f855b7fe640 1 --2- 172.21.15.76:0/4039179188 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x565419edad90 0x565419edd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.349+0000 7f853b7fe640 10 client.17172 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:03:13.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.349+0000 7f855b7fe640 1 --2- 172.21.15.76:0/4039179188 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x565419edad90 0x565419edd180 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1d1bf175b6a38d9f server_cookie=f864a24ecc2928c3 in_seq=0 out_seq=0 2024-03-12T06:03:13.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f853b7fe640 1 -- 172.21.15.76:0/4039179188 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x565419b61720 con 0x565419edad90 2024-03-12T06:03:13.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f853b7fe640 10 client.17172 handle_command_reply: tid=0 2024-03-12T06:03:13.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f85653e9740 1 client.17172 shutdown 2024-03-12T06:03:13.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f85653e9740 1 -- 172.21.15.76:0/4039179188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebcf40 msgr2=0x565419ec7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f85653e9740 1 --2- 172.21.15.76:0/4039179188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebcf40 0x565419ec7a80 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f854c0107a0 tx=0x7f854c010c50 comp rx=0 tx=0).stop 2024-03-12T06:03:13.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f85653e9740 1 -- 172.21.15.76:0/4039179188 shutdown_connections 2024-03-12T06:03:13.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f85653e9740 1 --2- 172.21.15.76:0/4039179188 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x565419edad90 0x565419edd180 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f85653e9740 1 --2- 172.21.15.76:0/4039179188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ecab60 0x565419ec8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f85653e9740 1 --2- 172.21.15.76:0/4039179188 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 0x565419ec7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f85653e9740 1 --2- 172.21.15.76:0/4039179188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebcf40 0x565419ec7a80 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f853b7fe640 0 client.17172 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:03:13.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f85653e9740 1 -- 172.21.15.76:0/4039179188 >> 172.21.15.76:0/4039179188 conn(0x565419eb7980 msgr2=0x565419eb96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f85653e9740 1 -- 172.21.15.76:0/4039179188 shutdown_connections 2024-03-12T06:03:13.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f85653e9740 1 -- 172.21.15.76:0/4039179188 wait complete. 2024-03-12T06:03:13.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.350+0000 7f85653e9740 20 client.17172 trim_cache size 0 max 16384 2024-03-12T06:03:13.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.361+0000 7f85653e9740 1 Processor -- start 2024-03-12T06:03:13.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.361+0000 7f85653e9740 1 -- start start 2024-03-12T06:03:13.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.361+0000 7f85653e9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ebcf40 0x565419ec6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.361+0000 7f85653e9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 0x565419ec8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.361+0000 7f85653e9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ecab60 0x565419eb6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.361+0000 7f85653e9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x565419daa130 con 0x565419ecab60 2024-03-12T06:03:13.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.361+0000 7f85653e9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x565419ec28f0 con 0x565419ebd860 2024-03-12T06:03:13.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.361+0000 7f85653e9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x565419ee23d0 con 0x565419ebcf40 2024-03-12T06:03:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.362+0000 7f8560cd7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ecab60 0x565419eb6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.362+0000 7f8560cd7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ecab60 0x565419eb6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49418/0 (socket says 172.21.15.76:49418) 2024-03-12T06:03:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.362+0000 7f8560cd7640 1 -- 172.21.15.76:0/3666549408 learned_addr learned my addr 172.21.15.76:0/3666549408 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.362+0000 7f8560cd7640 1 -- 172.21.15.76:0/3666549408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ebcf40 msgr2=0x565419ec6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.362+0000 7f855bfff640 1 --2- 172.21.15.76:0/3666549408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ebcf40 0x565419ec6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.362+0000 7f8560cd7640 1 --2- 172.21.15.76:0/3666549408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ebcf40 0x565419ec6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.362+0000 7f8560cd7640 1 -- 172.21.15.76:0/3666549408 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 msgr2=0x565419ec8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.362+0000 7f8560cd7640 1 --2- 172.21.15.76:0/3666549408 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 0x565419ec8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.362+0000 7f8560cd7640 1 -- 172.21.15.76:0/3666549408 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565419dbfc20 con 0x565419ecab60 2024-03-12T06:03:13.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.362+0000 7f855bfff640 1 --2- 172.21.15.76:0/3666549408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ebcf40 0x565419ec6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.362+0000 7f8560cd7640 1 --2- 172.21.15.76:0/3666549408 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ecab60 0x565419eb6a00 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f85540b8ba0 tx=0x7f855400a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f36efe882d1bf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.362+0000 7f8541ffb640 1 -- 172.21.15.76:0/3666549408 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85540c07e0 con 0x565419ecab60 2024-03-12T06:03:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.362+0000 7f8541ffb640 1 -- 172.21.15.76:0/3666549408 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85540c4e30 con 0x565419ecab60 2024-03-12T06:03:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.362+0000 7f8541ffb640 1 -- 172.21.15.76:0/3666549408 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85540b8f40 con 0x565419ecab60 2024-03-12T06:03:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 -- 172.21.15.76:0/3666549408 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ecab60 msgr2=0x565419eb6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 --2- 172.21.15.76:0/3666549408 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ecab60 0x565419eb6a00 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f85540b8ba0 tx=0x7f855400a6d0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 -- 172.21.15.76:0/3666549408 shutdown_connections 2024-03-12T06:03:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 --2- 172.21.15.76:0/3666549408 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ecab60 0x565419eb6a00 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 --2- 172.21.15.76:0/3666549408 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ebd860 0x565419ec8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 --2- 172.21.15.76:0/3666549408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ebcf40 0x565419ec6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 -- 172.21.15.76:0/3666549408 >> 172.21.15.76:0/3666549408 conn(0x565419eb7980 msgr2=0x565419eb62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 -- 172.21.15.76:0/3666549408 shutdown_connections 2024-03-12T06:03:13.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 -- 172.21.15.76:0/3666549408 wait complete. 2024-03-12T06:03:13.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 -- start start 2024-03-12T06:03:13.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 Processor -- start 2024-03-12T06:03:13.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ebcf40 0x565419ed0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebd860 0x565419ed0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ecab60 0x565419ed1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x565419f31470 con 0x565419ebd860 2024-03-12T06:03:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x565419f315e0 con 0x565419ecab60 2024-03-12T06:03:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x565419f317a0 con 0x565419ebcf40 2024-03-12T06:03:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f85653e9740 10 client.0 _ll_register_callbacks cb 0x7f854c00a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.363+0000 7f855b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebd860 0x565419ed0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f855b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebd860 0x565419ed0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49424/0 (socket says 172.21.15.76:49424) 2024-03-12T06:03:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f855b7fe640 1 -- 172.21.15.76:0/1473524225 learned_addr learned my addr 172.21.15.76:0/1473524225 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f8560cd7640 1 --2- 172.21.15.76:0/1473524225 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ecab60 0x565419ed1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f855b7fe640 1 -- 172.21.15.76:0/1473524225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ebcf40 msgr2=0x565419ed0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f855bfff640 1 --2- 172.21.15.76:0/1473524225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ebcf40 0x565419ed0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f855b7fe640 1 --2- 172.21.15.76:0/1473524225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ebcf40 0x565419ed0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f855b7fe640 1 -- 172.21.15.76:0/1473524225 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ecab60 msgr2=0x565419ed1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f855b7fe640 1 --2- 172.21.15.76:0/1473524225 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ecab60 0x565419ed1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f855b7fe640 1 -- 172.21.15.76:0/1473524225 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f855c19ce90 con 0x565419ebd860 2024-03-12T06:03:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f8560cd7640 1 --2- 172.21.15.76:0/1473524225 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ecab60 0x565419ed1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f855b7fe640 1 --2- 172.21.15.76:0/1473524225 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebd860 0x565419ed0c90 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x565419ebe160 tx=0x7f85500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9a3974e3fc2dec server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f853b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:13.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f853b7fe640 1 -- 172.21.15.76:0/1473524225 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8550021810 con 0x565419ebd860 2024-03-12T06:03:13.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f853b7fe640 1 -- 172.21.15.76:0/1473524225 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f8550019040 con 0x565419ebd860 2024-03-12T06:03:13.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f85653e9740 1 -- 172.21.15.76:0/1473524225 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f854c018b20 con 0x565419ebd860 2024-03-12T06:03:13.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f853b7fe640 1 -- 172.21.15.76:0/1473524225 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8550021970 con 0x565419ebd860 2024-03-12T06:03:13.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f85653e9740 10 client.17178 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:13.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f85653e9740 1 -- 172.21.15.76:0/1473524225 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x565419de58c0 con 0x565419ebd860 2024-03-12T06:03:13.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.364+0000 7f853b7fe640 1 -- 172.21.15.76:0/1473524225 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8550038020 con 0x565419ebd860 2024-03-12T06:03:13.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.365+0000 7f853b7fe640 1 -- 172.21.15.76:0/1473524225 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8550033cf0 con 0x565419ebd860 2024-03-12T06:03:13.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.365+0000 7f85653e9740 10 client.17178 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.365+0000 7f85653e9740 10 client.17178 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:03:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.365+0000 7f85653e9740 20 client.17178 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.365+0000 7f85653e9740 1 --2- 172.21.15.76:0/1473524225 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x565419f33000 0x565419f35420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.365+0000 7f85653e9740 4 client.17178 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:03:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.365+0000 7f85653e9740 1 -- 172.21.15.76:0/1473524225 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x565419bb0890 con 0x565419f33000 2024-03-12T06:03:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.365+0000 7f855bfff640 1 --2- 172.21.15.76:0/1473524225 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x565419f33000 0x565419f35420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.365+0000 7f855bfff640 1 --2- 172.21.15.76:0/1473524225 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x565419f33000 0x565419f35420 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=365f71b50ec73864 server_cookie=5ba7474617d5127e in_seq=0 out_seq=0 2024-03-12T06:03:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.365+0000 7f853b7fe640 10 client.17178 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:03:13.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.365+0000 7f853b7fe640 1 -- 172.21.15.76:0/1473524225 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x565419bb0890 con 0x565419f33000 2024-03-12T06:03:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.365+0000 7f853b7fe640 10 client.17178 handle_command_reply: tid=0 2024-03-12T06:03:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.365+0000 7f85653e9740 1 client.17178 shutdown 2024-03-12T06:03:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f85653e9740 1 -- 172.21.15.76:0/1473524225 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebd860 msgr2=0x565419ed0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f85653e9740 1 --2- 172.21.15.76:0/1473524225 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebd860 0x565419ed0c90 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x565419ebe160 tx=0x7f85500046c0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f85653e9740 1 -- 172.21.15.76:0/1473524225 shutdown_connections 2024-03-12T06:03:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f85653e9740 1 --2- 172.21.15.76:0/1473524225 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x565419f33000 0x565419f35420 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f85653e9740 1 --2- 172.21.15.76:0/1473524225 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x565419ecab60 0x565419ed1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f853b7fe640 0 client.17178 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:03:13.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f85653e9740 1 --2- 172.21.15.76:0/1473524225 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565419ebd860 0x565419ed0c90 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f85653e9740 1 --2- 172.21.15.76:0/1473524225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565419ebcf40 0x565419ed0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f85653e9740 1 -- 172.21.15.76:0/1473524225 >> 172.21.15.76:0/1473524225 conn(0x565419eb7980 msgr2=0x565419eb6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f85653e9740 1 -- 172.21.15.76:0/1473524225 shutdown_connections 2024-03-12T06:03:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f85653e9740 1 -- 172.21.15.76:0/1473524225 wait complete. 2024-03-12T06:03:13.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f85653e9740 20 client.17178 trim_cache size 0 max 16384 2024-03-12T06:03:13.383 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:03:13.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f856273a640 1 -- 172.21.15.76:0/3076842759 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f85440a1ba0 msgr2=0x7f85440a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f856273a640 1 --2- 172.21.15.76:0/3076842759 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f85440a1ba0 0x7f85440a4060 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f85500058a0 tx=0x7f85500103d0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f856273a640 1 -- 172.21.15.76:0/3076842759 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1058d0 msgr2=0x7f855c19c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f856273a640 1 --2- 172.21.15.76:0/3076842759 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1058d0 0x7f855c19c010 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8554004af0 tx=0x7f8554035670 comp rx=0 tx=0).stop 2024-03-12T06:03:13.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f856273a640 1 -- 172.21.15.76:0/3076842759 shutdown_connections 2024-03-12T06:03:13.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f856273a640 1 --2- 172.21.15.76:0/3076842759 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f85440a1ba0 0x7f85440a4060 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f856273a640 1 --2- 172.21.15.76:0/3076842759 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855c1058d0 0x7f855c19c010 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f856273a640 1 --2- 172.21.15.76:0/3076842759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f855c1004b0 0x7f855c197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f856273a640 1 --2- 172.21.15.76:0/3076842759 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f855c0ffaf0 0x7f855c197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f856273a640 1 -- 172.21.15.76:0/3076842759 >> 172.21.15.76:0/3076842759 conn(0x7f855c075360 msgr2=0x7f855c0fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f856273a640 1 -- 172.21.15.76:0/3076842759 shutdown_connections 2024-03-12T06:03:13.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.366+0000 7f856273a640 1 -- 172.21.15.76:0/3076842759 wait complete. 2024-03-12T06:03:13.386 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:03:13.387 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T06:03:13.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.487+0000 7fc853981640 1 -- 172.21.15.76:0/2811329325 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc84c105360 msgr2=0x7fc84c10cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.487+0000 7fc853981640 1 --2- 172.21.15.76:0/2811329325 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc84c105360 0x7fc84c10cf30 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc844009cd0 tx=0x7fc84402c0f0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.487+0000 7fc853981640 1 -- 172.21.15.76:0/2811329325 shutdown_connections 2024-03-12T06:03:13.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.487+0000 7fc853981640 1 --2- 172.21.15.76:0/2811329325 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc84c105360 0x7fc84c10cf30 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.487+0000 7fc853981640 1 --2- 172.21.15.76:0/2811329325 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc84c100c40 0x7fc84c104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.487+0000 7fc853981640 1 --2- 172.21.15.76:0/2811329325 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc84c100290 0x7fc84c100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.487+0000 7fc853981640 1 -- 172.21.15.76:0/2811329325 >> 172.21.15.76:0/2811329325 conn(0x7fc84c077230 msgr2=0x7fc84c077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.487+0000 7fc853981640 1 -- 172.21.15.76:0/2811329325 shutdown_connections 2024-03-12T06:03:13.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.487+0000 7fc853981640 1 -- 172.21.15.76:0/2811329325 wait complete. 2024-03-12T06:03:13.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc853981640 1 Processor -- start 2024-03-12T06:03:13.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc853981640 1 -- start start 2024-03-12T06:03:13.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc853981640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc84c100290 0x7fc84c19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc853981640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc84c105360 0x7fc84c19d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc853981640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc84c1975e0 0x7fc84c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc853981640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c197fd0 con 0x7fc84c100290 2024-03-12T06:03:13.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc853981640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c198140 con 0x7fc84c105360 2024-03-12T06:03:13.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc853981640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c198300 con 0x7fc84c1975e0 2024-03-12T06:03:13.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc85171d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc84c100290 0x7fc84c19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc85171d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc84c100290 0x7fc84c19d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49430/0 (socket says 172.21.15.76:49430) 2024-03-12T06:03:13.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc85171d640 1 -- 172.21.15.76:0/4175910551 learned_addr learned my addr 172.21.15.76:0/4175910551 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc850f1c640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc84c105360 0x7fc84c19d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc851f1e640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc84c1975e0 0x7fc84c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc85171d640 1 -- 172.21.15.76:0/4175910551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc84c1975e0 msgr2=0x7fc84c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc85171d640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc84c1975e0 0x7fc84c197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc85171d640 1 -- 172.21.15.76:0/4175910551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc84c105360 msgr2=0x7fc84c19d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc85171d640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc84c105360 0x7fc84c19d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc85171d640 1 -- 172.21.15.76:0/4175910551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc844009900 con 0x7fc84c100290 2024-03-12T06:03:13.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc850f1c640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc84c105360 0x7fc84c19d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:13.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc851f1e640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc84c1975e0 0x7fc84c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:13.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc85171d640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc84c100290 0x7fc84c19d430 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fc84800d950 tx=0x7fc84800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc8427fc640 1 -- 172.21.15.76:0/4175910551 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc848009a30 con 0x7fc84c100290 2024-03-12T06:03:13.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc8427fc640 1 -- 172.21.15.76:0/4175910551 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc848009b90 con 0x7fc84c100290 2024-03-12T06:03:13.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc8427fc640 1 -- 172.21.15.76:0/4175910551 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc848004a00 con 0x7fc84c100290 2024-03-12T06:03:13.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.488+0000 7fc853981640 1 -- 172.21.15.76:0/4175910551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84c1a42f0 con 0x7fc84c100290 2024-03-12T06:03:13.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.489+0000 7fc853981640 1 -- 172.21.15.76:0/4175910551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84c1a4850 con 0x7fc84c100290 2024-03-12T06:03:13.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.491+0000 7fc8427fc640 1 -- 172.21.15.76:0/4175910551 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc848009d00 con 0x7fc84c100290 2024-03-12T06:03:13.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc856632740 1 Processor -- start 2024-03-12T06:03:13.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc856632740 1 -- start start 2024-03-12T06:03:13.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc856632740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd152250 0x55d3cd152630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc856632740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 0x55d3cd15b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc856632740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd15b750 0x55d3cd15db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc856632740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3cd10d960 con 0x55d3cd152250 2024-03-12T06:03:13.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc856632740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3cd0c1310 con 0x55d3cd15b750 2024-03-12T06:03:13.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc856632740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3cd0be840 con 0x55d3cd152b70 2024-03-12T06:03:13.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc851f1e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd152250 0x55d3cd152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc851f1e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd152250 0x55d3cd152630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49436/0 (socket says 172.21.15.76:49436) 2024-03-12T06:03:13.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc851f1e640 1 -- 172.21.15.76:0/4144104061 learned_addr learned my addr 172.21.15.76:0/4144104061 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc8427fc640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc8380a1c20 0x7fc8380a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc850f1c640 1 --2- 172.21.15.76:0/4144104061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd15b750 0x55d3cd15db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc85171d640 1 --2- 172.21.15.76:0/4144104061 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 0x55d3cd15b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc851f1e640 1 -- 172.21.15.76:0/4144104061 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 msgr2=0x55d3cd15b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc851f1e640 1 --2- 172.21.15.76:0/4144104061 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 0x55d3cd15b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc851f1e640 1 -- 172.21.15.76:0/4144104061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd15b750 msgr2=0x55d3cd15db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc851f1e640 1 --2- 172.21.15.76:0/4144104061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd15b750 0x55d3cd15db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc8427fc640 1 -- 172.21.15.76:0/4175910551 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fc8480cd250 con 0x7fc84c100290 2024-03-12T06:03:13.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc851f1e640 1 -- 172.21.15.76:0/4144104061 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3cd07c060 con 0x55d3cd152250 2024-03-12T06:03:13.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc850f1c640 1 --2- 172.21.15.76:0/4144104061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd15b750 0x55d3cd15db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:13.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc85171d640 1 --2- 172.21.15.76:0/4144104061 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 0x55d3cd15b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:13.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.492+0000 7fc851f1e640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc8380a1c20 0x7fc8380a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.493+0000 7fc851f1e640 1 --2- 172.21.15.76:0/4144104061 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd152250 0x55d3cd152630 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fc844037040 tx=0x7fc844032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec193d96a9d39551 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.493+0000 7fc8377fe640 1 -- 172.21.15.76:0/4144104061 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844034c90 con 0x55d3cd152250 2024-03-12T06:03:13.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.493+0000 7fc8377fe640 1 -- 172.21.15.76:0/4144104061 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc844034df0 con 0x55d3cd152250 2024-03-12T06:03:13.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.493+0000 7fc8377fe640 1 -- 172.21.15.76:0/4144104061 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844004040 con 0x55d3cd152250 2024-03-12T06:03:13.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.493+0000 7fc851f1e640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc8380a1c20 0x7fc8380a40e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc84404b000 tx=0x7fc84404c000 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.493+0000 7fc856632740 1 -- 172.21.15.76:0/4144104061 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd152250 msgr2=0x55d3cd152630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.493+0000 7fc856632740 1 --2- 172.21.15.76:0/4144104061 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd152250 0x55d3cd152630 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fc844037040 tx=0x7fc844032ea0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.493+0000 7fc856632740 1 -- 172.21.15.76:0/4144104061 shutdown_connections 2024-03-12T06:03:13.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.493+0000 7fc856632740 1 --2- 172.21.15.76:0/4144104061 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd15b750 0x55d3cd15db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.493+0000 7fc856632740 1 --2- 172.21.15.76:0/4144104061 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 0x55d3cd15b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc856632740 1 --2- 172.21.15.76:0/4144104061 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd152250 0x55d3cd152630 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc856632740 1 -- 172.21.15.76:0/4144104061 >> 172.21.15.76:0/4144104061 conn(0x55d3cd14c980 msgr2=0x55d3cd14cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc856632740 1 -- 172.21.15.76:0/4144104061 shutdown_connections 2024-03-12T06:03:13.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc856632740 1 -- 172.21.15.76:0/4144104061 wait complete. 2024-03-12T06:03:13.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc856632740 1 -- start start 2024-03-12T06:03:13.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc856632740 1 Processor -- start 2024-03-12T06:03:13.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc856632740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152250 0x55d3cd1587d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc856632740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 0x55d3cd158d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc856632740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd15b750 0x55d3cd159250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc856632740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3cd0ba020 con 0x55d3cd15b750 2024-03-12T06:03:13.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc856632740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3ccd8d430 con 0x55d3cd152250 2024-03-12T06:03:13.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc856632740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3ccd49e00 con 0x55d3cd152b70 2024-03-12T06:03:13.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc856632740 10 client.0 _ll_register_callbacks cb 0x55d3cd061cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:13.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc851f1e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd15b750 0x55d3cd159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc851f1e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd15b750 0x55d3cd159250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49450/0 (socket says 172.21.15.76:49450) 2024-03-12T06:03:13.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.494+0000 7fc851f1e640 1 -- 172.21.15.76:0/4053527436 learned_addr learned my addr 172.21.15.76:0/4053527436 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc850f1c640 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 0x55d3cd158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc85171d640 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152250 0x55d3cd1587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc851f1e640 1 -- 172.21.15.76:0/4053527436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 msgr2=0x55d3cd158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc851f1e640 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 0x55d3cd158d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc851f1e640 1 -- 172.21.15.76:0/4053527436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152250 msgr2=0x55d3cd1587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc851f1e640 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152250 0x55d3cd1587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc851f1e640 1 -- 172.21.15.76:0/4053527436 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84c10d740 con 0x55d3cd15b750 2024-03-12T06:03:13.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc850f1c640 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 0x55d3cd158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:13.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc85171d640 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152250 0x55d3cd1587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:13.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc851f1e640 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd15b750 0x55d3cd159250 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fc844005e30 tx=0x7fc844034f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e789240e1b1b55ae server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc834ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:13.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc834ff9640 1 -- 172.21.15.76:0/4053527436 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844003970 con 0x55d3cd15b750 2024-03-12T06:03:13.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc834ff9640 1 -- 172.21.15.76:0/4053527436 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fc844003ad0 con 0x55d3cd15b750 2024-03-12T06:03:13.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc856632740 1 -- 172.21.15.76:0/4053527436 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d3ccdb97e0 con 0x55d3cd15b750 2024-03-12T06:03:13.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc834ff9640 1 -- 172.21.15.76:0/4053527436 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8440326b0 con 0x55d3cd15b750 2024-03-12T06:03:13.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc856632740 10 client.17190 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:13.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc856632740 1 -- 172.21.15.76:0/4053527436 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d3cd0b86a0 con 0x55d3cd15b750 2024-03-12T06:03:13.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.495+0000 7fc834ff9640 1 -- 172.21.15.76:0/4053527436 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d3ccdb97e0 con 0x55d3cd15b750 2024-03-12T06:03:13.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.496+0000 7fc834ff9640 1 -- 172.21.15.76:0/4053527436 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc84403e040 con 0x55d3cd15b750 2024-03-12T06:03:13.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.496+0000 7fc856632740 10 client.17190 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:13.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.496+0000 7fc856632740 10 client.17190 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:13.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.496+0000 7fc856632740 20 client.17190 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:13.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.496+0000 7fc856632740 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d3cd16b920 0x55d3cd16dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.496+0000 7fc856632740 4 client.17190 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:03:13.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.496+0000 7fc856632740 1 -- 172.21.15.76:0/4053527436 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d3ccdf6720 con 0x55d3cd16b920 2024-03-12T06:03:13.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.496+0000 7fc85171d640 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d3cd16b920 0x55d3cd16dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.496+0000 7fc834ff9640 10 client.17190 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:13.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.496+0000 7fc85171d640 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d3cd16b920 0x55d3cd16dd40 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=986ecd88c336ec81 server_cookie=1228bddc98ef0b84 in_seq=0 out_seq=0 2024-03-12T06:03:13.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc834ff9640 1 -- 172.21.15.76:0/4053527436 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d3ccdf6720 con 0x55d3cd16b920 2024-03-12T06:03:13.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc834ff9640 10 client.17190 handle_command_reply: tid=0 2024-03-12T06:03:13.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc856632740 1 client.17190 shutdown 2024-03-12T06:03:13.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc856632740 1 -- 172.21.15.76:0/4053527436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd15b750 msgr2=0x55d3cd159250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc856632740 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd15b750 0x55d3cd159250 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fc844005e30 tx=0x7fc844034f60 comp rx=0 tx=0).stop 2024-03-12T06:03:13.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc856632740 1 -- 172.21.15.76:0/4053527436 shutdown_connections 2024-03-12T06:03:13.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc856632740 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d3cd16b920 0x55d3cd16dd40 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc856632740 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd15b750 0x55d3cd159250 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc856632740 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 0x55d3cd158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc856632740 1 --2- 172.21.15.76:0/4053527436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152250 0x55d3cd1587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc856632740 1 -- 172.21.15.76:0/4053527436 >> 172.21.15.76:0/4053527436 conn(0x55d3cd14c980 msgr2=0x55d3cd14e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc834ff9640 0 client.17190 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:13.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc856632740 1 -- 172.21.15.76:0/4053527436 shutdown_connections 2024-03-12T06:03:13.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc856632740 1 -- 172.21.15.76:0/4053527436 wait complete. 2024-03-12T06:03:13.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.497+0000 7fc856632740 20 client.17190 trim_cache size 0 max 16384 2024-03-12T06:03:13.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.509+0000 7fc856632740 1 Processor -- start 2024-03-12T06:03:13.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.509+0000 7fc856632740 1 -- start start 2024-03-12T06:03:13.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.509+0000 7fc856632740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152250 0x55d3cd15af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.509+0000 7fc856632740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 0x55d3cd159070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.509+0000 7fc856632740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd15b750 0x55d3cd155a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.509+0000 7fc856632740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3cd03f130 con 0x55d3cd15b750 2024-03-12T06:03:13.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.509+0000 7fc856632740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3cd14d980 con 0x55d3cd152250 2024-03-12T06:03:13.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.509+0000 7fc856632740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3cd14c4b0 con 0x55d3cd152b70 2024-03-12T06:03:13.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.509+0000 7fc851f1e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd15b750 0x55d3cd155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.509+0000 7fc851f1e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd15b750 0x55d3cd155a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49466/0 (socket says 172.21.15.76:49466) 2024-03-12T06:03:13.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.509+0000 7fc851f1e640 1 -- 172.21.15.76:0/4232920271 learned_addr learned my addr 172.21.15.76:0/4232920271 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc851f1e640 1 -- 172.21.15.76:0/4232920271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 msgr2=0x55d3cd159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc851f1e640 1 --2- 172.21.15.76:0/4232920271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 0x55d3cd159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc851f1e640 1 -- 172.21.15.76:0/4232920271 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152250 msgr2=0x55d3cd15af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc851f1e640 1 --2- 172.21.15.76:0/4232920271 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152250 0x55d3cd15af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc851f1e640 1 -- 172.21.15.76:0/4232920271 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3cd054c20 con 0x55d3cd15b750 2024-03-12T06:03:13.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc851f1e640 1 --2- 172.21.15.76:0/4232920271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd15b750 0x55d3cd155a30 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fc844001220 tx=0x7fc844004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9043046dfedb89e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc836ffd640 1 -- 172.21.15.76:0/4232920271 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844034440 con 0x55d3cd15b750 2024-03-12T06:03:13.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc836ffd640 1 -- 172.21.15.76:0/4232920271 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc84402c810 con 0x55d3cd15b750 2024-03-12T06:03:13.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc836ffd640 1 -- 172.21.15.76:0/4232920271 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844032910 con 0x55d3cd15b750 2024-03-12T06:03:13.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc856632740 1 -- 172.21.15.76:0/4232920271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd15b750 msgr2=0x55d3cd155a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc856632740 1 --2- 172.21.15.76:0/4232920271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd15b750 0x55d3cd155a30 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fc844001220 tx=0x7fc844004a40 comp rx=0 tx=0).stop 2024-03-12T06:03:13.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc856632740 1 -- 172.21.15.76:0/4232920271 shutdown_connections 2024-03-12T06:03:13.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc856632740 1 --2- 172.21.15.76:0/4232920271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd15b750 0x55d3cd155a30 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc856632740 1 --2- 172.21.15.76:0/4232920271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd152b70 0x55d3cd159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc856632740 1 --2- 172.21.15.76:0/4232920271 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152250 0x55d3cd15af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.510+0000 7fc856632740 1 -- 172.21.15.76:0/4232920271 >> 172.21.15.76:0/4232920271 conn(0x55d3cd14c980 msgr2=0x55d3cd153a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.511+0000 7fc856632740 1 -- 172.21.15.76:0/4232920271 shutdown_connections 2024-03-12T06:03:13.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.511+0000 7fc856632740 1 -- 172.21.15.76:0/4232920271 wait complete. 2024-03-12T06:03:13.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.511+0000 7fc856632740 1 -- start start 2024-03-12T06:03:13.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.511+0000 7fc856632740 1 Processor -- start 2024-03-12T06:03:13.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.511+0000 7fc856632740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd152250 0x55d3cd162990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.511+0000 7fc856632740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152b70 0x55d3cd160d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.511+0000 7fc856632740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd15b750 0x55d3cd161270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.511+0000 7fc856632740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3cd161970 con 0x55d3cd152250 2024-03-12T06:03:13.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.511+0000 7fc856632740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3cd161ae0 con 0x55d3cd152b70 2024-03-12T06:03:13.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.511+0000 7fc856632740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3cd1c1e30 con 0x55d3cd15b750 2024-03-12T06:03:13.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.511+0000 7fc856632740 10 client.0 _ll_register_callbacks cb 0x7fc84c06aa10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:13.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.511+0000 7fc85171d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd152250 0x55d3cd162990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.511+0000 7fc85171d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd152250 0x55d3cd162990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49474/0 (socket says 172.21.15.76:49474) 2024-03-12T06:03:13.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.511+0000 7fc85171d640 1 -- 172.21.15.76:0/1848473459 learned_addr learned my addr 172.21.15.76:0/1848473459 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:13.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc850f1c640 1 --2- 172.21.15.76:0/1848473459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152b70 0x55d3cd160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc851f1e640 1 --2- 172.21.15.76:0/1848473459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd15b750 0x55d3cd161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc85171d640 1 -- 172.21.15.76:0/1848473459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd15b750 msgr2=0x55d3cd161270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc85171d640 1 --2- 172.21.15.76:0/1848473459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd15b750 0x55d3cd161270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc85171d640 1 -- 172.21.15.76:0/1848473459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152b70 msgr2=0x55d3cd160d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:13.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc85171d640 1 --2- 172.21.15.76:0/1848473459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152b70 0x55d3cd160d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc85171d640 1 -- 172.21.15.76:0/1848473459 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d3cd0b86a0 con 0x55d3cd152250 2024-03-12T06:03:13.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc851f1e640 1 --2- 172.21.15.76:0/1848473459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd15b750 0x55d3cd161270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:13.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc85171d640 1 --2- 172.21.15.76:0/1848473459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd152250 0x55d3cd162990 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc848099b80 tx=0x7fc848020f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4245a17549c0d565 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:13.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc834ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:13.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc834ff9640 1 -- 172.21.15.76:0/1848473459 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc848099e80 con 0x55d3cd152250 2024-03-12T06:03:13.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc834ff9640 1 -- 172.21.15.76:0/1848473459 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fc84809dfd0 con 0x55d3cd152250 2024-03-12T06:03:13.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc856632740 1 -- 172.21.15.76:0/1848473459 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc844034df0 con 0x55d3cd152250 2024-03-12T06:03:13.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc834ff9640 1 -- 172.21.15.76:0/1848473459 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8480a2620 con 0x55d3cd152250 2024-03-12T06:03:13.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc856632740 10 client.17196 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:13.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc856632740 1 -- 172.21.15.76:0/1848473459 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d3cd07a8c0 con 0x55d3cd152250 2024-03-12T06:03:13.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc834ff9640 1 -- 172.21.15.76:0/1848473459 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc844034df0 con 0x55d3cd152250 2024-03-12T06:03:13.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc834ff9640 1 -- 172.21.15.76:0/1848473459 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc84809a170 con 0x55d3cd152250 2024-03-12T06:03:13.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc856632740 10 client.17196 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:13.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc856632740 10 client.17196 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:13.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc856632740 20 client.17196 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:13.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc856632740 1 --2- 172.21.15.76:0/1848473459 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d3cd1c3540 0x55d3cd1c5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:13.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc856632740 4 client.17196 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:03:13.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.512+0000 7fc856632740 1 -- 172.21.15.76:0/1848473459 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d3cce45890 con 0x55d3cd1c3540 2024-03-12T06:03:13.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.513+0000 7fc850f1c640 1 --2- 172.21.15.76:0/1848473459 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d3cd1c3540 0x55d3cd1c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:13.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.513+0000 7fc834ff9640 10 client.17196 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:13.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.513+0000 7fc850f1c640 1 --2- 172.21.15.76:0/1848473459 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d3cd1c3540 0x55d3cd1c5960 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2a168aa3b3a4535 server_cookie=aacd097acb21294c in_seq=0 out_seq=0 2024-03-12T06:03:13.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.513+0000 7fc834ff9640 1 -- 172.21.15.76:0/1848473459 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d3cce45890 con 0x55d3cd1c3540 2024-03-12T06:03:13.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.513+0000 7fc834ff9640 10 client.17196 handle_command_reply: tid=0 2024-03-12T06:03:13.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.513+0000 7fc856632740 1 client.17196 shutdown 2024-03-12T06:03:13.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.513+0000 7fc856632740 1 -- 172.21.15.76:0/1848473459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd152250 msgr2=0x55d3cd162990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.513+0000 7fc856632740 1 --2- 172.21.15.76:0/1848473459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd152250 0x55d3cd162990 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc848099b80 tx=0x7fc848020f60 comp rx=0 tx=0).stop 2024-03-12T06:03:13.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.513+0000 7fc856632740 1 -- 172.21.15.76:0/1848473459 shutdown_connections 2024-03-12T06:03:13.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.513+0000 7fc856632740 1 --2- 172.21.15.76:0/1848473459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3cd15b750 0x55d3cd161270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.513+0000 7fc856632740 1 --2- 172.21.15.76:0/1848473459 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3cd152b70 0x55d3cd160d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc856632740 1 --2- 172.21.15.76:0/1848473459 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d3cd1c3540 0x55d3cd1c5960 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc856632740 1 --2- 172.21.15.76:0/1848473459 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3cd152250 0x55d3cd162990 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc834ff9640 0 client.17196 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:13.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc856632740 1 -- 172.21.15.76:0/1848473459 >> 172.21.15.76:0/1848473459 conn(0x55d3cd14c980 msgr2=0x55d3cd14b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc856632740 1 -- 172.21.15.76:0/1848473459 shutdown_connections 2024-03-12T06:03:13.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc856632740 1 -- 172.21.15.76:0/1848473459 wait complete. 2024-03-12T06:03:13.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc856632740 20 client.17196 trim_cache size 0 max 16384 2024-03-12T06:03:13.531 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:03:13.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc853981640 1 -- 172.21.15.76:0/4175910551 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc8380a1c20 msgr2=0x7fc8380a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc853981640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc8380a1c20 0x7fc8380a40e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc84404b000 tx=0x7fc84404c000 comp rx=0 tx=0).stop 2024-03-12T06:03:13.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc853981640 1 -- 172.21.15.76:0/4175910551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc84c100290 msgr2=0x7fc84c19d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:13.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc853981640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc84c100290 0x7fc84c19d430 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fc84800d950 tx=0x7fc84800de10 comp rx=0 tx=0).stop 2024-03-12T06:03:13.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc853981640 1 -- 172.21.15.76:0/4175910551 shutdown_connections 2024-03-12T06:03:13.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc853981640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc8380a1c20 0x7fc8380a40e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc853981640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc84c1975e0 0x7fc84c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc853981640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc84c105360 0x7fc84c19d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc853981640 1 --2- 172.21.15.76:0/4175910551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc84c100290 0x7fc84c19d430 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:13.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc853981640 1 -- 172.21.15.76:0/4175910551 >> 172.21.15.76:0/4175910551 conn(0x7fc84c077230 msgr2=0x7fc84c068790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:13.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc853981640 1 -- 172.21.15.76:0/4175910551 shutdown_connections 2024-03-12T06:03:13.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:13.514+0000 7fc853981640 1 -- 172.21.15.76:0/4175910551 wait complete. 2024-03-12T06:03:13.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:13 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1560883811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:03:13.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:13 smithi076 ceph-mon[32095]: pgmap v230: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-12T06:03:13.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:13 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1560883811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:03:13.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:13 smithi082 ceph-mon[36382]: pgmap v230: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-12T06:03:13.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:13 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1560883811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:03:13.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:13 smithi088 ceph-mon[36585]: pgmap v230: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-12T06:03:14.535 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T06:03:14.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:03:14.536 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T06:03:14.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.653+0000 7f1355793640 1 -- 172.21.15.76:0/3058432340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1350102d50 msgr2=0x7f13501031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.658 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.653+0000 7f1355793640 1 --2- 172.21.15.76:0/3058432340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1350102d50 0x7f13501031b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f1344009930 tx=0x7f134402bc30 comp rx=0 tx=0).stop 2024-03-12T06:03:14.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.653+0000 7f1355793640 1 -- 172.21.15.76:0/3058432340 shutdown_connections 2024-03-12T06:03:14.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.653+0000 7f1355793640 1 --2- 172.21.15.76:0/3058432340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13501036f0 0x7f135010e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.653+0000 7f1355793640 1 --2- 172.21.15.76:0/3058432340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1350102d50 0x7f13501031b0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.653+0000 7f1355793640 1 --2- 172.21.15.76:0/3058432340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1350101470 0x7f1350101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.653+0000 7f1355793640 1 -- 172.21.15.76:0/3058432340 >> 172.21.15.76:0/3058432340 conn(0x7f13500fcc10 msgr2=0x7f13500ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:14.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.653+0000 7f1355793640 1 -- 172.21.15.76:0/3058432340 shutdown_connections 2024-03-12T06:03:14.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.653+0000 7f1355793640 1 -- 172.21.15.76:0/3058432340 wait complete. 2024-03-12T06:03:14.659 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.654+0000 7f1355793640 1 Processor -- start 2024-03-12T06:03:14.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.654+0000 7f1355793640 1 -- start start 2024-03-12T06:03:14.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.654+0000 7f1355793640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1350101470 0x7f135019d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.654+0000 7f1355793640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1350102d50 0x7f135019db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.654+0000 7f1355793640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13501036f0 0x7f13501976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.654+0000 7f1355793640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1350197e50 con 0x7f1350102d50 2024-03-12T06:03:14.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.654+0000 7f1355793640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1350197fc0 con 0x7f13501036f0 2024-03-12T06:03:14.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.654+0000 7f1355793640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1350198180 con 0x7f1350101470 2024-03-12T06:03:14.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.654+0000 7f134e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1350102d50 0x7f135019db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.654+0000 7f134e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1350102d50 0x7f135019db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49482/0 (socket says 172.21.15.76:49482) 2024-03-12T06:03:14.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.654+0000 7f134e7fc640 1 -- 172.21.15.76:0/2932927663 learned_addr learned my addr 172.21.15.76:0/2932927663 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:14.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.654+0000 7f134effd640 1 --2- 172.21.15.76:0/2932927663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1350101470 0x7f135019d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.655+0000 7f134f7fe640 1 --2- 172.21.15.76:0/2932927663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13501036f0 0x7f13501976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.655+0000 7f134e7fc640 1 -- 172.21.15.76:0/2932927663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1350101470 msgr2=0x7f135019d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.655+0000 7f134e7fc640 1 --2- 172.21.15.76:0/2932927663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1350101470 0x7f135019d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.655+0000 7f134e7fc640 1 -- 172.21.15.76:0/2932927663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13501036f0 msgr2=0x7f13501976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.655+0000 7f134e7fc640 1 --2- 172.21.15.76:0/2932927663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13501036f0 0x7f13501976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.655+0000 7f134e7fc640 1 -- 172.21.15.76:0/2932927663 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1344009560 con 0x7f1350102d50 2024-03-12T06:03:14.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.655+0000 7f134e7fc640 1 --2- 172.21.15.76:0/2932927663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1350102d50 0x7f135019db40 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1344009230 tx=0x7f134402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:14.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.655+0000 7f1337fff640 1 -- 172.21.15.76:0/2932927663 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1344004350 con 0x7f1350102d50 2024-03-12T06:03:14.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.655+0000 7f1337fff640 1 -- 172.21.15.76:0/2932927663 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13440044b0 con 0x7f1350102d50 2024-03-12T06:03:14.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.655+0000 7f1355793640 1 -- 172.21.15.76:0/2932927663 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1350198400 con 0x7f1350102d50 2024-03-12T06:03:14.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.655+0000 7f1337fff640 1 -- 172.21.15.76:0/2932927663 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1344003d70 con 0x7f1350102d50 2024-03-12T06:03:14.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.655+0000 7f1355793640 1 -- 172.21.15.76:0/2932927663 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13501a4580 con 0x7f1350102d50 2024-03-12T06:03:14.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.657+0000 7f1337fff640 1 -- 172.21.15.76:0/2932927663 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f1344004df0 con 0x7f1350102d50 2024-03-12T06:03:14.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f135836e740 1 Processor -- start 2024-03-12T06:03:14.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f135836e740 1 -- start start 2024-03-12T06:03:14.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f135836e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5b1e0 0x5617b3f5b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f135836e740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5bb90 0x5617b3f646d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f135836e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 0x5617b3f65170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f135836e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b3ec9f90 con 0x5617b3f5b1e0 2024-03-12T06:03:14.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f135836e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b3ec74c0 con 0x5617b3f64cf0 2024-03-12T06:03:14.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f135836e740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b3ec2ca0 con 0x5617b3f5bb90 2024-03-12T06:03:14.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f1337fff640 1 --2- 172.21.15.76:0/2932927663 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f13380a1ba0 0x7f13380a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f134f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5b1e0 0x5617b3f5b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f134f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5b1e0 0x5617b3f5b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49488/0 (socket says 172.21.15.76:49488) 2024-03-12T06:03:14.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f134f7fe640 1 -- 172.21.15.76:0/1480077701 learned_addr learned my addr 172.21.15.76:0/1480077701 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:14.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f134f7fe640 1 --2- 172.21.15.76:0/2932927663 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f13380a1ba0 0x7f13380a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f134effd640 1 --2- 172.21.15.76:0/1480077701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5bb90 0x5617b3f646d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f1337fff640 1 -- 172.21.15.76:0/2932927663 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f13440ef830 con 0x7f1350102d50 2024-03-12T06:03:14.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f134e7fc640 1 --2- 172.21.15.76:0/1480077701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 0x5617b3f65170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f134f7fe640 1 -- 172.21.15.76:0/1480077701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5bb90 msgr2=0x5617b3f646d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f134f7fe640 1 --2- 172.21.15.76:0/1480077701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5bb90 0x5617b3f646d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.658+0000 7f134f7fe640 1 -- 172.21.15.76:0/1480077701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 msgr2=0x5617b3f65170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.659+0000 7f134f7fe640 1 --2- 172.21.15.76:0/1480077701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 0x5617b3f65170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.659+0000 7f134f7fe640 1 -- 172.21.15.76:0/1480077701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617b3e850b0 con 0x5617b3f5b1e0 2024-03-12T06:03:14.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.659+0000 7f134effd640 1 --2- 172.21.15.76:0/1480077701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5bb90 0x5617b3f646d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:14.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.659+0000 7f134e7fc640 1 --2- 172.21.15.76:0/1480077701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 0x5617b3f65170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:14.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.659+0000 7f134f7fe640 1 --2- 172.21.15.76:0/1480077701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5b1e0 0x5617b3f5b5c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1348014c40 tx=0x7f13480117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d7614f690ddeb47 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:14.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.659+0000 7f134f7fe640 1 --2- 172.21.15.76:0/2932927663 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f13380a1ba0 0x7f13380a4060 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1348011cb0 tx=0x7f1348016000 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:14.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.659+0000 7f13357fa640 1 -- 172.21.15.76:0/1480077701 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1348022070 con 0x5617b3f5b1e0 2024-03-12T06:03:14.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.659+0000 7f13357fa640 1 -- 172.21.15.76:0/1480077701 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f134801a040 con 0x5617b3f5b1e0 2024-03-12T06:03:14.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.659+0000 7f13357fa640 1 -- 172.21.15.76:0/1480077701 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1348009da0 con 0x5617b3f5b1e0 2024-03-12T06:03:14.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 -- 172.21.15.76:0/1480077701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5b1e0 msgr2=0x5617b3f5b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 --2- 172.21.15.76:0/1480077701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5b1e0 0x5617b3f5b5c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1348014c40 tx=0x7f13480117b0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 -- 172.21.15.76:0/1480077701 shutdown_connections 2024-03-12T06:03:14.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 --2- 172.21.15.76:0/1480077701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 0x5617b3f65170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 --2- 172.21.15.76:0/1480077701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5bb90 0x5617b3f646d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 --2- 172.21.15.76:0/1480077701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5b1e0 0x5617b3f5b5c0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 -- 172.21.15.76:0/1480077701 >> 172.21.15.76:0/1480077701 conn(0x5617b3f55840 msgr2=0x5617b3f55c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:14.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 -- 172.21.15.76:0/1480077701 shutdown_connections 2024-03-12T06:03:14.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 -- 172.21.15.76:0/1480077701 wait complete. 2024-03-12T06:03:14.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 -- start start 2024-03-12T06:03:14.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 Processor -- start 2024-03-12T06:03:14.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5b1e0 0x5617b3f68760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3f68ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 0x5617b3f691e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b3e67f40 con 0x5617b3f5bb90 2024-03-12T06:03:14.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b3b964b0 con 0x5617b3f64cf0 2024-03-12T06:03:14.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b3e48670 con 0x5617b3f5b1e0 2024-03-12T06:03:14.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.660+0000 7f135836e740 10 client.0 _ll_register_callbacks cb 0x5617b3eb5f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:14.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f134e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3f68ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f134e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3f68ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49500/0 (socket says 172.21.15.76:49500) 2024-03-12T06:03:14.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f134e7fc640 1 -- 172.21.15.76:0/1297417541 learned_addr learned my addr 172.21.15.76:0/1297417541 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:14.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f134f7fe640 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 0x5617b3f691e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f134effd640 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5b1e0 0x5617b3f68760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f134e7fc640 1 -- 172.21.15.76:0/1297417541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5b1e0 msgr2=0x5617b3f68760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f134e7fc640 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5b1e0 0x5617b3f68760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f134e7fc640 1 -- 172.21.15.76:0/1297417541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 msgr2=0x5617b3f691e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f134e7fc640 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 0x5617b3f691e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f134e7fc640 1 -- 172.21.15.76:0/1297417541 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1348014820 con 0x5617b3f5bb90 2024-03-12T06:03:14.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f134effd640 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5b1e0 0x5617b3f68760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:14.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f134f7fe640 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 0x5617b3f691e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:14.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f134e7fc640 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3f68ca0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f13440ed0f0 tx=0x7f1344009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1926f6c4358a3dc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:14.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f1332ff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:14.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f1332ff5640 1 -- 172.21.15.76:0/1297417541 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13440c06c0 con 0x5617b3f5bb90 2024-03-12T06:03:14.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f135836e740 1 -- 172.21.15.76:0/1297417541 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5617b3e9cf40 con 0x5617b3f5bb90 2024-03-12T06:03:14.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f1332ff5640 1 -- 172.21.15.76:0/1297417541 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f13440c4d10 con 0x5617b3f5bb90 2024-03-12T06:03:14.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f1332ff5640 1 -- 172.21.15.76:0/1297417541 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13440dc0b0 con 0x5617b3f5bb90 2024-03-12T06:03:14.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f135836e740 10 client.17208 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:14.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.661+0000 7f135836e740 1 -- 172.21.15.76:0/1297417541 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5617b3ec1320 con 0x5617b3f5bb90 2024-03-12T06:03:14.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.662+0000 7f1332ff5640 1 -- 172.21.15.76:0/1297417541 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5617b3e9cf40 con 0x5617b3f5bb90 2024-03-12T06:03:14.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.662+0000 7f1332ff5640 1 -- 172.21.15.76:0/1297417541 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f134403c070 con 0x5617b3f5bb90 2024-03-12T06:03:14.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.662+0000 7f135836e740 10 client.17208 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:14.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.662+0000 7f135836e740 10 client.17208 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:14.679 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.662+0000 7f135836e740 20 client.17208 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:14.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.662+0000 7f135836e740 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5617b3f79050 0x5617b3f7b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.662+0000 7f135836e740 4 client.17208 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:03:14.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.662+0000 7f135836e740 1 -- 172.21.15.76:0/1297417541 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5617b3eb9310 con 0x5617b3f79050 2024-03-12T06:03:14.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.662+0000 7f134effd640 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5617b3f79050 0x5617b3f7b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.662+0000 7f1332ff5640 10 client.17208 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:14.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.662+0000 7f134effd640 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5617b3f79050 0x5617b3f7b470 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53e00206dfcb87a8 server_cookie=c4aa691c8d0f51f5 in_seq=0 out_seq=0 2024-03-12T06:03:14.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f1332ff5640 1 -- 172.21.15.76:0/1297417541 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5617b3eb9310 con 0x5617b3f79050 2024-03-12T06:03:14.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f1332ff5640 10 client.17208 handle_command_reply: tid=0 2024-03-12T06:03:14.680 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f135836e740 1 client.17208 shutdown 2024-03-12T06:03:14.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f135836e740 1 -- 172.21.15.76:0/1297417541 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 msgr2=0x5617b3f68ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f135836e740 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3f68ca0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f13440ed0f0 tx=0x7f1344009260 comp rx=0 tx=0).stop 2024-03-12T06:03:14.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f135836e740 1 -- 172.21.15.76:0/1297417541 shutdown_connections 2024-03-12T06:03:14.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f135836e740 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 0x5617b3f691e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f135836e740 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5617b3f79050 0x5617b3f7b470 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f135836e740 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3f68ca0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f135836e740 1 --2- 172.21.15.76:0/1297417541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5b1e0 0x5617b3f68760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.681 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f1332ff5640 0 client.17208 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:14.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f135836e740 1 -- 172.21.15.76:0/1297417541 >> 172.21.15.76:0/1297417541 conn(0x5617b3f55840 msgr2=0x5617b3f577a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:14.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f135836e740 1 -- 172.21.15.76:0/1297417541 shutdown_connections 2024-03-12T06:03:14.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f135836e740 1 -- 172.21.15.76:0/1297417541 wait complete. 2024-03-12T06:03:14.682 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.663+0000 7f135836e740 20 client.17208 trim_cache size 0 max 16384 2024-03-12T06:03:14.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f135836e740 1 Processor -- start 2024-03-12T06:03:14.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f135836e740 1 -- start start 2024-03-12T06:03:14.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f135836e740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5b1e0 0x5617b3f6ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f135836e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3f660e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f135836e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 0x5617b3f69000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f135836e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b3f66620 con 0x5617b3f5bb90 2024-03-12T06:03:14.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f135836e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b3f66790 con 0x5617b3f64cf0 2024-03-12T06:03:14.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f135836e740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b3f69540 con 0x5617b3f5b1e0 2024-03-12T06:03:14.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f134e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3f660e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f134e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3f660e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49510/0 (socket says 172.21.15.76:49510) 2024-03-12T06:03:14.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f134e7fc640 1 -- 172.21.15.76:0/3864515965 learned_addr learned my addr 172.21.15.76:0/3864515965 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:14.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f134f7fe640 1 --2- 172.21.15.76:0/3864515965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 0x5617b3f69000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f134effd640 1 --2- 172.21.15.76:0/3864515965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5b1e0 0x5617b3f6ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f134e7fc640 1 -- 172.21.15.76:0/3864515965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5b1e0 msgr2=0x5617b3f6ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f134e7fc640 1 --2- 172.21.15.76:0/3864515965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5b1e0 0x5617b3f6ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f134e7fc640 1 -- 172.21.15.76:0/3864515965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 msgr2=0x5617b3f69000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f134e7fc640 1 --2- 172.21.15.76:0/3864515965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 0x5617b3f69000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f134e7fc640 1 -- 172.21.15.76:0/3864515965 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617b3e5dc00 con 0x5617b3f5bb90 2024-03-12T06:03:14.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f134effd640 1 --2- 172.21.15.76:0/3864515965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5b1e0 0x5617b3f6ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:14.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f134e7fc640 1 --2- 172.21.15.76:0/3864515965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3f660e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f13440ed580 tx=0x7f1344003ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a1f3ee2ea39f82 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:14.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f1334ff9640 1 -- 172.21.15.76:0/3864515965 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13440bc340 con 0x5617b3f5bb90 2024-03-12T06:03:14.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.679+0000 7f1334ff9640 1 -- 172.21.15.76:0/3864515965 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13440b8810 con 0x5617b3f5bb90 2024-03-12T06:03:14.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.680+0000 7f1334ff9640 1 -- 172.21.15.76:0/3864515965 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13440ba940 con 0x5617b3f5bb90 2024-03-12T06:03:14.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.680+0000 7f135836e740 1 -- 172.21.15.76:0/3864515965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 msgr2=0x5617b3f660e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.680+0000 7f135836e740 1 --2- 172.21.15.76:0/3864515965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3f660e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f13440ed580 tx=0x7f1344003ed0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.680+0000 7f135836e740 1 -- 172.21.15.76:0/3864515965 shutdown_connections 2024-03-12T06:03:14.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.680+0000 7f135836e740 1 --2- 172.21.15.76:0/3864515965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f64cf0 0x5617b3f69000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.680+0000 7f135836e740 1 --2- 172.21.15.76:0/3864515965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3f660e0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.680+0000 7f135836e740 1 --2- 172.21.15.76:0/3864515965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f5b1e0 0x5617b3f6ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.680+0000 7f135836e740 1 -- 172.21.15.76:0/3864515965 >> 172.21.15.76:0/3864515965 conn(0x5617b3f55840 msgr2=0x5617b3f626a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:14.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.680+0000 7f135836e740 1 -- 172.21.15.76:0/3864515965 shutdown_connections 2024-03-12T06:03:14.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.680+0000 7f135836e740 1 -- 172.21.15.76:0/3864515965 wait complete. 2024-03-12T06:03:14.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.680+0000 7f135836e740 1 -- start start 2024-03-12T06:03:14.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f135836e740 1 Processor -- start 2024-03-12T06:03:14.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f135836e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f5b1e0 0x5617b3fc9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f135836e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3fc9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f135836e740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f64cf0 0x5617b3f6f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f135836e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b3f6f600 con 0x5617b3f5bb90 2024-03-12T06:03:14.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f135836e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b3f6f770 con 0x5617b3f5b1e0 2024-03-12T06:03:14.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f135836e740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617b3f6f930 con 0x5617b3f64cf0 2024-03-12T06:03:14.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f135836e740 10 client.0 _ll_register_callbacks cb 0x7f134800cc40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:14.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f134e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3fc9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f134e7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3fc9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49522/0 (socket says 172.21.15.76:49522) 2024-03-12T06:03:14.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f134e7fc640 1 -- 172.21.15.76:0/3376783822 learned_addr learned my addr 172.21.15.76:0/3376783822 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:14.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f134f7fe640 1 --2- 172.21.15.76:0/3376783822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f64cf0 0x5617b3f6f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f134effd640 1 --2- 172.21.15.76:0/3376783822 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f5b1e0 0x5617b3fc9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f134e7fc640 1 -- 172.21.15.76:0/3376783822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f64cf0 msgr2=0x5617b3f6f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f134e7fc640 1 --2- 172.21.15.76:0/3376783822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f64cf0 0x5617b3f6f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f134e7fc640 1 -- 172.21.15.76:0/3376783822 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f5b1e0 msgr2=0x5617b3fc9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f134e7fc640 1 --2- 172.21.15.76:0/3376783822 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f5b1e0 0x5617b3fc9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f134e7fc640 1 -- 172.21.15.76:0/3376783822 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617b3ec1320 con 0x5617b3f5bb90 2024-03-12T06:03:14.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f134e7fc640 1 --2- 172.21.15.76:0/3376783822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3fc9e70 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f13440ed140 tx=0x7f1344009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=397afcb40cdde64a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:14.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f1332ff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:14.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f1332ff5640 1 -- 172.21.15.76:0/3376783822 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13440c4d10 con 0x5617b3f5bb90 2024-03-12T06:03:14.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.681+0000 7f1332ff5640 1 -- 172.21.15.76:0/3376783822 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f13440c06c0 con 0x5617b3f5bb90 2024-03-12T06:03:14.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.682+0000 7f135836e740 1 -- 172.21.15.76:0/3376783822 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5617b3ee8900 con 0x5617b3f5bb90 2024-03-12T06:03:14.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.682+0000 7f1332ff5640 1 -- 172.21.15.76:0/3376783822 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13440a70b0 con 0x5617b3f5bb90 2024-03-12T06:03:14.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.682+0000 7f135836e740 10 client.17214 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:14.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.682+0000 7f135836e740 1 -- 172.21.15.76:0/3376783822 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5617b3e83910 con 0x5617b3f5bb90 2024-03-12T06:03:14.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.682+0000 7f1332ff5640 1 -- 172.21.15.76:0/3376783822 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5617b3ee8900 con 0x5617b3f5bb90 2024-03-12T06:03:14.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.682+0000 7f1332ff5640 1 -- 172.21.15.76:0/3376783822 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f13440a1050 con 0x5617b3f5bb90 2024-03-12T06:03:14.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.682+0000 7f135836e740 10 client.17214 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:14.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.682+0000 7f135836e740 10 client.17214 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:14.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.682+0000 7f135836e740 20 client.17214 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:14.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.682+0000 7f135836e740 1 --2- 172.21.15.76:0/3376783822 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5617b3fd12e0 0x5617b3fd36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.682+0000 7f135836e740 4 client.17214 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T06:03:14.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.682+0000 7f135836e740 1 -- 172.21.15.76:0/3376783822 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5617b3bc2860 con 0x5617b3fd12e0 2024-03-12T06:03:14.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.682+0000 7f134effd640 1 --2- 172.21.15.76:0/3376783822 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5617b3fd12e0 0x5617b3fd36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.683+0000 7f1332ff5640 10 client.17214 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:14.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.683+0000 7f134effd640 1 --2- 172.21.15.76:0/3376783822 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5617b3fd12e0 0x5617b3fd36d0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e0f22ad0f1f3f8d server_cookie=7ea223709302dbce in_seq=0 out_seq=0 2024-03-12T06:03:14.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f1332ff5640 1 -- 172.21.15.76:0/3376783822 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5617b3bc2860 con 0x5617b3fd12e0 2024-03-12T06:03:14.697 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f1332ff5640 10 client.17214 handle_command_reply: tid=0 2024-03-12T06:03:14.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f135836e740 1 client.17214 shutdown 2024-03-12T06:03:14.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f135836e740 1 -- 172.21.15.76:0/3376783822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 msgr2=0x5617b3fc9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f135836e740 1 --2- 172.21.15.76:0/3376783822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3fc9e70 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f13440ed140 tx=0x7f1344009260 comp rx=0 tx=0).stop 2024-03-12T06:03:14.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f135836e740 1 -- 172.21.15.76:0/3376783822 shutdown_connections 2024-03-12T06:03:14.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f135836e740 1 --2- 172.21.15.76:0/3376783822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b3f64cf0 0x5617b3f6f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f135836e740 1 --2- 172.21.15.76:0/3376783822 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5617b3fd12e0 0x5617b3fd36d0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.698 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f135836e740 1 --2- 172.21.15.76:0/3376783822 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5617b3f5bb90 0x5617b3fc9e70 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f135836e740 1 --2- 172.21.15.76:0/3376783822 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5617b3f5b1e0 0x5617b3fc9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f1332ff5640 0 client.17214 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:14.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f135836e740 1 -- 172.21.15.76:0/3376783822 >> 172.21.15.76:0/3376783822 conn(0x5617b3f55840 msgr2=0x5617b3f66f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:14.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f135836e740 1 -- 172.21.15.76:0/3376783822 shutdown_connections 2024-03-12T06:03:14.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f135836e740 1 -- 172.21.15.76:0/3376783822 wait complete. 2024-03-12T06:03:14.699 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f135836e740 20 client.17214 trim_cache size 0 max 16384 2024-03-12T06:03:14.699 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:03:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T06:03:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "63592dc9-029e-43bc-bffc-3277137ee4ef", 2024-03-12T06:03:14.700 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T06:03:14.700 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:03:14.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f1355793640 1 -- 172.21.15.76:0/2932927663 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f13380a1ba0 msgr2=0x7f13380a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.684+0000 7f1355793640 1 --2- 172.21.15.76:0/2932927663 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f13380a1ba0 0x7f13380a4060 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1348011cb0 tx=0x7f1348016000 comp rx=0 tx=0).stop 2024-03-12T06:03:14.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.685+0000 7f1355793640 1 -- 172.21.15.76:0/2932927663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1350102d50 msgr2=0x7f135019db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.685+0000 7f1355793640 1 --2- 172.21.15.76:0/2932927663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1350102d50 0x7f135019db40 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1344009230 tx=0x7f134402c6c0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.685+0000 7f1355793640 1 -- 172.21.15.76:0/2932927663 shutdown_connections 2024-03-12T06:03:14.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.685+0000 7f1355793640 1 --2- 172.21.15.76:0/2932927663 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f13380a1ba0 0x7f13380a4060 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.685+0000 7f1355793640 1 --2- 172.21.15.76:0/2932927663 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f13501036f0 0x7f13501976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.685+0000 7f1355793640 1 --2- 172.21.15.76:0/2932927663 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1350102d50 0x7f135019db40 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.685+0000 7f1355793640 1 --2- 172.21.15.76:0/2932927663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1350101470 0x7f135019d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.685+0000 7f1355793640 1 -- 172.21.15.76:0/2932927663 >> 172.21.15.76:0/2932927663 conn(0x7f13500fcc10 msgr2=0x7f13500fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:14.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.685+0000 7f1355793640 1 -- 172.21.15.76:0/2932927663 shutdown_connections 2024-03-12T06:03:14.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.685+0000 7f1355793640 1 -- 172.21.15.76:0/2932927663 wait complete. 2024-03-12T06:03:14.703 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:03:14.703 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:03:14.831 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.825+0000 7f6e8700f640 1 -- 172.21.15.76:0/1954586170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e80069110 msgr2=0x7f6e80105de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.825+0000 7f6e8700f640 1 --2- 172.21.15.76:0/1954586170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e80069110 0x7f6e80105de0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009930 tx=0x7f6e7002bc30 comp rx=0 tx=0).stop 2024-03-12T06:03:14.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.825+0000 7f6e8700f640 1 -- 172.21.15.76:0/1954586170 shutdown_connections 2024-03-12T06:03:14.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.825+0000 7f6e8700f640 1 --2- 172.21.15.76:0/1954586170 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e801064c0 0x7f6e8010e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.825+0000 7f6e8700f640 1 --2- 172.21.15.76:0/1954586170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e80069110 0x7f6e80105de0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.825+0000 7f6e8700f640 1 --2- 172.21.15.76:0/1954586170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e80068760 0x7f6e80068b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.832 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.825+0000 7f6e8700f640 1 -- 172.21.15.76:0/1954586170 >> 172.21.15.76:0/1954586170 conn(0x7f6e80077180 msgr2=0x7f6e80077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:14.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.825+0000 7f6e8700f640 1 -- 172.21.15.76:0/1954586170 shutdown_connections 2024-03-12T06:03:14.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.825+0000 7f6e8700f640 1 -- 172.21.15.76:0/1954586170 wait complete. 2024-03-12T06:03:14.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.825+0000 7f6e8700f640 1 Processor -- start 2024-03-12T06:03:14.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e8700f640 1 -- start start 2024-03-12T06:03:14.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e8700f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e80068760 0x7f6e8019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e8700f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e801064c0 0x7f6e8019d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e8700f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e80197600 0x7f6e80197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.833 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e8700f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e80197ff0 con 0x7f6e801064c0 2024-03-12T06:03:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e8700f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e80198160 con 0x7f6e80197600 2024-03-12T06:03:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e8700f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e80198320 con 0x7f6e80068760 2024-03-12T06:03:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e84dab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e80068760 0x7f6e8019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e77fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e801064c0 0x7f6e8019d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e84dab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e80068760 0x7f6e8019d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.76:40220/0 (socket says 172.21.15.76:40220) 2024-03-12T06:03:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e84dab640 1 -- 172.21.15.76:0/396753554 learned_addr learned my addr 172.21.15.76:0/396753554 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e855ac640 1 --2- 172.21.15.76:0/396753554 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e80197600 0x7f6e80197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e77fff640 1 -- 172.21.15.76:0/396753554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e80068760 msgr2=0x7f6e8019d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e77fff640 1 --2- 172.21.15.76:0/396753554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e80068760 0x7f6e8019d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e77fff640 1 -- 172.21.15.76:0/396753554 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e80197600 msgr2=0x7f6e80197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e77fff640 1 --2- 172.21.15.76:0/396753554 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e80197600 0x7f6e80197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.826+0000 7f6e77fff640 1 -- 172.21.15.76:0/396753554 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e78009840 con 0x7f6e801064c0 2024-03-12T06:03:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.827+0000 7f6e77fff640 1 --2- 172.21.15.76:0/396753554 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e801064c0 0x7f6e8019d990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f6e70000c00 tx=0x7f6e70004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.827+0000 7f6e75ffb640 1 -- 172.21.15.76:0/396753554 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e70004450 con 0x7f6e801064c0 2024-03-12T06:03:14.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.827+0000 7f6e8700f640 1 -- 172.21.15.76:0/396753554 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e70009560 con 0x7f6e801064c0 2024-03-12T06:03:14.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.827+0000 7f6e8700f640 1 -- 172.21.15.76:0/396753554 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e80100260 con 0x7f6e801064c0 2024-03-12T06:03:14.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.827+0000 7f6e75ffb640 1 -- 172.21.15.76:0/396753554 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e7002c5d0 con 0x7f6e801064c0 2024-03-12T06:03:14.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.827+0000 7f6e75ffb640 1 -- 172.21.15.76:0/396753554 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e700039f0 con 0x7f6e801064c0 2024-03-12T06:03:14.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.830+0000 7f6e75ffb640 1 -- 172.21.15.76:0/396753554 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6e70003b50 con 0x7f6e801064c0 2024-03-12T06:03:14.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.830+0000 7f6e89bdb740 1 Processor -- start 2024-03-12T06:03:14.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.830+0000 7f6e89bdb740 1 -- start start 2024-03-12T06:03:14.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e89bdb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812cdf2b0 0x55b812cdf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e89bdb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e89bdb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812ce87b0 0x55b812ceaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e89bdb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b812c4e280 con 0x55b812cdf2b0 2024-03-12T06:03:14.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e89bdb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b812c4b7b0 con 0x55b812cdfbd0 2024-03-12T06:03:14.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e89bdb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b812c46f90 con 0x55b812ce87b0 2024-03-12T06:03:14.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e855ac640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812cdf2b0 0x55b812cdf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e84dab640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e84dab640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:38982/0 (socket says 172.21.15.76:38982) 2024-03-12T06:03:14.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e855ac640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812cdf2b0 0x55b812cdf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49552/0 (socket says 172.21.15.76:49552) 2024-03-12T06:03:14.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e84dab640 1 -- 172.21.15.76:0/2927197951 learned_addr learned my addr 172.21.15.76:0/2927197951 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:14.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e75ffb640 1 --2- 172.21.15.76:0/396753554 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f6e680a1ba0 0x7f6e680a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e77fff640 1 --2- 172.21.15.76:0/2927197951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812ce87b0 0x55b812ceaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e855ac640 1 --2- 172.21.15.76:0/396753554 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f6e680a1ba0 0x7f6e680a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e84dab640 1 -- 172.21.15.76:0/2927197951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812ce87b0 msgr2=0x55b812ceaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e84dab640 1 --2- 172.21.15.76:0/2927197951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812ce87b0 0x55b812ceaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e84dab640 1 -- 172.21.15.76:0/2927197951 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812cdf2b0 msgr2=0x55b812cdf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e84dab640 1 --2- 172.21.15.76:0/2927197951 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812cdf2b0 0x55b812cdf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e84dab640 1 -- 172.21.15.76:0/2927197951 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b812c05570 con 0x55b812cdfbd0 2024-03-12T06:03:14.843 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e75ffb640 1 -- 172.21.15.76:0/396753554 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f6e700f30d0 con 0x7f6e801064c0 2024-03-12T06:03:14.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.831+0000 7f6e84dab640 1 --2- 172.21.15.76:0/2927197951 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce8270 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c009e10 tx=0x7f6e7c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=516c3e006d0587c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:14.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.832+0000 7f6e855ac640 1 --2- 172.21.15.76:0/396753554 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f6e680a1ba0 0x7f6e680a4060 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009fd0 tx=0x7f6e78004ab0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:14.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.832+0000 7f6e66ffd640 1 -- 172.21.15.76:0/2927197951 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c019630 con 0x55b812cdfbd0 2024-03-12T06:03:14.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.832+0000 7f6e66ffd640 1 -- 172.21.15.76:0/2927197951 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e7c019c70 con 0x55b812cdfbd0 2024-03-12T06:03:14.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.832+0000 7f6e66ffd640 1 -- 172.21.15.76:0/2927197951 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c00fd20 con 0x55b812cdfbd0 2024-03-12T06:03:14.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.833+0000 7f6e89bdb740 1 -- 172.21.15.76:0/2927197951 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 msgr2=0x55b812ce8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.833+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/2927197951 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce8270 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c009e10 tx=0x7f6e7c011810 comp rx=0 tx=0).stop 2024-03-12T06:03:14.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.833+0000 7f6e89bdb740 1 -- 172.21.15.76:0/2927197951 shutdown_connections 2024-03-12T06:03:14.844 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.833+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/2927197951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812ce87b0 0x55b812ceaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.833+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/2927197951 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce8270 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.833+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/2927197951 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812cdf2b0 0x55b812cdf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.833+0000 7f6e89bdb740 1 -- 172.21.15.76:0/2927197951 >> 172.21.15.76:0/2927197951 conn(0x55b812cd9990 msgr2=0x55b812cd9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:14.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.833+0000 7f6e89bdb740 1 -- 172.21.15.76:0/2927197951 shutdown_connections 2024-03-12T06:03:14.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.833+0000 7f6e89bdb740 1 -- 172.21.15.76:0/2927197951 wait complete. 2024-03-12T06:03:14.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.833+0000 7f6e89bdb740 1 -- start start 2024-03-12T06:03:14.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.833+0000 7f6e89bdb740 1 Processor -- start 2024-03-12T06:03:14.845 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e89bdb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812cdf2b0 0x55b812ce81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e89bdb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e89bdb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812ce87b0 0x55b812ce5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e89bdb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b81291a430 con 0x55b812cdf2b0 2024-03-12T06:03:14.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e89bdb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8128d6e00 con 0x55b812cdfbd0 2024-03-12T06:03:14.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e84dab640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812cdf2b0 0x55b812ce81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e89bdb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b812bcbfd0 con 0x55b812ce87b0 2024-03-12T06:03:14.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e84dab640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812cdf2b0 0x55b812ce81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49568/0 (socket says 172.21.15.76:49568) 2024-03-12T06:03:14.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e89bdb740 10 client.0 _ll_register_callbacks cb 0x55b812c21fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:14.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e84dab640 1 -- 172.21.15.76:0/2873210425 learned_addr learned my addr 172.21.15.76:0/2873210425 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:14.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e855ac640 1 --2- 172.21.15.76:0/2873210425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812ce87b0 0x55b812ce5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e77fff640 1 --2- 172.21.15.76:0/2873210425 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e77fff640 1 -- 172.21.15.76:0/2873210425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812ce87b0 msgr2=0x55b812ce5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e77fff640 1 --2- 172.21.15.76:0/2873210425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812ce87b0 0x55b812ce5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e77fff640 1 -- 172.21.15.76:0/2873210425 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812cdf2b0 msgr2=0x55b812ce81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.834+0000 7f6e77fff640 1 --2- 172.21.15.76:0/2873210425 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812cdf2b0 0x55b812ce81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.835+0000 7f6e77fff640 1 -- 172.21.15.76:0/2873210425 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7c009a70 con 0x55b812cdfbd0 2024-03-12T06:03:14.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.835+0000 7f6e84dab640 1 --2- 172.21.15.76:0/2873210425 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812cdf2b0 0x55b812ce81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:03:14.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.835+0000 7f6e77fff640 1 --2- 172.21.15.76:0/2873210425 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce5550 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6e700f08b0 tx=0x7f6e700bf9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a92c9b853a7cbe2a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:14.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.835+0000 7f6e5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T06:03:14.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.835+0000 7f6e5ffff640 1 -- 172.21.15.76:0/2873210425 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e700f0c30 con 0x55b812cdfbd0 2024-03-12T06:03:14.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.835+0000 7f6e5ffff640 1 -- 172.21.15.76:0/2873210425 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f6e700c3ee0 con 0x55b812cdfbd0 2024-03-12T06:03:14.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.835+0000 7f6e89bdb740 1 -- 172.21.15.76:0/2873210425 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b8129467e0 con 0x55b812cdfbd0 2024-03-12T06:03:14.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.835+0000 7f6e5ffff640 1 -- 172.21.15.76:0/2873210425 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e700c8530 con 0x55b812cdfbd0 2024-03-12T06:03:14.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.835+0000 7f6e89bdb740 10 client.26330 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:14.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.835+0000 7f6e89bdb740 1 -- 172.21.15.76:0/2873210425 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b812c45610 con 0x55b812cdfbd0 2024-03-12T06:03:14.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.836+0000 7f6e5ffff640 1 -- 172.21.15.76:0/2873210425 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b8129467e0 con 0x55b812cdfbd0 2024-03-12T06:03:14.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.836+0000 7f6e5ffff640 1 -- 172.21.15.76:0/2873210425 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6e700b7040 con 0x55b812cdfbd0 2024-03-12T06:03:14.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.836+0000 7f6e89bdb740 10 client.26330 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:14.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.836+0000 7f6e89bdb740 10 client.26330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:14.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.836+0000 7f6e89bdb740 20 client.26330 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:14.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.836+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/2873210425 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55b812cf85d0 0x55b812cfa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.836+0000 7f6e89bdb740 4 client.26330 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:03:14.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.836+0000 7f6e89bdb740 1 -- 172.21.15.76:0/2873210425 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b812983720 con 0x55b812cf85d0 2024-03-12T06:03:14.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.836+0000 7f6e84dab640 1 --2- 172.21.15.76:0/2873210425 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55b812cf85d0 0x55b812cfa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.836+0000 7f6e5ffff640 10 client.26330 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:14.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.836+0000 7f6e84dab640 1 --2- 172.21.15.76:0/2873210425 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55b812cf85d0 0x55b812cfa9f0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dbba7d01f1f9a84 server_cookie=8e81784971fcbf77 in_seq=0 out_seq=0 2024-03-12T06:03:14.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.837+0000 7f6e5ffff640 1 -- 172.21.15.76:0/2873210425 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b812983720 con 0x55b812cf85d0 2024-03-12T06:03:14.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.837+0000 7f6e5ffff640 10 client.26330 handle_command_reply: tid=0 2024-03-12T06:03:14.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.837+0000 7f6e89bdb740 1 client.26330 shutdown 2024-03-12T06:03:14.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.837+0000 7f6e89bdb740 1 -- 172.21.15.76:0/2873210425 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 msgr2=0x55b812ce5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.837+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/2873210425 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce5550 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6e700f08b0 tx=0x7f6e700bf9d0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.837+0000 7f6e89bdb740 1 -- 172.21.15.76:0/2873210425 shutdown_connections 2024-03-12T06:03:14.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.837+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/2873210425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812ce87b0 0x55b812ce5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.837+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/2873210425 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce5550 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.837+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/2873210425 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55b812cf85d0 0x55b812cfa9f0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.838+0000 7f6e5ffff640 0 client.26330 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:14.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.838+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/2873210425 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812cdf2b0 0x55b812ce81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.838+0000 7f6e89bdb740 1 -- 172.21.15.76:0/2873210425 >> 172.21.15.76:0/2873210425 conn(0x55b812cd9990 msgr2=0x55b812cdb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:14.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.838+0000 7f6e89bdb740 1 -- 172.21.15.76:0/2873210425 shutdown_connections 2024-03-12T06:03:14.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.838+0000 7f6e89bdb740 1 -- 172.21.15.76:0/2873210425 wait complete. 2024-03-12T06:03:14.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.838+0000 7f6e89bdb740 20 client.26330 trim_cache size 0 max 16384 2024-03-12T06:03:14.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e89bdb740 1 Processor -- start 2024-03-12T06:03:14.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e89bdb740 1 -- start start 2024-03-12T06:03:14.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e89bdb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812cdf2b0 0x55b812ce7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e89bdb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e89bdb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812ce87b0 0x55b812ce3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e89bdb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b812cf54b0 con 0x55b812ce87b0 2024-03-12T06:03:14.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e89bdb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b812cdecf0 con 0x55b812cdfbd0 2024-03-12T06:03:14.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e89bdb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b812ce6160 con 0x55b812cdf2b0 2024-03-12T06:03:14.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e77fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e855ac640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812ce87b0 0x55b812ce3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e77fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:38990/0 (socket says 172.21.15.76:38990) 2024-03-12T06:03:14.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e77fff640 1 -- 172.21.15.76:0/821673254 learned_addr learned my addr 172.21.15.76:0/821673254 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:14.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e84dab640 1 --2- 172.21.15.76:0/821673254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812cdf2b0 0x55b812ce7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e77fff640 1 -- 172.21.15.76:0/821673254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812cdf2b0 msgr2=0x55b812ce7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e77fff640 1 --2- 172.21.15.76:0/821673254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812cdf2b0 0x55b812ce7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e77fff640 1 -- 172.21.15.76:0/821673254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812ce87b0 msgr2=0x55b812ce3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e77fff640 1 --2- 172.21.15.76:0/821673254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812ce87b0 0x55b812ce3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e77fff640 1 -- 172.21.15.76:0/821673254 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b812be1ac0 con 0x55b812cdfbd0 2024-03-12T06:03:14.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.857+0000 7f6e855ac640 1 --2- 172.21.15.76:0/821673254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812ce87b0 0x55b812ce3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:03:14.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.858+0000 7f6e77fff640 1 --2- 172.21.15.76:0/821673254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce81c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6e700bc710 tx=0x7f6e700bc160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d8cfe4a50e359a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:14.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.858+0000 7f6e667fc640 1 -- 172.21.15.76:0/821673254 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e700f0c30 con 0x55b812cdfbd0 2024-03-12T06:03:14.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.858+0000 7f6e667fc640 1 -- 172.21.15.76:0/821673254 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e700bf510 con 0x55b812cdfbd0 2024-03-12T06:03:14.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.858+0000 7f6e667fc640 1 -- 172.21.15.76:0/821673254 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e700bfff0 con 0x55b812cdfbd0 2024-03-12T06:03:14.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.859+0000 7f6e89bdb740 1 -- 172.21.15.76:0/821673254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 msgr2=0x55b812ce81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.859+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/821673254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce81c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6e700bc710 tx=0x7f6e700bc160 comp rx=0 tx=0).stop 2024-03-12T06:03:14.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.859+0000 7f6e89bdb740 1 -- 172.21.15.76:0/821673254 shutdown_connections 2024-03-12T06:03:14.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.859+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/821673254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812ce87b0 0x55b812ce3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.859+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/821673254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdfbd0 0x55b812ce81c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.859+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/821673254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812cdf2b0 0x55b812ce7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.859+0000 7f6e89bdb740 1 -- 172.21.15.76:0/821673254 >> 172.21.15.76:0/821673254 conn(0x55b812cd9990 msgr2=0x55b812ce0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:14.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.859+0000 7f6e89bdb740 1 -- 172.21.15.76:0/821673254 shutdown_connections 2024-03-12T06:03:14.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.859+0000 7f6e89bdb740 1 -- 172.21.15.76:0/821673254 wait complete. 2024-03-12T06:03:14.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.859+0000 7f6e89bdb740 1 -- start start 2024-03-12T06:03:14.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.859+0000 7f6e89bdb740 1 Processor -- start 2024-03-12T06:03:14.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e89bdb740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdf2b0 0x55b812d46d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e89bdb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812cdfbd0 0x55b812d472d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e89bdb740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812ce87b0 0x55b812ceba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e89bdb740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b812cec040 con 0x55b812ce87b0 2024-03-12T06:03:14.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e89bdb740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b812cec1b0 con 0x55b812cdf2b0 2024-03-12T06:03:14.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e89bdb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b812cec370 con 0x55b812cdfbd0 2024-03-12T06:03:14.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e89bdb740 10 client.0 _ll_register_callbacks cb 0x7f6e7c009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:14.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e855ac640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812ce87b0 0x55b812ceba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e84dab640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdf2b0 0x55b812d46d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e77fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812cdfbd0 0x55b812d472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e855ac640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812ce87b0 0x55b812ceba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:49592/0 (socket says 172.21.15.76:49592) 2024-03-12T06:03:14.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e84dab640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdf2b0 0x55b812d46d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:39004/0 (socket says 172.21.15.76:39004) 2024-03-12T06:03:14.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e855ac640 1 -- 172.21.15.76:0/1283205366 learned_addr learned my addr 172.21.15.76:0/1283205366 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:14.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e855ac640 1 -- 172.21.15.76:0/1283205366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812cdfbd0 msgr2=0x55b812d472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e855ac640 1 --2- 172.21.15.76:0/1283205366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812cdfbd0 0x55b812d472d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e855ac640 1 -- 172.21.15.76:0/1283205366 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdf2b0 msgr2=0x55b812d46d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:14.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e855ac640 1 --2- 172.21.15.76:0/1283205366 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdf2b0 0x55b812d46d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.860+0000 7f6e855ac640 1 -- 172.21.15.76:0/1283205366 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e78013120 con 0x55b812ce87b0 2024-03-12T06:03:14.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.861+0000 7f6e77fff640 1 --2- 172.21.15.76:0/1283205366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812cdfbd0 0x55b812d472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:03:14.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.861+0000 7f6e84dab640 1 --2- 172.21.15.76:0/1283205366 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdf2b0 0x55b812d46d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:03:14.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.861+0000 7f6e855ac640 1 --2- 172.21.15.76:0/1283205366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812ce87b0 0x55b812ceba80 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f6e78003c20 tx=0x7f6e78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2866782c9d1e2f1a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:14.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.861+0000 7f6e5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:14.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.861+0000 7f6e5ffff640 1 -- 172.21.15.76:0/1283205366 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78028070 con 0x55b812ce87b0 2024-03-12T06:03:14.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.861+0000 7f6e5ffff640 1 -- 172.21.15.76:0/1283205366 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f6e78022450 con 0x55b812ce87b0 2024-03-12T06:03:14.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.861+0000 7f6e89bdb740 1 -- 172.21.15.76:0/1283205366 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e7c019c70 con 0x55b812ce87b0 2024-03-12T06:03:14.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.861+0000 7f6e5ffff640 1 -- 172.21.15.76:0/1283205366 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e78003e60 con 0x55b812ce87b0 2024-03-12T06:03:14.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.861+0000 7f6e89bdb740 10 client.17232 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:14.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.861+0000 7f6e89bdb740 1 -- 172.21.15.76:0/1283205366 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f6e7c009a70 con 0x55b812ce87b0 2024-03-12T06:03:14.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.862+0000 7f6e5ffff640 1 -- 172.21.15.76:0/1283205366 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e7c019c70 con 0x55b812ce87b0 2024-03-12T06:03:14.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.862+0000 7f6e5ffff640 1 -- 172.21.15.76:0/1283205366 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6e780367b0 con 0x55b812ce87b0 2024-03-12T06:03:14.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.862+0000 7f6e89bdb740 10 client.17232 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:14.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.862+0000 7f6e89bdb740 10 client.17232 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:14.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.862+0000 7f6e89bdb740 20 client.17232 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:14.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.862+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/1283205366 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55b812cf85d0 0x55b812d51e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:14.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.862+0000 7f6e89bdb740 4 client.17232 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:03:14.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.862+0000 7f6e89bdb740 1 -- 172.21.15.76:0/1283205366 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b8129d2890 con 0x55b812cf85d0 2024-03-12T06:03:14.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.862+0000 7f6e84dab640 1 --2- 172.21.15.76:0/1283205366 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55b812cf85d0 0x55b812d51e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:14.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.862+0000 7f6e5ffff640 10 client.17232 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:14.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e84dab640 1 --2- 172.21.15.76:0/1283205366 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55b812cf85d0 0x55b812d51e50 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef05372142bb99d9 server_cookie=d34b4d029c2ff372 in_seq=0 out_seq=0 2024-03-12T06:03:14.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e5ffff640 1 -- 172.21.15.76:0/1283205366 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55b8129d2890 con 0x55b812cf85d0 2024-03-12T06:03:14.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e5ffff640 10 client.17232 handle_command_reply: tid=0 2024-03-12T06:03:14.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e89bdb740 1 client.17232 shutdown 2024-03-12T06:03:14.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e89bdb740 1 -- 172.21.15.76:0/1283205366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812ce87b0 msgr2=0x55b812ceba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/1283205366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812ce87b0 0x55b812ceba80 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f6e78003c20 tx=0x7f6e78004a40 comp rx=0 tx=0).stop 2024-03-12T06:03:14.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e89bdb740 1 -- 172.21.15.76:0/1283205366 shutdown_connections 2024-03-12T06:03:14.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/1283205366 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55b812cf85d0 0x55b812d51e50 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/1283205366 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b812ce87b0 0x55b812ceba80 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/1283205366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b812cdfbd0 0x55b812d472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e89bdb740 1 --2- 172.21.15.76:0/1283205366 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b812cdf2b0 0x55b812d46d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e89bdb740 1 -- 172.21.15.76:0/1283205366 >> 172.21.15.76:0/1283205366 conn(0x55b812cd9990 msgr2=0x55b812ce3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:14.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e5ffff640 0 client.17232 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:14.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e89bdb740 1 -- 172.21.15.76:0/1283205366 shutdown_connections 2024-03-12T06:03:14.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e89bdb740 1 -- 172.21.15.76:0/1283205366 wait complete. 2024-03-12T06:03:14.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.863+0000 7f6e89bdb740 20 client.17232 trim_cache size 0 max 16384 2024-03-12T06:03:14.878 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:03:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T06:03:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T06:03:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: "63592dc9-029e-43bc-bffc-3277137ee4ef": { 2024-03-12T06:03:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T06:03:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "63592dc9-029e-43bc-bffc-3277137ee4ef", 2024-03-12T06:03:14.878 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T06:03:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:03:14.879 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:03:14.879 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:03:14.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.864+0000 7f6e8700f640 1 -- 172.21.15.76:0/396753554 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f6e680a1ba0 msgr2=0x7f6e680a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.864+0000 7f6e8700f640 1 --2- 172.21.15.76:0/396753554 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f6e680a1ba0 0x7f6e680a4060 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009fd0 tx=0x7f6e78004ab0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.864+0000 7f6e8700f640 1 -- 172.21.15.76:0/396753554 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e801064c0 msgr2=0x7f6e8019d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:14.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.864+0000 7f6e8700f640 1 --2- 172.21.15.76:0/396753554 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e801064c0 0x7f6e8019d990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f6e70000c00 tx=0x7f6e70004950 comp rx=0 tx=0).stop 2024-03-12T06:03:14.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.864+0000 7f6e8700f640 1 -- 172.21.15.76:0/396753554 shutdown_connections 2024-03-12T06:03:14.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.864+0000 7f6e8700f640 1 --2- 172.21.15.76:0/396753554 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f6e680a1ba0 0x7f6e680a4060 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.864+0000 7f6e8700f640 1 --2- 172.21.15.76:0/396753554 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6e80197600 0x7f6e80197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.864+0000 7f6e8700f640 1 --2- 172.21.15.76:0/396753554 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6e801064c0 0x7f6e8019d990 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.864+0000 7f6e8700f640 1 --2- 172.21.15.76:0/396753554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e80068760 0x7f6e8019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:14.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.864+0000 7f6e8700f640 1 -- 172.21.15.76:0/396753554 >> 172.21.15.76:0/396753554 conn(0x7f6e80077180 msgr2=0x7f6e800fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:14.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.864+0000 7f6e8700f640 1 -- 172.21.15.76:0/396753554 shutdown_connections 2024-03-12T06:03:14.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:14.864+0000 7f6e8700f640 1 -- 172.21.15.76:0/396753554 wait complete. 2024-03-12T06:03:14.882 INFO:tasks.cephfs.filesystem:scrub status for tag:63592dc9-029e-43bc-bffc-3277137ee4ef - {'path': '/', 'tag': '63592dc9-029e-43bc-bffc-3277137ee4ef', 'options': 'recursive,force'} 2024-03-12T06:03:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:14 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:03:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:14 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:03:15.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:14 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:15.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:14 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:03:15.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:14 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:03:15.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:14 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:15.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:14 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:03:15.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:14 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:03:15.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:14 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:16.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:15 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T06:03:16.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:15 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:16.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:15 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:03:16.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:15 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:16.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:15 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:16.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:15 smithi088 ceph-mon[36585]: pgmap v231: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-12T06:03:16.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:15 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T06:03:16.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:15 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:16.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:15 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:03:16.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:15 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:16.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:15 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:16.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:15 smithi076 ceph-mon[32095]: pgmap v231: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-12T06:03:16.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:15 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T06:03:16.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:15 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:16.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:15 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:03:16.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:15 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:16.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:15 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:16.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:15 smithi082 ceph-mon[36382]: pgmap v231: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-12T06:03:18.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:18 smithi082 ceph-mon[36382]: pgmap v232: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-03-12T06:03:18.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:18 smithi088 ceph-mon[36585]: pgmap v232: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-03-12T06:03:18.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:18 smithi076 ceph-mon[32095]: pgmap v232: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-03-12T06:03:19.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:19 smithi088 ceph-mon[36585]: pgmap v233: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:03:19.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:19 smithi076 ceph-mon[32095]: pgmap v233: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:03:19.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:19 smithi082 ceph-mon[36382]: pgmap v233: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:03:22.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:22 smithi082 ceph-mon[36382]: pgmap v234: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:03:22.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:22 smithi088 ceph-mon[36585]: pgmap v234: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:03:22.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:22 smithi076 ceph-mon[32095]: pgmap v234: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:03:23.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:23 smithi088 ceph-mon[36585]: pgmap v235: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-12T06:03:23.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:23 smithi076 ceph-mon[32095]: pgmap v235: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-12T06:03:23.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:23 smithi082 ceph-mon[36382]: pgmap v235: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-12T06:03:26.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:26 smithi082 ceph-mon[36382]: pgmap v236: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:26.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:26 smithi088 ceph-mon[36585]: pgmap v236: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:26.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:26 smithi076 ceph-mon[32095]: pgmap v236: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:27.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:27 smithi082 ceph-mon[36382]: pgmap v237: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:27.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:27 smithi088 ceph-mon[36585]: pgmap v237: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:27.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:27 smithi076 ceph-mon[32095]: pgmap v237: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:28.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:28 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:03:28.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:28 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:03:28.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:28 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:03:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:28 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:03:28.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:28 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:03:28.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:28 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:03:28.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:28 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:03:28.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:28 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:03:28.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:28 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:03:29.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:29 smithi082 ceph-mon[36382]: pgmap v238: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:03:29.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:29 smithi088 ceph-mon[36585]: pgmap v238: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:03:29.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:29 smithi076 ceph-mon[32095]: pgmap v238: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:03:32.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:32 smithi082 ceph-mon[36382]: pgmap v239: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:32.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:32 smithi088 ceph-mon[36585]: pgmap v239: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:32.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:32 smithi076 ceph-mon[32095]: pgmap v239: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:33.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:33 smithi082 ceph-mon[36382]: pgmap v240: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:03:33.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:33 smithi088 ceph-mon[36585]: pgmap v240: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:03:33.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:33 smithi076 ceph-mon[32095]: pgmap v240: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:03:36.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:36 smithi082 ceph-mon[36382]: pgmap v241: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:36.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:36 smithi088 ceph-mon[36585]: pgmap v241: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:36.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:36 smithi076 ceph-mon[32095]: pgmap v241: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:37.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:37 smithi082 ceph-mon[36382]: pgmap v242: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:37.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:37 smithi088 ceph-mon[36585]: pgmap v242: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:37.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:37 smithi076 ceph-mon[32095]: pgmap v242: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:39.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:39 smithi088 ceph-mon[36585]: pgmap v243: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:03:39.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:39 smithi076 ceph-mon[32095]: pgmap v243: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:03:39.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:39 smithi082 ceph-mon[36382]: pgmap v243: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:03:41.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:41 smithi088 ceph-mon[36585]: pgmap v244: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:41.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:41 smithi076 ceph-mon[32095]: pgmap v244: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:41.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:41 smithi082 ceph-mon[36382]: pgmap v244: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:43.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:43 smithi088 ceph-mon[36585]: pgmap v245: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:03:43.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:43 smithi076 ceph-mon[32095]: pgmap v245: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:03:43.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:43 smithi082 ceph-mon[36382]: pgmap v245: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:03:44.883 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:03:44.884 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:03:45.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.992+0000 7f9a4ee0e640 1 -- 172.21.15.76:0/4040886204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a48100b50 msgr2=0x7f9a48100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.992+0000 7f9a4ee0e640 1 --2- 172.21.15.76:0/4040886204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a48100b50 0x7f9a48100fb0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009930 tx=0x7f9a3c02bc30 comp rx=0 tx=0).stop 2024-03-12T06:03:45.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 -- 172.21.15.76:0/4040886204 shutdown_connections 2024-03-12T06:03:45.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 --2- 172.21.15.76:0/4040886204 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a481014f0 0x7f9a4810d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 --2- 172.21.15.76:0/4040886204 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a48100b50 0x7f9a48100fb0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.000 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 --2- 172.21.15.76:0/4040886204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a48101fc0 0x7f9a481023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 -- 172.21.15.76:0/4040886204 >> 172.21.15.76:0/4040886204 conn(0x7f9a480753a0 msgr2=0x7f9a480757b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 -- 172.21.15.76:0/4040886204 shutdown_connections 2024-03-12T06:03:45.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 -- 172.21.15.76:0/4040886204 wait complete. 2024-03-12T06:03:45.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 Processor -- start 2024-03-12T06:03:45.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 -- start start 2024-03-12T06:03:45.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a48100b50 0x7f9a48196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a481014f0 0x7f9a48196c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.001 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a48101fc0 0x7f9a4819afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4819b5e0 con 0x7f9a48100b50 2024-03-12T06:03:45.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4819b750 con 0x7f9a481014f0 2024-03-12T06:03:45.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4ee0e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4819b910 con 0x7f9a48101fc0 2024-03-12T06:03:45.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4cbaa640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a48100b50 0x7f9a48196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.993+0000 7f9a4cbaa640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a48100b50 0x7f9a48196720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34578/0 (socket says 172.21.15.76:34578) 2024-03-12T06:03:45.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.994+0000 7f9a4cbaa640 1 -- 172.21.15.76:0/2227369208 learned_addr learned my addr 172.21.15.76:0/2227369208 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.994+0000 7f9a4d3ab640 1 --2- 172.21.15.76:0/2227369208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a48101fc0 0x7f9a4819afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.994+0000 7f9a47fff640 1 --2- 172.21.15.76:0/2227369208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a481014f0 0x7f9a48196c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.002 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.994+0000 7f9a4cbaa640 1 -- 172.21.15.76:0/2227369208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a48101fc0 msgr2=0x7f9a4819afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.994+0000 7f9a4cbaa640 1 --2- 172.21.15.76:0/2227369208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a48101fc0 0x7f9a4819afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.994+0000 7f9a4cbaa640 1 -- 172.21.15.76:0/2227369208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a481014f0 msgr2=0x7f9a48196c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.994+0000 7f9a4cbaa640 1 --2- 172.21.15.76:0/2227369208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a481014f0 0x7f9a48196c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.994+0000 7f9a4cbaa640 1 -- 172.21.15.76:0/2227369208 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c009560 con 0x7f9a48100b50 2024-03-12T06:03:45.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.994+0000 7f9a4cbaa640 1 --2- 172.21.15.76:0/2227369208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a48100b50 0x7f9a48196720 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f9a3800d770 tx=0x7f9a3800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.994+0000 7f9a45ffb640 1 -- 172.21.15.76:0/2227369208 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a38009c30 con 0x7f9a48100b50 2024-03-12T06:03:45.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.994+0000 7f9a45ffb640 1 -- 172.21.15.76:0/2227369208 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a3800de80 con 0x7f9a48100b50 2024-03-12T06:03:45.003 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.994+0000 7f9a45ffb640 1 -- 172.21.15.76:0/2227369208 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a38004e80 con 0x7f9a48100b50 2024-03-12T06:03:45.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.994+0000 7f9a4ee0e640 1 -- 172.21.15.76:0/2227369208 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a4819bbf0 con 0x7f9a48100b50 2024-03-12T06:03:45.004 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.994+0000 7f9a4ee0e640 1 -- 172.21.15.76:0/2227369208 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a481a3470 con 0x7f9a48100b50 2024-03-12T06:03:45.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.996+0000 7f9a45ffb640 1 -- 172.21.15.76:0/2227369208 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9a38005020 con 0x7f9a48100b50 2024-03-12T06:03:45.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a45ffb640 1 --2- 172.21.15.76:0/2227369208 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9a300a1ba0 0x7f9a300a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a519de740 1 Processor -- start 2024-03-12T06:03:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a519de740 1 -- start start 2024-03-12T06:03:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a47fff640 1 --2- 172.21.15.76:0/2227369208 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9a300a1ba0 0x7f9a300a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a45ffb640 1 -- 172.21.15.76:0/2227369208 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9a380cd240 con 0x7f9a48100b50 2024-03-12T06:03:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a519de740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b42b0 0x5563e64b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a519de740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b4bd0 0x5563e64bd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a519de740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e64bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a519de740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e6423280 con 0x5563e64b4bd0 2024-03-12T06:03:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a519de740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e64207b0 con 0x5563e64b42b0 2024-03-12T06:03:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a519de740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e641bf90 con 0x5563e64bd7b0 2024-03-12T06:03:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a47fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b4bd0 0x5563e64bd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a47fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b4bd0 0x5563e64bd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34582/0 (socket says 172.21.15.76:34582) 2024-03-12T06:03:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a47fff640 1 -- 172.21.15.76:0/828023254 learned_addr learned my addr 172.21.15.76:0/828023254 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a4d3ab640 1 --2- 172.21.15.76:0/828023254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e64bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.997+0000 7f9a4cbaa640 1 --2- 172.21.15.76:0/828023254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b42b0 0x5563e64b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.998+0000 7f9a47fff640 1 --2- 172.21.15.76:0/2227369208 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9a300a1ba0 0x7f9a300a4060 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c036040 tx=0x7f9a3c004a60 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.998+0000 7f9a47fff640 1 -- 172.21.15.76:0/828023254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 msgr2=0x5563e64bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.013 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.998+0000 7f9a47fff640 1 --2- 172.21.15.76:0/828023254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e64bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.998+0000 7f9a47fff640 1 -- 172.21.15.76:0/828023254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b42b0 msgr2=0x5563e64b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.998+0000 7f9a47fff640 1 --2- 172.21.15.76:0/828023254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b42b0 0x5563e64b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.998+0000 7f9a47fff640 1 -- 172.21.15.76:0/828023254 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563e63da570 con 0x5563e64b4bd0 2024-03-12T06:03:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.998+0000 7f9a4d3ab640 1 --2- 172.21.15.76:0/828023254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e64bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.998+0000 7f9a47fff640 1 --2- 172.21.15.76:0/828023254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b4bd0 0x5563e64bd270 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009930 tx=0x7f9a3c04a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8734032eb394e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.998+0000 7f9a2effd640 1 -- 172.21.15.76:0/828023254 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c004df0 con 0x5563e64b4bd0 2024-03-12T06:03:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.998+0000 7f9a2effd640 1 -- 172.21.15.76:0/828023254 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a3c005430 con 0x5563e64b4bd0 2024-03-12T06:03:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.998+0000 7f9a2effd640 1 -- 172.21.15.76:0/828023254 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c003d60 con 0x5563e64b4bd0 2024-03-12T06:03:45.014 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.999+0000 7f9a519de740 1 -- 172.21.15.76:0/828023254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b4bd0 msgr2=0x5563e64bd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.999+0000 7f9a519de740 1 --2- 172.21.15.76:0/828023254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b4bd0 0x5563e64bd270 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009930 tx=0x7f9a3c04a000 comp rx=0 tx=0).stop 2024-03-12T06:03:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.999+0000 7f9a519de740 1 -- 172.21.15.76:0/828023254 shutdown_connections 2024-03-12T06:03:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.999+0000 7f9a519de740 1 --2- 172.21.15.76:0/828023254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e64bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.999+0000 7f9a519de740 1 --2- 172.21.15.76:0/828023254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b4bd0 0x5563e64bd270 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.999+0000 7f9a519de740 1 --2- 172.21.15.76:0/828023254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b42b0 0x5563e64b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.999+0000 7f9a519de740 1 -- 172.21.15.76:0/828023254 >> 172.21.15.76:0/828023254 conn(0x5563e64ae990 msgr2=0x5563e64aed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.999+0000 7f9a519de740 1 -- 172.21.15.76:0/828023254 shutdown_connections 2024-03-12T06:03:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.999+0000 7f9a519de740 1 -- 172.21.15.76:0/828023254 wait complete. 2024-03-12T06:03:45.015 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.999+0000 7f9a519de740 1 -- start start 2024-03-12T06:03:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:44.999+0000 7f9a519de740 1 Processor -- start 2024-03-12T06:03:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a519de740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b42b0 0x5563e64bd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a519de740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64b4bd0 0x5563e64ba560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a519de740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64bd7b0 0x5563e64bab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a519de740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e60ef430 con 0x5563e64bd7b0 2024-03-12T06:03:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a519de740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e60abe00 con 0x5563e64b42b0 2024-03-12T06:03:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a519de740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e63a0fd0 con 0x5563e64b4bd0 2024-03-12T06:03:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a519de740 10 client.0 _ll_register_callbacks cb 0x5563e63f6fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a4d3ab640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64bd7b0 0x5563e64bab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a4d3ab640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64bd7b0 0x5563e64bab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34594/0 (socket says 172.21.15.76:34594) 2024-03-12T06:03:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a4d3ab640 1 -- 172.21.15.76:0/2913778547 learned_addr learned my addr 172.21.15.76:0/2913778547 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a4cbaa640 1 --2- 172.21.15.76:0/2913778547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b42b0 0x5563e64bd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a47fff640 1 --2- 172.21.15.76:0/2913778547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64b4bd0 0x5563e64ba560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a4d3ab640 1 -- 172.21.15.76:0/2913778547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64b4bd0 msgr2=0x5563e64ba560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a4d3ab640 1 --2- 172.21.15.76:0/2913778547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64b4bd0 0x5563e64ba560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a4d3ab640 1 -- 172.21.15.76:0/2913778547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b42b0 msgr2=0x5563e64bd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a4d3ab640 1 --2- 172.21.15.76:0/2913778547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b42b0 0x5563e64bd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a4d3ab640 1 -- 172.21.15.76:0/2913778547 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4810d880 con 0x5563e64bd7b0 2024-03-12T06:03:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a4d3ab640 1 --2- 172.21.15.76:0/2913778547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64bd7b0 0x5563e64bab70 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f9a40012720 tx=0x7f9a40012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fb7b0982aa2097 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a27fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a27fff640 1 -- 172.21.15.76:0/2913778547 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a40010d60 con 0x5563e64bd7b0 2024-03-12T06:03:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a27fff640 1 -- 172.21.15.76:0/2913778547 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9a4001a3f0 con 0x5563e64bd7b0 2024-03-12T06:03:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a519de740 1 -- 172.21.15.76:0/2913778547 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5563e611b7e0 con 0x5563e64bd7b0 2024-03-12T06:03:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.000+0000 7f9a27fff640 1 -- 172.21.15.76:0/2913778547 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a40010ec0 con 0x5563e64bd7b0 2024-03-12T06:03:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.001+0000 7f9a519de740 10 client.17244 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.001+0000 7f9a519de740 1 -- 172.21.15.76:0/2913778547 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5563e641a610 con 0x5563e64bd7b0 2024-03-12T06:03:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.001+0000 7f9a27fff640 1 -- 172.21.15.76:0/2913778547 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a40027020 con 0x5563e64bd7b0 2024-03-12T06:03:45.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.001+0000 7f9a27fff640 1 -- 172.21.15.76:0/2913778547 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9a4000e530 con 0x5563e64bd7b0 2024-03-12T06:03:45.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.001+0000 7f9a519de740 10 client.17244 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:45.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.001+0000 7f9a519de740 10 client.17244 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:45.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.001+0000 7f9a519de740 20 client.17244 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:45.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.001+0000 7f9a519de740 1 --2- 172.21.15.76:0/2913778547 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5563e64cda00 0x5563e64cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.001+0000 7f9a519de740 4 client.17244 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:03:45.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.001+0000 7f9a519de740 1 -- 172.21.15.76:0/2913778547 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5563e6158720 con 0x5563e64cda00 2024-03-12T06:03:45.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.001+0000 7f9a4cbaa640 1 --2- 172.21.15.76:0/2913778547 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5563e64cda00 0x5563e64cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.001+0000 7f9a4cbaa640 1 --2- 172.21.15.76:0/2913778547 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5563e64cda00 0x5563e64cfe20 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3993efd3587d4bff server_cookie=f2998216adf4e8fc in_seq=0 out_seq=0 2024-03-12T06:03:45.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.001+0000 7f9a27fff640 10 client.17244 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:45.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a27fff640 1 -- 172.21.15.76:0/2913778547 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5563e6158720 con 0x5563e64cda00 2024-03-12T06:03:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a27fff640 10 client.17244 handle_command_reply: tid=0 2024-03-12T06:03:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a519de740 1 client.17244 shutdown 2024-03-12T06:03:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a519de740 1 -- 172.21.15.76:0/2913778547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64bd7b0 msgr2=0x5563e64bab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a519de740 1 --2- 172.21.15.76:0/2913778547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64bd7b0 0x5563e64bab70 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f9a40012720 tx=0x7f9a40012be0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a519de740 1 -- 172.21.15.76:0/2913778547 shutdown_connections 2024-03-12T06:03:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a519de740 1 --2- 172.21.15.76:0/2913778547 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5563e64cda00 0x5563e64cfe20 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a519de740 1 --2- 172.21.15.76:0/2913778547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64bd7b0 0x5563e64bab70 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a519de740 1 --2- 172.21.15.76:0/2913778547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64b4bd0 0x5563e64ba560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a27fff640 0 client.17244 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:45.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a519de740 1 --2- 172.21.15.76:0/2913778547 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b42b0 0x5563e64bd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a519de740 1 -- 172.21.15.76:0/2913778547 >> 172.21.15.76:0/2913778547 conn(0x5563e64ae990 msgr2=0x5563e64b0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a519de740 1 -- 172.21.15.76:0/2913778547 shutdown_connections 2024-03-12T06:03:45.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a519de740 1 -- 172.21.15.76:0/2913778547 wait complete. 2024-03-12T06:03:45.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.002+0000 7f9a519de740 20 client.17244 trim_cache size 0 max 16384 2024-03-12T06:03:45.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.017+0000 7f9a519de740 1 Processor -- start 2024-03-12T06:03:45.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.017+0000 7f9a519de740 1 -- start start 2024-03-12T06:03:45.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.017+0000 7f9a519de740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b42b0 0x5563e64bcc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.017+0000 7f9a519de740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b4bd0 0x5563e64bd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.017+0000 7f9a519de740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e64c4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.017+0000 7f9a519de740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e64cd7d0 con 0x5563e64b42b0 2024-03-12T06:03:45.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.017+0000 7f9a519de740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e64afa10 con 0x5563e64b4bd0 2024-03-12T06:03:45.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.017+0000 7f9a519de740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e64af7c0 con 0x5563e64bd7b0 2024-03-12T06:03:45.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.018+0000 7f9a4cbaa640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b42b0 0x5563e64bcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.018+0000 7f9a4cbaa640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b42b0 0x5563e64bcc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34596/0 (socket says 172.21.15.76:34596) 2024-03-12T06:03:45.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.018+0000 7f9a4cbaa640 1 -- 172.21.15.76:0/643601116 learned_addr learned my addr 172.21.15.76:0/643601116 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.018+0000 7f9a4d3ab640 1 --2- 172.21.15.76:0/643601116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e64c4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.018+0000 7f9a4cbaa640 1 -- 172.21.15.76:0/643601116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 msgr2=0x5563e64c4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.018+0000 7f9a4cbaa640 1 --2- 172.21.15.76:0/643601116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e64c4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.018+0000 7f9a4cbaa640 1 -- 172.21.15.76:0/643601116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b4bd0 msgr2=0x5563e64bd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.018+0000 7f9a4cbaa640 1 --2- 172.21.15.76:0/643601116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b4bd0 0x5563e64bd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.018+0000 7f9a4cbaa640 1 -- 172.21.15.76:0/643601116 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563e63b6ac0 con 0x5563e64b42b0 2024-03-12T06:03:45.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.018+0000 7f9a4d3ab640 1 --2- 172.21.15.76:0/643601116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e64c4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:45.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.018+0000 7f9a4cbaa640 1 --2- 172.21.15.76:0/643601116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b42b0 0x5563e64bcc90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f9a38096840 tx=0x7f9a380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=676359a0b996bc30 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.018+0000 7f9a2e7fc640 1 -- 172.21.15.76:0/643601116 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a380b6070 con 0x5563e64b42b0 2024-03-12T06:03:45.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.018+0000 7f9a2e7fc640 1 -- 172.21.15.76:0/643601116 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a38009e50 con 0x5563e64b42b0 2024-03-12T06:03:45.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.018+0000 7f9a2e7fc640 1 -- 172.21.15.76:0/643601116 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a380b2ba0 con 0x5563e64b42b0 2024-03-12T06:03:45.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.019+0000 7f9a519de740 1 -- 172.21.15.76:0/643601116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b42b0 msgr2=0x5563e64bcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.019+0000 7f9a519de740 1 --2- 172.21.15.76:0/643601116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b42b0 0x5563e64bcc90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f9a38096840 tx=0x7f9a380046c0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.019+0000 7f9a519de740 1 -- 172.21.15.76:0/643601116 shutdown_connections 2024-03-12T06:03:45.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.019+0000 7f9a519de740 1 --2- 172.21.15.76:0/643601116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e64c4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.019+0000 7f9a519de740 1 --2- 172.21.15.76:0/643601116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b4bd0 0x5563e64bd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.019+0000 7f9a519de740 1 --2- 172.21.15.76:0/643601116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b42b0 0x5563e64bcc90 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.019+0000 7f9a519de740 1 -- 172.21.15.76:0/643601116 >> 172.21.15.76:0/643601116 conn(0x5563e64ae990 msgr2=0x5563e64b8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.019+0000 7f9a519de740 1 -- 172.21.15.76:0/643601116 shutdown_connections 2024-03-12T06:03:45.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.019+0000 7f9a519de740 1 -- 172.21.15.76:0/643601116 wait complete. 2024-03-12T06:03:45.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.019+0000 7f9a519de740 1 -- start start 2024-03-12T06:03:45.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.019+0000 7f9a519de740 1 Processor -- start 2024-03-12T06:03:45.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a519de740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b42b0 0x5563e64c3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a519de740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b4bd0 0x5563e64c3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a519de740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e6525f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a519de740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e6526460 con 0x5563e64b42b0 2024-03-12T06:03:45.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a519de740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e65265a0 con 0x5563e64b4bd0 2024-03-12T06:03:45.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a519de740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e65266e0 con 0x5563e64bd7b0 2024-03-12T06:03:45.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a519de740 10 client.0 _ll_register_callbacks cb 0x7f9a3c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:45.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a4cbaa640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b42b0 0x5563e64c3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.032 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a4cbaa640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b42b0 0x5563e64c3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34600/0 (socket says 172.21.15.76:34600) 2024-03-12T06:03:45.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a4cbaa640 1 -- 172.21.15.76:0/759021640 learned_addr learned my addr 172.21.15.76:0/759021640 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a47fff640 1 --2- 172.21.15.76:0/759021640 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b4bd0 0x5563e64c3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a4d3ab640 1 --2- 172.21.15.76:0/759021640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e6525f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a4cbaa640 1 -- 172.21.15.76:0/759021640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 msgr2=0x5563e6525f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a4cbaa640 1 --2- 172.21.15.76:0/759021640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e6525f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a4cbaa640 1 -- 172.21.15.76:0/759021640 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b4bd0 msgr2=0x5563e64c3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a4cbaa640 1 --2- 172.21.15.76:0/759021640 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b4bd0 0x5563e64c3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a4cbaa640 1 -- 172.21.15.76:0/759021640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a481a3470 con 0x5563e64b42b0 2024-03-12T06:03:45.033 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a4d3ab640 1 --2- 172.21.15.76:0/759021640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e6525f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:45.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a4cbaa640 1 --2- 172.21.15.76:0/759021640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b42b0 0x5563e64c3720 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f9a38099f20 tx=0x7f9a38004910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4681a4bcb8ff8bf server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a27fff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:45.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a27fff640 1 -- 172.21.15.76:0/759021640 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a380b6070 con 0x5563e64b42b0 2024-03-12T06:03:45.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a27fff640 1 -- 172.21.15.76:0/759021640 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9a380b82d0 con 0x5563e64b42b0 2024-03-12T06:03:45.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a519de740 1 -- 172.21.15.76:0/759021640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a3c005430 con 0x5563e64b42b0 2024-03-12T06:03:45.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a27fff640 1 -- 172.21.15.76:0/759021640 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a380b2ba0 con 0x5563e64b42b0 2024-03-12T06:03:45.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a519de740 10 client.17250 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:45.034 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.020+0000 7f9a519de740 1 -- 172.21.15.76:0/759021640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5563e63dc6a0 con 0x5563e64b42b0 2024-03-12T06:03:45.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.021+0000 7f9a27fff640 1 -- 172.21.15.76:0/759021640 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a3c005430 con 0x5563e64b42b0 2024-03-12T06:03:45.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.021+0000 7f9a27fff640 1 -- 172.21.15.76:0/759021640 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9a380992c0 con 0x5563e64b42b0 2024-03-12T06:03:45.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.021+0000 7f9a519de740 10 client.17250 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:45.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.021+0000 7f9a519de740 10 client.17250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:45.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.021+0000 7f9a519de740 20 client.17250 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:45.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.021+0000 7f9a519de740 1 --2- 172.21.15.76:0/759021640 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5563e6527950 0x5563e6529d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.021+0000 7f9a519de740 4 client.17250 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:03:45.035 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.021+0000 7f9a519de740 1 -- 172.21.15.76:0/759021640 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5563e61a7890 con 0x5563e6527950 2024-03-12T06:03:45.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.021+0000 7f9a47fff640 1 --2- 172.21.15.76:0/759021640 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5563e6527950 0x5563e6529d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a27fff640 10 client.17250 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:45.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.021+0000 7f9a47fff640 1 --2- 172.21.15.76:0/759021640 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5563e6527950 0x5563e6529d70 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef4849a9289a8d6e server_cookie=40dd5a61853d4dd1 in_seq=0 out_seq=0 2024-03-12T06:03:45.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a27fff640 1 -- 172.21.15.76:0/759021640 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5563e61a7890 con 0x5563e6527950 2024-03-12T06:03:45.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a27fff640 10 client.17250 handle_command_reply: tid=0 2024-03-12T06:03:45.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a519de740 1 client.17250 shutdown 2024-03-12T06:03:45.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a519de740 1 -- 172.21.15.76:0/759021640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b42b0 msgr2=0x5563e64c3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.036 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a519de740 1 --2- 172.21.15.76:0/759021640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b42b0 0x5563e64c3720 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f9a38099f20 tx=0x7f9a38004910 comp rx=0 tx=0).stop 2024-03-12T06:03:45.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a519de740 1 -- 172.21.15.76:0/759021640 shutdown_connections 2024-03-12T06:03:45.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a519de740 1 --2- 172.21.15.76:0/759021640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e64bd7b0 0x5563e6525f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a519de740 1 --2- 172.21.15.76:0/759021640 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5563e64b4bd0 0x5563e64c3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a519de740 1 --2- 172.21.15.76:0/759021640 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5563e6527950 0x5563e6529d70 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a519de740 1 --2- 172.21.15.76:0/759021640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5563e64b42b0 0x5563e64c3720 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a27fff640 0 client.17250 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:45.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a519de740 1 -- 172.21.15.76:0/759021640 >> 172.21.15.76:0/759021640 conn(0x5563e64ae990 msgr2=0x5563e64ad1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a519de740 1 -- 172.21.15.76:0/759021640 shutdown_connections 2024-03-12T06:03:45.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a519de740 1 -- 172.21.15.76:0/759021640 wait complete. 2024-03-12T06:03:45.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.022+0000 7f9a519de740 20 client.17250 trim_cache size 0 max 16384 2024-03-12T06:03:45.038 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:03:45.038 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T06:03:45.038 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T06:03:45.038 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:03:45.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.023+0000 7f9a4ee0e640 1 -- 172.21.15.76:0/2227369208 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9a300a1ba0 msgr2=0x7f9a300a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.023+0000 7f9a4ee0e640 1 --2- 172.21.15.76:0/2227369208 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9a300a1ba0 0x7f9a300a4060 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c036040 tx=0x7f9a3c004a60 comp rx=0 tx=0).stop 2024-03-12T06:03:45.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.023+0000 7f9a4ee0e640 1 -- 172.21.15.76:0/2227369208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a48100b50 msgr2=0x7f9a48196720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.023+0000 7f9a4ee0e640 1 --2- 172.21.15.76:0/2227369208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a48100b50 0x7f9a48196720 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f9a3800d770 tx=0x7f9a3800dc30 comp rx=0 tx=0).stop 2024-03-12T06:03:45.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.023+0000 7f9a4ee0e640 1 -- 172.21.15.76:0/2227369208 shutdown_connections 2024-03-12T06:03:45.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.023+0000 7f9a4ee0e640 1 --2- 172.21.15.76:0/2227369208 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9a300a1ba0 0x7f9a300a4060 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.023+0000 7f9a4ee0e640 1 --2- 172.21.15.76:0/2227369208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a48101fc0 0x7f9a4819afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.023+0000 7f9a4ee0e640 1 --2- 172.21.15.76:0/2227369208 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9a481014f0 0x7f9a48196c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.023+0000 7f9a4ee0e640 1 --2- 172.21.15.76:0/2227369208 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a48100b50 0x7f9a48196720 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.023+0000 7f9a4ee0e640 1 -- 172.21.15.76:0/2227369208 >> 172.21.15.76:0/2227369208 conn(0x7f9a480753a0 msgr2=0x7f9a480fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.023+0000 7f9a4ee0e640 1 -- 172.21.15.76:0/2227369208 shutdown_connections 2024-03-12T06:03:45.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.023+0000 7f9a4ee0e640 1 -- 172.21.15.76:0/2227369208 wait complete. 2024-03-12T06:03:45.041 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T06:03:45.041 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T06:03:45.146 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.138+0000 7f02af817640 1 -- 172.21.15.76:0/3797502764 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02a80ff9f0 msgr2=0x7f02a80ffe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.138+0000 7f02af817640 1 --2- 172.21.15.76:0/3797502764 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02a80ff9f0 0x7f02a80ffe50 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009900 tx=0x7f02a402bc30 comp rx=0 tx=0).stop 2024-03-12T06:03:45.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.138+0000 7f02af817640 1 -- 172.21.15.76:0/3797502764 shutdown_connections 2024-03-12T06:03:45.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.138+0000 7f02af817640 1 --2- 172.21.15.76:0/3797502764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02a8100390 0x7f02a810c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.138+0000 7f02af817640 1 --2- 172.21.15.76:0/3797502764 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02a80ff9f0 0x7f02a80ffe50 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.138+0000 7f02af817640 1 --2- 172.21.15.76:0/3797502764 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f02a81058d0 0x7f02a8105cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.138+0000 7f02af817640 1 -- 172.21.15.76:0/3797502764 >> 172.21.15.76:0/3797502764 conn(0x7f02a8076540 msgr2=0x7f02a8076950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.138+0000 7f02af817640 1 -- 172.21.15.76:0/3797502764 shutdown_connections 2024-03-12T06:03:45.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.139+0000 7f02af817640 1 -- 172.21.15.76:0/3797502764 wait complete. 2024-03-12T06:03:45.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.139+0000 7f02af817640 1 Processor -- start 2024-03-12T06:03:45.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.139+0000 7f02af817640 1 -- start start 2024-03-12T06:03:45.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.139+0000 7f02af817640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f02a80ff9f0 0x7f02a8198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.139+0000 7f02af817640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02a8100390 0x7f02a8198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.139+0000 7f02af817640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02a81058d0 0x7f02a81a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.139+0000 7f02af817640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a8199440 con 0x7f02a81058d0 2024-03-12T06:03:45.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.139+0000 7f02af817640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a81a3c20 con 0x7f02a80ff9f0 2024-03-12T06:03:45.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.139+0000 7f02af817640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a81a3d60 con 0x7f02a8100390 2024-03-12T06:03:45.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.139+0000 7f02addb4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02a81058d0 0x7f02a81a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.139+0000 7f02addb4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02a81058d0 0x7f02a81a36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34626/0 (socket says 172.21.15.76:34626) 2024-03-12T06:03:45.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.139+0000 7f02addb4640 1 -- 172.21.15.76:0/2565532998 learned_addr learned my addr 172.21.15.76:0/2565532998 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.140+0000 7f02acdb2640 1 --2- 172.21.15.76:0/2565532998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02a8100390 0x7f02a8198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.140+0000 7f02ad5b3640 1 --2- 172.21.15.76:0/2565532998 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f02a80ff9f0 0x7f02a8198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.140+0000 7f02addb4640 1 -- 172.21.15.76:0/2565532998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02a8100390 msgr2=0x7f02a8198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.140+0000 7f02addb4640 1 --2- 172.21.15.76:0/2565532998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02a8100390 0x7f02a8198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.140+0000 7f02addb4640 1 -- 172.21.15.76:0/2565532998 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f02a80ff9f0 msgr2=0x7f02a8198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.140+0000 7f02addb4640 1 --2- 172.21.15.76:0/2565532998 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f02a80ff9f0 0x7f02a8198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.140+0000 7f02addb4640 1 -- 172.21.15.76:0/2565532998 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a4009560 con 0x7f02a81058d0 2024-03-12T06:03:45.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.140+0000 7f02addb4640 1 --2- 172.21.15.76:0/2565532998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02a81058d0 0x7f02a81a36e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f02a000ead0 tx=0x7f02a000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.140+0000 7f029e7fc640 1 -- 172.21.15.76:0/2565532998 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a000cde0 con 0x7f02a81058d0 2024-03-12T06:03:45.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.140+0000 7f029e7fc640 1 -- 172.21.15.76:0/2565532998 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02a0005020 con 0x7f02a81058d0 2024-03-12T06:03:45.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.140+0000 7f02af817640 1 -- 172.21.15.76:0/2565532998 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a81a4010 con 0x7f02a81058d0 2024-03-12T06:03:45.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.140+0000 7f029e7fc640 1 -- 172.21.15.76:0/2565532998 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a00116b0 con 0x7f02a81058d0 2024-03-12T06:03:45.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.140+0000 7f02af817640 1 -- 172.21.15.76:0/2565532998 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a81a44f0 con 0x7f02a81058d0 2024-03-12T06:03:45.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.141+0000 7f02af817640 1 -- 172.21.15.76:0/2565532998 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a8068450 con 0x7f02a81058d0 2024-03-12T06:03:45.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.142+0000 7f029e7fc640 1 -- 172.21.15.76:0/2565532998 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f02a0005520 con 0x7f02a81058d0 2024-03-12T06:03:45.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.143+0000 7f029e7fc640 1 --2- 172.21.15.76:0/2565532998 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f02940a1c20 0x7f02940a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.143+0000 7f02ad5b3640 1 --2- 172.21.15.76:0/2565532998 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f02940a1c20 0x7f02940a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.143+0000 7f029e7fc640 1 -- 172.21.15.76:0/2565532998 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f02a00cd970 con 0x7f02a81058d0 2024-03-12T06:03:45.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.143+0000 7f02ad5b3640 1 --2- 172.21.15.76:0/2565532998 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f02940a1c20 0x7f02940a40e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0298005e00 tx=0x7f0298005c50 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.147+0000 7f029e7fc640 1 -- 172.21.15.76:0/2565532998 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f02a0096840 con 0x7f02a81058d0 2024-03-12T06:03:45.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.360+0000 7f02af817640 1 -- 172.21.15.76:0/2565532998 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02a80fe9f0 con 0x7f02a81058d0 2024-03-12T06:03:45.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.361+0000 7f029e7fc640 1 -- 172.21.15.76:0/2565532998 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11328 (secure 0 0 0) 0x7f02a0099f00 con 0x7f02a81058d0 2024-03-12T06:03:45.366 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:03:45.366 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T06:01:33.842001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T06:03:45.366 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 28 2024-03-12T06:03:45.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.363+0000 7f02af817640 1 -- 172.21.15.76:0/2565532998 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f02940a1c20 msgr2=0x7f02940a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.363+0000 7f02af817640 1 --2- 172.21.15.76:0/2565532998 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f02940a1c20 0x7f02940a40e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0298005e00 tx=0x7f0298005c50 comp rx=0 tx=0).stop 2024-03-12T06:03:45.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.363+0000 7f02af817640 1 -- 172.21.15.76:0/2565532998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02a81058d0 msgr2=0x7f02a81a36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.363+0000 7f02af817640 1 --2- 172.21.15.76:0/2565532998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02a81058d0 0x7f02a81a36e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f02a000ead0 tx=0x7f02a000ef90 comp rx=0 tx=0).stop 2024-03-12T06:03:45.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.363+0000 7f02af817640 1 -- 172.21.15.76:0/2565532998 shutdown_connections 2024-03-12T06:03:45.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.363+0000 7f02af817640 1 --2- 172.21.15.76:0/2565532998 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f02940a1c20 0x7f02940a40e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.363+0000 7f02af817640 1 --2- 172.21.15.76:0/2565532998 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02a81058d0 0x7f02a81a36e0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.363+0000 7f02af817640 1 --2- 172.21.15.76:0/2565532998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02a8100390 0x7f02a8198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.363+0000 7f02af817640 1 --2- 172.21.15.76:0/2565532998 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f02a80ff9f0 0x7f02a8198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.363+0000 7f02af817640 1 -- 172.21.15.76:0/2565532998 >> 172.21.15.76:0/2565532998 conn(0x7f02a8076540 msgr2=0x7f02a81013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.363+0000 7f02af817640 1 -- 172.21.15.76:0/2565532998 shutdown_connections 2024-03-12T06:03:45.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.363+0000 7f02af817640 1 -- 172.21.15.76:0/2565532998 wait complete. 2024-03-12T06:03:45.378 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:03:45.379 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T06:03:45.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.478+0000 7effdfad4640 1 -- 172.21.15.76:0/2406220707 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effd8101d30 msgr2=0x7effd8105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.478+0000 7effdfad4640 1 --2- 172.21.15.76:0/2406220707 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effd8101d30 0x7effd8105d70 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7effc8009930 tx=0x7effc802bc30 comp rx=0 tx=0).stop 2024-03-12T06:03:45.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.478+0000 7effdfad4640 1 -- 172.21.15.76:0/2406220707 shutdown_connections 2024-03-12T06:03:45.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.478+0000 7effdfad4640 1 --2- 172.21.15.76:0/2406220707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effd8106450 0x7effd810e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.478+0000 7effdfad4640 1 --2- 172.21.15.76:0/2406220707 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effd8101d30 0x7effd8105d70 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.478+0000 7effdfad4640 1 --2- 172.21.15.76:0/2406220707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7effd8101380 0x7effd8101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.478+0000 7effdfad4640 1 -- 172.21.15.76:0/2406220707 >> 172.21.15.76:0/2406220707 conn(0x7effd80fcc20 msgr2=0x7effd80ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.478+0000 7effdfad4640 1 -- 172.21.15.76:0/2406220707 shutdown_connections 2024-03-12T06:03:45.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.478+0000 7effdfad4640 1 -- 172.21.15.76:0/2406220707 wait complete. 2024-03-12T06:03:45.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.479+0000 7effdfad4640 1 Processor -- start 2024-03-12T06:03:45.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.479+0000 7effdfad4640 1 -- start start 2024-03-12T06:03:45.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.479+0000 7effdfad4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7effd8101380 0x7effd819d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.479+0000 7effdfad4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effd8106450 0x7effd819d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.479+0000 7effdfad4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effd81975d0 0x7effd8197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.479+0000 7effdfad4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd8197fc0 con 0x7effd81975d0 2024-03-12T06:03:45.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.479+0000 7effdfad4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd8198130 con 0x7effd8101380 2024-03-12T06:03:45.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.479+0000 7effdfad4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd81982f0 con 0x7effd8106450 2024-03-12T06:03:45.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.479+0000 7effde071640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effd81975d0 0x7effd8197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.479+0000 7effde071640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effd81975d0 0x7effd8197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34644/0 (socket says 172.21.15.76:34644) 2024-03-12T06:03:45.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.479+0000 7effde071640 1 -- 172.21.15.76:0/62368127 learned_addr learned my addr 172.21.15.76:0/62368127 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.479+0000 7effdd870640 1 --2- 172.21.15.76:0/62368127 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7effd8101380 0x7effd819d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.479+0000 7effdd06f640 1 --2- 172.21.15.76:0/62368127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effd8106450 0x7effd819d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.480+0000 7effde071640 1 -- 172.21.15.76:0/62368127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effd8106450 msgr2=0x7effd819d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.480+0000 7effde071640 1 --2- 172.21.15.76:0/62368127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effd8106450 0x7effd819d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.480+0000 7effde071640 1 -- 172.21.15.76:0/62368127 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7effd8101380 msgr2=0x7effd819d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.480+0000 7effde071640 1 --2- 172.21.15.76:0/62368127 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7effd8101380 0x7effd819d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.480+0000 7effde071640 1 -- 172.21.15.76:0/62368127 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc8009560 con 0x7effd81975d0 2024-03-12T06:03:45.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.480+0000 7effde071640 1 --2- 172.21.15.76:0/62368127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effd81975d0 0x7effd8197a80 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7effd000cce0 tx=0x7effd000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.480+0000 7effceffd640 1 -- 172.21.15.76:0/62368127 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd000f450 con 0x7effd81975d0 2024-03-12T06:03:45.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.480+0000 7effceffd640 1 -- 172.21.15.76:0/62368127 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effd000fa90 con 0x7effd81975d0 2024-03-12T06:03:45.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.480+0000 7effdfad4640 1 -- 172.21.15.76:0/62368127 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd81a42e0 con 0x7effd81975d0 2024-03-12T06:03:45.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.480+0000 7effceffd640 1 -- 172.21.15.76:0/62368127 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0018b90 con 0x7effd81975d0 2024-03-12T06:03:45.489 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.480+0000 7effdfad4640 1 -- 172.21.15.76:0/62368127 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd81a47f0 con 0x7effd81975d0 2024-03-12T06:03:45.490 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:45 smithi088 ceph-mon[36585]: pgmap v246: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:45.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.483+0000 7effceffd640 1 -- 172.21.15.76:0/62368127 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7effd000fc00 con 0x7effd81975d0 2024-03-12T06:03:45.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.483+0000 7effe277f740 1 Processor -- start 2024-03-12T06:03:45.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.483+0000 7effe277f740 1 -- start start 2024-03-12T06:03:45.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.483+0000 7effe277f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602250 0x558ede602630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.483+0000 7effe277f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede602b70 0x558ede60b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.492 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.483+0000 7effe277f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede60b750 0x558ede60db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.483+0000 7effe277f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ede5bd960 con 0x558ede60b750 2024-03-12T06:03:45.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.483+0000 7effe277f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ede571310 con 0x558ede602b70 2024-03-12T06:03:45.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.483+0000 7effe277f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ede56e840 con 0x558ede602250 2024-03-12T06:03:45.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effdd06f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede60b750 0x558ede60db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effdd06f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede60b750 0x558ede60db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34656/0 (socket says 172.21.15.76:34656) 2024-03-12T06:03:45.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effdd06f640 1 -- 172.21.15.76:0/1427981841 learned_addr learned my addr 172.21.15.76:0/1427981841 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effceffd640 1 --2- 172.21.15.76:0/62368127 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7effc40a1c20 0x7effc40a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effdd06f640 1 --2- 172.21.15.76:0/1427981841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602250 0x558ede602630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.493 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effdd870640 1 --2- 172.21.15.76:0/1427981841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede602b70 0x558ede60b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effde071640 1 --2- 172.21.15.76:0/62368127 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7effc40a1c20 0x7effc40a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effdd06f640 1 -- 172.21.15.76:0/1427981841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602250 msgr2=0x558ede602630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effdd06f640 1 --2- 172.21.15.76:0/1427981841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602250 0x558ede602630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effdd06f640 1 -- 172.21.15.76:0/1427981841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede602b70 msgr2=0x558ede60b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effdd06f640 1 --2- 172.21.15.76:0/1427981841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede602b70 0x558ede60b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effdd06f640 1 -- 172.21.15.76:0/1427981841 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ede52c060 con 0x558ede60b750 2024-03-12T06:03:45.494 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effceffd640 1 -- 172.21.15.76:0/62368127 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7effd00cdee0 con 0x7effd81975d0 2024-03-12T06:03:45.495 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:45 smithi076 ceph-mon[32095]: pgmap v246: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:45.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effdd06f640 1 --2- 172.21.15.76:0/1427981841 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede60b750 0x558ede60db40 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7effd806a8e0 tx=0x7effc8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1098e15b0a63c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effde071640 1 --2- 172.21.15.76:0/62368127 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7effc40a1c20 0x7effc40a40e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7effd00cb6e0 tx=0x7effd006abb0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effc3fff640 1 -- 172.21.15.76:0/1427981841 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc802ca90 con 0x558ede60b750 2024-03-12T06:03:45.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effc3fff640 1 -- 172.21.15.76:0/1427981841 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effc8004510 con 0x558ede60b750 2024-03-12T06:03:45.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.484+0000 7effc3fff640 1 -- 172.21.15.76:0/1427981841 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8004df0 con 0x558ede60b750 2024-03-12T06:03:45.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.485+0000 7effe277f740 1 -- 172.21.15.76:0/1427981841 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede60b750 msgr2=0x558ede60db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.485+0000 7effe277f740 1 --2- 172.21.15.76:0/1427981841 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede60b750 0x558ede60db40 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7effd806a8e0 tx=0x7effc8000bc0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.485+0000 7effe277f740 1 -- 172.21.15.76:0/1427981841 shutdown_connections 2024-03-12T06:03:45.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.485+0000 7effe277f740 1 --2- 172.21.15.76:0/1427981841 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede60b750 0x558ede60db40 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.485+0000 7effe277f740 1 --2- 172.21.15.76:0/1427981841 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede602b70 0x558ede60b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.485+0000 7effe277f740 1 --2- 172.21.15.76:0/1427981841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602250 0x558ede602630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.485+0000 7effe277f740 1 -- 172.21.15.76:0/1427981841 >> 172.21.15.76:0/1427981841 conn(0x558ede5fc980 msgr2=0x558ede5fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.485+0000 7effe277f740 1 -- 172.21.15.76:0/1427981841 shutdown_connections 2024-03-12T06:03:45.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.485+0000 7effe277f740 1 -- 172.21.15.76:0/1427981841 wait complete. 2024-03-12T06:03:45.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effe277f740 1 -- start start 2024-03-12T06:03:45.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effe277f740 1 Processor -- start 2024-03-12T06:03:45.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effe277f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede602250 0x558ede608810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effe277f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede602b70 0x558ede608d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effe277f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede60b750 0x558ede609290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effe277f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ede56a020 con 0x558ede602250 2024-03-12T06:03:45.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effe277f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ede23d430 con 0x558ede602b70 2024-03-12T06:03:45.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effe277f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ede1f9e00 con 0x558ede60b750 2024-03-12T06:03:45.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effe277f740 10 client.0 _ll_register_callbacks cb 0x558ede511cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:45.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effdd06f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede602250 0x558ede608810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effdd06f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede602250 0x558ede608810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34668/0 (socket says 172.21.15.76:34668) 2024-03-12T06:03:45.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effdd06f640 1 -- 172.21.15.76:0/247970295 learned_addr learned my addr 172.21.15.76:0/247970295 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effdd06f640 1 -- 172.21.15.76:0/247970295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede60b750 msgr2=0x558ede609290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effdd06f640 1 --2- 172.21.15.76:0/247970295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede60b750 0x558ede609290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effdd06f640 1 -- 172.21.15.76:0/247970295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede602b70 msgr2=0x558ede608d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effdd06f640 1 --2- 172.21.15.76:0/247970295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede602b70 0x558ede608d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.486+0000 7effdd06f640 1 -- 172.21.15.76:0/247970295 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd810e830 con 0x558ede602250 2024-03-12T06:03:45.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effdd06f640 1 --2- 172.21.15.76:0/247970295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede602250 0x558ede608810 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7effc8047040 tx=0x7effc8009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcef04fceca733e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effc17fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:45.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effc17fa640 1 -- 172.21.15.76:0/247970295 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8003c90 con 0x558ede602250 2024-03-12T06:03:45.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effc17fa640 1 -- 172.21.15.76:0/247970295 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7effc8000e40 con 0x558ede602250 2024-03-12T06:03:45.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effe277f740 1 -- 172.21.15.76:0/247970295 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558ede2697e0 con 0x558ede602250 2024-03-12T06:03:45.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effc17fa640 1 -- 172.21.15.76:0/247970295 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7effc8003df0 con 0x558ede602250 2024-03-12T06:03:45.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effe277f740 10 client.17268 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:45.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effe277f740 1 -- 172.21.15.76:0/247970295 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558ede5686a0 con 0x558ede602250 2024-03-12T06:03:45.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effc17fa640 1 -- 172.21.15.76:0/247970295 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558ede2697e0 con 0x558ede602250 2024-03-12T06:03:45.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effc17fa640 1 -- 172.21.15.76:0/247970295 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7effc80053d0 con 0x558ede602250 2024-03-12T06:03:45.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effe277f740 10 client.17268 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:45.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effe277f740 10 client.17268 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:03:45.504 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effe277f740 20 client.17268 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:45.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effe277f740 1 --2- 172.21.15.76:0/247970295 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x558ede61b7e0 0x558ede61dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effe277f740 4 client.17268 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:03:45.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.487+0000 7effe277f740 1 -- 172.21.15.76:0/247970295 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558ede2a6720 con 0x558ede61b7e0 2024-03-12T06:03:45.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.488+0000 7effdd870640 1 --2- 172.21.15.76:0/247970295 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x558ede61b7e0 0x558ede61dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.488+0000 7effc17fa640 10 client.17268 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:03:45.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.488+0000 7effdd870640 1 --2- 172.21.15.76:0/247970295 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x558ede61b7e0 0x558ede61dc00 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3fdff4a210eef23 server_cookie=bbaa0f5f45091dcb in_seq=0 out_seq=0 2024-03-12T06:03:45.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effc17fa640 1 -- 172.21.15.76:0/247970295 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558ede2a6720 con 0x558ede61b7e0 2024-03-12T06:03:45.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effc17fa640 10 client.17268 handle_command_reply: tid=0 2024-03-12T06:03:45.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effe277f740 1 client.17268 shutdown 2024-03-12T06:03:45.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effe277f740 1 -- 172.21.15.76:0/247970295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede602250 msgr2=0x558ede608810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effe277f740 1 --2- 172.21.15.76:0/247970295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede602250 0x558ede608810 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7effc8047040 tx=0x7effc8009230 comp rx=0 tx=0).stop 2024-03-12T06:03:45.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effe277f740 1 -- 172.21.15.76:0/247970295 shutdown_connections 2024-03-12T06:03:45.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effe277f740 1 --2- 172.21.15.76:0/247970295 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x558ede61b7e0 0x558ede61dc00 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effe277f740 1 --2- 172.21.15.76:0/247970295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede60b750 0x558ede609290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effe277f740 1 --2- 172.21.15.76:0/247970295 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede602b70 0x558ede608d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effc17fa640 0 client.17268 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:03:45.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effe277f740 1 --2- 172.21.15.76:0/247970295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede602250 0x558ede608810 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effe277f740 1 -- 172.21.15.76:0/247970295 >> 172.21.15.76:0/247970295 conn(0x558ede5fc980 msgr2=0x558ede5fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effe277f740 1 -- 172.21.15.76:0/247970295 shutdown_connections 2024-03-12T06:03:45.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effe277f740 1 -- 172.21.15.76:0/247970295 wait complete. 2024-03-12T06:03:45.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.489+0000 7effe277f740 20 client.17268 trim_cache size 0 max 16384 2024-03-12T06:03:45.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effe277f740 1 Processor -- start 2024-03-12T06:03:45.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effe277f740 1 -- start start 2024-03-12T06:03:45.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effe277f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602250 0x558ede60af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effe277f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede602b70 0x558ede603a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effe277f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede60b750 0x558ede603fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effe277f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ede4ef130 con 0x558ede602b70 2024-03-12T06:03:45.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effe277f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ede5fd780 con 0x558ede60b750 2024-03-12T06:03:45.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effe277f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ede601eb0 con 0x558ede602250 2024-03-12T06:03:45.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effdd870640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede602b70 0x558ede603a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effdd870640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede602b70 0x558ede603a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34676/0 (socket says 172.21.15.76:34676) 2024-03-12T06:03:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effdd870640 1 -- 172.21.15.76:0/2393672265 learned_addr learned my addr 172.21.15.76:0/2393672265 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effdd06f640 1 --2- 172.21.15.76:0/2393672265 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602250 0x558ede60af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effdd06f640 1 --2- 172.21.15.76:0/2393672265 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede60b750 0x558ede603fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effdd870640 1 -- 172.21.15.76:0/2393672265 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602250 msgr2=0x558ede60af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effdd870640 1 --2- 172.21.15.76:0/2393672265 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602250 0x558ede60af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effdd870640 1 -- 172.21.15.76:0/2393672265 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede60b750 msgr2=0x558ede603fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effdd870640 1 --2- 172.21.15.76:0/2393672265 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede60b750 0x558ede603fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effdd870640 1 -- 172.21.15.76:0/2393672265 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ede504c20 con 0x558ede602b70 2024-03-12T06:03:45.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effdd06f640 1 --2- 172.21.15.76:0/2393672265 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede60b750 0x558ede603fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:45.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.501+0000 7effdd870640 1 --2- 172.21.15.76:0/2393672265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede602b70 0x558ede603a90 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7effd4005d00 tx=0x7effd4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73a9c6c94ff54d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.502+0000 7effc37fe640 1 -- 172.21.15.76:0/2393672265 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4016040 con 0x558ede602b70 2024-03-12T06:03:45.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.502+0000 7effc37fe640 1 -- 172.21.15.76:0/2393672265 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effd401f070 con 0x558ede602b70 2024-03-12T06:03:45.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.502+0000 7effc37fe640 1 -- 172.21.15.76:0/2393672265 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4004b20 con 0x558ede602b70 2024-03-12T06:03:45.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.502+0000 7effe277f740 1 -- 172.21.15.76:0/2393672265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede602b70 msgr2=0x558ede603a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.502+0000 7effe277f740 1 --2- 172.21.15.76:0/2393672265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede602b70 0x558ede603a90 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7effd4005d00 tx=0x7effd4011810 comp rx=0 tx=0).stop 2024-03-12T06:03:45.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.502+0000 7effe277f740 1 -- 172.21.15.76:0/2393672265 shutdown_connections 2024-03-12T06:03:45.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.502+0000 7effe277f740 1 --2- 172.21.15.76:0/2393672265 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede60b750 0x558ede603fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.502+0000 7effe277f740 1 --2- 172.21.15.76:0/2393672265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede602b70 0x558ede603a90 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.502+0000 7effe277f740 1 --2- 172.21.15.76:0/2393672265 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602250 0x558ede60af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.502+0000 7effe277f740 1 -- 172.21.15.76:0/2393672265 >> 172.21.15.76:0/2393672265 conn(0x558ede5fc980 msgr2=0x558ede5f7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.502+0000 7effe277f740 1 -- 172.21.15.76:0/2393672265 shutdown_connections 2024-03-12T06:03:45.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.502+0000 7effe277f740 1 -- 172.21.15.76:0/2393672265 wait complete. 2024-03-12T06:03:45.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.502+0000 7effe277f740 1 -- start start 2024-03-12T06:03:45.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effe277f740 1 Processor -- start 2024-03-12T06:03:45.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effe277f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede602250 0x558ede6112c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effe277f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602b70 0x558ede611800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effe277f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede60b750 0x558ede671d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effe277f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ede611f60 con 0x558ede60b750 2024-03-12T06:03:45.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effe277f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ede6120d0 con 0x558ede602250 2024-03-12T06:03:45.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effe277f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ede672250 con 0x558ede602b70 2024-03-12T06:03:45.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effe277f740 10 client.0 _ll_register_callbacks cb 0x7effc8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:45.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effdd06f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede60b750 0x558ede671d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effdd06f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede60b750 0x558ede671d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34690/0 (socket says 172.21.15.76:34690) 2024-03-12T06:03:45.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effdd06f640 1 -- 172.21.15.76:0/3678599586 learned_addr learned my addr 172.21.15.76:0/3678599586 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effdd06f640 1 --2- 172.21.15.76:0/3678599586 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede602250 0x558ede6112c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effdd870640 1 --2- 172.21.15.76:0/3678599586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602b70 0x558ede611800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effdd06f640 1 -- 172.21.15.76:0/3678599586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602b70 msgr2=0x558ede611800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effdd06f640 1 --2- 172.21.15.76:0/3678599586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602b70 0x558ede611800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effdd06f640 1 -- 172.21.15.76:0/3678599586 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede602250 msgr2=0x558ede6112c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effdd06f640 1 --2- 172.21.15.76:0/3678599586 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede602250 0x558ede6112c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.503+0000 7effdd06f640 1 -- 172.21.15.76:0/3678599586 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd400be10 con 0x558ede60b750 2024-03-12T06:03:45.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effdd06f640 1 --2- 172.21.15.76:0/3678599586 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede60b750 0x558ede671d10 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7effc8047040 tx=0x7effc804f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c5c4ecc8f3a04d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effc17fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:45.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effc17fa640 1 -- 172.21.15.76:0/3678599586 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8066070 con 0x558ede60b750 2024-03-12T06:03:45.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effc17fa640 1 -- 172.21.15.76:0/3678599586 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7effc8000e40 con 0x558ede60b750 2024-03-12T06:03:45.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effe277f740 1 -- 172.21.15.76:0/3678599586 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7effc8004510 con 0x558ede60b750 2024-03-12T06:03:45.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effc17fa640 1 -- 172.21.15.76:0/3678599586 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7effc802c5d0 con 0x558ede60b750 2024-03-12T06:03:45.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effe277f740 10 client.17274 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:45.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effe277f740 1 -- 172.21.15.76:0/3678599586 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558ede52a8c0 con 0x558ede60b750 2024-03-12T06:03:45.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effc17fa640 1 -- 172.21.15.76:0/3678599586 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effc8004510 con 0x558ede60b750 2024-03-12T06:03:45.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effc17fa640 1 -- 172.21.15.76:0/3678599586 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7effc8069020 con 0x558ede60b750 2024-03-12T06:03:45.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effe277f740 10 client.17274 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:45.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effe277f740 10 client.17274 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:03:45.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effe277f740 20 client.17274 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:45.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effe277f740 1 --2- 172.21.15.76:0/3678599586 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x558ede673590 0x558ede6759b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effe277f740 4 client.17274 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:03:45.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.504+0000 7effe277f740 1 -- 172.21.15.76:0/3678599586 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558ede2f5890 con 0x558ede673590 2024-03-12T06:03:45.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effdd870640 1 --2- 172.21.15.76:0/3678599586 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x558ede673590 0x558ede6759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effc17fa640 10 client.17274 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:03:45.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effdd870640 1 --2- 172.21.15.76:0/3678599586 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x558ede673590 0x558ede6759b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac63b310d977bde server_cookie=b36b3b6a3c697184 in_seq=0 out_seq=0 2024-03-12T06:03:45.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effc17fa640 1 -- 172.21.15.76:0/3678599586 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558ede2f5890 con 0x558ede673590 2024-03-12T06:03:45.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effc17fa640 10 client.17274 handle_command_reply: tid=0 2024-03-12T06:03:45.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effe277f740 1 client.17274 shutdown 2024-03-12T06:03:45.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effe277f740 1 -- 172.21.15.76:0/3678599586 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede60b750 msgr2=0x558ede671d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effe277f740 1 --2- 172.21.15.76:0/3678599586 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede60b750 0x558ede671d10 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7effc8047040 tx=0x7effc804f040 comp rx=0 tx=0).stop 2024-03-12T06:03:45.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effe277f740 1 -- 172.21.15.76:0/3678599586 shutdown_connections 2024-03-12T06:03:45.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effe277f740 1 --2- 172.21.15.76:0/3678599586 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x558ede673590 0x558ede6759b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effe277f740 1 --2- 172.21.15.76:0/3678599586 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558ede60b750 0x558ede671d10 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effe277f740 1 --2- 172.21.15.76:0/3678599586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ede602b70 0x558ede611800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effe277f740 1 --2- 172.21.15.76:0/3678599586 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558ede602250 0x558ede6112c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effc17fa640 0 client.17274 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:03:45.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effe277f740 1 -- 172.21.15.76:0/3678599586 >> 172.21.15.76:0/3678599586 conn(0x558ede5fc980 msgr2=0x558ede603a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effe277f740 1 -- 172.21.15.76:0/3678599586 shutdown_connections 2024-03-12T06:03:45.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effe277f740 1 -- 172.21.15.76:0/3678599586 wait complete. 2024-03-12T06:03:45.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.505+0000 7effe277f740 20 client.17274 trim_cache size 0 max 16384 2024-03-12T06:03:45.524 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:03:45.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.506+0000 7effdfad4640 1 -- 172.21.15.76:0/62368127 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7effc40a1c20 msgr2=0x7effc40a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.506+0000 7effdfad4640 1 --2- 172.21.15.76:0/62368127 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7effc40a1c20 0x7effc40a40e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7effd00cb6e0 tx=0x7effd006abb0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.506+0000 7effdfad4640 1 -- 172.21.15.76:0/62368127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effd81975d0 msgr2=0x7effd8197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.506+0000 7effdfad4640 1 --2- 172.21.15.76:0/62368127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effd81975d0 0x7effd8197a80 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7effd000cce0 tx=0x7effd000a620 comp rx=0 tx=0).stop 2024-03-12T06:03:45.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.506+0000 7effdfad4640 1 -- 172.21.15.76:0/62368127 shutdown_connections 2024-03-12T06:03:45.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.506+0000 7effdfad4640 1 --2- 172.21.15.76:0/62368127 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7effc40a1c20 0x7effc40a40e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.506+0000 7effdfad4640 1 --2- 172.21.15.76:0/62368127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effd81975d0 0x7effd8197a80 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.506+0000 7effdfad4640 1 --2- 172.21.15.76:0/62368127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effd8106450 0x7effd819d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.506+0000 7effdfad4640 1 --2- 172.21.15.76:0/62368127 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7effd8101380 0x7effd819d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.506+0000 7effdfad4640 1 -- 172.21.15.76:0/62368127 >> 172.21.15.76:0/62368127 conn(0x7effd80fcc20 msgr2=0x7effd80fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.506+0000 7effdfad4640 1 -- 172.21.15.76:0/62368127 shutdown_connections 2024-03-12T06:03:45.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.506+0000 7effdfad4640 1 -- 172.21.15.76:0/62368127 wait complete. 2024-03-12T06:03:45.527 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:03:45.527 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T06:03:45.633 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.626+0000 7f53928a3640 1 -- 172.21.15.76:0/1391662053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f538c105360 msgr2=0x7f538c10cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.626+0000 7f53928a3640 1 --2- 172.21.15.76:0/1391662053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f538c105360 0x7f538c10cf30 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f5384009cd0 tx=0x7f538402c0f0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.626+0000 7f53928a3640 1 -- 172.21.15.76:0/1391662053 shutdown_connections 2024-03-12T06:03:45.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.626+0000 7f53928a3640 1 --2- 172.21.15.76:0/1391662053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f538c105360 0x7f538c10cf30 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.626+0000 7f53928a3640 1 --2- 172.21.15.76:0/1391662053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f538c100c40 0x7f538c104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.626+0000 7f53928a3640 1 --2- 172.21.15.76:0/1391662053 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f538c100290 0x7f538c100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.626+0000 7f53928a3640 1 -- 172.21.15.76:0/1391662053 >> 172.21.15.76:0/1391662053 conn(0x7f538c077230 msgr2=0x7f538c077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.626+0000 7f53928a3640 1 -- 172.21.15.76:0/1391662053 shutdown_connections 2024-03-12T06:03:45.634 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.626+0000 7f53928a3640 1 -- 172.21.15.76:0/1391662053 wait complete. 2024-03-12T06:03:45.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.627+0000 7f53928a3640 1 Processor -- start 2024-03-12T06:03:45.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.627+0000 7f53928a3640 1 -- start start 2024-03-12T06:03:45.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.627+0000 7f53928a3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f538c100290 0x7f538c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.627+0000 7f53928a3640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f538c105360 0x7f538c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.627+0000 7f53928a3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f538c199160 0x7f538c1a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.627+0000 7f53928a3640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c1a3db0 con 0x7f538c199160 2024-03-12T06:03:45.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.627+0000 7f53928a3640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c1a3f20 con 0x7f538c105360 2024-03-12T06:03:45.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.627+0000 7f53928a3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c1a40e0 con 0x7f538c100290 2024-03-12T06:03:45.635 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.627+0000 7f5390e40640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f538c199160 0x7f538c1a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f5390e40640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f538c199160 0x7f538c1a3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34706/0 (socket says 172.21.15.76:34706) 2024-03-12T06:03:45.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f5390e40640 1 -- 172.21.15.76:0/784243636 learned_addr learned my addr 172.21.15.76:0/784243636 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f538b7fe640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f538c105360 0x7f538c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f538bfff640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f538c100290 0x7f538c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f5390e40640 1 -- 172.21.15.76:0/784243636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f538c100290 msgr2=0x7f538c198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f5390e40640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f538c100290 0x7f538c198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f5390e40640 1 -- 172.21.15.76:0/784243636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f538c105360 msgr2=0x7f538c198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f5390e40640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f538c105360 0x7f538c198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f5390e40640 1 -- 172.21.15.76:0/784243636 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5384009900 con 0x7f538c199160 2024-03-12T06:03:45.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f538b7fe640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f538c105360 0x7f538c198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:45.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f538bfff640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f538c100290 0x7f538c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:45.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f5390e40640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f538c199160 0x7f538c1a3710 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f5384004af0 tx=0x7f5384034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f53897fa640 1 -- 172.21.15.76:0/784243636 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53840328c0 con 0x7f538c199160 2024-03-12T06:03:45.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f53897fa640 1 -- 172.21.15.76:0/784243636 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5384003e50 con 0x7f538c199160 2024-03-12T06:03:45.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f53928a3640 1 -- 172.21.15.76:0/784243636 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538c1a4360 con 0x7f538c199160 2024-03-12T06:03:45.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f53897fa640 1 -- 172.21.15.76:0/784243636 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384004860 con 0x7f538c199160 2024-03-12T06:03:45.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.628+0000 7f53928a3640 1 -- 172.21.15.76:0/784243636 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f538c1a4810 con 0x7f538c199160 2024-03-12T06:03:45.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.630+0000 7f53897fa640 1 -- 172.21.15.76:0/784243636 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f538400a040 con 0x7f538c199160 2024-03-12T06:03:45.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.630+0000 7f5395471740 1 Processor -- start 2024-03-12T06:03:45.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f5395471740 1 -- start start 2024-03-12T06:03:45.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f5395471740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04250 0x560807a04630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f5395471740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a04b70 0x560807a0d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f5395471740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a0d750 0x560807a0fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f5395471740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608079bf960 con 0x560807a0d750 2024-03-12T06:03:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f5395471740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560807973310 con 0x560807a04250 2024-03-12T06:03:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f5395471740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560807970840 con 0x560807a04b70 2024-03-12T06:03:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f538b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a0d750 0x560807a0fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f538b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a0d750 0x560807a0fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34718/0 (socket says 172.21.15.76:34718) 2024-03-12T06:03:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f538b7fe640 1 -- 172.21.15.76:0/4207632035 learned_addr learned my addr 172.21.15.76:0/4207632035 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f53897fa640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f53740a1ba0 0x7f53740a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f538bfff640 1 --2- 172.21.15.76:0/4207632035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a04b70 0x560807a0d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f538b7fe640 1 --2- 172.21.15.76:0/4207632035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04250 0x560807a04630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f5390e40640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f53740a1ba0 0x7f53740a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f538b7fe640 1 -- 172.21.15.76:0/4207632035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a04b70 msgr2=0x560807a0d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f538b7fe640 1 --2- 172.21.15.76:0/4207632035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a04b70 0x560807a0d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f538b7fe640 1 -- 172.21.15.76:0/4207632035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04250 msgr2=0x560807a04630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f538b7fe640 1 --2- 172.21.15.76:0/4207632035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04250 0x560807a04630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f538b7fe640 1 -- 172.21.15.76:0/4207632035 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56080792e060 con 0x560807a0d750 2024-03-12T06:03:45.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.631+0000 7f53897fa640 1 -- 172.21.15.76:0/784243636 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f53840f0530 con 0x7f538c199160 2024-03-12T06:03:45.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.632+0000 7f538b7fe640 1 --2- 172.21.15.76:0/4207632035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a0d750 0x560807a0fb40 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f53800144f0 tx=0x7f538000d510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3116a83ac8cd961 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.646 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.632+0000 7f5390e40640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f53740a1ba0 0x7f53740a4060 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f53840ede40 tx=0x7f5384004300 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.632+0000 7f53727fc640 1 -- 172.21.15.76:0/4207632035 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380012710 con 0x560807a0d750 2024-03-12T06:03:45.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.632+0000 7f53727fc640 1 -- 172.21.15.76:0/4207632035 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5380018020 con 0x560807a0d750 2024-03-12T06:03:45.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.632+0000 7f53727fc640 1 -- 172.21.15.76:0/4207632035 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380011aa0 con 0x560807a0d750 2024-03-12T06:03:45.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.632+0000 7f5395471740 1 -- 172.21.15.76:0/4207632035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a0d750 msgr2=0x560807a0fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.632+0000 7f5395471740 1 --2- 172.21.15.76:0/4207632035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a0d750 0x560807a0fb40 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f53800144f0 tx=0x7f538000d510 comp rx=0 tx=0).stop 2024-03-12T06:03:45.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.632+0000 7f5395471740 1 -- 172.21.15.76:0/4207632035 shutdown_connections 2024-03-12T06:03:45.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.632+0000 7f5395471740 1 --2- 172.21.15.76:0/4207632035 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a0d750 0x560807a0fb40 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.647 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.632+0000 7f5395471740 1 --2- 172.21.15.76:0/4207632035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a04b70 0x560807a0d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.632+0000 7f5395471740 1 --2- 172.21.15.76:0/4207632035 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04250 0x560807a04630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.632+0000 7f5395471740 1 -- 172.21.15.76:0/4207632035 >> 172.21.15.76:0/4207632035 conn(0x5608079fe980 msgr2=0x5608079fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.632+0000 7f5395471740 1 -- 172.21.15.76:0/4207632035 shutdown_connections 2024-03-12T06:03:45.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.632+0000 7f5395471740 1 -- 172.21.15.76:0/4207632035 wait complete. 2024-03-12T06:03:45.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.633+0000 7f5395471740 1 -- start start 2024-03-12T06:03:45.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.633+0000 7f5395471740 1 Processor -- start 2024-03-12T06:03:45.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.633+0000 7f5395471740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04250 0x560807a0a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.633+0000 7f5395471740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04b70 0x560807a0ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.648 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.633+0000 7f5395471740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a0d750 0x560807a0b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.633+0000 7f5395471740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56080796c020 con 0x560807a04b70 2024-03-12T06:03:45.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.633+0000 7f5395471740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56080763f430 con 0x560807a04250 2024-03-12T06:03:45.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.633+0000 7f5395471740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608075fbe00 con 0x560807a0d750 2024-03-12T06:03:45.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.633+0000 7f5395471740 10 client.0 _ll_register_callbacks cb 0x560807913cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:45.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.633+0000 7f538bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04b70 0x560807a0ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.633+0000 7f538bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04b70 0x560807a0ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34730/0 (socket says 172.21.15.76:34730) 2024-03-12T06:03:45.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.633+0000 7f538bfff640 1 -- 172.21.15.76:0/1738641239 learned_addr learned my addr 172.21.15.76:0/1738641239 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.633+0000 7f538b7fe640 1 --2- 172.21.15.76:0/1738641239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04250 0x560807a0a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.649 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f538b7fe640 1 --2- 172.21.15.76:0/1738641239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a0d750 0x560807a0b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f538bfff640 1 -- 172.21.15.76:0/1738641239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a0d750 msgr2=0x560807a0b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f538bfff640 1 --2- 172.21.15.76:0/1738641239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a0d750 0x560807a0b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f538bfff640 1 -- 172.21.15.76:0/1738641239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04250 msgr2=0x560807a0a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f538bfff640 1 --2- 172.21.15.76:0/1738641239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04250 0x560807a0a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f538bfff640 1 -- 172.21.15.76:0/1738641239 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f538000dbe0 con 0x560807a04b70 2024-03-12T06:03:45.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f538b7fe640 1 --2- 172.21.15.76:0/1738641239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04250 0x560807a0a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:45.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f538bfff640 1 --2- 172.21.15.76:0/1738641239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04b70 0x560807a0ad10 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f537c009fc0 tx=0x7f537c011870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1927a96b11291562 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.650 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f536b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:45.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f536b7fe640 1 -- 172.21.15.76:0/1738641239 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537c012920 con 0x560807a04b70 2024-03-12T06:03:45.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f536b7fe640 1 -- 172.21.15.76:0/1738641239 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f537c012a80 con 0x560807a04b70 2024-03-12T06:03:45.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f5395471740 1 -- 172.21.15.76:0/1738641239 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56080766b7e0 con 0x560807a04b70 2024-03-12T06:03:45.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f536b7fe640 1 -- 172.21.15.76:0/1738641239 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f537c012c40 con 0x560807a04b70 2024-03-12T06:03:45.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f5395471740 10 client.17286 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:45.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f5395471740 1 -- 172.21.15.76:0/1738641239 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56080796a6a0 con 0x560807a04b70 2024-03-12T06:03:45.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f536b7fe640 1 -- 172.21.15.76:0/1738641239 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56080766b7e0 con 0x560807a04b70 2024-03-12T06:03:45.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f536b7fe640 1 -- 172.21.15.76:0/1738641239 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f537c00c070 con 0x560807a04b70 2024-03-12T06:03:45.651 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f5395471740 10 client.17286 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:45.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f5395471740 10 client.17286 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:03:45.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f5395471740 20 client.17286 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:45.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f5395471740 1 --2- 172.21.15.76:0/1738641239 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560807a1dc50 0x560807a20070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f5395471740 4 client.17286 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:03:45.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.634+0000 7f5395471740 1 -- 172.21.15.76:0/1738641239 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5608076a8720 con 0x560807a1dc50 2024-03-12T06:03:45.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.635+0000 7f538b7fe640 1 --2- 172.21.15.76:0/1738641239 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560807a1dc50 0x560807a20070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.635+0000 7f536b7fe640 10 client.17286 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:03:45.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.635+0000 7f538b7fe640 1 --2- 172.21.15.76:0/1738641239 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560807a1dc50 0x560807a20070 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bb80db71a38d927a server_cookie=7ec49cd3c1179b8f in_seq=0 out_seq=0 2024-03-12T06:03:45.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f536b7fe640 1 -- 172.21.15.76:0/1738641239 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5608076a8720 con 0x560807a1dc50 2024-03-12T06:03:45.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f536b7fe640 10 client.17286 handle_command_reply: tid=0 2024-03-12T06:03:45.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f5395471740 1 client.17286 shutdown 2024-03-12T06:03:45.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f5395471740 1 -- 172.21.15.76:0/1738641239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04b70 msgr2=0x560807a0ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f5395471740 1 --2- 172.21.15.76:0/1738641239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04b70 0x560807a0ad10 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f537c009fc0 tx=0x7f537c011870 comp rx=0 tx=0).stop 2024-03-12T06:03:45.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f5395471740 1 -- 172.21.15.76:0/1738641239 shutdown_connections 2024-03-12T06:03:45.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f5395471740 1 --2- 172.21.15.76:0/1738641239 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560807a1dc50 0x560807a20070 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.653 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f5395471740 1 --2- 172.21.15.76:0/1738641239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a0d750 0x560807a0b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f5395471740 1 --2- 172.21.15.76:0/1738641239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04b70 0x560807a0ad10 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f5395471740 1 --2- 172.21.15.76:0/1738641239 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04250 0x560807a0a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f536b7fe640 0 client.17286 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:03:45.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f5395471740 1 -- 172.21.15.76:0/1738641239 >> 172.21.15.76:0/1738641239 conn(0x5608079fe980 msgr2=0x560807a006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f5395471740 1 -- 172.21.15.76:0/1738641239 shutdown_connections 2024-03-12T06:03:45.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f5395471740 1 -- 172.21.15.76:0/1738641239 wait complete. 2024-03-12T06:03:45.654 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.636+0000 7f5395471740 20 client.17286 trim_cache size 0 max 16384 2024-03-12T06:03:45.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f5395471740 1 Processor -- start 2024-03-12T06:03:45.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f5395471740 1 -- start start 2024-03-12T06:03:45.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f5395471740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04250 0x560807a0cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.660 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f5395471740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04b70 0x560807a0b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f5395471740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a0d750 0x560807a07a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f5395471740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608078f1130 con 0x560807a04250 2024-03-12T06:03:45.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f5395471740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560807a03c90 con 0x560807a04b70 2024-03-12T06:03:45.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f5395471740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560807a252a0 con 0x560807a0d750 2024-03-12T06:03:45.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f538b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04250 0x560807a0cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f538b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04250 0x560807a0cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34732/0 (socket says 172.21.15.76:34732) 2024-03-12T06:03:45.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f538b7fe640 1 -- 172.21.15.76:0/657007314 learned_addr learned my addr 172.21.15.76:0/657007314 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f538bfff640 1 --2- 172.21.15.76:0/657007314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04b70 0x560807a0b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.661 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f538b7fe640 1 -- 172.21.15.76:0/657007314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a0d750 msgr2=0x560807a07a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f538b7fe640 1 --2- 172.21.15.76:0/657007314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a0d750 0x560807a07a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f538b7fe640 1 -- 172.21.15.76:0/657007314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04b70 msgr2=0x560807a0b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f538b7fe640 1 --2- 172.21.15.76:0/657007314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04b70 0x560807a0b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f538b7fe640 1 -- 172.21.15.76:0/657007314 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560807906c20 con 0x560807a04250 2024-03-12T06:03:45.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f538b7fe640 1 --2- 172.21.15.76:0/657007314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04250 0x560807a0cf00 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5380019000 tx=0x7f538000de00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d602ab553a8f167 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f5371ffb640 1 -- 172.21.15.76:0/657007314 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380014380 con 0x560807a04250 2024-03-12T06:03:45.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f5371ffb640 1 -- 172.21.15.76:0/657007314 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5380005b50 con 0x560807a04250 2024-03-12T06:03:45.662 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.648+0000 7f5371ffb640 1 -- 172.21.15.76:0/657007314 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380011e10 con 0x560807a04250 2024-03-12T06:03:45.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.649+0000 7f5395471740 1 -- 172.21.15.76:0/657007314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04250 msgr2=0x560807a0cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.649+0000 7f5395471740 1 --2- 172.21.15.76:0/657007314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04250 0x560807a0cf00 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5380019000 tx=0x7f538000de00 comp rx=0 tx=0).stop 2024-03-12T06:03:45.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.649+0000 7f5395471740 1 -- 172.21.15.76:0/657007314 shutdown_connections 2024-03-12T06:03:45.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.649+0000 7f5395471740 1 --2- 172.21.15.76:0/657007314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a0d750 0x560807a07a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.649+0000 7f5395471740 1 --2- 172.21.15.76:0/657007314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a04b70 0x560807a0b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.649+0000 7f5395471740 1 --2- 172.21.15.76:0/657007314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04250 0x560807a0cf00 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.649+0000 7f5395471740 1 -- 172.21.15.76:0/657007314 >> 172.21.15.76:0/657007314 conn(0x5608079fe980 msgr2=0x560807a05a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.649+0000 7f5395471740 1 -- 172.21.15.76:0/657007314 shutdown_connections 2024-03-12T06:03:45.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.649+0000 7f5395471740 1 -- 172.21.15.76:0/657007314 wait complete. 2024-03-12T06:03:45.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.649+0000 7f5395471740 1 -- start start 2024-03-12T06:03:45.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f5395471740 1 Processor -- start 2024-03-12T06:03:45.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f5395471740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04250 0x560807a132f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f5395471740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a04b70 0x560807a13830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f5395471740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a0d750 0x560807a13db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f5395471740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560807a74340 con 0x560807a04250 2024-03-12T06:03:45.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f5395471740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560807a744b0 con 0x560807a0d750 2024-03-12T06:03:45.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f5395471740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560807a74670 con 0x560807a04b70 2024-03-12T06:03:45.664 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f5395471740 10 client.0 _ll_register_callbacks cb 0x7f538000d3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:45.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f538b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04250 0x560807a132f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f538b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04250 0x560807a132f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34748/0 (socket says 172.21.15.76:34748) 2024-03-12T06:03:45.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f538b7fe640 1 -- 172.21.15.76:0/4242425724 learned_addr learned my addr 172.21.15.76:0/4242425724 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f538bfff640 1 --2- 172.21.15.76:0/4242425724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a04b70 0x560807a13830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f538b7fe640 1 --2- 172.21.15.76:0/4242425724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a0d750 0x560807a13db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f538b7fe640 1 -- 172.21.15.76:0/4242425724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a04b70 msgr2=0x560807a13830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f538b7fe640 1 --2- 172.21.15.76:0/4242425724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a04b70 0x560807a13830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f538b7fe640 1 -- 172.21.15.76:0/4242425724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a0d750 msgr2=0x560807a13db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f538b7fe640 1 --2- 172.21.15.76:0/4242425724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a0d750 0x560807a13db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f538b7fe640 1 -- 172.21.15.76:0/4242425724 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56080792e060 con 0x560807a04250 2024-03-12T06:03:45.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f538b7fe640 1 --2- 172.21.15.76:0/4242425724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04250 0x560807a132f0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f538002c040 tx=0x7f5380011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc4f48a5812e7f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f536b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:45.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.650+0000 7f536b7fe640 1 -- 172.21.15.76:0/4242425724 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53800117e0 con 0x560807a04250 2024-03-12T06:03:45.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.651+0000 7f536b7fe640 1 -- 172.21.15.76:0/4242425724 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5380025070 con 0x560807a04250 2024-03-12T06:03:45.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.651+0000 7f5395471740 1 -- 172.21.15.76:0/4242425724 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5380018020 con 0x560807a04250 2024-03-12T06:03:45.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.651+0000 7f536b7fe640 1 -- 172.21.15.76:0/4242425724 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53800293c0 con 0x560807a04250 2024-03-12T06:03:45.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.651+0000 7f5395471740 10 client.17292 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:45.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.651+0000 7f5395471740 1 -- 172.21.15.76:0/4242425724 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56080792c8c0 con 0x560807a04250 2024-03-12T06:03:45.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.651+0000 7f536b7fe640 1 -- 172.21.15.76:0/4242425724 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5380018020 con 0x560807a04250 2024-03-12T06:03:45.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.651+0000 7f536b7fe640 1 -- 172.21.15.76:0/4242425724 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5380012ba0 con 0x560807a04250 2024-03-12T06:03:45.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.651+0000 7f5395471740 10 client.17292 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:45.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.651+0000 7f5395471740 10 client.17292 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:03:45.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.651+0000 7f5395471740 20 client.17292 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:45.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.651+0000 7f5395471740 1 --2- 172.21.15.76:0/4242425724 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560807a75ac0 0x560807a77ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.667 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.651+0000 7f5395471740 4 client.17292 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:03:45.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.651+0000 7f5395471740 1 -- 172.21.15.76:0/4242425724 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5608076f7890 con 0x560807a75ac0 2024-03-12T06:03:45.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.651+0000 7f538bfff640 1 --2- 172.21.15.76:0/4242425724 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560807a75ac0 0x560807a77ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f536b7fe640 10 client.17292 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:03:45.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f538bfff640 1 --2- 172.21.15.76:0/4242425724 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560807a75ac0 0x560807a77ee0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ed48ecdee84e5afe server_cookie=8af8df10061804b8 in_seq=0 out_seq=0 2024-03-12T06:03:45.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f536b7fe640 1 -- 172.21.15.76:0/4242425724 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5608076f7890 con 0x560807a75ac0 2024-03-12T06:03:45.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f536b7fe640 10 client.17292 handle_command_reply: tid=0 2024-03-12T06:03:45.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f5395471740 1 client.17292 shutdown 2024-03-12T06:03:45.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f5395471740 1 -- 172.21.15.76:0/4242425724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04250 msgr2=0x560807a132f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.668 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f5395471740 1 --2- 172.21.15.76:0/4242425724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04250 0x560807a132f0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f538002c040 tx=0x7f5380011660 comp rx=0 tx=0).stop 2024-03-12T06:03:45.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f5395471740 1 -- 172.21.15.76:0/4242425724 shutdown_connections 2024-03-12T06:03:45.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f5395471740 1 --2- 172.21.15.76:0/4242425724 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560807a75ac0 0x560807a77ee0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f5395471740 1 --2- 172.21.15.76:0/4242425724 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560807a0d750 0x560807a13db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f5395471740 1 --2- 172.21.15.76:0/4242425724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560807a04b70 0x560807a13830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f536b7fe640 0 client.17292 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:03:45.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f5395471740 1 --2- 172.21.15.76:0/4242425724 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560807a04250 0x560807a132f0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f5395471740 1 -- 172.21.15.76:0/4242425724 >> 172.21.15.76:0/4242425724 conn(0x5608079fe980 msgr2=0x5608079fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.669 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f5395471740 1 -- 172.21.15.76:0/4242425724 shutdown_connections 2024-03-12T06:03:45.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f5395471740 1 -- 172.21.15.76:0/4242425724 wait complete. 2024-03-12T06:03:45.670 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.652+0000 7f5395471740 20 client.17292 trim_cache size 0 max 16384 2024-03-12T06:03:45.670 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:03:45.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.653+0000 7f53928a3640 1 -- 172.21.15.76:0/784243636 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f53740a1ba0 msgr2=0x7f53740a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.653+0000 7f53928a3640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f53740a1ba0 0x7f53740a4060 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f53840ede40 tx=0x7f5384004300 comp rx=0 tx=0).stop 2024-03-12T06:03:45.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.653+0000 7f53928a3640 1 -- 172.21.15.76:0/784243636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f538c199160 msgr2=0x7f538c1a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.653+0000 7f53928a3640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f538c199160 0x7f538c1a3710 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f5384004af0 tx=0x7f5384034cc0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.653+0000 7f53928a3640 1 -- 172.21.15.76:0/784243636 shutdown_connections 2024-03-12T06:03:45.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.653+0000 7f53928a3640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f53740a1ba0 0x7f53740a4060 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.653+0000 7f53928a3640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f538c199160 0x7f538c1a3710 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.653+0000 7f53928a3640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f538c105360 0x7f538c198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.653+0000 7f53928a3640 1 --2- 172.21.15.76:0/784243636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f538c100290 0x7f538c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.653+0000 7f53928a3640 1 -- 172.21.15.76:0/784243636 >> 172.21.15.76:0/784243636 conn(0x7f538c077230 msgr2=0x7f538c106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.653+0000 7f53928a3640 1 -- 172.21.15.76:0/784243636 shutdown_connections 2024-03-12T06:03:45.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.653+0000 7f53928a3640 1 -- 172.21.15.76:0/784243636 wait complete. 2024-03-12T06:03:45.673 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:03:45.673 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T06:03:45.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:45 smithi082 ceph-mon[36382]: pgmap v246: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:45.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.773+0000 7fc1f04ae640 1 -- 172.21.15.76:0/3131163153 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1e8101d30 msgr2=0x7fc1e8105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.773+0000 7fc1f04ae640 1 --2- 172.21.15.76:0/3131163153 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1e8101d30 0x7fc1e8105d70 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009930 tx=0x7fc1d802bc30 comp rx=0 tx=0).stop 2024-03-12T06:03:45.780 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.773+0000 7fc1f04ae640 1 -- 172.21.15.76:0/3131163153 shutdown_connections 2024-03-12T06:03:45.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.773+0000 7fc1f04ae640 1 --2- 172.21.15.76:0/3131163153 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1e8106450 0x7fc1e810e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.773+0000 7fc1f04ae640 1 --2- 172.21.15.76:0/3131163153 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1e8101d30 0x7fc1e8105d70 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.773+0000 7fc1f04ae640 1 --2- 172.21.15.76:0/3131163153 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc1e8101380 0x7fc1e8101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.773+0000 7fc1f04ae640 1 -- 172.21.15.76:0/3131163153 >> 172.21.15.76:0/3131163153 conn(0x7fc1e80fcc20 msgr2=0x7fc1e80ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.773+0000 7fc1f04ae640 1 -- 172.21.15.76:0/3131163153 shutdown_connections 2024-03-12T06:03:45.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.773+0000 7fc1f04ae640 1 -- 172.21.15.76:0/3131163153 wait complete. 2024-03-12T06:03:45.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.773+0000 7fc1f04ae640 1 Processor -- start 2024-03-12T06:03:45.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1f04ae640 1 -- start start 2024-03-12T06:03:45.781 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1f04ae640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1e8101380 0x7fc1e819d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1f04ae640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc1e8106450 0x7fc1e819d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1f04ae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1e81975d0 0x7fc1e8197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1f04ae640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e8197fc0 con 0x7fc1e8101380 2024-03-12T06:03:45.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1f04ae640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e8198130 con 0x7fc1e8106450 2024-03-12T06:03:45.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1f04ae640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e81982f0 con 0x7fc1e81975d0 2024-03-12T06:03:45.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1ee24a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1e8101380 0x7fc1e819d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1ee24a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1e8101380 0x7fc1e819d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34764/0 (socket says 172.21.15.76:34764) 2024-03-12T06:03:45.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1ee24a640 1 -- 172.21.15.76:0/1720059584 learned_addr learned my addr 172.21.15.76:0/1720059584 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1eea4b640 1 --2- 172.21.15.76:0/1720059584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1e81975d0 0x7fc1e8197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1eda49640 1 --2- 172.21.15.76:0/1720059584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc1e8106450 0x7fc1e819d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1ee24a640 1 -- 172.21.15.76:0/1720059584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1e81975d0 msgr2=0x7fc1e8197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1ee24a640 1 --2- 172.21.15.76:0/1720059584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1e81975d0 0x7fc1e8197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1ee24a640 1 -- 172.21.15.76:0/1720059584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc1e8106450 msgr2=0x7fc1e819d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1ee24a640 1 --2- 172.21.15.76:0/1720059584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc1e8106450 0x7fc1e819d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1ee24a640 1 -- 172.21.15.76:0/1720059584 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d8009560 con 0x7fc1e8101380 2024-03-12T06:03:45.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1ee24a640 1 --2- 172.21.15.76:0/1720059584 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1e8101380 0x7fc1e819d420 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000db80 tx=0x7fc1e000b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1df7fe640 1 -- 172.21.15.76:0/1720059584 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0003b70 con 0x7fc1e8101380 2024-03-12T06:03:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1df7fe640 1 -- 172.21.15.76:0/1720059584 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1e000de80 con 0x7fc1e8101380 2024-03-12T06:03:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1f04ae640 1 -- 172.21.15.76:0/1720059584 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e81a42e0 con 0x7fc1e8101380 2024-03-12T06:03:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.774+0000 7fc1df7fe640 1 -- 172.21.15.76:0/1720059584 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e00106c0 con 0x7fc1e8101380 2024-03-12T06:03:45.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.775+0000 7fc1f04ae640 1 -- 172.21.15.76:0/1720059584 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e81a4810 con 0x7fc1e8101380 2024-03-12T06:03:45.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.777+0000 7fc1df7fe640 1 -- 172.21.15.76:0/1720059584 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc1e0004020 con 0x7fc1e8101380 2024-03-12T06:03:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.777+0000 7fc1f3080740 1 Processor -- start 2024-03-12T06:03:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.777+0000 7fc1f3080740 1 -- start start 2024-03-12T06:03:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.777+0000 7fc1f3080740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5250 0x5585694a5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.777+0000 7fc1f3080740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 0x5585694ae210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.777+0000 7fc1f3080740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694ae750 0x5585694b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.777+0000 7fc1f3080740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558569460960 con 0x5585694ae750 2024-03-12T06:03:45.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.777+0000 7fc1f3080740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558569414310 con 0x5585694a5250 2024-03-12T06:03:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.777+0000 7fc1f3080740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558569411840 con 0x5585694a5b70 2024-03-12T06:03:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1eda49640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694ae750 0x5585694b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1df7fe640 1 --2- 172.21.15.76:0/1720059584 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc1d40a1c20 0x7fc1d40a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1eda49640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694ae750 0x5585694b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34780/0 (socket says 172.21.15.76:34780) 2024-03-12T06:03:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1ee24a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 0x5585694ae210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1eda49640 1 -- 172.21.15.76:0/2309484799 learned_addr learned my addr 172.21.15.76:0/2309484799 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1eea4b640 1 --2- 172.21.15.76:0/1720059584 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc1d40a1c20 0x7fc1d40a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1eea4b640 1 --2- 172.21.15.76:0/2309484799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5250 0x5585694a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1eda49640 1 -- 172.21.15.76:0/2309484799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 msgr2=0x5585694ae210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1eda49640 1 --2- 172.21.15.76:0/2309484799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 0x5585694ae210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1df7fe640 1 -- 172.21.15.76:0/1720059584 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fc1e00cd8c0 con 0x7fc1e8101380 2024-03-12T06:03:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1eda49640 1 -- 172.21.15.76:0/2309484799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5250 msgr2=0x5585694a5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1eda49640 1 --2- 172.21.15.76:0/2309484799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5250 0x5585694a5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1eda49640 1 -- 172.21.15.76:0/2309484799 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585693cf060 con 0x5585694ae750 2024-03-12T06:03:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1eda49640 1 --2- 172.21.15.76:0/2309484799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694ae750 0x5585694b0b40 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc1e806a8e0 tx=0x7fc1d802c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151818b26d9ad263 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1eea4b640 1 --2- 172.21.15.76:0/1720059584 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc1d40a1c20 0x7fc1d40a40e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4005fd0 tx=0x7fc1e4005e80 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1dcbfe640 1 -- 172.21.15.76:0/2309484799 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8004a40 con 0x5585694ae750 2024-03-12T06:03:45.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1dcbfe640 1 -- 172.21.15.76:0/2309484799 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1d8004030 con 0x5585694ae750 2024-03-12T06:03:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.778+0000 7fc1dcbfe640 1 -- 172.21.15.76:0/2309484799 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d80053c0 con 0x5585694ae750 2024-03-12T06:03:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.779+0000 7fc1f3080740 1 -- 172.21.15.76:0/2309484799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694ae750 msgr2=0x5585694b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.779+0000 7fc1f3080740 1 --2- 172.21.15.76:0/2309484799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694ae750 0x5585694b0b40 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc1e806a8e0 tx=0x7fc1d802c6f0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.779+0000 7fc1f3080740 1 -- 172.21.15.76:0/2309484799 shutdown_connections 2024-03-12T06:03:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.779+0000 7fc1f3080740 1 --2- 172.21.15.76:0/2309484799 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694ae750 0x5585694b0b40 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.779+0000 7fc1f3080740 1 --2- 172.21.15.76:0/2309484799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 0x5585694ae210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.779+0000 7fc1f3080740 1 --2- 172.21.15.76:0/2309484799 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5250 0x5585694a5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.779+0000 7fc1f3080740 1 -- 172.21.15.76:0/2309484799 >> 172.21.15.76:0/2309484799 conn(0x55856949f980 msgr2=0x55856949fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.779+0000 7fc1f3080740 1 -- 172.21.15.76:0/2309484799 shutdown_connections 2024-03-12T06:03:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.779+0000 7fc1f3080740 1 -- 172.21.15.76:0/2309484799 wait complete. 2024-03-12T06:03:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.779+0000 7fc1f3080740 1 -- start start 2024-03-12T06:03:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1f3080740 1 Processor -- start 2024-03-12T06:03:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1f3080740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694a5250 0x5585694ab730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1f3080740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5b70 0x5585694abc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1f3080740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694ae750 0x5585694ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1f3080740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55856940d020 con 0x5585694a5250 2024-03-12T06:03:45.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1f3080740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585690e0430 con 0x5585694a5b70 2024-03-12T06:03:45.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1f3080740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55856909ce00 con 0x5585694ae750 2024-03-12T06:03:45.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1f3080740 10 client.0 _ll_register_callbacks cb 0x5585693b4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:45.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1ee24a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694a5250 0x5585694ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1ee24a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694a5250 0x5585694ab730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34786/0 (socket says 172.21.15.76:34786) 2024-03-12T06:03:45.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1ee24a640 1 -- 172.21.15.76:0/684718123 learned_addr learned my addr 172.21.15.76:0/684718123 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1eda49640 1 --2- 172.21.15.76:0/684718123 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5b70 0x5585694abc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1ee24a640 1 -- 172.21.15.76:0/684718123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694ae750 msgr2=0x5585694ac1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1eea4b640 1 --2- 172.21.15.76:0/684718123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694ae750 0x5585694ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1ee24a640 1 --2- 172.21.15.76:0/684718123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694ae750 0x5585694ac1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1ee24a640 1 -- 172.21.15.76:0/684718123 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5b70 msgr2=0x5585694abc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1ee24a640 1 --2- 172.21.15.76:0/684718123 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5b70 0x5585694abc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1ee24a640 1 -- 172.21.15.76:0/684718123 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e810e830 con 0x5585694a5250 2024-03-12T06:03:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1ee24a640 1 --2- 172.21.15.76:0/684718123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694a5250 0x5585694ab730 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0000f80 tx=0x7fc1e009a030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef842edf6035b2ed server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.780+0000 7fc1d1ffb640 1 -- 172.21.15.76:0/684718123 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e000be60 con 0x5585694a5250 2024-03-12T06:03:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.781+0000 7fc1d1ffb640 1 -- 172.21.15.76:0/684718123 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fc1e009abf0 con 0x5585694a5250 2024-03-12T06:03:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.781+0000 7fc1f3080740 1 -- 172.21.15.76:0/684718123 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55856910c7e0 con 0x5585694a5250 2024-03-12T06:03:45.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.781+0000 7fc1d1ffb640 1 -- 172.21.15.76:0/684718123 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1e00212a0 con 0x5585694a5250 2024-03-12T06:03:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.781+0000 7fc1f3080740 10 client.17304 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.781+0000 7fc1f3080740 1 -- 172.21.15.76:0/684718123 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55856940b6a0 con 0x5585694a5250 2024-03-12T06:03:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.781+0000 7fc1d1ffb640 1 -- 172.21.15.76:0/684718123 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55856910c7e0 con 0x5585694a5250 2024-03-12T06:03:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.781+0000 7fc1d1ffb640 1 -- 172.21.15.76:0/684718123 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc1e00cb290 con 0x5585694a5250 2024-03-12T06:03:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.781+0000 7fc1f3080740 10 client.17304 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.781+0000 7fc1f3080740 10 client.17304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.781+0000 7fc1f3080740 20 client.17304 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:45.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.781+0000 7fc1f3080740 1 --2- 172.21.15.76:0/684718123 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5585694beb10 0x5585694c0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.781+0000 7fc1f3080740 4 client.17304 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:03:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.781+0000 7fc1f3080740 1 -- 172.21.15.76:0/684718123 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558569149720 con 0x5585694beb10 2024-03-12T06:03:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.781+0000 7fc1eda49640 1 --2- 172.21.15.76:0/684718123 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5585694beb10 0x5585694c0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.782+0000 7fc1eda49640 1 --2- 172.21.15.76:0/684718123 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5585694beb10 0x5585694c0f20 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cac206cfe59d7a1 server_cookie=12801030614dd426 in_seq=0 out_seq=0 2024-03-12T06:03:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.782+0000 7fc1d1ffb640 10 client.17304 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.782+0000 7fc1d1ffb640 1 -- 172.21.15.76:0/684718123 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558569149720 con 0x5585694beb10 2024-03-12T06:03:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.782+0000 7fc1d1ffb640 10 client.17304 handle_command_reply: tid=0 2024-03-12T06:03:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.782+0000 7fc1f3080740 1 client.17304 shutdown 2024-03-12T06:03:45.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.783+0000 7fc1f3080740 1 -- 172.21.15.76:0/684718123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694a5250 msgr2=0x5585694ab730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.783+0000 7fc1f3080740 1 --2- 172.21.15.76:0/684718123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694a5250 0x5585694ab730 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0000f80 tx=0x7fc1e009a030 comp rx=0 tx=0).stop 2024-03-12T06:03:45.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.783+0000 7fc1f3080740 1 -- 172.21.15.76:0/684718123 shutdown_connections 2024-03-12T06:03:45.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.783+0000 7fc1f3080740 1 --2- 172.21.15.76:0/684718123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694ae750 0x5585694ac1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.783+0000 7fc1f3080740 1 --2- 172.21.15.76:0/684718123 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5b70 0x5585694abc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.783+0000 7fc1f3080740 1 --2- 172.21.15.76:0/684718123 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5585694beb10 0x5585694c0f20 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.783+0000 7fc1f3080740 1 --2- 172.21.15.76:0/684718123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694a5250 0x5585694ab730 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.783+0000 7fc1d1ffb640 0 client.17304 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:45.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.783+0000 7fc1f3080740 1 -- 172.21.15.76:0/684718123 >> 172.21.15.76:0/684718123 conn(0x55856949f980 msgr2=0x5585694a16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.783+0000 7fc1f3080740 1 -- 172.21.15.76:0/684718123 shutdown_connections 2024-03-12T06:03:45.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.783+0000 7fc1f3080740 1 -- 172.21.15.76:0/684718123 wait complete. 2024-03-12T06:03:45.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.783+0000 7fc1f3080740 20 client.17304 trim_cache size 0 max 16384 2024-03-12T06:03:45.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.794+0000 7fc1f3080740 1 Processor -- start 2024-03-12T06:03:45.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.794+0000 7fc1f3080740 1 -- start start 2024-03-12T06:03:45.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.794+0000 7fc1f3080740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5250 0x5585694abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.794+0000 7fc1f3080740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 0x5585694a14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.794+0000 7fc1f3080740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694ae750 0x5585694b55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.794+0000 7fc1f3080740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558569392130 con 0x5585694ae750 2024-03-12T06:03:45.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.794+0000 7fc1f3080740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585694a4cc0 con 0x5585694a5250 2024-03-12T06:03:45.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.794+0000 7fc1f3080740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55856949b5c0 con 0x5585694a5b70 2024-03-12T06:03:45.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1eea4b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694ae750 0x5585694b55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1eea4b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694ae750 0x5585694b55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34792/0 (socket says 172.21.15.76:34792) 2024-03-12T06:03:45.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1eea4b640 1 -- 172.21.15.76:0/2451879787 learned_addr learned my addr 172.21.15.76:0/2451879787 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1eda49640 1 --2- 172.21.15.76:0/2451879787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 0x5585694a14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1ee24a640 1 --2- 172.21.15.76:0/2451879787 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5250 0x5585694abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1eea4b640 1 -- 172.21.15.76:0/2451879787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 msgr2=0x5585694a14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1eea4b640 1 --2- 172.21.15.76:0/2451879787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 0x5585694a14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1eea4b640 1 -- 172.21.15.76:0/2451879787 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5250 msgr2=0x5585694abfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1eea4b640 1 --2- 172.21.15.76:0/2451879787 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5250 0x5585694abfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1eea4b640 1 -- 172.21.15.76:0/2451879787 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585693a7c20 con 0x5585694ae750 2024-03-12T06:03:45.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1eda49640 1 --2- 172.21.15.76:0/2451879787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 0x5585694a14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:45.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1eea4b640 1 --2- 172.21.15.76:0/2451879787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694ae750 0x5585694b55a0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400bed0 tx=0x7fc1e4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c13b63ef3f5ac22 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1d3fff640 1 -- 172.21.15.76:0/2451879787 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4028660 con 0x5585694ae750 2024-03-12T06:03:45.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1d3fff640 1 -- 172.21.15.76:0/2451879787 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1e4028ca0 con 0x5585694ae750 2024-03-12T06:03:45.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.795+0000 7fc1d3fff640 1 -- 172.21.15.76:0/2451879787 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e401fc60 con 0x5585694ae750 2024-03-12T06:03:45.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 -- 172.21.15.76:0/2451879787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694ae750 msgr2=0x5585694b55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 --2- 172.21.15.76:0/2451879787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694ae750 0x5585694b55a0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400bed0 tx=0x7fc1e4012000 comp rx=0 tx=0).stop 2024-03-12T06:03:45.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 -- 172.21.15.76:0/2451879787 shutdown_connections 2024-03-12T06:03:45.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 --2- 172.21.15.76:0/2451879787 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694ae750 0x5585694b55a0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 --2- 172.21.15.76:0/2451879787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 0x5585694a14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 --2- 172.21.15.76:0/2451879787 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694a5250 0x5585694abfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 -- 172.21.15.76:0/2451879787 >> 172.21.15.76:0/2451879787 conn(0x55856949f980 msgr2=0x5585694a0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 -- 172.21.15.76:0/2451879787 shutdown_connections 2024-03-12T06:03:45.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 -- 172.21.15.76:0/2451879787 wait complete. 2024-03-12T06:03:45.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 -- start start 2024-03-12T06:03:45.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 Processor -- start 2024-03-12T06:03:45.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694a5250 0x5585694b4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 0x5585694b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694ae750 0x55856950f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55856950f640 con 0x5585694a5250 2024-03-12T06:03:45.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55856950f7b0 con 0x5585694ae750 2024-03-12T06:03:45.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55856950f970 con 0x5585694a5b70 2024-03-12T06:03:45.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1f3080740 10 client.0 _ll_register_callbacks cb 0x7fc1d8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:45.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1ee24a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694a5250 0x5585694b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1ee24a640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694a5250 0x5585694b4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34802/0 (socket says 172.21.15.76:34802) 2024-03-12T06:03:45.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.796+0000 7fc1ee24a640 1 -- 172.21.15.76:0/226655334 learned_addr learned my addr 172.21.15.76:0/226655334 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:45.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1eda49640 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 0x5585694b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1eea4b640 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694ae750 0x55856950f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1ee24a640 1 -- 172.21.15.76:0/226655334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 msgr2=0x5585694b4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1ee24a640 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 0x5585694b4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1ee24a640 1 -- 172.21.15.76:0/226655334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694ae750 msgr2=0x55856950f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:45.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1ee24a640 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694ae750 0x55856950f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1ee24a640 1 -- 172.21.15.76:0/226655334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e4018070 con 0x5585694a5250 2024-03-12T06:03:45.813 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1eda49640 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 0x5585694b4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:45.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1eea4b640 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694ae750 0x55856950f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:45.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1ee24a640 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694a5250 0x5585694b4930 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0096ec0 tx=0x7fc1e0003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90362b9d682d4cc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:45.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:45.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1d1ffb640 1 -- 172.21.15.76:0/226655334 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0096d20 con 0x5585694a5250 2024-03-12T06:03:45.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1d1ffb640 1 -- 172.21.15.76:0/226655334 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fc1e009abf0 con 0x5585694a5250 2024-03-12T06:03:45.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1f3080740 1 -- 172.21.15.76:0/226655334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc1d8004030 con 0x5585694a5250 2024-03-12T06:03:45.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1d1ffb640 1 -- 172.21.15.76:0/226655334 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1e000be60 con 0x5585694a5250 2024-03-12T06:03:45.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1f3080740 10 client.17310 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:45.814 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1f3080740 1 -- 172.21.15.76:0/226655334 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5585693cd8c0 con 0x5585694a5250 2024-03-12T06:03:45.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1d1ffb640 1 -- 172.21.15.76:0/226655334 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1d8004030 con 0x5585694a5250 2024-03-12T06:03:45.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1d1ffb640 1 -- 172.21.15.76:0/226655334 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc1e00b5020 con 0x5585694a5250 2024-03-12T06:03:45.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1f3080740 10 client.17310 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:45.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1f3080740 10 client.17310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:45.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1f3080740 20 client.17310 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:45.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1f3080740 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558569516e60 0x558569519280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:45.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1f3080740 4 client.17310 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:03:45.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.797+0000 7fc1f3080740 1 -- 172.21.15.76:0/226655334 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558569198890 con 0x558569516e60 2024-03-12T06:03:45.815 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1eda49640 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558569516e60 0x558569519280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:45.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1d1ffb640 10 client.17310 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:45.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1eda49640 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558569516e60 0x558569519280 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=541497f757b8f0e3 server_cookie=3a703dbed40b5574 in_seq=0 out_seq=0 2024-03-12T06:03:45.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1d1ffb640 1 -- 172.21.15.76:0/226655334 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558569198890 con 0x558569516e60 2024-03-12T06:03:45.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1d1ffb640 10 client.17310 handle_command_reply: tid=0 2024-03-12T06:03:45.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1f3080740 1 client.17310 shutdown 2024-03-12T06:03:45.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1f3080740 1 -- 172.21.15.76:0/226655334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694a5250 msgr2=0x5585694b4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1f3080740 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694a5250 0x5585694b4930 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0096ec0 tx=0x7fc1e0003cd0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1f3080740 1 -- 172.21.15.76:0/226655334 shutdown_connections 2024-03-12T06:03:45.816 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1f3080740 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5585694ae750 0x55856950f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1f3080740 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5585694a5b70 0x5585694b4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1f3080740 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558569516e60 0x558569519280 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1f3080740 1 --2- 172.21.15.76:0/226655334 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5585694a5250 0x5585694b4930 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1d1ffb640 0 client.17310 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:45.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1f3080740 1 -- 172.21.15.76:0/226655334 >> 172.21.15.76:0/226655334 conn(0x55856949f980 msgr2=0x55856949eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1f3080740 1 -- 172.21.15.76:0/226655334 shutdown_connections 2024-03-12T06:03:45.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.798+0000 7fc1f3080740 1 -- 172.21.15.76:0/226655334 wait complete. 2024-03-12T06:03:45.817 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.799+0000 7fc1f3080740 20 client.17310 trim_cache size 0 max 16384 2024-03-12T06:03:45.818 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:03:45.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.799+0000 7fc1f04ae640 1 -- 172.21.15.76:0/1720059584 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc1d40a1c20 msgr2=0x7fc1d40a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.799+0000 7fc1f04ae640 1 --2- 172.21.15.76:0/1720059584 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc1d40a1c20 0x7fc1d40a40e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4005fd0 tx=0x7fc1e4005e80 comp rx=0 tx=0).stop 2024-03-12T06:03:45.819 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.799+0000 7fc1f04ae640 1 -- 172.21.15.76:0/1720059584 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1e8101380 msgr2=0x7fc1e819d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:45.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.799+0000 7fc1f04ae640 1 --2- 172.21.15.76:0/1720059584 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1e8101380 0x7fc1e819d420 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000db80 tx=0x7fc1e000b630 comp rx=0 tx=0).stop 2024-03-12T06:03:45.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.799+0000 7fc1f04ae640 1 -- 172.21.15.76:0/1720059584 shutdown_connections 2024-03-12T06:03:45.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.799+0000 7fc1f04ae640 1 --2- 172.21.15.76:0/1720059584 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc1d40a1c20 0x7fc1d40a40e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.799+0000 7fc1f04ae640 1 --2- 172.21.15.76:0/1720059584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1e81975d0 0x7fc1e8197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.799+0000 7fc1f04ae640 1 --2- 172.21.15.76:0/1720059584 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc1e8106450 0x7fc1e819d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.799+0000 7fc1f04ae640 1 --2- 172.21.15.76:0/1720059584 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1e8101380 0x7fc1e819d420 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:45.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.799+0000 7fc1f04ae640 1 -- 172.21.15.76:0/1720059584 >> 172.21.15.76:0/1720059584 conn(0x7fc1e80fcc20 msgr2=0x7fc1e80fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:45.820 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.799+0000 7fc1f04ae640 1 -- 172.21.15.76:0/1720059584 shutdown_connections 2024-03-12T06:03:45.821 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:45.799+0000 7fc1f04ae640 1 -- 172.21.15.76:0/1720059584 wait complete. 2024-03-12T06:03:46.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:46 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2565532998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:03:46.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:46 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2565532998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:03:46.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:46 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2565532998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:03:46.822 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T06:03:46.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:03:46.823 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T06:03:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.936+0000 7fcc9804e640 1 -- 172.21.15.76:0/4275702374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc901054a0 msgr2=0x7fcc90105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.936+0000 7fcc9804e640 1 --2- 172.21.15.76:0/4275702374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc901054a0 0x7fcc90105920 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fcc88009d00 tx=0x7fcc8802c0f0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.936+0000 7fcc9804e640 1 -- 172.21.15.76:0/4275702374 shutdown_connections 2024-03-12T06:03:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.936+0000 7fcc9804e640 1 --2- 172.21.15.76:0/4275702374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc901054a0 0x7fcc90105920 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.936+0000 7fcc9804e640 1 --2- 172.21.15.76:0/4275702374 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc90100830 0x7fcc90104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.943 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.936+0000 7fcc9804e640 1 --2- 172.21.15.76:0/4275702374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc900fff10 0x7fcc901002f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.936+0000 7fcc9804e640 1 -- 172.21.15.76:0/4275702374 >> 172.21.15.76:0/4275702374 conn(0x7fcc90077260 msgr2=0x7fcc90077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.936+0000 7fcc9804e640 1 -- 172.21.15.76:0/4275702374 shutdown_connections 2024-03-12T06:03:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.936+0000 7fcc9804e640 1 -- 172.21.15.76:0/4275702374 wait complete. 2024-03-12T06:03:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.936+0000 7fcc9804e640 1 Processor -- start 2024-03-12T06:03:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.936+0000 7fcc9804e640 1 -- start start 2024-03-12T06:03:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc9804e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc900fff10 0x7fcc9019ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc9804e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc90100830 0x7fcc9019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.944 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc9804e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc901054a0 0x7fcc901a2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc9804e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc901a2d90 con 0x7fcc901054a0 2024-03-12T06:03:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc9804e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc901a2f00 con 0x7fcc90100830 2024-03-12T06:03:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc9804e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc901a30c0 con 0x7fcc900fff10 2024-03-12T06:03:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc965eb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc901054a0 0x7fcc901a2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc965eb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc901054a0 0x7fcc901a2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34830/0 (socket says 172.21.15.76:34830) 2024-03-12T06:03:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc965eb640 1 -- 172.21.15.76:0/3970857456 learned_addr learned my addr 172.21.15.76:0/3970857456 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc955e9640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc90100830 0x7fcc9019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc95dea640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc900fff10 0x7fcc9019ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc965eb640 1 -- 172.21.15.76:0/3970857456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc900fff10 msgr2=0x7fcc9019ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc965eb640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc900fff10 0x7fcc9019ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc965eb640 1 -- 172.21.15.76:0/3970857456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc90100830 msgr2=0x7fcc9019e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc965eb640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc90100830 0x7fcc9019e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc965eb640 1 -- 172.21.15.76:0/3970857456 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc88009900 con 0x7fcc901054a0 2024-03-12T06:03:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc955e9640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc90100830 0x7fcc9019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc95dea640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc900fff10 0x7fcc9019ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:03:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc965eb640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc901054a0 0x7fcc901a2790 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fcc88005cc0 tx=0x7fcc88034a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:46.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc86ffd640 1 -- 172.21.15.76:0/3970857456 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc880328b0 con 0x7fcc901054a0 2024-03-12T06:03:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc86ffd640 1 -- 172.21.15.76:0/3970857456 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc88003e50 con 0x7fcc901054a0 2024-03-12T06:03:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc9804e640 1 -- 172.21.15.76:0/3970857456 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc901a3340 con 0x7fcc901054a0 2024-03-12T06:03:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc86ffd640 1 -- 172.21.15.76:0/3970857456 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc880047f0 con 0x7fcc901054a0 2024-03-12T06:03:46.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.937+0000 7fcc9804e640 1 -- 172.21.15.76:0/3970857456 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc901aac20 con 0x7fcc901054a0 2024-03-12T06:03:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.939+0000 7fcc86ffd640 1 -- 172.21.15.76:0/3970857456 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fcc8800a040 con 0x7fcc901054a0 2024-03-12T06:03:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.940+0000 7fcc9ac25740 1 Processor -- start 2024-03-12T06:03:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.940+0000 7fcc9ac25740 1 -- start start 2024-03-12T06:03:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.940+0000 7fcc86ffd640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcc7c0a1c20 0x7fcc7c0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.940+0000 7fcc9ac25740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5380 0x56079c4b5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.940+0000 7fcc9ac25740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5d30 0x56079c4be2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.940+0000 7fcc9ac25740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 0x56079c4c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.940+0000 7fcc9ac25740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079c423f90 con 0x56079c4b5d30 2024-03-12T06:03:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.940+0000 7fcc9ac25740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079c4214c0 con 0x56079c4b5380 2024-03-12T06:03:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.940+0000 7fcc9ac25740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079c41cca0 con 0x56079c4be8b0 2024-03-12T06:03:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.940+0000 7fcc955e9640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcc7c0a1c20 0x7fcc7c0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.940+0000 7fcc86ffd640 1 -- 172.21.15.76:0/3970857456 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fcc880f04a0 con 0x7fcc901054a0 2024-03-12T06:03:46.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.940+0000 7fcc955e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5d30 0x56079c4be2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.940+0000 7fcc955e9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5d30 0x56079c4be2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34846/0 (socket says 172.21.15.76:34846) 2024-03-12T06:03:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.940+0000 7fcc955e9640 1 -- 172.21.15.76:0/1017109955 learned_addr learned my addr 172.21.15.76:0/1017109955 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.941+0000 7fcc95dea640 1 --2- 172.21.15.76:0/1017109955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5380 0x56079c4b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.941+0000 7fcc965eb640 1 --2- 172.21.15.76:0/1017109955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 0x56079c4c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.941+0000 7fcc955e9640 1 -- 172.21.15.76:0/1017109955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 msgr2=0x56079c4c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.941+0000 7fcc955e9640 1 --2- 172.21.15.76:0/1017109955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 0x56079c4c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.941+0000 7fcc955e9640 1 -- 172.21.15.76:0/1017109955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5380 msgr2=0x56079c4b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.941+0000 7fcc955e9640 1 --2- 172.21.15.76:0/1017109955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5380 0x56079c4b5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.941+0000 7fcc955e9640 1 -- 172.21.15.76:0/1017109955 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56079c3df0b0 con 0x56079c4b5d30 2024-03-12T06:03:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.941+0000 7fcc95dea640 1 --2- 172.21.15.76:0/1017109955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5380 0x56079c4b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.941+0000 7fcc955e9640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcc7c0a1c20 0x7fcc7c0a40e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcc800149a0 tx=0x7fcc8000bb60 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.941+0000 7fcc955e9640 1 --2- 172.21.15.76:0/1017109955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5d30 0x56079c4be2e0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fcc800117b0 tx=0x7fcc8001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=328e3fb767ba08b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.941+0000 7fcc7bfff640 1 -- 172.21.15.76:0/1017109955 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80011ee0 con 0x56079c4b5d30 2024-03-12T06:03:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.941+0000 7fcc7bfff640 1 -- 172.21.15.76:0/1017109955 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc8001a040 con 0x56079c4b5d30 2024-03-12T06:03:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.941+0000 7fcc7bfff640 1 -- 172.21.15.76:0/1017109955 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80005020 con 0x56079c4b5d30 2024-03-12T06:03:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.942+0000 7fcc9ac25740 1 -- 172.21.15.76:0/1017109955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5d30 msgr2=0x56079c4be2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.942+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/1017109955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5d30 0x56079c4be2e0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fcc800117b0 tx=0x7fcc8001b000 comp rx=0 tx=0).stop 2024-03-12T06:03:46.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.942+0000 7fcc9ac25740 1 -- 172.21.15.76:0/1017109955 shutdown_connections 2024-03-12T06:03:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.942+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/1017109955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 0x56079c4c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.942+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/1017109955 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5d30 0x56079c4be2e0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.942+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/1017109955 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5380 0x56079c4b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.942+0000 7fcc9ac25740 1 -- 172.21.15.76:0/1017109955 >> 172.21.15.76:0/1017109955 conn(0x56079c4af840 msgr2=0x56079c4afc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.942+0000 7fcc9ac25740 1 -- 172.21.15.76:0/1017109955 shutdown_connections 2024-03-12T06:03:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.942+0000 7fcc9ac25740 1 -- 172.21.15.76:0/1017109955 wait complete. 2024-03-12T06:03:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.942+0000 7fcc9ac25740 1 -- start start 2024-03-12T06:03:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.942+0000 7fcc9ac25740 1 Processor -- start 2024-03-12T06:03:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc9ac25740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5380 0x56079c4bb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc9ac25740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4b5d30 0x56079c4bbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc9ac25740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4be8b0 0x56079c4bc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc9ac25740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079c3c1f40 con 0x56079c4be8b0 2024-03-12T06:03:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc9ac25740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079c0f04b0 con 0x56079c4b5380 2024-03-12T06:03:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc9ac25740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079c3a2670 con 0x56079c4b5d30 2024-03-12T06:03:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc9ac25740 10 client.0 _ll_register_callbacks cb 0x56079c40ff70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc965eb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4be8b0 0x56079c4bc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc965eb640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4be8b0 0x56079c4bc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34856/0 (socket says 172.21.15.76:34856) 2024-03-12T06:03:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc965eb640 1 -- 172.21.15.76:0/3745941849 learned_addr learned my addr 172.21.15.76:0/3745941849 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc955e9640 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4b5d30 0x56079c4bbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc95dea640 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5380 0x56079c4bb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc965eb640 1 -- 172.21.15.76:0/3745941849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4b5d30 msgr2=0x56079c4bbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc965eb640 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4b5d30 0x56079c4bbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc965eb640 1 -- 172.21.15.76:0/3745941849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5380 msgr2=0x56079c4bb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc965eb640 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5380 0x56079c4bb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc965eb640 1 -- 172.21.15.76:0/3745941849 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc800119e0 con 0x56079c4be8b0 2024-03-12T06:03:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc95dea640 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5380 0x56079c4bb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc955e9640 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4b5d30 0x56079c4bbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc965eb640 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4be8b0 0x56079c4bc2f0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fcc880eddb0 tx=0x7fcc880bcc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de024c9759df1e82 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc797fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.943+0000 7fcc797fa640 1 -- 172.21.15.76:0/3745941849 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc880c1150 con 0x56079c4be8b0 2024-03-12T06:03:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.944+0000 7fcc797fa640 1 -- 172.21.15.76:0/3745941849 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fcc880bd1e0 con 0x56079c4be8b0 2024-03-12T06:03:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.944+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3745941849 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56079c3f6f40 con 0x56079c4be8b0 2024-03-12T06:03:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.944+0000 7fcc797fa640 1 -- 172.21.15.76:0/3745941849 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc880c1330 con 0x56079c4be8b0 2024-03-12T06:03:46.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.944+0000 7fcc9ac25740 10 client.17322 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:46.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.944+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3745941849 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56079c41b320 con 0x56079c4be8b0 2024-03-12T06:03:46.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.944+0000 7fcc797fa640 1 -- 172.21.15.76:0/3745941849 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56079c3f6f40 con 0x56079c4be8b0 2024-03-12T06:03:46.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.944+0000 7fcc797fa640 1 -- 172.21.15.76:0/3745941849 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcc880bd440 con 0x56079c4be8b0 2024-03-12T06:03:46.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.944+0000 7fcc9ac25740 10 client.17322 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:46.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.944+0000 7fcc9ac25740 10 client.17322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:46.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.944+0000 7fcc9ac25740 20 client.17322 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:46.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.944+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56079c4ceb60 0x56079c4d0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.962 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.944+0000 7fcc9ac25740 4 client.17322 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:03:46.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.944+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3745941849 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56079c413310 con 0x56079c4ceb60 2024-03-12T06:03:46.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.944+0000 7fcc95dea640 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56079c4ceb60 0x56079c4d0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.945+0000 7fcc95dea640 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56079c4ceb60 0x56079c4d0f80 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf4b8aa373ca8f2c server_cookie=430961702d93c5bc in_seq=0 out_seq=0 2024-03-12T06:03:46.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.945+0000 7fcc797fa640 10 client.17322 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:46.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.945+0000 7fcc797fa640 1 -- 172.21.15.76:0/3745941849 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56079c413310 con 0x56079c4ceb60 2024-03-12T06:03:46.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.945+0000 7fcc797fa640 10 client.17322 handle_command_reply: tid=0 2024-03-12T06:03:46.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.945+0000 7fcc9ac25740 1 client.17322 shutdown 2024-03-12T06:03:46.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.946+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3745941849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4be8b0 msgr2=0x56079c4bc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:46.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.946+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4be8b0 0x56079c4bc2f0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fcc880eddb0 tx=0x7fcc880bcc00 comp rx=0 tx=0).stop 2024-03-12T06:03:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.946+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3745941849 shutdown_connections 2024-03-12T06:03:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.946+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56079c4ceb60 0x56079c4d0f80 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.946+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4be8b0 0x56079c4bc2f0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.946+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4b5d30 0x56079c4bbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.946+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/3745941849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5380 0x56079c4bb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.946+0000 7fcc797fa640 0 client.17322 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.946+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3745941849 >> 172.21.15.76:0/3745941849 conn(0x56079c4af840 msgr2=0x56079c4b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.946+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3745941849 shutdown_connections 2024-03-12T06:03:46.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.946+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3745941849 wait complete. 2024-03-12T06:03:46.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.946+0000 7fcc9ac25740 20 client.17322 trim_cache size 0 max 16384 2024-03-12T06:03:46.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc9ac25740 1 Processor -- start 2024-03-12T06:03:46.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc9ac25740 1 -- start start 2024-03-12T06:03:46.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc9ac25740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5380 0x56079c4bc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc9ac25740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5d30 0x56079c4ae760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc9ac25740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 0x56079c4c5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc9ac25740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079c413310 con 0x56079c4b5380 2024-03-12T06:03:46.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc9ac25740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079c3f6f40 con 0x56079c4b5d30 2024-03-12T06:03:46.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc9ac25740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079c4c1700 con 0x56079c4be8b0 2024-03-12T06:03:46.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc95dea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5380 0x56079c4bc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc95dea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5380 0x56079c4bc110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34868/0 (socket says 172.21.15.76:34868) 2024-03-12T06:03:46.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc95dea640 1 -- 172.21.15.76:0/1535627373 learned_addr learned my addr 172.21.15.76:0/1535627373 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:46.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc965eb640 1 --2- 172.21.15.76:0/1535627373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 0x56079c4c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc955e9640 1 --2- 172.21.15.76:0/1535627373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5d30 0x56079c4ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc95dea640 1 -- 172.21.15.76:0/1535627373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 msgr2=0x56079c4c5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:46.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc95dea640 1 --2- 172.21.15.76:0/1535627373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 0x56079c4c5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc95dea640 1 -- 172.21.15.76:0/1535627373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5d30 msgr2=0x56079c4ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:46.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc95dea640 1 --2- 172.21.15.76:0/1535627373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5d30 0x56079c4ae760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc95dea640 1 -- 172.21.15.76:0/1535627373 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56079c3b7c00 con 0x56079c4b5380 2024-03-12T06:03:46.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc965eb640 1 --2- 172.21.15.76:0/1535627373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 0x56079c4c5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:46.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.961+0000 7fcc95dea640 1 --2- 172.21.15.76:0/1535627373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5380 0x56079c4bc110 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c009c80 tx=0x7fcc8c014840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e6bab0e83b93bd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:46.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.962+0000 7fcc7b7fe640 1 -- 172.21.15.76:0/1535627373 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c017040 con 0x56079c4b5380 2024-03-12T06:03:46.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.962+0000 7fcc7b7fe640 1 -- 172.21.15.76:0/1535627373 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc8c021070 con 0x56079c4b5380 2024-03-12T06:03:46.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.962+0000 7fcc7b7fe640 1 -- 172.21.15.76:0/1535627373 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c003850 con 0x56079c4b5380 2024-03-12T06:03:46.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.962+0000 7fcc9ac25740 1 -- 172.21.15.76:0/1535627373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5380 msgr2=0x56079c4bc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:46.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.962+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/1535627373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5380 0x56079c4bc110 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c009c80 tx=0x7fcc8c014840 comp rx=0 tx=0).stop 2024-03-12T06:03:46.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.962+0000 7fcc9ac25740 1 -- 172.21.15.76:0/1535627373 shutdown_connections 2024-03-12T06:03:46.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.962+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/1535627373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 0x56079c4c5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.962+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/1535627373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5d30 0x56079c4ae760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.962+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/1535627373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5380 0x56079c4bc110 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.962+0000 7fcc9ac25740 1 -- 172.21.15.76:0/1535627373 >> 172.21.15.76:0/1535627373 conn(0x56079c4af840 msgr2=0x56079c4ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:46.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.962+0000 7fcc9ac25740 1 -- 172.21.15.76:0/1535627373 shutdown_connections 2024-03-12T06:03:46.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.962+0000 7fcc9ac25740 1 -- 172.21.15.76:0/1535627373 wait complete. 2024-03-12T06:03:46.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc9ac25740 1 -- start start 2024-03-12T06:03:46.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc9ac25740 1 Processor -- start 2024-03-12T06:03:46.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc9ac25740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5380 0x56079c4c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc9ac25740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5d30 0x56079c4c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc9ac25740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 0x56079c4c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc9ac25740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079c525150 con 0x56079c4b5380 2024-03-12T06:03:46.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc9ac25740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079c5252c0 con 0x56079c4b5d30 2024-03-12T06:03:46.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc9ac25740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56079c525480 con 0x56079c4be8b0 2024-03-12T06:03:46.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc9ac25740 10 client.0 _ll_register_callbacks cb 0x7fcc800144b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:46.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc95dea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5380 0x56079c4c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc95dea640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5380 0x56079c4c44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34878/0 (socket says 172.21.15.76:34878) 2024-03-12T06:03:46.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc95dea640 1 -- 172.21.15.76:0/3035565936 learned_addr learned my addr 172.21.15.76:0/3035565936 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:46.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc965eb640 1 --2- 172.21.15.76:0/3035565936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 0x56079c4c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc955e9640 1 --2- 172.21.15.76:0/3035565936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5d30 0x56079c4c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc95dea640 1 -- 172.21.15.76:0/3035565936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 msgr2=0x56079c4c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:46.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc95dea640 1 --2- 172.21.15.76:0/3035565936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 0x56079c4c4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc95dea640 1 -- 172.21.15.76:0/3035565936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5d30 msgr2=0x56079c4c4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:46.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc95dea640 1 --2- 172.21.15.76:0/3035565936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5d30 0x56079c4c4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.963+0000 7fcc95dea640 1 -- 172.21.15.76:0/3035565936 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8c009980 con 0x56079c4b5380 2024-03-12T06:03:46.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc95dea640 1 --2- 172.21.15.76:0/3035565936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5380 0x56079c4c44f0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c0142e0 tx=0x7fcc8c014310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f412ea6dbb71da4b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:46.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc797fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:03:46.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc797fa640 1 -- 172.21.15.76:0/3035565936 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c017050 con 0x56079c4b5380 2024-03-12T06:03:46.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc797fa640 1 -- 172.21.15.76:0/3035565936 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fcc8c004350 con 0x56079c4b5380 2024-03-12T06:03:46.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3035565936 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56079c442900 con 0x56079c4b5380 2024-03-12T06:03:46.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc797fa640 1 -- 172.21.15.76:0/3035565936 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc8c004510 con 0x56079c4b5380 2024-03-12T06:03:46.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc9ac25740 10 client.17328 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:46.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3035565936 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56079c3dd910 con 0x56079c4b5380 2024-03-12T06:03:46.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc797fa640 1 -- 172.21.15.76:0/3035565936 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc8c03b080 con 0x56079c4b5380 2024-03-12T06:03:46.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc797fa640 1 -- 172.21.15.76:0/3035565936 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcc8c035030 con 0x56079c4b5380 2024-03-12T06:03:46.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc9ac25740 10 client.17328 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:46.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc9ac25740 10 client.17328 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:46.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc9ac25740 20 client.17328 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:46.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/3035565936 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56079c526c60 0x56079c529080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:46.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc9ac25740 4 client.17328 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T06:03:46.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.964+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3035565936 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56079c11c860 con 0x56079c526c60 2024-03-12T06:03:46.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.965+0000 7fcc955e9640 1 --2- 172.21.15.76:0/3035565936 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56079c526c60 0x56079c529080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:46.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.965+0000 7fcc797fa640 10 client.17328 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:46.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.965+0000 7fcc955e9640 1 --2- 172.21.15.76:0/3035565936 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56079c526c60 0x56079c529080 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=907ab4ce8e004eef server_cookie=bb916ca1a2a31a6b in_seq=0 out_seq=0 2024-03-12T06:03:46.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.966+0000 7fcc797fa640 1 -- 172.21.15.76:0/3035565936 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56079c11c860 con 0x56079c526c60 2024-03-12T06:03:46.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.966+0000 7fcc797fa640 10 client.17328 handle_command_reply: tid=0 2024-03-12T06:03:46.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.966+0000 7fcc9ac25740 1 client.17328 shutdown 2024-03-12T06:03:46.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.966+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3035565936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5380 msgr2=0x56079c4c44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:46.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.966+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/3035565936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5380 0x56079c4c44f0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c0142e0 tx=0x7fcc8c014310 comp rx=0 tx=0).stop 2024-03-12T06:03:46.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3035565936 shutdown_connections 2024-03-12T06:03:46.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/3035565936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56079c4be8b0 0x56079c4c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/3035565936 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56079c4b5d30 0x56079c4c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/3035565936 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56079c526c60 0x56079c529080 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9ac25740 1 --2- 172.21.15.76:0/3035565936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56079c4b5380 0x56079c4c44f0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc797fa640 0 client.17328 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:46.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3035565936 >> 172.21.15.76:0/3035565936 conn(0x56079c4af840 msgr2=0x56079c4ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:46.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3035565936 shutdown_connections 2024-03-12T06:03:46.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9ac25740 1 -- 172.21.15.76:0/3035565936 wait complete. 2024-03-12T06:03:46.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9ac25740 20 client.17328 trim_cache size 0 max 16384 2024-03-12T06:03:46.981 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:03:46.981 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T06:03:46.981 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "af51ee4f-ac53-44b7-94cc-5317a745152b", 2024-03-12T06:03:46.981 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T06:03:46.982 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:03:46.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9804e640 1 -- 172.21.15.76:0/3970857456 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcc7c0a1c20 msgr2=0x7fcc7c0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:46.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9804e640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcc7c0a1c20 0x7fcc7c0a40e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcc800149a0 tx=0x7fcc8000bb60 comp rx=0 tx=0).stop 2024-03-12T06:03:46.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9804e640 1 -- 172.21.15.76:0/3970857456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc901054a0 msgr2=0x7fcc901a2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:46.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9804e640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc901054a0 0x7fcc901a2790 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fcc88005cc0 tx=0x7fcc88034a90 comp rx=0 tx=0).stop 2024-03-12T06:03:46.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9804e640 1 -- 172.21.15.76:0/3970857456 shutdown_connections 2024-03-12T06:03:46.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9804e640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fcc7c0a1c20 0x7fcc7c0a40e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9804e640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc901054a0 0x7fcc901a2790 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9804e640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc90100830 0x7fcc9019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9804e640 1 --2- 172.21.15.76:0/3970857456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc900fff10 0x7fcc9019ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:46.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9804e640 1 -- 172.21.15.76:0/3970857456 >> 172.21.15.76:0/3970857456 conn(0x7fcc90077260 msgr2=0x7fcc90102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:46.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9804e640 1 -- 172.21.15.76:0/3970857456 shutdown_connections 2024-03-12T06:03:46.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:46.967+0000 7fcc9804e640 1 -- 172.21.15.76:0/3970857456 wait complete. 2024-03-12T06:03:46.984 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:03:46.985 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:03:47.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.105+0000 7f5cc88c0640 1 -- 172.21.15.76:0/3124411215 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc4100780 msgr2=0x7f5cc4100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:47.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.105+0000 7f5cc88c0640 1 --2- 172.21.15.76:0/3124411215 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc4100780 0x7f5cc4100be0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009900 tx=0x7f5cb402bc80 comp rx=0 tx=0).stop 2024-03-12T06:03:47.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.105+0000 7f5cc88c0640 1 -- 172.21.15.76:0/3124411215 shutdown_connections 2024-03-12T06:03:47.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.105+0000 7f5cc88c0640 1 --2- 172.21.15.76:0/3124411215 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cc4101120 0x7f5cc410ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.105+0000 7f5cc88c0640 1 --2- 172.21.15.76:0/3124411215 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc4100780 0x7f5cc4100be0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.105+0000 7f5cc88c0640 1 --2- 172.21.15.76:0/3124411215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cc4104fe0 0x7f5cc41053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.105+0000 7f5cc88c0640 1 -- 172.21.15.76:0/3124411215 >> 172.21.15.76:0/3124411215 conn(0x7f5cc40775d0 msgr2=0x7f5cc40fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:47.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.105+0000 7f5cc88c0640 1 -- 172.21.15.76:0/3124411215 shutdown_connections 2024-03-12T06:03:47.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.105+0000 7f5cc88c0640 1 -- 172.21.15.76:0/3124411215 wait complete. 2024-03-12T06:03:47.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.106+0000 7f5cc88c0640 1 Processor -- start 2024-03-12T06:03:47.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.106+0000 7f5cc88c0640 1 -- start start 2024-03-12T06:03:47.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.106+0000 7f5cc88c0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cc4100780 0x7f5cc4108650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.106+0000 7f5cc88c0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cc4101120 0x7f5cc4108b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.106+0000 7f5cc88c0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc4104fe0 0x7f5cc41a9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.106+0000 7f5cc88c0640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc41092f0 con 0x7f5cc4104fe0 2024-03-12T06:03:47.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.106+0000 7f5cc88c0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc41aa100 con 0x7f5cc4101120 2024-03-12T06:03:47.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.106+0000 7f5cc88c0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc41aa2c0 con 0x7f5cc4100780 2024-03-12T06:03:47.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.107+0000 7f5cc1d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cc4101120 0x7f5cc4108b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.107+0000 7f5cc1d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cc4101120 0x7f5cc4108b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:57270/0 (socket says 172.21.15.76:57270) 2024-03-12T06:03:47.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.107+0000 7f5cc1d9b640 1 -- 172.21.15.76:0/3249216351 learned_addr learned my addr 172.21.15.76:0/3249216351 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:47.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.107+0000 7f5cc259c640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cc4100780 0x7f5cc4108650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.107+0000 7f5cc2d9d640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc4104fe0 0x7f5cc41a9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.107+0000 7f5cc1d9b640 1 -- 172.21.15.76:0/3249216351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cc4100780 msgr2=0x7f5cc4108650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:47.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.107+0000 7f5cc1d9b640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cc4100780 0x7f5cc4108650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.107+0000 7f5cc1d9b640 1 -- 172.21.15.76:0/3249216351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc4104fe0 msgr2=0x7f5cc41a9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:47.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.107+0000 7f5cc1d9b640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc4104fe0 0x7f5cc41a9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.107+0000 7f5cc1d9b640 1 -- 172.21.15.76:0/3249216351 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb4009560 con 0x7f5cc4101120 2024-03-12T06:03:47.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.107+0000 7f5cc2d9d640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc4104fe0 0x7f5cc41a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:47.115 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.107+0000 7f5cc1d9b640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cc4101120 0x7f5cc4108b90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4005940 tx=0x7f5cb402c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:47.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.107+0000 7f5cb37fe640 1 -- 172.21.15.76:0/3249216351 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb40042d0 con 0x7f5cc4101120 2024-03-12T06:03:47.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.108+0000 7f5cb37fe640 1 -- 172.21.15.76:0/3249216351 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5cb4004430 con 0x7f5cc4101120 2024-03-12T06:03:47.117 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.108+0000 7f5cb37fe640 1 -- 172.21.15.76:0/3249216351 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4003c50 con 0x7f5cc4101120 2024-03-12T06:03:47.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.108+0000 7f5cc88c0640 1 -- 172.21.15.76:0/3249216351 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc41aa540 con 0x7f5cc4101120 2024-03-12T06:03:47.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.108+0000 7f5cc88c0640 1 -- 172.21.15.76:0/3249216351 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc41aaa00 con 0x7f5cc4101120 2024-03-12T06:03:47.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.110+0000 7f5ccb56c740 1 Processor -- start 2024-03-12T06:03:47.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5ccb56c740 1 -- start start 2024-03-12T06:03:47.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5ccb56c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da32b0 0x558368da3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5ccb56c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da3bd0 0x558368dac270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.118 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5ccb56c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368dac7b0 0x558368daeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5ccb56c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558368d12280 con 0x558368da32b0 2024-03-12T06:03:47.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5ccb56c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558368d0f7b0 con 0x558368da3bd0 2024-03-12T06:03:47.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5ccb56c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558368d0af90 con 0x558368dac7b0 2024-03-12T06:03:47.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5cb37fe640 1 -- 172.21.15.76:0/3249216351 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5cb40054f0 con 0x7f5cc4101120 2024-03-12T06:03:47.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5cc259c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da3bd0 0x558368dac270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5cc2d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da32b0 0x558368da3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5cc259c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da3bd0 0x558368dac270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:57272/0 (socket says 172.21.15.76:57272) 2024-03-12T06:03:47.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5cc259c640 1 -- 172.21.15.76:0/1893128529 learned_addr learned my addr 172.21.15.76:0/1893128529 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:47.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5cc2d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da32b0 0x558368da3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34902/0 (socket says 172.21.15.76:34902) 2024-03-12T06:03:47.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5cc1d9b640 1 --2- 172.21.15.76:0/1893128529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368dac7b0 0x558368daeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5cc259c640 1 -- 172.21.15.76:0/1893128529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368dac7b0 msgr2=0x558368daeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:47.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5cc259c640 1 --2- 172.21.15.76:0/1893128529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368dac7b0 0x558368daeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.111+0000 7f5cc259c640 1 -- 172.21.15.76:0/1893128529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da32b0 msgr2=0x558368da3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:47.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.112+0000 7f5cc259c640 1 --2- 172.21.15.76:0/1893128529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da32b0 0x558368da3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.112+0000 7f5cc259c640 1 -- 172.21.15.76:0/1893128529 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558368cc9570 con 0x558368da3bd0 2024-03-12T06:03:47.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.112+0000 7f5cc2d9d640 1 --2- 172.21.15.76:0/1893128529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da32b0 0x558368da3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:03:47.120 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.112+0000 7f5cc259c640 1 --2- 172.21.15.76:0/1893128529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da3bd0 0x558368dac270 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8009d20 tx=0x7f5cb8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aeed64b40b29f8a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:47.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.112+0000 7f5cb37fe640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5cac0a1cf0 0x7f5cac0a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.112+0000 7f5cb0ff9640 1 -- 172.21.15.76:0/1893128529 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8019630 con 0x558368da3bd0 2024-03-12T06:03:47.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.112+0000 7f5cb0ff9640 1 -- 172.21.15.76:0/1893128529 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5cb8019c70 con 0x558368da3bd0 2024-03-12T06:03:47.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.112+0000 7f5cb37fe640 1 -- 172.21.15.76:0/3249216351 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5cb40ef6e0 con 0x7f5cc4101120 2024-03-12T06:03:47.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.112+0000 7f5cb0ff9640 1 -- 172.21.15.76:0/1893128529 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb800ec70 con 0x558368da3bd0 2024-03-12T06:03:47.121 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.112+0000 7f5cc2d9d640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5cac0a1cf0 0x7f5cac0a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.113+0000 7f5ccb56c740 1 -- 172.21.15.76:0/1893128529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da3bd0 msgr2=0x558368dac270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:47.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.113+0000 7f5cc2d9d640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5cac0a1cf0 0x7f5cac0a41b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc005fd0 tx=0x7f5cbc009500 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:47.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.113+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/1893128529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da3bd0 0x558368dac270 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8009d20 tx=0x7f5cb8011810 comp rx=0 tx=0).stop 2024-03-12T06:03:47.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.113+0000 7f5ccb56c740 1 -- 172.21.15.76:0/1893128529 shutdown_connections 2024-03-12T06:03:47.125 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.113+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/1893128529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368dac7b0 0x558368daeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.113+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/1893128529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da3bd0 0x558368dac270 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.113+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/1893128529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da32b0 0x558368da3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.113+0000 7f5ccb56c740 1 -- 172.21.15.76:0/1893128529 >> 172.21.15.76:0/1893128529 conn(0x558368d9d990 msgr2=0x558368d9dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:47.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.113+0000 7f5ccb56c740 1 -- 172.21.15.76:0/1893128529 shutdown_connections 2024-03-12T06:03:47.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.113+0000 7f5ccb56c740 1 -- 172.21.15.76:0/1893128529 wait complete. 2024-03-12T06:03:47.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.114+0000 7f5ccb56c740 1 -- start start 2024-03-12T06:03:47.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.114+0000 7f5ccb56c740 1 Processor -- start 2024-03-12T06:03:47.126 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.114+0000 7f5ccb56c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368da32b0 0x558368da9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.114+0000 7f5ccb56c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da3bd0 0x558368da9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.114+0000 7f5ccb56c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368dac7b0 0x558368daa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.114+0000 7f5ccb56c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583689de430 con 0x558368dac7b0 2024-03-12T06:03:47.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.114+0000 7f5ccb56c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55836899ae00 con 0x558368da3bd0 2024-03-12T06:03:47.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.114+0000 7f5ccb56c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558368c8ffd0 con 0x558368da32b0 2024-03-12T06:03:47.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5cc1d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da3bd0 0x558368da9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5cc1d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da3bd0 0x558368da9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:57280/0 (socket says 172.21.15.76:57280) 2024-03-12T06:03:47.127 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5cc1d9b640 1 -- 172.21.15.76:0/3438237448 learned_addr learned my addr 172.21.15.76:0/3438237448 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:47.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5cc259c640 1 --2- 172.21.15.76:0/3438237448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368da32b0 0x558368da9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5cc2d9d640 1 --2- 172.21.15.76:0/3438237448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368dac7b0 0x558368daa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5ccb56c740 10 client.0 _ll_register_callbacks cb 0x558368ce5fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:47.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5cc1d9b640 1 -- 172.21.15.76:0/3438237448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368da32b0 msgr2=0x558368da9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:47.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5cc1d9b640 1 --2- 172.21.15.76:0/3438237448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368da32b0 0x558368da9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5cc1d9b640 1 -- 172.21.15.76:0/3438237448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368dac7b0 msgr2=0x558368daa0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:47.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5cc1d9b640 1 --2- 172.21.15.76:0/3438237448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368dac7b0 0x558368daa0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5cc1d9b640 1 -- 172.21.15.76:0/3438237448 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb8009950 con 0x558368da3bd0 2024-03-12T06:03:47.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5cc1d9b640 1 --2- 172.21.15.76:0/3438237448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da3bd0 0x558368da9b60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4000c00 tx=0x7f5cb40b8660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88ad01ef8a027578 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:47.128 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5ca9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T06:03:47.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5ca9ffb640 1 -- 172.21.15.76:0/3438237448 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb40ed250 con 0x558368da3bd0 2024-03-12T06:03:47.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3438237448 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558368a0a7e0 con 0x558368da3bd0 2024-03-12T06:03:47.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.115+0000 7f5ca9ffb640 1 -- 172.21.15.76:0/3438237448 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5cb40bcac0 con 0x558368da3bd0 2024-03-12T06:03:47.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.116+0000 7f5ca9ffb640 1 -- 172.21.15.76:0/3438237448 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5cb40bbd20 con 0x558368da3bd0 2024-03-12T06:03:47.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.116+0000 7f5ccb56c740 10 client.26426 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:47.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.116+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3438237448 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558368d09610 con 0x558368da3bd0 2024-03-12T06:03:47.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.116+0000 7f5ca9ffb640 1 -- 172.21.15.76:0/3438237448 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558368a0a7e0 con 0x558368da3bd0 2024-03-12T06:03:47.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.116+0000 7f5ca9ffb640 1 -- 172.21.15.76:0/3438237448 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5cb403c070 con 0x558368da3bd0 2024-03-12T06:03:47.129 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.116+0000 7f5ccb56c740 10 client.26426 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:47.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.116+0000 7f5ccb56c740 10 client.26426 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:47.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.116+0000 7f5ccb56c740 20 client.26426 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:47.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.116+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/3438237448 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558368dbc7e0 0x558368dbebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.117+0000 7f5ccb56c740 4 client.26426 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:03:47.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.117+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3438237448 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558368a47720 con 0x558368dbc7e0 2024-03-12T06:03:47.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.117+0000 7f5cc259c640 1 --2- 172.21.15.76:0/3438237448 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558368dbc7e0 0x558368dbebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.117+0000 7f5ca9ffb640 10 client.26426 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:47.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.117+0000 7f5cc259c640 1 --2- 172.21.15.76:0/3438237448 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558368dbc7e0 0x558368dbebd0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=476ce898608253cc server_cookie=6ba8d963060e364b in_seq=0 out_seq=0 2024-03-12T06:03:47.130 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ca9ffb640 1 -- 172.21.15.76:0/3438237448 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558368a47720 con 0x558368dbc7e0 2024-03-12T06:03:47.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ca9ffb640 10 client.26426 handle_command_reply: tid=0 2024-03-12T06:03:47.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ccb56c740 1 client.26426 shutdown 2024-03-12T06:03:47.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3438237448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da3bd0 msgr2=0x558368da9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:47.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/3438237448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da3bd0 0x558368da9b60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4000c00 tx=0x7f5cb40b8660 comp rx=0 tx=0).stop 2024-03-12T06:03:47.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3438237448 shutdown_connections 2024-03-12T06:03:47.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/3438237448 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558368dbc7e0 0x558368dbebd0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/3438237448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368dac7b0 0x558368daa0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/3438237448 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da3bd0 0x558368da9b60 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.131 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ca9ffb640 0 client.26426 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:47.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/3438237448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368da32b0 0x558368da9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3438237448 >> 172.21.15.76:0/3438237448 conn(0x558368d9d990 msgr2=0x558368d9f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:47.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3438237448 shutdown_connections 2024-03-12T06:03:47.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3438237448 wait complete. 2024-03-12T06:03:47.132 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.118+0000 7f5ccb56c740 20 client.26426 trim_cache size 0 max 16384 2024-03-12T06:03:47.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.139+0000 7f5ccb56c740 1 Processor -- start 2024-03-12T06:03:47.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.139+0000 7f5ccb56c740 1 -- start start 2024-03-12T06:03:47.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.139+0000 7f5ccb56c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368da32b0 0x558368dabce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.139+0000 7f5ccb56c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da3bd0 0x558368da8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.139+0000 7f5ccb56c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368dac7b0 0x558368da6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.139+0000 7f5ccb56c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558368a0a7e0 con 0x558368da3bd0 2024-03-12T06:03:47.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.139+0000 7f5ccb56c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558368d990f0 con 0x558368dac7b0 2024-03-12T06:03:47.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.139+0000 7f5ccb56c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558368dbc5e0 con 0x558368da32b0 2024-03-12T06:03:47.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.139+0000 7f5cc1d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da3bd0 0x558368da8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.139+0000 7f5cc2d9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368dac7b0 0x558368da6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.139+0000 7f5cc1d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da3bd0 0x558368da8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:34920/0 (socket says 172.21.15.76:34920) 2024-03-12T06:03:47.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.139+0000 7f5cc259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368da32b0 0x558368dabce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.145 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.139+0000 7f5cc1d9b640 1 -- 172.21.15.76:0/2936491170 learned_addr learned my addr 172.21.15.76:0/2936491170 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:47.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.140+0000 7f5cc1d9b640 1 -- 172.21.15.76:0/2936491170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368da32b0 msgr2=0x558368dabce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:47.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.140+0000 7f5cc1d9b640 1 --2- 172.21.15.76:0/2936491170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368da32b0 0x558368dabce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.140+0000 7f5cc1d9b640 1 -- 172.21.15.76:0/2936491170 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368dac7b0 msgr2=0x558368da6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:47.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.140+0000 7f5cc1d9b640 1 --2- 172.21.15.76:0/2936491170 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368dac7b0 0x558368da6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.140+0000 7f5cc1d9b640 1 -- 172.21.15.76:0/2936491170 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558368ca5ac0 con 0x558368da3bd0 2024-03-12T06:03:47.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.140+0000 7f5cc1d9b640 1 --2- 172.21.15.76:0/2936491170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da3bd0 0x558368da8600 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40036a0 tx=0x7f5cb40ba8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18719dfcb4b07cdc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:47.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.140+0000 7f5cabfff640 1 -- 172.21.15.76:0/2936491170 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb40bbd20 con 0x558368da3bd0 2024-03-12T06:03:47.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.140+0000 7f5cabfff640 1 -- 172.21.15.76:0/2936491170 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5cb40bc190 con 0x558368da3bd0 2024-03-12T06:03:47.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.140+0000 7f5cabfff640 1 -- 172.21.15.76:0/2936491170 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb40c4bf0 con 0x558368da3bd0 2024-03-12T06:03:47.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.141+0000 7f5ccb56c740 1 -- 172.21.15.76:0/2936491170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da3bd0 msgr2=0x558368da8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:47.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.141+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/2936491170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da3bd0 0x558368da8600 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40036a0 tx=0x7f5cb40ba8d0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.141+0000 7f5ccb56c740 1 -- 172.21.15.76:0/2936491170 shutdown_connections 2024-03-12T06:03:47.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.141+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/2936491170 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368dac7b0 0x558368da6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.141+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/2936491170 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da3bd0 0x558368da8600 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.141+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/2936491170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368da32b0 0x558368dabce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.141+0000 7f5ccb56c740 1 -- 172.21.15.76:0/2936491170 >> 172.21.15.76:0/2936491170 conn(0x558368d9d990 msgr2=0x558368da7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:47.150 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5ccb56c740 1 -- 172.21.15.76:0/2936491170 shutdown_connections 2024-03-12T06:03:47.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5ccb56c740 1 -- 172.21.15.76:0/2936491170 wait complete. 2024-03-12T06:03:47.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5ccb56c740 1 -- start start 2024-03-12T06:03:47.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5ccb56c740 1 Processor -- start 2024-03-12T06:03:47.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5ccb56c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da32b0 0x558368db2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5ccb56c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da3bd0 0x558368db28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5ccb56c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368dac7b0 0x558368e12c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5ccb56c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x558368db3060 con 0x558368da3bd0 2024-03-12T06:03:47.151 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5ccb56c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x558368db31d0 con 0x558368da32b0 2024-03-12T06:03:47.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5ccb56c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x558368e131d0 con 0x558368dac7b0 2024-03-12T06:03:47.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5ccb56c740 10 client.0 _ll_register_callbacks cb 0x7f5cb8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:03:47.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5cc259c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da32b0 0x558368db2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5cc259c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da32b0 0x558368db2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:57304/0 (socket says 172.21.15.76:57304) 2024-03-12T06:03:47.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5cc259c640 1 -- 172.21.15.76:0/3453795384 learned_addr learned my addr 172.21.15.76:0/3453795384 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:03:47.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.142+0000 7f5cc1d9b640 1 --2- 172.21.15.76:0/3453795384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da3bd0 0x558368db28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5cc2d9d640 1 --2- 172.21.15.76:0/3453795384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368dac7b0 0x558368e12c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.152 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5cc259c640 1 -- 172.21.15.76:0/3453795384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368dac7b0 msgr2=0x558368e12c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:47.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5cc259c640 1 --2- 172.21.15.76:0/3453795384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368dac7b0 0x558368e12c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5cc259c640 1 -- 172.21.15.76:0/3453795384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da3bd0 msgr2=0x558368db28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:03:47.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5cc259c640 1 --2- 172.21.15.76:0/3453795384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da3bd0 0x558368db28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5cc259c640 1 -- 172.21.15.76:0/3453795384 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558368d09610 con 0x558368da32b0 2024-03-12T06:03:47.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5cc2d9d640 1 --2- 172.21.15.76:0/3453795384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368dac7b0 0x558368e12c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:03:47.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5cc259c640 1 --2- 172.21.15.76:0/3453795384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da32b0 0x558368db2390 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5cb801c040 tx=0x7f5cb800e910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6e805af79fe3238 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:03:47.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5ca9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T06:03:47.153 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5ca9ffb640 1 -- 172.21.15.76:0/3453795384 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb800eb80 con 0x558368da32b0 2024-03-12T06:03:47.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5ca9ffb640 1 -- 172.21.15.76:0/3453795384 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5cb800ece0 con 0x558368da32b0 2024-03-12T06:03:47.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3453795384 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5cb8019c70 con 0x558368da32b0 2024-03-12T06:03:47.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5ca9ffb640 1 -- 172.21.15.76:0/3453795384 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5cb8009ed0 con 0x558368da32b0 2024-03-12T06:03:47.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5ccb56c740 10 client.26432 fetch_fsmap learned FSMap version 28 2024-03-12T06:03:47.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.143+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3453795384 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558368ccb6a0 con 0x558368da32b0 2024-03-12T06:03:47.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.144+0000 7f5ca9ffb640 1 -- 172.21.15.76:0/3453795384 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5cb8019c70 con 0x558368da32b0 2024-03-12T06:03:47.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.144+0000 7f5ca9ffb640 1 -- 172.21.15.76:0/3453795384 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5cb80035f0 con 0x558368da32b0 2024-03-12T06:03:47.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.144+0000 7f5ccb56c740 10 client.26432 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:03:47.154 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.144+0000 7f5ccb56c740 10 client.26432 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:03:47.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.144+0000 7f5ccb56c740 20 client.26432 populate_metadata read hostname 'smithi076' 2024-03-12T06:03:47.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.144+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/3453795384 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558368e14910 0x558368e16d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:03:47.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.144+0000 7f5ccb56c740 4 client.26432 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:03:47.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.144+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3453795384 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558368a96890 con 0x558368e14910 2024-03-12T06:03:47.156 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.144+0000 7f5cc1d9b640 1 --2- 172.21.15.76:0/3453795384 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558368e14910 0x558368e16d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:03:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ca9ffb640 10 client.26432 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5cc1d9b640 1 --2- 172.21.15.76:0/3453795384 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558368e14910 0x558368e16d30 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9876b628d1c12e server_cookie=e419d18b25b46f4a in_seq=0 out_seq=0 2024-03-12T06:03:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ca9ffb640 1 -- 172.21.15.76:0/3453795384 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558368a96890 con 0x558368e14910 2024-03-12T06:03:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ca9ffb640 10 client.26432 handle_command_reply: tid=0 2024-03-12T06:03:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ccb56c740 1 client.26432 shutdown 2024-03-12T06:03:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3453795384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da32b0 msgr2=0x558368db2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/3453795384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da32b0 0x558368db2390 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5cb801c040 tx=0x7f5cb800e910 comp rx=0 tx=0).stop 2024-03-12T06:03:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3453795384 shutdown_connections 2024-03-12T06:03:47.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/3453795384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558368dac7b0 0x558368e12c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/3453795384 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x558368e14910 0x558368e16d30 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/3453795384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x558368da3bd0 0x558368db28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ccb56c740 1 --2- 172.21.15.76:0/3453795384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x558368da32b0 0x558368db2390 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3453795384 >> 172.21.15.76:0/3453795384 conn(0x558368d9d990 msgr2=0x558368da8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:47.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ca9ffb640 0 client.26432 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:03:47.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3453795384 shutdown_connections 2024-03-12T06:03:47.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ccb56c740 1 -- 172.21.15.76:0/3453795384 wait complete. 2024-03-12T06:03:47.158 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.145+0000 7f5ccb56c740 20 client.26432 trim_cache size 0 max 16384 2024-03-12T06:03:47.159 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:03:47.159 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T06:03:47.159 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T06:03:47.159 INFO:teuthology.orchestra.run.smithi076.stdout: "af51ee4f-ac53-44b7-94cc-5317a745152b": { 2024-03-12T06:03:47.159 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T06:03:47.159 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "af51ee4f-ac53-44b7-94cc-5317a745152b", 2024-03-12T06:03:47.160 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T06:03:47.160 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:03:47.160 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:03:47.160 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:03:47.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.146+0000 7f5cc88c0640 1 -- 172.21.15.76:0/3249216351 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5cac0a1cf0 msgr2=0x7f5cac0a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:47.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.146+0000 7f5cc88c0640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5cac0a1cf0 0x7f5cac0a41b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc005fd0 tx=0x7f5cbc009500 comp rx=0 tx=0).stop 2024-03-12T06:03:47.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.146+0000 7f5cc88c0640 1 -- 172.21.15.76:0/3249216351 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cc4101120 msgr2=0x7f5cc4108b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:03:47.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.146+0000 7f5cc88c0640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cc4101120 0x7f5cc4108b90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4005940 tx=0x7f5cb402c660 comp rx=0 tx=0).stop 2024-03-12T06:03:47.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.146+0000 7f5cc88c0640 1 -- 172.21.15.76:0/3249216351 shutdown_connections 2024-03-12T06:03:47.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.146+0000 7f5cc88c0640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5cac0a1cf0 0x7f5cac0a41b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.146+0000 7f5cc88c0640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5cc4104fe0 0x7f5cc41a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.161 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.146+0000 7f5cc88c0640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cc4101120 0x7f5cc4108b90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.146+0000 7f5cc88c0640 1 --2- 172.21.15.76:0/3249216351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cc4100780 0x7f5cc4108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:03:47.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.146+0000 7f5cc88c0640 1 -- 172.21.15.76:0/3249216351 >> 172.21.15.76:0/3249216351 conn(0x7f5cc40775d0 msgr2=0x7f5cc4101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:03:47.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.146+0000 7f5cc88c0640 1 -- 172.21.15.76:0/3249216351 shutdown_connections 2024-03-12T06:03:47.162 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:03:47.146+0000 7f5cc88c0640 1 -- 172.21.15.76:0/3249216351 wait complete. 2024-03-12T06:03:47.165 INFO:tasks.cephfs.filesystem:scrub status for tag:af51ee4f-ac53-44b7-94cc-5317a745152b - {'path': '/', 'tag': 'af51ee4f-ac53-44b7-94cc-5317a745152b', 'options': 'recursive,force'} 2024-03-12T06:03:47.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:47 smithi076 ceph-mon[32095]: pgmap v247: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:47.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:47 smithi082 ceph-mon[36382]: pgmap v247: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:47.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:47 smithi088 ceph-mon[36585]: pgmap v247: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:03:48.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:48 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T06:03:48.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:48 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:48.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:48 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:03:48.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:48 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:03:48.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:48 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:03:48.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:48 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:48.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:48 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:48.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:48 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:48.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:48 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T06:03:48.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:48 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:48.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:48 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:03:48.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:48 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:03:48.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:48 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:03:48.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:48 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:48.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:48 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:48.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:48 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:48.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:48 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T06:03:48.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:48 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:48.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:48 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:03:48.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:48 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:03:48.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:48 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:03:48.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:48 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:48.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:48 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:48.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:48 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:03:49.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:49 smithi082 ceph-mon[36382]: pgmap v248: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:03:49.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:49 smithi088 ceph-mon[36585]: pgmap v248: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:03:49.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:49 smithi076 ceph-mon[32095]: pgmap v248: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:03:51.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:51 smithi088 ceph-mon[36585]: pgmap v249: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:51.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:51 smithi076 ceph-mon[32095]: pgmap v249: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:51.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:51 smithi082 ceph-mon[36382]: pgmap v249: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:53.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:53 smithi088 ceph-mon[36585]: pgmap v250: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:03:53.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:53 smithi076 ceph-mon[32095]: pgmap v250: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:03:53.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:53 smithi082 ceph-mon[36382]: pgmap v250: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:03:55.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:55 smithi088 ceph-mon[36585]: pgmap v251: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:55.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:55 smithi076 ceph-mon[32095]: pgmap v251: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:55.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:55 smithi082 ceph-mon[36382]: pgmap v251: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:57.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:57 smithi088 ceph-mon[36585]: pgmap v252: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:57.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:57 smithi076 ceph-mon[32095]: pgmap v252: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:57.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:57 smithi082 ceph-mon[36382]: pgmap v252: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:03:58.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:58 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:03:58.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:58 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:03:58.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:58 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:03:58.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:58 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:03:58.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:58 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:03:58.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:58 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:03:58.597 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:58 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:03:58.597 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:58 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:03:58.597 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:58 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:03:59.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:59 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:03:59.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:59 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:03:59.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:59 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:03:59.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:59 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:03:59.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:03:59 smithi088 ceph-mon[36585]: pgmap v253: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:03:59.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:59 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:03:59.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:59 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:03:59.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:59 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:03:59.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:59 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:03:59.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:03:59 smithi076 ceph-mon[32095]: pgmap v253: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:03:59.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:59 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:03:59.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:59 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:03:59.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:59 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:03:59.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:59 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:03:59.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:03:59 smithi082 ceph-mon[36382]: pgmap v253: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:04:01.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:01 smithi088 ceph-mon[36585]: pgmap v254: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:01.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:01 smithi076 ceph-mon[32095]: pgmap v254: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:01.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:01 smithi082 ceph-mon[36382]: pgmap v254: 97 pgs: 97 active+clean; 80 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:03.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:03 smithi088 ceph-mon[36585]: pgmap v255: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-12T06:04:03.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:03 smithi076 ceph-mon[32095]: pgmap v255: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-12T06:04:03.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:03 smithi082 ceph-mon[36382]: pgmap v255: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-12T06:04:05.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:05 smithi088 ceph-mon[36585]: pgmap v256: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:04:05.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:05 smithi076 ceph-mon[32095]: pgmap v256: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:04:05.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:05 smithi082 ceph-mon[36382]: pgmap v256: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:04:07.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:07 smithi088 ceph-mon[36585]: pgmap v257: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:04:07.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:07 smithi088 ceph-mon[36585]: osdmap e94: 12 total, 12 up, 12 in 2024-03-12T06:04:07.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:07 smithi076 ceph-mon[32095]: pgmap v257: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:04:07.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:07 smithi076 ceph-mon[32095]: osdmap e94: 12 total, 12 up, 12 in 2024-03-12T06:04:07.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:07 smithi082 ceph-mon[36382]: pgmap v257: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:04:07.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:07 smithi082 ceph-mon[36382]: osdmap e94: 12 total, 12 up, 12 in 2024-03-12T06:04:09.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:09 smithi088 ceph-mon[36585]: pgmap v259: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:04:09.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:09 smithi076 ceph-mon[32095]: pgmap v259: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:04:09.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:09 smithi082 ceph-mon[36382]: pgmap v259: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:04:11.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:11 smithi088 ceph-mon[36585]: pgmap v260: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:04:11.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:11 smithi076 ceph-mon[32095]: pgmap v260: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:04:11.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:11 smithi082 ceph-mon[36382]: pgmap v260: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:04:12.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:12 smithi082 ceph-mon[36382]: osdmap e95: 12 total, 12 up, 12 in 2024-03-12T06:04:12.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:12 smithi088 ceph-mon[36585]: osdmap e95: 12 total, 12 up, 12 in 2024-03-12T06:04:12.989 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:12 smithi076 ceph-mon[32095]: osdmap e95: 12 total, 12 up, 12 in 2024-03-12T06:04:13.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:13 smithi082 ceph-mon[36382]: pgmap v262: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-12T06:04:13.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:13 smithi088 ceph-mon[36585]: pgmap v262: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-12T06:04:13.989 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:13 smithi076 ceph-mon[32095]: pgmap v262: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-12T06:04:15.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:15 smithi088 ceph-mon[36585]: pgmap v263: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-12T06:04:15.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:15 smithi076 ceph-mon[32095]: pgmap v263: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-12T06:04:15.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:15 smithi082 ceph-mon[36382]: pgmap v263: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-12T06:04:17.166 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:04:17.167 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:04:17.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.280+0000 7f67723b6640 1 -- 172.21.15.76:0/1566230032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c0ffa10 msgr2=0x7f676c0ffe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.280+0000 7f67723b6640 1 --2- 172.21.15.76:0/1566230032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c0ffa10 0x7f676c0ffe70 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f675c009900 tx=0x7f675c02bc30 comp rx=0 tx=0).stop 2024-03-12T06:04:17.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.280+0000 7f67723b6640 1 -- 172.21.15.76:0/1566230032 shutdown_connections 2024-03-12T06:04:17.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.280+0000 7f67723b6640 1 --2- 172.21.15.76:0/1566230032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c1003b0 0x7f676c10c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.280+0000 7f67723b6640 1 --2- 172.21.15.76:0/1566230032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c0ffa10 0x7f676c0ffe70 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.280+0000 7f67723b6640 1 --2- 172.21.15.76:0/1566230032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f676c104780 0x7f676c104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.280+0000 7f67723b6640 1 -- 172.21.15.76:0/1566230032 >> 172.21.15.76:0/1566230032 conn(0x7f676c0752a0 msgr2=0x7f676c0756b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.280+0000 7f67723b6640 1 -- 172.21.15.76:0/1566230032 shutdown_connections 2024-03-12T06:04:17.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.280+0000 7f67723b6640 1 -- 172.21.15.76:0/1566230032 wait complete. 2024-03-12T06:04:17.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.280+0000 7f67723b6640 1 Processor -- start 2024-03-12T06:04:17.286 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.280+0000 7f67723b6640 1 -- start start 2024-03-12T06:04:17.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f67723b6640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c0ffa10 0x7f676c197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.288 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f67723b6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c1003b0 0x7f676c197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f67723b6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f676c104780 0x7f676c19bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f67723b6640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c19c5f0 con 0x7f676c1003b0 2024-03-12T06:04:17.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f67723b6640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c19c760 con 0x7f676c0ffa10 2024-03-12T06:04:17.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f67723b6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c19c920 con 0x7f676c104780 2024-03-12T06:04:17.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f676b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c1003b0 0x7f676c197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f676b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c1003b0 0x7f676c197cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37566/0 (socket says 172.21.15.76:37566) 2024-03-12T06:04:17.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f676b7fe640 1 -- 172.21.15.76:0/3065028827 learned_addr learned my addr 172.21.15.76:0/3065028827 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f676bfff640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c0ffa10 0x7f676c197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f6770953640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f676c104780 0x7f676c19bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f676b7fe640 1 -- 172.21.15.76:0/3065028827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f676c104780 msgr2=0x7f676c19bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f676b7fe640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f676c104780 0x7f676c19bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f676b7fe640 1 -- 172.21.15.76:0/3065028827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c0ffa10 msgr2=0x7f676c197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f676b7fe640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c0ffa10 0x7f676c197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f676b7fe640 1 -- 172.21.15.76:0/3065028827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c009560 con 0x7f676c1003b0 2024-03-12T06:04:17.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f6770953640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f676c104780 0x7f676c19bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:17.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.281+0000 7f676b7fe640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c1003b0 0x7f676c197cc0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f675c005940 tx=0x7f675c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.282+0000 7f67697fa640 1 -- 172.21.15.76:0/3065028827 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c004440 con 0x7f676c1003b0 2024-03-12T06:04:17.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.282+0000 7f67697fa640 1 -- 172.21.15.76:0/3065028827 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f675c0052d0 con 0x7f676c1003b0 2024-03-12T06:04:17.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.282+0000 7f67723b6640 1 -- 172.21.15.76:0/3065028827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c19cba0 con 0x7f676c1003b0 2024-03-12T06:04:17.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.282+0000 7f67697fa640 1 -- 172.21.15.76:0/3065028827 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c003da0 con 0x7f676c1003b0 2024-03-12T06:04:17.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.282+0000 7f67723b6640 1 -- 172.21.15.76:0/3065028827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c1a4480 con 0x7f676c1003b0 2024-03-12T06:04:17.291 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.284+0000 7f67697fa640 1 -- 172.21.15.76:0/3065028827 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f675c005440 con 0x7f676c1003b0 2024-03-12T06:04:17.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.284+0000 7f6774f87740 1 Processor -- start 2024-03-12T06:04:17.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.284+0000 7f6774f87740 1 -- start start 2024-03-12T06:04:17.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.284+0000 7f67697fa640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f67580a1ba0 0x7f67580a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.284+0000 7f6774f87740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92b52b0 0x55a6d92b5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.284+0000 7f6774f87740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b5bd0 0x55a6d92be270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.284+0000 7f6774f87740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92be7b0 0x55a6d92c0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.284+0000 7f6770953640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f67580a1ba0 0x7f67580a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.284+0000 7f6774f87740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d9224280 con 0x55a6d92b5bd0 2024-03-12T06:04:17.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.284+0000 7f6774f87740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d92217b0 con 0x55a6d92be7b0 2024-03-12T06:04:17.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.284+0000 7f6774f87740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d921cf90 con 0x55a6d92b52b0 2024-03-12T06:04:17.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f67697fa640 1 -- 172.21.15.76:0/3065028827 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f675c0ef9c0 con 0x7f676c1003b0 2024-03-12T06:04:17.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f676b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b5bd0 0x55a6d92be270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f676b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b5bd0 0x55a6d92be270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37574/0 (socket says 172.21.15.76:37574) 2024-03-12T06:04:17.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f676b7fe640 1 -- 172.21.15.76:0/489473751 learned_addr learned my addr 172.21.15.76:0/489473751 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f676bfff640 1 --2- 172.21.15.76:0/489473751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92b52b0 0x55a6d92b5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f676b7fe640 1 -- 172.21.15.76:0/489473751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92b52b0 msgr2=0x55a6d92b5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f676b7fe640 1 --2- 172.21.15.76:0/489473751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92b52b0 0x55a6d92b5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f676b7fe640 1 -- 172.21.15.76:0/489473751 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92be7b0 msgr2=0x55a6d92c0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f676b7fe640 1 --2- 172.21.15.76:0/489473751 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92be7b0 0x55a6d92c0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f676b7fe640 1 -- 172.21.15.76:0/489473751 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6d91db570 con 0x55a6d92b5bd0 2024-03-12T06:04:17.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f6770953640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f67580a1ba0 0x7f67580a4060 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f676400af30 tx=0x7f676400a8b0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f676b7fe640 1 --2- 172.21.15.76:0/489473751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b5bd0 0x55a6d92be270 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f675c0bc690 tx=0x7f675c0bc590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67b6e688930bfca server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.300 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f67627fc640 1 -- 172.21.15.76:0/489473751 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c0bce00 con 0x55a6d92b5bd0 2024-03-12T06:04:17.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f67627fc640 1 -- 172.21.15.76:0/489473751 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f675c0bc700 con 0x55a6d92b5bd0 2024-03-12T06:04:17.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.285+0000 7f67627fc640 1 -- 172.21.15.76:0/489473751 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c0b8d70 con 0x55a6d92b5bd0 2024-03-12T06:04:17.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.286+0000 7f6774f87740 1 -- 172.21.15.76:0/489473751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b5bd0 msgr2=0x55a6d92be270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.286+0000 7f6774f87740 1 --2- 172.21.15.76:0/489473751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b5bd0 0x55a6d92be270 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f675c0bc690 tx=0x7f675c0bc590 comp rx=0 tx=0).stop 2024-03-12T06:04:17.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.286+0000 7f6774f87740 1 -- 172.21.15.76:0/489473751 shutdown_connections 2024-03-12T06:04:17.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.286+0000 7f6774f87740 1 --2- 172.21.15.76:0/489473751 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92be7b0 0x55a6d92c0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.286+0000 7f6774f87740 1 --2- 172.21.15.76:0/489473751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b5bd0 0x55a6d92be270 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.286+0000 7f6774f87740 1 --2- 172.21.15.76:0/489473751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92b52b0 0x55a6d92b5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.286+0000 7f6774f87740 1 -- 172.21.15.76:0/489473751 >> 172.21.15.76:0/489473751 conn(0x55a6d92af990 msgr2=0x55a6d92afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.286+0000 7f6774f87740 1 -- 172.21.15.76:0/489473751 shutdown_connections 2024-03-12T06:04:17.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.286+0000 7f6774f87740 1 -- 172.21.15.76:0/489473751 wait complete. 2024-03-12T06:04:17.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.286+0000 7f6774f87740 1 -- start start 2024-03-12T06:04:17.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f6774f87740 1 Processor -- start 2024-03-12T06:04:17.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f6774f87740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b52b0 0x55a6d92be260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f6774f87740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92b5bd0 0x55a6d92bb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.302 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f6774f87740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92be7b0 0x55a6d92bbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f6774f87740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d8ef0430 con 0x55a6d92b52b0 2024-03-12T06:04:17.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f6774f87740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d8eace00 con 0x55a6d92be7b0 2024-03-12T06:04:17.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f6774f87740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d91a1fd0 con 0x55a6d92b5bd0 2024-03-12T06:04:17.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f6774f87740 10 client.0 _ll_register_callbacks cb 0x55a6d91f7fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:17.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f676bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b52b0 0x55a6d92be260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f676bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b52b0 0x55a6d92be260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37582/0 (socket says 172.21.15.76:37582) 2024-03-12T06:04:17.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f676bfff640 1 -- 172.21.15.76:0/1089842121 learned_addr learned my addr 172.21.15.76:0/1089842121 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f676b7fe640 1 --2- 172.21.15.76:0/1089842121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92b5bd0 0x55a6d92bb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.303 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f6770953640 1 --2- 172.21.15.76:0/1089842121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92be7b0 0x55a6d92bbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f676bfff640 1 -- 172.21.15.76:0/1089842121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92b5bd0 msgr2=0x55a6d92bb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f676bfff640 1 --2- 172.21.15.76:0/1089842121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92b5bd0 0x55a6d92bb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f676bfff640 1 -- 172.21.15.76:0/1089842121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92be7b0 msgr2=0x55a6d92bbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f676bfff640 1 --2- 172.21.15.76:0/1089842121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92be7b0 0x55a6d92bbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f676bfff640 1 -- 172.21.15.76:0/1089842121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c1a4480 con 0x55a6d92b52b0 2024-03-12T06:04:17.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f676bfff640 1 --2- 172.21.15.76:0/1089842121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b52b0 0x55a6d92be260 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f675400fa50 tx=0x7f675400ff10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30de0c7ec0954f23 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f674b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:17.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.287+0000 7f674b7fe640 1 -- 172.21.15.76:0/1089842121 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6754014070 con 0x55a6d92b52b0 2024-03-12T06:04:17.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.288+0000 7f674b7fe640 1 -- 172.21.15.76:0/1089842121 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f675400dbd0 con 0x55a6d92b52b0 2024-03-12T06:04:17.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.288+0000 7f6774f87740 1 -- 172.21.15.76:0/1089842121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a6d8f1c7e0 con 0x55a6d92b52b0 2024-03-12T06:04:17.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.288+0000 7f674b7fe640 1 -- 172.21.15.76:0/1089842121 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f675400ddd0 con 0x55a6d92b52b0 2024-03-12T06:04:17.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.288+0000 7f6774f87740 10 client.17358 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:17.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.288+0000 7f6774f87740 1 -- 172.21.15.76:0/1089842121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a6d921b610 con 0x55a6d92b52b0 2024-03-12T06:04:17.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.288+0000 7f674b7fe640 1 -- 172.21.15.76:0/1089842121 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6754025020 con 0x55a6d92b52b0 2024-03-12T06:04:17.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.288+0000 7f674b7fe640 1 -- 172.21.15.76:0/1089842121 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f675400c4f0 con 0x55a6d92b52b0 2024-03-12T06:04:17.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.288+0000 7f6774f87740 10 client.17358 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:17.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.288+0000 7f6774f87740 10 client.17358 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:17.305 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.288+0000 7f6774f87740 20 client.17358 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.288+0000 7f6774f87740 1 --2- 172.21.15.76:0/1089842121 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a6d92ce910 0x55a6d92d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.288+0000 7f6774f87740 4 client.17358 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:04:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.288+0000 7f6774f87740 1 -- 172.21.15.76:0/1089842121 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a6d8f59720 con 0x55a6d92ce910 2024-03-12T06:04:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.288+0000 7f676b7fe640 1 --2- 172.21.15.76:0/1089842121 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a6d92ce910 0x55a6d92d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.289+0000 7f674b7fe640 10 client.17358 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.289+0000 7f676b7fe640 1 --2- 172.21.15.76:0/1089842121 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a6d92ce910 0x55a6d92d0d30 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e87e25b6c1c70ade server_cookie=dd55e79270585b5f in_seq=0 out_seq=0 2024-03-12T06:04:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.289+0000 7f674b7fe640 1 -- 172.21.15.76:0/1089842121 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a6d8f59720 con 0x55a6d92ce910 2024-03-12T06:04:17.306 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.289+0000 7f674b7fe640 10 client.17358 handle_command_reply: tid=0 2024-03-12T06:04:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.290+0000 7f6774f87740 1 client.17358 shutdown 2024-03-12T06:04:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.290+0000 7f6774f87740 1 -- 172.21.15.76:0/1089842121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b52b0 msgr2=0x55a6d92be260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.290+0000 7f6774f87740 1 --2- 172.21.15.76:0/1089842121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b52b0 0x55a6d92be260 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f675400fa50 tx=0x7f675400ff10 comp rx=0 tx=0).stop 2024-03-12T06:04:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.290+0000 7f6774f87740 1 -- 172.21.15.76:0/1089842121 shutdown_connections 2024-03-12T06:04:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.290+0000 7f6774f87740 1 --2- 172.21.15.76:0/1089842121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92be7b0 0x55a6d92bbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.290+0000 7f6774f87740 1 --2- 172.21.15.76:0/1089842121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92b5bd0 0x55a6d92bb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.290+0000 7f6774f87740 1 --2- 172.21.15.76:0/1089842121 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a6d92ce910 0x55a6d92d0d30 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.290+0000 7f6774f87740 1 --2- 172.21.15.76:0/1089842121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b52b0 0x55a6d92be260 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.307 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.290+0000 7f674b7fe640 0 client.17358 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.290+0000 7f6774f87740 1 -- 172.21.15.76:0/1089842121 >> 172.21.15.76:0/1089842121 conn(0x55a6d92af990 msgr2=0x55a6d92b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.290+0000 7f6774f87740 1 -- 172.21.15.76:0/1089842121 shutdown_connections 2024-03-12T06:04:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.290+0000 7f6774f87740 1 -- 172.21.15.76:0/1089842121 wait complete. 2024-03-12T06:04:17.308 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.290+0000 7f6774f87740 20 client.17358 trim_cache size 0 max 16384 2024-03-12T06:04:17.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f6774f87740 1 Processor -- start 2024-03-12T06:04:17.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f6774f87740 1 -- start start 2024-03-12T06:04:17.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f6774f87740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92b52b0 0x55a6d92c5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.315 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f6774f87740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b5bd0 0x55a6d92c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f6774f87740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92be7b0 0x55a6d92bdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f6774f87740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d92a9e90 con 0x55a6d92b5bd0 2024-03-12T06:04:17.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f6774f87740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d92d5f20 con 0x55a6d92b52b0 2024-03-12T06:04:17.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f6774f87740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d92ab2b0 con 0x55a6d92be7b0 2024-03-12T06:04:17.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f676b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b5bd0 0x55a6d92c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f676b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b5bd0 0x55a6d92c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37594/0 (socket says 172.21.15.76:37594) 2024-03-12T06:04:17.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f676b7fe640 1 -- 172.21.15.76:0/3009588549 learned_addr learned my addr 172.21.15.76:0/3009588549 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f676bfff640 1 --2- 172.21.15.76:0/3009588549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92b52b0 0x55a6d92c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.316 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f676b7fe640 1 -- 172.21.15.76:0/3009588549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92be7b0 msgr2=0x55a6d92bdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f6770953640 1 --2- 172.21.15.76:0/3009588549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92be7b0 0x55a6d92bdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f676b7fe640 1 --2- 172.21.15.76:0/3009588549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92be7b0 0x55a6d92bdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f676b7fe640 1 -- 172.21.15.76:0/3009588549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92b52b0 msgr2=0x55a6d92c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f676b7fe640 1 --2- 172.21.15.76:0/3009588549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92b52b0 0x55a6d92c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f676b7fe640 1 -- 172.21.15.76:0/3009588549 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6d91b7ac0 con 0x55a6d92b5bd0 2024-03-12T06:04:17.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f676bfff640 1 --2- 172.21.15.76:0/3009588549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92b52b0 0x55a6d92c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:17.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.305+0000 7f676b7fe640 1 --2- 172.21.15.76:0/3009588549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b5bd0 0x55a6d92c5a80 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f675c0b90d0 tx=0x7f675c003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ff9e67d5f5c7e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.306+0000 7f6761ffb640 1 -- 172.21.15.76:0/3009588549 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c0c4ea0 con 0x55a6d92b5bd0 2024-03-12T06:04:17.317 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.306+0000 7f6761ffb640 1 -- 172.21.15.76:0/3009588549 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f675c0baa00 con 0x55a6d92b5bd0 2024-03-12T06:04:17.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.306+0000 7f6761ffb640 1 -- 172.21.15.76:0/3009588549 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c0dc0c0 con 0x55a6d92b5bd0 2024-03-12T06:04:17.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.306+0000 7f6774f87740 1 -- 172.21.15.76:0/3009588549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b5bd0 msgr2=0x55a6d92c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.306+0000 7f6774f87740 1 --2- 172.21.15.76:0/3009588549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b5bd0 0x55a6d92c5a80 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f675c0b90d0 tx=0x7f675c003f90 comp rx=0 tx=0).stop 2024-03-12T06:04:17.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.306+0000 7f6774f87740 1 -- 172.21.15.76:0/3009588549 shutdown_connections 2024-03-12T06:04:17.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.306+0000 7f6774f87740 1 --2- 172.21.15.76:0/3009588549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92be7b0 0x55a6d92bdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.306+0000 7f6774f87740 1 --2- 172.21.15.76:0/3009588549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b5bd0 0x55a6d92c5a80 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.306+0000 7f6774f87740 1 --2- 172.21.15.76:0/3009588549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92b52b0 0x55a6d92c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.318 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.306+0000 7f6774f87740 1 -- 172.21.15.76:0/3009588549 >> 172.21.15.76:0/3009588549 conn(0x55a6d92af990 msgr2=0x55a6d92b68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.306+0000 7f6774f87740 1 -- 172.21.15.76:0/3009588549 shutdown_connections 2024-03-12T06:04:17.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.306+0000 7f6774f87740 1 -- 172.21.15.76:0/3009588549 wait complete. 2024-03-12T06:04:17.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.306+0000 7f6774f87740 1 -- start start 2024-03-12T06:04:17.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f6774f87740 1 Processor -- start 2024-03-12T06:04:17.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f6774f87740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b52b0 0x55a6d92c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f6774f87740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92b5bd0 0x55a6d92c3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f6774f87740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92be7b0 0x55a6d92c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f6774f87740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d92c44e0 con 0x55a6d92b52b0 2024-03-12T06:04:17.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f6774f87740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d92c4650 con 0x55a6d92b5bd0 2024-03-12T06:04:17.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f6774f87740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6d92c4810 con 0x55a6d92be7b0 2024-03-12T06:04:17.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f6774f87740 10 client.0 _ll_register_callbacks cb 0x7f675c037fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:17.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f676bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b52b0 0x55a6d92c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f676bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b52b0 0x55a6d92c5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37604/0 (socket says 172.21.15.76:37604) 2024-03-12T06:04:17.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f676bfff640 1 -- 172.21.15.76:0/4240875575 learned_addr learned my addr 172.21.15.76:0/4240875575 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f676b7fe640 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92b5bd0 0x55a6d92c3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f6770953640 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92be7b0 0x55a6d92c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f676bfff640 1 -- 172.21.15.76:0/4240875575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92be7b0 msgr2=0x55a6d92c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.320 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f676bfff640 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92be7b0 0x55a6d92c3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f676bfff640 1 -- 172.21.15.76:0/4240875575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92b5bd0 msgr2=0x55a6d92c3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f676bfff640 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92b5bd0 0x55a6d92c3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f676bfff640 1 -- 172.21.15.76:0/4240875575 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6d91db570 con 0x55a6d92b52b0 2024-03-12T06:04:17.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f6770953640 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92be7b0 0x55a6d92c3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:17.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f676b7fe640 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92b5bd0 0x55a6d92c3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:17.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f676bfff640 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b52b0 0x55a6d92c5670 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f675400b850 tx=0x7f675400bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d1f4aa66e454ec server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.307+0000 7f674b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:17.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.308+0000 7f674b7fe640 1 -- 172.21.15.76:0/4240875575 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6754014040 con 0x55a6d92b52b0 2024-03-12T06:04:17.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.308+0000 7f674b7fe640 1 -- 172.21.15.76:0/4240875575 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f6754026080 con 0x55a6d92b52b0 2024-03-12T06:04:17.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.308+0000 7f6774f87740 1 -- 172.21.15.76:0/4240875575 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f675c0bc700 con 0x55a6d92b52b0 2024-03-12T06:04:17.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.308+0000 7f674b7fe640 1 -- 172.21.15.76:0/4240875575 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6754005bb0 con 0x55a6d92b52b0 2024-03-12T06:04:17.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.308+0000 7f6774f87740 10 client.17364 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:17.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.308+0000 7f6774f87740 1 -- 172.21.15.76:0/4240875575 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a6d91dd6a0 con 0x55a6d92b52b0 2024-03-12T06:04:17.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.308+0000 7f674b7fe640 1 -- 172.21.15.76:0/4240875575 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6754036080 con 0x55a6d92b52b0 2024-03-12T06:04:17.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.308+0000 7f674b7fe640 1 -- 172.21.15.76:0/4240875575 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f67540038f0 con 0x55a6d92b52b0 2024-03-12T06:04:17.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.308+0000 7f6774f87740 10 client.17364 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:17.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.308+0000 7f6774f87740 10 client.17364 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:17.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.308+0000 7f6774f87740 20 client.17364 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:17.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.308+0000 7f6774f87740 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a6d9326380 0x55a6d93287a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.308+0000 7f6774f87740 4 client.17364 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:04:17.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.308+0000 7f6774f87740 1 -- 172.21.15.76:0/4240875575 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a6d8fa8890 con 0x55a6d9326380 2024-03-12T06:04:17.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f676b7fe640 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a6d9326380 0x55a6d93287a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f676b7fe640 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a6d9326380 0x55a6d93287a0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9953dade3b3f88a3 server_cookie=19f5a2d13352a157 in_seq=0 out_seq=0 2024-03-12T06:04:17.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f674b7fe640 10 client.17364 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:17.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f674b7fe640 1 -- 172.21.15.76:0/4240875575 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a6d8fa8890 con 0x55a6d9326380 2024-03-12T06:04:17.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f674b7fe640 10 client.17364 handle_command_reply: tid=0 2024-03-12T06:04:17.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f6774f87740 1 client.17364 shutdown 2024-03-12T06:04:17.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f6774f87740 1 -- 172.21.15.76:0/4240875575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b52b0 msgr2=0x55a6d92c5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f6774f87740 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b52b0 0x55a6d92c5670 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f675400b850 tx=0x7f675400bd80 comp rx=0 tx=0).stop 2024-03-12T06:04:17.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f6774f87740 1 -- 172.21.15.76:0/4240875575 shutdown_connections 2024-03-12T06:04:17.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f6774f87740 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a6d92be7b0 0x55a6d92c3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f6774f87740 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a6d92b5bd0 0x55a6d92c3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f6774f87740 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a6d9326380 0x55a6d93287a0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f6774f87740 1 --2- 172.21.15.76:0/4240875575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a6d92b52b0 0x55a6d92c5670 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f674b7fe640 0 client.17364 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:17.324 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.309+0000 7f6774f87740 1 -- 172.21.15.76:0/4240875575 >> 172.21.15.76:0/4240875575 conn(0x55a6d92af990 msgr2=0x55a6d92bae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f6774f87740 1 -- 172.21.15.76:0/4240875575 shutdown_connections 2024-03-12T06:04:17.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f6774f87740 1 -- 172.21.15.76:0/4240875575 wait complete. 2024-03-12T06:04:17.325 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f6774f87740 20 client.17364 trim_cache size 0 max 16384 2024-03-12T06:04:17.325 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:04:17.325 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T06:04:17.325 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T06:04:17.325 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:04:17.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f67723b6640 1 -- 172.21.15.76:0/3065028827 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f67580a1ba0 msgr2=0x7f67580a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f67723b6640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f67580a1ba0 0x7f67580a4060 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f676400af30 tx=0x7f676400a8b0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f67723b6640 1 -- 172.21.15.76:0/3065028827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c1003b0 msgr2=0x7f676c197cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f67723b6640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c1003b0 0x7f676c197cc0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f675c005940 tx=0x7f675c004040 comp rx=0 tx=0).stop 2024-03-12T06:04:17.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f67723b6640 1 -- 172.21.15.76:0/3065028827 shutdown_connections 2024-03-12T06:04:17.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f67723b6640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f67580a1ba0 0x7f67580a4060 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.327 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f67723b6640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f676c104780 0x7f676c19bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f67723b6640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f676c1003b0 0x7f676c197cc0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f67723b6640 1 --2- 172.21.15.76:0/3065028827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f676c0ffa10 0x7f676c197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f67723b6640 1 -- 172.21.15.76:0/3065028827 >> 172.21.15.76:0/3065028827 conn(0x7f676c0752a0 msgr2=0x7f676c0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f67723b6640 1 -- 172.21.15.76:0/3065028827 shutdown_connections 2024-03-12T06:04:17.328 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.310+0000 7f67723b6640 1 -- 172.21.15.76:0/3065028827 wait complete. 2024-03-12T06:04:17.328 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T06:04:17.328 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T06:04:17.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.426+0000 7fe0665a0640 1 -- 172.21.15.76:0/827512891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe060106340 msgr2=0x7fe06010df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.426+0000 7fe0665a0640 1 --2- 172.21.15.76:0/827512891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe060106340 0x7fe06010df10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fe058009cd0 tx=0x7fe05802c0f0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.426+0000 7fe0665a0640 1 -- 172.21.15.76:0/827512891 shutdown_connections 2024-03-12T06:04:17.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.426+0000 7fe0665a0640 1 --2- 172.21.15.76:0/827512891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe060106340 0x7fe06010df10 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.426+0000 7fe0665a0640 1 --2- 172.21.15.76:0/827512891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe060101c20 0x7fe060105c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.426+0000 7fe0665a0640 1 --2- 172.21.15.76:0/827512891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe060101270 0x7fe060101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.426+0000 7fe0665a0640 1 -- 172.21.15.76:0/827512891 >> 172.21.15.76:0/827512891 conn(0x7fe0600fcb50 msgr2=0x7fe0600fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.426+0000 7fe0665a0640 1 -- 172.21.15.76:0/827512891 shutdown_connections 2024-03-12T06:04:17.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.426+0000 7fe0665a0640 1 -- 172.21.15.76:0/827512891 wait complete. 2024-03-12T06:04:17.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe0665a0640 1 Processor -- start 2024-03-12T06:04:17.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe0665a0640 1 -- start start 2024-03-12T06:04:17.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe0665a0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe060101270 0x7fe060073480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe0665a0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe060106340 0x7fe0600739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe0665a0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe060073f00 0x7fe0601a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe0665a0640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0601a5e10 con 0x7fe060073f00 2024-03-12T06:04:17.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe0665a0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0601a5f80 con 0x7fe060106340 2024-03-12T06:04:17.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe0665a0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0601a6140 con 0x7fe060101270 2024-03-12T06:04:17.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe065d9f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe060073f00 0x7fe0601a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe065d9f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe060073f00 0x7fe0601a5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37620/0 (socket says 172.21.15.76:37620) 2024-03-12T06:04:17.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe065d9f640 1 -- 172.21.15.76:0/99050328 learned_addr learned my addr 172.21.15.76:0/99050328 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe064d9d640 1 --2- 172.21.15.76:0/99050328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe060106340 0x7fe0600739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe06559e640 1 --2- 172.21.15.76:0/99050328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe060101270 0x7fe060073480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe065d9f640 1 -- 172.21.15.76:0/99050328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe060101270 msgr2=0x7fe060073480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe065d9f640 1 --2- 172.21.15.76:0/99050328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe060101270 0x7fe060073480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe065d9f640 1 -- 172.21.15.76:0/99050328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe060106340 msgr2=0x7fe0600739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe065d9f640 1 --2- 172.21.15.76:0/99050328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe060106340 0x7fe0600739c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.427+0000 7fe065d9f640 1 -- 172.21.15.76:0/99050328 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe058009900 con 0x7fe060073f00 2024-03-12T06:04:17.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.428+0000 7fe065d9f640 1 --2- 172.21.15.76:0/99050328 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe060073f00 0x7fe0601a5840 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe058004af0 tx=0x7fe058032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.428+0000 7fe0567fc640 1 -- 172.21.15.76:0/99050328 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0580328b0 con 0x7fe060073f00 2024-03-12T06:04:17.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.428+0000 7fe0567fc640 1 -- 172.21.15.76:0/99050328 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe058003e50 con 0x7fe060073f00 2024-03-12T06:04:17.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.428+0000 7fe0665a0640 1 -- 172.21.15.76:0/99050328 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0601a63c0 con 0x7fe060073f00 2024-03-12T06:04:17.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.428+0000 7fe0567fc640 1 -- 172.21.15.76:0/99050328 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058004740 con 0x7fe060073f00 2024-03-12T06:04:17.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.428+0000 7fe0665a0640 1 -- 172.21.15.76:0/99050328 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0601a6870 con 0x7fe060073f00 2024-03-12T06:04:17.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.429+0000 7fe0665a0640 1 -- 172.21.15.76:0/99050328 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe06004ef50 con 0x7fe060073f00 2024-03-12T06:04:17.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.430+0000 7fe0567fc640 1 -- 172.21.15.76:0/99050328 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe05800a040 con 0x7fe060073f00 2024-03-12T06:04:17.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.430+0000 7fe0567fc640 1 --2- 172.21.15.76:0/99050328 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe04c0a1bc0 0x7fe04c0a4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.431+0000 7fe06559e640 1 --2- 172.21.15.76:0/99050328 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe04c0a1bc0 0x7fe04c0a4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.431+0000 7fe0567fc640 1 -- 172.21.15.76:0/99050328 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fe0580efec0 con 0x7fe060073f00 2024-03-12T06:04:17.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.431+0000 7fe06559e640 1 --2- 172.21.15.76:0/99050328 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe04c0a1bc0 0x7fe04c0a4080 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe05c009bd0 tx=0x7fe05c005f90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.435+0000 7fe0567fc640 1 -- 172.21.15.76:0/99050328 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe0580b8ea0 con 0x7fe060073f00 2024-03-12T06:04:17.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:17 smithi088 ceph-mon[36585]: pgmap v264: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-03-12T06:04:17.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:17 smithi076 ceph-mon[32095]: pgmap v264: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-03-12T06:04:17.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.647+0000 7fe0665a0640 1 -- 172.21.15.76:0/99050328 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe060109570 con 0x7fe060073f00 2024-03-12T06:04:17.652 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.648+0000 7fe0567fc640 1 -- 172.21.15.76:0/99050328 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11328 (secure 0 0 0) 0x7fe0580bc560 con 0x7fe060073f00 2024-03-12T06:04:17.653 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:04:17.653 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T06:01:33.842001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T06:04:17.653 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 28 2024-03-12T06:04:17.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.650+0000 7fe0665a0640 1 -- 172.21.15.76:0/99050328 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe04c0a1bc0 msgr2=0x7fe04c0a4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.650+0000 7fe0665a0640 1 --2- 172.21.15.76:0/99050328 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe04c0a1bc0 0x7fe04c0a4080 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe05c009bd0 tx=0x7fe05c005f90 comp rx=0 tx=0).stop 2024-03-12T06:04:17.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.650+0000 7fe0665a0640 1 -- 172.21.15.76:0/99050328 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe060073f00 msgr2=0x7fe0601a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.650+0000 7fe0665a0640 1 --2- 172.21.15.76:0/99050328 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe060073f00 0x7fe0601a5840 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe058004af0 tx=0x7fe058032570 comp rx=0 tx=0).stop 2024-03-12T06:04:17.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.650+0000 7fe0665a0640 1 -- 172.21.15.76:0/99050328 shutdown_connections 2024-03-12T06:04:17.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.650+0000 7fe0665a0640 1 --2- 172.21.15.76:0/99050328 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe04c0a1bc0 0x7fe04c0a4080 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.655 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.650+0000 7fe0665a0640 1 --2- 172.21.15.76:0/99050328 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe060073f00 0x7fe0601a5840 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.650+0000 7fe0665a0640 1 --2- 172.21.15.76:0/99050328 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe060106340 0x7fe0600739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.650+0000 7fe0665a0640 1 --2- 172.21.15.76:0/99050328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe060101270 0x7fe060073480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.650+0000 7fe0665a0640 1 -- 172.21.15.76:0/99050328 >> 172.21.15.76:0/99050328 conn(0x7fe0600fcb50 msgr2=0x7fe0600fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.650+0000 7fe0665a0640 1 -- 172.21.15.76:0/99050328 shutdown_connections 2024-03-12T06:04:17.656 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.650+0000 7fe0665a0640 1 -- 172.21.15.76:0/99050328 wait complete. 2024-03-12T06:04:17.665 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:04:17.666 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T06:04:17.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:17 smithi082 ceph-mon[36382]: pgmap v264: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-03-12T06:04:17.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.765+0000 7feed4fac640 1 -- 172.21.15.76:0/745913519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feed0100280 msgr2=0x7feed0100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.765+0000 7feed4fac640 1 --2- 172.21.15.76:0/745913519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feed0100280 0x7feed0100660 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7feec0009930 tx=0x7feec002bc30 comp rx=0 tx=0).stop 2024-03-12T06:04:17.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.765+0000 7feed4fac640 1 -- 172.21.15.76:0/745913519 shutdown_connections 2024-03-12T06:04:17.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.765+0000 7feed4fac640 1 --2- 172.21.15.76:0/745913519 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feed0105460 0x7feed010d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.765+0000 7feed4fac640 1 --2- 172.21.15.76:0/745913519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feed0100c30 0x7feed0104d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.765+0000 7feed4fac640 1 --2- 172.21.15.76:0/745913519 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feed0100280 0x7feed0100660 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.765+0000 7feed4fac640 1 -- 172.21.15.76:0/745913519 >> 172.21.15.76:0/745913519 conn(0x7feed00771a0 msgr2=0x7feed00775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.765+0000 7feed4fac640 1 -- 172.21.15.76:0/745913519 shutdown_connections 2024-03-12T06:04:17.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.766+0000 7feed4fac640 1 -- 172.21.15.76:0/745913519 wait complete. 2024-03-12T06:04:17.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.766+0000 7feed4fac640 1 Processor -- start 2024-03-12T06:04:17.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.766+0000 7feed4fac640 1 -- start start 2024-03-12T06:04:17.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.766+0000 7feed4fac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feed0100280 0x7feed019d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.766+0000 7feed4fac640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feed0105460 0x7feed019db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.766+0000 7feed4fac640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feed0197710 0x7feed0197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.766+0000 7feed4fac640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed01981e0 con 0x7feed0197710 2024-03-12T06:04:17.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.766+0000 7feed4fac640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed0198350 con 0x7feed0105460 2024-03-12T06:04:17.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.766+0000 7feed4fac640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed0198510 con 0x7feed0100280 2024-03-12T06:04:17.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feeced9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feed0197710 0x7feed0197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.774 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feeced9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feed0197710 0x7feed0197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37636/0 (socket says 172.21.15.76:37636) 2024-03-12T06:04:17.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feeced9d640 1 -- 172.21.15.76:0/3476715464 learned_addr learned my addr 172.21.15.76:0/3476715464 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feece59c640 1 --2- 172.21.15.76:0/3476715464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feed0100280 0x7feed019d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feeced9d640 1 -- 172.21.15.76:0/3476715464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feed0100280 msgr2=0x7feed019d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feeced9d640 1 --2- 172.21.15.76:0/3476715464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feed0100280 0x7feed019d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feecdd9b640 1 --2- 172.21.15.76:0/3476715464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feed0105460 0x7feed019db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feeced9d640 1 -- 172.21.15.76:0/3476715464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feed0105460 msgr2=0x7feed019db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feeced9d640 1 --2- 172.21.15.76:0/3476715464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feed0105460 0x7feed019db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feeced9d640 1 -- 172.21.15.76:0/3476715464 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec0009560 con 0x7feed0197710 2024-03-12T06:04:17.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feeced9d640 1 --2- 172.21.15.76:0/3476715464 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feed0197710 0x7feed0197b90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7feec800cce0 tx=0x7feec800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.775 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feebf7fe640 1 -- 172.21.15.76:0/3476715464 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec800f450 con 0x7feed0197710 2024-03-12T06:04:17.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feebf7fe640 1 -- 172.21.15.76:0/3476715464 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feec800fa90 con 0x7feed0197710 2024-03-12T06:04:17.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feed4fac640 1 -- 172.21.15.76:0/3476715464 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed01a44b0 con 0x7feed0197710 2024-03-12T06:04:17.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feebf7fe640 1 -- 172.21.15.76:0/3476715464 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec8018b90 con 0x7feed0197710 2024-03-12T06:04:17.776 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.767+0000 7feed4fac640 1 -- 172.21.15.76:0/3476715464 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed01a4940 con 0x7feed0197710 2024-03-12T06:04:17.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.769+0000 7feebf7fe640 1 -- 172.21.15.76:0/3476715464 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7feec800fc00 con 0x7feed0197710 2024-03-12T06:04:17.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feed7b7f740 1 Processor -- start 2024-03-12T06:04:17.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feed7b7f740 1 -- start start 2024-03-12T06:04:17.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feed7b7f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952ec30 0x55bd7952f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feed7b7f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952f5e0 0x55bd7953c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.782 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feed7b7f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd7953ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feed7b7f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd794ea960 con 0x55bd7953cb60 2024-03-12T06:04:17.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feed7b7f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd7949e310 con 0x55bd7952ec30 2024-03-12T06:04:17.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feed7b7f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd7949b840 con 0x55bd7952f5e0 2024-03-12T06:04:17.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feebf7fe640 1 --2- 172.21.15.76:0/3476715464 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7feeb80a1ba0 0x7feeb80a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feecdd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd7953ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feecdd9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952ec30 0x55bd7952f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feeced9d640 1 --2- 172.21.15.76:0/3476715464 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7feeb80a1ba0 0x7feeb80a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.783 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feecdd9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952ec30 0x55bd7952f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:50992/0 (socket says 172.21.15.76:50992) 2024-03-12T06:04:17.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feecdd9b640 1 -- 172.21.15.76:0/1525051269 learned_addr learned my addr 172.21.15.76:0/1525051269 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feebf7fe640 1 -- 172.21.15.76:0/3476715464 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7feec80ce050 con 0x7feed0197710 2024-03-12T06:04:17.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.770+0000 7feece59c640 1 --2- 172.21.15.76:0/1525051269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952f5e0 0x55bd7953c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.771+0000 7feecdd9b640 1 -- 172.21.15.76:0/1525051269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952f5e0 msgr2=0x55bd7953c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.771+0000 7feecdd9b640 1 --2- 172.21.15.76:0/1525051269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952f5e0 0x55bd7953c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.771+0000 7feecdd9b640 1 -- 172.21.15.76:0/1525051269 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 msgr2=0x55bd7953ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.771+0000 7feecdd9b640 1 --2- 172.21.15.76:0/1525051269 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd7953ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.771+0000 7feecdd9b640 1 -- 172.21.15.76:0/1525051269 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd79459060 con 0x55bd7952ec30 2024-03-12T06:04:17.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.771+0000 7feece59c640 1 --2- 172.21.15.76:0/1525051269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952f5e0 0x55bd7953c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:17.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.771+0000 7feeced9d640 1 --2- 172.21.15.76:0/3476715464 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7feeb80a1ba0 0x7feeb80a4060 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feec80cb850 tx=0x7feec806df30 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.771+0000 7feecdd9b640 1 --2- 172.21.15.76:0/1525051269 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952ec30 0x55bd7952f010 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7feec400bf20 tx=0x7feec4006b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f36b3f743e8702f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.771+0000 7feebcff9640 1 -- 172.21.15.76:0/1525051269 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec401b070 con 0x55bd7952ec30 2024-03-12T06:04:17.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.771+0000 7feebcff9640 1 -- 172.21.15.76:0/1525051269 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feec40178f0 con 0x55bd7952ec30 2024-03-12T06:04:17.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.771+0000 7feebcff9640 1 -- 172.21.15.76:0/1525051269 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec4011560 con 0x55bd7952ec30 2024-03-12T06:04:17.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 -- 172.21.15.76:0/1525051269 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952ec30 msgr2=0x55bd7952f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 --2- 172.21.15.76:0/1525051269 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952ec30 0x55bd7952f010 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7feec400bf20 tx=0x7feec4006b20 comp rx=0 tx=0).stop 2024-03-12T06:04:17.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 -- 172.21.15.76:0/1525051269 shutdown_connections 2024-03-12T06:04:17.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 --2- 172.21.15.76:0/1525051269 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd7953ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 --2- 172.21.15.76:0/1525051269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952f5e0 0x55bd7953c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 --2- 172.21.15.76:0/1525051269 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952ec30 0x55bd7952f010 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 -- 172.21.15.76:0/1525051269 >> 172.21.15.76:0/1525051269 conn(0x55bd79529980 msgr2=0x55bd79529d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 -- 172.21.15.76:0/1525051269 shutdown_connections 2024-03-12T06:04:17.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 -- 172.21.15.76:0/1525051269 wait complete. 2024-03-12T06:04:17.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 -- start start 2024-03-12T06:04:17.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 Processor -- start 2024-03-12T06:04:17.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7952ec30 0x55bd79539b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952f5e0 0x55bd7953a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7953cb60 0x55bd7953a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd79497020 con 0x55bd7952ec30 2024-03-12T06:04:17.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd7916a430 con 0x55bd7952f5e0 2024-03-12T06:04:17.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd79126e00 con 0x55bd7953cb60 2024-03-12T06:04:17.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.772+0000 7feed7b7f740 10 client.0 _ll_register_callbacks cb 0x55bd7943ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:17.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feecdd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7952ec30 0x55bd79539b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feecdd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7952ec30 0x55bd79539b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37652/0 (socket says 172.21.15.76:37652) 2024-03-12T06:04:17.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feecdd9b640 1 -- 172.21.15.76:0/2453735128 learned_addr learned my addr 172.21.15.76:0/2453735128 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feece59c640 1 --2- 172.21.15.76:0/2453735128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952f5e0 0x55bd7953a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feecdd9b640 1 -- 172.21.15.76:0/2453735128 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7953cb60 msgr2=0x55bd7953a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feecdd9b640 1 --2- 172.21.15.76:0/2453735128 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7953cb60 0x55bd7953a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feecdd9b640 1 -- 172.21.15.76:0/2453735128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952f5e0 msgr2=0x55bd7953a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feecdd9b640 1 --2- 172.21.15.76:0/2453735128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952f5e0 0x55bd7953a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feecdd9b640 1 -- 172.21.15.76:0/2453735128 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec4015300 con 0x55bd7952ec30 2024-03-12T06:04:17.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feecdd9b640 1 --2- 172.21.15.76:0/2453735128 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7952ec30 0x55bd79539b60 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7feec4004810 tx=0x7feec400b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ba8911d60fcbb1c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feeb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:17.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feeb5ffb640 1 -- 172.21.15.76:0/2453735128 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec401b040 con 0x55bd7952ec30 2024-03-12T06:04:17.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feeb5ffb640 1 -- 172.21.15.76:0/2453735128 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7feec4020640 con 0x55bd7952ec30 2024-03-12T06:04:17.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feed7b7f740 1 -- 172.21.15.76:0/2453735128 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bd791967e0 con 0x55bd7952ec30 2024-03-12T06:04:17.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feeb5ffb640 1 -- 172.21.15.76:0/2453735128 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7feec4017ea0 con 0x55bd7952ec30 2024-03-12T06:04:17.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feed7b7f740 10 client.17382 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:17.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feed7b7f740 1 -- 172.21.15.76:0/2453735128 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bd794956a0 con 0x55bd7952ec30 2024-03-12T06:04:17.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.773+0000 7feeb5ffb640 1 -- 172.21.15.76:0/2453735128 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55bd791967e0 con 0x55bd7952ec30 2024-03-12T06:04:17.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.774+0000 7feeb5ffb640 1 -- 172.21.15.76:0/2453735128 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7feec4029970 con 0x55bd7952ec30 2024-03-12T06:04:17.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.774+0000 7feed7b7f740 10 client.17382 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:17.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.774+0000 7feed7b7f740 10 client.17382 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:04:17.791 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.774+0000 7feed7b7f740 20 client.17382 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:17.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.774+0000 7feed7b7f740 1 --2- 172.21.15.76:0/2453735128 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55bd7954cd20 0x55bd7954f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.774+0000 7feed7b7f740 4 client.17382 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:04:17.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.774+0000 7feed7b7f740 1 -- 172.21.15.76:0/2453735128 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bd791d3720 con 0x55bd7954cd20 2024-03-12T06:04:17.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.774+0000 7feece59c640 1 --2- 172.21.15.76:0/2453735128 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55bd7954cd20 0x55bd7954f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.774+0000 7feeb5ffb640 10 client.17382 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:04:17.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.774+0000 7feece59c640 1 --2- 172.21.15.76:0/2453735128 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55bd7954cd20 0x55bd7954f110 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ab8520feb463546 server_cookie=956300d0d80b9431 in_seq=0 out_seq=0 2024-03-12T06:04:17.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feeb5ffb640 1 -- 172.21.15.76:0/2453735128 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55bd791d3720 con 0x55bd7954cd20 2024-03-12T06:04:17.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feeb5ffb640 10 client.17382 handle_command_reply: tid=0 2024-03-12T06:04:17.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feed7b7f740 1 client.17382 shutdown 2024-03-12T06:04:17.792 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feed7b7f740 1 -- 172.21.15.76:0/2453735128 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7952ec30 msgr2=0x55bd79539b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feed7b7f740 1 --2- 172.21.15.76:0/2453735128 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7952ec30 0x55bd79539b60 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7feec4004810 tx=0x7feec400b9b0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feed7b7f740 1 -- 172.21.15.76:0/2453735128 shutdown_connections 2024-03-12T06:04:17.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feed7b7f740 1 --2- 172.21.15.76:0/2453735128 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55bd7954cd20 0x55bd7954f110 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feed7b7f740 1 --2- 172.21.15.76:0/2453735128 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7953cb60 0x55bd7953a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feed7b7f740 1 --2- 172.21.15.76:0/2453735128 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952f5e0 0x55bd7953a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feeb5ffb640 0 client.17382 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:04:17.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feed7b7f740 1 --2- 172.21.15.76:0/2453735128 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7952ec30 0x55bd79539b60 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feed7b7f740 1 -- 172.21.15.76:0/2453735128 >> 172.21.15.76:0/2453735128 conn(0x55bd79529980 msgr2=0x55bd7952b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.793 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feed7b7f740 1 -- 172.21.15.76:0/2453735128 shutdown_connections 2024-03-12T06:04:17.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feed7b7f740 1 -- 172.21.15.76:0/2453735128 wait complete. 2024-03-12T06:04:17.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.775+0000 7feed7b7f740 20 client.17382 trim_cache size 0 max 16384 2024-03-12T06:04:17.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feed7b7f740 1 Processor -- start 2024-03-12T06:04:17.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feed7b7f740 1 -- start start 2024-03-12T06:04:17.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feed7b7f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952ec30 0x55bd7953c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feed7b7f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952f5e0 0x55bd79538d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feed7b7f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd79528a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feed7b7f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd7941c130 con 0x55bd7953cb60 2024-03-12T06:04:17.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feed7b7f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd7953afc0 con 0x55bd7952f5e0 2024-03-12T06:04:17.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feed7b7f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd7953ab30 con 0x55bd7952ec30 2024-03-12T06:04:17.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feecdd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd79528a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feecdd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd79528a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37658/0 (socket says 172.21.15.76:37658) 2024-03-12T06:04:17.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feecdd9b640 1 -- 172.21.15.76:0/1469896384 learned_addr learned my addr 172.21.15.76:0/1469896384 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feecdd9b640 1 --2- 172.21.15.76:0/1469896384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952ec30 0x55bd7953c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feece59c640 1 --2- 172.21.15.76:0/1469896384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952f5e0 0x55bd79538d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feecdd9b640 1 -- 172.21.15.76:0/1469896384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952ec30 msgr2=0x55bd7953c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feecdd9b640 1 --2- 172.21.15.76:0/1469896384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952ec30 0x55bd7953c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feecdd9b640 1 -- 172.21.15.76:0/1469896384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952f5e0 msgr2=0x55bd79538d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.801 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feecdd9b640 1 --2- 172.21.15.76:0/1469896384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952f5e0 0x55bd79538d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feecdd9b640 1 -- 172.21.15.76:0/1469896384 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd79431c20 con 0x55bd7953cb60 2024-03-12T06:04:17.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feecdd9b640 1 --2- 172.21.15.76:0/1469896384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd79528a00 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7feec400bbf0 tx=0x7feec4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5e4816b22870a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.787+0000 7feeb7fff640 1 -- 172.21.15.76:0/1469896384 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec401b030 con 0x55bd7953cb60 2024-03-12T06:04:17.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.788+0000 7feeb7fff640 1 -- 172.21.15.76:0/1469896384 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feec400b1a0 con 0x55bd7953cb60 2024-03-12T06:04:17.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.788+0000 7feeb7fff640 1 -- 172.21.15.76:0/1469896384 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec401f5e0 con 0x55bd7953cb60 2024-03-12T06:04:17.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.788+0000 7feed7b7f740 1 -- 172.21.15.76:0/1469896384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 msgr2=0x55bd79528a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.788+0000 7feed7b7f740 1 --2- 172.21.15.76:0/1469896384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd79528a00 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7feec400bbf0 tx=0x7feec4000bc0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.788+0000 7feed7b7f740 1 -- 172.21.15.76:0/1469896384 shutdown_connections 2024-03-12T06:04:17.802 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.788+0000 7feed7b7f740 1 --2- 172.21.15.76:0/1469896384 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd79528a00 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.788+0000 7feed7b7f740 1 --2- 172.21.15.76:0/1469896384 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952f5e0 0x55bd79538d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.788+0000 7feed7b7f740 1 --2- 172.21.15.76:0/1469896384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952ec30 0x55bd7953c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.788+0000 7feed7b7f740 1 -- 172.21.15.76:0/1469896384 >> 172.21.15.76:0/1469896384 conn(0x55bd79529980 msgr2=0x55bd795282f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.788+0000 7feed7b7f740 1 -- 172.21.15.76:0/1469896384 shutdown_connections 2024-03-12T06:04:17.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.788+0000 7feed7b7f740 1 -- 172.21.15.76:0/1469896384 wait complete. 2024-03-12T06:04:17.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.788+0000 7feed7b7f740 1 -- start start 2024-03-12T06:04:17.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.788+0000 7feed7b7f740 1 Processor -- start 2024-03-12T06:04:17.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feed7b7f740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952ec30 0x55bd795427e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feed7b7f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952f5e0 0x55bd79542d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.803 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feed7b7f740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd795a31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feed7b7f740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd795435f0 con 0x55bd7953cb60 2024-03-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feed7b7f740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd795a3700 con 0x55bd7952ec30 2024-03-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feed7b7f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd795a3840 con 0x55bd7952f5e0 2024-03-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feed7b7f740 10 client.0 _ll_register_callbacks cb 0x7feec400b6f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feecdd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd795a31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feecdd9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd795a31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37660/0 (socket says 172.21.15.76:37660) 2024-03-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feecdd9b640 1 -- 172.21.15.76:0/3347536850 learned_addr learned my addr 172.21.15.76:0/3347536850 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feece59c640 1 --2- 172.21.15.76:0/3347536850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952f5e0 0x55bd79542d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feecdd9b640 1 --2- 172.21.15.76:0/3347536850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952ec30 0x55bd795427e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.804 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feecdd9b640 1 -- 172.21.15.76:0/3347536850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952f5e0 msgr2=0x55bd79542d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feecdd9b640 1 --2- 172.21.15.76:0/3347536850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952f5e0 0x55bd79542d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feecdd9b640 1 -- 172.21.15.76:0/3347536850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952ec30 msgr2=0x55bd795427e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feecdd9b640 1 --2- 172.21.15.76:0/3347536850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952ec30 0x55bd795427e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feecdd9b640 1 -- 172.21.15.76:0/3347536850 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bd794956a0 con 0x55bd7953cb60 2024-03-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feecdd9b640 1 --2- 172.21.15.76:0/3347536850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd795a31c0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7feec400bd80 tx=0x7feec40298a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b185595c912687 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feeb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feeb5ffb640 1 -- 172.21.15.76:0/3347536850 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec4039070 con 0x55bd7953cb60 2024-03-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feeb5ffb640 1 -- 172.21.15.76:0/3347536850 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7feec403e060 con 0x55bd7953cb60 2024-03-12T06:04:17.805 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.789+0000 7feed7b7f740 1 -- 172.21.15.76:0/3347536850 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bd795250f0 con 0x55bd7953cb60 2024-03-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.790+0000 7feeb5ffb640 1 -- 172.21.15.76:0/3347536850 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7feec401b040 con 0x55bd7953cb60 2024-03-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.790+0000 7feed7b7f740 10 client.17388 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.790+0000 7feed7b7f740 1 -- 172.21.15.76:0/3347536850 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bd794578c0 con 0x55bd7953cb60 2024-03-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.790+0000 7feeb5ffb640 1 -- 172.21.15.76:0/3347536850 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feec4046020 con 0x55bd7953cb60 2024-03-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.790+0000 7feeb5ffb640 1 -- 172.21.15.76:0/3347536850 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7feec4020440 con 0x55bd7953cb60 2024-03-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.790+0000 7feed7b7f740 10 client.17388 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.790+0000 7feed7b7f740 10 client.17388 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.790+0000 7feed7b7f740 20 client.17388 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:17.806 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.790+0000 7feed7b7f740 1 --2- 172.21.15.76:0/3347536850 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55bd795a4b20 0x55bd795a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.790+0000 7feed7b7f740 4 client.17388 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:04:17.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.790+0000 7feed7b7f740 1 -- 172.21.15.76:0/3347536850 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7feec40178f0 con 0x55bd795a4b20 2024-03-12T06:04:17.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.790+0000 7feece59c640 1 --2- 172.21.15.76:0/3347536850 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55bd795a4b20 0x55bd795a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.790+0000 7feeb5ffb640 10 client.17388 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:04:17.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.790+0000 7feece59c640 1 --2- 172.21.15.76:0/3347536850 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55bd795a4b20 0x55bd795a6f40 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=18cd869a574a5341 server_cookie=9e429801530b22d0 in_seq=0 out_seq=0 2024-03-12T06:04:17.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feeb5ffb640 1 -- 172.21.15.76:0/3347536850 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7feec40178f0 con 0x55bd795a4b20 2024-03-12T06:04:17.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feeb5ffb640 10 client.17388 handle_command_reply: tid=0 2024-03-12T06:04:17.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed7b7f740 1 client.17388 shutdown 2024-03-12T06:04:17.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed7b7f740 1 -- 172.21.15.76:0/3347536850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 msgr2=0x55bd795a31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.807 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed7b7f740 1 --2- 172.21.15.76:0/3347536850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd795a31c0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7feec400bd80 tx=0x7feec40298a0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed7b7f740 1 -- 172.21.15.76:0/3347536850 shutdown_connections 2024-03-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed7b7f740 1 --2- 172.21.15.76:0/3347536850 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55bd795a4b20 0x55bd795a6f40 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed7b7f740 1 --2- 172.21.15.76:0/3347536850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55bd7953cb60 0x55bd795a31c0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feeb5ffb640 0 client.17388 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed7b7f740 1 --2- 172.21.15.76:0/3347536850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd7952f5e0 0x55bd79542d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed7b7f740 1 --2- 172.21.15.76:0/3347536850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55bd7952ec30 0x55bd795427e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed7b7f740 1 -- 172.21.15.76:0/3347536850 >> 172.21.15.76:0/3347536850 conn(0x55bd79529980 msgr2=0x55bd79528b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.808 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed7b7f740 1 -- 172.21.15.76:0/3347536850 shutdown_connections 2024-03-12T06:04:17.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed7b7f740 1 -- 172.21.15.76:0/3347536850 wait complete. 2024-03-12T06:04:17.809 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed7b7f740 20 client.17388 trim_cache size 0 max 16384 2024-03-12T06:04:17.809 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:04:17.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed4fac640 1 -- 172.21.15.76:0/3476715464 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7feeb80a1ba0 msgr2=0x7feeb80a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed4fac640 1 --2- 172.21.15.76:0/3476715464 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7feeb80a1ba0 0x7feeb80a4060 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feec80cb850 tx=0x7feec806df30 comp rx=0 tx=0).stop 2024-03-12T06:04:17.810 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed4fac640 1 -- 172.21.15.76:0/3476715464 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feed0197710 msgr2=0x7feed0197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed4fac640 1 --2- 172.21.15.76:0/3476715464 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feed0197710 0x7feed0197b90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7feec800cce0 tx=0x7feec800a620 comp rx=0 tx=0).stop 2024-03-12T06:04:17.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed4fac640 1 -- 172.21.15.76:0/3476715464 shutdown_connections 2024-03-12T06:04:17.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed4fac640 1 --2- 172.21.15.76:0/3476715464 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7feeb80a1ba0 0x7feeb80a4060 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed4fac640 1 --2- 172.21.15.76:0/3476715464 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feed0197710 0x7feed0197b90 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed4fac640 1 --2- 172.21.15.76:0/3476715464 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feed0105460 0x7feed019db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed4fac640 1 --2- 172.21.15.76:0/3476715464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feed0100280 0x7feed019d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.791+0000 7feed4fac640 1 -- 172.21.15.76:0/3476715464 >> 172.21.15.76:0/3476715464 conn(0x7feed00771a0 msgr2=0x7feed0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.811 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.792+0000 7feed4fac640 1 -- 172.21.15.76:0/3476715464 shutdown_connections 2024-03-12T06:04:17.812 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.792+0000 7feed4fac640 1 -- 172.21.15.76:0/3476715464 wait complete. 2024-03-12T06:04:17.812 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:04:17.812 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T06:04:17.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.912+0000 7f5a4a440640 1 -- 172.21.15.76:0/3640051101 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a44102cc0 msgr2=0x7f5a4410b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.912+0000 7f5a4a440640 1 --2- 172.21.15.76:0/3640051101 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a44102cc0 0x7f5a4410b1a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f5a38009900 tx=0x7f5a3802bcd0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.912+0000 7f5a4a440640 1 -- 172.21.15.76:0/3640051101 shutdown_connections 2024-03-12T06:04:17.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.912+0000 7f5a4a440640 1 --2- 172.21.15.76:0/3640051101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a4410b770 0x7f5a4410dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.912+0000 7f5a4a440640 1 --2- 172.21.15.76:0/3640051101 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a44102cc0 0x7f5a4410b1a0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.917 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.912+0000 7f5a4a440640 1 --2- 172.21.15.76:0/3640051101 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a441023a0 0x7f5a44102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.912+0000 7f5a4a440640 1 -- 172.21.15.76:0/3640051101 >> 172.21.15.76:0/3640051101 conn(0x7f5a44077230 msgr2=0x7f5a44077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.912+0000 7f5a4a440640 1 -- 172.21.15.76:0/3640051101 shutdown_connections 2024-03-12T06:04:17.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.912+0000 7f5a4a440640 1 -- 172.21.15.76:0/3640051101 wait complete. 2024-03-12T06:04:17.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.912+0000 7f5a4a440640 1 Processor -- start 2024-03-12T06:04:17.918 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.912+0000 7f5a4a440640 1 -- start start 2024-03-12T06:04:17.926 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a4a440640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a441023a0 0x7f5a4419d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a4a440640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a44102cc0 0x7f5a4419d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a4a440640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a4410b770 0x7f5a441975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a4a440640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a44197c20 con 0x7f5a441023a0 2024-03-12T06:04:17.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a4a440640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a44197d90 con 0x7f5a4410b770 2024-03-12T06:04:17.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a4a440640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a44197f50 con 0x7f5a44102cc0 2024-03-12T06:04:17.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a43fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a441023a0 0x7f5a4419d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a43fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a441023a0 0x7f5a4419d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37674/0 (socket says 172.21.15.76:37674) 2024-03-12T06:04:17.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a43fff640 1 -- 172.21.15.76:0/3100662529 learned_addr learned my addr 172.21.15.76:0/3100662529 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.927 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a437fe640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a44102cc0 0x7f5a4419d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a43fff640 1 -- 172.21.15.76:0/3100662529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a44102cc0 msgr2=0x7f5a4419d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a43fff640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a44102cc0 0x7f5a4419d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a43fff640 1 -- 172.21.15.76:0/3100662529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a4410b770 msgr2=0x7f5a441975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a489dd640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a4410b770 0x7f5a441975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a43fff640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a4410b770 0x7f5a441975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a43fff640 1 -- 172.21.15.76:0/3100662529 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a38009560 con 0x7f5a441023a0 2024-03-12T06:04:17.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a437fe640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a44102cc0 0x7f5a4419d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:17.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a43fff640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a441023a0 0x7f5a4419d420 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400d950 tx=0x7f5a3400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a417fa640 1 -- 172.21.15.76:0/3100662529 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a34009c30 con 0x7f5a441023a0 2024-03-12T06:04:17.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a4a440640 1 -- 172.21.15.76:0/3100662529 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a44198230 con 0x7f5a441023a0 2024-03-12T06:04:17.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a417fa640 1 -- 172.21.15.76:0/3100662529 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a34009d90 con 0x7f5a441023a0 2024-03-12T06:04:17.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a417fa640 1 -- 172.21.15.76:0/3100662529 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3400f7e0 con 0x7f5a441023a0 2024-03-12T06:04:17.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.913+0000 7f5a4a440640 1 -- 172.21.15.76:0/3100662529 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a441a44c0 con 0x7f5a441023a0 2024-03-12T06:04:17.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.915+0000 7f5a417fa640 1 -- 172.21.15.76:0/3100662529 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5a34010ad0 con 0x7f5a441023a0 2024-03-12T06:04:17.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a4d012740 1 Processor -- start 2024-03-12T06:04:17.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a4d012740 1 -- start start 2024-03-12T06:04:17.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a417fa640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5a2c0a1ba0 0x7f5a2c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a4d012740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68250 0x55c434c68630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a4d012740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c68b70 0x55c434c71210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a4d012740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c71750 0x55c434c73b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a4d012740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c434c23960 con 0x55c434c68b70 2024-03-12T06:04:17.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a4d012740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c434bd7310 con 0x55c434c68250 2024-03-12T06:04:17.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a4d012740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c434bd4840 con 0x55c434c71750 2024-03-12T06:04:17.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a489dd640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5a2c0a1ba0 0x7f5a2c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a417fa640 1 -- 172.21.15.76:0/3100662529 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5a340cd540 con 0x7f5a441023a0 2024-03-12T06:04:17.930 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c68b70 0x55c434c71210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a437fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c68b70 0x55c434c71210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37686/0 (socket says 172.21.15.76:37686) 2024-03-12T06:04:17.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a437fe640 1 -- 172.21.15.76:0/729803672 learned_addr learned my addr 172.21.15.76:0/729803672 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a489dd640 1 --2- 172.21.15.76:0/729803672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c71750 0x55c434c73b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a43fff640 1 --2- 172.21.15.76:0/729803672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68250 0x55c434c68630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a437fe640 1 -- 172.21.15.76:0/729803672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c71750 msgr2=0x55c434c73b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a437fe640 1 --2- 172.21.15.76:0/729803672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c71750 0x55c434c73b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a437fe640 1 -- 172.21.15.76:0/729803672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68250 msgr2=0x55c434c68630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.931 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a437fe640 1 --2- 172.21.15.76:0/729803672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68250 0x55c434c68630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.916+0000 7f5a437fe640 1 -- 172.21.15.76:0/729803672 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c434b92060 con 0x55c434c68b70 2024-03-12T06:04:17.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.917+0000 7f5a489dd640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5a2c0a1ba0 0x7f5a2c0a4060 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0092f0 tx=0x7f5a3c013a60 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.917+0000 7f5a437fe640 1 --2- 172.21.15.76:0/729803672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c68b70 0x55c434c71210 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f5a3802c210 tx=0x7f5a380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73ca411df8020e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.917+0000 7f5a2a7fc640 1 -- 172.21.15.76:0/729803672 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38004a30 con 0x55c434c68b70 2024-03-12T06:04:17.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.917+0000 7f5a2a7fc640 1 -- 172.21.15.76:0/729803672 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a38004040 con 0x55c434c68b70 2024-03-12T06:04:17.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.917+0000 7f5a2a7fc640 1 -- 172.21.15.76:0/729803672 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38005330 con 0x55c434c68b70 2024-03-12T06:04:17.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.917+0000 7f5a4d012740 1 -- 172.21.15.76:0/729803672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c68b70 msgr2=0x55c434c71210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.917+0000 7f5a4d012740 1 --2- 172.21.15.76:0/729803672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c68b70 0x55c434c71210 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f5a3802c210 tx=0x7f5a380046c0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.932 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.917+0000 7f5a4d012740 1 -- 172.21.15.76:0/729803672 shutdown_connections 2024-03-12T06:04:17.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.917+0000 7f5a4d012740 1 --2- 172.21.15.76:0/729803672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c71750 0x55c434c73b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.917+0000 7f5a4d012740 1 --2- 172.21.15.76:0/729803672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c68b70 0x55c434c71210 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.917+0000 7f5a4d012740 1 --2- 172.21.15.76:0/729803672 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68250 0x55c434c68630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.917+0000 7f5a4d012740 1 -- 172.21.15.76:0/729803672 >> 172.21.15.76:0/729803672 conn(0x55c434c62980 msgr2=0x55c434c62d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.918+0000 7f5a4d012740 1 -- 172.21.15.76:0/729803672 shutdown_connections 2024-03-12T06:04:17.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.918+0000 7f5a4d012740 1 -- 172.21.15.76:0/729803672 wait complete. 2024-03-12T06:04:17.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.918+0000 7f5a4d012740 1 -- start start 2024-03-12T06:04:17.933 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.918+0000 7f5a4d012740 1 Processor -- start 2024-03-12T06:04:17.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.918+0000 7f5a4d012740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68250 0x55c434c6e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.918+0000 7f5a4d012740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c68b70 0x55c434c6ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.918+0000 7f5a4d012740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c71750 0x55c434c6f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.918+0000 7f5a4d012740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c434bd0020 con 0x55c434c71750 2024-03-12T06:04:17.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.918+0000 7f5a4d012740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4348a3430 con 0x55c434c68250 2024-03-12T06:04:17.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.918+0000 7f5a4d012740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c43485fe00 con 0x55c434c68b70 2024-03-12T06:04:17.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.918+0000 7f5a4d012740 10 client.0 _ll_register_callbacks cb 0x55c434b77cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:17.934 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.918+0000 7f5a489dd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c71750 0x55c434c6f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.918+0000 7f5a489dd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c71750 0x55c434c6f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37702/0 (socket says 172.21.15.76:37702) 2024-03-12T06:04:17.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.918+0000 7f5a489dd640 1 -- 172.21.15.76:0/401679133 learned_addr learned my addr 172.21.15.76:0/401679133 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a437fe640 1 --2- 172.21.15.76:0/401679133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c68b70 0x55c434c6ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a43fff640 1 --2- 172.21.15.76:0/401679133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68250 0x55c434c6e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a489dd640 1 -- 172.21.15.76:0/401679133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c68b70 msgr2=0x55c434c6ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a489dd640 1 --2- 172.21.15.76:0/401679133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c68b70 0x55c434c6ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a489dd640 1 -- 172.21.15.76:0/401679133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68250 msgr2=0x55c434c6e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a489dd640 1 --2- 172.21.15.76:0/401679133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68250 0x55c434c6e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.935 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a489dd640 1 -- 172.21.15.76:0/401679133 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4410e820 con 0x55c434c71750 2024-03-12T06:04:17.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a43fff640 1 --2- 172.21.15.76:0/401679133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68250 0x55c434c6e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:17.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a489dd640 1 --2- 172.21.15.76:0/401679133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c71750 0x55c434c6f200 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c003ac0 tx=0x7f5a3c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fedf25c95cdae051 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a237fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:17.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a237fe640 1 -- 172.21.15.76:0/401679133 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c011e50 con 0x55c434c71750 2024-03-12T06:04:17.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a237fe640 1 -- 172.21.15.76:0/401679133 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5a3c004b20 con 0x55c434c71750 2024-03-12T06:04:17.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a4d012740 1 -- 172.21.15.76:0/401679133 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c4348cf7e0 con 0x55c434c71750 2024-03-12T06:04:17.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a237fe640 1 -- 172.21.15.76:0/401679133 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a3c004ce0 con 0x55c434c71750 2024-03-12T06:04:17.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a4d012740 10 client.17400 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:17.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a4d012740 1 -- 172.21.15.76:0/401679133 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c434bce6a0 con 0x55c434c71750 2024-03-12T06:04:17.936 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a237fe640 1 -- 172.21.15.76:0/401679133 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c4348cf7e0 con 0x55c434c71750 2024-03-12T06:04:17.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.919+0000 7f5a237fe640 1 -- 172.21.15.76:0/401679133 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5a3c0053f0 con 0x55c434c71750 2024-03-12T06:04:17.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.920+0000 7f5a4d012740 10 client.17400 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:17.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.920+0000 7f5a4d012740 10 client.17400 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:04:17.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.920+0000 7f5a4d012740 20 client.17400 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:17.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.920+0000 7f5a4d012740 1 --2- 172.21.15.76:0/401679133 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55c434c81a30 0x55c434c83e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.920+0000 7f5a4d012740 4 client.17400 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:04:17.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.920+0000 7f5a4d012740 1 -- 172.21.15.76:0/401679133 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c43490c720 con 0x55c434c81a30 2024-03-12T06:04:17.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.920+0000 7f5a43fff640 1 --2- 172.21.15.76:0/401679133 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55c434c81a30 0x55c434c83e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.937 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.920+0000 7f5a43fff640 1 --2- 172.21.15.76:0/401679133 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55c434c81a30 0x55c434c83e20 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4b58c736359b8b7e server_cookie=a49de9676417e2d1 in_seq=0 out_seq=0 2024-03-12T06:04:17.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.920+0000 7f5a237fe640 10 client.17400 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:04:17.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a237fe640 1 -- 172.21.15.76:0/401679133 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c43490c720 con 0x55c434c81a30 2024-03-12T06:04:17.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a237fe640 10 client.17400 handle_command_reply: tid=0 2024-03-12T06:04:17.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a4d012740 1 client.17400 shutdown 2024-03-12T06:04:17.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a4d012740 1 -- 172.21.15.76:0/401679133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c71750 msgr2=0x55c434c6f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a4d012740 1 --2- 172.21.15.76:0/401679133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c71750 0x55c434c6f200 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c003ac0 tx=0x7f5a3c004ab0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a4d012740 1 -- 172.21.15.76:0/401679133 shutdown_connections 2024-03-12T06:04:17.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a4d012740 1 --2- 172.21.15.76:0/401679133 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55c434c81a30 0x55c434c83e20 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.938 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a4d012740 1 --2- 172.21.15.76:0/401679133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c71750 0x55c434c6f200 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a4d012740 1 --2- 172.21.15.76:0/401679133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c68b70 0x55c434c6ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a4d012740 1 --2- 172.21.15.76:0/401679133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68250 0x55c434c6e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a237fe640 0 client.17400 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:04:17.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a4d012740 1 -- 172.21.15.76:0/401679133 >> 172.21.15.76:0/401679133 conn(0x55c434c62980 msgr2=0x55c434c646d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a4d012740 1 -- 172.21.15.76:0/401679133 shutdown_connections 2024-03-12T06:04:17.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a4d012740 1 -- 172.21.15.76:0/401679133 wait complete. 2024-03-12T06:04:17.939 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.921+0000 7f5a4d012740 20 client.17400 trim_cache size 0 max 16384 2024-03-12T06:04:17.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a4d012740 1 Processor -- start 2024-03-12T06:04:17.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a4d012740 1 -- start start 2024-03-12T06:04:17.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a4d012740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c68250 0x55c434c61a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a4d012740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c68b70 0x55c434c78480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a4d012740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c71750 0x55c434c789c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a4d012740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c434b55130 con 0x55c434c68250 2024-03-12T06:04:17.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a4d012740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c43490c720 con 0x55c434c71750 2024-03-12T06:04:17.945 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a4d012740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c434c54a40 con 0x55c434c68b70 2024-03-12T06:04:17.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a43fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c68250 0x55c434c61a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a43fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c68250 0x55c434c61a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37714/0 (socket says 172.21.15.76:37714) 2024-03-12T06:04:17.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a43fff640 1 -- 172.21.15.76:0/1903996414 learned_addr learned my addr 172.21.15.76:0/1903996414 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a43fff640 1 -- 172.21.15.76:0/1903996414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c68b70 msgr2=0x55c434c78480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a43fff640 1 --2- 172.21.15.76:0/1903996414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c68b70 0x55c434c78480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a43fff640 1 -- 172.21.15.76:0/1903996414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c71750 msgr2=0x55c434c789c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a43fff640 1 --2- 172.21.15.76:0/1903996414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c71750 0x55c434c789c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.946 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a43fff640 1 -- 172.21.15.76:0/1903996414 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c434b6ac20 con 0x55c434c68250 2024-03-12T06:04:17.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a43fff640 1 --2- 172.21.15.76:0/1903996414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c68250 0x55c434c61a00 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f5a3409a100 tx=0x7f5a3400b530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d0c841c6d60b38 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a29ffb640 1 -- 172.21.15.76:0/1903996414 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3409a170 con 0x55c434c68250 2024-03-12T06:04:17.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a29ffb640 1 -- 172.21.15.76:0/1903996414 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a3409e2c0 con 0x55c434c68250 2024-03-12T06:04:17.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.933+0000 7f5a29ffb640 1 -- 172.21.15.76:0/1903996414 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a340967e0 con 0x55c434c68250 2024-03-12T06:04:17.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.934+0000 7f5a4d012740 1 -- 172.21.15.76:0/1903996414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c68250 msgr2=0x55c434c61a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.934+0000 7f5a4d012740 1 --2- 172.21.15.76:0/1903996414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c68250 0x55c434c61a00 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f5a3409a100 tx=0x7f5a3400b530 comp rx=0 tx=0).stop 2024-03-12T06:04:17.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.934+0000 7f5a4d012740 1 -- 172.21.15.76:0/1903996414 shutdown_connections 2024-03-12T06:04:17.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.934+0000 7f5a4d012740 1 --2- 172.21.15.76:0/1903996414 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c71750 0x55c434c789c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.947 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.934+0000 7f5a4d012740 1 --2- 172.21.15.76:0/1903996414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c68b70 0x55c434c78480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.934+0000 7f5a4d012740 1 --2- 172.21.15.76:0/1903996414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c68250 0x55c434c61a00 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.934+0000 7f5a4d012740 1 -- 172.21.15.76:0/1903996414 >> 172.21.15.76:0/1903996414 conn(0x55c434c62980 msgr2=0x55c434c612f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.934+0000 7f5a4d012740 1 -- 172.21.15.76:0/1903996414 shutdown_connections 2024-03-12T06:04:17.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.934+0000 7f5a4d012740 1 -- 172.21.15.76:0/1903996414 wait complete. 2024-03-12T06:04:17.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.934+0000 7f5a4d012740 1 -- start start 2024-03-12T06:04:17.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a4d012740 1 Processor -- start 2024-03-12T06:04:17.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a4d012740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c68250 0x55c434cd11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a4d012740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68b70 0x55c434cd1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a4d012740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c71750 0x55c434c77720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.948 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a4d012740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c434c77ce0 con 0x55c434c71750 2024-03-12T06:04:17.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a4d012740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c434c77e50 con 0x55c434c68b70 2024-03-12T06:04:17.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a4d012740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c434c78010 con 0x55c434c68250 2024-03-12T06:04:17.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a4d012740 10 client.0 _ll_register_callbacks cb 0x7f5a38009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:17.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a489dd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c71750 0x55c434c77720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a489dd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c71750 0x55c434c77720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37726/0 (socket says 172.21.15.76:37726) 2024-03-12T06:04:17.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a489dd640 1 -- 172.21.15.76:0/2749172843 learned_addr learned my addr 172.21.15.76:0/2749172843 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:17.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a437fe640 1 --2- 172.21.15.76:0/2749172843 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68b70 0x55c434cd1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a43fff640 1 --2- 172.21.15.76:0/2749172843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c68250 0x55c434cd11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.949 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a489dd640 1 -- 172.21.15.76:0/2749172843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c68250 msgr2=0x55c434cd11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a489dd640 1 --2- 172.21.15.76:0/2749172843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c68250 0x55c434cd11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a489dd640 1 -- 172.21.15.76:0/2749172843 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68b70 msgr2=0x55c434cd1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:17.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a489dd640 1 --2- 172.21.15.76:0/2749172843 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68b70 0x55c434cd1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a489dd640 1 -- 172.21.15.76:0/2749172843 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a441a44c0 con 0x55c434c71750 2024-03-12T06:04:17.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a489dd640 1 --2- 172.21.15.76:0/2749172843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c71750 0x55c434c77720 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c005820 tx=0x7f5a3c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e330460edfd166dd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:17.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a237fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:17.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a237fe640 1 -- 172.21.15.76:0/2749172843 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c011e50 con 0x55c434c71750 2024-03-12T06:04:17.950 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.935+0000 7f5a237fe640 1 -- 172.21.15.76:0/2749172843 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5a3c005990 con 0x55c434c71750 2024-03-12T06:04:17.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.936+0000 7f5a4d012740 1 -- 172.21.15.76:0/2749172843 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a38004040 con 0x55c434c71750 2024-03-12T06:04:17.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.936+0000 7f5a237fe640 1 -- 172.21.15.76:0/2749172843 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a3c004250 con 0x55c434c71750 2024-03-12T06:04:17.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.936+0000 7f5a4d012740 10 client.17406 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:17.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.936+0000 7f5a4d012740 1 -- 172.21.15.76:0/2749172843 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c434b908c0 con 0x55c434c71750 2024-03-12T06:04:17.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.936+0000 7f5a237fe640 1 -- 172.21.15.76:0/2749172843 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a38004040 con 0x55c434c71750 2024-03-12T06:04:17.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.936+0000 7f5a237fe640 1 -- 172.21.15.76:0/2749172843 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5a3c020c70 con 0x55c434c71750 2024-03-12T06:04:17.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.936+0000 7f5a4d012740 10 client.17406 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:17.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.936+0000 7f5a4d012740 10 client.17406 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:04:17.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.936+0000 7f5a4d012740 20 client.17406 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:17.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.936+0000 7f5a4d012740 1 --2- 172.21.15.76:0/2749172843 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55c434cda170 0x55c434cdc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:17.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.936+0000 7f5a4d012740 4 client.17406 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:04:17.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.936+0000 7f5a4d012740 1 -- 172.21.15.76:0/2749172843 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c43495b890 con 0x55c434cda170 2024-03-12T06:04:17.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.936+0000 7f5a43fff640 1 --2- 172.21.15.76:0/2749172843 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55c434cda170 0x55c434cdc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:17.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a237fe640 10 client.17406 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:04:17.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a43fff640 1 --2- 172.21.15.76:0/2749172843 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55c434cda170 0x55c434cdc590 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e96e2dce5b4daac5 server_cookie=9a8e66ce8bc7cd45 in_seq=0 out_seq=0 2024-03-12T06:04:17.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a237fe640 1 -- 172.21.15.76:0/2749172843 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c43495b890 con 0x55c434cda170 2024-03-12T06:04:17.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a237fe640 10 client.17406 handle_command_reply: tid=0 2024-03-12T06:04:17.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a4d012740 1 client.17406 shutdown 2024-03-12T06:04:17.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a4d012740 1 -- 172.21.15.76:0/2749172843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c71750 msgr2=0x55c434c77720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a4d012740 1 --2- 172.21.15.76:0/2749172843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c71750 0x55c434c77720 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c005820 tx=0x7f5a3c011a50 comp rx=0 tx=0).stop 2024-03-12T06:04:17.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a4d012740 1 -- 172.21.15.76:0/2749172843 shutdown_connections 2024-03-12T06:04:17.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a4d012740 1 --2- 172.21.15.76:0/2749172843 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x55c434cda170 0x55c434cdc590 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a4d012740 1 --2- 172.21.15.76:0/2749172843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c434c71750 0x55c434c77720 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a4d012740 1 --2- 172.21.15.76:0/2749172843 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c434c68b70 0x55c434cd1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a4d012740 1 --2- 172.21.15.76:0/2749172843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c434c68250 0x55c434cd11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a237fe640 0 client.17406 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:04:17.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a4d012740 1 -- 172.21.15.76:0/2749172843 >> 172.21.15.76:0/2749172843 conn(0x55c434c62980 msgr2=0x55c434c61b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a4d012740 1 -- 172.21.15.76:0/2749172843 shutdown_connections 2024-03-12T06:04:17.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a4d012740 1 -- 172.21.15.76:0/2749172843 wait complete. 2024-03-12T06:04:17.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.937+0000 7f5a4d012740 20 client.17406 trim_cache size 0 max 16384 2024-03-12T06:04:17.954 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:04:17.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.938+0000 7f5a4a440640 1 -- 172.21.15.76:0/3100662529 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5a2c0a1ba0 msgr2=0x7f5a2c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.938+0000 7f5a4a440640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5a2c0a1ba0 0x7f5a2c0a4060 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0092f0 tx=0x7f5a3c013a60 comp rx=0 tx=0).stop 2024-03-12T06:04:17.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.938+0000 7f5a4a440640 1 -- 172.21.15.76:0/3100662529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a441023a0 msgr2=0x7f5a4419d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:17.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.938+0000 7f5a4a440640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a441023a0 0x7f5a4419d420 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400d950 tx=0x7f5a3400de10 comp rx=0 tx=0).stop 2024-03-12T06:04:17.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.938+0000 7f5a4a440640 1 -- 172.21.15.76:0/3100662529 shutdown_connections 2024-03-12T06:04:17.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.938+0000 7f5a4a440640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5a2c0a1ba0 0x7f5a2c0a4060 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.938+0000 7f5a4a440640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5a4410b770 0x7f5a441975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.938+0000 7f5a4a440640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a44102cc0 0x7f5a4419d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.938+0000 7f5a4a440640 1 --2- 172.21.15.76:0/3100662529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a441023a0 0x7f5a4419d420 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:17.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.938+0000 7f5a4a440640 1 -- 172.21.15.76:0/3100662529 >> 172.21.15.76:0/3100662529 conn(0x7f5a44077230 msgr2=0x7f5a440fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:17.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.938+0000 7f5a4a440640 1 -- 172.21.15.76:0/3100662529 shutdown_connections 2024-03-12T06:04:17.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:17.938+0000 7f5a4a440640 1 -- 172.21.15.76:0/3100662529 wait complete. 2024-03-12T06:04:17.957 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:04:17.957 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T06:04:18.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.058+0000 7fc07189d640 1 -- 172.21.15.76:0/2334564925 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc06c105370 msgr2=0x7fc06c10cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:18.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.058+0000 7fc07189d640 1 --2- 172.21.15.76:0/2334564925 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc06c105370 0x7fc06c10cf40 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fc064009cd0 tx=0x7fc06402c0f0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.058+0000 7fc07189d640 1 -- 172.21.15.76:0/2334564925 shutdown_connections 2024-03-12T06:04:18.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.058+0000 7fc07189d640 1 --2- 172.21.15.76:0/2334564925 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc06c105370 0x7fc06c10cf40 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.058+0000 7fc07189d640 1 --2- 172.21.15.76:0/2334564925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc06c100ca0 0x7fc06c104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.058+0000 7fc07189d640 1 --2- 172.21.15.76:0/2334564925 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc06c1002f0 0x7fc06c1006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.058+0000 7fc07189d640 1 -- 172.21.15.76:0/2334564925 >> 172.21.15.76:0/2334564925 conn(0x7fc06c077250 msgr2=0x7fc06c077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:18.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.058+0000 7fc07189d640 1 -- 172.21.15.76:0/2334564925 shutdown_connections 2024-03-12T06:04:18.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.058+0000 7fc07189d640 1 -- 172.21.15.76:0/2334564925 wait complete. 2024-03-12T06:04:18.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.058+0000 7fc07189d640 1 Processor -- start 2024-03-12T06:04:18.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc07189d640 1 -- start start 2024-03-12T06:04:18.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc07189d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc06c1002f0 0x7fc06c19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc07189d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc06c105370 0x7fc06c19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc07189d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc06c1975f0 0x7fc06c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc07189d640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c197fe0 con 0x7fc06c1002f0 2024-03-12T06:04:18.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc07189d640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c198150 con 0x7fc06c105370 2024-03-12T06:04:18.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc07189d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c198310 con 0x7fc06c1975f0 2024-03-12T06:04:18.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc06affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc06c1002f0 0x7fc06c19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc06affd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc06c1002f0 0x7fc06c19d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37730/0 (socket says 172.21.15.76:37730) 2024-03-12T06:04:18.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc06affd640 1 -- 172.21.15.76:0/1683187719 learned_addr learned my addr 172.21.15.76:0/1683187719 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:18.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc06b7fe640 1 --2- 172.21.15.76:0/1683187719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc06c1975f0 0x7fc06c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc06a7fc640 1 --2- 172.21.15.76:0/1683187719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc06c105370 0x7fc06c19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc06affd640 1 -- 172.21.15.76:0/1683187719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc06c1975f0 msgr2=0x7fc06c197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:18.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc06affd640 1 --2- 172.21.15.76:0/1683187719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc06c1975f0 0x7fc06c197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc06affd640 1 -- 172.21.15.76:0/1683187719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc06c105370 msgr2=0x7fc06c19d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:18.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc06affd640 1 --2- 172.21.15.76:0/1683187719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc06c105370 0x7fc06c19d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc06affd640 1 -- 172.21.15.76:0/1683187719 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc064009900 con 0x7fc06c1002f0 2024-03-12T06:04:18.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc06affd640 1 --2- 172.21.15.76:0/1683187719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc06c1002f0 0x7fc06c19d3f0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00e9f0 tx=0x7fc05c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:18.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc07089b640 1 -- 172.21.15.76:0/1683187719 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c009700 con 0x7fc06c1002f0 2024-03-12T06:04:18.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc07089b640 1 -- 172.21.15.76:0/1683187719 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc05c00cd80 con 0x7fc06c1002f0 2024-03-12T06:04:18.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc07189d640 1 -- 172.21.15.76:0/1683187719 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06c1a42b0 con 0x7fc06c1002f0 2024-03-12T06:04:18.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc07089b640 1 -- 172.21.15.76:0/1683187719 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c018920 con 0x7fc06c1002f0 2024-03-12T06:04:18.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.059+0000 7fc07189d640 1 -- 172.21.15.76:0/1683187719 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06c1a47c0 con 0x7fc06c1002f0 2024-03-12T06:04:18.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc07089b640 1 -- 172.21.15.76:0/1683187719 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc05c018a80 con 0x7fc06c1002f0 2024-03-12T06:04:18.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc074478740 1 Processor -- start 2024-03-12T06:04:18.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc074478740 1 -- start start 2024-03-12T06:04:18.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc07089b640 1 --2- 172.21.15.76:0/1683187719 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc0540a1ba0 0x7fc0540a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc074478740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cb250 0x5638297cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc074478740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297d4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc074478740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297d4750 0x5638297d6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc074478740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563829786960 con 0x5638297cbb70 2024-03-12T06:04:18.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc074478740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382973a310 con 0x5638297cb250 2024-03-12T06:04:18.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc074478740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563829737840 con 0x5638297d4750 2024-03-12T06:04:18.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc06b7fe640 1 --2- 172.21.15.76:0/1683187719 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc0540a1ba0 0x7fc0540a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc07089b640 1 -- 172.21.15.76:0/1683187719 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fc05c014070 con 0x7fc06c1002f0 2024-03-12T06:04:18.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc06a7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc06a7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297d4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37736/0 (socket says 172.21.15.76:37736) 2024-03-12T06:04:18.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.062+0000 7fc06a7fc640 1 -- 172.21.15.76:0/1648809381 learned_addr learned my addr 172.21.15.76:0/1648809381 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:18.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.063+0000 7fc06affd640 1 --2- 172.21.15.76:0/1648809381 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cb250 0x5638297cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.063+0000 7fc06b7fe640 1 --2- 172.21.15.76:0/1648809381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297d4750 0x5638297d6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.063+0000 7fc06a7fc640 1 -- 172.21.15.76:0/1648809381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297d4750 msgr2=0x5638297d6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:18.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.063+0000 7fc06a7fc640 1 --2- 172.21.15.76:0/1648809381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297d4750 0x5638297d6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.063+0000 7fc06a7fc640 1 -- 172.21.15.76:0/1648809381 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cb250 msgr2=0x5638297cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:18.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.063+0000 7fc06a7fc640 1 --2- 172.21.15.76:0/1648809381 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cb250 0x5638297cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.063+0000 7fc06a7fc640 1 -- 172.21.15.76:0/1648809381 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638296f5060 con 0x5638297cbb70 2024-03-12T06:04:18.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.063+0000 7fc06b7fe640 1 --2- 172.21.15.76:0/1683187719 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc0540a1ba0 0x7fc0540a4060 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc064005cc0 tx=0x7fc064005f10 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:18.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.063+0000 7fc06a7fc640 1 --2- 172.21.15.76:0/1648809381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297d4210 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fc060009ba0 tx=0x7fc060011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed96ea5ee7a76c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:18.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.063+0000 7fc051ffb640 1 -- 172.21.15.76:0/1648809381 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060004230 con 0x5638297cbb70 2024-03-12T06:04:18.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.063+0000 7fc051ffb640 1 -- 172.21.15.76:0/1648809381 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc060009e10 con 0x5638297cbb70 2024-03-12T06:04:18.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.063+0000 7fc051ffb640 1 -- 172.21.15.76:0/1648809381 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060004d00 con 0x5638297cbb70 2024-03-12T06:04:18.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 -- 172.21.15.76:0/1648809381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 msgr2=0x5638297d4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:18.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 --2- 172.21.15.76:0/1648809381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297d4210 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fc060009ba0 tx=0x7fc060011810 comp rx=0 tx=0).stop 2024-03-12T06:04:18.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 -- 172.21.15.76:0/1648809381 shutdown_connections 2024-03-12T06:04:18.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 --2- 172.21.15.76:0/1648809381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297d4750 0x5638297d6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 --2- 172.21.15.76:0/1648809381 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297d4210 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 --2- 172.21.15.76:0/1648809381 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cb250 0x5638297cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 -- 172.21.15.76:0/1648809381 >> 172.21.15.76:0/1648809381 conn(0x5638297c5980 msgr2=0x5638297c5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:18.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 -- 172.21.15.76:0/1648809381 shutdown_connections 2024-03-12T06:04:18.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 -- 172.21.15.76:0/1648809381 wait complete. 2024-03-12T06:04:18.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 -- start start 2024-03-12T06:04:18.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 Processor -- start 2024-03-12T06:04:18.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cb250 0x5638297d17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297d1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297d4750 0x5638297d2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563829733020 con 0x5638297cbb70 2024-03-12T06:04:18.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563829406430 con 0x5638297cb250 2024-03-12T06:04:18.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.064+0000 7fc074478740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638293c2e00 con 0x5638297d4750 2024-03-12T06:04:18.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc074478740 10 client.0 _ll_register_callbacks cb 0x5638296dacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:18.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc06a7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297d1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc06a7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297d1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37748/0 (socket says 172.21.15.76:37748) 2024-03-12T06:04:18.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc06a7fc640 1 -- 172.21.15.76:0/1746103867 learned_addr learned my addr 172.21.15.76:0/1746103867 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:18.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc06b7fe640 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297d4750 0x5638297d2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc06affd640 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cb250 0x5638297d17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc06a7fc640 1 -- 172.21.15.76:0/1746103867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297d4750 msgr2=0x5638297d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:18.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc06a7fc640 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297d4750 0x5638297d2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc06a7fc640 1 -- 172.21.15.76:0/1746103867 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cb250 msgr2=0x5638297d17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:18.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc06a7fc640 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cb250 0x5638297d17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc06a7fc640 1 -- 172.21.15.76:0/1746103867 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc060009850 con 0x5638297cbb70 2024-03-12T06:04:18.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc06affd640 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cb250 0x5638297d17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:18.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc06b7fe640 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297d4750 0x5638297d2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:18.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc06a7fc640 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297d1d00 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fc0600052b0 tx=0x7fc0600052e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=755cfde402ced089 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:18.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc04affd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:18.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc04affd640 1 -- 172.21.15.76:0/1746103867 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060024670 con 0x5638297cbb70 2024-03-12T06:04:18.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc04affd640 1 -- 172.21.15.76:0/1746103867 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fc0600247d0 con 0x5638297cbb70 2024-03-12T06:04:18.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc074478740 1 -- 172.21.15.76:0/1746103867 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5638294327e0 con 0x5638297cbb70 2024-03-12T06:04:18.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc04affd640 1 -- 172.21.15.76:0/1746103867 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc060005500 con 0x5638297cbb70 2024-03-12T06:04:18.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc074478740 10 client.17418 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:18.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc074478740 1 -- 172.21.15.76:0/1746103867 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5638297316a0 con 0x5638297cbb70 2024-03-12T06:04:18.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.065+0000 7fc04affd640 1 -- 172.21.15.76:0/1746103867 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5638294327e0 con 0x5638297cbb70 2024-03-12T06:04:18.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.066+0000 7fc04affd640 1 -- 172.21.15.76:0/1746103867 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc060019990 con 0x5638297cbb70 2024-03-12T06:04:18.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.066+0000 7fc074478740 10 client.17418 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:18.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.066+0000 7fc074478740 10 client.17418 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:18.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.066+0000 7fc074478740 20 client.17418 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:18.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.066+0000 7fc074478740 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5638297e4c00 0x5638297e7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.066+0000 7fc074478740 4 client.17418 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:04:18.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.066+0000 7fc074478740 1 -- 172.21.15.76:0/1746103867 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56382946f720 con 0x5638297e4c00 2024-03-12T06:04:18.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.066+0000 7fc06affd640 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5638297e4c00 0x5638297e7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.066+0000 7fc04affd640 10 client.17418 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:18.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.066+0000 7fc06affd640 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5638297e4c00 0x5638297e7020 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a47f8dd0d27365b1 server_cookie=42579659ff42ceaa in_seq=0 out_seq=0 2024-03-12T06:04:18.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc04affd640 1 -- 172.21.15.76:0/1746103867 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56382946f720 con 0x5638297e4c00 2024-03-12T06:04:18.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc04affd640 10 client.17418 handle_command_reply: tid=0 2024-03-12T06:04:18.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc074478740 1 client.17418 shutdown 2024-03-12T06:04:18.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc074478740 1 -- 172.21.15.76:0/1746103867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 msgr2=0x5638297d1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:18.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc074478740 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297d1d00 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fc0600052b0 tx=0x7fc0600052e0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc074478740 1 -- 172.21.15.76:0/1746103867 shutdown_connections 2024-03-12T06:04:18.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc074478740 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297d4750 0x5638297d2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc074478740 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5638297e4c00 0x5638297e7020 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc074478740 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297d1d00 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc074478740 1 --2- 172.21.15.76:0/1746103867 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cb250 0x5638297d17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc04affd640 0 client.17418 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:18.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc074478740 1 -- 172.21.15.76:0/1746103867 >> 172.21.15.76:0/1746103867 conn(0x5638297c5980 msgr2=0x5638297c76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:18.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc074478740 1 -- 172.21.15.76:0/1746103867 shutdown_connections 2024-03-12T06:04:18.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc074478740 1 -- 172.21.15.76:0/1746103867 wait complete. 2024-03-12T06:04:18.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.067+0000 7fc074478740 20 client.17418 trim_cache size 0 max 16384 2024-03-12T06:04:18.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc074478740 1 Processor -- start 2024-03-12T06:04:18.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc074478740 1 -- start start 2024-03-12T06:04:18.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc074478740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297cb250 0x5638297d3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc074478740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cbb70 0x5638297d2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc074478740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297d4750 0x5638297ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc074478740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638296b8130 con 0x5638297d4750 2024-03-12T06:04:18.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc074478740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638297d7110 con 0x5638297cbb70 2024-03-12T06:04:18.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc074478740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638297ec230 con 0x5638297cb250 2024-03-12T06:04:18.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc06b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297d4750 0x5638297ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc06b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297d4750 0x5638297ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37760/0 (socket says 172.21.15.76:37760) 2024-03-12T06:04:18.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc06b7fe640 1 -- 172.21.15.76:0/547443245 learned_addr learned my addr 172.21.15.76:0/547443245 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:18.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc06affd640 1 --2- 172.21.15.76:0/547443245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297cb250 0x5638297d3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc06b7fe640 1 -- 172.21.15.76:0/547443245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297cb250 msgr2=0x5638297d3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:18.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc06b7fe640 1 --2- 172.21.15.76:0/547443245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297cb250 0x5638297d3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc06b7fe640 1 -- 172.21.15.76:0/547443245 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cbb70 msgr2=0x5638297d2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:18.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc06b7fe640 1 --2- 172.21.15.76:0/547443245 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cbb70 0x5638297d2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.079+0000 7fc06b7fe640 1 -- 172.21.15.76:0/547443245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638296cdc20 con 0x5638297d4750 2024-03-12T06:04:18.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.080+0000 7fc06b7fe640 1 --2- 172.21.15.76:0/547443245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297d4750 0x5638297ccaf0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fc064036000 tx=0x7fc064043040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abba4829f6ca1e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:18.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.080+0000 7fc0517fa640 1 -- 172.21.15.76:0/547443245 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc064034c90 con 0x5638297d4750 2024-03-12T06:04:18.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.080+0000 7fc0517fa640 1 -- 172.21.15.76:0/547443245 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc064034df0 con 0x5638297d4750 2024-03-12T06:04:18.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.080+0000 7fc0517fa640 1 -- 172.21.15.76:0/547443245 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0640042a0 con 0x5638297d4750 2024-03-12T06:04:18.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.080+0000 7fc074478740 1 -- 172.21.15.76:0/547443245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297d4750 msgr2=0x5638297ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:18.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.080+0000 7fc074478740 1 --2- 172.21.15.76:0/547443245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297d4750 0x5638297ccaf0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fc064036000 tx=0x7fc064043040 comp rx=0 tx=0).stop 2024-03-12T06:04:18.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.080+0000 7fc074478740 1 -- 172.21.15.76:0/547443245 shutdown_connections 2024-03-12T06:04:18.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.080+0000 7fc074478740 1 --2- 172.21.15.76:0/547443245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297d4750 0x5638297ccaf0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.080+0000 7fc074478740 1 --2- 172.21.15.76:0/547443245 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297cbb70 0x5638297d2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.080+0000 7fc074478740 1 --2- 172.21.15.76:0/547443245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297cb250 0x5638297d3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.080+0000 7fc074478740 1 -- 172.21.15.76:0/547443245 >> 172.21.15.76:0/547443245 conn(0x5638297c5980 msgr2=0x5638297c0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:18.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.080+0000 7fc074478740 1 -- 172.21.15.76:0/547443245 shutdown_connections 2024-03-12T06:04:18.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.080+0000 7fc074478740 1 -- 172.21.15.76:0/547443245 wait complete. 2024-03-12T06:04:18.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc074478740 1 -- start start 2024-03-12T06:04:18.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc074478740 1 Processor -- start 2024-03-12T06:04:18.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc074478740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297cb250 0x5638297da6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc074478740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297dac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc074478740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297d4750 0x56382983b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc074478740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382983b650 con 0x5638297cbb70 2024-03-12T06:04:18.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc074478740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382983b790 con 0x5638297d4750 2024-03-12T06:04:18.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc074478740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56382983b8d0 con 0x5638297cb250 2024-03-12T06:04:18.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc074478740 10 client.0 _ll_register_callbacks cb 0x7fc0600097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:18.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc06a7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297dac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc06a7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297dac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37772/0 (socket says 172.21.15.76:37772) 2024-03-12T06:04:18.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc06a7fc640 1 -- 172.21.15.76:0/3744551995 learned_addr learned my addr 172.21.15.76:0/3744551995 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:18.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc06affd640 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297cb250 0x5638297da6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc06b7fe640 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297d4750 0x56382983b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc06a7fc640 1 -- 172.21.15.76:0/3744551995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297cb250 msgr2=0x5638297da6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:18.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc06a7fc640 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297cb250 0x5638297da6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc06a7fc640 1 -- 172.21.15.76:0/3744551995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297d4750 msgr2=0x56382983b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:18.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc06a7fc640 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297d4750 0x56382983b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc06a7fc640 1 -- 172.21.15.76:0/3744551995 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06c069840 con 0x5638297cbb70 2024-03-12T06:04:18.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc06affd640 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297cb250 0x5638297da6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:18.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.081+0000 7fc06b7fe640 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297d4750 0x56382983b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:18.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc06a7fc640 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297dac30 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fc06001c040 tx=0x7fc060009af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f6458f6acafdb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:18.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc04affd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:18.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc04affd640 1 -- 172.21.15.76:0/3744551995 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0600052e0 con 0x5638297cbb70 2024-03-12T06:04:18.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc04affd640 1 -- 172.21.15.76:0/3744551995 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fc060000e40 con 0x5638297cbb70 2024-03-12T06:04:18.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc074478740 1 -- 172.21.15.76:0/3744551995 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc060009e10 con 0x5638297cbb70 2024-03-12T06:04:18.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc04affd640 1 -- 172.21.15.76:0/3744551995 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc060005440 con 0x5638297cbb70 2024-03-12T06:04:18.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc074478740 10 client.17424 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:18.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc074478740 1 -- 172.21.15.76:0/3744551995 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5638296f38c0 con 0x5638297cbb70 2024-03-12T06:04:18.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc04affd640 1 -- 172.21.15.76:0/3744551995 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc060009e10 con 0x5638297cbb70 2024-03-12T06:04:18.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc04affd640 1 -- 172.21.15.76:0/3744551995 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc06003d070 con 0x5638297cbb70 2024-03-12T06:04:18.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc074478740 10 client.17424 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:18.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc074478740 10 client.17424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:18.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc074478740 20 client.17424 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:18.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc074478740 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56382983cd70 0x56382983f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:18.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc074478740 4 client.17424 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:04:18.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc074478740 1 -- 172.21.15.76:0/3744551995 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5638294be890 con 0x56382983cd70 2024-03-12T06:04:18.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.082+0000 7fc06affd640 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56382983cd70 0x56382983f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:18.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc04affd640 10 client.17424 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:18.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc06affd640 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56382983cd70 0x56382983f190 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f29d35f894138e54 server_cookie=329278ff9246485 in_seq=0 out_seq=0 2024-03-12T06:04:18.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc04affd640 1 -- 172.21.15.76:0/3744551995 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5638294be890 con 0x56382983cd70 2024-03-12T06:04:18.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc04affd640 10 client.17424 handle_command_reply: tid=0 2024-03-12T06:04:18.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc074478740 1 client.17424 shutdown 2024-03-12T06:04:18.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc074478740 1 -- 172.21.15.76:0/3744551995 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 msgr2=0x5638297dac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:18.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc074478740 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297dac30 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fc06001c040 tx=0x7fc060009af0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc074478740 1 -- 172.21.15.76:0/3744551995 shutdown_connections 2024-03-12T06:04:18.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc074478740 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5638297d4750 0x56382983b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc074478740 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x56382983cd70 0x56382983f190 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc074478740 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5638297cbb70 0x5638297dac30 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc074478740 1 --2- 172.21.15.76:0/3744551995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638297cb250 0x5638297da6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc04affd640 0 client.17424 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:18.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc074478740 1 -- 172.21.15.76:0/3744551995 >> 172.21.15.76:0/3744551995 conn(0x5638297c5980 msgr2=0x5638297ccaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:18.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc074478740 1 -- 172.21.15.76:0/3744551995 shutdown_connections 2024-03-12T06:04:18.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc074478740 1 -- 172.21.15.76:0/3744551995 wait complete. 2024-03-12T06:04:18.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.083+0000 7fc074478740 20 client.17424 trim_cache size 0 max 16384 2024-03-12T06:04:18.100 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:04:18.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.084+0000 7fc07189d640 1 -- 172.21.15.76:0/1683187719 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc0540a1ba0 msgr2=0x7fc0540a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:18.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.084+0000 7fc07189d640 1 --2- 172.21.15.76:0/1683187719 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc0540a1ba0 0x7fc0540a4060 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc064005cc0 tx=0x7fc064005f10 comp rx=0 tx=0).stop 2024-03-12T06:04:18.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.084+0000 7fc07189d640 1 -- 172.21.15.76:0/1683187719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc06c1002f0 msgr2=0x7fc06c19d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:18.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.084+0000 7fc07189d640 1 --2- 172.21.15.76:0/1683187719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc06c1002f0 0x7fc06c19d3f0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00e9f0 tx=0x7fc05c00eeb0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.084+0000 7fc07189d640 1 -- 172.21.15.76:0/1683187719 shutdown_connections 2024-03-12T06:04:18.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.084+0000 7fc07189d640 1 --2- 172.21.15.76:0/1683187719 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc0540a1ba0 0x7fc0540a4060 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.084+0000 7fc07189d640 1 --2- 172.21.15.76:0/1683187719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc06c1975f0 0x7fc06c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.084+0000 7fc07189d640 1 --2- 172.21.15.76:0/1683187719 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc06c105370 0x7fc06c19d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.084+0000 7fc07189d640 1 --2- 172.21.15.76:0/1683187719 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc06c1002f0 0x7fc06c19d3f0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:18.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.084+0000 7fc07189d640 1 -- 172.21.15.76:0/1683187719 >> 172.21.15.76:0/1683187719 conn(0x7fc06c077250 msgr2=0x7fc06c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:18.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.084+0000 7fc07189d640 1 -- 172.21.15.76:0/1683187719 shutdown_connections 2024-03-12T06:04:18.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:18.084+0000 7fc07189d640 1 -- 172.21.15.76:0/1683187719 wait complete. 2024-03-12T06:04:18.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:18 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/99050328' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:04:18.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:18 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/99050328' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:04:18.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:18 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/99050328' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:04:19.104 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T06:04:19.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:04:19.105 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T06:04:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.219+0000 7fe055615640 1 -- 172.21.15.76:0/1631042249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe05010b750 msgr2=0x7fe05010dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.219+0000 7fe055615640 1 --2- 172.21.15.76:0/1631042249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe05010b750 0x7fe05010dba0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fe048009ca0 tx=0x7fe04802c180 comp rx=0 tx=0).stop 2024-03-12T06:04:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.219+0000 7fe055615640 1 -- 172.21.15.76:0/1631042249 shutdown_connections 2024-03-12T06:04:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.219+0000 7fe055615640 1 --2- 172.21.15.76:0/1631042249 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe05010b750 0x7fe05010dba0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.219+0000 7fe055615640 1 --2- 172.21.15.76:0/1631042249 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe050102ca0 0x7fe05010b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.219+0000 7fe055615640 1 --2- 172.21.15.76:0/1631042249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe050102380 0x7fe050102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.219+0000 7fe055615640 1 -- 172.21.15.76:0/1631042249 >> 172.21.15.76:0/1631042249 conn(0x7fe0500fbae0 msgr2=0x7fe0500fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:19.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.219+0000 7fe055615640 1 -- 172.21.15.76:0/1631042249 shutdown_connections 2024-03-12T06:04:19.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.219+0000 7fe055615640 1 -- 172.21.15.76:0/1631042249 wait complete. 2024-03-12T06:04:19.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.219+0000 7fe055615640 1 Processor -- start 2024-03-12T06:04:19.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe055615640 1 -- start start 2024-03-12T06:04:19.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe055615640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe050102380 0x7fe050197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe055615640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe050102ca0 0x7fe050197cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe055615640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe05010b750 0x7fe05019c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe055615640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05019c600 con 0x7fe050102380 2024-03-12T06:04:19.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe055615640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05019c770 con 0x7fe050102ca0 2024-03-12T06:04:19.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe055615640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05019c930 con 0x7fe05010b750 2024-03-12T06:04:19.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe04effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe050102380 0x7fe050197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe04effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe050102380 0x7fe050197790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37796/0 (socket says 172.21.15.76:37796) 2024-03-12T06:04:19.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe04effd640 1 -- 172.21.15.76:0/3249794285 learned_addr learned my addr 172.21.15.76:0/3249794285 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:19.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe04e7fc640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe050102ca0 0x7fe050197cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe04f7fe640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe05010b750 0x7fe05019c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe04effd640 1 -- 172.21.15.76:0/3249794285 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe05010b750 msgr2=0x7fe05019c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe04effd640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe05010b750 0x7fe05019c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe04effd640 1 -- 172.21.15.76:0/3249794285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe050102ca0 msgr2=0x7fe050197cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe04effd640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe050102ca0 0x7fe050197cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe04effd640 1 -- 172.21.15.76:0/3249794285 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe048009900 con 0x7fe050102380 2024-03-12T06:04:19.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe04f7fe640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe05010b750 0x7fe05019c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:19.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe04e7fc640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe050102ca0 0x7fe050197cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:19.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.220+0000 7fe04effd640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe050102380 0x7fe050197790 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fe04000d950 tx=0x7fe04000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:19.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.221+0000 7fe037fff640 1 -- 172.21.15.76:0/3249794285 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe040009a30 con 0x7fe050102380 2024-03-12T06:04:19.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.221+0000 7fe055615640 1 -- 172.21.15.76:0/3249794285 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe05019cc10 con 0x7fe050102380 2024-03-12T06:04:19.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.221+0000 7fe037fff640 1 -- 172.21.15.76:0/3249794285 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe040009b90 con 0x7fe050102380 2024-03-12T06:04:19.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.221+0000 7fe037fff640 1 -- 172.21.15.76:0/3249794285 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe040004a00 con 0x7fe050102380 2024-03-12T06:04:19.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.221+0000 7fe055615640 1 -- 172.21.15.76:0/3249794285 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0501a4490 con 0x7fe050102380 2024-03-12T06:04:19.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.223+0000 7fe037fff640 1 -- 172.21.15.76:0/3249794285 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fe040009d00 con 0x7fe050102380 2024-03-12T06:04:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.223+0000 7fe0581e1740 1 Processor -- start 2024-03-12T06:04:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.223+0000 7fe0581e1740 1 -- start start 2024-03-12T06:04:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.223+0000 7fe0581e1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189380 0x55a410189760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.223+0000 7fe0581e1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189d30 0x55a4101922e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.223+0000 7fe0581e1740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 0x55a410194d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.223+0000 7fe0581e1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4100f7f90 con 0x55a410189d30 2024-03-12T06:04:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.223+0000 7fe0581e1740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4100f54c0 con 0x55a4101928b0 2024-03-12T06:04:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.223+0000 7fe0581e1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4100f0ca0 con 0x55a410189380 2024-03-12T06:04:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe04effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189d30 0x55a4101922e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe037fff640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe0380a1ba0 0x7fe0380a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe04effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189d30 0x55a4101922e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37812/0 (socket says 172.21.15.76:37812) 2024-03-12T06:04:19.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe04effd640 1 -- 172.21.15.76:0/2071785691 learned_addr learned my addr 172.21.15.76:0/2071785691 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:19.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe04f7fe640 1 --2- 172.21.15.76:0/2071785691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189380 0x55a410189760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe04e7fc640 1 --2- 172.21.15.76:0/2071785691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 0x55a410194d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe04f7fe640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe0380a1ba0 0x7fe0380a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe037fff640 1 -- 172.21.15.76:0/3249794285 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fe0400cd340 con 0x7fe050102380 2024-03-12T06:04:19.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe04effd640 1 -- 172.21.15.76:0/2071785691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189380 msgr2=0x55a410189760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe04effd640 1 --2- 172.21.15.76:0/2071785691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189380 0x55a410189760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe04effd640 1 -- 172.21.15.76:0/2071785691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 msgr2=0x55a410194d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe04effd640 1 --2- 172.21.15.76:0/2071785691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 0x55a410194d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe04effd640 1 -- 172.21.15.76:0/2071785691 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a4100b30b0 con 0x55a410189d30 2024-03-12T06:04:19.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe04effd640 1 --2- 172.21.15.76:0/2071785691 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189d30 0x55a4101922e0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fe0400ca9a0 tx=0x7fe040000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d99fc68dab0a88 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:19.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe04f7fe640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe0380a1ba0 0x7fe0380a4060 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe048035fd0 tx=0x7fe0480095c0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:19.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe0357fa640 1 -- 172.21.15.76:0/2071785691 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04009a610 con 0x55a410189d30 2024-03-12T06:04:19.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe0357fa640 1 -- 172.21.15.76:0/2071785691 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe040099f10 con 0x55a410189d30 2024-03-12T06:04:19.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.224+0000 7fe0357fa640 1 -- 172.21.15.76:0/2071785691 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0400caef0 con 0x55a410189d30 2024-03-12T06:04:19.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.225+0000 7fe0581e1740 1 -- 172.21.15.76:0/2071785691 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189d30 msgr2=0x55a4101922e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.225+0000 7fe0581e1740 1 --2- 172.21.15.76:0/2071785691 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189d30 0x55a4101922e0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fe0400ca9a0 tx=0x7fe040000bc0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.225+0000 7fe0581e1740 1 -- 172.21.15.76:0/2071785691 shutdown_connections 2024-03-12T06:04:19.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.225+0000 7fe0581e1740 1 --2- 172.21.15.76:0/2071785691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 0x55a410194d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.225+0000 7fe0581e1740 1 --2- 172.21.15.76:0/2071785691 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189d30 0x55a4101922e0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.225+0000 7fe0581e1740 1 --2- 172.21.15.76:0/2071785691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189380 0x55a410189760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.225+0000 7fe0581e1740 1 -- 172.21.15.76:0/2071785691 >> 172.21.15.76:0/2071785691 conn(0x55a410183840 msgr2=0x55a410183c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:19.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.225+0000 7fe0581e1740 1 -- 172.21.15.76:0/2071785691 shutdown_connections 2024-03-12T06:04:19.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.225+0000 7fe0581e1740 1 -- 172.21.15.76:0/2071785691 wait complete. 2024-03-12T06:04:19.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.225+0000 7fe0581e1740 1 -- start start 2024-03-12T06:04:19.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe0581e1740 1 Processor -- start 2024-03-12T06:04:19.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe0581e1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189380 0x55a41018f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe0581e1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189d30 0x55a41018fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe0581e1740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 0x55a410190300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe0581e1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a410095f40 con 0x55a410189380 2024-03-12T06:04:19.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe0581e1740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a40fdc44b0 con 0x55a4101928b0 2024-03-12T06:04:19.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe0581e1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a410076670 con 0x55a410189d30 2024-03-12T06:04:19.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe0581e1740 10 client.0 _ll_register_callbacks cb 0x55a4100e3f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:19.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe04effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189380 0x55a41018f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe04effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189380 0x55a41018f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37818/0 (socket says 172.21.15.76:37818) 2024-03-12T06:04:19.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe04effd640 1 -- 172.21.15.76:0/548587608 learned_addr learned my addr 172.21.15.76:0/548587608 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:19.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe04e7fc640 1 --2- 172.21.15.76:0/548587608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189d30 0x55a41018fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe04f7fe640 1 --2- 172.21.15.76:0/548587608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 0x55a410190300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe04effd640 1 -- 172.21.15.76:0/548587608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189d30 msgr2=0x55a41018fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe04effd640 1 --2- 172.21.15.76:0/548587608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189d30 0x55a41018fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe04effd640 1 -- 172.21.15.76:0/548587608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 msgr2=0x55a410190300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe04effd640 1 --2- 172.21.15.76:0/548587608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 0x55a410190300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe04effd640 1 -- 172.21.15.76:0/548587608 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0501a4490 con 0x55a410189380 2024-03-12T06:04:19.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe04effd640 1 --2- 172.21.15.76:0/548587608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189380 0x55a41018f880 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fe040096370 tx=0x7fe04009a700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8635908a388728 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:19.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe032ff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:19.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe032ff5640 1 -- 172.21.15.76:0/548587608 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0400caeb0 con 0x55a410189380 2024-03-12T06:04:19.241 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe032ff5640 1 -- 172.21.15.76:0/548587608 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fe040005680 con 0x55a410189380 2024-03-12T06:04:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.226+0000 7fe0581e1740 1 -- 172.21.15.76:0/548587608 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a4100caf40 con 0x55a410189380 2024-03-12T06:04:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.227+0000 7fe032ff5640 1 -- 172.21.15.76:0/548587608 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0400961b0 con 0x55a410189380 2024-03-12T06:04:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.227+0000 7fe0581e1740 10 client.17436 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.227+0000 7fe0581e1740 1 -- 172.21.15.76:0/548587608 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a4100ef320 con 0x55a410189380 2024-03-12T06:04:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.227+0000 7fe032ff5640 1 -- 172.21.15.76:0/548587608 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0400b6020 con 0x55a410189380 2024-03-12T06:04:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.227+0000 7fe032ff5640 1 -- 172.21.15.76:0/548587608 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe0400a9b40 con 0x55a410189380 2024-03-12T06:04:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.227+0000 7fe0581e1740 10 client.17436 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.227+0000 7fe0581e1740 10 client.17436 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:19.242 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.227+0000 7fe0581e1740 20 client.17436 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.227+0000 7fe0581e1740 1 --2- 172.21.15.76:0/548587608 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a4101a2a70 0x55a4101a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.227+0000 7fe0581e1740 4 client.17436 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.227+0000 7fe0581e1740 1 -- 172.21.15.76:0/548587608 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a4100e7310 con 0x55a4101a2a70 2024-03-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.227+0000 7fe04e7fc640 1 --2- 172.21.15.76:0/548587608 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a4101a2a70 0x55a4101a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.228+0000 7fe032ff5640 10 client.17436 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.228+0000 7fe04e7fc640 1 --2- 172.21.15.76:0/548587608 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a4101a2a70 0x55a4101a4e90 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84baa5a560b9a462 server_cookie=cf26863d97d7ab40 in_seq=0 out_seq=0 2024-03-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.228+0000 7fe032ff5640 1 -- 172.21.15.76:0/548587608 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a4100e7310 con 0x55a4101a2a70 2024-03-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.228+0000 7fe032ff5640 10 client.17436 handle_command_reply: tid=0 2024-03-12T06:04:19.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.228+0000 7fe0581e1740 1 client.17436 shutdown 2024-03-12T06:04:19.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.228+0000 7fe0581e1740 1 -- 172.21.15.76:0/548587608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189380 msgr2=0x55a41018f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.228+0000 7fe0581e1740 1 --2- 172.21.15.76:0/548587608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189380 0x55a41018f880 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fe040096370 tx=0x7fe04009a700 comp rx=0 tx=0).stop 2024-03-12T06:04:19.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.228+0000 7fe0581e1740 1 -- 172.21.15.76:0/548587608 shutdown_connections 2024-03-12T06:04:19.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.228+0000 7fe0581e1740 1 --2- 172.21.15.76:0/548587608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 0x55a410190300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.228+0000 7fe0581e1740 1 --2- 172.21.15.76:0/548587608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189d30 0x55a41018fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.228+0000 7fe0581e1740 1 --2- 172.21.15.76:0/548587608 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a4101a2a70 0x55a4101a4e90 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.228+0000 7fe0581e1740 1 --2- 172.21.15.76:0/548587608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189380 0x55a41018f880 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.228+0000 7fe032ff5640 0 client.17436 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:19.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.228+0000 7fe0581e1740 1 -- 172.21.15.76:0/548587608 >> 172.21.15.76:0/548587608 conn(0x55a410183840 msgr2=0x55a4101857a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:19.244 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.229+0000 7fe0581e1740 1 -- 172.21.15.76:0/548587608 shutdown_connections 2024-03-12T06:04:19.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.229+0000 7fe0581e1740 1 -- 172.21.15.76:0/548587608 wait complete. 2024-03-12T06:04:19.245 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.229+0000 7fe0581e1740 20 client.17436 trim_cache size 0 max 16384 2024-03-12T06:04:19.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.244+0000 7fe0581e1740 1 Processor -- start 2024-03-12T06:04:19.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.244+0000 7fe0581e1740 1 -- start start 2024-03-12T06:04:19.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.244+0000 7fe0581e1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189380 0x55a410191fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.244+0000 7fe0581e1740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a410189d30 0x55a410190120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe0581e1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4101928b0 0x55a41018cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe0581e1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a410190660 con 0x55a4101928b0 2024-03-12T06:04:19.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe0581e1740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4101907d0 con 0x55a410189d30 2024-03-12T06:04:19.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe0581e1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a410183e50 con 0x55a410189380 2024-03-12T06:04:19.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe04f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4101928b0 0x55a41018cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe04f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4101928b0 0x55a41018cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37832/0 (socket says 172.21.15.76:37832) 2024-03-12T06:04:19.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe04f7fe640 1 -- 172.21.15.76:0/1586967592 learned_addr learned my addr 172.21.15.76:0/1586967592 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:19.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe04e7fc640 1 --2- 172.21.15.76:0/1586967592 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a410189d30 0x55a410190120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe04effd640 1 --2- 172.21.15.76:0/1586967592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189380 0x55a410191fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe04f7fe640 1 -- 172.21.15.76:0/1586967592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189380 msgr2=0x55a410191fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe04f7fe640 1 --2- 172.21.15.76:0/1586967592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189380 0x55a410191fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe04f7fe640 1 -- 172.21.15.76:0/1586967592 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a410189d30 msgr2=0x55a410190120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe04f7fe640 1 --2- 172.21.15.76:0/1586967592 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a410189d30 0x55a410190120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe04f7fe640 1 -- 172.21.15.76:0/1586967592 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a41008bc00 con 0x55a4101928b0 2024-03-12T06:04:19.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe04f7fe640 1 --2- 172.21.15.76:0/1586967592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4101928b0 0x55a41018cb20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe048036000 tx=0x7fe04803a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c19081c8e639e45 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:19.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe034ff9640 1 -- 172.21.15.76:0/1586967592 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe048003970 con 0x55a4101928b0 2024-03-12T06:04:19.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe034ff9640 1 -- 172.21.15.76:0/1586967592 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe048032790 con 0x55a4101928b0 2024-03-12T06:04:19.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.245+0000 7fe034ff9640 1 -- 172.21.15.76:0/1586967592 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0480042a0 con 0x55a4101928b0 2024-03-12T06:04:19.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.246+0000 7fe0581e1740 1 -- 172.21.15.76:0/1586967592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4101928b0 msgr2=0x55a41018cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.246+0000 7fe0581e1740 1 --2- 172.21.15.76:0/1586967592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4101928b0 0x55a41018cb20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe048036000 tx=0x7fe04803a000 comp rx=0 tx=0).stop 2024-03-12T06:04:19.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.246+0000 7fe0581e1740 1 -- 172.21.15.76:0/1586967592 shutdown_connections 2024-03-12T06:04:19.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.246+0000 7fe0581e1740 1 --2- 172.21.15.76:0/1586967592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a4101928b0 0x55a41018cb20 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.246+0000 7fe0581e1740 1 --2- 172.21.15.76:0/1586967592 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a410189d30 0x55a410190120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.246+0000 7fe0581e1740 1 --2- 172.21.15.76:0/1586967592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189380 0x55a410191fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.246+0000 7fe0581e1740 1 -- 172.21.15.76:0/1586967592 >> 172.21.15.76:0/1586967592 conn(0x55a410183840 msgr2=0x55a41018e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:19.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.246+0000 7fe0581e1740 1 -- 172.21.15.76:0/1586967592 shutdown_connections 2024-03-12T06:04:19.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.246+0000 7fe0581e1740 1 -- 172.21.15.76:0/1586967592 wait complete. 2024-03-12T06:04:19.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.246+0000 7fe0581e1740 1 -- start start 2024-03-12T06:04:19.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe0581e1740 1 Processor -- start 2024-03-12T06:04:19.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe0581e1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189380 0x55a4101f3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe0581e1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189d30 0x55a4101f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe0581e1740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 0x55a410198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe0581e1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4101995e0 con 0x55a410189380 2024-03-12T06:04:19.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe0581e1740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a410199750 con 0x55a4101928b0 2024-03-12T06:04:19.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe0581e1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a410199910 con 0x55a410189d30 2024-03-12T06:04:19.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe0581e1740 10 client.0 _ll_register_callbacks cb 0x7fe040099ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:19.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe04effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189380 0x55a4101f3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe04effd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189380 0x55a4101f3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37842/0 (socket says 172.21.15.76:37842) 2024-03-12T06:04:19.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe04effd640 1 -- 172.21.15.76:0/845552747 learned_addr learned my addr 172.21.15.76:0/845552747 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:19.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe04e7fc640 1 --2- 172.21.15.76:0/845552747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189d30 0x55a4101f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe04f7fe640 1 --2- 172.21.15.76:0/845552747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 0x55a410198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe04effd640 1 -- 172.21.15.76:0/845552747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189d30 msgr2=0x55a4101f3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe04effd640 1 --2- 172.21.15.76:0/845552747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189d30 0x55a4101f3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe04effd640 1 -- 172.21.15.76:0/845552747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 msgr2=0x55a410198fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe04effd640 1 --2- 172.21.15.76:0/845552747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 0x55a410198fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe04effd640 1 -- 172.21.15.76:0/845552747 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05010e870 con 0x55a410189380 2024-03-12T06:04:19.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe04f7fe640 1 --2- 172.21.15.76:0/845552747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 0x55a410198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:19.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe04effd640 1 --2- 172.21.15.76:0/845552747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189380 0x55a4101f3700 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fe0400cab70 tx=0x7fe04009a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2eea988d9c1e7ed server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:19.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe032ff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:19.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe032ff5640 1 -- 172.21.15.76:0/845552747 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0400caeb0 con 0x55a410189380 2024-03-12T06:04:19.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe032ff5640 1 -- 172.21.15.76:0/845552747 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fe0400b7080 con 0x55a410189380 2024-03-12T06:04:19.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.247+0000 7fe0581e1740 1 -- 172.21.15.76:0/845552747 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a410116900 con 0x55a410189380 2024-03-12T06:04:19.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.248+0000 7fe032ff5640 1 -- 172.21.15.76:0/845552747 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe040099f10 con 0x55a410189380 2024-03-12T06:04:19.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.248+0000 7fe0581e1740 10 client.17442 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:19.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.248+0000 7fe0581e1740 1 -- 172.21.15.76:0/845552747 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a4100b1910 con 0x55a410189380 2024-03-12T06:04:19.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.248+0000 7fe032ff5640 1 -- 172.21.15.76:0/845552747 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a410116900 con 0x55a410189380 2024-03-12T06:04:19.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.248+0000 7fe032ff5640 1 -- 172.21.15.76:0/845552747 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe0400a9b40 con 0x55a410189380 2024-03-12T06:04:19.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.248+0000 7fe0581e1740 10 client.17442 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:19.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.248+0000 7fe0581e1740 10 client.17442 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:19.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.248+0000 7fe0581e1740 20 client.17442 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:19.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.248+0000 7fe0581e1740 1 --2- 172.21.15.76:0/845552747 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a4101fb140 0x55a4101fd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.248+0000 7fe0581e1740 4 client.17442 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T06:04:19.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.248+0000 7fe0581e1740 1 -- 172.21.15.76:0/845552747 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a40fdf0860 con 0x55a4101fb140 2024-03-12T06:04:19.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.248+0000 7fe04e7fc640 1 --2- 172.21.15.76:0/845552747 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a4101fb140 0x55a4101fd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.249+0000 7fe032ff5640 10 client.17442 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:19.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.249+0000 7fe04e7fc640 1 --2- 172.21.15.76:0/845552747 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a4101fb140 0x55a4101fd530 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b55e5d3a36fea9a4 server_cookie=c6a1244de02b8322 in_seq=0 out_seq=0 2024-03-12T06:04:19.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe032ff5640 1 -- 172.21.15.76:0/845552747 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a40fdf0860 con 0x55a4101fb140 2024-03-12T06:04:19.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe032ff5640 10 client.17442 handle_command_reply: tid=0 2024-03-12T06:04:19.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe0581e1740 1 client.17442 shutdown 2024-03-12T06:04:19.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe0581e1740 1 -- 172.21.15.76:0/845552747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189380 msgr2=0x55a4101f3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe0581e1740 1 --2- 172.21.15.76:0/845552747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189380 0x55a4101f3700 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fe0400cab70 tx=0x7fe04009a610 comp rx=0 tx=0).stop 2024-03-12T06:04:19.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe0581e1740 1 -- 172.21.15.76:0/845552747 shutdown_connections 2024-03-12T06:04:19.262 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe0581e1740 1 --2- 172.21.15.76:0/845552747 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a4101928b0 0x55a410198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe0581e1740 1 --2- 172.21.15.76:0/845552747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a410189d30 0x55a4101f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe0581e1740 1 --2- 172.21.15.76:0/845552747 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a4101fb140 0x55a4101fd530 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe0581e1740 1 --2- 172.21.15.76:0/845552747 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a410189380 0x55a4101f3700 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe032ff5640 0 client.17442 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:19.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe0581e1740 1 -- 172.21.15.76:0/845552747 >> 172.21.15.76:0/845552747 conn(0x55a410183840 msgr2=0x55a410184ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:19.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe0581e1740 1 -- 172.21.15.76:0/845552747 shutdown_connections 2024-03-12T06:04:19.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe0581e1740 1 -- 172.21.15.76:0/845552747 wait complete. 2024-03-12T06:04:19.263 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.250+0000 7fe0581e1740 20 client.17442 trim_cache size 0 max 16384 2024-03-12T06:04:19.264 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:04:19.264 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T06:04:19.264 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "eb3b7b37-b297-46ae-945b-66ae5d97b4c3", 2024-03-12T06:04:19.264 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T06:04:19.264 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:04:19.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.251+0000 7fe055615640 1 -- 172.21.15.76:0/3249794285 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe0380a1ba0 msgr2=0x7fe0380a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.251+0000 7fe055615640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe0380a1ba0 0x7fe0380a4060 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe048035fd0 tx=0x7fe0480095c0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.251+0000 7fe055615640 1 -- 172.21.15.76:0/3249794285 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe050102380 msgr2=0x7fe050197790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.251+0000 7fe055615640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe050102380 0x7fe050197790 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fe04000d950 tx=0x7fe04000de10 comp rx=0 tx=0).stop 2024-03-12T06:04:19.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.251+0000 7fe055615640 1 -- 172.21.15.76:0/3249794285 shutdown_connections 2024-03-12T06:04:19.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.251+0000 7fe055615640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fe0380a1ba0 0x7fe0380a4060 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.251+0000 7fe055615640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe05010b750 0x7fe05019c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.251+0000 7fe055615640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe050102ca0 0x7fe050197cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.251+0000 7fe055615640 1 --2- 172.21.15.76:0/3249794285 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe050102380 0x7fe050197790 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.251+0000 7fe055615640 1 -- 172.21.15.76:0/3249794285 >> 172.21.15.76:0/3249794285 conn(0x7fe0500fbae0 msgr2=0x7fe0500fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:19.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.251+0000 7fe055615640 1 -- 172.21.15.76:0/3249794285 shutdown_connections 2024-03-12T06:04:19.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.251+0000 7fe055615640 1 -- 172.21.15.76:0/3249794285 wait complete. 2024-03-12T06:04:19.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:04:19.267 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:04:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.388+0000 7f6faa1b6640 1 -- 172.21.15.76:0/1099784316 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fa4106430 msgr2=0x7f6fa410e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.388+0000 7f6faa1b6640 1 --2- 172.21.15.76:0/1099784316 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fa4106430 0x7f6fa410e000 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009c10 tx=0x7f6f9c02bef0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.388+0000 7f6faa1b6640 1 -- 172.21.15.76:0/1099784316 shutdown_connections 2024-03-12T06:04:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.388+0000 7f6faa1b6640 1 --2- 172.21.15.76:0/1099784316 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fa4106430 0x7f6fa410e000 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.388+0000 7f6faa1b6640 1 --2- 172.21.15.76:0/1099784316 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fa40690f0 0x7f6fa4105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.388+0000 7f6faa1b6640 1 --2- 172.21.15.76:0/1099784316 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6fa4068740 0x7f6fa4068b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.388+0000 7f6faa1b6640 1 -- 172.21.15.76:0/1099784316 >> 172.21.15.76:0/1099784316 conn(0x7f6fa4077180 msgr2=0x7f6fa4077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.388+0000 7f6faa1b6640 1 -- 172.21.15.76:0/1099784316 shutdown_connections 2024-03-12T06:04:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.388+0000 7f6faa1b6640 1 -- 172.21.15.76:0/1099784316 wait complete. 2024-03-12T06:04:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.389+0000 7f6faa1b6640 1 Processor -- start 2024-03-12T06:04:19.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.389+0000 7f6faa1b6640 1 -- start start 2024-03-12T06:04:19.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.389+0000 7f6faa1b6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6fa4068740 0x7f6fa4100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.389+0000 7f6faa1b6640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fa4106430 0x7f6fa4100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.389+0000 7f6faa1b6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fa4100ac0 0x7f6fa41a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.389+0000 7f6faa1b6640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa41a5ea0 con 0x7f6fa4068740 2024-03-12T06:04:19.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.389+0000 7f6faa1b6640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa41a6010 con 0x7f6fa4106430 2024-03-12T06:04:19.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.389+0000 7f6faa1b6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa41a61d0 con 0x7f6fa4100ac0 2024-03-12T06:04:19.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.390+0000 7f6fa37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6fa4068740 0x7f6fa4100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.390+0000 7f6fa2ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fa4106430 0x7f6fa4100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.390+0000 7f6fa2ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fa4106430 0x7f6fa4100580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:51176/0 (socket says 172.21.15.76:51176) 2024-03-12T06:04:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.390+0000 7f6fa37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6fa4068740 0x7f6fa4100040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37870/0 (socket says 172.21.15.76:37870) 2024-03-12T06:04:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.390+0000 7f6fa2ffd640 1 -- 172.21.15.76:0/3609101638 learned_addr learned my addr 172.21.15.76:0/3609101638 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.390+0000 7f6fa3fff640 1 --2- 172.21.15.76:0/3609101638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fa4100ac0 0x7f6fa41a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.390+0000 7f6fa2ffd640 1 -- 172.21.15.76:0/3609101638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fa4100ac0 msgr2=0x7f6fa41a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.390+0000 7f6fa2ffd640 1 --2- 172.21.15.76:0/3609101638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fa4100ac0 0x7f6fa41a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.390+0000 7f6fa2ffd640 1 -- 172.21.15.76:0/3609101638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6fa4068740 msgr2=0x7f6fa4100040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.390+0000 7f6fa2ffd640 1 --2- 172.21.15.76:0/3609101638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6fa4068740 0x7f6fa4100040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.390+0000 7f6fa2ffd640 1 -- 172.21.15.76:0/3609101638 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c009840 con 0x7f6fa4106430 2024-03-12T06:04:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.390+0000 7f6fa37fe640 1 --2- 172.21.15.76:0/3609101638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6fa4068740 0x7f6fa4100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:04:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.390+0000 7f6fa2ffd640 1 --2- 172.21.15.76:0/3609101638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fa4106430 0x7f6fa4100580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800e9f0 tx=0x7f6f9800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.391+0000 7f6fa0ff9640 1 -- 172.21.15.76:0/3609101638 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f980096c0 con 0x7f6fa4106430 2024-03-12T06:04:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.391+0000 7f6fa0ff9640 1 -- 172.21.15.76:0/3609101638 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f98005500 con 0x7f6fa4106430 2024-03-12T06:04:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.391+0000 7f6faa1b6640 1 -- 172.21.15.76:0/3609101638 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa41a64b0 con 0x7f6fa4106430 2024-03-12T06:04:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.391+0000 7f6fa0ff9640 1 -- 172.21.15.76:0/3609101638 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f98010430 con 0x7f6fa4106430 2024-03-12T06:04:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.391+0000 7f6faa1b6640 1 -- 172.21.15.76:0/3609101638 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa41a6970 con 0x7f6fa4106430 2024-03-12T06:04:19.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.394+0000 7f6fa0ff9640 1 -- 172.21.15.76:0/3609101638 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f6f98005020 con 0x7f6fa4106430 2024-03-12T06:04:19.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.394+0000 7f6facd85740 1 Processor -- start 2024-03-12T06:04:19.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.394+0000 7f6facd85740 1 -- start start 2024-03-12T06:04:19.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.394+0000 7f6facd85740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb485c90 0x5653bb486070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.394+0000 7f6facd85740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb486640 0x5653bb497a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.394+0000 7f6facd85740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb497fc0 0x5653bb49a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.394+0000 7f6facd85740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653bb3f5280 con 0x5653bb486640 2024-03-12T06:04:19.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.394+0000 7f6facd85740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653bb3f27b0 con 0x5653bb485c90 2024-03-12T06:04:19.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.394+0000 7f6facd85740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653bb3edf90 con 0x5653bb497fc0 2024-03-12T06:04:19.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa0ff9640 1 --2- 172.21.15.76:0/3609101638 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f6f8c0a1bd0 0x7f6f8c0a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb486640 0x5653bb497a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb486640 0x5653bb497a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37874/0 (socket says 172.21.15.76:37874) 2024-03-12T06:04:19.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa37fe640 1 -- 172.21.15.76:0/692949806 learned_addr learned my addr 172.21.15.76:0/692949806 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:19.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa0ff9640 1 -- 172.21.15.76:0/3609101638 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f6f98014070 con 0x7f6fa4106430 2024-03-12T06:04:19.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa2ffd640 1 --2- 172.21.15.76:0/692949806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb497fc0 0x5653bb49a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa3fff640 1 --2- 172.21.15.76:0/692949806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb485c90 0x5653bb486070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa37fe640 1 -- 172.21.15.76:0/692949806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb497fc0 msgr2=0x5653bb49a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa37fe640 1 --2- 172.21.15.76:0/692949806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb497fc0 0x5653bb49a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa37fe640 1 -- 172.21.15.76:0/692949806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb485c90 msgr2=0x5653bb486070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa37fe640 1 --2- 172.21.15.76:0/692949806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb485c90 0x5653bb486070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa37fe640 1 -- 172.21.15.76:0/692949806 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653bb3ac570 con 0x5653bb486640 2024-03-12T06:04:19.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa3fff640 1 --2- 172.21.15.76:0/692949806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb485c90 0x5653bb486070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:19.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa3fff640 1 --2- 172.21.15.76:0/3609101638 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f6f8c0a1bd0 0x7f6f8c0a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6fa37fe640 1 --2- 172.21.15.76:0/692949806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb486640 0x5653bb497a80 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f6f94011510 tx=0x7f6f940119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7ba8fec33d2bd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:19.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.395+0000 7f6f89ffb640 1 -- 172.21.15.76:0/692949806 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f94004020 con 0x5653bb486640 2024-03-12T06:04:19.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.396+0000 7f6f89ffb640 1 -- 172.21.15.76:0/692949806 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f94004180 con 0x5653bb486640 2024-03-12T06:04:19.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.396+0000 7f6f89ffb640 1 -- 172.21.15.76:0/692949806 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9400f820 con 0x5653bb486640 2024-03-12T06:04:19.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.396+0000 7f6facd85740 1 -- 172.21.15.76:0/692949806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb486640 msgr2=0x5653bb497a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.396+0000 7f6facd85740 1 --2- 172.21.15.76:0/692949806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb486640 0x5653bb497a80 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f6f94011510 tx=0x7f6f940119d0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.396+0000 7f6fa3fff640 1 --2- 172.21.15.76:0/3609101638 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f6f8c0a1bd0 0x7f6f8c0a4090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c006010 tx=0x7f6f9c02e040 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:19.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.396+0000 7f6facd85740 1 -- 172.21.15.76:0/692949806 shutdown_connections 2024-03-12T06:04:19.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.396+0000 7f6facd85740 1 --2- 172.21.15.76:0/692949806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb497fc0 0x5653bb49a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.396+0000 7f6facd85740 1 --2- 172.21.15.76:0/692949806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb486640 0x5653bb497a80 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.396+0000 7f6facd85740 1 --2- 172.21.15.76:0/692949806 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb485c90 0x5653bb486070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.396+0000 7f6facd85740 1 -- 172.21.15.76:0/692949806 >> 172.21.15.76:0/692949806 conn(0x5653bb480990 msgr2=0x5653bb480d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:19.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.396+0000 7f6facd85740 1 -- 172.21.15.76:0/692949806 shutdown_connections 2024-03-12T06:04:19.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.396+0000 7f6facd85740 1 -- 172.21.15.76:0/692949806 wait complete. 2024-03-12T06:04:19.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.397+0000 7f6facd85740 1 -- start start 2024-03-12T06:04:19.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.397+0000 7f6facd85740 1 Processor -- start 2024-03-12T06:04:19.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.397+0000 7f6facd85740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb485c90 0x5653bb4979a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.397+0000 7f6facd85740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb486640 0x5653bb494d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.397+0000 7f6facd85740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb497fc0 0x5653bb495340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.397+0000 7f6facd85740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653bb0c1430 con 0x5653bb485c90 2024-03-12T06:04:19.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.397+0000 7f6facd85740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653bb07de00 con 0x5653bb486640 2024-03-12T06:04:19.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.397+0000 7f6facd85740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653bb372fd0 con 0x5653bb497fc0 2024-03-12T06:04:19.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.397+0000 7f6fa37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb485c90 0x5653bb4979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.398+0000 7f6facd85740 10 client.0 _ll_register_callbacks cb 0x5653bb3c8fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:19.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.398+0000 7f6fa2ffd640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb486640 0x5653bb494d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.398+0000 7f6fa37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb485c90 0x5653bb4979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37888/0 (socket says 172.21.15.76:37888) 2024-03-12T06:04:19.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.398+0000 7f6fa37fe640 1 -- 172.21.15.76:0/3494316397 learned_addr learned my addr 172.21.15.76:0/3494316397 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:19.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.398+0000 7f6fa3fff640 1 --2- 172.21.15.76:0/3494316397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb497fc0 0x5653bb495340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.398+0000 7f6fa37fe640 1 -- 172.21.15.76:0/3494316397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb497fc0 msgr2=0x5653bb495340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.398+0000 7f6fa37fe640 1 --2- 172.21.15.76:0/3494316397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb497fc0 0x5653bb495340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.398+0000 7f6fa37fe640 1 -- 172.21.15.76:0/3494316397 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb486640 msgr2=0x5653bb494d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.398+0000 7f6fa37fe640 1 --2- 172.21.15.76:0/3494316397 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb486640 0x5653bb494d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.398+0000 7f6fa37fe640 1 -- 172.21.15.76:0/3494316397 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f94009d00 con 0x5653bb485c90 2024-03-12T06:04:19.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.398+0000 7f6fa2ffd640 1 --2- 172.21.15.76:0/3494316397 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb486640 0x5653bb494d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:04:19.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.398+0000 7f6fa37fe640 1 --2- 172.21.15.76:0/3494316397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb485c90 0x5653bb4979a0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f6f94003e00 tx=0x7f6f940038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9ed72cec066ef56 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:19.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.398+0000 7f6f82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:19.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.398+0000 7f6f82ffd640 1 -- 172.21.15.76:0/3494316397 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f94005040 con 0x5653bb485c90 2024-03-12T06:04:19.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.399+0000 7f6f82ffd640 1 -- 172.21.15.76:0/3494316397 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f6f940051a0 con 0x5653bb485c90 2024-03-12T06:04:19.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.399+0000 7f6facd85740 1 -- 172.21.15.76:0/3494316397 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5653bb0ed7e0 con 0x5653bb485c90 2024-03-12T06:04:19.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.399+0000 7f6f82ffd640 1 -- 172.21.15.76:0/3494316397 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f94005360 con 0x5653bb485c90 2024-03-12T06:04:19.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.399+0000 7f6facd85740 10 client.17454 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:19.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.399+0000 7f6facd85740 1 -- 172.21.15.76:0/3494316397 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5653bb3ec610 con 0x5653bb485c90 2024-03-12T06:04:19.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.399+0000 7f6f82ffd640 1 -- 172.21.15.76:0/3494316397 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5653bb0ed7e0 con 0x5653bb485c90 2024-03-12T06:04:19.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.399+0000 7f6f82ffd640 1 -- 172.21.15.76:0/3494316397 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6f94021040 con 0x5653bb485c90 2024-03-12T06:04:19.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.399+0000 7f6facd85740 10 client.17454 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:19.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.399+0000 7f6facd85740 10 client.17454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:19.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.399+0000 7f6facd85740 20 client.17454 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:19.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.399+0000 7f6facd85740 1 --2- 172.21.15.76:0/3494316397 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653bb4a8110 0x5653bb4aa500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.399+0000 7f6facd85740 4 client.17454 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:04:19.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.399+0000 7f6facd85740 1 -- 172.21.15.76:0/3494316397 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5653bb12a720 con 0x5653bb4a8110 2024-03-12T06:04:19.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.400+0000 7f6fa2ffd640 1 --2- 172.21.15.76:0/3494316397 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653bb4a8110 0x5653bb4aa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.400+0000 7f6f82ffd640 10 client.17454 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:19.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.400+0000 7f6fa2ffd640 1 --2- 172.21.15.76:0/3494316397 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653bb4a8110 0x5653bb4aa500 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec4cbae391d434c8 server_cookie=a00e99c3bcea2e61 in_seq=0 out_seq=0 2024-03-12T06:04:19.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6f82ffd640 1 -- 172.21.15.76:0/3494316397 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5653bb12a720 con 0x5653bb4a8110 2024-03-12T06:04:19.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6f82ffd640 10 client.17454 handle_command_reply: tid=0 2024-03-12T06:04:19.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6facd85740 1 client.17454 shutdown 2024-03-12T06:04:19.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6facd85740 1 -- 172.21.15.76:0/3494316397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb485c90 msgr2=0x5653bb4979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6facd85740 1 --2- 172.21.15.76:0/3494316397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb485c90 0x5653bb4979a0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f6f94003e00 tx=0x7f6f940038a0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6facd85740 1 -- 172.21.15.76:0/3494316397 shutdown_connections 2024-03-12T06:04:19.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6facd85740 1 --2- 172.21.15.76:0/3494316397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb497fc0 0x5653bb495340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6facd85740 1 --2- 172.21.15.76:0/3494316397 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb486640 0x5653bb494d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6facd85740 1 --2- 172.21.15.76:0/3494316397 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653bb4a8110 0x5653bb4aa500 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6facd85740 1 --2- 172.21.15.76:0/3494316397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb485c90 0x5653bb4979a0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6facd85740 1 -- 172.21.15.76:0/3494316397 >> 172.21.15.76:0/3494316397 conn(0x5653bb480990 msgr2=0x5653bb482730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:19.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6f82ffd640 0 client.17454 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:19.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6facd85740 1 -- 172.21.15.76:0/3494316397 shutdown_connections 2024-03-12T06:04:19.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6facd85740 1 -- 172.21.15.76:0/3494316397 wait complete. 2024-03-12T06:04:19.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.401+0000 7f6facd85740 20 client.17454 trim_cache size 0 max 16384 2024-03-12T06:04:19.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.421+0000 7f6facd85740 1 Processor -- start 2024-03-12T06:04:19.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.421+0000 7f6facd85740 1 -- start start 2024-03-12T06:04:19.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.421+0000 7f6facd85740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb485c90 0x5653bb497460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.421+0000 7f6facd85740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb486640 0x5653bb4979a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.426 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.421+0000 7f6facd85740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb497fc0 0x5653bb4945c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.421+0000 7f6facd85740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653bb4a4630 con 0x5653bb486640 2024-03-12T06:04:19.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.421+0000 7f6facd85740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653bb487090 con 0x5653bb497fc0 2024-03-12T06:04:19.430 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.421+0000 7f6facd85740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653bb4855f0 con 0x5653bb485c90 2024-03-12T06:04:19.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.421+0000 7f6fa2ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb486640 0x5653bb4979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.421+0000 7f6fa3fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb497fc0 0x5653bb4945c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.421+0000 7f6fa2ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb486640 0x5653bb4979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:37902/0 (socket says 172.21.15.76:37902) 2024-03-12T06:04:19.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.421+0000 7f6fa2ffd640 1 -- 172.21.15.76:0/1540466004 learned_addr learned my addr 172.21.15.76:0/1540466004 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:19.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.422+0000 7f6fa37fe640 1 --2- 172.21.15.76:0/1540466004 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb485c90 0x5653bb497460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.422+0000 7f6fa3fff640 1 -- 172.21.15.76:0/1540466004 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb485c90 msgr2=0x5653bb497460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.431 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.422+0000 7f6fa3fff640 1 --2- 172.21.15.76:0/1540466004 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb485c90 0x5653bb497460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.422+0000 7f6fa3fff640 1 -- 172.21.15.76:0/1540466004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb486640 msgr2=0x5653bb4979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.422+0000 7f6fa3fff640 1 --2- 172.21.15.76:0/1540466004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb486640 0x5653bb4979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.422+0000 7f6fa3fff640 1 -- 172.21.15.76:0/1540466004 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653bb388ac0 con 0x5653bb497fc0 2024-03-12T06:04:19.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.422+0000 7f6fa2ffd640 1 --2- 172.21.15.76:0/1540466004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb486640 0x5653bb4979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:04:19.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.422+0000 7f6fa3fff640 1 --2- 172.21.15.76:0/1540466004 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb497fc0 0x5653bb4945c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c02c430 tx=0x7f6f9c041070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=162e8ac03a74f58d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:19.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.422+0000 7f6f897fa640 1 -- 172.21.15.76:0/1540466004 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c0337e0 con 0x5653bb497fc0 2024-03-12T06:04:19.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.422+0000 7f6f897fa640 1 -- 172.21.15.76:0/1540466004 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f9c003e50 con 0x5653bb497fc0 2024-03-12T06:04:19.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.422+0000 7f6f897fa640 1 -- 172.21.15.76:0/1540466004 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c04c690 con 0x5653bb497fc0 2024-03-12T06:04:19.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.423+0000 7f6facd85740 1 -- 172.21.15.76:0/1540466004 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb497fc0 msgr2=0x5653bb4945c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.423+0000 7f6facd85740 1 --2- 172.21.15.76:0/1540466004 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb497fc0 0x5653bb4945c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c02c430 tx=0x7f6f9c041070 comp rx=0 tx=0).stop 2024-03-12T06:04:19.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.423+0000 7f6facd85740 1 -- 172.21.15.76:0/1540466004 shutdown_connections 2024-03-12T06:04:19.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.423+0000 7f6facd85740 1 --2- 172.21.15.76:0/1540466004 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb497fc0 0x5653bb4945c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.423+0000 7f6facd85740 1 --2- 172.21.15.76:0/1540466004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb486640 0x5653bb4979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.423+0000 7f6facd85740 1 --2- 172.21.15.76:0/1540466004 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb485c90 0x5653bb497460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.423+0000 7f6facd85740 1 -- 172.21.15.76:0/1540466004 >> 172.21.15.76:0/1540466004 conn(0x5653bb480990 msgr2=0x5653bb490020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:19.433 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.423+0000 7f6facd85740 1 -- 172.21.15.76:0/1540466004 shutdown_connections 2024-03-12T06:04:19.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.423+0000 7f6facd85740 1 -- 172.21.15.76:0/1540466004 wait complete. 2024-03-12T06:04:19.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.424+0000 7f6facd85740 1 -- start start 2024-03-12T06:04:19.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.424+0000 7f6facd85740 1 Processor -- start 2024-03-12T06:04:19.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.424+0000 7f6facd85740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb485c90 0x5653bb49f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.424+0000 7f6facd85740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb486640 0x5653bb49b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.424+0000 7f6facd85740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb497fc0 0x5653bb49b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.424+0000 7f6facd85740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653bb49bf40 con 0x5653bb497fc0 2024-03-12T06:04:19.434 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.424+0000 7f6facd85740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653bb49c0b0 con 0x5653bb485c90 2024-03-12T06:04:19.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.424+0000 7f6facd85740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653bb49c270 con 0x5653bb486640 2024-03-12T06:04:19.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.424+0000 7f6facd85740 10 client.0 _ll_register_callbacks cb 0x7f6f94009b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:19.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6fa37fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb485c90 0x5653bb49f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6fa2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb486640 0x5653bb49b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6fa3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb497fc0 0x5653bb49b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6fa37fe640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb485c90 0x5653bb49f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:51212/0 (socket says 172.21.15.76:51212) 2024-03-12T06:04:19.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6fa37fe640 1 -- 172.21.15.76:0/1938315386 learned_addr learned my addr 172.21.15.76:0/1938315386 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:19.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6fa37fe640 1 -- 172.21.15.76:0/1938315386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb486640 msgr2=0x5653bb49b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6fa37fe640 1 --2- 172.21.15.76:0/1938315386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb486640 0x5653bb49b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6fa37fe640 1 -- 172.21.15.76:0/1938315386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb497fc0 msgr2=0x5653bb49b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6fa37fe640 1 --2- 172.21.15.76:0/1938315386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb497fc0 0x5653bb49b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6fa37fe640 1 -- 172.21.15.76:0/1938315386 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f98083e70 con 0x5653bb485c90 2024-03-12T06:04:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6fa3fff640 1 --2- 172.21.15.76:0/1938315386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb497fc0 0x5653bb49b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:04:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6fa37fe640 1 --2- 172.21.15.76:0/1938315386 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb485c90 0x5653bb49f0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6f94011510 tx=0x7f6f94005df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bc65edfe9e667ab server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6f82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T06:04:19.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6f82ffd640 1 -- 172.21.15.76:0/1938315386 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f94005240 con 0x5653bb485c90 2024-03-12T06:04:19.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6facd85740 1 -- 172.21.15.76:0/1938315386 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6f94004180 con 0x5653bb485c90 2024-03-12T06:04:19.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6f82ffd640 1 -- 172.21.15.76:0/1938315386 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f6f940053a0 con 0x5653bb485c90 2024-03-12T06:04:19.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.425+0000 7f6f82ffd640 1 -- 172.21.15.76:0/1938315386 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f94003910 con 0x5653bb485c90 2024-03-12T06:04:19.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.426+0000 7f6facd85740 10 client.26516 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:19.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.426+0000 7f6facd85740 1 -- 172.21.15.76:0/1938315386 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f6fa410e810 con 0x5653bb485c90 2024-03-12T06:04:19.437 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.426+0000 7f6f82ffd640 1 -- 172.21.15.76:0/1938315386 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f94043080 con 0x5653bb485c90 2024-03-12T06:04:19.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.426+0000 7f6f82ffd640 1 -- 172.21.15.76:0/1938315386 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6f9403d030 con 0x5653bb485c90 2024-03-12T06:04:19.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.426+0000 7f6facd85740 10 client.26516 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:19.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.426+0000 7f6facd85740 10 client.26516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:19.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.426+0000 7f6facd85740 20 client.26516 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:19.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.426+0000 7f6facd85740 1 --2- 172.21.15.76:0/1938315386 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653bb4ff760 0x5653bb501b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:19.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.426+0000 7f6facd85740 4 client.26516 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:04:19.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.426+0000 7f6facd85740 1 -- 172.21.15.76:0/1938315386 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5653bb179890 con 0x5653bb4ff760 2024-03-12T06:04:19.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.426+0000 7f6fa2ffd640 1 --2- 172.21.15.76:0/1938315386 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653bb4ff760 0x5653bb501b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:19.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6fa2ffd640 1 --2- 172.21.15.76:0/1938315386 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653bb4ff760 0x5653bb501b50 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e02e68c28d0987d0 server_cookie=723f847187241abe in_seq=0 out_seq=0 2024-03-12T06:04:19.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6f82ffd640 10 client.26516 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:19.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6f82ffd640 1 -- 172.21.15.76:0/1938315386 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5653bb179890 con 0x5653bb4ff760 2024-03-12T06:04:19.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6f82ffd640 10 client.26516 handle_command_reply: tid=0 2024-03-12T06:04:19.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6facd85740 1 client.26516 shutdown 2024-03-12T06:04:19.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6facd85740 1 -- 172.21.15.76:0/1938315386 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb485c90 msgr2=0x5653bb49f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6facd85740 1 --2- 172.21.15.76:0/1938315386 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb485c90 0x5653bb49f0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6f94011510 tx=0x7f6f94005df0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6facd85740 1 -- 172.21.15.76:0/1938315386 shutdown_connections 2024-03-12T06:04:19.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6facd85740 1 --2- 172.21.15.76:0/1938315386 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5653bb4ff760 0x5653bb501b50 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6facd85740 1 --2- 172.21.15.76:0/1938315386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5653bb497fc0 0x5653bb49b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6facd85740 1 --2- 172.21.15.76:0/1938315386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653bb486640 0x5653bb49b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6facd85740 1 --2- 172.21.15.76:0/1938315386 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5653bb485c90 0x5653bb49f0c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6f82ffd640 0 client.26516 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:19.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6facd85740 1 -- 172.21.15.76:0/1938315386 >> 172.21.15.76:0/1938315386 conn(0x5653bb480990 msgr2=0x5653bb47f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:19.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6facd85740 1 -- 172.21.15.76:0/1938315386 shutdown_connections 2024-03-12T06:04:19.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6facd85740 1 -- 172.21.15.76:0/1938315386 wait complete. 2024-03-12T06:04:19.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.427+0000 7f6facd85740 20 client.26516 trim_cache size 0 max 16384 2024-03-12T06:04:19.442 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:04:19.442 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T06:04:19.443 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T06:04:19.443 INFO:teuthology.orchestra.run.smithi076.stdout: "eb3b7b37-b297-46ae-945b-66ae5d97b4c3": { 2024-03-12T06:04:19.443 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T06:04:19.443 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "eb3b7b37-b297-46ae-945b-66ae5d97b4c3", 2024-03-12T06:04:19.443 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T06:04:19.443 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:04:19.443 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:04:19.443 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:04:19.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.428+0000 7f6faa1b6640 1 -- 172.21.15.76:0/3609101638 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f6f8c0a1bd0 msgr2=0x7f6f8c0a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.428+0000 7f6faa1b6640 1 --2- 172.21.15.76:0/3609101638 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f6f8c0a1bd0 0x7f6f8c0a4090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c006010 tx=0x7f6f9c02e040 comp rx=0 tx=0).stop 2024-03-12T06:04:19.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.428+0000 7f6faa1b6640 1 -- 172.21.15.76:0/3609101638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fa4106430 msgr2=0x7f6fa4100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:19.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.428+0000 7f6faa1b6640 1 --2- 172.21.15.76:0/3609101638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fa4106430 0x7f6fa4100580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800e9f0 tx=0x7f6f9800edf0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.428+0000 7f6faa1b6640 1 -- 172.21.15.76:0/3609101638 shutdown_connections 2024-03-12T06:04:19.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.428+0000 7f6faa1b6640 1 --2- 172.21.15.76:0/3609101638 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f6f8c0a1bd0 0x7f6f8c0a4090 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.428+0000 7f6faa1b6640 1 --2- 172.21.15.76:0/3609101638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fa4100ac0 0x7f6fa41a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.428+0000 7f6faa1b6640 1 --2- 172.21.15.76:0/3609101638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fa4106430 0x7f6fa4100580 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.428+0000 7f6faa1b6640 1 --2- 172.21.15.76:0/3609101638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6fa4068740 0x7f6fa4100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:19.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.428+0000 7f6faa1b6640 1 -- 172.21.15.76:0/3609101638 >> 172.21.15.76:0/3609101638 conn(0x7f6fa4077180 msgr2=0x7f6fa40fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:19.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.428+0000 7f6faa1b6640 1 -- 172.21.15.76:0/3609101638 shutdown_connections 2024-03-12T06:04:19.445 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:19.429+0000 7f6faa1b6640 1 -- 172.21.15.76:0/3609101638 wait complete. 2024-03-12T06:04:19.446 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:19 smithi076 ceph-mon[32095]: pgmap v265: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:04:19.446 INFO:tasks.cephfs.filesystem:scrub status for tag:eb3b7b37-b297-46ae-945b-66ae5d97b4c3 - {'path': '/', 'tag': 'eb3b7b37-b297-46ae-945b-66ae5d97b4c3', 'options': 'recursive,force'} 2024-03-12T06:04:19.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:19 smithi082 ceph-mon[36382]: pgmap v265: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:04:19.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:19 smithi088 ceph-mon[36585]: pgmap v265: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:04:20.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:20 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:04:20.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:20 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:04:20.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:20 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:20.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:20 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:20.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:20 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:04:20.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:20 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:04:20.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:20 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:20.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:20 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:20.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:20 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:04:20.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:20 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:04:20.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:20 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:20.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:20 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:21.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:21 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T06:04:21.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:21 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:21.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:21 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:04:21.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:21 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:21.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:21 smithi082 ceph-mon[36382]: pgmap v266: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:04:21.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:21 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T06:04:21.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:21 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:21.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:21 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:04:21.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:21 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:21.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:21 smithi088 ceph-mon[36585]: pgmap v266: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:04:21.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:21 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T06:04:21.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:21 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:21.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:21 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:04:21.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:21 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:21.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:21 smithi076 ceph-mon[32095]: pgmap v266: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:04:23.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:23 smithi076 ceph-mon[32095]: pgmap v267: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-12T06:04:23.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:23 smithi082 ceph-mon[36382]: pgmap v267: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-12T06:04:23.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:23 smithi088 ceph-mon[36585]: pgmap v267: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 87 B/s wr, 8 op/s 2024-03-12T06:04:25.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:25 smithi076 ceph-mon[32095]: pgmap v268: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:25.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:25 smithi082 ceph-mon[36382]: pgmap v268: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:25.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:25 smithi088 ceph-mon[36585]: pgmap v268: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:27.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:27 smithi076 ceph-mon[32095]: pgmap v269: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:27.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:27 smithi082 ceph-mon[36382]: pgmap v269: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:27.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:27 smithi088 ceph-mon[36585]: pgmap v269: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:29.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:29 smithi076 ceph-mon[32095]: pgmap v270: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:04:29.498 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:29 smithi082 ceph-mon[36382]: pgmap v270: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:04:29.538 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:29 smithi088 ceph-mon[36585]: pgmap v270: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:04:31.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:31 smithi076 ceph-mon[32095]: pgmap v271: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:31.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:31 smithi082 ceph-mon[36382]: pgmap v271: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:31.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:31 smithi088 ceph-mon[36585]: pgmap v271: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:32.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:32 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:04:32.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:32 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:04:32.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:32 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:04:33.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:33 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:04:33.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:33 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:04:33.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:33 smithi082 ceph-mon[36382]: pgmap v272: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:04:33.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:33 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:04:33.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:33 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:04:33.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:33 smithi088 ceph-mon[36585]: pgmap v272: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:04:33.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:33 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:04:33.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:33 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:04:33.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:33 smithi076 ceph-mon[32095]: pgmap v272: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:04:35.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:35 smithi076 ceph-mon[32095]: pgmap v273: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:35.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:35 smithi082 ceph-mon[36382]: pgmap v273: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:35.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:35 smithi088 ceph-mon[36585]: pgmap v273: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:37.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:37 smithi076 ceph-mon[32095]: pgmap v274: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:37.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:37 smithi082 ceph-mon[36382]: pgmap v274: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:37.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:37 smithi088 ceph-mon[36585]: pgmap v274: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:39.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:39 smithi076 ceph-mon[32095]: pgmap v275: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:04:39.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:39 smithi082 ceph-mon[36382]: pgmap v275: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:04:39.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:39 smithi088 ceph-mon[36585]: pgmap v275: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:04:41.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:41 smithi076 ceph-mon[32095]: pgmap v276: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:41.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:41 smithi082 ceph-mon[36382]: pgmap v276: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:41.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:41 smithi088 ceph-mon[36585]: pgmap v276: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:43.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:43 smithi076 ceph-mon[32095]: pgmap v277: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:04:43.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:43 smithi082 ceph-mon[36382]: pgmap v277: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:04:43.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:43 smithi088 ceph-mon[36585]: pgmap v277: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:04:45.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:45 smithi082 ceph-mon[36382]: pgmap v278: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:45.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:45 smithi088 ceph-mon[36585]: pgmap v278: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:45.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:45 smithi076 ceph-mon[32095]: pgmap v278: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:47.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:47 smithi082 ceph-mon[36382]: pgmap v279: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:47.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:47 smithi088 ceph-mon[36585]: pgmap v279: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:47.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:47 smithi076 ceph-mon[32095]: pgmap v279: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:49.447 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:04:49.447 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:04:49.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:49 smithi076 ceph-mon[32095]: pgmap v280: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:04:49.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.557+0000 7f9c124ae640 1 -- 172.21.15.76:0/3110641891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c0c100c10 msgr2=0x7f9c0c104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:49.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.557+0000 7f9c124ae640 1 --2- 172.21.15.76:0/3110641891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c0c100c10 0x7f9c0c104c50 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009930 tx=0x7f9c0002bc30 comp rx=0 tx=0).stop 2024-03-12T06:04:49.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.557+0000 7f9c124ae640 1 -- 172.21.15.76:0/3110641891 shutdown_connections 2024-03-12T06:04:49.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.557+0000 7f9c124ae640 1 --2- 172.21.15.76:0/3110641891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c0c105330 0x7f9c0c10cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.557+0000 7f9c124ae640 1 --2- 172.21.15.76:0/3110641891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c0c100c10 0x7f9c0c104c50 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.557+0000 7f9c124ae640 1 --2- 172.21.15.76:0/3110641891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c0c100260 0x7f9c0c100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.557+0000 7f9c124ae640 1 -- 172.21.15.76:0/3110641891 >> 172.21.15.76:0/3110641891 conn(0x7f9c0c0771b0 msgr2=0x7f9c0c0775c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:49.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.557+0000 7f9c124ae640 1 -- 172.21.15.76:0/3110641891 shutdown_connections 2024-03-12T06:04:49.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.557+0000 7f9c124ae640 1 -- 172.21.15.76:0/3110641891 wait complete. 2024-03-12T06:04:49.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.558+0000 7f9c124ae640 1 Processor -- start 2024-03-12T06:04:49.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.558+0000 7f9c124ae640 1 -- start start 2024-03-12T06:04:49.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.558+0000 7f9c124ae640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c0c100260 0x7f9c0c19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.558+0000 7f9c124ae640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c0c105330 0x7f9c0c19d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.558+0000 7f9c124ae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c0c197610 0x7f9c0c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.558+0000 7f9c124ae640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c198000 con 0x7f9c0c105330 2024-03-12T06:04:49.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.558+0000 7f9c124ae640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c198170 con 0x7f9c0c100260 2024-03-12T06:04:49.566 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.558+0000 7f9c124ae640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c198330 con 0x7f9c0c197610 2024-03-12T06:04:49.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.558+0000 7f9c0b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c0c105330 0x7f9c0c19d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.558+0000 7f9c0b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c0c105330 0x7f9c0c19d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52814/0 (socket says 172.21.15.76:52814) 2024-03-12T06:04:49.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.558+0000 7f9c0b7fe640 1 -- 172.21.15.76:0/4184033875 learned_addr learned my addr 172.21.15.76:0/4184033875 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:49.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.558+0000 7f9c0bfff640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c0c100260 0x7f9c0c19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.558+0000 7f9c0b7fe640 1 -- 172.21.15.76:0/4184033875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c0c197610 msgr2=0x7f9c0c197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:49.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.558+0000 7f9c10a4b640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c0c197610 0x7f9c0c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.559+0000 7f9c0b7fe640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c0c197610 0x7f9c0c197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.559+0000 7f9c0b7fe640 1 -- 172.21.15.76:0/4184033875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c0c100260 msgr2=0x7f9c0c19d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:49.567 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.559+0000 7f9c0b7fe640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c0c100260 0x7f9c0c19d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.559+0000 7f9c0b7fe640 1 -- 172.21.15.76:0/4184033875 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c00009560 con 0x7f9c0c105330 2024-03-12T06:04:49.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.559+0000 7f9c10a4b640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c0c197610 0x7f9c0c197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:49.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.559+0000 7f9c0b7fe640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c0c105330 0x7f9c0c19d9a0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9c00000c00 tx=0x7f9c00004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:49.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.559+0000 7f9c097fa640 1 -- 172.21.15.76:0/4184033875 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c00004df0 con 0x7f9c0c105330 2024-03-12T06:04:49.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.559+0000 7f9c124ae640 1 -- 172.21.15.76:0/4184033875 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c0c1a4320 con 0x7f9c0c105330 2024-03-12T06:04:49.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.559+0000 7f9c097fa640 1 -- 172.21.15.76:0/4184033875 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c00005430 con 0x7f9c0c105330 2024-03-12T06:04:49.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.559+0000 7f9c097fa640 1 -- 172.21.15.76:0/4184033875 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0003e450 con 0x7f9c0c105330 2024-03-12T06:04:49.568 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.559+0000 7f9c124ae640 1 -- 172.21.15.76:0/4184033875 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0c1a47d0 con 0x7f9c0c105330 2024-03-12T06:04:49.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c097fa640 1 -- 172.21.15.76:0/4184033875 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f9c00003680 con 0x7f9c0c105330 2024-03-12T06:04:49.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c15080740 1 Processor -- start 2024-03-12T06:04:49.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c15080740 1 -- start start 2024-03-12T06:04:49.575 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c15080740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b302b0 0x5605a3b30690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c15080740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b30bd0 0x5605a3b39270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c15080740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b397b0 0x5605a3b3bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c097fa640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9bf40a1ba0 0x7f9bf40a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c15080740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a3a9f280 con 0x5605a3b30bd0 2024-03-12T06:04:49.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c15080740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a3a9c7b0 con 0x5605a3b397b0 2024-03-12T06:04:49.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c15080740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a3a97f90 con 0x5605a3b302b0 2024-03-12T06:04:49.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c0bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b30bd0 0x5605a3b39270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c10a4b640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9bf40a1ba0 0x7f9bf40a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c0bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b30bd0 0x5605a3b39270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52830/0 (socket says 172.21.15.76:52830) 2024-03-12T06:04:49.576 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c0bfff640 1 -- 172.21.15.76:0/3506655333 learned_addr learned my addr 172.21.15.76:0/3506655333 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:49.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c10a4b640 1 --2- 172.21.15.76:0/3506655333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b302b0 0x5605a3b30690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.562+0000 7f9c097fa640 1 -- 172.21.15.76:0/4184033875 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9c000efbf0 con 0x7f9c0c105330 2024-03-12T06:04:49.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.563+0000 7f9c0b7fe640 1 --2- 172.21.15.76:0/3506655333 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b397b0 0x5605a3b3bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.563+0000 7f9c0bfff640 1 -- 172.21.15.76:0/3506655333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b302b0 msgr2=0x5605a3b30690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:49.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.563+0000 7f9c0bfff640 1 --2- 172.21.15.76:0/3506655333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b302b0 0x5605a3b30690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.563+0000 7f9c0bfff640 1 -- 172.21.15.76:0/3506655333 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b397b0 msgr2=0x5605a3b3bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:49.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.563+0000 7f9c0bfff640 1 --2- 172.21.15.76:0/3506655333 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b397b0 0x5605a3b3bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.563+0000 7f9c0bfff640 1 -- 172.21.15.76:0/3506655333 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605a3a56570 con 0x5605a3b30bd0 2024-03-12T06:04:49.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.563+0000 7f9c0b7fe640 1 --2- 172.21.15.76:0/3506655333 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b397b0 0x5605a3b3bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:49.577 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.563+0000 7f9c10a4b640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9bf40a1ba0 0x7f9bf40a4060 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c040129c0 tx=0x7f9c0400ac00 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:49.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.563+0000 7f9c0bfff640 1 --2- 172.21.15.76:0/3506655333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b30bd0 0x5605a3b39270 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00ac20 tx=0x7f9bfc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b10942d5e9e396a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:49.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.563+0000 7f9bf27fc640 1 -- 172.21.15.76:0/3506655333 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc00a720 con 0x5605a3b30bd0 2024-03-12T06:04:49.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.563+0000 7f9bf27fc640 1 -- 172.21.15.76:0/3506655333 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9bfc018b20 con 0x5605a3b30bd0 2024-03-12T06:04:49.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.563+0000 7f9bf27fc640 1 -- 172.21.15.76:0/3506655333 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc010ba0 con 0x5605a3b30bd0 2024-03-12T06:04:49.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.564+0000 7f9c15080740 1 -- 172.21.15.76:0/3506655333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b30bd0 msgr2=0x5605a3b39270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:49.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.564+0000 7f9c15080740 1 --2- 172.21.15.76:0/3506655333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b30bd0 0x5605a3b39270 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00ac20 tx=0x7f9bfc00f790 comp rx=0 tx=0).stop 2024-03-12T06:04:49.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.564+0000 7f9c15080740 1 -- 172.21.15.76:0/3506655333 shutdown_connections 2024-03-12T06:04:49.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.564+0000 7f9c15080740 1 --2- 172.21.15.76:0/3506655333 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b397b0 0x5605a3b3bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.578 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.564+0000 7f9c15080740 1 --2- 172.21.15.76:0/3506655333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b30bd0 0x5605a3b39270 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.564+0000 7f9c15080740 1 --2- 172.21.15.76:0/3506655333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b302b0 0x5605a3b30690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.564+0000 7f9c15080740 1 -- 172.21.15.76:0/3506655333 >> 172.21.15.76:0/3506655333 conn(0x5605a3b2a990 msgr2=0x5605a3b2ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:49.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.564+0000 7f9c15080740 1 -- 172.21.15.76:0/3506655333 shutdown_connections 2024-03-12T06:04:49.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.564+0000 7f9c15080740 1 -- 172.21.15.76:0/3506655333 wait complete. 2024-03-12T06:04:49.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.564+0000 7f9c15080740 1 -- start start 2024-03-12T06:04:49.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.564+0000 7f9c15080740 1 Processor -- start 2024-03-12T06:04:49.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.564+0000 7f9c15080740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b302b0 0x5605a3b39220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c15080740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b30bd0 0x5605a3b365b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c15080740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b397b0 0x5605a3b36cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.579 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c15080740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a376b430 con 0x5605a3b397b0 2024-03-12T06:04:49.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c15080740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a3727e00 con 0x5605a3b302b0 2024-03-12T06:04:49.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c15080740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a3a1cfd0 con 0x5605a3b30bd0 2024-03-12T06:04:49.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c15080740 10 client.0 _ll_register_callbacks cb 0x5605a3a72fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:49.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c10a4b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b397b0 0x5605a3b36cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c10a4b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b397b0 0x5605a3b36cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52836/0 (socket says 172.21.15.76:52836) 2024-03-12T06:04:49.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c10a4b640 1 -- 172.21.15.76:0/825313829 learned_addr learned my addr 172.21.15.76:0/825313829 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:49.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c0bfff640 1 --2- 172.21.15.76:0/825313829 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b302b0 0x5605a3b39220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c0b7fe640 1 --2- 172.21.15.76:0/825313829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b30bd0 0x5605a3b365b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c10a4b640 1 -- 172.21.15.76:0/825313829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b30bd0 msgr2=0x5605a3b365b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:49.580 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c10a4b640 1 --2- 172.21.15.76:0/825313829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b30bd0 0x5605a3b365b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c10a4b640 1 -- 172.21.15.76:0/825313829 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b302b0 msgr2=0x5605a3b39220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:49.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c10a4b640 1 --2- 172.21.15.76:0/825313829 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b302b0 0x5605a3b39220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c10a4b640 1 -- 172.21.15.76:0/825313829 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bfc00a880 con 0x5605a3b397b0 2024-03-12T06:04:49.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c10a4b640 1 --2- 172.21.15.76:0/825313829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b397b0 0x5605a3b36cf0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400fe90 tx=0x7f9c04005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c23d4a2f82d4bd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:49.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9beb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:49.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9beb7fe640 1 -- 172.21.15.76:0/825313829 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c04028430 con 0x5605a3b397b0 2024-03-12T06:04:49.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9beb7fe640 1 -- 172.21.15.76:0/825313829 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9c0401e3f0 con 0x5605a3b397b0 2024-03-12T06:04:49.581 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c15080740 1 -- 172.21.15.76:0/825313829 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5605a37977e0 con 0x5605a3b397b0 2024-03-12T06:04:49.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9beb7fe640 1 -- 172.21.15.76:0/825313829 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c0401e5b0 con 0x5605a3b397b0 2024-03-12T06:04:49.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c15080740 10 client.17472 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:49.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.565+0000 7f9c15080740 1 -- 172.21.15.76:0/825313829 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5605a3a96610 con 0x5605a3b397b0 2024-03-12T06:04:49.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.566+0000 7f9beb7fe640 1 -- 172.21.15.76:0/825313829 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c04034080 con 0x5605a3b397b0 2024-03-12T06:04:49.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.566+0000 7f9beb7fe640 1 -- 172.21.15.76:0/825313829 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9c0402e030 con 0x5605a3b397b0 2024-03-12T06:04:49.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.566+0000 7f9c15080740 10 client.17472 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:49.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.566+0000 7f9c15080740 10 client.17472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:49.582 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.566+0000 7f9c15080740 20 client.17472 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:49.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.566+0000 7f9c15080740 1 --2- 172.21.15.76:0/825313829 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5605a3b49970 0x5605a3b4bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.566+0000 7f9c15080740 4 client.17472 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:04:49.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.566+0000 7f9c15080740 1 -- 172.21.15.76:0/825313829 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5605a37d4720 con 0x5605a3b49970 2024-03-12T06:04:49.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.566+0000 7f9c0bfff640 1 --2- 172.21.15.76:0/825313829 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5605a3b49970 0x5605a3b4bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.566+0000 7f9c0bfff640 1 --2- 172.21.15.76:0/825313829 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5605a3b49970 0x5605a3b4bd90 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42095ff562d16627 server_cookie=93024d5d2c435638 in_seq=0 out_seq=0 2024-03-12T06:04:49.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.566+0000 7f9beb7fe640 10 client.17472 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:49.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.567+0000 7f9beb7fe640 1 -- 172.21.15.76:0/825313829 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5605a37d4720 con 0x5605a3b49970 2024-03-12T06:04:49.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.567+0000 7f9beb7fe640 10 client.17472 handle_command_reply: tid=0 2024-03-12T06:04:49.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.567+0000 7f9c15080740 1 client.17472 shutdown 2024-03-12T06:04:49.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.567+0000 7f9c15080740 1 -- 172.21.15.76:0/825313829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b397b0 msgr2=0x5605a3b36cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:49.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.567+0000 7f9c15080740 1 --2- 172.21.15.76:0/825313829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b397b0 0x5605a3b36cf0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400fe90 tx=0x7f9c04005c30 comp rx=0 tx=0).stop 2024-03-12T06:04:49.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.567+0000 7f9c15080740 1 -- 172.21.15.76:0/825313829 shutdown_connections 2024-03-12T06:04:49.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.567+0000 7f9c15080740 1 --2- 172.21.15.76:0/825313829 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5605a3b49970 0x5605a3b4bd90 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.567+0000 7f9c15080740 1 --2- 172.21.15.76:0/825313829 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b397b0 0x5605a3b36cf0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.567+0000 7f9c15080740 1 --2- 172.21.15.76:0/825313829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b30bd0 0x5605a3b365b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.567+0000 7f9beb7fe640 0 client.17472 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:49.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.567+0000 7f9c15080740 1 --2- 172.21.15.76:0/825313829 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b302b0 0x5605a3b39220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.567+0000 7f9c15080740 1 -- 172.21.15.76:0/825313829 >> 172.21.15.76:0/825313829 conn(0x5605a3b2a990 msgr2=0x5605a3b2c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:49.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.567+0000 7f9c15080740 1 -- 172.21.15.76:0/825313829 shutdown_connections 2024-03-12T06:04:49.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.567+0000 7f9c15080740 1 -- 172.21.15.76:0/825313829 wait complete. 2024-03-12T06:04:49.585 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.568+0000 7f9c15080740 20 client.17472 trim_cache size 0 max 16384 2024-03-12T06:04:49.593 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.582+0000 7f9c15080740 1 Processor -- start 2024-03-12T06:04:49.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.582+0000 7f9c15080740 1 -- start start 2024-03-12T06:04:49.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c15080740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b302b0 0x5605a3b36550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c15080740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b30bd0 0x5605a3b38ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c15080740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b397b0 0x5605a3b39220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c15080740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a3b260f0 con 0x5605a3b397b0 2024-03-12T06:04:49.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c15080740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a3b49720 con 0x5605a3b302b0 2024-03-12T06:04:49.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c15080740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a3b34b00 con 0x5605a3b30bd0 2024-03-12T06:04:49.594 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c10a4b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b397b0 0x5605a3b39220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c10a4b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b397b0 0x5605a3b39220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52838/0 (socket says 172.21.15.76:52838) 2024-03-12T06:04:49.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c10a4b640 1 -- 172.21.15.76:0/2496129593 learned_addr learned my addr 172.21.15.76:0/2496129593 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:49.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c10a4b640 1 -- 172.21.15.76:0/2496129593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b30bd0 msgr2=0x5605a3b38ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:49.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c0b7fe640 1 --2- 172.21.15.76:0/2496129593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b30bd0 0x5605a3b38ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c0bfff640 1 --2- 172.21.15.76:0/2496129593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b302b0 0x5605a3b36550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c10a4b640 1 --2- 172.21.15.76:0/2496129593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b30bd0 0x5605a3b38ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c10a4b640 1 -- 172.21.15.76:0/2496129593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b302b0 msgr2=0x5605a3b36550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:49.595 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c10a4b640 1 --2- 172.21.15.76:0/2496129593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b302b0 0x5605a3b36550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c10a4b640 1 -- 172.21.15.76:0/2496129593 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605a3a32ac0 con 0x5605a3b397b0 2024-03-12T06:04:49.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c0bfff640 1 --2- 172.21.15.76:0/2496129593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b302b0 0x5605a3b36550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:49.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c0b7fe640 1 --2- 172.21.15.76:0/2496129593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b30bd0 0x5605a3b38ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:49.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9c10a4b640 1 --2- 172.21.15.76:0/2496129593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b397b0 0x5605a3b39220 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400a330 tx=0x7f9c0400a5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d32b1e211ae501a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:49.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9bf1ffb640 1 -- 172.21.15.76:0/2496129593 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c04012680 con 0x5605a3b397b0 2024-03-12T06:04:49.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9bf1ffb640 1 -- 172.21.15.76:0/2496129593 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5605a37977e0 con 0x5605a3b397b0 2024-03-12T06:04:49.596 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.583+0000 7f9bf1ffb640 1 -- 172.21.15.76:0/2496129593 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0403b8f0 con 0x5605a3b397b0 2024-03-12T06:04:49.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.584+0000 7f9c15080740 1 -- 172.21.15.76:0/2496129593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b397b0 msgr2=0x5605a3b39220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:49.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.584+0000 7f9c15080740 1 --2- 172.21.15.76:0/2496129593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b397b0 0x5605a3b39220 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400a330 tx=0x7f9c0400a5e0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.584+0000 7f9c15080740 1 -- 172.21.15.76:0/2496129593 shutdown_connections 2024-03-12T06:04:49.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.584+0000 7f9c15080740 1 --2- 172.21.15.76:0/2496129593 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b397b0 0x5605a3b39220 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.584+0000 7f9c15080740 1 --2- 172.21.15.76:0/2496129593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b30bd0 0x5605a3b38ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.584+0000 7f9c15080740 1 --2- 172.21.15.76:0/2496129593 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b302b0 0x5605a3b36550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.584+0000 7f9c15080740 1 -- 172.21.15.76:0/2496129593 >> 172.21.15.76:0/2496129593 conn(0x5605a3b2a990 msgr2=0x5605a3b35e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:49.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.584+0000 7f9c15080740 1 -- 172.21.15.76:0/2496129593 shutdown_connections 2024-03-12T06:04:49.597 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.584+0000 7f9c15080740 1 -- 172.21.15.76:0/2496129593 wait complete. 2024-03-12T06:04:49.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.584+0000 7f9c15080740 1 -- start start 2024-03-12T06:04:49.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.584+0000 7f9c15080740 1 Processor -- start 2024-03-12T06:04:49.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c15080740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b302b0 0x5605a3b40390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c15080740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b30bd0 0x5605a3b408d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c15080740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b397b0 0x5605a3b3d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c15080740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a3b3df10 con 0x5605a3b30bd0 2024-03-12T06:04:49.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c15080740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a3b3e080 con 0x5605a3b397b0 2024-03-12T06:04:49.598 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c15080740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a3b3e240 con 0x5605a3b302b0 2024-03-12T06:04:49.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c15080740 10 client.0 _ll_register_callbacks cb 0x7f9bfc00a670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:49.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c0b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b30bd0 0x5605a3b408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c0b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b30bd0 0x5605a3b408d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52854/0 (socket says 172.21.15.76:52854) 2024-03-12T06:04:49.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c0b7fe640 1 -- 172.21.15.76:0/54207486 learned_addr learned my addr 172.21.15.76:0/54207486 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:49.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c10a4b640 1 --2- 172.21.15.76:0/54207486 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b397b0 0x5605a3b3d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c0bfff640 1 --2- 172.21.15.76:0/54207486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b302b0 0x5605a3b40390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c0b7fe640 1 -- 172.21.15.76:0/54207486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b302b0 msgr2=0x5605a3b40390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:49.599 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c0b7fe640 1 --2- 172.21.15.76:0/54207486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b302b0 0x5605a3b40390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c0b7fe640 1 -- 172.21.15.76:0/54207486 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b397b0 msgr2=0x5605a3b3d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:49.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c0b7fe640 1 --2- 172.21.15.76:0/54207486 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b397b0 0x5605a3b3d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c0b7fe640 1 -- 172.21.15.76:0/54207486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605a3a96610 con 0x5605a3b30bd0 2024-03-12T06:04:49.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c0b7fe640 1 --2- 172.21.15.76:0/54207486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b30bd0 0x5605a3b408d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f9c00005ec0 tx=0x7f9c000bc530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a1e2f641f0cc28 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:49.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9beb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:49.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9beb7fe640 1 -- 172.21.15.76:0/54207486 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c000bc930 con 0x5605a3b30bd0 2024-03-12T06:04:49.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9beb7fe640 1 -- 172.21.15.76:0/54207486 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9c000c0a80 con 0x5605a3b30bd0 2024-03-12T06:04:49.600 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.585+0000 7f9c15080740 1 -- 172.21.15.76:0/54207486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9bfc018b20 con 0x5605a3b30bd0 2024-03-12T06:04:49.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.586+0000 7f9beb7fe640 1 -- 172.21.15.76:0/54207486 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c000c50d0 con 0x5605a3b30bd0 2024-03-12T06:04:49.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.586+0000 7f9c15080740 10 client.17478 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:49.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.586+0000 7f9c15080740 1 -- 172.21.15.76:0/54207486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5605a3a586a0 con 0x5605a3b30bd0 2024-03-12T06:04:49.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.586+0000 7f9beb7fe640 1 -- 172.21.15.76:0/54207486 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9bfc018b20 con 0x5605a3b30bd0 2024-03-12T06:04:49.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.586+0000 7f9beb7fe640 1 -- 172.21.15.76:0/54207486 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9c0003c070 con 0x5605a3b30bd0 2024-03-12T06:04:49.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.586+0000 7f9c15080740 10 client.17478 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:49.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.586+0000 7f9c15080740 10 client.17478 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:49.601 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.586+0000 7f9c15080740 20 client.17478 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:49.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.586+0000 7f9c15080740 1 --2- 172.21.15.76:0/54207486 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5605a3ba1ec0 0x5605a3ba42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.586+0000 7f9c15080740 4 client.17478 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:04:49.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.586+0000 7f9c15080740 1 -- 172.21.15.76:0/54207486 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5605a3823890 con 0x5605a3ba1ec0 2024-03-12T06:04:49.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.586+0000 7f9c0bfff640 1 --2- 172.21.15.76:0/54207486 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5605a3ba1ec0 0x5605a3ba42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9beb7fe640 10 client.17478 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:49.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9c0bfff640 1 --2- 172.21.15.76:0/54207486 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5605a3ba1ec0 0x5605a3ba42e0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=981248f2509f9521 server_cookie=4c46193e6b2f9c67 in_seq=0 out_seq=0 2024-03-12T06:04:49.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9beb7fe640 1 -- 172.21.15.76:0/54207486 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5605a3823890 con 0x5605a3ba1ec0 2024-03-12T06:04:49.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9beb7fe640 10 client.17478 handle_command_reply: tid=0 2024-03-12T06:04:49.602 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9c15080740 1 client.17478 shutdown 2024-03-12T06:04:49.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9c15080740 1 -- 172.21.15.76:0/54207486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b30bd0 msgr2=0x5605a3b408d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:49.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9c15080740 1 --2- 172.21.15.76:0/54207486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b30bd0 0x5605a3b408d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f9c00005ec0 tx=0x7f9c000bc530 comp rx=0 tx=0).stop 2024-03-12T06:04:49.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9c15080740 1 -- 172.21.15.76:0/54207486 shutdown_connections 2024-03-12T06:04:49.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9c15080740 1 --2- 172.21.15.76:0/54207486 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5605a3b397b0 0x5605a3b3d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9c15080740 1 --2- 172.21.15.76:0/54207486 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5605a3ba1ec0 0x5605a3ba42e0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9c15080740 1 --2- 172.21.15.76:0/54207486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5605a3b30bd0 0x5605a3b408d0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9beb7fe640 0 client.17478 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:49.603 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9c15080740 1 --2- 172.21.15.76:0/54207486 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605a3b302b0 0x5605a3b40390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9c15080740 1 -- 172.21.15.76:0/54207486 >> 172.21.15.76:0/54207486 conn(0x5605a3b2a990 msgr2=0x5605a3b34e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:49.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9c15080740 1 -- 172.21.15.76:0/54207486 shutdown_connections 2024-03-12T06:04:49.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9c15080740 1 -- 172.21.15.76:0/54207486 wait complete. 2024-03-12T06:04:49.604 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.587+0000 7f9c15080740 20 client.17478 trim_cache size 0 max 16384 2024-03-12T06:04:49.604 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:04:49.604 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T06:04:49.604 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T06:04:49.605 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:04:49.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.588+0000 7f9c124ae640 1 -- 172.21.15.76:0/4184033875 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9bf40a1ba0 msgr2=0x7f9bf40a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:49.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.588+0000 7f9c124ae640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9bf40a1ba0 0x7f9bf40a4060 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c040129c0 tx=0x7f9c0400ac00 comp rx=0 tx=0).stop 2024-03-12T06:04:49.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.588+0000 7f9c124ae640 1 -- 172.21.15.76:0/4184033875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c0c105330 msgr2=0x7f9c0c19d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:49.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.588+0000 7f9c124ae640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c0c105330 0x7f9c0c19d9a0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f9c00000c00 tx=0x7f9c00004270 comp rx=0 tx=0).stop 2024-03-12T06:04:49.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.588+0000 7f9c124ae640 1 -- 172.21.15.76:0/4184033875 shutdown_connections 2024-03-12T06:04:49.606 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.588+0000 7f9c124ae640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f9bf40a1ba0 0x7f9bf40a4060 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.588+0000 7f9c124ae640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c0c197610 0x7f9c0c197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.588+0000 7f9c124ae640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9c0c105330 0x7f9c0c19d9a0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.588+0000 7f9c124ae640 1 --2- 172.21.15.76:0/4184033875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9c0c100260 0x7f9c0c19d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.588+0000 7f9c124ae640 1 -- 172.21.15.76:0/4184033875 >> 172.21.15.76:0/4184033875 conn(0x7f9c0c0771b0 msgr2=0x7f9c0c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:49.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.588+0000 7f9c124ae640 1 -- 172.21.15.76:0/4184033875 shutdown_connections 2024-03-12T06:04:49.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.588+0000 7f9c124ae640 1 -- 172.21.15.76:0/4184033875 wait complete. 2024-03-12T06:04:49.607 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T06:04:49.608 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T06:04:49.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:49 smithi082 ceph-mon[36382]: pgmap v280: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:04:49.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.705+0000 7f2d70d3e640 1 -- 172.21.15.76:0/1236135872 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d6c105370 msgr2=0x7f2d6c10cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:49.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.705+0000 7f2d70d3e640 1 --2- 172.21.15.76:0/1236135872 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d6c105370 0x7f2d6c10cf40 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f2d64009cd0 tx=0x7f2d6402c0f0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.705+0000 7f2d70d3e640 1 -- 172.21.15.76:0/1236135872 shutdown_connections 2024-03-12T06:04:49.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.705+0000 7f2d70d3e640 1 --2- 172.21.15.76:0/1236135872 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d6c105370 0x7f2d6c10cf40 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.705+0000 7f2d70d3e640 1 --2- 172.21.15.76:0/1236135872 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d6c100c50 0x7f2d6c104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.705+0000 7f2d70d3e640 1 --2- 172.21.15.76:0/1236135872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d6c1002a0 0x7f2d6c100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.705+0000 7f2d70d3e640 1 -- 172.21.15.76:0/1236135872 >> 172.21.15.76:0/1236135872 conn(0x7f2d6c077200 msgr2=0x7f2d6c077610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:49.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.705+0000 7f2d70d3e640 1 -- 172.21.15.76:0/1236135872 shutdown_connections 2024-03-12T06:04:49.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.705+0000 7f2d70d3e640 1 -- 172.21.15.76:0/1236135872 wait complete. 2024-03-12T06:04:49.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d70d3e640 1 Processor -- start 2024-03-12T06:04:49.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d70d3e640 1 -- start start 2024-03-12T06:04:49.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d70d3e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d6c1002a0 0x7f2d6c10d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d70d3e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d6c105370 0x7f2d6c10dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d70d3e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d6c10e2a0 0x7f2d6c1a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d70d3e640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c1a5ed0 con 0x7f2d6c10e2a0 2024-03-12T06:04:49.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d70d3e640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c1a6040 con 0x7f2d6c105370 2024-03-12T06:04:49.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d70d3e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c1a6200 con 0x7f2d6c1002a0 2024-03-12T06:04:49.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d6ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d6c10e2a0 0x7f2d6c1a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d6ad9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d6c10e2a0 0x7f2d6c1a5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52882/0 (socket says 172.21.15.76:52882) 2024-03-12T06:04:49.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d6ad9d640 1 -- 172.21.15.76:0/704777745 learned_addr learned my addr 172.21.15.76:0/704777745 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:49.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d6a59c640 1 --2- 172.21.15.76:0/704777745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d6c1002a0 0x7f2d6c10d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d69d9b640 1 --2- 172.21.15.76:0/704777745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d6c105370 0x7f2d6c10dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d6ad9d640 1 -- 172.21.15.76:0/704777745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d6c1002a0 msgr2=0x7f2d6c10d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:49.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d6ad9d640 1 --2- 172.21.15.76:0/704777745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d6c1002a0 0x7f2d6c10d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d6ad9d640 1 -- 172.21.15.76:0/704777745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d6c105370 msgr2=0x7f2d6c10dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:49.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d6ad9d640 1 --2- 172.21.15.76:0/704777745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d6c105370 0x7f2d6c10dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.706+0000 7f2d6ad9d640 1 -- 172.21.15.76:0/704777745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d64009900 con 0x7f2d6c10e2a0 2024-03-12T06:04:49.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.707+0000 7f2d6ad9d640 1 --2- 172.21.15.76:0/704777745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d6c10e2a0 0x7f2d6c1a5900 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f2d640348f0 tx=0x7f2d64034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:49.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.707+0000 7f2d5b7fe640 1 -- 172.21.15.76:0/704777745 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d640328c0 con 0x7f2d6c10e2a0 2024-03-12T06:04:49.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.707+0000 7f2d70d3e640 1 -- 172.21.15.76:0/704777745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d6c1a6480 con 0x7f2d6c10e2a0 2024-03-12T06:04:49.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.707+0000 7f2d5b7fe640 1 -- 172.21.15.76:0/704777745 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d64003e50 con 0x7f2d6c10e2a0 2024-03-12T06:04:49.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.707+0000 7f2d5b7fe640 1 -- 172.21.15.76:0/704777745 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64040450 con 0x7f2d6c10e2a0 2024-03-12T06:04:49.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.707+0000 7f2d70d3e640 1 -- 172.21.15.76:0/704777745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d6c1a6980 con 0x7f2d6c10e2a0 2024-03-12T06:04:49.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.708+0000 7f2d70d3e640 1 -- 172.21.15.76:0/704777745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6c068450 con 0x7f2d6c10e2a0 2024-03-12T06:04:49.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.709+0000 7f2d5b7fe640 1 -- 172.21.15.76:0/704777745 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f2d6400a040 con 0x7f2d6c10e2a0 2024-03-12T06:04:49.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.710+0000 7f2d5b7fe640 1 --2- 172.21.15.76:0/704777745 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f2d540a1bd0 0x7f2d540a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:49.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.710+0000 7f2d6a59c640 1 --2- 172.21.15.76:0/704777745 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f2d540a1bd0 0x7f2d540a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:49.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.710+0000 7f2d5b7fe640 1 -- 172.21.15.76:0/704777745 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f2d640f09c0 con 0x7f2d6c10e2a0 2024-03-12T06:04:49.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.710+0000 7f2d6a59c640 1 --2- 172.21.15.76:0/704777745 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f2d540a1bd0 0x7f2d540a4090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c005fd0 tx=0x7f2d5c005850 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:49.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.713+0000 7f2d5b7fe640 1 -- 172.21.15.76:0/704777745 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2d640b99a0 con 0x7f2d6c10e2a0 2024-03-12T06:04:49.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:49 smithi088 ceph-mon[36585]: pgmap v280: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:04:49.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.920+0000 7f2d70d3e640 1 -- 172.21.15.76:0/704777745 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d6c108600 con 0x7f2d6c10e2a0 2024-03-12T06:04:49.925 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.921+0000 7f2d5b7fe640 1 -- 172.21.15.76:0/704777745 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11328 (secure 0 0 0) 0x7f2d640bd060 con 0x7f2d6c10e2a0 2024-03-12T06:04:49.926 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:04:49.926 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T06:01:33.842001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T06:04:49.926 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 28 2024-03-12T06:04:49.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.923+0000 7f2d70d3e640 1 -- 172.21.15.76:0/704777745 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f2d540a1bd0 msgr2=0x7f2d540a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:49.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.923+0000 7f2d70d3e640 1 --2- 172.21.15.76:0/704777745 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f2d540a1bd0 0x7f2d540a4090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c005fd0 tx=0x7f2d5c005850 comp rx=0 tx=0).stop 2024-03-12T06:04:49.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.923+0000 7f2d70d3e640 1 -- 172.21.15.76:0/704777745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d6c10e2a0 msgr2=0x7f2d6c1a5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:49.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.923+0000 7f2d70d3e640 1 --2- 172.21.15.76:0/704777745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d6c10e2a0 0x7f2d6c1a5900 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f2d640348f0 tx=0x7f2d64034cc0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.923+0000 7f2d70d3e640 1 -- 172.21.15.76:0/704777745 shutdown_connections 2024-03-12T06:04:49.928 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.923+0000 7f2d70d3e640 1 --2- 172.21.15.76:0/704777745 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f2d540a1bd0 0x7f2d540a4090 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.923+0000 7f2d70d3e640 1 --2- 172.21.15.76:0/704777745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d6c10e2a0 0x7f2d6c1a5900 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.923+0000 7f2d70d3e640 1 --2- 172.21.15.76:0/704777745 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2d6c105370 0x7f2d6c10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.923+0000 7f2d70d3e640 1 --2- 172.21.15.76:0/704777745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d6c1002a0 0x7f2d6c10d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:49.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.923+0000 7f2d70d3e640 1 -- 172.21.15.76:0/704777745 >> 172.21.15.76:0/704777745 conn(0x7f2d6c077200 msgr2=0x7f2d6c0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:49.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.923+0000 7f2d70d3e640 1 -- 172.21.15.76:0/704777745 shutdown_connections 2024-03-12T06:04:49.929 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:49.923+0000 7f2d70d3e640 1 -- 172.21.15.76:0/704777745 wait complete. 2024-03-12T06:04:49.939 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:04:49.939 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T06:04:50.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.038+0000 7f24fbe76640 1 -- 172.21.15.76:0/350376297 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24f4102c00 msgr2=0x7f24f410b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.038+0000 7f24fbe76640 1 --2- 172.21.15.76:0/350376297 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24f4102c00 0x7f24f410b0e0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009900 tx=0x7f24e402bcd0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.038+0000 7f24fbe76640 1 -- 172.21.15.76:0/350376297 shutdown_connections 2024-03-12T06:04:50.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.038+0000 7f24fbe76640 1 --2- 172.21.15.76:0/350376297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24f410b700 0x7f24f410dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.038+0000 7f24fbe76640 1 --2- 172.21.15.76:0/350376297 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24f4102c00 0x7f24f410b0e0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.038+0000 7f24fbe76640 1 --2- 172.21.15.76:0/350376297 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24f41022e0 0x7f24f41026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.038+0000 7f24fbe76640 1 -- 172.21.15.76:0/350376297 >> 172.21.15.76:0/350376297 conn(0x7f24f40fbae0 msgr2=0x7f24f40fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.039+0000 7f24fbe76640 1 -- 172.21.15.76:0/350376297 shutdown_connections 2024-03-12T06:04:50.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.039+0000 7f24fbe76640 1 -- 172.21.15.76:0/350376297 wait complete. 2024-03-12T06:04:50.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.039+0000 7f24fbe76640 1 Processor -- start 2024-03-12T06:04:50.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.039+0000 7f24fbe76640 1 -- start start 2024-03-12T06:04:50.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.039+0000 7f24fbe76640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24f41022e0 0x7f24f419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.039+0000 7f24fbe76640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24f4102c00 0x7f24f419d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.039+0000 7f24fbe76640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24f410b700 0x7f24f4197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.039+0000 7f24fbe76640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f4197cb0 con 0x7f24f4102c00 2024-03-12T06:04:50.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.039+0000 7f24fbe76640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f4197e20 con 0x7f24f41022e0 2024-03-12T06:04:50.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.039+0000 7f24fbe76640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f4197fe0 con 0x7f24f410b700 2024-03-12T06:04:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.039+0000 7f24f9411640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24f4102c00 0x7f24f419d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.039+0000 7f24f9411640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24f4102c00 0x7f24f419d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52892/0 (socket says 172.21.15.76:52892) 2024-03-12T06:04:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.039+0000 7f24f9411640 1 -- 172.21.15.76:0/4040322189 learned_addr learned my addr 172.21.15.76:0/4040322189 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.040+0000 7f24fa413640 1 --2- 172.21.15.76:0/4040322189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24f410b700 0x7f24f4197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.040+0000 7f24f9c12640 1 --2- 172.21.15.76:0/4040322189 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24f41022e0 0x7f24f419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.040+0000 7f24f9411640 1 -- 172.21.15.76:0/4040322189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24f410b700 msgr2=0x7f24f4197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.040+0000 7f24f9411640 1 --2- 172.21.15.76:0/4040322189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24f410b700 0x7f24f4197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.040+0000 7f24f9411640 1 -- 172.21.15.76:0/4040322189 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24f41022e0 msgr2=0x7f24f419d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.040+0000 7f24f9411640 1 --2- 172.21.15.76:0/4040322189 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24f41022e0 0x7f24f419d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.040+0000 7f24f9411640 1 -- 172.21.15.76:0/4040322189 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e4009560 con 0x7f24f4102c00 2024-03-12T06:04:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.040+0000 7f24f9411640 1 --2- 172.21.15.76:0/4040322189 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24f4102c00 0x7f24f419d9a0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f24e4005940 tx=0x7f24e402c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.040+0000 7f24eaffd640 1 -- 172.21.15.76:0/4040322189 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e40043d0 con 0x7f24f4102c00 2024-03-12T06:04:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.040+0000 7f24fbe76640 1 -- 172.21.15.76:0/4040322189 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f4198260 con 0x7f24f4102c00 2024-03-12T06:04:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.040+0000 7f24eaffd640 1 -- 172.21.15.76:0/4040322189 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24e40052d0 con 0x7f24f4102c00 2024-03-12T06:04:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.040+0000 7f24eaffd640 1 -- 172.21.15.76:0/4040322189 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e4003d30 con 0x7f24f4102c00 2024-03-12T06:04:50.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.040+0000 7f24fbe76640 1 -- 172.21.15.76:0/4040322189 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f41a4500 con 0x7f24f4102c00 2024-03-12T06:04:50.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.042+0000 7f24eaffd640 1 -- 172.21.15.76:0/4040322189 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f24e4005440 con 0x7f24f4102c00 2024-03-12T06:04:50.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24fea46740 1 Processor -- start 2024-03-12T06:04:50.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24fea46740 1 -- start start 2024-03-12T06:04:50.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24fea46740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498250 0x559cfd498630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24fea46740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd498b70 0x559cfd4a1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24eaffd640 1 --2- 172.21.15.76:0/4040322189 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f24e00a1ba0 0x7f24e00a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24fea46740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd4a1750 0x559cfd4a3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24fea46740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cfd453960 con 0x559cfd498b70 2024-03-12T06:04:50.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24fea46740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cfd407310 con 0x559cfd498250 2024-03-12T06:04:50.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24fea46740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cfd404840 con 0x559cfd4a1750 2024-03-12T06:04:50.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24f9c12640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd498b70 0x559cfd4a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24fa413640 1 --2- 172.21.15.76:0/4040322189 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f24e00a1ba0 0x7f24e00a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24f9c12640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd498b70 0x559cfd4a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52896/0 (socket says 172.21.15.76:52896) 2024-03-12T06:04:50.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24f9c12640 1 -- 172.21.15.76:0/3188305815 learned_addr learned my addr 172.21.15.76:0/3188305815 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24eaffd640 1 -- 172.21.15.76:0/4040322189 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f24e40effa0 con 0x7f24f4102c00 2024-03-12T06:04:50.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24f9411640 1 --2- 172.21.15.76:0/3188305815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd4a1750 0x559cfd4a3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.043+0000 7f24fa413640 1 --2- 172.21.15.76:0/3188305815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498250 0x559cfd498630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.044+0000 7f24f9c12640 1 -- 172.21.15.76:0/3188305815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd4a1750 msgr2=0x559cfd4a3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.044+0000 7f24f9c12640 1 --2- 172.21.15.76:0/3188305815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd4a1750 0x559cfd4a3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.044+0000 7f24f9c12640 1 -- 172.21.15.76:0/3188305815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498250 msgr2=0x559cfd498630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.044+0000 7f24f9c12640 1 --2- 172.21.15.76:0/3188305815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498250 0x559cfd498630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.044+0000 7f24f9c12640 1 -- 172.21.15.76:0/3188305815 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559cfd3c2060 con 0x559cfd498b70 2024-03-12T06:04:50.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.044+0000 7f24fa413640 1 --2- 172.21.15.76:0/4040322189 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f24e00a1ba0 0x7f24e00a4060 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f24ec005c40 tx=0x7f24ec0129b0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.044+0000 7f24f9c12640 1 --2- 172.21.15.76:0/3188305815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd498b70 0x559cfd4a1210 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f24f000cbf0 tx=0x7f24f000e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eade2ca69515deae server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.044+0000 7f24dffff640 1 -- 172.21.15.76:0/3188305815 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f0010430 con 0x559cfd498b70 2024-03-12T06:04:50.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.044+0000 7f24dffff640 1 -- 172.21.15.76:0/3188305815 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24f0010a70 con 0x559cfd498b70 2024-03-12T06:04:50.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.044+0000 7f24dffff640 1 -- 172.21.15.76:0/3188305815 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f000fb30 con 0x559cfd498b70 2024-03-12T06:04:50.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.045+0000 7f24fea46740 1 -- 172.21.15.76:0/3188305815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd498b70 msgr2=0x559cfd4a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.045+0000 7f24fea46740 1 --2- 172.21.15.76:0/3188305815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd498b70 0x559cfd4a1210 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f24f000cbf0 tx=0x7f24f000e350 comp rx=0 tx=0).stop 2024-03-12T06:04:50.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.045+0000 7f24fea46740 1 -- 172.21.15.76:0/3188305815 shutdown_connections 2024-03-12T06:04:50.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.045+0000 7f24fea46740 1 --2- 172.21.15.76:0/3188305815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd4a1750 0x559cfd4a3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.045+0000 7f24fea46740 1 --2- 172.21.15.76:0/3188305815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd498b70 0x559cfd4a1210 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.045+0000 7f24fea46740 1 --2- 172.21.15.76:0/3188305815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498250 0x559cfd498630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.045+0000 7f24fea46740 1 -- 172.21.15.76:0/3188305815 >> 172.21.15.76:0/3188305815 conn(0x559cfd492980 msgr2=0x559cfd492d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.045+0000 7f24fea46740 1 -- 172.21.15.76:0/3188305815 shutdown_connections 2024-03-12T06:04:50.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.045+0000 7f24fea46740 1 -- 172.21.15.76:0/3188305815 wait complete. 2024-03-12T06:04:50.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.045+0000 7f24fea46740 1 -- start start 2024-03-12T06:04:50.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.045+0000 7f24fea46740 1 Processor -- start 2024-03-12T06:04:50.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fea46740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd498250 0x559cfd49e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fea46740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 0x559cfd49ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fea46740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd4a1750 0x559cfd49f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fea46740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cfd400020 con 0x559cfd4a1750 2024-03-12T06:04:50.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fea46740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cfd0d3430 con 0x559cfd498b70 2024-03-12T06:04:50.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fea46740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cfd08fe00 con 0x559cfd498250 2024-03-12T06:04:50.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fea46740 10 client.0 _ll_register_callbacks cb 0x559cfd3a7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:50.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fa413640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd4a1750 0x559cfd49f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.061 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fa413640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd4a1750 0x559cfd49f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52900/0 (socket says 172.21.15.76:52900) 2024-03-12T06:04:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fa413640 1 -- 172.21.15.76:0/2813495107 learned_addr learned my addr 172.21.15.76:0/2813495107 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24f9c12640 1 --2- 172.21.15.76:0/2813495107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd498250 0x559cfd49e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24f9411640 1 --2- 172.21.15.76:0/2813495107 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 0x559cfd49ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fa413640 1 -- 172.21.15.76:0/2813495107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd498250 msgr2=0x559cfd49e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fa413640 1 --2- 172.21.15.76:0/2813495107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd498250 0x559cfd49e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fa413640 1 -- 172.21.15.76:0/2813495107 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 msgr2=0x559cfd49ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fa413640 1 --2- 172.21.15.76:0/2813495107 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 0x559cfd49ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fa413640 1 -- 172.21.15.76:0/2813495107 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f000c850 con 0x559cfd4a1750 2024-03-12T06:04:50.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24f9411640 1 --2- 172.21.15.76:0/2813495107 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 0x559cfd49ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:50.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fa413640 1 --2- 172.21.15.76:0/2813495107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd4a1750 0x559cfd49f200 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f24ec010a10 tx=0x7f24ec01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5341875f99385c1a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:50.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24dd7fa640 1 -- 172.21.15.76:0/2813495107 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec017040 con 0x559cfd4a1750 2024-03-12T06:04:50.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24dd7fa640 1 -- 172.21.15.76:0/2813495107 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f24ec01cd30 con 0x559cfd4a1750 2024-03-12T06:04:50.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.046+0000 7f24fea46740 1 -- 172.21.15.76:0/2813495107 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559cfd0ff7e0 con 0x559cfd4a1750 2024-03-12T06:04:50.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.047+0000 7f24dd7fa640 1 -- 172.21.15.76:0/2813495107 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24ec021460 con 0x559cfd4a1750 2024-03-12T06:04:50.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.047+0000 7f24fea46740 10 client.17496 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:50.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.047+0000 7f24fea46740 1 -- 172.21.15.76:0/2813495107 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559cfd3fe6a0 con 0x559cfd4a1750 2024-03-12T06:04:50.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.047+0000 7f24dd7fa640 1 -- 172.21.15.76:0/2813495107 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559cfd0ff7e0 con 0x559cfd4a1750 2024-03-12T06:04:50.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.047+0000 7f24dd7fa640 1 -- 172.21.15.76:0/2813495107 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f24ec01f710 con 0x559cfd4a1750 2024-03-12T06:04:50.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.047+0000 7f24fea46740 10 client.17496 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:50.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.047+0000 7f24fea46740 10 client.17496 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:04:50.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.047+0000 7f24fea46740 20 client.17496 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:50.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.047+0000 7f24fea46740 1 --2- 172.21.15.76:0/2813495107 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559cfd4b1c40 0x559cfd4b4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.047+0000 7f24fea46740 4 client.17496 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:04:50.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.047+0000 7f24fea46740 1 -- 172.21.15.76:0/2813495107 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559cfd13c720 con 0x559cfd4b1c40 2024-03-12T06:04:50.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.047+0000 7f24f9c12640 1 --2- 172.21.15.76:0/2813495107 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559cfd4b1c40 0x559cfd4b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.047+0000 7f24dd7fa640 10 client.17496 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:04:50.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.047+0000 7f24f9c12640 1 --2- 172.21.15.76:0/2813495107 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559cfd4b1c40 0x559cfd4b4060 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7f128bfdb9013c03 server_cookie=27be432bd95f5923 in_seq=0 out_seq=0 2024-03-12T06:04:50.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.048+0000 7f24dd7fa640 1 -- 172.21.15.76:0/2813495107 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559cfd13c720 con 0x559cfd4b1c40 2024-03-12T06:04:50.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.048+0000 7f24dd7fa640 10 client.17496 handle_command_reply: tid=0 2024-03-12T06:04:50.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.048+0000 7f24fea46740 1 client.17496 shutdown 2024-03-12T06:04:50.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.048+0000 7f24fea46740 1 -- 172.21.15.76:0/2813495107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd4a1750 msgr2=0x559cfd49f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.048+0000 7f24fea46740 1 --2- 172.21.15.76:0/2813495107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd4a1750 0x559cfd49f200 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f24ec010a10 tx=0x7f24ec01a000 comp rx=0 tx=0).stop 2024-03-12T06:04:50.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.048+0000 7f24fea46740 1 -- 172.21.15.76:0/2813495107 shutdown_connections 2024-03-12T06:04:50.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.048+0000 7f24fea46740 1 --2- 172.21.15.76:0/2813495107 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559cfd4b1c40 0x559cfd4b4060 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.048+0000 7f24fea46740 1 --2- 172.21.15.76:0/2813495107 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd4a1750 0x559cfd49f200 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.048+0000 7f24fea46740 1 --2- 172.21.15.76:0/2813495107 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 0x559cfd49ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.048+0000 7f24fea46740 1 --2- 172.21.15.76:0/2813495107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd498250 0x559cfd49e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.048+0000 7f24dd7fa640 0 client.17496 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:04:50.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.048+0000 7f24fea46740 1 -- 172.21.15.76:0/2813495107 >> 172.21.15.76:0/2813495107 conn(0x559cfd492980 msgr2=0x559cfd4946d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.048+0000 7f24fea46740 1 -- 172.21.15.76:0/2813495107 shutdown_connections 2024-03-12T06:04:50.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.048+0000 7f24fea46740 1 -- 172.21.15.76:0/2813495107 wait complete. 2024-03-12T06:04:50.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.049+0000 7f24fea46740 20 client.17496 trim_cache size 0 max 16384 2024-03-12T06:04:50.073 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24fea46740 1 Processor -- start 2024-03-12T06:04:50.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24fea46740 1 -- start start 2024-03-12T06:04:50.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24fea46740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd498250 0x559cfd4a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24fea46740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 0x559cfd49f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24fea46740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd4a1750 0x559cfd49b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24fea46740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cfd385130 con 0x559cfd498250 2024-03-12T06:04:50.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24fea46740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cfd48e080 con 0x559cfd498b70 2024-03-12T06:04:50.074 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24fea46740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cfd49ccd0 con 0x559cfd4a1750 2024-03-12T06:04:50.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24f9c12640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd498250 0x559cfd4a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24f9c12640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd498250 0x559cfd4a0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52916/0 (socket says 172.21.15.76:52916) 2024-03-12T06:04:50.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24f9c12640 1 -- 172.21.15.76:0/3516583688 learned_addr learned my addr 172.21.15.76:0/3516583688 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24f9c12640 1 -- 172.21.15.76:0/3516583688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd4a1750 msgr2=0x559cfd49b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24fa413640 1 --2- 172.21.15.76:0/3516583688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd4a1750 0x559cfd49b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24f9c12640 1 --2- 172.21.15.76:0/3516583688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd4a1750 0x559cfd49b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24f9c12640 1 -- 172.21.15.76:0/3516583688 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 msgr2=0x559cfd49f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24f9411640 1 --2- 172.21.15.76:0/3516583688 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 0x559cfd49f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.075 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24f9c12640 1 --2- 172.21.15.76:0/3516583688 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 0x559cfd49f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24f9c12640 1 -- 172.21.15.76:0/3516583688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559cfd39ac20 con 0x559cfd498250 2024-03-12T06:04:50.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.060+0000 7f24fa413640 1 --2- 172.21.15.76:0/3516583688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd4a1750 0x559cfd49b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:50.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.061+0000 7f24f9c12640 1 --2- 172.21.15.76:0/3516583688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd498250 0x559cfd4a0eb0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f24f000cb40 tx=0x7f24f000c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390944283a8f2b03 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.061+0000 7f24df7fe640 1 -- 172.21.15.76:0/3516583688 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f0020050 con 0x559cfd498250 2024-03-12T06:04:50.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.061+0000 7f24df7fe640 1 -- 172.21.15.76:0/3516583688 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24f0021040 con 0x559cfd498250 2024-03-12T06:04:50.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.061+0000 7f24df7fe640 1 -- 172.21.15.76:0/3516583688 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f000f780 con 0x559cfd498250 2024-03-12T06:04:50.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.061+0000 7f24fea46740 1 -- 172.21.15.76:0/3516583688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd498250 msgr2=0x559cfd4a0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.061+0000 7f24fea46740 1 --2- 172.21.15.76:0/3516583688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd498250 0x559cfd4a0eb0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f24f000cb40 tx=0x7f24f000c4f0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.061+0000 7f24fea46740 1 -- 172.21.15.76:0/3516583688 shutdown_connections 2024-03-12T06:04:50.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.061+0000 7f24fea46740 1 --2- 172.21.15.76:0/3516583688 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd4a1750 0x559cfd49b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.061+0000 7f24fea46740 1 --2- 172.21.15.76:0/3516583688 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 0x559cfd49f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.061+0000 7f24fea46740 1 --2- 172.21.15.76:0/3516583688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd498250 0x559cfd4a0eb0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.061+0000 7f24fea46740 1 -- 172.21.15.76:0/3516583688 >> 172.21.15.76:0/3516583688 conn(0x559cfd492980 msgr2=0x559cfd4b89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.061+0000 7f24fea46740 1 -- 172.21.15.76:0/3516583688 shutdown_connections 2024-03-12T06:04:50.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.061+0000 7f24fea46740 1 -- 172.21.15.76:0/3516583688 wait complete. 2024-03-12T06:04:50.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fea46740 1 -- start start 2024-03-12T06:04:50.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fea46740 1 Processor -- start 2024-03-12T06:04:50.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fea46740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd498250 0x559cfd5014e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fea46740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 0x559cfd501a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fea46740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd4a1750 0x559cfd4a7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fea46740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cfd4a7f00 con 0x559cfd4a1750 2024-03-12T06:04:50.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fea46740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cfd4a8070 con 0x559cfd498b70 2024-03-12T06:04:50.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fea46740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cfd4a8230 con 0x559cfd498250 2024-03-12T06:04:50.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fea46740 10 client.0 _ll_register_callbacks cb 0x7f24f000c390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:50.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fa413640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd4a1750 0x559cfd4a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fa413640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd4a1750 0x559cfd4a7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52926/0 (socket says 172.21.15.76:52926) 2024-03-12T06:04:50.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fa413640 1 -- 172.21.15.76:0/551442116 learned_addr learned my addr 172.21.15.76:0/551442116 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24f9c12640 1 --2- 172.21.15.76:0/551442116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd498250 0x559cfd5014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24f9411640 1 --2- 172.21.15.76:0/551442116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 0x559cfd501a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fa413640 1 -- 172.21.15.76:0/551442116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd498250 msgr2=0x559cfd5014e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fa413640 1 --2- 172.21.15.76:0/551442116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd498250 0x559cfd5014e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fa413640 1 -- 172.21.15.76:0/551442116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 msgr2=0x559cfd501a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fa413640 1 --2- 172.21.15.76:0/551442116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 0x559cfd501a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24fa413640 1 -- 172.21.15.76:0/551442116 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559cfd3c2060 con 0x559cfd4a1750 2024-03-12T06:04:50.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.062+0000 7f24f9411640 1 --2- 172.21.15.76:0/551442116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 0x559cfd501a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:50.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24fa413640 1 --2- 172.21.15.76:0/551442116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd4a1750 0x559cfd4a7830 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f24ec012ed0 tx=0x7f24ec012460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80fe9625106ea58e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:50.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24dd7fa640 1 -- 172.21.15.76:0/551442116 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec017070 con 0x559cfd4a1750 2024-03-12T06:04:50.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24dd7fa640 1 -- 172.21.15.76:0/551442116 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f24ec0105f0 con 0x559cfd4a1750 2024-03-12T06:04:50.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24fea46740 1 -- 172.21.15.76:0/551442116 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f24f0010a70 con 0x559cfd4a1750 2024-03-12T06:04:50.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24dd7fa640 1 -- 172.21.15.76:0/551442116 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24ec01c770 con 0x559cfd4a1750 2024-03-12T06:04:50.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24fea46740 10 client.17502 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:50.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24fea46740 1 -- 172.21.15.76:0/551442116 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559cfd3c08c0 con 0x559cfd4a1750 2024-03-12T06:04:50.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24dd7fa640 1 -- 172.21.15.76:0/551442116 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24f0010a70 con 0x559cfd4a1750 2024-03-12T06:04:50.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24dd7fa640 1 -- 172.21.15.76:0/551442116 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f24ec0051d0 con 0x559cfd4a1750 2024-03-12T06:04:50.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24fea46740 10 client.17502 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:50.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24fea46740 10 client.17502 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:04:50.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24fea46740 20 client.17502 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:50.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24fea46740 1 --2- 172.21.15.76:0/551442116 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559cfd50a2e0 0x559cfd50c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24fea46740 4 client.17502 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:04:50.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24fea46740 1 -- 172.21.15.76:0/551442116 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559cfd18b890 con 0x559cfd50a2e0 2024-03-12T06:04:50.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.063+0000 7f24f9c12640 1 --2- 172.21.15.76:0/551442116 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559cfd50a2e0 0x559cfd50c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24dd7fa640 10 client.17502 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:04:50.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24f9c12640 1 --2- 172.21.15.76:0/551442116 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559cfd50a2e0 0x559cfd50c700 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=22beac401d433170 server_cookie=7353ef9549de8042 in_seq=0 out_seq=0 2024-03-12T06:04:50.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24dd7fa640 1 -- 172.21.15.76:0/551442116 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559cfd18b890 con 0x559cfd50a2e0 2024-03-12T06:04:50.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24dd7fa640 10 client.17502 handle_command_reply: tid=0 2024-03-12T06:04:50.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24fea46740 1 client.17502 shutdown 2024-03-12T06:04:50.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24fea46740 1 -- 172.21.15.76:0/551442116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd4a1750 msgr2=0x559cfd4a7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24fea46740 1 --2- 172.21.15.76:0/551442116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd4a1750 0x559cfd4a7830 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f24ec012ed0 tx=0x7f24ec012460 comp rx=0 tx=0).stop 2024-03-12T06:04:50.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24fea46740 1 -- 172.21.15.76:0/551442116 shutdown_connections 2024-03-12T06:04:50.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24fea46740 1 --2- 172.21.15.76:0/551442116 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559cfd50a2e0 0x559cfd50c700 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24fea46740 1 --2- 172.21.15.76:0/551442116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x559cfd4a1750 0x559cfd4a7830 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24fea46740 1 --2- 172.21.15.76:0/551442116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x559cfd498b70 0x559cfd501a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24fea46740 1 --2- 172.21.15.76:0/551442116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cfd498250 0x559cfd5014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24dd7fa640 0 client.17502 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:04:50.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24fea46740 1 -- 172.21.15.76:0/551442116 >> 172.21.15.76:0/551442116 conn(0x559cfd492980 msgr2=0x559cfd4912f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24fea46740 1 -- 172.21.15.76:0/551442116 shutdown_connections 2024-03-12T06:04:50.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24fea46740 1 -- 172.21.15.76:0/551442116 wait complete. 2024-03-12T06:04:50.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.064+0000 7f24fea46740 20 client.17502 trim_cache size 0 max 16384 2024-03-12T06:04:50.085 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:04:50.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.065+0000 7f24fbe76640 1 -- 172.21.15.76:0/4040322189 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f24e00a1ba0 msgr2=0x7f24e00a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.065+0000 7f24fbe76640 1 --2- 172.21.15.76:0/4040322189 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f24e00a1ba0 0x7f24e00a4060 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f24ec005c40 tx=0x7f24ec0129b0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.065+0000 7f24fbe76640 1 -- 172.21.15.76:0/4040322189 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24f4102c00 msgr2=0x7f24f419d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.065+0000 7f24fbe76640 1 --2- 172.21.15.76:0/4040322189 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24f4102c00 0x7f24f419d9a0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f24e4005940 tx=0x7f24e402c780 comp rx=0 tx=0).stop 2024-03-12T06:04:50.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.065+0000 7f24fbe76640 1 -- 172.21.15.76:0/4040322189 shutdown_connections 2024-03-12T06:04:50.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.065+0000 7f24fbe76640 1 --2- 172.21.15.76:0/4040322189 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f24e00a1ba0 0x7f24e00a4060 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.065+0000 7f24fbe76640 1 --2- 172.21.15.76:0/4040322189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24f410b700 0x7f24f4197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.065+0000 7f24fbe76640 1 --2- 172.21.15.76:0/4040322189 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24f4102c00 0x7f24f419d9a0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.065+0000 7f24fbe76640 1 --2- 172.21.15.76:0/4040322189 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f24f41022e0 0x7f24f419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.065+0000 7f24fbe76640 1 -- 172.21.15.76:0/4040322189 >> 172.21.15.76:0/4040322189 conn(0x7f24f40fbae0 msgr2=0x7f24f40fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.065+0000 7f24fbe76640 1 -- 172.21.15.76:0/4040322189 shutdown_connections 2024-03-12T06:04:50.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.065+0000 7f24fbe76640 1 -- 172.21.15.76:0/4040322189 wait complete. 2024-03-12T06:04:50.088 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:04:50.089 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T06:04:50.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.188+0000 7fbc78281640 1 -- 172.21.15.76:0/1922856562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc70105360 msgr2=0x7fbc7010cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.188+0000 7fbc78281640 1 --2- 172.21.15.76:0/1922856562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc70105360 0x7fbc7010cf30 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fbc68009cd0 tx=0x7fbc6802c0f0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.188+0000 7fbc78281640 1 -- 172.21.15.76:0/1922856562 shutdown_connections 2024-03-12T06:04:50.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.188+0000 7fbc78281640 1 --2- 172.21.15.76:0/1922856562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc70105360 0x7fbc7010cf30 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.188+0000 7fbc78281640 1 --2- 172.21.15.76:0/1922856562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc70100c40 0x7fbc70104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.188+0000 7fbc78281640 1 --2- 172.21.15.76:0/1922856562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc70100290 0x7fbc70100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.188+0000 7fbc78281640 1 -- 172.21.15.76:0/1922856562 >> 172.21.15.76:0/1922856562 conn(0x7fbc70077230 msgr2=0x7fbc70077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.188+0000 7fbc78281640 1 -- 172.21.15.76:0/1922856562 shutdown_connections 2024-03-12T06:04:50.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.188+0000 7fbc78281640 1 -- 172.21.15.76:0/1922856562 wait complete. 2024-03-12T06:04:50.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.188+0000 7fbc78281640 1 Processor -- start 2024-03-12T06:04:50.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc78281640 1 -- start start 2024-03-12T06:04:50.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc78281640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc70100290 0x7fbc70196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc78281640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc70105360 0x7fbc70196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc78281640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc7019aea0 0x7fbc7019b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc78281640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7019b890 con 0x7fbc7019aea0 2024-03-12T06:04:50.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc78281640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7019ba00 con 0x7fbc70100290 2024-03-12T06:04:50.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc78281640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7019bb70 con 0x7fbc70105360 2024-03-12T06:04:50.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc7681e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc7019aea0 0x7fbc7019b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc7681e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc7019aea0 0x7fbc7019b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52946/0 (socket says 172.21.15.76:52946) 2024-03-12T06:04:50.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc7681e640 1 -- 172.21.15.76:0/3716787513 learned_addr learned my addr 172.21.15.76:0/3716787513 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc7581c640 1 --2- 172.21.15.76:0/3716787513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc70105360 0x7fbc70196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc7601d640 1 --2- 172.21.15.76:0/3716787513 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc70100290 0x7fbc70196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc7681e640 1 -- 172.21.15.76:0/3716787513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc70105360 msgr2=0x7fbc70196b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc7681e640 1 --2- 172.21.15.76:0/3716787513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc70105360 0x7fbc70196b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc7681e640 1 -- 172.21.15.76:0/3716787513 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc70100290 msgr2=0x7fbc70196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc7681e640 1 --2- 172.21.15.76:0/3716787513 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc70100290 0x7fbc70196630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc7681e640 1 -- 172.21.15.76:0/3716787513 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc68009900 con 0x7fbc7019aea0 2024-03-12T06:04:50.198 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc7681e640 1 --2- 172.21.15.76:0/3716787513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc7019aea0 0x7fbc7019b350 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fbc68037040 tx=0x7fbc68032fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.189+0000 7fbc677fe640 1 -- 172.21.15.76:0/3716787513 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68032ac0 con 0x7fbc7019aea0 2024-03-12T06:04:50.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.190+0000 7fbc677fe640 1 -- 172.21.15.76:0/3716787513 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc68003e50 con 0x7fbc7019aea0 2024-03-12T06:04:50.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.190+0000 7fbc78281640 1 -- 172.21.15.76:0/3716787513 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc701a3150 con 0x7fbc7019aea0 2024-03-12T06:04:50.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.190+0000 7fbc677fe640 1 -- 172.21.15.76:0/3716787513 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68040470 con 0x7fbc7019aea0 2024-03-12T06:04:50.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.190+0000 7fbc78281640 1 -- 172.21.15.76:0/3716787513 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc701a3570 con 0x7fbc7019aea0 2024-03-12T06:04:50.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.192+0000 7fbc677fe640 1 -- 172.21.15.76:0/3716787513 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fbc6800a040 con 0x7fbc7019aea0 2024-03-12T06:04:50.206 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.192+0000 7fbc7ae51740 1 Processor -- start 2024-03-12T06:04:50.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.192+0000 7fbc7ae51740 1 -- start start 2024-03-12T06:04:50.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.192+0000 7fbc677fe640 1 --2- 172.21.15.76:0/3716787513 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbc5c0a1ba0 0x7fbc5c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7ae51740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e446f00 0x56186e4472e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7ae51740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e452420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7ae51740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e452960 0x56186e454d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7ae51740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186e402960 con 0x56186e447820 2024-03-12T06:04:50.207 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7ae51740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186e3b6310 con 0x56186e452960 2024-03-12T06:04:50.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7ae51740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186e3b3840 con 0x56186e446f00 2024-03-12T06:04:50.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7581c640 1 --2- 172.21.15.76:0/3716787513 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbc5c0a1ba0 0x7fbc5c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc677fe640 1 -- 172.21.15.76:0/3716787513 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fbc680f0200 con 0x7fbc7019aea0 2024-03-12T06:04:50.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7581c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e452420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7581c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e452420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52950/0 (socket says 172.21.15.76:52950) 2024-03-12T06:04:50.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7581c640 1 -- 172.21.15.76:0/3131648255 learned_addr learned my addr 172.21.15.76:0/3131648255 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7601d640 1 --2- 172.21.15.76:0/3131648255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e446f00 0x56186e4472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7581c640 1 -- 172.21.15.76:0/3131648255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e446f00 msgr2=0x56186e4472e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.208 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7581c640 1 --2- 172.21.15.76:0/3131648255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e446f00 0x56186e4472e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7681e640 1 --2- 172.21.15.76:0/3131648255 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e452960 0x56186e454d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7581c640 1 -- 172.21.15.76:0/3131648255 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e452960 msgr2=0x56186e454d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7581c640 1 --2- 172.21.15.76:0/3131648255 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e452960 0x56186e454d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7581c640 1 -- 172.21.15.76:0/3131648255 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56186e371060 con 0x56186e447820 2024-03-12T06:04:50.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7601d640 1 --2- 172.21.15.76:0/3131648255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e446f00 0x56186e4472e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:50.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7581c640 1 --2- 172.21.15.76:0/3716787513 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbc5c0a1ba0 0x7fbc5c0a4060 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c005900 tx=0x7fbc6c00aad0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc7581c640 1 --2- 172.21.15.76:0/3131648255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e452420 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c005bc0 tx=0x7fbc6c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a385016054316f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.209 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc64bfe640 1 -- 172.21.15.76:0/3131648255 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00fe50 con 0x56186e447820 2024-03-12T06:04:50.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc64bfe640 1 -- 172.21.15.76:0/3131648255 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc6c01ee60 con 0x56186e447820 2024-03-12T06:04:50.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.193+0000 7fbc64bfe640 1 -- 172.21.15.76:0/3131648255 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c01fce0 con 0x56186e447820 2024-03-12T06:04:50.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.194+0000 7fbc7ae51740 1 -- 172.21.15.76:0/3131648255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 msgr2=0x56186e452420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.194+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/3131648255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e452420 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c005bc0 tx=0x7fbc6c01b000 comp rx=0 tx=0).stop 2024-03-12T06:04:50.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.194+0000 7fbc7ae51740 1 -- 172.21.15.76:0/3131648255 shutdown_connections 2024-03-12T06:04:50.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.194+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/3131648255 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e452960 0x56186e454d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.194+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/3131648255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e452420 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.210 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.194+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/3131648255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e446f00 0x56186e4472e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.194+0000 7fbc7ae51740 1 -- 172.21.15.76:0/3131648255 >> 172.21.15.76:0/3131648255 conn(0x56186e441980 msgr2=0x56186e441d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.194+0000 7fbc7ae51740 1 -- 172.21.15.76:0/3131648255 shutdown_connections 2024-03-12T06:04:50.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.194+0000 7fbc7ae51740 1 -- 172.21.15.76:0/3131648255 wait complete. 2024-03-12T06:04:50.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.194+0000 7fbc7ae51740 1 -- start start 2024-03-12T06:04:50.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7ae51740 1 Processor -- start 2024-03-12T06:04:50.211 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7ae51740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e446f00 0x56186e44f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7ae51740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e44fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7ae51740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e452960 0x56186e450410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7ae51740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186e3af020 con 0x56186e447820 2024-03-12T06:04:50.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7ae51740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186e082430 con 0x56186e446f00 2024-03-12T06:04:50.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7ae51740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186e03ee00 con 0x56186e452960 2024-03-12T06:04:50.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7ae51740 10 client.0 _ll_register_callbacks cb 0x56186e356cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:50.212 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7581c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e44fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7581c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e44fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52964/0 (socket says 172.21.15.76:52964) 2024-03-12T06:04:50.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7581c640 1 -- 172.21.15.76:0/2977934362 learned_addr learned my addr 172.21.15.76:0/2977934362 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7681e640 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e452960 0x56186e450410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7601d640 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e446f00 0x56186e44f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7581c640 1 -- 172.21.15.76:0/2977934362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e452960 msgr2=0x56186e450410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7581c640 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e452960 0x56186e450410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7581c640 1 -- 172.21.15.76:0/2977934362 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e446f00 msgr2=0x56186e44f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.213 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7581c640 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e446f00 0x56186e44f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7581c640 1 -- 172.21.15.76:0/2977934362 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc6c019040 con 0x56186e447820 2024-03-12T06:04:50.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7681e640 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e452960 0x56186e450410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:50.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7601d640 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e446f00 0x56186e44f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:50.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7581c640 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e44fed0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c01e740 tx=0x7fbc6c00ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8689ff5e8193825d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:50.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc59ffb640 1 -- 172.21.15.76:0/2977934362 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c01ec50 con 0x56186e447820 2024-03-12T06:04:50.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc59ffb640 1 -- 172.21.15.76:0/2977934362 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fbc6c01fa20 con 0x56186e447820 2024-03-12T06:04:50.214 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.195+0000 7fbc7ae51740 1 -- 172.21.15.76:0/2977934362 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56186e0ae7e0 con 0x56186e447820 2024-03-12T06:04:50.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.196+0000 7fbc59ffb640 1 -- 172.21.15.76:0/2977934362 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc6c01fbe0 con 0x56186e447820 2024-03-12T06:04:50.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.196+0000 7fbc7ae51740 10 client.17514 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:50.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.196+0000 7fbc7ae51740 1 -- 172.21.15.76:0/2977934362 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56186e3ad6a0 con 0x56186e447820 2024-03-12T06:04:50.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.196+0000 7fbc59ffb640 1 -- 172.21.15.76:0/2977934362 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc6c03d080 con 0x56186e447820 2024-03-12T06:04:50.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.196+0000 7fbc59ffb640 1 -- 172.21.15.76:0/2977934362 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbc6c037030 con 0x56186e447820 2024-03-12T06:04:50.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.196+0000 7fbc7ae51740 10 client.17514 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:50.215 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.196+0000 7fbc7ae51740 10 client.17514 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:04:50.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.196+0000 7fbc7ae51740 20 client.17514 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:50.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.196+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x56186e462b00 0x56186e464f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.196+0000 7fbc7ae51740 4 client.17514 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:04:50.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.196+0000 7fbc7ae51740 1 -- 172.21.15.76:0/2977934362 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56186e0eb720 con 0x56186e462b00 2024-03-12T06:04:50.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.196+0000 7fbc7601d640 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x56186e462b00 0x56186e464f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.197+0000 7fbc59ffb640 10 client.17514 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:04:50.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.197+0000 7fbc7601d640 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x56186e462b00 0x56186e464f40 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d7f7b4fc2f3af482 server_cookie=e4b0d56598545c8e in_seq=0 out_seq=0 2024-03-12T06:04:50.216 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.197+0000 7fbc59ffb640 1 -- 172.21.15.76:0/2977934362 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56186e0eb720 con 0x56186e462b00 2024-03-12T06:04:50.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.197+0000 7fbc59ffb640 10 client.17514 handle_command_reply: tid=0 2024-03-12T06:04:50.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.197+0000 7fbc7ae51740 1 client.17514 shutdown 2024-03-12T06:04:50.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.197+0000 7fbc7ae51740 1 -- 172.21.15.76:0/2977934362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 msgr2=0x56186e44fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.197+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e44fed0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c01e740 tx=0x7fbc6c00ff90 comp rx=0 tx=0).stop 2024-03-12T06:04:50.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.198+0000 7fbc7ae51740 1 -- 172.21.15.76:0/2977934362 shutdown_connections 2024-03-12T06:04:50.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.198+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x56186e462b00 0x56186e464f40 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.198+0000 7fbc59ffb640 0 client.17514 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:04:50.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.198+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e452960 0x56186e450410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.198+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e44fed0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.198+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/2977934362 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e446f00 0x56186e44f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.198+0000 7fbc7ae51740 1 -- 172.21.15.76:0/2977934362 >> 172.21.15.76:0/2977934362 conn(0x56186e441980 msgr2=0x56186e4436d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.198+0000 7fbc7ae51740 1 -- 172.21.15.76:0/2977934362 shutdown_connections 2024-03-12T06:04:50.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.198+0000 7fbc7ae51740 1 -- 172.21.15.76:0/2977934362 wait complete. 2024-03-12T06:04:50.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.198+0000 7fbc7ae51740 20 client.17514 trim_cache size 0 max 16384 2024-03-12T06:04:50.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.209+0000 7fbc7ae51740 1 Processor -- start 2024-03-12T06:04:50.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.209+0000 7fbc7ae51740 1 -- start start 2024-03-12T06:04:50.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.209+0000 7fbc7ae51740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e446f00 0x56186e4520c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.209+0000 7fbc7ae51740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e450230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.209+0000 7fbc7ae51740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e452960 0x56186e44cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.209+0000 7fbc7ae51740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186e334130 con 0x56186e447820 2024-03-12T06:04:50.224 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.209+0000 7fbc7ae51740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186e46a150 con 0x56186e446f00 2024-03-12T06:04:50.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.209+0000 7fbc7ae51740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186e446570 con 0x56186e452960 2024-03-12T06:04:50.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.210+0000 7fbc7581c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e450230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.210+0000 7fbc7581c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e450230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52966/0 (socket says 172.21.15.76:52966) 2024-03-12T06:04:50.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.210+0000 7fbc7581c640 1 -- 172.21.15.76:0/3213056883 learned_addr learned my addr 172.21.15.76:0/3213056883 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.210+0000 7fbc7601d640 1 --2- 172.21.15.76:0/3213056883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e446f00 0x56186e4520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.210+0000 7fbc7581c640 1 -- 172.21.15.76:0/3213056883 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e452960 msgr2=0x56186e44cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.210+0000 7fbc7581c640 1 --2- 172.21.15.76:0/3213056883 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e452960 0x56186e44cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.225 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.210+0000 7fbc7581c640 1 -- 172.21.15.76:0/3213056883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e446f00 msgr2=0x56186e4520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.210+0000 7fbc7581c640 1 --2- 172.21.15.76:0/3213056883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e446f00 0x56186e4520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.210+0000 7fbc7581c640 1 -- 172.21.15.76:0/3213056883 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56186e349c20 con 0x56186e447820 2024-03-12T06:04:50.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.210+0000 7fbc7601d640 1 --2- 172.21.15.76:0/3213056883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e446f00 0x56186e4520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:50.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.210+0000 7fbc7581c640 1 --2- 172.21.15.76:0/3213056883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e450230 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0058d0 tx=0x7fbc6c00a970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10a5cd37a82d905 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.210+0000 7fbc5bfff640 1 -- 172.21.15.76:0/3213056883 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00aec0 con 0x56186e447820 2024-03-12T06:04:50.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.210+0000 7fbc5bfff640 1 -- 172.21.15.76:0/3213056883 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56186e0ae7e0 con 0x56186e447820 2024-03-12T06:04:50.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.210+0000 7fbc5bfff640 1 -- 172.21.15.76:0/3213056883 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c01f6c0 con 0x56186e447820 2024-03-12T06:04:50.226 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 -- 172.21.15.76:0/3213056883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 msgr2=0x56186e450230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/3213056883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e450230 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0058d0 tx=0x7fbc6c00a970 comp rx=0 tx=0).stop 2024-03-12T06:04:50.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 -- 172.21.15.76:0/3213056883 shutdown_connections 2024-03-12T06:04:50.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/3213056883 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e452960 0x56186e44cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/3213056883 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e447820 0x56186e450230 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/3213056883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e446f00 0x56186e4520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 -- 172.21.15.76:0/3213056883 >> 172.21.15.76:0/3213056883 conn(0x56186e441980 msgr2=0x56186e44aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 -- 172.21.15.76:0/3213056883 shutdown_connections 2024-03-12T06:04:50.227 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 -- 172.21.15.76:0/3213056883 wait complete. 2024-03-12T06:04:50.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 -- start start 2024-03-12T06:04:50.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 Processor -- start 2024-03-12T06:04:50.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e446f00 0x56186e458550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e447820 0x56186e458a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e452960 0x56186e458ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186e4b91d0 con 0x56186e446f00 2024-03-12T06:04:50.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186e4b9340 con 0x56186e452960 2024-03-12T06:04:50.228 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186e4b9500 con 0x56186e447820 2024-03-12T06:04:50.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7ae51740 10 client.0 _ll_register_callbacks cb 0x7fbc6c0114c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:50.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7601d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e446f00 0x56186e458550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7601d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e446f00 0x56186e458550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52974/0 (socket says 172.21.15.76:52974) 2024-03-12T06:04:50.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7601d640 1 -- 172.21.15.76:0/1667633252 learned_addr learned my addr 172.21.15.76:0/1667633252 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7601d640 1 -- 172.21.15.76:0/1667633252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e447820 msgr2=0x56186e458a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.211+0000 7fbc7601d640 1 --2- 172.21.15.76:0/1667633252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e447820 0x56186e458a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc7601d640 1 -- 172.21.15.76:0/1667633252 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e452960 msgr2=0x56186e458ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc7601d640 1 --2- 172.21.15.76:0/1667633252 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e452960 0x56186e458ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc7601d640 1 -- 172.21.15.76:0/1667633252 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56186e3ad6a0 con 0x56186e446f00 2024-03-12T06:04:50.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc7601d640 1 --2- 172.21.15.76:0/1667633252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e446f00 0x56186e458550 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000b010 tx=0x7fbc60019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a87e59db5ea703 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:50.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc59ffb640 1 -- 172.21.15.76:0/1667633252 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc60004020 con 0x56186e446f00 2024-03-12T06:04:50.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc59ffb640 1 -- 172.21.15.76:0/1667633252 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fbc60004180 con 0x56186e446f00 2024-03-12T06:04:50.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc7ae51740 1 -- 172.21.15.76:0/1667633252 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbc6c01ee60 con 0x56186e446f00 2024-03-12T06:04:50.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc59ffb640 1 -- 172.21.15.76:0/1667633252 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc60004380 con 0x56186e446f00 2024-03-12T06:04:50.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc7ae51740 10 client.17520 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:50.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc7ae51740 1 -- 172.21.15.76:0/1667633252 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56186e36f8c0 con 0x56186e446f00 2024-03-12T06:04:50.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc59ffb640 1 -- 172.21.15.76:0/1667633252 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc6c01ee60 con 0x56186e446f00 2024-03-12T06:04:50.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc59ffb640 1 -- 172.21.15.76:0/1667633252 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbc60003b00 con 0x56186e446f00 2024-03-12T06:04:50.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc7ae51740 10 client.17520 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:50.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc7ae51740 10 client.17520 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:04:50.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc7ae51740 20 client.17520 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:50.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/1667633252 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x56186e462b00 0x56186e4bcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc7ae51740 4 client.17520 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:04:50.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.212+0000 7fbc7ae51740 1 -- 172.21.15.76:0/1667633252 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56186e13a890 con 0x56186e462b00 2024-03-12T06:04:50.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc7581c640 1 --2- 172.21.15.76:0/1667633252 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x56186e462b00 0x56186e4bcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc59ffb640 10 client.17520 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:04:50.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc7581c640 1 --2- 172.21.15.76:0/1667633252 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x56186e462b00 0x56186e4bcd80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7d6b27e0966372ef server_cookie=459295649ed2ea83 in_seq=0 out_seq=0 2024-03-12T06:04:50.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc59ffb640 1 -- 172.21.15.76:0/1667633252 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fbc6c03d020 con 0x56186e462b00 2024-03-12T06:04:50.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc59ffb640 10 client.17520 handle_command_reply: tid=0 2024-03-12T06:04:50.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc7ae51740 1 client.17520 shutdown 2024-03-12T06:04:50.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc7ae51740 1 -- 172.21.15.76:0/1667633252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e446f00 msgr2=0x56186e458550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/1667633252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e446f00 0x56186e458550 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000b010 tx=0x7fbc60019650 comp rx=0 tx=0).stop 2024-03-12T06:04:50.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc7ae51740 1 -- 172.21.15.76:0/1667633252 shutdown_connections 2024-03-12T06:04:50.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/1667633252 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x56186e462b00 0x56186e4bcd80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/1667633252 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x56186e452960 0x56186e458ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/1667633252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56186e447820 0x56186e458a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc7ae51740 1 --2- 172.21.15.76:0/1667633252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56186e446f00 0x56186e458550 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc59ffb640 0 client.17520 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:04:50.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc7ae51740 1 -- 172.21.15.76:0/1667633252 >> 172.21.15.76:0/1667633252 conn(0x56186e441980 msgr2=0x56186e4402f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc7ae51740 1 -- 172.21.15.76:0/1667633252 shutdown_connections 2024-03-12T06:04:50.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc7ae51740 1 -- 172.21.15.76:0/1667633252 wait complete. 2024-03-12T06:04:50.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.213+0000 7fbc7ae51740 20 client.17520 trim_cache size 0 max 16384 2024-03-12T06:04:50.235 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:04:50.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.214+0000 7fbc78281640 1 -- 172.21.15.76:0/3716787513 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbc5c0a1ba0 msgr2=0x7fbc5c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.214+0000 7fbc78281640 1 --2- 172.21.15.76:0/3716787513 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbc5c0a1ba0 0x7fbc5c0a4060 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c005900 tx=0x7fbc6c00aad0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.214+0000 7fbc78281640 1 -- 172.21.15.76:0/3716787513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc7019aea0 msgr2=0x7fbc7019b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.214+0000 7fbc78281640 1 --2- 172.21.15.76:0/3716787513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc7019aea0 0x7fbc7019b350 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fbc68037040 tx=0x7fbc68032fc0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.214+0000 7fbc78281640 1 -- 172.21.15.76:0/3716787513 shutdown_connections 2024-03-12T06:04:50.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.214+0000 7fbc78281640 1 --2- 172.21.15.76:0/3716787513 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fbc5c0a1ba0 0x7fbc5c0a4060 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.214+0000 7fbc78281640 1 --2- 172.21.15.76:0/3716787513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc7019aea0 0x7fbc7019b350 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.214+0000 7fbc78281640 1 --2- 172.21.15.76:0/3716787513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc70105360 0x7fbc70196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.214+0000 7fbc78281640 1 --2- 172.21.15.76:0/3716787513 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbc70100290 0x7fbc70196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.214+0000 7fbc78281640 1 -- 172.21.15.76:0/3716787513 >> 172.21.15.76:0/3716787513 conn(0x7fbc70077230 msgr2=0x7fbc700fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.214+0000 7fbc78281640 1 -- 172.21.15.76:0/3716787513 shutdown_connections 2024-03-12T06:04:50.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.214+0000 7fbc78281640 1 -- 172.21.15.76:0/3716787513 wait complete. 2024-03-12T06:04:50.238 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:04:50.238 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T06:04:50.343 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.336+0000 7f175d6c0640 1 -- 172.21.15.76:0/3591048941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1758100290 msgr2=0x7f1758100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.336+0000 7f175d6c0640 1 --2- 172.21.15.76:0/3591048941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1758100290 0x7f1758100670 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f1750009930 tx=0x7f175002bc20 comp rx=0 tx=0).stop 2024-03-12T06:04:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.336+0000 7f175d6c0640 1 -- 172.21.15.76:0/3591048941 shutdown_connections 2024-03-12T06:04:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.336+0000 7f175d6c0640 1 --2- 172.21.15.76:0/3591048941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1758105360 0x7f175810cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.336+0000 7f175d6c0640 1 --2- 172.21.15.76:0/3591048941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1758100c40 0x7f1758104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.336+0000 7f175d6c0640 1 --2- 172.21.15.76:0/3591048941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1758100290 0x7f1758100670 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.336+0000 7f175d6c0640 1 -- 172.21.15.76:0/3591048941 >> 172.21.15.76:0/3591048941 conn(0x7f1758077230 msgr2=0x7f1758077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.344 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.336+0000 7f175d6c0640 1 -- 172.21.15.76:0/3591048941 shutdown_connections 2024-03-12T06:04:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.337+0000 7f175d6c0640 1 -- 172.21.15.76:0/3591048941 wait complete. 2024-03-12T06:04:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.337+0000 7f175d6c0640 1 Processor -- start 2024-03-12T06:04:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.337+0000 7f175d6c0640 1 -- start start 2024-03-12T06:04:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.337+0000 7f175d6c0640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1758100290 0x7f1758196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.337+0000 7f175d6c0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1758105360 0x7f1758196c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.337+0000 7f175d6c0640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f175819afe0 0x7f175819b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.337+0000 7f175d6c0640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175819b9d0 con 0x7f175819afe0 2024-03-12T06:04:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.337+0000 7f175d6c0640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175819bb40 con 0x7f1758100290 2024-03-12T06:04:50.345 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.337+0000 7f175d6c0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175819bcb0 con 0x7f1758105360 2024-03-12T06:04:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f17577fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f175819afe0 0x7f175819b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f17577fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f175819afe0 0x7f175819b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53002/0 (socket says 172.21.15.76:53002) 2024-03-12T06:04:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f17577fe640 1 -- 172.21.15.76:0/1082253070 learned_addr learned my addr 172.21.15.76:0/1082253070 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f17567fc640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1758105360 0x7f1758196c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f17577fe640 1 -- 172.21.15.76:0/1082253070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1758105360 msgr2=0x7f1758196c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f17577fe640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1758105360 0x7f1758196c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f1756ffd640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1758100290 0x7f1758196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.346 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f17577fe640 1 -- 172.21.15.76:0/1082253070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1758100290 msgr2=0x7f1758196720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f17577fe640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1758100290 0x7f1758196720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f17577fe640 1 -- 172.21.15.76:0/1082253070 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1750009560 con 0x7f175819afe0 2024-03-12T06:04:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f17567fc640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1758105360 0x7f1758196c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f17577fe640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f175819afe0 0x7f175819b490 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f174800cab0 tx=0x7f174800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f173ffff640 1 -- 172.21.15.76:0/1082253070 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174800ae20 con 0x7f175819afe0 2024-03-12T06:04:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f173ffff640 1 -- 172.21.15.76:0/1082253070 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f174800f930 con 0x7f175819afe0 2024-03-12T06:04:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f175d6c0640 1 -- 172.21.15.76:0/1082253070 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17581a3290 con 0x7f175819afe0 2024-03-12T06:04:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f173ffff640 1 -- 172.21.15.76:0/1082253070 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1748018a70 con 0x7f175819afe0 2024-03-12T06:04:50.347 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.338+0000 7f175d6c0640 1 -- 172.21.15.76:0/1082253070 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17581a36f0 con 0x7f175819afe0 2024-03-12T06:04:50.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.340+0000 7f173ffff640 1 -- 172.21.15.76:0/1082253070 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f174800faa0 con 0x7f175819afe0 2024-03-12T06:04:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.340+0000 7f176036e740 1 Processor -- start 2024-03-12T06:04:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f176036e740 1 -- start start 2024-03-12T06:04:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f176036e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264460 0x563ae6264840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f176036e740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264d80 0x563ae626d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f176036e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae626d960 0x563ae626fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f176036e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae621d960 con 0x563ae6264460 2024-03-12T06:04:50.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f176036e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae61d1310 con 0x563ae626d960 2024-03-12T06:04:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f176036e740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae61ce840 con 0x563ae6264d80 2024-03-12T06:04:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f173ffff640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f17400a1ba0 0x7f17400a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f17567fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264460 0x563ae6264840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f17567fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264460 0x563ae6264840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53004/0 (socket says 172.21.15.76:53004) 2024-03-12T06:04:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f17567fc640 1 -- 172.21.15.76:0/3614702722 learned_addr learned my addr 172.21.15.76:0/3614702722 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f1756ffd640 1 --2- 172.21.15.76:0/3614702722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264d80 0x563ae626d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f17577fe640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f17400a1ba0 0x7f17400a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f17567fc640 1 --2- 172.21.15.76:0/3614702722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae626d960 0x563ae626fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f173ffff640 1 -- 172.21.15.76:0/1082253070 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f17480ce0e0 con 0x7f175819afe0 2024-03-12T06:04:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f17567fc640 1 -- 172.21.15.76:0/3614702722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264d80 msgr2=0x563ae626d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f17567fc640 1 --2- 172.21.15.76:0/3614702722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264d80 0x563ae626d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f17567fc640 1 -- 172.21.15.76:0/3614702722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae626d960 msgr2=0x563ae626fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f17567fc640 1 --2- 172.21.15.76:0/3614702722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae626d960 0x563ae626fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f17567fc640 1 -- 172.21.15.76:0/3614702722 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ae618c060 con 0x563ae6264460 2024-03-12T06:04:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f17567fc640 1 --2- 172.21.15.76:0/3614702722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264460 0x563ae6264840 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f174c00afa0 tx=0x7f174c00a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4564f3c437ed106f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.341+0000 7f17577fe640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f17400a1ba0 0x7f17400a4060 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f17480cb980 tx=0x7f174806a190 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.342+0000 7f173d7fa640 1 -- 172.21.15.76:0/3614702722 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c00ee50 con 0x563ae6264460 2024-03-12T06:04:50.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.342+0000 7f173d7fa640 1 -- 172.21.15.76:0/3614702722 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f174c00c070 con 0x563ae6264460 2024-03-12T06:04:50.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.342+0000 7f173d7fa640 1 -- 172.21.15.76:0/3614702722 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c015710 con 0x563ae6264460 2024-03-12T06:04:50.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.342+0000 7f176036e740 1 -- 172.21.15.76:0/3614702722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264460 msgr2=0x563ae6264840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.342+0000 7f176036e740 1 --2- 172.21.15.76:0/3614702722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264460 0x563ae6264840 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f174c00afa0 tx=0x7f174c00a3f0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.342+0000 7f176036e740 1 -- 172.21.15.76:0/3614702722 shutdown_connections 2024-03-12T06:04:50.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.342+0000 7f176036e740 1 --2- 172.21.15.76:0/3614702722 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae626d960 0x563ae626fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.342+0000 7f176036e740 1 --2- 172.21.15.76:0/3614702722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264d80 0x563ae626d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.342+0000 7f176036e740 1 --2- 172.21.15.76:0/3614702722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264460 0x563ae6264840 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.342+0000 7f176036e740 1 -- 172.21.15.76:0/3614702722 >> 172.21.15.76:0/3614702722 conn(0x563ae625eb90 msgr2=0x563ae625ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.342+0000 7f176036e740 1 -- 172.21.15.76:0/3614702722 shutdown_connections 2024-03-12T06:04:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.342+0000 7f176036e740 1 -- 172.21.15.76:0/3614702722 wait complete. 2024-03-12T06:04:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f176036e740 1 -- start start 2024-03-12T06:04:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f176036e740 1 Processor -- start 2024-03-12T06:04:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f176036e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae6264460 0x563ae626a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f176036e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264d80 0x563ae626aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f176036e740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae626d960 0x563ae626b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f176036e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae61ca020 con 0x563ae6264d80 2024-03-12T06:04:50.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f176036e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae5e9d430 con 0x563ae6264460 2024-03-12T06:04:50.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f176036e740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae5e59e00 con 0x563ae626d960 2024-03-12T06:04:50.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f176036e740 10 client.0 _ll_register_callbacks cb 0x563ae6171cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:50.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f1756ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264d80 0x563ae626aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f1756ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264d80 0x563ae626aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53010/0 (socket says 172.21.15.76:53010) 2024-03-12T06:04:50.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f1756ffd640 1 -- 172.21.15.76:0/1779855402 learned_addr learned my addr 172.21.15.76:0/1779855402 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f17567fc640 1 --2- 172.21.15.76:0/1779855402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae626d960 0x563ae626b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f17567fc640 1 --2- 172.21.15.76:0/1779855402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae6264460 0x563ae626a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f1756ffd640 1 -- 172.21.15.76:0/1779855402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae626d960 msgr2=0x563ae626b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f1756ffd640 1 --2- 172.21.15.76:0/1779855402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae626d960 0x563ae626b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f1756ffd640 1 -- 172.21.15.76:0/1779855402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae6264460 msgr2=0x563ae626a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f1756ffd640 1 --2- 172.21.15.76:0/1779855402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae6264460 0x563ae626a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f1756ffd640 1 -- 172.21.15.76:0/1779855402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174c00ab20 con 0x563ae6264d80 2024-03-12T06:04:50.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f17567fc640 1 --2- 172.21.15.76:0/1779855402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae6264460 0x563ae626a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:50.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f1756ffd640 1 --2- 172.21.15.76:0/1779855402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264d80 0x563ae626aeb0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f1750009900 tx=0x7f175002ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36401877fb0d35a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.343+0000 7f173aff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:50.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f173aff5640 1 -- 172.21.15.76:0/1779855402 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175002cc30 con 0x563ae6264d80 2024-03-12T06:04:50.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f173aff5640 1 -- 172.21.15.76:0/1779855402 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f1750004250 con 0x563ae6264d80 2024-03-12T06:04:50.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f176036e740 1 -- 172.21.15.76:0/1779855402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563ae5ec97e0 con 0x563ae6264d80 2024-03-12T06:04:50.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f173aff5640 1 -- 172.21.15.76:0/1779855402 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1750004410 con 0x563ae6264d80 2024-03-12T06:04:50.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f176036e740 10 client.17532 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:50.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f176036e740 1 -- 172.21.15.76:0/1779855402 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563ae61c86a0 con 0x563ae6264d80 2024-03-12T06:04:50.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f173aff5640 1 -- 172.21.15.76:0/1779855402 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563ae5ec97e0 con 0x563ae6264d80 2024-03-12T06:04:50.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f173aff5640 1 -- 172.21.15.76:0/1779855402 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f175003c050 con 0x563ae6264d80 2024-03-12T06:04:50.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f176036e740 10 client.17532 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:50.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f176036e740 10 client.17532 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:50.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f176036e740 20 client.17532 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:50.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f176036e740 1 --2- 172.21.15.76:0/1779855402 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563ae627dcb0 0x563ae62800d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f176036e740 4 client.17532 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:04:50.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f176036e740 1 -- 172.21.15.76:0/1779855402 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563ae5f06720 con 0x563ae627dcb0 2024-03-12T06:04:50.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.344+0000 7f17567fc640 1 --2- 172.21.15.76:0/1779855402 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563ae627dcb0 0x563ae62800d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.345+0000 7f17567fc640 1 --2- 172.21.15.76:0/1779855402 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563ae627dcb0 0x563ae62800d0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c4adb587be31659 server_cookie=b72043cf72099183 in_seq=0 out_seq=0 2024-03-12T06:04:50.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.345+0000 7f173aff5640 10 client.17532 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:50.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.345+0000 7f173aff5640 1 -- 172.21.15.76:0/1779855402 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563ae5f06720 con 0x563ae627dcb0 2024-03-12T06:04:50.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.345+0000 7f173aff5640 10 client.17532 handle_command_reply: tid=0 2024-03-12T06:04:50.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.345+0000 7f176036e740 1 client.17532 shutdown 2024-03-12T06:04:50.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.346+0000 7f176036e740 1 -- 172.21.15.76:0/1779855402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264d80 msgr2=0x563ae626aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.346+0000 7f176036e740 1 --2- 172.21.15.76:0/1779855402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264d80 0x563ae626aeb0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f1750009900 tx=0x7f175002ca00 comp rx=0 tx=0).stop 2024-03-12T06:04:50.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.346+0000 7f176036e740 1 -- 172.21.15.76:0/1779855402 shutdown_connections 2024-03-12T06:04:50.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.346+0000 7f176036e740 1 --2- 172.21.15.76:0/1779855402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae626d960 0x563ae626b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.346+0000 7f176036e740 1 --2- 172.21.15.76:0/1779855402 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563ae627dcb0 0x563ae62800d0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.346+0000 7f176036e740 1 --2- 172.21.15.76:0/1779855402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264d80 0x563ae626aeb0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.346+0000 7f176036e740 1 --2- 172.21.15.76:0/1779855402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae6264460 0x563ae626a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.346+0000 7f176036e740 1 -- 172.21.15.76:0/1779855402 >> 172.21.15.76:0/1779855402 conn(0x563ae625eb90 msgr2=0x563ae6260050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.346+0000 7f173aff5640 0 client.17532 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:50.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.346+0000 7f176036e740 1 -- 172.21.15.76:0/1779855402 shutdown_connections 2024-03-12T06:04:50.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.346+0000 7f176036e740 1 -- 172.21.15.76:0/1779855402 wait complete. 2024-03-12T06:04:50.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.346+0000 7f176036e740 20 client.17532 trim_cache size 0 max 16384 2024-03-12T06:04:50.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.357+0000 7f176036e740 1 Processor -- start 2024-03-12T06:04:50.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.357+0000 7f176036e740 1 -- start start 2024-03-12T06:04:50.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.357+0000 7f176036e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264460 0x563ae626b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.357+0000 7f176036e740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264d80 0x563ae6274810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.357+0000 7f176036e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae626d960 0x563ae6274d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.357+0000 7f176036e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae614f130 con 0x563ae6264460 2024-03-12T06:04:50.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.357+0000 7f176036e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae627a850 con 0x563ae626d960 2024-03-12T06:04:50.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.357+0000 7f176036e740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae6263ea0 con 0x563ae6264d80 2024-03-12T06:04:50.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.357+0000 7f17567fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264460 0x563ae626b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.358+0000 7f17567fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264460 0x563ae626b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53024/0 (socket says 172.21.15.76:53024) 2024-03-12T06:04:50.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.358+0000 7f17567fc640 1 -- 172.21.15.76:0/428515613 learned_addr learned my addr 172.21.15.76:0/428515613 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.358+0000 7f17567fc640 1 -- 172.21.15.76:0/428515613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264d80 msgr2=0x563ae6274810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.358+0000 7f17567fc640 1 --2- 172.21.15.76:0/428515613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264d80 0x563ae6274810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.358+0000 7f17567fc640 1 -- 172.21.15.76:0/428515613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae626d960 msgr2=0x563ae6274d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.358+0000 7f17567fc640 1 --2- 172.21.15.76:0/428515613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae626d960 0x563ae6274d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.358+0000 7f17567fc640 1 -- 172.21.15.76:0/428515613 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ae6164c20 con 0x563ae6264460 2024-03-12T06:04:50.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.358+0000 7f17567fc640 1 --2- 172.21.15.76:0/428515613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264460 0x563ae626b210 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f174c00f010 tx=0x7f174c00a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779ac0ee4377935f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.358+0000 7f173cff9640 1 -- 172.21.15.76:0/428515613 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c02ad00 con 0x563ae6264460 2024-03-12T06:04:50.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.358+0000 7f173cff9640 1 -- 172.21.15.76:0/428515613 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f174c028070 con 0x563ae6264460 2024-03-12T06:04:50.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.358+0000 7f173cff9640 1 -- 172.21.15.76:0/428515613 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f174c016870 con 0x563ae6264460 2024-03-12T06:04:50.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 -- 172.21.15.76:0/428515613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264460 msgr2=0x563ae626b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 --2- 172.21.15.76:0/428515613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264460 0x563ae626b210 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f174c00f010 tx=0x7f174c00a3f0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 -- 172.21.15.76:0/428515613 shutdown_connections 2024-03-12T06:04:50.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 --2- 172.21.15.76:0/428515613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae626d960 0x563ae6274d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 --2- 172.21.15.76:0/428515613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264d80 0x563ae6274810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 --2- 172.21.15.76:0/428515613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264460 0x563ae626b210 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 -- 172.21.15.76:0/428515613 >> 172.21.15.76:0/428515613 conn(0x563ae625eb90 msgr2=0x563ae6284a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 -- 172.21.15.76:0/428515613 shutdown_connections 2024-03-12T06:04:50.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 -- 172.21.15.76:0/428515613 wait complete. 2024-03-12T06:04:50.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 -- start start 2024-03-12T06:04:50.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 Processor -- start 2024-03-12T06:04:50.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264460 0x563ae6274400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264d80 0x563ae6274940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae626d960 0x563ae6272910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae6272f20 con 0x563ae6264d80 2024-03-12T06:04:50.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae6273090 con 0x563ae626d960 2024-03-12T06:04:50.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae6273250 con 0x563ae6264460 2024-03-12T06:04:50.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f176036e740 10 client.0 _ll_register_callbacks cb 0x7f174c00a250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:50.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f1756ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264d80 0x563ae6274940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f1756ffd640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264d80 0x563ae6274940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53030/0 (socket says 172.21.15.76:53030) 2024-03-12T06:04:50.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.359+0000 7f1756ffd640 1 -- 172.21.15.76:0/2299658306 learned_addr learned my addr 172.21.15.76:0/2299658306 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:50.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f17567fc640 1 --2- 172.21.15.76:0/2299658306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264460 0x563ae6274400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f1756ffd640 1 -- 172.21.15.76:0/2299658306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264460 msgr2=0x563ae6274400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f1756ffd640 1 --2- 172.21.15.76:0/2299658306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264460 0x563ae6274400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f1756ffd640 1 -- 172.21.15.76:0/2299658306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae626d960 msgr2=0x563ae6272910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:50.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f1756ffd640 1 --2- 172.21.15.76:0/2299658306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae626d960 0x563ae6272910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f1756ffd640 1 -- 172.21.15.76:0/2299658306 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ae618c060 con 0x563ae6264d80 2024-03-12T06:04:50.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f17567fc640 1 --2- 172.21.15.76:0/2299658306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264460 0x563ae6274400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:50.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f1756ffd640 1 --2- 172.21.15.76:0/2299658306 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264d80 0x563ae6274940 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f1750009850 tx=0x7f1750005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51a1159b069fea22 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:50.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f173aff5640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:50.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f173aff5640 1 -- 172.21.15.76:0/2299658306 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175002c9e0 con 0x563ae6264d80 2024-03-12T06:04:50.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f173aff5640 1 -- 172.21.15.76:0/2299658306 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f1750000e40 con 0x563ae6264d80 2024-03-12T06:04:50.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f176036e740 1 -- 172.21.15.76:0/2299658306 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f174c00c070 con 0x563ae6264d80 2024-03-12T06:04:50.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f173aff5640 1 -- 172.21.15.76:0/2299658306 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1750003e10 con 0x563ae6264d80 2024-03-12T06:04:50.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f176036e740 10 client.17538 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:50.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f176036e740 1 -- 172.21.15.76:0/2299658306 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563ae618a8c0 con 0x563ae6264d80 2024-03-12T06:04:50.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f173aff5640 1 -- 172.21.15.76:0/2299658306 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f175005e080 con 0x563ae6264d80 2024-03-12T06:04:50.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f173aff5640 1 -- 172.21.15.76:0/2299658306 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f175003c040 con 0x563ae6264d80 2024-03-12T06:04:50.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f176036e740 10 client.17538 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:50.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f176036e740 10 client.17538 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:50.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f176036e740 20 client.17538 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:50.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f176036e740 1 --2- 172.21.15.76:0/2299658306 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563ae62d50c0 0x563ae62d74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:50.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f176036e740 4 client.17538 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:04:50.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.360+0000 7f176036e740 1 -- 172.21.15.76:0/2299658306 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563ae5f55890 con 0x563ae62d50c0 2024-03-12T06:04:50.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f17567fc640 1 --2- 172.21.15.76:0/2299658306 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563ae62d50c0 0x563ae62d74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:50.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f17567fc640 1 --2- 172.21.15.76:0/2299658306 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563ae62d50c0 0x563ae62d74e0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=427629bead85d756 server_cookie=19ea66d370604fdf in_seq=0 out_seq=0 2024-03-12T06:04:50.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f173aff5640 10 client.17538 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:50.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f173aff5640 1 -- 172.21.15.76:0/2299658306 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563ae5f55890 con 0x563ae62d50c0 2024-03-12T06:04:50.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f173aff5640 10 client.17538 handle_command_reply: tid=0 2024-03-12T06:04:50.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f176036e740 1 client.17538 shutdown 2024-03-12T06:04:50.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f176036e740 1 -- 172.21.15.76:0/2299658306 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264d80 msgr2=0x563ae6274940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f176036e740 1 --2- 172.21.15.76:0/2299658306 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264d80 0x563ae6274940 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f1750009850 tx=0x7f1750005bc0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f176036e740 1 -- 172.21.15.76:0/2299658306 shutdown_connections 2024-03-12T06:04:50.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f176036e740 1 --2- 172.21.15.76:0/2299658306 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x563ae626d960 0x563ae6272910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f176036e740 1 --2- 172.21.15.76:0/2299658306 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x563ae62d50c0 0x563ae62d74e0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f176036e740 1 --2- 172.21.15.76:0/2299658306 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x563ae6264d80 0x563ae6274940 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f176036e740 1 --2- 172.21.15.76:0/2299658306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ae6264460 0x563ae6274400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f176036e740 1 -- 172.21.15.76:0/2299658306 >> 172.21.15.76:0/2299658306 conn(0x563ae625eb90 msgr2=0x563ae625b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f173aff5640 0 client.17538 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:50.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f176036e740 1 -- 172.21.15.76:0/2299658306 shutdown_connections 2024-03-12T06:04:50.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f176036e740 1 -- 172.21.15.76:0/2299658306 wait complete. 2024-03-12T06:04:50.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.361+0000 7f176036e740 20 client.17538 trim_cache size 0 max 16384 2024-03-12T06:04:50.382 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:04:50.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.362+0000 7f175d6c0640 1 -- 172.21.15.76:0/1082253070 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f17400a1ba0 msgr2=0x7f17400a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.362+0000 7f175d6c0640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f17400a1ba0 0x7f17400a4060 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f17480cb980 tx=0x7f174806a190 comp rx=0 tx=0).stop 2024-03-12T06:04:50.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.362+0000 7f175d6c0640 1 -- 172.21.15.76:0/1082253070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f175819afe0 msgr2=0x7f175819b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:50.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.362+0000 7f175d6c0640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f175819afe0 0x7f175819b490 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f174800cab0 tx=0x7f174800cf70 comp rx=0 tx=0).stop 2024-03-12T06:04:50.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.362+0000 7f175d6c0640 1 -- 172.21.15.76:0/1082253070 shutdown_connections 2024-03-12T06:04:50.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.362+0000 7f175d6c0640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f17400a1ba0 0x7f17400a4060 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.362+0000 7f175d6c0640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f175819afe0 0x7f175819b490 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.362+0000 7f175d6c0640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1758105360 0x7f1758196c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.362+0000 7f175d6c0640 1 --2- 172.21.15.76:0/1082253070 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1758100290 0x7f1758196720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:50.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.362+0000 7f175d6c0640 1 -- 172.21.15.76:0/1082253070 >> 172.21.15.76:0/1082253070 conn(0x7f1758077230 msgr2=0x7f17580fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:50.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.362+0000 7f175d6c0640 1 -- 172.21.15.76:0/1082253070 shutdown_connections 2024-03-12T06:04:50.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:50.362+0000 7f175d6c0640 1 -- 172.21.15.76:0/1082253070 wait complete. 2024-03-12T06:04:50.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:50 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/704777745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:04:50.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:50 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/704777745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:04:50.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:50 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/704777745' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:04:51.385 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T06:04:51.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:04:51.386 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T06:04:51.501 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.496+0000 7f4a9a288640 1 -- 172.21.15.76:0/1771312347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a9406a550 msgr2=0x7f4a94105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.496+0000 7f4a9a288640 1 --2- 172.21.15.76:0/1771312347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a9406a550 0x7f4a94105d80 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f4a88009930 tx=0x7f4a8802bc30 comp rx=0 tx=0).stop 2024-03-12T06:04:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.496+0000 7f4a9a288640 1 -- 172.21.15.76:0/1771312347 shutdown_connections 2024-03-12T06:04:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.496+0000 7f4a9a288640 1 --2- 172.21.15.76:0/1771312347 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a94106460 0x7f4a9410e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.496+0000 7f4a9a288640 1 --2- 172.21.15.76:0/1771312347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a9406a550 0x7f4a94105d80 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.496+0000 7f4a9a288640 1 --2- 172.21.15.76:0/1771312347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a94069ba0 0x7f4a94069f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.496+0000 7f4a9a288640 1 -- 172.21.15.76:0/1771312347 >> 172.21.15.76:0/1771312347 conn(0x7f4a94077200 msgr2=0x7f4a94077610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.496+0000 7f4a9a288640 1 -- 172.21.15.76:0/1771312347 shutdown_connections 2024-03-12T06:04:51.502 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.496+0000 7f4a9a288640 1 -- 172.21.15.76:0/1771312347 wait complete. 2024-03-12T06:04:51.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a9a288640 1 Processor -- start 2024-03-12T06:04:51.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a9a288640 1 -- start start 2024-03-12T06:04:51.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a9a288640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a94069ba0 0x7f4a9419d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a9a288640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a94106460 0x7f4a9419dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a9a288640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a94197630 0x7f4a94197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a9a288640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a94198100 con 0x7f4a94069ba0 2024-03-12T06:04:51.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a9a288640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a94198270 con 0x7f4a94106460 2024-03-12T06:04:51.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a9a288640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a94198430 con 0x7f4a94197630 2024-03-12T06:04:51.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a93fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a94069ba0 0x7f4a9419d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a93fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a94069ba0 0x7f4a9419d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46786/0 (socket says 172.21.15.76:46786) 2024-03-12T06:04:51.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a93fff640 1 -- 172.21.15.76:0/2473396194 learned_addr learned my addr 172.21.15.76:0/2473396194 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:51.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a98825640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a94197630 0x7f4a94197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a937fe640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a94106460 0x7f4a9419dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a93fff640 1 -- 172.21.15.76:0/2473396194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a94197630 msgr2=0x7f4a94197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a93fff640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a94197630 0x7f4a94197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.497+0000 7f4a93fff640 1 -- 172.21.15.76:0/2473396194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a94106460 msgr2=0x7f4a9419dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.498+0000 7f4a93fff640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a94106460 0x7f4a9419dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.498+0000 7f4a93fff640 1 -- 172.21.15.76:0/2473396194 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a88009560 con 0x7f4a94069ba0 2024-03-12T06:04:51.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.498+0000 7f4a937fe640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a94106460 0x7f4a9419dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:51.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.498+0000 7f4a98825640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a94197630 0x7f4a94197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:04:51.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.498+0000 7f4a93fff640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a94069ba0 0x7f4a9419d570 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f4a8400d770 tx=0x7f4a8400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:51.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.498+0000 7f4a917fa640 1 -- 172.21.15.76:0/2473396194 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a84009c30 con 0x7f4a94069ba0 2024-03-12T06:04:51.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.498+0000 7f4a917fa640 1 -- 172.21.15.76:0/2473396194 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a8400de80 con 0x7f4a94069ba0 2024-03-12T06:04:51.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.498+0000 7f4a9a288640 1 -- 172.21.15.76:0/2473396194 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a941a43d0 con 0x7f4a94069ba0 2024-03-12T06:04:51.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.498+0000 7f4a917fa640 1 -- 172.21.15.76:0/2473396194 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a84004e80 con 0x7f4a94069ba0 2024-03-12T06:04:51.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.498+0000 7f4a9a288640 1 -- 172.21.15.76:0/2473396194 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a941a4860 con 0x7f4a94069ba0 2024-03-12T06:04:51.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.500+0000 7f4a917fa640 1 -- 172.21.15.76:0/2473396194 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f4a84005020 con 0x7f4a94069ba0 2024-03-12T06:04:51.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.500+0000 7f4a9ce64740 1 Processor -- start 2024-03-12T06:04:51.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.500+0000 7f4a9ce64740 1 -- start start 2024-03-12T06:04:51.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a9ce64740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cf5380 0x556745cf5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a917fa640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4a7c0a1ba0 0x7f4a7c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a9ce64740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5d30 0x556745cfe2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a9ce64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cfe8b0 0x556745d00d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.514 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a9ce64740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556745c63f90 con 0x556745cf5d30 2024-03-12T06:04:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a9ce64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x556745c614c0 con 0x556745cfe8b0 2024-03-12T06:04:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a9ce64740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556745c5cca0 con 0x556745cf5380 2024-03-12T06:04:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a937fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5d30 0x556745cfe2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a93fff640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4a7c0a1ba0 0x7f4a7c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a937fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5d30 0x556745cfe2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46800/0 (socket says 172.21.15.76:46800) 2024-03-12T06:04:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a937fe640 1 -- 172.21.15.76:0/2835198092 learned_addr learned my addr 172.21.15.76:0/2835198092 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a98825640 1 --2- 172.21.15.76:0/2835198092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cf5380 0x556745cf5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a917fa640 1 -- 172.21.15.76:0/2473396194 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f4a840cd2f0 con 0x7f4a94069ba0 2024-03-12T06:04:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a98825640 1 --2- 172.21.15.76:0/2835198092 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cfe8b0 0x556745d00d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a937fe640 1 -- 172.21.15.76:0/2835198092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cf5380 msgr2=0x556745cf5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a937fe640 1 --2- 172.21.15.76:0/2835198092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cf5380 0x556745cf5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a937fe640 1 -- 172.21.15.76:0/2835198092 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cfe8b0 msgr2=0x556745d00d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a937fe640 1 --2- 172.21.15.76:0/2835198092 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cfe8b0 0x556745d00d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a937fe640 1 -- 172.21.15.76:0/2835198092 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556745c1f0b0 con 0x556745cf5d30 2024-03-12T06:04:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a98825640 1 --2- 172.21.15.76:0/2835198092 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cfe8b0 0x556745d00d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a937fe640 1 --2- 172.21.15.76:0/2835198092 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5d30 0x556745cfe2e0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f4a88005c40 tx=0x7f4a88004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91d870436a055f28 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:51.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a93fff640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4a7c0a1ba0 0x7f4a7c0a4060 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4a84099eb0 tx=0x7f4a8408f5f0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a7a7fc640 1 -- 172.21.15.76:0/2835198092 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8802cea0 con 0x556745cf5d30 2024-03-12T06:04:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a7a7fc640 1 -- 172.21.15.76:0/2835198092 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a88004040 con 0x556745cf5d30 2024-03-12T06:04:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.501+0000 7f4a7a7fc640 1 -- 172.21.15.76:0/2835198092 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88005270 con 0x556745cf5d30 2024-03-12T06:04:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.502+0000 7f4a9ce64740 1 -- 172.21.15.76:0/2835198092 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5d30 msgr2=0x556745cfe2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.502+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/2835198092 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5d30 0x556745cfe2e0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f4a88005c40 tx=0x7f4a88004740 comp rx=0 tx=0).stop 2024-03-12T06:04:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.502+0000 7f4a9ce64740 1 -- 172.21.15.76:0/2835198092 shutdown_connections 2024-03-12T06:04:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.502+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/2835198092 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cfe8b0 0x556745d00d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.502+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/2835198092 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5d30 0x556745cfe2e0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.502+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/2835198092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cf5380 0x556745cf5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.502+0000 7f4a9ce64740 1 -- 172.21.15.76:0/2835198092 >> 172.21.15.76:0/2835198092 conn(0x556745cef840 msgr2=0x556745cefc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.502+0000 7f4a9ce64740 1 -- 172.21.15.76:0/2835198092 shutdown_connections 2024-03-12T06:04:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.502+0000 7f4a9ce64740 1 -- 172.21.15.76:0/2835198092 wait complete. 2024-03-12T06:04:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.502+0000 7f4a9ce64740 1 -- start start 2024-03-12T06:04:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a9ce64740 1 Processor -- start 2024-03-12T06:04:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a9ce64740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5380 0x556745cfb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a9ce64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cf5d30 0x556745cfbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a9ce64740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cfe8b0 0x556745cfc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a9ce64740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556745c01f40 con 0x556745cf5380 2024-03-12T06:04:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a9ce64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567459304b0 con 0x556745cf5d30 2024-03-12T06:04:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a9ce64740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556745be2670 con 0x556745cfe8b0 2024-03-12T06:04:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a9ce64740 10 client.0 _ll_register_callbacks cb 0x556745c4ff70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a98825640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5380 0x556745cfb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a98825640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5380 0x556745cfb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46816/0 (socket says 172.21.15.76:46816) 2024-03-12T06:04:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a98825640 1 -- 172.21.15.76:0/1128690557 learned_addr learned my addr 172.21.15.76:0/1128690557 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:51.519 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a937fe640 1 --2- 172.21.15.76:0/1128690557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cf5d30 0x556745cfbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a98825640 1 --2- 172.21.15.76:0/1128690557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cfe8b0 0x556745cfc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a98825640 1 -- 172.21.15.76:0/1128690557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cfe8b0 msgr2=0x556745cfc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a98825640 1 --2- 172.21.15.76:0/1128690557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cfe8b0 0x556745cfc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a98825640 1 -- 172.21.15.76:0/1128690557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cf5d30 msgr2=0x556745cfbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a98825640 1 --2- 172.21.15.76:0/1128690557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cf5d30 0x556745cfbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a98825640 1 -- 172.21.15.76:0/1128690557 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9410e840 con 0x556745cf5380 2024-03-12T06:04:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a937fe640 1 --2- 172.21.15.76:0/1128690557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cf5d30 0x556745cfbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:04:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a98825640 1 --2- 172.21.15.76:0/1128690557 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5380 0x556745cfb810 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c014f50 tx=0x7f4a8c009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2a7980de743e27e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:51.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.503+0000 7f4a737fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.504+0000 7f4a737fe640 1 -- 172.21.15.76:0/1128690557 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c011e70 con 0x556745cf5380 2024-03-12T06:04:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.504+0000 7f4a737fe640 1 -- 172.21.15.76:0/1128690557 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f4a8c004b20 con 0x556745cf5380 2024-03-12T06:04:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.504+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1128690557 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556745c36f40 con 0x556745cf5380 2024-03-12T06:04:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.504+0000 7f4a737fe640 1 -- 172.21.15.76:0/1128690557 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a8c004ce0 con 0x556745cf5380 2024-03-12T06:04:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.504+0000 7f4a9ce64740 10 client.17550 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.504+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1128690557 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556745c5b320 con 0x556745cf5380 2024-03-12T06:04:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.504+0000 7f4a737fe640 1 -- 172.21.15.76:0/1128690557 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x556745c36f40 con 0x556745cf5380 2024-03-12T06:04:51.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.504+0000 7f4a737fe640 1 -- 172.21.15.76:0/1128690557 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4a8c012780 con 0x556745cf5380 2024-03-12T06:04:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.504+0000 7f4a9ce64740 10 client.17550 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.504+0000 7f4a9ce64740 10 client.17550 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.504+0000 7f4a9ce64740 20 client.17550 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.504+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/1128690557 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556745d0ee50 0x556745d11270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.504+0000 7f4a9ce64740 4 client.17550 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:04:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.504+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1128690557 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556745c53310 con 0x556745d0ee50 2024-03-12T06:04:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.505+0000 7f4a937fe640 1 --2- 172.21.15.76:0/1128690557 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556745d0ee50 0x556745d11270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.505+0000 7f4a937fe640 1 --2- 172.21.15.76:0/1128690557 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556745d0ee50 0x556745d11270 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f7666a34ec11057 server_cookie=4ec5b2b5ef9a221 in_seq=0 out_seq=0 2024-03-12T06:04:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.505+0000 7f4a737fe640 10 client.17550 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a737fe640 1 -- 172.21.15.76:0/1128690557 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556745c53310 con 0x556745d0ee50 2024-03-12T06:04:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a737fe640 10 client.17550 handle_command_reply: tid=0 2024-03-12T06:04:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a9ce64740 1 client.17550 shutdown 2024-03-12T06:04:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1128690557 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5380 msgr2=0x556745cfb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/1128690557 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5380 0x556745cfb810 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c014f50 tx=0x7f4a8c009d70 comp rx=0 tx=0).stop 2024-03-12T06:04:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1128690557 shutdown_connections 2024-03-12T06:04:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/1128690557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cfe8b0 0x556745cfc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/1128690557 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cf5d30 0x556745cfbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/1128690557 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556745d0ee50 0x556745d11270 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/1128690557 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5380 0x556745cfb810 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a737fe640 0 client.17550 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:51.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1128690557 >> 172.21.15.76:0/1128690557 conn(0x556745cef840 msgr2=0x556745cf17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:51.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1128690557 shutdown_connections 2024-03-12T06:04:51.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1128690557 wait complete. 2024-03-12T06:04:51.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.506+0000 7f4a9ce64740 20 client.17550 trim_cache size 0 max 16384 2024-03-12T06:04:51.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a9ce64740 1 Processor -- start 2024-03-12T06:04:51.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a9ce64740 1 -- start start 2024-03-12T06:04:51.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a9ce64740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5380 0x556745cfdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a9ce64740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cf5d30 0x556745cf8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a9ce64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cfe8b0 0x556745cf8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a9ce64740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556745cfc980 con 0x556745cf5380 2024-03-12T06:04:51.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a9ce64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x556745c53310 con 0x556745cfe8b0 2024-03-12T06:04:51.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a9ce64740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556745cf0640 con 0x556745cf5d30 2024-03-12T06:04:51.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a98825640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5380 0x556745cfdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a98825640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5380 0x556745cfdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46822/0 (socket says 172.21.15.76:46822) 2024-03-12T06:04:51.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a98825640 1 -- 172.21.15.76:0/3535609956 learned_addr learned my addr 172.21.15.76:0/3535609956 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:51.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a98825640 1 -- 172.21.15.76:0/3535609956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cf5d30 msgr2=0x556745cf8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a98825640 1 --2- 172.21.15.76:0/3535609956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cf5d30 0x556745cf8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a98825640 1 -- 172.21.15.76:0/3535609956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cfe8b0 msgr2=0x556745cf8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a98825640 1 --2- 172.21.15.76:0/3535609956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cfe8b0 0x556745cf8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.521+0000 7f4a98825640 1 -- 172.21.15.76:0/3535609956 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556745bf7c00 con 0x556745cf5380 2024-03-12T06:04:51.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.522+0000 7f4a98825640 1 --2- 172.21.15.76:0/3535609956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5380 0x556745cfdf40 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00a010 tx=0x7f4a8c014c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b403cbcbfa6512cf server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:51.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.522+0000 7f4a79ffb640 1 -- 172.21.15.76:0/3535609956 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c009640 con 0x556745cf5380 2024-03-12T06:04:51.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.522+0000 7f4a79ffb640 1 -- 172.21.15.76:0/3535609956 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a8c01c030 con 0x556745cf5380 2024-03-12T06:04:51.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.522+0000 7f4a79ffb640 1 -- 172.21.15.76:0/3535609956 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c012890 con 0x556745cf5380 2024-03-12T06:04:51.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.522+0000 7f4a9ce64740 1 -- 172.21.15.76:0/3535609956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5380 msgr2=0x556745cfdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.522+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/3535609956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5380 0x556745cfdf40 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00a010 tx=0x7f4a8c014c20 comp rx=0 tx=0).stop 2024-03-12T06:04:51.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.522+0000 7f4a9ce64740 1 -- 172.21.15.76:0/3535609956 shutdown_connections 2024-03-12T06:04:51.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.522+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/3535609956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cfe8b0 0x556745cf8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.522+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/3535609956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cf5d30 0x556745cf8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.522+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/3535609956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cf5380 0x556745cfdf40 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.522+0000 7f4a9ce64740 1 -- 172.21.15.76:0/3535609956 >> 172.21.15.76:0/3535609956 conn(0x556745cef840 msgr2=0x556745d15bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:51.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.522+0000 7f4a9ce64740 1 -- 172.21.15.76:0/3535609956 shutdown_connections 2024-03-12T06:04:51.533 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.522+0000 7f4a9ce64740 1 -- 172.21.15.76:0/3535609956 wait complete. 2024-03-12T06:04:51.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.523+0000 7f4a9ce64740 1 -- start start 2024-03-12T06:04:51.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.523+0000 7f4a9ce64740 1 Processor -- start 2024-03-12T06:04:51.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.523+0000 7f4a9ce64740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cf5380 0x556745d050b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.523+0000 7f4a9ce64740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cf5d30 0x556745d055f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.523+0000 7f4a9ce64740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cfe8b0 0x556745d629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.523+0000 7f4a9ce64740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556745d62f00 con 0x556745cfe8b0 2024-03-12T06:04:51.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.523+0000 7f4a9ce64740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x556745d63070 con 0x556745cf5d30 2024-03-12T06:04:51.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.523+0000 7f4a9ce64740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556745d63230 con 0x556745cf5380 2024-03-12T06:04:51.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.523+0000 7f4a9ce64740 10 client.0 _ll_register_callbacks cb 0x7f4a88009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:51.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.523+0000 7f4a98825640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cfe8b0 0x556745d629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.523+0000 7f4a98825640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cfe8b0 0x556745d629c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46824/0 (socket says 172.21.15.76:46824) 2024-03-12T06:04:51.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.523+0000 7f4a98825640 1 -- 172.21.15.76:0/1487199612 learned_addr learned my addr 172.21.15.76:0/1487199612 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:51.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.523+0000 7f4a937fe640 1 --2- 172.21.15.76:0/1487199612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cf5d30 0x556745d055f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a98825640 1 --2- 172.21.15.76:0/1487199612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cf5380 0x556745d050b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a98825640 1 -- 172.21.15.76:0/1487199612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cf5380 msgr2=0x556745d050b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a98825640 1 --2- 172.21.15.76:0/1487199612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cf5380 0x556745d050b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a98825640 1 -- 172.21.15.76:0/1487199612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cf5d30 msgr2=0x556745d055f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a98825640 1 --2- 172.21.15.76:0/1487199612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cf5d30 0x556745d055f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a98825640 1 -- 172.21.15.76:0/1487199612 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556745c5b320 con 0x556745cfe8b0 2024-03-12T06:04:51.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a98825640 1 --2- 172.21.15.76:0/1487199612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cfe8b0 0x556745d629c0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x556745d06080 tx=0x7f4a8c012530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0d2f2904f6065 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:51.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a737fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:51.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a737fe640 1 -- 172.21.15.76:0/1487199612 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c03a070 con 0x556745cfe8b0 2024-03-12T06:04:51.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a737fe640 1 -- 172.21.15.76:0/1487199612 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f4a8c009ae0 con 0x556745cfe8b0 2024-03-12T06:04:51.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1487199612 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556745c82900 con 0x556745cfe8b0 2024-03-12T06:04:51.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a737fe640 1 -- 172.21.15.76:0/1487199612 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a8c012820 con 0x556745cfe8b0 2024-03-12T06:04:51.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a9ce64740 10 client.17556 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:51.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1487199612 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556745c1d910 con 0x556745cfe8b0 2024-03-12T06:04:51.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a737fe640 1 -- 172.21.15.76:0/1487199612 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x556745c82900 con 0x556745cfe8b0 2024-03-12T06:04:51.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a737fe640 1 -- 172.21.15.76:0/1487199612 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4a8c03d020 con 0x556745cfe8b0 2024-03-12T06:04:51.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a9ce64740 10 client.17556 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:51.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.524+0000 7f4a9ce64740 10 client.17556 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:51.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.525+0000 7f4a9ce64740 20 client.17556 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:51.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.525+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/1487199612 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556745d6b810 0x556745d6dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.537 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.525+0000 7f4a9ce64740 4 client.17556 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T06:04:51.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.525+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1487199612 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55674595c860 con 0x556745d6b810 2024-03-12T06:04:51.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.525+0000 7f4a937fe640 1 --2- 172.21.15.76:0/1487199612 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556745d6b810 0x556745d6dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.525+0000 7f4a937fe640 1 --2- 172.21.15.76:0/1487199612 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556745d6b810 0x556745d6dc00 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4be5ec8298b8afe server_cookie=b96b56cb0673c310 in_seq=0 out_seq=0 2024-03-12T06:04:51.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.525+0000 7f4a737fe640 10 client.17556 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:51.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.526+0000 7f4a737fe640 1 -- 172.21.15.76:0/1487199612 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55674595c860 con 0x556745d6b810 2024-03-12T06:04:51.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.526+0000 7f4a737fe640 10 client.17556 handle_command_reply: tid=0 2024-03-12T06:04:51.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.526+0000 7f4a9ce64740 1 client.17556 shutdown 2024-03-12T06:04:51.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.526+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1487199612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cfe8b0 msgr2=0x556745d629c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.526+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/1487199612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cfe8b0 0x556745d629c0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x556745d06080 tx=0x7f4a8c012530 comp rx=0 tx=0).stop 2024-03-12T06:04:51.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.526+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1487199612 shutdown_connections 2024-03-12T06:04:51.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.526+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/1487199612 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556745d6b810 0x556745d6dc00 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.526+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/1487199612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556745cfe8b0 0x556745d629c0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.526+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/1487199612 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556745cf5d30 0x556745d055f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.526+0000 7f4a737fe640 0 client.17556 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:51.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.526+0000 7f4a9ce64740 1 --2- 172.21.15.76:0/1487199612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556745cf5380 0x556745d050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1487199612 >> 172.21.15.76:0/1487199612 conn(0x556745cef840 msgr2=0x556745cee1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:51.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1487199612 shutdown_connections 2024-03-12T06:04:51.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9ce64740 1 -- 172.21.15.76:0/1487199612 wait complete. 2024-03-12T06:04:51.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9ce64740 20 client.17556 trim_cache size 0 max 16384 2024-03-12T06:04:51.541 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:04:51.541 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T06:04:51.541 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "dedeb2a0-6233-4b1a-85c9-ee93de2bf48f", 2024-03-12T06:04:51.542 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T06:04:51.542 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:04:51.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9a288640 1 -- 172.21.15.76:0/2473396194 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4a7c0a1ba0 msgr2=0x7f4a7c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9a288640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4a7c0a1ba0 0x7f4a7c0a4060 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4a84099eb0 tx=0x7f4a8408f5f0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9a288640 1 -- 172.21.15.76:0/2473396194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a94069ba0 msgr2=0x7f4a9419d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9a288640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a94069ba0 0x7f4a9419d570 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f4a8400d770 tx=0x7f4a8400dc30 comp rx=0 tx=0).stop 2024-03-12T06:04:51.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9a288640 1 -- 172.21.15.76:0/2473396194 shutdown_connections 2024-03-12T06:04:51.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9a288640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f4a7c0a1ba0 0x7f4a7c0a4060 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9a288640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a94197630 0x7f4a94197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9a288640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a94106460 0x7f4a9419dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9a288640 1 --2- 172.21.15.76:0/2473396194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a94069ba0 0x7f4a9419d570 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9a288640 1 -- 172.21.15.76:0/2473396194 >> 172.21.15.76:0/2473396194 conn(0x7f4a94077200 msgr2=0x7f4a940fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:51.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9a288640 1 -- 172.21.15.76:0/2473396194 shutdown_connections 2024-03-12T06:04:51.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.527+0000 7f4a9a288640 1 -- 172.21.15.76:0/2473396194 wait complete. 2024-03-12T06:04:51.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:51 smithi076 ceph-mon[32095]: pgmap v281: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:51.545 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:04:51.546 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:04:51.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.665+0000 7ff020edf640 1 -- 172.21.15.76:0/695695786 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff01c105330 msgr2=0x7ff01c10cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.665+0000 7ff020edf640 1 --2- 172.21.15.76:0/695695786 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff01c105330 0x7ff01c10cf00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff014009cd0 tx=0x7ff01402c0f0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.665+0000 7ff020edf640 1 -- 172.21.15.76:0/695695786 shutdown_connections 2024-03-12T06:04:51.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.665+0000 7ff020edf640 1 --2- 172.21.15.76:0/695695786 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff01c105330 0x7ff01c10cf00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.665+0000 7ff020edf640 1 --2- 172.21.15.76:0/695695786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff01c100c10 0x7ff01c104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.671 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.665+0000 7ff020edf640 1 --2- 172.21.15.76:0/695695786 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff01c100260 0x7ff01c100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.665+0000 7ff020edf640 1 -- 172.21.15.76:0/695695786 >> 172.21.15.76:0/695695786 conn(0x7ff01c077180 msgr2=0x7ff01c077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:51.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.665+0000 7ff020edf640 1 -- 172.21.15.76:0/695695786 shutdown_connections 2024-03-12T06:04:51.672 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.665+0000 7ff020edf640 1 -- 172.21.15.76:0/695695786 wait complete. 2024-03-12T06:04:51.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.666+0000 7ff020edf640 1 Processor -- start 2024-03-12T06:04:51.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.666+0000 7ff020edf640 1 -- start start 2024-03-12T06:04:51.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff020edf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff01c100260 0x7ff01c195630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff020edf640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff01c105330 0x7ff01c195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff020edf640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff01c199ea0 0x7ff01c19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff020edf640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c19a890 con 0x7ff01c105330 2024-03-12T06:04:51.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff020edf640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c19aa00 con 0x7ff01c199ea0 2024-03-12T06:04:51.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff020edf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c19ab70 con 0x7ff01c100260 2024-03-12T06:04:51.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff019d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff01c105330 0x7ff01c195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff019d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff01c105330 0x7ff01c195b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46844/0 (socket says 172.21.15.76:46844) 2024-03-12T06:04:51.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff01ad9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff01c199ea0 0x7ff01c19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff019d9b640 1 -- 172.21.15.76:0/1567252875 learned_addr learned my addr 172.21.15.76:0/1567252875 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:51.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff01a59c640 1 --2- 172.21.15.76:0/1567252875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff01c100260 0x7ff01c195630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff01ad9d640 1 -- 172.21.15.76:0/1567252875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff01c100260 msgr2=0x7ff01c195630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff01ad9d640 1 --2- 172.21.15.76:0/1567252875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff01c100260 0x7ff01c195630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff01ad9d640 1 -- 172.21.15.76:0/1567252875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff01c105330 msgr2=0x7ff01c195b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff01ad9d640 1 --2- 172.21.15.76:0/1567252875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff01c105330 0x7ff01c195b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff01ad9d640 1 -- 172.21.15.76:0/1567252875 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c009490 con 0x7ff01c199ea0 2024-03-12T06:04:51.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.667+0000 7ff01ad9d640 1 --2- 172.21.15.76:0/1567252875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff01c199ea0 0x7ff01c19a350 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff014037040 tx=0x7ff0140325c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:51.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.668+0000 7ff00b7fe640 1 -- 172.21.15.76:0/1567252875 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0140328b0 con 0x7ff01c199ea0 2024-03-12T06:04:51.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.668+0000 7ff00b7fe640 1 -- 172.21.15.76:0/1567252875 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff014034de0 con 0x7ff01c199ea0 2024-03-12T06:04:51.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.668+0000 7ff020edf640 1 -- 172.21.15.76:0/1567252875 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff014009900 con 0x7ff01c199ea0 2024-03-12T06:04:51.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.668+0000 7ff00b7fe640 1 -- 172.21.15.76:0/1567252875 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0140046e0 con 0x7ff01c199ea0 2024-03-12T06:04:51.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.668+0000 7ff020edf640 1 -- 172.21.15.76:0/1567252875 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01c1a2440 con 0x7ff01c199ea0 2024-03-12T06:04:51.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.670+0000 7ff00b7fe640 1 -- 172.21.15.76:0/1567252875 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff01400a040 con 0x7ff01c199ea0 2024-03-12T06:04:51.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.670+0000 7ff023b90740 1 Processor -- start 2024-03-12T06:04:51.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.671+0000 7ff023b90740 1 -- start start 2024-03-12T06:04:51.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.671+0000 7ff023b90740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c6041622b0 0x55c604162690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.671+0000 7ff023b90740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c604162bd0 0x55c60416b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.671+0000 7ff023b90740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c60416b7b0 0x55c60416dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.671+0000 7ff023b90740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6040d1280 con 0x55c60416b7b0 2024-03-12T06:04:51.683 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.671+0000 7ff023b90740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6040ce7b0 con 0x55c604162bd0 2024-03-12T06:04:51.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.671+0000 7ff023b90740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6040c9f90 con 0x55c6041622b0 2024-03-12T06:04:51.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.671+0000 7ff01a59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c604162bd0 0x55c60416b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.671+0000 7ff01a59c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c604162bd0 0x55c60416b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:52980/0 (socket says 172.21.15.76:52980) 2024-03-12T06:04:51.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.671+0000 7ff01a59c640 1 -- 172.21.15.76:0/376832992 learned_addr learned my addr 172.21.15.76:0/376832992 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:51.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.671+0000 7ff019d9b640 1 --2- 172.21.15.76:0/376832992 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c60416b7b0 0x55c60416dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.671+0000 7ff00b7fe640 1 --2- 172.21.15.76:0/1567252875 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff0040a6070 0x7ff0040a8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.672+0000 7ff019d9b640 1 --2- 172.21.15.76:0/376832992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c6041622b0 0x55c604162690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.672+0000 7ff01a59c640 1 -- 172.21.15.76:0/376832992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c6041622b0 msgr2=0x55c604162690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.684 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.672+0000 7ff01a59c640 1 --2- 172.21.15.76:0/376832992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c6041622b0 0x55c604162690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.672+0000 7ff01a59c640 1 -- 172.21.15.76:0/376832992 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c60416b7b0 msgr2=0x55c60416dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.672+0000 7ff01a59c640 1 --2- 172.21.15.76:0/376832992 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c60416b7b0 0x55c60416dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.672+0000 7ff01a59c640 1 -- 172.21.15.76:0/376832992 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c604088570 con 0x55c604162bd0 2024-03-12T06:04:51.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.672+0000 7ff01ad9d640 1 --2- 172.21.15.76:0/1567252875 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff0040a6070 0x7ff0040a8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.672+0000 7ff00b7fe640 1 -- 172.21.15.76:0/1567252875 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7ff0140f00a0 con 0x7ff01c199ea0 2024-03-12T06:04:51.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.672+0000 7ff01a59c640 1 --2- 172.21.15.76:0/376832992 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c604162bd0 0x55c60416b270 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff00c009e80 tx=0x7ff00c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42c56e6cd494bbf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:51.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.672+0000 7ff008ff9640 1 -- 172.21.15.76:0/376832992 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c021410 con 0x55c604162bd0 2024-03-12T06:04:51.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.672+0000 7ff008ff9640 1 -- 172.21.15.76:0/376832992 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff00c021a50 con 0x55c604162bd0 2024-03-12T06:04:51.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.672+0000 7ff008ff9640 1 -- 172.21.15.76:0/376832992 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c020a70 con 0x55c604162bd0 2024-03-12T06:04:51.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.673+0000 7ff023b90740 1 -- 172.21.15.76:0/376832992 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c604162bd0 msgr2=0x55c60416b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.673+0000 7ff023b90740 1 --2- 172.21.15.76:0/376832992 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c604162bd0 0x55c60416b270 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff00c009e80 tx=0x7ff00c00f790 comp rx=0 tx=0).stop 2024-03-12T06:04:51.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.673+0000 7ff01ad9d640 1 --2- 172.21.15.76:0/1567252875 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff0040a6070 0x7ff0040a8530 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff0140eddf0 tx=0x7ff0140b24b0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:51.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.673+0000 7ff023b90740 1 -- 172.21.15.76:0/376832992 shutdown_connections 2024-03-12T06:04:51.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.673+0000 7ff023b90740 1 --2- 172.21.15.76:0/376832992 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c60416b7b0 0x55c60416dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.673+0000 7ff023b90740 1 --2- 172.21.15.76:0/376832992 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c604162bd0 0x55c60416b270 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.673+0000 7ff023b90740 1 --2- 172.21.15.76:0/376832992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c6041622b0 0x55c604162690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.673+0000 7ff023b90740 1 -- 172.21.15.76:0/376832992 >> 172.21.15.76:0/376832992 conn(0x55c60415c990 msgr2=0x55c60415cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:51.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.673+0000 7ff023b90740 1 -- 172.21.15.76:0/376832992 shutdown_connections 2024-03-12T06:04:51.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.673+0000 7ff023b90740 1 -- 172.21.15.76:0/376832992 wait complete. 2024-03-12T06:04:51.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.674+0000 7ff023b90740 1 -- start start 2024-03-12T06:04:51.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.674+0000 7ff023b90740 1 Processor -- start 2024-03-12T06:04:51.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.674+0000 7ff023b90740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6041622b0 0x55c60416b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.674+0000 7ff023b90740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c604162bd0 0x55c6041685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.674+0000 7ff023b90740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 0x55c604168d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.674+0000 7ff023b90740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c603d9d430 con 0x55c604162bd0 2024-03-12T06:04:51.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.674+0000 7ff023b90740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c603d59e00 con 0x55c6041622b0 2024-03-12T06:04:51.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.674+0000 7ff023b90740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c60404efd0 con 0x55c60416b7b0 2024-03-12T06:04:51.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff023b90740 10 client.0 _ll_register_callbacks cb 0x55c6040a4fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:51.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff019d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6041622b0 0x55c60416b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff019d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6041622b0 0x55c60416b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:52994/0 (socket says 172.21.15.76:52994) 2024-03-12T06:04:51.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff019d9b640 1 -- 172.21.15.76:0/1670366815 learned_addr learned my addr 172.21.15.76:0/1670366815 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:51.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff019d9b640 1 --2- 172.21.15.76:0/1670366815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 0x55c604168d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff01a59c640 1 --2- 172.21.15.76:0/1670366815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c604162bd0 0x55c6041685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff019d9b640 1 -- 172.21.15.76:0/1670366815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 msgr2=0x55c604168d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff019d9b640 1 --2- 172.21.15.76:0/1670366815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 0x55c604168d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff019d9b640 1 -- 172.21.15.76:0/1670366815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c604162bd0 msgr2=0x55c6041685f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff019d9b640 1 --2- 172.21.15.76:0/1670366815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c604162bd0 0x55c6041685f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff019d9b640 1 -- 172.21.15.76:0/1670366815 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00c009ae0 con 0x55c6041622b0 2024-03-12T06:04:51.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff019d9b640 1 --2- 172.21.15.76:0/1670366815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6041622b0 0x55c60416b260 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff010010e40 tx=0x7ff010004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad89828e0791263e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:51.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff001ffb640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T06:04:51.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff001ffb640 1 -- 172.21.15.76:0/1670366815 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff010005020 con 0x55c6041622b0 2024-03-12T06:04:51.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff001ffb640 1 -- 172.21.15.76:0/1670366815 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7ff010005180 con 0x55c6041622b0 2024-03-12T06:04:51.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.675+0000 7ff023b90740 1 -- 172.21.15.76:0/1670366815 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c603dc97e0 con 0x55c6041622b0 2024-03-12T06:04:51.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.676+0000 7ff001ffb640 1 -- 172.21.15.76:0/1670366815 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff010005340 con 0x55c6041622b0 2024-03-12T06:04:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.676+0000 7ff023b90740 10 client.26570 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.676+0000 7ff023b90740 1 -- 172.21.15.76:0/1670366815 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c6040c8610 con 0x55c6041622b0 2024-03-12T06:04:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.676+0000 7ff001ffb640 1 -- 172.21.15.76:0/1670366815 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c603dc97e0 con 0x55c6041622b0 2024-03-12T06:04:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.676+0000 7ff001ffb640 1 -- 172.21.15.76:0/1670366815 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff01000e5c0 con 0x55c6041622b0 2024-03-12T06:04:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.676+0000 7ff023b90740 10 client.26570 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.676+0000 7ff023b90740 10 client.26570 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.676+0000 7ff023b90740 20 client.26570 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.676+0000 7ff023b90740 1 --2- 172.21.15.76:0/1670366815 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c60417b960 0x55c60417dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.676+0000 7ff023b90740 4 client.26570 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:04:51.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.676+0000 7ff023b90740 1 -- 172.21.15.76:0/1670366815 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c603e06720 con 0x55c60417b960 2024-03-12T06:04:51.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.676+0000 7ff01a59c640 1 --2- 172.21.15.76:0/1670366815 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c60417b960 0x55c60417dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.677+0000 7ff01a59c640 1 --2- 172.21.15.76:0/1670366815 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c60417b960 0x55c60417dd50 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c57e22d7a24db3dc server_cookie=5ff81924b2969cbc in_seq=0 out_seq=0 2024-03-12T06:04:51.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.677+0000 7ff001ffb640 10 client.26570 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:51.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.677+0000 7ff001ffb640 1 -- 172.21.15.76:0/1670366815 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c603e06720 con 0x55c60417b960 2024-03-12T06:04:51.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.677+0000 7ff001ffb640 10 client.26570 handle_command_reply: tid=0 2024-03-12T06:04:51.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.677+0000 7ff023b90740 1 client.26570 shutdown 2024-03-12T06:04:51.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.677+0000 7ff023b90740 1 -- 172.21.15.76:0/1670366815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6041622b0 msgr2=0x55c60416b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.677+0000 7ff023b90740 1 --2- 172.21.15.76:0/1670366815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6041622b0 0x55c60416b260 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff010010e40 tx=0x7ff010004730 comp rx=0 tx=0).stop 2024-03-12T06:04:51.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.677+0000 7ff023b90740 1 -- 172.21.15.76:0/1670366815 shutdown_connections 2024-03-12T06:04:51.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.677+0000 7ff023b90740 1 --2- 172.21.15.76:0/1670366815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 0x55c604168d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.677+0000 7ff023b90740 1 --2- 172.21.15.76:0/1670366815 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c60417b960 0x55c60417dd50 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.678+0000 7ff001ffb640 0 client.26570 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:51.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.677+0000 7ff023b90740 1 --2- 172.21.15.76:0/1670366815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c604162bd0 0x55c6041685f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.678+0000 7ff023b90740 1 --2- 172.21.15.76:0/1670366815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6041622b0 0x55c60416b260 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.678+0000 7ff023b90740 1 -- 172.21.15.76:0/1670366815 >> 172.21.15.76:0/1670366815 conn(0x55c60415c990 msgr2=0x55c60415e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:51.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.678+0000 7ff023b90740 1 -- 172.21.15.76:0/1670366815 shutdown_connections 2024-03-12T06:04:51.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.678+0000 7ff023b90740 1 -- 172.21.15.76:0/1670366815 wait complete. 2024-03-12T06:04:51.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.678+0000 7ff023b90740 20 client.26570 trim_cache size 0 max 16384 2024-03-12T06:04:51.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:51 smithi082 ceph-mon[36382]: pgmap v281: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:51.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.697+0000 7ff023b90740 1 Processor -- start 2024-03-12T06:04:51.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.697+0000 7ff023b90740 1 -- start start 2024-03-12T06:04:51.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.697+0000 7ff023b90740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6041622b0 0x55c604168d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.697+0000 7ff023b90740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c604162bd0 0x55c604172660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.697+0000 7ff023b90740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 0x55c604172ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.697+0000 7ff023b90740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6041580f0 con 0x55c604162bd0 2024-03-12T06:04:51.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.697+0000 7ff023b90740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6041613a0 con 0x55c6041622b0 2024-03-12T06:04:51.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.697+0000 7ff023b90740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c604177570 con 0x55c60416b7b0 2024-03-12T06:04:51.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.697+0000 7ff01a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c604162bd0 0x55c604172660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.698+0000 7ff019d9b640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6041622b0 0x55c604168d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.698+0000 7ff01a59c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c604162bd0 0x55c604172660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46872/0 (socket says 172.21.15.76:46872) 2024-03-12T06:04:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.698+0000 7ff01a59c640 1 -- 172.21.15.76:0/3504852941 learned_addr learned my addr 172.21.15.76:0/3504852941 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.698+0000 7ff019d9b640 1 --2- 172.21.15.76:0/3504852941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 0x55c604172ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.698+0000 7ff019d9b640 1 -- 172.21.15.76:0/3504852941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 msgr2=0x55c604172ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.698+0000 7ff019d9b640 1 --2- 172.21.15.76:0/3504852941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 0x55c604172ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.698+0000 7ff019d9b640 1 -- 172.21.15.76:0/3504852941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c604162bd0 msgr2=0x55c604172660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.698+0000 7ff019d9b640 1 --2- 172.21.15.76:0/3504852941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c604162bd0 0x55c604172660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.698+0000 7ff019d9b640 1 -- 172.21.15.76:0/3504852941 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c604064ac0 con 0x55c6041622b0 2024-03-12T06:04:51.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.698+0000 7ff01a59c640 1 --2- 172.21.15.76:0/3504852941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c604162bd0 0x55c604172660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:04:51.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.698+0000 7ff019d9b640 1 --2- 172.21.15.76:0/3504852941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6041622b0 0x55c604168d10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff010009c70 tx=0x7ff01000ff90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c6e110f8a8d4263 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:51.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.698+0000 7ff003fff640 1 -- 172.21.15.76:0/3504852941 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff010009e20 con 0x55c6041622b0 2024-03-12T06:04:51.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.698+0000 7ff003fff640 1 -- 172.21.15.76:0/3504852941 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff010005500 con 0x55c6041622b0 2024-03-12T06:04:51.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.698+0000 7ff003fff640 1 -- 172.21.15.76:0/3504852941 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff010004a00 con 0x55c6041622b0 2024-03-12T06:04:51.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.699+0000 7ff023b90740 1 -- 172.21.15.76:0/3504852941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6041622b0 msgr2=0x55c604168d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.699+0000 7ff023b90740 1 --2- 172.21.15.76:0/3504852941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6041622b0 0x55c604168d10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff010009c70 tx=0x7ff01000ff90 comp rx=0 tx=0).stop 2024-03-12T06:04:51.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.699+0000 7ff023b90740 1 -- 172.21.15.76:0/3504852941 shutdown_connections 2024-03-12T06:04:51.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.699+0000 7ff023b90740 1 --2- 172.21.15.76:0/3504852941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 0x55c604172ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.699+0000 7ff023b90740 1 --2- 172.21.15.76:0/3504852941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c604162bd0 0x55c604172660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.699+0000 7ff023b90740 1 --2- 172.21.15.76:0/3504852941 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c6041622b0 0x55c604168d10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.699+0000 7ff023b90740 1 -- 172.21.15.76:0/3504852941 >> 172.21.15.76:0/3504852941 conn(0x55c60415c990 msgr2=0x55c6041638b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:51.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.699+0000 7ff023b90740 1 -- 172.21.15.76:0/3504852941 shutdown_connections 2024-03-12T06:04:51.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.700+0000 7ff023b90740 1 -- 172.21.15.76:0/3504852941 wait complete. 2024-03-12T06:04:51.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.700+0000 7ff023b90740 1 -- start start 2024-03-12T06:04:51.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.700+0000 7ff023b90740 1 Processor -- start 2024-03-12T06:04:51.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.700+0000 7ff023b90740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c6041622b0 0x55c6041710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.700+0000 7ff023b90740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c604162bd0 0x55c6041715f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.700+0000 7ff023b90740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 0x55c604171b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.700+0000 7ff023b90740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6041d1e80 con 0x55c6041622b0 2024-03-12T06:04:51.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.700+0000 7ff023b90740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6041d1fc0 con 0x55c604162bd0 2024-03-12T06:04:51.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.700+0000 7ff023b90740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6041d2100 con 0x55c60416b7b0 2024-03-12T06:04:51.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.700+0000 7ff023b90740 10 client.0 _ll_register_callbacks cb 0x7ff00c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:04:51.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.700+0000 7ff019d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c6041622b0 0x55c6041710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.700+0000 7ff019d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c6041622b0 0x55c6041710b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:46880/0 (socket says 172.21.15.76:46880) 2024-03-12T06:04:51.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.700+0000 7ff019d9b640 1 -- 172.21.15.76:0/2822190399 learned_addr learned my addr 172.21.15.76:0/2822190399 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:04:51.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff01a59c640 1 --2- 172.21.15.76:0/2822190399 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c604162bd0 0x55c6041715f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff019d9b640 1 --2- 172.21.15.76:0/2822190399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 0x55c604171b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff019d9b640 1 -- 172.21.15.76:0/2822190399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 msgr2=0x55c604171b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff019d9b640 1 --2- 172.21.15.76:0/2822190399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 0x55c604171b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff019d9b640 1 -- 172.21.15.76:0/2822190399 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c604162bd0 msgr2=0x55c6041715f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:04:51.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff019d9b640 1 --2- 172.21.15.76:0/2822190399 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c604162bd0 0x55c6041715f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff019d9b640 1 -- 172.21.15.76:0/2822190399 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c604088570 con 0x55c6041622b0 2024-03-12T06:04:51.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff019d9b640 1 --2- 172.21.15.76:0/2822190399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c6041622b0 0x55c6041710b0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7ff010010c10 tx=0x7ff0100106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7835647391d7e76f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:04:51.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff001ffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:04:51.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff001ffb640 1 -- 172.21.15.76:0/2822190399 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01003f030 con 0x55c6041622b0 2024-03-12T06:04:51.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff001ffb640 1 -- 172.21.15.76:0/2822190399 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7ff010040070 con 0x55c6041622b0 2024-03-12T06:04:51.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff023b90740 1 -- 172.21.15.76:0/2822190399 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff00c021a50 con 0x55c6041622b0 2024-03-12T06:04:51.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff001ffb640 1 -- 172.21.15.76:0/2822190399 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff010005670 con 0x55c6041622b0 2024-03-12T06:04:51.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff023b90740 10 client.17574 fetch_fsmap learned FSMap version 28 2024-03-12T06:04:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff023b90740 1 -- 172.21.15.76:0/2822190399 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c6040c8610 con 0x55c6041622b0 2024-03-12T06:04:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.701+0000 7ff001ffb640 1 -- 172.21.15.76:0/2822190399 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff00c021a50 con 0x55c6041622b0 2024-03-12T06:04:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.702+0000 7ff001ffb640 1 -- 172.21.15.76:0/2822190399 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff010039410 con 0x55c6041622b0 2024-03-12T06:04:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.702+0000 7ff023b90740 10 client.17574 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:04:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.702+0000 7ff023b90740 10 client.17574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:04:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.702+0000 7ff023b90740 20 client.17574 populate_metadata read hostname 'smithi076' 2024-03-12T06:04:51.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.702+0000 7ff023b90740 1 --2- 172.21.15.76:0/2822190399 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c60417b960 0x55c6041d5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:04:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.702+0000 7ff023b90740 4 client.17574 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:04:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.702+0000 7ff023b90740 1 -- 172.21.15.76:0/2822190399 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c603e55890 con 0x55c60417b960 2024-03-12T06:04:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.702+0000 7ff01a59c640 1 --2- 172.21.15.76:0/2822190399 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c60417b960 0x55c6041d5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:04:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.702+0000 7ff01a59c640 1 --2- 172.21.15.76:0/2822190399 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c60417b960 0x55c6041d5640 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8df10fe363041f4 server_cookie=6d984f359e90fea0 in_seq=0 out_seq=0 2024-03-12T06:04:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.702+0000 7ff001ffb640 10 client.17574 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff001ffb640 1 -- 172.21.15.76:0/2822190399 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c603e55890 con 0x55c60417b960 2024-03-12T06:04:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff001ffb640 10 client.17574 handle_command_reply: tid=0 2024-03-12T06:04:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff023b90740 1 client.17574 shutdown 2024-03-12T06:04:51.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff023b90740 1 -- 172.21.15.76:0/2822190399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c6041622b0 msgr2=0x55c6041710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff023b90740 1 --2- 172.21.15.76:0/2822190399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c6041622b0 0x55c6041710b0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7ff010010c10 tx=0x7ff0100106b0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff023b90740 1 -- 172.21.15.76:0/2822190399 shutdown_connections 2024-03-12T06:04:51.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff023b90740 1 --2- 172.21.15.76:0/2822190399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c60416b7b0 0x55c604171b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff023b90740 1 --2- 172.21.15.76:0/2822190399 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55c604162bd0 0x55c6041715f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff023b90740 1 --2- 172.21.15.76:0/2822190399 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55c60417b960 0x55c6041d5640 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff023b90740 1 --2- 172.21.15.76:0/2822190399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55c6041622b0 0x55c6041710b0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff023b90740 1 -- 172.21.15.76:0/2822190399 >> 172.21.15.76:0/2822190399 conn(0x55c60415c990 msgr2=0x55c604166df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:51.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff001ffb640 0 client.17574 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:04:51.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff023b90740 1 -- 172.21.15.76:0/2822190399 shutdown_connections 2024-03-12T06:04:51.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff023b90740 1 -- 172.21.15.76:0/2822190399 wait complete. 2024-03-12T06:04:51.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.703+0000 7ff023b90740 20 client.17574 trim_cache size 0 max 16384 2024-03-12T06:04:51.718 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:04:51.719 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T06:04:51.719 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T06:04:51.719 INFO:teuthology.orchestra.run.smithi076.stdout: "dedeb2a0-6233-4b1a-85c9-ee93de2bf48f": { 2024-03-12T06:04:51.719 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T06:04:51.719 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "dedeb2a0-6233-4b1a-85c9-ee93de2bf48f", 2024-03-12T06:04:51.719 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T06:04:51.719 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:04:51.720 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:04:51.720 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:04:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.704+0000 7ff020edf640 1 -- 172.21.15.76:0/1567252875 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff0040a6070 msgr2=0x7ff0040a8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.704+0000 7ff020edf640 1 --2- 172.21.15.76:0/1567252875 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff0040a6070 0x7ff0040a8530 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff0140eddf0 tx=0x7ff0140b24b0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.704+0000 7ff020edf640 1 -- 172.21.15.76:0/1567252875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff01c199ea0 msgr2=0x7ff01c19a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:04:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.704+0000 7ff020edf640 1 --2- 172.21.15.76:0/1567252875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff01c199ea0 0x7ff01c19a350 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff014037040 tx=0x7ff0140325c0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.704+0000 7ff020edf640 1 -- 172.21.15.76:0/1567252875 shutdown_connections 2024-03-12T06:04:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.704+0000 7ff020edf640 1 --2- 172.21.15.76:0/1567252875 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff0040a6070 0x7ff0040a8530 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.704+0000 7ff020edf640 1 --2- 172.21.15.76:0/1567252875 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff01c199ea0 0x7ff01c19a350 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.704+0000 7ff020edf640 1 --2- 172.21.15.76:0/1567252875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff01c105330 0x7ff01c195b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.704+0000 7ff020edf640 1 --2- 172.21.15.76:0/1567252875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff01c100260 0x7ff01c195630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:04:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.704+0000 7ff020edf640 1 -- 172.21.15.76:0/1567252875 >> 172.21.15.76:0/1567252875 conn(0x7ff01c077180 msgr2=0x7ff01c0fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:04:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.704+0000 7ff020edf640 1 -- 172.21.15.76:0/1567252875 shutdown_connections 2024-03-12T06:04:51.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:04:51.704+0000 7ff020edf640 1 -- 172.21.15.76:0/1567252875 wait complete. 2024-03-12T06:04:51.725 INFO:tasks.cephfs.filesystem:scrub status for tag:dedeb2a0-6233-4b1a-85c9-ee93de2bf48f - {'path': '/', 'tag': 'dedeb2a0-6233-4b1a-85c9-ee93de2bf48f', 'options': 'recursive,force'} 2024-03-12T06:04:51.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:51 smithi088 ceph-mon[36585]: pgmap v281: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:04:52.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:52 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:04:52.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:52 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:04:52.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:52 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:52.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:52 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:52.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:52 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:04:52.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:52 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:04:52.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:52 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:52.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:52 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:52.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:52 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:04:52.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:52 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:04:52.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:52 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:52.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:52 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:53.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:53 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T06:04:53.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:53 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:53.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:53 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:04:53.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:53 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:53.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:53 smithi082 ceph-mon[36382]: pgmap v282: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:04:53.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:53 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T06:04:53.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:53 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:53.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:53 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:04:53.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:53 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:53.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:53 smithi088 ceph-mon[36585]: pgmap v282: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:04:53.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:53 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T06:04:53.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:53 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:53.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:53 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:04:53.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:53 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:04:53.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:53 smithi076 ceph-mon[32095]: pgmap v282: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:04:55.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:55 smithi082 ceph-mon[36382]: pgmap v283: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:55.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:55 smithi088 ceph-mon[36585]: pgmap v283: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:55.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:55 smithi076 ceph-mon[32095]: pgmap v283: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:57.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:57 smithi082 ceph-mon[36382]: pgmap v284: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:57.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:57 smithi088 ceph-mon[36585]: pgmap v284: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:57.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:57 smithi076 ceph-mon[32095]: pgmap v284: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:04:59.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:04:59 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:04:59.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:04:59 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:04:59.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:04:59 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:05:00.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:00 smithi082 ceph-mon[36382]: pgmap v285: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:05:00.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:00 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:05:00.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:00 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:05:00.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:00 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:05:00.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:00 smithi088 ceph-mon[36585]: pgmap v285: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:05:00.485 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:00 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:05:00.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:00 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:05:00.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:00 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:05:00.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:00 smithi076 ceph-mon[32095]: pgmap v285: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:05:00.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:00 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:05:00.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:00 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:05:00.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:00 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:05:01.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:01 smithi082 ceph-mon[36382]: pgmap v286: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:05:01.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:01 smithi088 ceph-mon[36585]: pgmap v286: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:05:01.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:01 smithi076 ceph-mon[32095]: pgmap v286: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:05:02.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:02 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:05:02.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:02 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:05:02.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:02 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:05:03.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:03 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:05:03.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:03 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:05:03.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:03 smithi082 ceph-mon[36382]: pgmap v287: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:05:03.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:03 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:05:03.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:03 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:05:03.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:03 smithi088 ceph-mon[36585]: pgmap v287: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:05:03.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:03 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:05:03.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:03 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:05:03.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:03 smithi076 ceph-mon[32095]: pgmap v287: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:05:05.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:05 smithi082 ceph-mon[36382]: pgmap v288: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:05:05.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:05 smithi088 ceph-mon[36585]: pgmap v288: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:05:05.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:05 smithi076 ceph-mon[32095]: pgmap v288: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:05:07.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:07 smithi082 ceph-mon[36382]: pgmap v289: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:05:07.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:07 smithi088 ceph-mon[36585]: pgmap v289: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:05:07.989 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:07 smithi076 ceph-mon[32095]: pgmap v289: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:05:09.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:09 smithi082 ceph-mon[36382]: pgmap v290: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:05:09.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:09 smithi088 ceph-mon[36585]: pgmap v290: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:05:09.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:09 smithi076 ceph-mon[32095]: pgmap v290: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:05:11.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:11 smithi082 ceph-mon[36382]: pgmap v291: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:05:11.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:11 smithi088 ceph-mon[36585]: pgmap v291: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:05:11.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:11 smithi076 ceph-mon[32095]: pgmap v291: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-12T06:05:13.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:13 smithi082 ceph-mon[36382]: pgmap v292: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:05:13.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:13 smithi088 ceph-mon[36585]: pgmap v292: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:05:13.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:13 smithi076 ceph-mon[32095]: pgmap v292: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-12T06:05:15.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:15 smithi082 ceph-mon[36382]: pgmap v293: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:05:15.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:15 smithi088 ceph-mon[36585]: pgmap v293: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:05:15.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:15 smithi076 ceph-mon[32095]: pgmap v293: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:05:17.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:17 smithi082 ceph-mon[36382]: pgmap v294: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:05:17.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:17 smithi088 ceph-mon[36585]: pgmap v294: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:05:17.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:17 smithi076 ceph-mon[32095]: pgmap v294: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-12T06:05:19.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:19 smithi076 ceph-mon[32095]: pgmap v295: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-12T06:05:19.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:19 smithi082 ceph-mon[36382]: pgmap v295: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-12T06:05:19.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:19 smithi088 ceph-mon[36585]: pgmap v295: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-12T06:05:21.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:21 smithi082 ceph-mon[36382]: pgmap v296: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:21.726 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:05:21.726 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:05:21.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:21 smithi088 ceph-mon[36585]: pgmap v296: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:21.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:21 smithi076 ceph-mon[32095]: pgmap v296: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:21.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.842+0000 7f150bb93640 1 -- 172.21.15.76:0/2318254876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1504100260 msgr2=0x7f1504100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.842+0000 7f150bb93640 1 --2- 172.21.15.76:0/2318254876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1504100260 0x7f1504100640 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f1500009cd0 tx=0x7f150002c0f0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.842+0000 7f150bb93640 1 -- 172.21.15.76:0/2318254876 shutdown_connections 2024-03-12T06:05:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.842+0000 7f150bb93640 1 --2- 172.21.15.76:0/2318254876 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1504105330 0x7f150410cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.842+0000 7f150bb93640 1 --2- 172.21.15.76:0/2318254876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1504100c10 0x7f1504104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.842+0000 7f150bb93640 1 --2- 172.21.15.76:0/2318254876 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1504100260 0x7f1504100640 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.842+0000 7f150bb93640 1 -- 172.21.15.76:0/2318254876 >> 172.21.15.76:0/2318254876 conn(0x7f1504077180 msgr2=0x7f1504077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.842+0000 7f150bb93640 1 -- 172.21.15.76:0/2318254876 shutdown_connections 2024-03-12T06:05:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.842+0000 7f150bb93640 1 -- 172.21.15.76:0/2318254876 wait complete. 2024-03-12T06:05:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150bb93640 1 Processor -- start 2024-03-12T06:05:21.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150bb93640 1 -- start start 2024-03-12T06:05:21.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150bb93640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1504100260 0x7f15041966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150bb93640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1504105330 0x7f1504196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150bb93640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f150419af70 0x7f150419b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150bb93640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150419b960 con 0x7f1504100260 2024-03-12T06:05:21.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150bb93640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150419bad0 con 0x7f150419af70 2024-03-12T06:05:21.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150bb93640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150419bc40 con 0x7f1504105330 2024-03-12T06:05:21.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150992f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1504100260 0x7f15041966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150992f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1504100260 0x7f15041966b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57738/0 (socket says 172.21.15.76:57738) 2024-03-12T06:05:21.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150992f640 1 -- 172.21.15.76:0/4292768613 learned_addr learned my addr 172.21.15.76:0/4292768613 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:21.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150912e640 1 --2- 172.21.15.76:0/4292768613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1504105330 0x7f1504196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150a130640 1 --2- 172.21.15.76:0/4292768613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f150419af70 0x7f150419b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150992f640 1 -- 172.21.15.76:0/4292768613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1504105330 msgr2=0x7f1504196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:21.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150992f640 1 --2- 172.21.15.76:0/4292768613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1504105330 0x7f1504196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150992f640 1 -- 172.21.15.76:0/4292768613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f150419af70 msgr2=0x7f150419b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:21.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150992f640 1 --2- 172.21.15.76:0/4292768613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f150419af70 0x7f150419b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.843+0000 7f150992f640 1 -- 172.21.15.76:0/4292768613 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1500009900 con 0x7f1504100260 2024-03-12T06:05:21.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.844+0000 7f150992f640 1 --2- 172.21.15.76:0/4292768613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1504100260 0x7f15041966b0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f1500004af0 tx=0x7f1500034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:21.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.844+0000 7f14faffd640 1 -- 172.21.15.76:0/4292768613 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500032ac0 con 0x7f1504100260 2024-03-12T06:05:21.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.844+0000 7f14faffd640 1 -- 172.21.15.76:0/4292768613 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1500003e50 con 0x7f1504100260 2024-03-12T06:05:21.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.844+0000 7f14faffd640 1 -- 172.21.15.76:0/4292768613 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500040450 con 0x7f1504100260 2024-03-12T06:05:21.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.844+0000 7f150bb93640 1 -- 172.21.15.76:0/4292768613 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15041a3220 con 0x7f1504100260 2024-03-12T06:05:21.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.844+0000 7f150bb93640 1 -- 172.21.15.76:0/4292768613 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15041a3640 con 0x7f1504100260 2024-03-12T06:05:21.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.846+0000 7f14faffd640 1 -- 172.21.15.76:0/4292768613 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f150000a040 con 0x7f1504100260 2024-03-12T06:05:21.859 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.847+0000 7f150e76c740 1 Processor -- start 2024-03-12T06:05:21.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.847+0000 7f150e76c740 1 -- start start 2024-03-12T06:05:21.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.847+0000 7f14faffd640 1 --2- 172.21.15.76:0/4292768613 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f14f00a1ba0 0x7f14f00a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.847+0000 7f150e76c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddb84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.847+0000 7f150e76c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddc3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.847+0000 7f150e76c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.847+0000 7f150e76c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad6dd27280 con 0x55ad6ddc39c0 2024-03-12T06:05:21.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.847+0000 7f150e76c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad6dd247b0 con 0x55ad6ddb8a70 2024-03-12T06:05:21.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.847+0000 7f150e76c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad6dd1ff90 con 0x55ad6ddb80c0 2024-03-12T06:05:21.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f150a130640 1 --2- 172.21.15.76:0/4292768613 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f14f00a1ba0 0x7f14f00a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f14faffd640 1 -- 172.21.15.76:0/4292768613 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f15000f04c0 con 0x7f1504100260 2024-03-12T06:05:21.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f150a130640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f150a130640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57742/0 (socket says 172.21.15.76:57742) 2024-03-12T06:05:21.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f150a130640 1 -- 172.21.15.76:0/3842172193 learned_addr learned my addr 172.21.15.76:0/3842172193 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:21.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f150912e640 1 --2- 172.21.15.76:0/3842172193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddc3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f150992f640 1 --2- 172.21.15.76:0/3842172193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddb84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.861 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f150a130640 1 -- 172.21.15.76:0/3842172193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 msgr2=0x55ad6ddb84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:21.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f150a130640 1 --2- 172.21.15.76:0/3842172193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddb84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f150a130640 1 -- 172.21.15.76:0/3842172193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddb8a70 msgr2=0x55ad6ddc3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:21.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f150a130640 1 --2- 172.21.15.76:0/3842172193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddc3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f150a130640 1 -- 172.21.15.76:0/3842172193 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad6dcde570 con 0x55ad6ddc39c0 2024-03-12T06:05:21.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f150992f640 1 --2- 172.21.15.76:0/3842172193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddb84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:05:21.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f150a130640 1 --2- 172.21.15.76:0/4292768613 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f14f00a1ba0 0x7f14f00a4060 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0128a0 tx=0x7f14fc00bab0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:21.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f150a130640 1 --2- 172.21.15.76:0/3842172193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc5db0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f14fc000b60 tx=0x7f14fc01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ba02cd9535bbfb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:21.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.848+0000 7f14effff640 1 -- 172.21.15.76:0/3842172193 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc010e00 con 0x55ad6ddc39c0 2024-03-12T06:05:21.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.849+0000 7f14effff640 1 -- 172.21.15.76:0/3842172193 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14fc014e90 con 0x55ad6ddc39c0 2024-03-12T06:05:21.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.849+0000 7f14effff640 1 -- 172.21.15.76:0/3842172193 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc0208f0 con 0x55ad6ddc39c0 2024-03-12T06:05:21.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.849+0000 7f150e76c740 1 -- 172.21.15.76:0/3842172193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddc39c0 msgr2=0x55ad6ddc5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:21.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.849+0000 7f150e76c740 1 --2- 172.21.15.76:0/3842172193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc5db0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f14fc000b60 tx=0x7f14fc01c000 comp rx=0 tx=0).stop 2024-03-12T06:05:21.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.849+0000 7f150e76c740 1 -- 172.21.15.76:0/3842172193 shutdown_connections 2024-03-12T06:05:21.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.849+0000 7f150e76c740 1 --2- 172.21.15.76:0/3842172193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc5db0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.849+0000 7f150e76c740 1 --2- 172.21.15.76:0/3842172193 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddc3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.849+0000 7f150e76c740 1 --2- 172.21.15.76:0/3842172193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddb84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.849+0000 7f150e76c740 1 -- 172.21.15.76:0/3842172193 >> 172.21.15.76:0/3842172193 conn(0x55ad6ddb2990 msgr2=0x55ad6ddb2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:21.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.849+0000 7f150e76c740 1 -- 172.21.15.76:0/3842172193 shutdown_connections 2024-03-12T06:05:21.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.849+0000 7f150e76c740 1 -- 172.21.15.76:0/3842172193 wait complete. 2024-03-12T06:05:21.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.850+0000 7f150e76c740 1 -- start start 2024-03-12T06:05:21.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.850+0000 7f150e76c740 1 Processor -- start 2024-03-12T06:05:21.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.850+0000 7f150e76c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddc3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.850+0000 7f150e76c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddc07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.850+0000 7f150e76c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.850+0000 7f150e76c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad6d9f3430 con 0x55ad6ddc39c0 2024-03-12T06:05:21.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.850+0000 7f150e76c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad6d9afe00 con 0x55ad6ddb8a70 2024-03-12T06:05:21.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.850+0000 7f150e76c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad6dca4fd0 con 0x55ad6ddb80c0 2024-03-12T06:05:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.850+0000 7f150e76c740 10 client.0 _ll_register_callbacks cb 0x55ad6dcfafd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.850+0000 7f150a130640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.850+0000 7f150a130640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57754/0 (socket says 172.21.15.76:57754) 2024-03-12T06:05:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f150a130640 1 -- 172.21.15.76:0/3288495684 learned_addr learned my addr 172.21.15.76:0/3288495684 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.850+0000 7f150912e640 1 --2- 172.21.15.76:0/3288495684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddc07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f150992f640 1 --2- 172.21.15.76:0/3288495684 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddc3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f150a130640 1 -- 172.21.15.76:0/3288495684 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 msgr2=0x55ad6ddc3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f150a130640 1 --2- 172.21.15.76:0/3288495684 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddc3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f150a130640 1 -- 172.21.15.76:0/3288495684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddb8a70 msgr2=0x55ad6ddc07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f150a130640 1 --2- 172.21.15.76:0/3288495684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddc07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f150a130640 1 -- 172.21.15.76:0/3288495684 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14fc01a040 con 0x55ad6ddc39c0 2024-03-12T06:05:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f150992f640 1 --2- 172.21.15.76:0/3288495684 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddc3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f150a130640 1 --2- 172.21.15.76:0/3288495684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc0ed0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f14fc016010 tx=0x7f14fc020e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a615b98b203c7abb server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f14ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f14ed7fa640 1 -- 172.21.15.76:0/3288495684 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc00b600 con 0x55ad6ddc39c0 2024-03-12T06:05:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f14ed7fa640 1 -- 172.21.15.76:0/3288495684 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f14fc010e00 con 0x55ad6ddc39c0 2024-03-12T06:05:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f150e76c740 1 -- 172.21.15.76:0/3288495684 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ad6da1f7e0 con 0x55ad6ddc39c0 2024-03-12T06:05:21.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f14ed7fa640 1 -- 172.21.15.76:0/3288495684 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14fc020880 con 0x55ad6ddc39c0 2024-03-12T06:05:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f150e76c740 10 client.17586 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.851+0000 7f150e76c740 1 -- 172.21.15.76:0/3288495684 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ad6dd1e610 con 0x55ad6ddc39c0 2024-03-12T06:05:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.852+0000 7f14ed7fa640 1 -- 172.21.15.76:0/3288495684 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ad6da1f7e0 con 0x55ad6ddc39c0 2024-03-12T06:05:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.852+0000 7f14ed7fa640 1 -- 172.21.15.76:0/3288495684 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f14fc005480 con 0x55ad6ddc39c0 2024-03-12T06:05:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.852+0000 7f150e76c740 10 client.17586 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.852+0000 7f150e76c740 10 client.17586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:05:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.852+0000 7f150e76c740 20 client.17586 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.852+0000 7f150e76c740 1 --2- 172.21.15.76:0/3288495684 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ad6ddd3a90 0x55ad6ddd5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.852+0000 7f150e76c740 4 client.17586 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:05:21.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.852+0000 7f150e76c740 1 -- 172.21.15.76:0/3288495684 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad6da5c720 con 0x55ad6ddd3a90 2024-03-12T06:05:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.852+0000 7f150992f640 1 --2- 172.21.15.76:0/3288495684 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ad6ddd3a90 0x55ad6ddd5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.852+0000 7f14ed7fa640 10 client.17586 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.852+0000 7f150992f640 1 --2- 172.21.15.76:0/3288495684 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ad6ddd3a90 0x55ad6ddd5eb0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4298062a8b1ad2b1 server_cookie=c14bdf31b328f49 in_seq=0 out_seq=0 2024-03-12T06:05:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.853+0000 7f14ed7fa640 1 -- 172.21.15.76:0/3288495684 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ad6da5c720 con 0x55ad6ddd3a90 2024-03-12T06:05:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.853+0000 7f14ed7fa640 10 client.17586 handle_command_reply: tid=0 2024-03-12T06:05:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.853+0000 7f150e76c740 1 client.17586 shutdown 2024-03-12T06:05:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.853+0000 7f150e76c740 1 -- 172.21.15.76:0/3288495684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddc39c0 msgr2=0x55ad6ddc0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.853+0000 7f150e76c740 1 --2- 172.21.15.76:0/3288495684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc0ed0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f14fc016010 tx=0x7f14fc020e60 comp rx=0 tx=0).stop 2024-03-12T06:05:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.853+0000 7f150e76c740 1 -- 172.21.15.76:0/3288495684 shutdown_connections 2024-03-12T06:05:21.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.853+0000 7f150e76c740 1 --2- 172.21.15.76:0/3288495684 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ad6ddd3a90 0x55ad6ddd5eb0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.853+0000 7f150e76c740 1 --2- 172.21.15.76:0/3288495684 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc0ed0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.853+0000 7f150e76c740 1 --2- 172.21.15.76:0/3288495684 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddc07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.853+0000 7f14ed7fa640 0 client.17586 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.853+0000 7f150e76c740 1 --2- 172.21.15.76:0/3288495684 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddc3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.853+0000 7f150e76c740 1 -- 172.21.15.76:0/3288495684 >> 172.21.15.76:0/3288495684 conn(0x55ad6ddb2990 msgr2=0x55ad6ddb4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.853+0000 7f150e76c740 1 -- 172.21.15.76:0/3288495684 shutdown_connections 2024-03-12T06:05:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.853+0000 7f150e76c740 1 -- 172.21.15.76:0/3288495684 wait complete. 2024-03-12T06:05:21.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.854+0000 7f150e76c740 20 client.17586 trim_cache size 0 max 16384 2024-03-12T06:05:21.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150e76c740 1 Processor -- start 2024-03-12T06:05:21.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150e76c740 1 -- start start 2024-03-12T06:05:21.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150e76c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddc2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150e76c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddc3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150e76c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150e76c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad6ddae120 con 0x55ad6ddb8a70 2024-03-12T06:05:21.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150e76c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad6da5c750 con 0x55ad6ddc39c0 2024-03-12T06:05:21.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150e76c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad6ddb73a0 con 0x55ad6ddb80c0 2024-03-12T06:05:21.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150912e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddc3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150912e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddc3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57758/0 (socket says 172.21.15.76:57758) 2024-03-12T06:05:21.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150912e640 1 -- 172.21.15.76:0/3390834177 learned_addr learned my addr 172.21.15.76:0/3390834177 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:21.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150a130640 1 --2- 172.21.15.76:0/3390834177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150912e640 1 -- 172.21.15.76:0/3390834177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 msgr2=0x55ad6ddc2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:21.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150912e640 1 --2- 172.21.15.76:0/3390834177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddc2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150912e640 1 -- 172.21.15.76:0/3390834177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddc39c0 msgr2=0x55ad6ddc0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:21.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150912e640 1 --2- 172.21.15.76:0/3390834177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150912e640 1 -- 172.21.15.76:0/3390834177 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad6dcbaac0 con 0x55ad6ddb8a70 2024-03-12T06:05:21.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150a130640 1 --2- 172.21.15.76:0/3390834177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:21.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f150912e640 1 --2- 172.21.15.76:0/3390834177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddc3450 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f14f4005fd0 tx=0x7f14f400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7496c1305871695b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:21.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.869+0000 7f14ef7fe640 1 -- 172.21.15.76:0/3390834177 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f401f070 con 0x55ad6ddb8a70 2024-03-12T06:05:21.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.870+0000 7f14ef7fe640 1 -- 172.21.15.76:0/3390834177 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14f4017040 con 0x55ad6ddb8a70 2024-03-12T06:05:21.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.870+0000 7f14ef7fe640 1 -- 172.21.15.76:0/3390834177 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4016070 con 0x55ad6ddb8a70 2024-03-12T06:05:21.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.870+0000 7f150e76c740 1 -- 172.21.15.76:0/3390834177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddb8a70 msgr2=0x55ad6ddc3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:21.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.870+0000 7f150e76c740 1 --2- 172.21.15.76:0/3390834177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddc3450 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f14f4005fd0 tx=0x7f14f400be20 comp rx=0 tx=0).stop 2024-03-12T06:05:21.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.870+0000 7f150e76c740 1 -- 172.21.15.76:0/3390834177 shutdown_connections 2024-03-12T06:05:21.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.870+0000 7f150e76c740 1 --2- 172.21.15.76:0/3390834177 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.870+0000 7f150e76c740 1 --2- 172.21.15.76:0/3390834177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddc3450 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.870+0000 7f150e76c740 1 --2- 172.21.15.76:0/3390834177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddc2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.870+0000 7f150e76c740 1 -- 172.21.15.76:0/3390834177 >> 172.21.15.76:0/3390834177 conn(0x55ad6ddb2990 msgr2=0x55ad6ddbf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:21.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.870+0000 7f150e76c740 1 -- 172.21.15.76:0/3390834177 shutdown_connections 2024-03-12T06:05:21.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.870+0000 7f150e76c740 1 -- 172.21.15.76:0/3390834177 wait complete. 2024-03-12T06:05:21.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150e76c740 1 -- start start 2024-03-12T06:05:21.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150e76c740 1 Processor -- start 2024-03-12T06:05:21.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150e76c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddca850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.882 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150e76c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddcad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150e76c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150e76c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad6ddc8480 con 0x55ad6ddb80c0 2024-03-12T06:05:21.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150e76c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad6ddc85f0 con 0x55ad6ddc39c0 2024-03-12T06:05:21.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150e76c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad6ddc87b0 con 0x55ad6ddb8a70 2024-03-12T06:05:21.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150e76c740 10 client.0 _ll_register_callbacks cb 0x7f14fc0123f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:21.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150992f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150992f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddca850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57760/0 (socket says 172.21.15.76:57760) 2024-03-12T06:05:21.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150992f640 1 -- 172.21.15.76:0/1192281553 learned_addr learned my addr 172.21.15.76:0/1192281553 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:21.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150992f640 1 -- 172.21.15.76:0/1192281553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb8a70 msgr2=0x55ad6ddcad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150992f640 1 --2- 172.21.15.76:0/1192281553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddcad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150992f640 1 -- 172.21.15.76:0/1192281553 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddc39c0 msgr2=0x55ad6ddc7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150992f640 1 --2- 172.21.15.76:0/1192281553 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.871+0000 7f150992f640 1 -- 172.21.15.76:0/1192281553 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f400b670 con 0x55ad6ddb80c0 2024-03-12T06:05:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f150992f640 1 --2- 172.21.15.76:0/1192281553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddca850 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f15000edd80 tx=0x7f15000bce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da68a0caacaa9f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f14ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f14ed7fa640 1 -- 172.21.15.76:0/1192281553 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15000c1170 con 0x55ad6ddb80c0 2024-03-12T06:05:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f14ed7fa640 1 -- 172.21.15.76:0/1192281553 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f15000bdb90 con 0x55ad6ddb80c0 2024-03-12T06:05:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f150e76c740 1 -- 172.21.15.76:0/1192281553 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14fc014e90 con 0x55ad6ddb80c0 2024-03-12T06:05:21.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f14ed7fa640 1 -- 172.21.15.76:0/1192281553 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15000c1350 con 0x55ad6ddb80c0 2024-03-12T06:05:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f150e76c740 10 client.17592 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f150e76c740 1 -- 172.21.15.76:0/1192281553 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ad6dce06a0 con 0x55ad6ddb80c0 2024-03-12T06:05:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f14ed7fa640 1 -- 172.21.15.76:0/1192281553 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14fc014e90 con 0x55ad6ddb80c0 2024-03-12T06:05:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f14ed7fa640 1 -- 172.21.15.76:0/1192281553 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f15000d3070 con 0x55ad6ddb80c0 2024-03-12T06:05:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f150e76c740 10 client.17592 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f150e76c740 10 client.17592 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:05:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f150e76c740 20 client.17592 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f150e76c740 1 --2- 172.21.15.76:0/1192281553 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ad6de2c150 0x55ad6de2e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f150e76c740 4 client.17592 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:05:21.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.872+0000 7f150e76c740 1 -- 172.21.15.76:0/1192281553 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ad6daab890 con 0x55ad6de2c150 2024-03-12T06:05:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.873+0000 7f150912e640 1 --2- 172.21.15.76:0/1192281553 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ad6de2c150 0x55ad6de2e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.873+0000 7f14ed7fa640 10 client.17592 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.873+0000 7f150912e640 1 --2- 172.21.15.76:0/1192281553 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ad6de2c150 0x55ad6de2e570 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1256612d80f03fc server_cookie=4f6601d3a66d19ad in_seq=0 out_seq=0 2024-03-12T06:05:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.873+0000 7f14ed7fa640 1 -- 172.21.15.76:0/1192281553 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ad6daab890 con 0x55ad6de2c150 2024-03-12T06:05:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.873+0000 7f14ed7fa640 10 client.17592 handle_command_reply: tid=0 2024-03-12T06:05:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.873+0000 7f150e76c740 1 client.17592 shutdown 2024-03-12T06:05:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.873+0000 7f150e76c740 1 -- 172.21.15.76:0/1192281553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddb80c0 msgr2=0x55ad6ddca850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:21.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.873+0000 7f150e76c740 1 --2- 172.21.15.76:0/1192281553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddca850 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f15000edd80 tx=0x7f15000bce60 comp rx=0 tx=0).stop 2024-03-12T06:05:21.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150e76c740 1 -- 172.21.15.76:0/1192281553 shutdown_connections 2024-03-12T06:05:21.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150e76c740 1 --2- 172.21.15.76:0/1192281553 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ad6ddc39c0 0x55ad6ddc7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150e76c740 1 --2- 172.21.15.76:0/1192281553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad6ddb8a70 0x55ad6ddcad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150e76c740 1 --2- 172.21.15.76:0/1192281553 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55ad6de2c150 0x55ad6de2e570 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150e76c740 1 --2- 172.21.15.76:0/1192281553 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ad6ddb80c0 0x55ad6ddca850 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f14ed7fa640 0 client.17592 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:21.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150e76c740 1 -- 172.21.15.76:0/1192281553 >> 172.21.15.76:0/1192281553 conn(0x55ad6ddb2990 msgr2=0x55ad6ddc0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:21.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150e76c740 1 -- 172.21.15.76:0/1192281553 shutdown_connections 2024-03-12T06:05:21.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150e76c740 1 -- 172.21.15.76:0/1192281553 wait complete. 2024-03-12T06:05:21.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150e76c740 20 client.17592 trim_cache size 0 max 16384 2024-03-12T06:05:21.888 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:05:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T06:05:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T06:05:21.888 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:05:21.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150bb93640 1 -- 172.21.15.76:0/4292768613 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f14f00a1ba0 msgr2=0x7f14f00a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:21.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150bb93640 1 --2- 172.21.15.76:0/4292768613 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f14f00a1ba0 0x7f14f00a4060 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0128a0 tx=0x7f14fc00bab0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150bb93640 1 -- 172.21.15.76:0/4292768613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1504100260 msgr2=0x7f15041966b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:21.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150bb93640 1 --2- 172.21.15.76:0/4292768613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1504100260 0x7f15041966b0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f1500004af0 tx=0x7f1500034cc0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150bb93640 1 -- 172.21.15.76:0/4292768613 shutdown_connections 2024-03-12T06:05:21.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150bb93640 1 --2- 172.21.15.76:0/4292768613 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f14f00a1ba0 0x7f14f00a4060 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150bb93640 1 --2- 172.21.15.76:0/4292768613 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f150419af70 0x7f150419b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150bb93640 1 --2- 172.21.15.76:0/4292768613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1504105330 0x7f1504196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150bb93640 1 --2- 172.21.15.76:0/4292768613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1504100260 0x7f15041966b0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:21.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150bb93640 1 -- 172.21.15.76:0/4292768613 >> 172.21.15.76:0/4292768613 conn(0x7f1504077180 msgr2=0x7f1504106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:21.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150bb93640 1 -- 172.21.15.76:0/4292768613 shutdown_connections 2024-03-12T06:05:21.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:21.874+0000 7f150bb93640 1 -- 172.21.15.76:0/4292768613 wait complete. 2024-03-12T06:05:21.891 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T06:05:21.891 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T06:05:22.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.030+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/1759237816 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc59c100c50 msgr2=0x7fc59c104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.030+0000 7fc5a2e9d640 1 --2- 172.21.15.76:0/1759237816 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc59c100c50 0x7fc59c104c90 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc588009930 tx=0x7fc58802bc30 comp rx=0 tx=0).stop 2024-03-12T06:05:22.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.030+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/1759237816 shutdown_connections 2024-03-12T06:05:22.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.030+0000 7fc5a2e9d640 1 --2- 172.21.15.76:0/1759237816 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc59c105370 0x7fc59c10cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.030+0000 7fc5a2e9d640 1 --2- 172.21.15.76:0/1759237816 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc59c100c50 0x7fc59c104c90 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.030+0000 7fc5a2e9d640 1 --2- 172.21.15.76:0/1759237816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc59c1002a0 0x7fc59c100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.030+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/1759237816 >> 172.21.15.76:0/1759237816 conn(0x7fc59c077200 msgr2=0x7fc59c077610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.030+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/1759237816 shutdown_connections 2024-03-12T06:05:22.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.030+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/1759237816 wait complete. 2024-03-12T06:05:22.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc5a2e9d640 1 Processor -- start 2024-03-12T06:05:22.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc5a2e9d640 1 -- start start 2024-03-12T06:05:22.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc5a2e9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc59c1002a0 0x7fc59c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc5a2e9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc59c105370 0x7fc59c198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc5a2e9d640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc59c199100 0x7fc59c1a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc5a2e9d640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1a3d80 con 0x7fc59c105370 2024-03-12T06:05:22.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc5a2e9d640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1a3ef0 con 0x7fc59c199100 2024-03-12T06:05:22.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc5a2e9d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1a40b0 con 0x7fc59c1002a0 2024-03-12T06:05:22.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc59bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc59c105370 0x7fc59c198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc59bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc59c105370 0x7fc59c198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57772/0 (socket says 172.21.15.76:57772) 2024-03-12T06:05:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc59bfff640 1 -- 172.21.15.76:0/2620602984 learned_addr learned my addr 172.21.15.76:0/2620602984 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc5a0c39640 1 --2- 172.21.15.76:0/2620602984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc59c1002a0 0x7fc59c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc5a143a640 1 --2- 172.21.15.76:0/2620602984 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc59c199100 0x7fc59c1a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc59bfff640 1 -- 172.21.15.76:0/2620602984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc59c1002a0 msgr2=0x7fc59c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc59bfff640 1 --2- 172.21.15.76:0/2620602984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc59c1002a0 0x7fc59c198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc59bfff640 1 -- 172.21.15.76:0/2620602984 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc59c199100 msgr2=0x7fc59c1a36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc59bfff640 1 --2- 172.21.15.76:0/2620602984 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc59c199100 0x7fc59c1a36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.031+0000 7fc59bfff640 1 -- 172.21.15.76:0/2620602984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc588009560 con 0x7fc59c105370 2024-03-12T06:05:22.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.032+0000 7fc59bfff640 1 --2- 172.21.15.76:0/2620602984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc59c105370 0x7fc59c198b70 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fc588009230 tx=0x7fc588004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.032+0000 7fc599ffb640 1 -- 172.21.15.76:0/2620602984 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58802cbc0 con 0x7fc59c105370 2024-03-12T06:05:22.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.032+0000 7fc599ffb640 1 -- 172.21.15.76:0/2620602984 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5880052d0 con 0x7fc59c105370 2024-03-12T06:05:22.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.032+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/2620602984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59c1a4330 con 0x7fc59c105370 2024-03-12T06:05:22.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.032+0000 7fc599ffb640 1 -- 172.21.15.76:0/2620602984 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc588003e30 con 0x7fc59c105370 2024-03-12T06:05:22.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.032+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/2620602984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59c1a4840 con 0x7fc59c105370 2024-03-12T06:05:22.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.033+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/2620602984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59c068000 con 0x7fc59c105370 2024-03-12T06:05:22.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.034+0000 7fc599ffb640 1 -- 172.21.15.76:0/2620602984 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fc588005440 con 0x7fc59c105370 2024-03-12T06:05:22.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.034+0000 7fc599ffb640 1 --2- 172.21.15.76:0/2620602984 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc58c0a1ba0 0x7fc58c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.035+0000 7fc5a0c39640 1 --2- 172.21.15.76:0/2620602984 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc58c0a1ba0 0x7fc58c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.035+0000 7fc599ffb640 1 -- 172.21.15.76:0/2620602984 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fc5880efb60 con 0x7fc59c105370 2024-03-12T06:05:22.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.035+0000 7fc5a0c39640 1 --2- 172.21.15.76:0/2620602984 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc58c0a1ba0 0x7fc58c0a4060 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc5900058c0 tx=0x7fc59000a2d0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.039+0000 7fc599ffb640 1 -- 172.21.15.76:0/2620602984 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc5880b8b40 con 0x7fc59c105370 2024-03-12T06:05:22.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.251+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/2620602984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc59c1085a0 con 0x7fc59c105370 2024-03-12T06:05:22.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.252+0000 7fc599ffb640 1 -- 172.21.15.76:0/2620602984 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11328 (secure 0 0 0) 0x7fc5880bc200 con 0x7fc59c105370 2024-03-12T06:05:22.257 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:05:22.257 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T06:01:33.842001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T06:05:22.258 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 28 2024-03-12T06:05:22.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.254+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/2620602984 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc58c0a1ba0 msgr2=0x7fc58c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.254+0000 7fc5a2e9d640 1 --2- 172.21.15.76:0/2620602984 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc58c0a1ba0 0x7fc58c0a4060 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc5900058c0 tx=0x7fc59000a2d0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.254+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/2620602984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc59c105370 msgr2=0x7fc59c198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.254+0000 7fc5a2e9d640 1 --2- 172.21.15.76:0/2620602984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc59c105370 0x7fc59c198b70 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fc588009230 tx=0x7fc588004270 comp rx=0 tx=0).stop 2024-03-12T06:05:22.259 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.254+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/2620602984 shutdown_connections 2024-03-12T06:05:22.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.254+0000 7fc5a2e9d640 1 --2- 172.21.15.76:0/2620602984 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fc58c0a1ba0 0x7fc58c0a4060 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.254+0000 7fc5a2e9d640 1 --2- 172.21.15.76:0/2620602984 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc59c199100 0x7fc59c1a36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.254+0000 7fc5a2e9d640 1 --2- 172.21.15.76:0/2620602984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc59c105370 0x7fc59c198b70 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.254+0000 7fc5a2e9d640 1 --2- 172.21.15.76:0/2620602984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc59c1002a0 0x7fc59c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.254+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/2620602984 >> 172.21.15.76:0/2620602984 conn(0x7fc59c077200 msgr2=0x7fc59c106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.254+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/2620602984 shutdown_connections 2024-03-12T06:05:22.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.254+0000 7fc5a2e9d640 1 -- 172.21.15.76:0/2620602984 wait complete. 2024-03-12T06:05:22.270 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:05:22.270 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T06:05:22.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.371+0000 7fea5cba7640 1 -- 172.21.15.76:0/3179050869 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea58100290 msgr2=0x7fea58100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.371+0000 7fea5cba7640 1 --2- 172.21.15.76:0/3179050869 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea58100290 0x7fea58100670 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fea48009930 tx=0x7fea4802bc20 comp rx=0 tx=0).stop 2024-03-12T06:05:22.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.371+0000 7fea5cba7640 1 -- 172.21.15.76:0/3179050869 shutdown_connections 2024-03-12T06:05:22.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.371+0000 7fea5cba7640 1 --2- 172.21.15.76:0/3179050869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea58105360 0x7fea5810cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.371+0000 7fea5cba7640 1 --2- 172.21.15.76:0/3179050869 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fea58100c40 0x7fea58104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.371+0000 7fea5cba7640 1 --2- 172.21.15.76:0/3179050869 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea58100290 0x7fea58100670 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.371+0000 7fea5cba7640 1 -- 172.21.15.76:0/3179050869 >> 172.21.15.76:0/3179050869 conn(0x7fea58077230 msgr2=0x7fea58077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.371+0000 7fea5cba7640 1 -- 172.21.15.76:0/3179050869 shutdown_connections 2024-03-12T06:05:22.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.371+0000 7fea5cba7640 1 -- 172.21.15.76:0/3179050869 wait complete. 2024-03-12T06:05:22.379 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.371+0000 7fea5cba7640 1 Processor -- start 2024-03-12T06:05:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.371+0000 7fea5cba7640 1 -- start start 2024-03-12T06:05:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5cba7640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea58100290 0x7fea5819d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5cba7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea58105360 0x7fea5819d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5cba7640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fea58197620 0x7fea58197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5cba7640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea58198010 con 0x7fea58100290 2024-03-12T06:05:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5cba7640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea58198180 con 0x7fea58197620 2024-03-12T06:05:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5cba7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea58198340 con 0x7fea58105360 2024-03-12T06:05:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea58100290 0x7fea5819d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea58100290 0x7fea5819d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57784/0 (socket says 172.21.15.76:57784) 2024-03-12T06:05:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5659c640 1 -- 172.21.15.76:0/3937382980 learned_addr learned my addr 172.21.15.76:0/3937382980 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea55d9b640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea58105360 0x7fea5819d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea56d9d640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fea58197620 0x7fea58197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5659c640 1 -- 172.21.15.76:0/3937382980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea58105360 msgr2=0x7fea5819d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5659c640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea58105360 0x7fea5819d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5659c640 1 -- 172.21.15.76:0/3937382980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fea58197620 msgr2=0x7fea58197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5659c640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fea58197620 0x7fea58197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5659c640 1 -- 172.21.15.76:0/3937382980 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea48009560 con 0x7fea58100290 2024-03-12T06:05:22.381 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea56d9d640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fea58197620 0x7fea58197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea55d9b640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea58105360 0x7fea5819d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5659c640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea58100290 0x7fea5819d470 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fea48009230 tx=0x7fea48004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea477fe640 1 -- 172.21.15.76:0/3937382980 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea48004440 con 0x7fea58100290 2024-03-12T06:05:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea477fe640 1 -- 172.21.15.76:0/3937382980 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea480052d0 con 0x7fea58100290 2024-03-12T06:05:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5cba7640 1 -- 172.21.15.76:0/3937382980 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea581a4330 con 0x7fea58100290 2024-03-12T06:05:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea477fe640 1 -- 172.21.15.76:0/3937382980 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea48003d70 con 0x7fea58100290 2024-03-12T06:05:22.382 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.372+0000 7fea5cba7640 1 -- 172.21.15.76:0/3937382980 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea581a47e0 con 0x7fea58100290 2024-03-12T06:05:22.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.374+0000 7fea477fe640 1 -- 172.21.15.76:0/3937382980 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fea48005440 con 0x7fea58100290 2024-03-12T06:05:22.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea5f776740 1 Processor -- start 2024-03-12T06:05:22.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea5f776740 1 -- start start 2024-03-12T06:05:22.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea5f776740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e4c30 0x5595196e5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea5f776740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e55e0 0x5595196f2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea5f776740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196f2b60 0x5595196f4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea5f776740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595196a0960 con 0x5595196e55e0 2024-03-12T06:05:22.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea5f776740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559519654310 con 0x5595196e4c30 2024-03-12T06:05:22.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea5f776740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559519651840 con 0x5595196f2b60 2024-03-12T06:05:22.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea5659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e55e0 0x5595196f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea477fe640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fea400a1ba0 0x7fea400a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea5659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e55e0 0x5595196f2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57796/0 (socket says 172.21.15.76:57796) 2024-03-12T06:05:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea5659c640 1 -- 172.21.15.76:0/3194116049 learned_addr learned my addr 172.21.15.76:0/3194116049 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea56d9d640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fea400a1ba0 0x7fea400a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea55d9b640 1 --2- 172.21.15.76:0/3194116049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196f2b60 0x5595196f4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.375+0000 7fea56d9d640 1 --2- 172.21.15.76:0/3194116049 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e4c30 0x5595196e5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.376+0000 7fea477fe640 1 -- 172.21.15.76:0/3937382980 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fea480ef9b0 con 0x7fea58100290 2024-03-12T06:05:22.390 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.376+0000 7fea5659c640 1 -- 172.21.15.76:0/3194116049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196f2b60 msgr2=0x5595196f4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.376+0000 7fea5659c640 1 --2- 172.21.15.76:0/3194116049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196f2b60 0x5595196f4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.376+0000 7fea5659c640 1 -- 172.21.15.76:0/3194116049 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e4c30 msgr2=0x5595196e5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.376+0000 7fea5659c640 1 --2- 172.21.15.76:0/3194116049 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e4c30 0x5595196e5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.376+0000 7fea5659c640 1 -- 172.21.15.76:0/3194116049 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55951960f060 con 0x5595196e55e0 2024-03-12T06:05:22.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.376+0000 7fea55d9b640 1 --2- 172.21.15.76:0/3194116049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196f2b60 0x5595196f4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:22.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.376+0000 7fea56d9d640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fea400a1ba0 0x7fea400a4060 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fea50005c40 tx=0x7fea50009e20 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.376+0000 7fea5659c640 1 --2- 172.21.15.76:0/3194116049 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e55e0 0x5595196f2620 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fea480bcd50 tx=0x7fea48009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67162ff8a6f7b951 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.376+0000 7fea44ff9640 1 -- 172.21.15.76:0/3194116049 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea480bc050 con 0x5595196e55e0 2024-03-12T06:05:22.391 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.376+0000 7fea44ff9640 1 -- 172.21.15.76:0/3194116049 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea480bc6f0 con 0x5595196e55e0 2024-03-12T06:05:22.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.376+0000 7fea44ff9640 1 -- 172.21.15.76:0/3194116049 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea480baaa0 con 0x5595196e55e0 2024-03-12T06:05:22.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 -- 172.21.15.76:0/3194116049 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e55e0 msgr2=0x5595196f2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 --2- 172.21.15.76:0/3194116049 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e55e0 0x5595196f2620 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fea480bcd50 tx=0x7fea48009260 comp rx=0 tx=0).stop 2024-03-12T06:05:22.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 -- 172.21.15.76:0/3194116049 shutdown_connections 2024-03-12T06:05:22.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 --2- 172.21.15.76:0/3194116049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196f2b60 0x5595196f4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 --2- 172.21.15.76:0/3194116049 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e55e0 0x5595196f2620 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 --2- 172.21.15.76:0/3194116049 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e4c30 0x5595196e5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 -- 172.21.15.76:0/3194116049 >> 172.21.15.76:0/3194116049 conn(0x5595196df980 msgr2=0x5595196dfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 -- 172.21.15.76:0/3194116049 shutdown_connections 2024-03-12T06:05:22.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 -- 172.21.15.76:0/3194116049 wait complete. 2024-03-12T06:05:22.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 -- start start 2024-03-12T06:05:22.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 Processor -- start 2024-03-12T06:05:22.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e4c30 0x5595196efb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196e55e0 0x5595196f00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196f2b60 0x5595196f05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55951964d020 con 0x5595196e4c30 2024-03-12T06:05:22.393 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559519320430 con 0x5595196f2b60 2024-03-12T06:05:22.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.377+0000 7fea5f776740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595192dce00 con 0x5595196e55e0 2024-03-12T06:05:22.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea5f776740 10 client.0 _ll_register_callbacks cb 0x5595195f4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:22.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea5659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e4c30 0x5595196efb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea5659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e4c30 0x5595196efb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57804/0 (socket says 172.21.15.76:57804) 2024-03-12T06:05:22.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea5659c640 1 -- 172.21.15.76:0/570602892 learned_addr learned my addr 172.21.15.76:0/570602892 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea56d9d640 1 --2- 172.21.15.76:0/570602892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196f2b60 0x5595196f05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea55d9b640 1 --2- 172.21.15.76:0/570602892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196e55e0 0x5595196f00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea5659c640 1 -- 172.21.15.76:0/570602892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196e55e0 msgr2=0x5595196f00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea5659c640 1 --2- 172.21.15.76:0/570602892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196e55e0 0x5595196f00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea5659c640 1 -- 172.21.15.76:0/570602892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196f2b60 msgr2=0x5595196f05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea5659c640 1 --2- 172.21.15.76:0/570602892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196f2b60 0x5595196f05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea5659c640 1 -- 172.21.15.76:0/570602892 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea581a47e0 con 0x5595196e4c30 2024-03-12T06:05:22.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea5659c640 1 --2- 172.21.15.76:0/570602892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e4c30 0x5595196efb70 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fea480b90c0 tx=0x7fea480bc120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a599d35d960c92 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:22.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea3dffb640 1 -- 172.21.15.76:0/570602892 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea480bce70 con 0x5595196e4c30 2024-03-12T06:05:22.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea3dffb640 1 -- 172.21.15.76:0/570602892 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fea480c0840 con 0x5595196e4c30 2024-03-12T06:05:22.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea5f776740 1 -- 172.21.15.76:0/570602892 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55951934c7e0 con 0x5595196e4c30 2024-03-12T06:05:22.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea3dffb640 1 -- 172.21.15.76:0/570602892 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea480c4e90 con 0x5595196e4c30 2024-03-12T06:05:22.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea5f776740 10 client.17610 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:22.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea5f776740 1 -- 172.21.15.76:0/570602892 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55951964b6a0 con 0x5595196e4c30 2024-03-12T06:05:22.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.378+0000 7fea3dffb640 1 -- 172.21.15.76:0/570602892 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55951934c7e0 con 0x5595196e4c30 2024-03-12T06:05:22.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.379+0000 7fea3dffb640 1 -- 172.21.15.76:0/570602892 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fea4803c070 con 0x5595196e4c30 2024-03-12T06:05:22.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.379+0000 7fea5f776740 10 client.17610 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:22.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.379+0000 7fea5f776740 10 client.17610 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:05:22.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.379+0000 7fea5f776740 20 client.17610 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:22.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.379+0000 7fea5f776740 1 --2- 172.21.15.76:0/570602892 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559519702820 0x559519704c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.379+0000 7fea5f776740 4 client.17610 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:05:22.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.379+0000 7fea5f776740 1 -- 172.21.15.76:0/570602892 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559519389720 con 0x559519702820 2024-03-12T06:05:22.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.379+0000 7fea55d9b640 1 --2- 172.21.15.76:0/570602892 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559519702820 0x559519704c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.379+0000 7fea3dffb640 10 client.17610 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:05:22.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.379+0000 7fea55d9b640 1 --2- 172.21.15.76:0/570602892 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559519702820 0x559519704c10 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8e61aa65226f0c3 server_cookie=c3eca9413d08893f in_seq=0 out_seq=0 2024-03-12T06:05:22.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.380+0000 7fea3dffb640 1 -- 172.21.15.76:0/570602892 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559519389720 con 0x559519702820 2024-03-12T06:05:22.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.380+0000 7fea3dffb640 10 client.17610 handle_command_reply: tid=0 2024-03-12T06:05:22.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.380+0000 7fea5f776740 1 client.17610 shutdown 2024-03-12T06:05:22.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.380+0000 7fea5f776740 1 -- 172.21.15.76:0/570602892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e4c30 msgr2=0x5595196efb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.380+0000 7fea5f776740 1 --2- 172.21.15.76:0/570602892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e4c30 0x5595196efb70 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fea480b90c0 tx=0x7fea480bc120 comp rx=0 tx=0).stop 2024-03-12T06:05:22.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.380+0000 7fea5f776740 1 -- 172.21.15.76:0/570602892 shutdown_connections 2024-03-12T06:05:22.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.380+0000 7fea5f776740 1 --2- 172.21.15.76:0/570602892 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x559519702820 0x559519704c10 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.380+0000 7fea5f776740 1 --2- 172.21.15.76:0/570602892 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196f2b60 0x5595196f05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.380+0000 7fea5f776740 1 --2- 172.21.15.76:0/570602892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196e55e0 0x5595196f00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.380+0000 7fea3dffb640 0 client.17610 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:05:22.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.380+0000 7fea5f776740 1 --2- 172.21.15.76:0/570602892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e4c30 0x5595196efb70 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.380+0000 7fea5f776740 1 -- 172.21.15.76:0/570602892 >> 172.21.15.76:0/570602892 conn(0x5595196df980 msgr2=0x5595196e16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.380+0000 7fea5f776740 1 -- 172.21.15.76:0/570602892 shutdown_connections 2024-03-12T06:05:22.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.380+0000 7fea5f776740 1 -- 172.21.15.76:0/570602892 wait complete. 2024-03-12T06:05:22.399 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.381+0000 7fea5f776740 20 client.17610 trim_cache size 0 max 16384 2024-03-12T06:05:22.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.392+0000 7fea5f776740 1 Processor -- start 2024-03-12T06:05:22.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.392+0000 7fea5f776740 1 -- start start 2024-03-12T06:05:22.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.392+0000 7fea5f776740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e4c30 0x5595196f0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.392+0000 7fea5f776740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196e55e0 0x5595196e14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.392+0000 7fea5f776740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196f2b60 0x5595196f97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.392+0000 7fea5f776740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595195d2130 con 0x5595196f2b60 2024-03-12T06:05:22.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.392+0000 7fea5f776740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595196e4590 con 0x5595196e4c30 2024-03-12T06:05:22.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.392+0000 7fea5f776740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559519709d10 con 0x5595196e55e0 2024-03-12T06:05:22.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.392+0000 7fea56d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196f2b60 0x5595196f97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.392+0000 7fea56d9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196f2b60 0x5595196f97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57814/0 (socket says 172.21.15.76:57814) 2024-03-12T06:05:22.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.392+0000 7fea56d9d640 1 -- 172.21.15.76:0/2367733827 learned_addr learned my addr 172.21.15.76:0/2367733827 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.392+0000 7fea55d9b640 1 --2- 172.21.15.76:0/2367733827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196e55e0 0x5595196e14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.392+0000 7fea56d9d640 1 -- 172.21.15.76:0/2367733827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196e55e0 msgr2=0x5595196e14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.392+0000 7fea56d9d640 1 --2- 172.21.15.76:0/2367733827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196e55e0 0x5595196e14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.409 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea5659c640 1 --2- 172.21.15.76:0/2367733827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e4c30 0x5595196f0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea56d9d640 1 -- 172.21.15.76:0/2367733827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e4c30 msgr2=0x5595196f0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea56d9d640 1 --2- 172.21.15.76:0/2367733827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e4c30 0x5595196f0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea56d9d640 1 -- 172.21.15.76:0/2367733827 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595195e7c20 con 0x5595196f2b60 2024-03-12T06:05:22.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea56d9d640 1 --2- 172.21.15.76:0/2367733827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196f2b60 0x5595196f97e0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fea500098d0 tx=0x7fea50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da46384d0738c28 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea3ffff640 1 -- 172.21.15.76:0/2367733827 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50023040 con 0x5595196f2b60 2024-03-12T06:05:22.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea3ffff640 1 -- 172.21.15.76:0/2367733827 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea50012650 con 0x5595196f2b60 2024-03-12T06:05:22.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea3ffff640 1 -- 172.21.15.76:0/2367733827 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50029cc0 con 0x5595196f2b60 2024-03-12T06:05:22.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea5f776740 1 -- 172.21.15.76:0/2367733827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196f2b60 msgr2=0x5595196f97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea5f776740 1 --2- 172.21.15.76:0/2367733827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196f2b60 0x5595196f97e0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fea500098d0 tx=0x7fea50004a40 comp rx=0 tx=0).stop 2024-03-12T06:05:22.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea5f776740 1 -- 172.21.15.76:0/2367733827 shutdown_connections 2024-03-12T06:05:22.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea5f776740 1 --2- 172.21.15.76:0/2367733827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196f2b60 0x5595196f97e0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea5f776740 1 --2- 172.21.15.76:0/2367733827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196e55e0 0x5595196e14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea5f776740 1 --2- 172.21.15.76:0/2367733827 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e4c30 0x5595196f0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea5f776740 1 -- 172.21.15.76:0/2367733827 >> 172.21.15.76:0/2367733827 conn(0x5595196df980 msgr2=0x5595196e0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea5f776740 1 -- 172.21.15.76:0/2367733827 shutdown_connections 2024-03-12T06:05:22.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.393+0000 7fea5f776740 1 -- 172.21.15.76:0/2367733827 wait complete. 2024-03-12T06:05:22.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5f776740 1 -- start start 2024-03-12T06:05:22.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5f776740 1 Processor -- start 2024-03-12T06:05:22.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5f776740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e4c30 0x5595196f8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5f776740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e55e0 0x5595196f8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5f776740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196f2b60 0x559519758bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5f776740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595196f93a0 con 0x5595196e4c30 2024-03-12T06:05:22.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5f776740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x559519759130 con 0x5595196e55e0 2024-03-12T06:05:22.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5f776740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559519759270 con 0x5595196f2b60 2024-03-12T06:05:22.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5f776740 10 client.0 _ll_register_callbacks cb 0x7fea480bc4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:22.412 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e4c30 0x5595196f8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5659c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e4c30 0x5595196f8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57824/0 (socket says 172.21.15.76:57824) 2024-03-12T06:05:22.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5659c640 1 -- 172.21.15.76:0/2003981938 learned_addr learned my addr 172.21.15.76:0/2003981938 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea55d9b640 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e55e0 0x5595196f8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5659c640 1 -- 172.21.15.76:0/2003981938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196f2b60 msgr2=0x559519758bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea56d9d640 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196f2b60 0x559519758bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5659c640 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196f2b60 0x559519758bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5659c640 1 -- 172.21.15.76:0/2003981938 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e55e0 msgr2=0x5595196f8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5659c640 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e55e0 0x5595196f8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea5659c640 1 -- 172.21.15.76:0/2003981938 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea5001a040 con 0x5595196e4c30 2024-03-12T06:05:22.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea55d9b640 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e55e0 0x5595196f8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:05:22.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.394+0000 7fea56d9d640 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196f2b60 0x559519758bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:05:22.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea5659c640 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e4c30 0x5595196f8590 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fea480bc3f0 tx=0x7fea4802bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1671df26c4c31e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea3dffb640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:22.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea3dffb640 1 -- 172.21.15.76:0/2003981938 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea480bc230 con 0x5595196e4c30 2024-03-12T06:05:22.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea3dffb640 1 -- 172.21.15.76:0/2003981938 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fea480bcdf0 con 0x5595196e4c30 2024-03-12T06:05:22.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea5f776740 1 -- 172.21.15.76:0/2003981938 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fea480bc6f0 con 0x5595196e4c30 2024-03-12T06:05:22.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea3dffb640 1 -- 172.21.15.76:0/2003981938 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea480dc0b0 con 0x5595196e4c30 2024-03-12T06:05:22.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea5f776740 10 client.17616 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:22.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea5f776740 1 -- 172.21.15.76:0/2003981938 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55951960d8c0 con 0x5595196e4c30 2024-03-12T06:05:22.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea3dffb640 1 -- 172.21.15.76:0/2003981938 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea480bc6f0 con 0x5595196e4c30 2024-03-12T06:05:22.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea3dffb640 1 -- 172.21.15.76:0/2003981938 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fea4803c070 con 0x5595196e4c30 2024-03-12T06:05:22.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea5f776740 10 client.17616 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:22.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea5f776740 10 client.17616 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:05:22.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea5f776740 20 client.17616 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:22.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea5f776740 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55951975a4e0 0x55951975c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea5f776740 4 client.17616 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:05:22.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea5f776740 1 -- 172.21.15.76:0/2003981938 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5595193d8890 con 0x55951975a4e0 2024-03-12T06:05:22.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.395+0000 7fea55d9b640 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55951975a4e0 0x55951975c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea3dffb640 10 client.17616 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:05:22.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea55d9b640 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55951975a4e0 0x55951975c8d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e8b34e1684cdd6d5 server_cookie=2a6b2836988e3210 in_seq=0 out_seq=0 2024-03-12T06:05:22.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea3dffb640 1 -- 172.21.15.76:0/2003981938 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5595193d8890 con 0x55951975a4e0 2024-03-12T06:05:22.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea3dffb640 10 client.17616 handle_command_reply: tid=0 2024-03-12T06:05:22.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea5f776740 1 client.17616 shutdown 2024-03-12T06:05:22.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea5f776740 1 -- 172.21.15.76:0/2003981938 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e4c30 msgr2=0x5595196f8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea5f776740 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e4c30 0x5595196f8590 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fea480bc3f0 tx=0x7fea4802bc20 comp rx=0 tx=0).stop 2024-03-12T06:05:22.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea5f776740 1 -- 172.21.15.76:0/2003981938 shutdown_connections 2024-03-12T06:05:22.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea5f776740 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55951975a4e0 0x55951975c8d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea5f776740 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5595196f2b60 0x559519758bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea5f776740 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5595196e55e0 0x5595196f8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea5f776740 1 --2- 172.21.15.76:0/2003981938 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5595196e4c30 0x5595196f8590 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea3dffb640 0 client.17616 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:05:22.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea5f776740 1 -- 172.21.15.76:0/2003981938 >> 172.21.15.76:0/2003981938 conn(0x5595196df980 msgr2=0x5595196deb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea5f776740 1 -- 172.21.15.76:0/2003981938 shutdown_connections 2024-03-12T06:05:22.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea5f776740 1 -- 172.21.15.76:0/2003981938 wait complete. 2024-03-12T06:05:22.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.396+0000 7fea5f776740 20 client.17616 trim_cache size 0 max 16384 2024-03-12T06:05:22.418 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:05:22.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.397+0000 7fea5cba7640 1 -- 172.21.15.76:0/3937382980 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fea400a1ba0 msgr2=0x7fea400a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.397+0000 7fea5cba7640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fea400a1ba0 0x7fea400a4060 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fea50005c40 tx=0x7fea50009e20 comp rx=0 tx=0).stop 2024-03-12T06:05:22.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.397+0000 7fea5cba7640 1 -- 172.21.15.76:0/3937382980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea58100290 msgr2=0x7fea5819d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.397+0000 7fea5cba7640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea58100290 0x7fea5819d470 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fea48009230 tx=0x7fea48004040 comp rx=0 tx=0).stop 2024-03-12T06:05:22.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.397+0000 7fea5cba7640 1 -- 172.21.15.76:0/3937382980 shutdown_connections 2024-03-12T06:05:22.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.397+0000 7fea5cba7640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fea400a1ba0 0x7fea400a4060 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.397+0000 7fea5cba7640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fea58197620 0x7fea58197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.397+0000 7fea5cba7640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea58105360 0x7fea5819d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.397+0000 7fea5cba7640 1 --2- 172.21.15.76:0/3937382980 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea58100290 0x7fea5819d470 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.397+0000 7fea5cba7640 1 -- 172.21.15.76:0/3937382980 >> 172.21.15.76:0/3937382980 conn(0x7fea58077230 msgr2=0x7fea580fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.420 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.397+0000 7fea5cba7640 1 -- 172.21.15.76:0/3937382980 shutdown_connections 2024-03-12T06:05:22.421 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.397+0000 7fea5cba7640 1 -- 172.21.15.76:0/3937382980 wait complete. 2024-03-12T06:05:22.421 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:05:22.421 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T06:05:22.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.521+0000 7f7d5b464640 1 -- 172.21.15.76:0/345476402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d54106520 msgr2=0x7f7d5410e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.521+0000 7f7d5b464640 1 --2- 172.21.15.76:0/345476402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d54106520 0x7f7d5410e110 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009930 tx=0x7f7d4c02bc30 comp rx=0 tx=0).stop 2024-03-12T06:05:22.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.521+0000 7f7d5b464640 1 -- 172.21.15.76:0/345476402 shutdown_connections 2024-03-12T06:05:22.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.521+0000 7f7d5b464640 1 --2- 172.21.15.76:0/345476402 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d54106520 0x7f7d5410e110 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.521+0000 7f7d5b464640 1 --2- 172.21.15.76:0/345476402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d54100710 0x7f7d54105e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.521+0000 7f7d5b464640 1 --2- 172.21.15.76:0/345476402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d540ffd60 0x7f7d54100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.521+0000 7f7d5b464640 1 -- 172.21.15.76:0/345476402 >> 172.21.15.76:0/345476402 conn(0x7f7d54075360 msgr2=0x7f7d54075770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.521+0000 7f7d5b464640 1 -- 172.21.15.76:0/345476402 shutdown_connections 2024-03-12T06:05:22.527 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.521+0000 7f7d5b464640 1 -- 172.21.15.76:0/345476402 wait complete. 2024-03-12T06:05:22.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.521+0000 7f7d5b464640 1 Processor -- start 2024-03-12T06:05:22.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d5b464640 1 -- start start 2024-03-12T06:05:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d5b464640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d540ffd60 0x7f7d5419d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d5b464640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d54106520 0x7f7d5419dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d5b464640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d54197650 0x7f7d54197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d5b464640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d54198120 con 0x7f7d540ffd60 2024-03-12T06:05:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d5b464640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d54198290 con 0x7f7d54197650 2024-03-12T06:05:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d5b464640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d54198450 con 0x7f7d54106520 2024-03-12T06:05:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d59200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d540ffd60 0x7f7d5419d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d59200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d540ffd60 0x7f7d5419d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57838/0 (socket says 172.21.15.76:57838) 2024-03-12T06:05:22.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d59200640 1 -- 172.21.15.76:0/2718637455 learned_addr learned my addr 172.21.15.76:0/2718637455 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d59a01640 1 --2- 172.21.15.76:0/2718637455 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d54197650 0x7f7d54197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d589ff640 1 --2- 172.21.15.76:0/2718637455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d54106520 0x7f7d5419dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d59200640 1 -- 172.21.15.76:0/2718637455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d54106520 msgr2=0x7f7d5419dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d59200640 1 --2- 172.21.15.76:0/2718637455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d54106520 0x7f7d5419dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d59200640 1 -- 172.21.15.76:0/2718637455 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d54197650 msgr2=0x7f7d54197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d59200640 1 --2- 172.21.15.76:0/2718637455 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d54197650 0x7f7d54197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d59200640 1 -- 172.21.15.76:0/2718637455 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c009560 con 0x7f7d540ffd60 2024-03-12T06:05:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d59200640 1 --2- 172.21.15.76:0/2718637455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d540ffd60 0x7f7d5419d590 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7d4400eb90 tx=0x7f7d4400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d527fc640 1 -- 172.21.15.76:0/2718637455 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d440099b0 con 0x7f7d540ffd60 2024-03-12T06:05:22.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d527fc640 1 -- 172.21.15.76:0/2718637455 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d4400ee90 con 0x7f7d540ffd60 2024-03-12T06:05:22.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d527fc640 1 -- 172.21.15.76:0/2718637455 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d440107a0 con 0x7f7d540ffd60 2024-03-12T06:05:22.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.522+0000 7f7d5b464640 1 -- 172.21.15.76:0/2718637455 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d541a43f0 con 0x7f7d540ffd60 2024-03-12T06:05:22.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.523+0000 7f7d5b464640 1 -- 172.21.15.76:0/2718637455 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d541a4880 con 0x7f7d540ffd60 2024-03-12T06:05:22.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.525+0000 7f7d527fc640 1 -- 172.21.15.76:0/2718637455 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f7d44005020 con 0x7f7d540ffd60 2024-03-12T06:05:22.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.525+0000 7f7d5e03c740 1 Processor -- start 2024-03-12T06:05:22.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.525+0000 7f7d5e03c740 1 -- start start 2024-03-12T06:05:22.538 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.525+0000 7f7d527fc640 1 --2- 172.21.15.76:0/2718637455 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7d3c0a1ba0 0x7f7d3c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.525+0000 7f7d5e03c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9250 0x5561be1e9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.525+0000 7f7d5e03c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9b70 0x5561be1f2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.525+0000 7f7d5e03c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.525+0000 7f7d5e03c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561be1a4960 con 0x5561be1f2750 2024-03-12T06:05:22.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.525+0000 7f7d5e03c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561be158310 con 0x5561be1e9250 2024-03-12T06:05:22.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.525+0000 7f7d5e03c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561be155840 con 0x5561be1e9b70 2024-03-12T06:05:22.539 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.525+0000 7f7d59a01640 1 --2- 172.21.15.76:0/2718637455 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7d3c0a1ba0 0x7f7d3c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d527fc640 1 -- 172.21.15.76:0/2718637455 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7d44014070 con 0x7f7d540ffd60 2024-03-12T06:05:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d59a01640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d59200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9250 0x5561be1e9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d59200640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9250 0x5561be1e9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:44878/0 (socket says 172.21.15.76:44878) 2024-03-12T06:05:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d59a01640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57844/0 (socket says 172.21.15.76:57844) 2024-03-12T06:05:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d59200640 1 -- 172.21.15.76:0/1811304629 learned_addr learned my addr 172.21.15.76:0/1811304629 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d589ff640 1 --2- 172.21.15.76:0/1811304629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9b70 0x5561be1f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d59200640 1 -- 172.21.15.76:0/1811304629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9b70 msgr2=0x5561be1f2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d59200640 1 --2- 172.21.15.76:0/1811304629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9b70 0x5561be1f2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.540 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d59200640 1 -- 172.21.15.76:0/1811304629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 msgr2=0x5561be1f4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d59200640 1 --2- 172.21.15.76:0/1811304629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d59a01640 1 --2- 172.21.15.76:0/2718637455 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7d3c0a1ba0 0x7f7d3c0a4060 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c005940 tx=0x7f7d4c004a90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d59200640 1 -- 172.21.15.76:0/1811304629 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561be113060 con 0x5561be1e9250 2024-03-12T06:05:22.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d59200640 1 --2- 172.21.15.76:0/1811304629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9250 0x5561be1e9630 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7d44097570 tx=0x7f7d4409a5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7d7b66d66b77d82 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d3b7fe640 1 -- 172.21.15.76:0/1811304629 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4409ecf0 con 0x5561be1e9250 2024-03-12T06:05:22.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d3b7fe640 1 -- 172.21.15.76:0/1811304629 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d440a3340 con 0x5561be1e9250 2024-03-12T06:05:22.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.526+0000 7f7d3b7fe640 1 -- 172.21.15.76:0/1811304629 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d440cbb00 con 0x5561be1e9250 2024-03-12T06:05:22.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.527+0000 7f7d5e03c740 1 -- 172.21.15.76:0/1811304629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9250 msgr2=0x5561be1e9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.527+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/1811304629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9250 0x5561be1e9630 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7d44097570 tx=0x7f7d4409a5c0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.527+0000 7f7d5e03c740 1 -- 172.21.15.76:0/1811304629 shutdown_connections 2024-03-12T06:05:22.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.527+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/1811304629 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.527+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/1811304629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9b70 0x5561be1f2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.527+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/1811304629 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9250 0x5561be1e9630 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.527+0000 7f7d5e03c740 1 -- 172.21.15.76:0/1811304629 >> 172.21.15.76:0/1811304629 conn(0x5561be1e3980 msgr2=0x5561be1e3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.527+0000 7f7d5e03c740 1 -- 172.21.15.76:0/1811304629 shutdown_connections 2024-03-12T06:05:22.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.527+0000 7f7d5e03c740 1 -- 172.21.15.76:0/1811304629 wait complete. 2024-03-12T06:05:22.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.527+0000 7f7d5e03c740 1 -- start start 2024-03-12T06:05:22.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.527+0000 7f7d5e03c740 1 Processor -- start 2024-03-12T06:05:22.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d5e03c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9250 0x5561be1ef740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d5e03c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9b70 0x5561be1efc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d5e03c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d5e03c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561be151020 con 0x5561be1f2750 2024-03-12T06:05:22.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d5e03c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561bde24430 con 0x5561be1e9b70 2024-03-12T06:05:22.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d5e03c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561bdde0e00 con 0x5561be1e9250 2024-03-12T06:05:22.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d5e03c740 10 client.0 _ll_register_callbacks cb 0x5561be0f8cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:22.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d59a01640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d59a01640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57852/0 (socket says 172.21.15.76:57852) 2024-03-12T06:05:22.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d59a01640 1 -- 172.21.15.76:0/500034919 learned_addr learned my addr 172.21.15.76:0/500034919 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d589ff640 1 --2- 172.21.15.76:0/500034919 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9b70 0x5561be1efc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d59200640 1 --2- 172.21.15.76:0/500034919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9250 0x5561be1ef740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d59a01640 1 -- 172.21.15.76:0/500034919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9250 msgr2=0x5561be1ef740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d59a01640 1 --2- 172.21.15.76:0/500034919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9250 0x5561be1ef740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d59a01640 1 -- 172.21.15.76:0/500034919 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9b70 msgr2=0x5561be1efc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d59a01640 1 --2- 172.21.15.76:0/500034919 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9b70 0x5561be1efc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d59a01640 1 -- 172.21.15.76:0/500034919 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d541a4880 con 0x5561be1f2750 2024-03-12T06:05:22.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.528+0000 7f7d59a01640 1 --2- 172.21.15.76:0/500034919 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f01c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c005ab0 tx=0x7f7d4c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa832b806959dd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:22.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d38ff9640 1 -- 172.21.15.76:0/500034919 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c004df0 con 0x5561be1f2750 2024-03-12T06:05:22.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d38ff9640 1 -- 172.21.15.76:0/500034919 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7d4c004f50 con 0x5561be1f2750 2024-03-12T06:05:22.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d5e03c740 1 -- 172.21.15.76:0/500034919 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5561bde507e0 con 0x5561be1f2750 2024-03-12T06:05:22.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d38ff9640 1 -- 172.21.15.76:0/500034919 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d4c005110 con 0x5561be1f2750 2024-03-12T06:05:22.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d5e03c740 10 client.17628 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:22.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d5e03c740 1 -- 172.21.15.76:0/500034919 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5561be14f6a0 con 0x5561be1f2750 2024-03-12T06:05:22.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d38ff9640 1 -- 172.21.15.76:0/500034919 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5561bde507e0 con 0x5561be1f2750 2024-03-12T06:05:22.545 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d38ff9640 1 -- 172.21.15.76:0/500034919 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7d4c03c040 con 0x5561be1f2750 2024-03-12T06:05:22.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d5e03c740 10 client.17628 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:22.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d5e03c740 10 client.17628 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:05:22.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d5e03c740 20 client.17628 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:22.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/500034919 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5561be202ac0 0x5561be204ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d5e03c740 4 client.17628 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:05:22.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.529+0000 7f7d5e03c740 1 -- 172.21.15.76:0/500034919 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5561bde8d720 con 0x5561be202ac0 2024-03-12T06:05:22.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.530+0000 7f7d59200640 1 --2- 172.21.15.76:0/500034919 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5561be202ac0 0x5561be204ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.530+0000 7f7d38ff9640 10 client.17628 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:05:22.546 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.530+0000 7f7d59200640 1 --2- 172.21.15.76:0/500034919 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5561be202ac0 0x5561be204ee0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a74c5c8def7223d8 server_cookie=2678c772ecfa2b01 in_seq=0 out_seq=0 2024-03-12T06:05:22.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.530+0000 7f7d38ff9640 1 -- 172.21.15.76:0/500034919 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5561bde8d720 con 0x5561be202ac0 2024-03-12T06:05:22.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.530+0000 7f7d38ff9640 10 client.17628 handle_command_reply: tid=0 2024-03-12T06:05:22.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.531+0000 7f7d5e03c740 1 client.17628 shutdown 2024-03-12T06:05:22.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.531+0000 7f7d5e03c740 1 -- 172.21.15.76:0/500034919 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 msgr2=0x5561be1f01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.531+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/500034919 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f01c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c005ab0 tx=0x7f7d4c02bc30 comp rx=0 tx=0).stop 2024-03-12T06:05:22.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.531+0000 7f7d5e03c740 1 -- 172.21.15.76:0/500034919 shutdown_connections 2024-03-12T06:05:22.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.531+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/500034919 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5561be202ac0 0x5561be204ee0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.531+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/500034919 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f01c0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.547 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.531+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/500034919 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9b70 0x5561be1efc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.531+0000 7f7d38ff9640 0 client.17628 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:05:22.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.531+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/500034919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9250 0x5561be1ef740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.531+0000 7f7d5e03c740 1 -- 172.21.15.76:0/500034919 >> 172.21.15.76:0/500034919 conn(0x5561be1e3980 msgr2=0x5561be1e56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.531+0000 7f7d5e03c740 1 -- 172.21.15.76:0/500034919 shutdown_connections 2024-03-12T06:05:22.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.531+0000 7f7d5e03c740 1 -- 172.21.15.76:0/500034919 wait complete. 2024-03-12T06:05:22.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.531+0000 7f7d5e03c740 20 client.17628 trim_cache size 0 max 16384 2024-03-12T06:05:22.554 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.542+0000 7f7d5e03c740 1 Processor -- start 2024-03-12T06:05:22.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d5e03c740 1 -- start start 2024-03-12T06:05:22.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d5e03c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9250 0x5561be1effe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d5e03c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9b70 0x5561be1e2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d5e03c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d5e03c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561be0d6130 con 0x5561be1f2750 2024-03-12T06:05:22.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d5e03c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561be1e37f0 con 0x5561be1e9250 2024-03-12T06:05:22.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d5e03c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561be1e8340 con 0x5561be1e9b70 2024-03-12T06:05:22.555 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d59a01640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d59a01640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57864/0 (socket says 172.21.15.76:57864) 2024-03-12T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d59a01640 1 -- 172.21.15.76:0/1220955479 learned_addr learned my addr 172.21.15.76:0/1220955479 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d589ff640 1 --2- 172.21.15.76:0/1220955479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9b70 0x5561be1e2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d59a01640 1 -- 172.21.15.76:0/1220955479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9b70 msgr2=0x5561be1e2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d59a01640 1 --2- 172.21.15.76:0/1220955479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9b70 0x5561be1e2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d59a01640 1 -- 172.21.15.76:0/1220955479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9250 msgr2=0x5561be1effe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d59a01640 1 --2- 172.21.15.76:0/1220955479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9250 0x5561be1effe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d59a01640 1 -- 172.21.15.76:0/1220955479 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561be0ebc20 con 0x5561be1f2750 2024-03-12T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d589ff640 1 --2- 172.21.15.76:0/1220955479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9b70 0x5561be1e2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d59a01640 1 --2- 172.21.15.76:0/1220955479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f9600 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009800 tx=0x7f7d4c004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e284e2917c1430 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d3affd640 1 -- 172.21.15.76:0/1220955479 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c064070 con 0x5561be1f2750 2024-03-12T06:05:22.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d3affd640 1 -- 172.21.15.76:0/1220955479 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d4c003e90 con 0x5561be1f2750 2024-03-12T06:05:22.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.543+0000 7f7d3affd640 1 -- 172.21.15.76:0/1220955479 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c05fae0 con 0x5561be1f2750 2024-03-12T06:05:22.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.544+0000 7f7d5e03c740 1 -- 172.21.15.76:0/1220955479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 msgr2=0x5561be1f9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.544+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/1220955479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f9600 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009800 tx=0x7f7d4c004610 comp rx=0 tx=0).stop 2024-03-12T06:05:22.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.544+0000 7f7d5e03c740 1 -- 172.21.15.76:0/1220955479 shutdown_connections 2024-03-12T06:05:22.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.544+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/1220955479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be1f9600 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.544+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/1220955479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9b70 0x5561be1e2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.544+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/1220955479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9250 0x5561be1effe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.557 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.544+0000 7f7d5e03c740 1 -- 172.21.15.76:0/1220955479 >> 172.21.15.76:0/1220955479 conn(0x5561be1e3980 msgr2=0x5561be1e22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.544+0000 7f7d5e03c740 1 -- 172.21.15.76:0/1220955479 shutdown_connections 2024-03-12T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.544+0000 7f7d5e03c740 1 -- 172.21.15.76:0/1220955479 wait complete. 2024-03-12T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.544+0000 7f7d5e03c740 1 -- start start 2024-03-12T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.544+0000 7f7d5e03c740 1 Processor -- start 2024-03-12T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d5e03c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9250 0x5561be1f86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d5e03c740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9b70 0x5561be1f8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d5e03c740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be258fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d5e03c740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561be1f9320 con 0x5561be1f2750 2024-03-12T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d5e03c740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561be259510 con 0x5561be1e9b70 2024-03-12T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d5e03c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561be259680 con 0x5561be1e9250 2024-03-12T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d5e03c740 10 client.0 _ll_register_callbacks cb 0x7f7d44004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d59a01640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be258fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d59a01640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be258fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57868/0 (socket says 172.21.15.76:57868) 2024-03-12T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d59a01640 1 -- 172.21.15.76:0/3354271101 learned_addr learned my addr 172.21.15.76:0/3354271101 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d589ff640 1 --2- 172.21.15.76:0/3354271101 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9b70 0x5561be1f8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d59200640 1 --2- 172.21.15.76:0/3354271101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9250 0x5561be1f86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d59a01640 1 -- 172.21.15.76:0/3354271101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9250 msgr2=0x5561be1f86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d59a01640 1 --2- 172.21.15.76:0/3354271101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9250 0x5561be1f86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d59a01640 1 -- 172.21.15.76:0/3354271101 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9b70 msgr2=0x5561be1f8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d59a01640 1 --2- 172.21.15.76:0/3354271101 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9b70 0x5561be1f8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d59a01640 1 -- 172.21.15.76:0/3354271101 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561be14f6a0 con 0x5561be1f2750 2024-03-12T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d59a01640 1 --2- 172.21.15.76:0/3354271101 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be258fd0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c02c170 tx=0x7f7d4c02c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c0cd7ecae4be2b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d38ff9640 1 -- 172.21.15.76:0/3354271101 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c064020 con 0x5561be1f2750 2024-03-12T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d38ff9640 1 -- 172.21.15.76:0/3354271101 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7d4c02c9a0 con 0x5561be1f2750 2024-03-12T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.545+0000 7f7d5e03c740 1 -- 172.21.15.76:0/3354271101 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d440a3340 con 0x5561be1f2750 2024-03-12T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.546+0000 7f7d38ff9640 1 -- 172.21.15.76:0/3354271101 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d4c005200 con 0x5561be1f2750 2024-03-12T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.546+0000 7f7d5e03c740 10 client.17634 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.546+0000 7f7d5e03c740 1 -- 172.21.15.76:0/3354271101 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5561be1118c0 con 0x5561be1f2750 2024-03-12T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.546+0000 7f7d38ff9640 1 -- 172.21.15.76:0/3354271101 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d440a3340 con 0x5561be1f2750 2024-03-12T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.546+0000 7f7d38ff9640 1 -- 172.21.15.76:0/3354271101 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7d4c03c070 con 0x5561be1f2750 2024-03-12T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.546+0000 7f7d5e03c740 10 client.17634 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.546+0000 7f7d5e03c740 10 client.17634 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.546+0000 7f7d5e03c740 20 client.17634 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.546+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/3354271101 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5561be25a9c0 0x5561be25cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.546+0000 7f7d5e03c740 4 client.17634 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.546+0000 7f7d5e03c740 1 -- 172.21.15.76:0/3354271101 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5561bdedc890 con 0x5561be25a9c0 2024-03-12T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.546+0000 7f7d59200640 1 --2- 172.21.15.76:0/3354271101 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5561be25a9c0 0x5561be25cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d38ff9640 10 client.17634 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d59200640 1 --2- 172.21.15.76:0/3354271101 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5561be25a9c0 0x5561be25cdb0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=828a4fd382d69f15 server_cookie=7360cb8862236d8d in_seq=0 out_seq=0 2024-03-12T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d38ff9640 1 -- 172.21.15.76:0/3354271101 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5561bdedc890 con 0x5561be25a9c0 2024-03-12T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d38ff9640 10 client.17634 handle_command_reply: tid=0 2024-03-12T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d5e03c740 1 client.17634 shutdown 2024-03-12T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d5e03c740 1 -- 172.21.15.76:0/3354271101 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 msgr2=0x5561be258fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/3354271101 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be258fd0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c02c170 tx=0x7f7d4c02c2b0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d5e03c740 1 -- 172.21.15.76:0/3354271101 shutdown_connections 2024-03-12T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/3354271101 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x5561be25a9c0 0x5561be25cdb0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/3354271101 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5561be1f2750 0x5561be258fd0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d38ff9640 0 client.17634 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/3354271101 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5561be1e9b70 0x5561be1f8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d5e03c740 1 --2- 172.21.15.76:0/3354271101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561be1e9250 0x5561be1f86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d5e03c740 1 -- 172.21.15.76:0/3354271101 >> 172.21.15.76:0/3354271101 conn(0x5561be1e3980 msgr2=0x5561be1e2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d5e03c740 1 -- 172.21.15.76:0/3354271101 shutdown_connections 2024-03-12T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d5e03c740 1 -- 172.21.15.76:0/3354271101 wait complete. 2024-03-12T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.547+0000 7f7d5e03c740 20 client.17634 trim_cache size 0 max 16384 2024-03-12T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:05:22.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.548+0000 7f7d5b464640 1 -- 172.21.15.76:0/2718637455 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7d3c0a1ba0 msgr2=0x7f7d3c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.548+0000 7f7d5b464640 1 --2- 172.21.15.76:0/2718637455 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7d3c0a1ba0 0x7f7d3c0a4060 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c005940 tx=0x7f7d4c004a90 comp rx=0 tx=0).stop 2024-03-12T06:05:22.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.548+0000 7f7d5b464640 1 -- 172.21.15.76:0/2718637455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d540ffd60 msgr2=0x7f7d5419d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.548+0000 7f7d5b464640 1 --2- 172.21.15.76:0/2718637455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d540ffd60 0x7f7d5419d590 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7d4400eb90 tx=0x7f7d4400c6a0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.548+0000 7f7d5b464640 1 -- 172.21.15.76:0/2718637455 shutdown_connections 2024-03-12T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.548+0000 7f7d5b464640 1 --2- 172.21.15.76:0/2718637455 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7d3c0a1ba0 0x7f7d3c0a4060 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.548+0000 7f7d5b464640 1 --2- 172.21.15.76:0/2718637455 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7d54197650 0x7f7d54197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.548+0000 7f7d5b464640 1 --2- 172.21.15.76:0/2718637455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d54106520 0x7f7d5419dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.548+0000 7f7d5b464640 1 --2- 172.21.15.76:0/2718637455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d540ffd60 0x7f7d5419d590 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.548+0000 7f7d5b464640 1 -- 172.21.15.76:0/2718637455 >> 172.21.15.76:0/2718637455 conn(0x7f7d54075360 msgr2=0x7f7d540fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.548+0000 7f7d5b464640 1 -- 172.21.15.76:0/2718637455 shutdown_connections 2024-03-12T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.548+0000 7f7d5b464640 1 -- 172.21.15.76:0/2718637455 wait complete. 2024-03-12T06:05:22.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:22 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2620602984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:05:22.566 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:05:22.567 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T06:05:22.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.666+0000 7fb665049640 1 -- 172.21.15.76:0/1763144851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6601022e0 msgr2=0x7fb6601026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.666+0000 7fb665049640 1 --2- 172.21.15.76:0/1763144851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6601022e0 0x7fb6601026c0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fb650009900 tx=0x7fb65002bcd0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.666+0000 7fb665049640 1 -- 172.21.15.76:0/1763144851 shutdown_connections 2024-03-12T06:05:22.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.666+0000 7fb665049640 1 --2- 172.21.15.76:0/1763144851 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb66010b700 0x7fb66010dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.666+0000 7fb665049640 1 --2- 172.21.15.76:0/1763144851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb660102c00 0x7fb66010b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.666+0000 7fb665049640 1 --2- 172.21.15.76:0/1763144851 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6601022e0 0x7fb6601026c0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.666+0000 7fb665049640 1 -- 172.21.15.76:0/1763144851 >> 172.21.15.76:0/1763144851 conn(0x7fb6600fbae0 msgr2=0x7fb6600fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.666+0000 7fb665049640 1 -- 172.21.15.76:0/1763144851 shutdown_connections 2024-03-12T06:05:22.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.666+0000 7fb665049640 1 -- 172.21.15.76:0/1763144851 wait complete. 2024-03-12T06:05:22.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.667+0000 7fb665049640 1 Processor -- start 2024-03-12T06:05:22.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.667+0000 7fb665049640 1 -- start start 2024-03-12T06:05:22.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.667+0000 7fb665049640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6601022e0 0x7fb6601977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.667+0000 7fb665049640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb660102c00 0x7fb660197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.675 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.667+0000 7fb665049640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb66010b700 0x7fb66019c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.667+0000 7fb665049640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66019c610 con 0x7fb6601022e0 2024-03-12T06:05:22.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.667+0000 7fb665049640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66019c780 con 0x7fb66010b700 2024-03-12T06:05:22.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.667+0000 7fb665049640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66019c940 con 0x7fb660102c00 2024-03-12T06:05:22.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.667+0000 7fb65ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6601022e0 0x7fb6601977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.667+0000 7fb65ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6601022e0 0x7fb6601977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57878/0 (socket says 172.21.15.76:57878) 2024-03-12T06:05:22.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.667+0000 7fb65ed9d640 1 -- 172.21.15.76:0/4042184896 learned_addr learned my addr 172.21.15.76:0/4042184896 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.667+0000 7fb65f59e640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb66010b700 0x7fb66019c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.668+0000 7fb657fff640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb660102c00 0x7fb660197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.668+0000 7fb65ed9d640 1 -- 172.21.15.76:0/4042184896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb660102c00 msgr2=0x7fb660197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.668+0000 7fb65ed9d640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb660102c00 0x7fb660197ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.668+0000 7fb65ed9d640 1 -- 172.21.15.76:0/4042184896 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb66010b700 msgr2=0x7fb66019c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.668+0000 7fb65ed9d640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb66010b700 0x7fb66019c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.668+0000 7fb65ed9d640 1 -- 172.21.15.76:0/4042184896 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb650009560 con 0x7fb6601022e0 2024-03-12T06:05:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.668+0000 7fb657fff640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb660102c00 0x7fb660197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:05:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.668+0000 7fb65ed9d640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6601022e0 0x7fb6601977a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fb650005940 tx=0x7fb650004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.668+0000 7fb65cd99640 1 -- 172.21.15.76:0/4042184896 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb650004440 con 0x7fb6601022e0 2024-03-12T06:05:22.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.668+0000 7fb665049640 1 -- 172.21.15.76:0/4042184896 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb66019cbc0 con 0x7fb6601022e0 2024-03-12T06:05:22.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.668+0000 7fb65cd99640 1 -- 172.21.15.76:0/4042184896 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6500052d0 con 0x7fb6601022e0 2024-03-12T06:05:22.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.668+0000 7fb65cd99640 1 -- 172.21.15.76:0/4042184896 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb650003da0 con 0x7fb6601022e0 2024-03-12T06:05:22.678 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.668+0000 7fb665049640 1 -- 172.21.15.76:0/4042184896 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6601a44a0 con 0x7fb6601022e0 2024-03-12T06:05:22.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.670+0000 7fb65cd99640 1 -- 172.21.15.76:0/4042184896 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb650005440 con 0x7fb6601022e0 2024-03-12T06:05:22.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.670+0000 7fb667c16740 1 Processor -- start 2024-03-12T06:05:22.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.670+0000 7fb667c16740 1 -- start start 2024-03-12T06:05:22.685 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb667c16740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771a6250 0x55d3771a6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb667c16740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d3771af210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb667c16740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771af750 0x55d3771b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb667c16740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d377161960 con 0x55d3771a6b70 2024-03-12T06:05:22.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb667c16740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d377115310 con 0x55d3771a6250 2024-03-12T06:05:22.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb667c16740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d377112840 con 0x55d3771af750 2024-03-12T06:05:22.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb65cd99640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb64c0a1ba0 0x7fb64c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb65ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d3771af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.686 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb65ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d3771af210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57886/0 (socket says 172.21.15.76:57886) 2024-03-12T06:05:22.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb65f59e640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771a6250 0x55d3771a6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb65ed9d640 1 -- 172.21.15.76:0/3059604815 learned_addr learned my addr 172.21.15.76:0/3059604815 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb65f59e640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb64c0a1ba0 0x7fb64c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb657fff640 1 --2- 172.21.15.76:0/3059604815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771af750 0x55d3771b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb65cd99640 1 -- 172.21.15.76:0/4042184896 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb6500f0090 con 0x7fb6601022e0 2024-03-12T06:05:22.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb65ed9d640 1 -- 172.21.15.76:0/3059604815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771af750 msgr2=0x55d3771b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb65ed9d640 1 --2- 172.21.15.76:0/3059604815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771af750 0x55d3771b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.687 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb65ed9d640 1 -- 172.21.15.76:0/3059604815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771a6250 msgr2=0x55d3771a6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb65ed9d640 1 --2- 172.21.15.76:0/3059604815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771a6250 0x55d3771a6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb65ed9d640 1 -- 172.21.15.76:0/3059604815 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3770d0060 con 0x55d3771a6b70 2024-03-12T06:05:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.671+0000 7fb65f59e640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb64c0a1ba0 0x7fb64c0a4060 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb658005cc0 tx=0x7fb65800b8c0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.672+0000 7fb65ed9d640 1 --2- 172.21.15.76:0/3059604815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d3771af210 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fb650006010 tx=0x7fb65004f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ff743cc0538945 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.672+0000 7fb6557fa640 1 -- 172.21.15.76:0/3059604815 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65003ce30 con 0x55d3771a6b70 2024-03-12T06:05:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.672+0000 7fb6557fa640 1 -- 172.21.15.76:0/3059604815 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6500bd4d0 con 0x55d3771a6b70 2024-03-12T06:05:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.672+0000 7fb6557fa640 1 -- 172.21.15.76:0/3059604815 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6500b9440 con 0x55d3771a6b70 2024-03-12T06:05:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.672+0000 7fb667c16740 1 -- 172.21.15.76:0/3059604815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 msgr2=0x55d3771af210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.672+0000 7fb667c16740 1 --2- 172.21.15.76:0/3059604815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d3771af210 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fb650006010 tx=0x7fb65004f000 comp rx=0 tx=0).stop 2024-03-12T06:05:22.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.672+0000 7fb667c16740 1 -- 172.21.15.76:0/3059604815 shutdown_connections 2024-03-12T06:05:22.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.672+0000 7fb667c16740 1 --2- 172.21.15.76:0/3059604815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771af750 0x55d3771b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.672+0000 7fb667c16740 1 --2- 172.21.15.76:0/3059604815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d3771af210 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.672+0000 7fb667c16740 1 --2- 172.21.15.76:0/3059604815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771a6250 0x55d3771a6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.672+0000 7fb667c16740 1 -- 172.21.15.76:0/3059604815 >> 172.21.15.76:0/3059604815 conn(0x55d3771a0980 msgr2=0x55d3771a0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb667c16740 1 -- 172.21.15.76:0/3059604815 shutdown_connections 2024-03-12T06:05:22.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb667c16740 1 -- 172.21.15.76:0/3059604815 wait complete. 2024-03-12T06:05:22.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb667c16740 1 -- start start 2024-03-12T06:05:22.689 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb667c16740 1 Processor -- start 2024-03-12T06:05:22.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb667c16740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6250 0x55d3771ac740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb667c16740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6b70 0x55d3771acc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb667c16740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 0x55d3771ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb667c16740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d37710e020 con 0x55d3771a6250 2024-03-12T06:05:22.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb667c16740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d376de1430 con 0x55d3771af750 2024-03-12T06:05:22.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb667c16740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d376d9de00 con 0x55d3771a6b70 2024-03-12T06:05:22.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb667c16740 10 client.0 _ll_register_callbacks cb 0x55d3770b5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:22.690 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb65ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6250 0x55d3771ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb65ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6250 0x55d3771ac740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57890/0 (socket says 172.21.15.76:57890) 2024-03-12T06:05:22.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb65ed9d640 1 -- 172.21.15.76:0/2014049498 learned_addr learned my addr 172.21.15.76:0/2014049498 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb657fff640 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6b70 0x55d3771acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb65f59e640 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 0x55d3771ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb65ed9d640 1 -- 172.21.15.76:0/2014049498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6b70 msgr2=0x55d3771acc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb65ed9d640 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6b70 0x55d3771acc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.691 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb65ed9d640 1 -- 172.21.15.76:0/2014049498 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 msgr2=0x55d3771ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.673+0000 7fb65ed9d640 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 0x55d3771ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb65ed9d640 1 -- 172.21.15.76:0/2014049498 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6601a44a0 con 0x55d3771a6250 2024-03-12T06:05:22.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb657fff640 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6b70 0x55d3771acc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:22.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb65f59e640 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 0x55d3771ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:05:22.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb65ed9d640 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6250 0x55d3771ac740 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fb6500bcad0 tx=0x7fb65002bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756646b3c5c04cb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb6467fc640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:22.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb6467fc640 1 -- 172.21.15.76:0/2014049498 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6500d4070 con 0x55d3771a6250 2024-03-12T06:05:22.692 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb6467fc640 1 -- 172.21.15.76:0/2014049498 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb6500bcdd0 con 0x55d3771a6250 2024-03-12T06:05:22.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb667c16740 1 -- 172.21.15.76:0/2014049498 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d376e0d7e0 con 0x55d3771a6250 2024-03-12T06:05:22.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb6467fc640 1 -- 172.21.15.76:0/2014049498 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb65003ce30 con 0x55d3771a6250 2024-03-12T06:05:22.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb667c16740 10 client.17646 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:22.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb667c16740 1 -- 172.21.15.76:0/2014049498 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d37710c6a0 con 0x55d3771a6250 2024-03-12T06:05:22.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb6467fc640 1 -- 172.21.15.76:0/2014049498 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d376e0d7e0 con 0x55d3771a6250 2024-03-12T06:05:22.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb6467fc640 1 -- 172.21.15.76:0/2014049498 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb65003c070 con 0x55d3771a6250 2024-03-12T06:05:22.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb667c16740 10 client.17646 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:22.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb667c16740 10 client.17646 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:05:22.693 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb667c16740 20 client.17646 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:22.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb667c16740 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d3771bf2c0 0x55d3771c16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb667c16740 4 client.17646 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:05:22.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.674+0000 7fb667c16740 1 -- 172.21.15.76:0/2014049498 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d376e4a720 con 0x55d3771bf2c0 2024-03-12T06:05:22.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.675+0000 7fb657fff640 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d3771bf2c0 0x55d3771c16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.675+0000 7fb6467fc640 10 client.17646 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:22.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.675+0000 7fb657fff640 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d3771bf2c0 0x55d3771c16b0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60f40826e6b7ba52 server_cookie=324ff066f320e9ae in_seq=0 out_seq=0 2024-03-12T06:05:22.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb6467fc640 1 -- 172.21.15.76:0/2014049498 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d376e4a720 con 0x55d3771bf2c0 2024-03-12T06:05:22.694 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb6467fc640 10 client.17646 handle_command_reply: tid=0 2024-03-12T06:05:22.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb667c16740 1 client.17646 shutdown 2024-03-12T06:05:22.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb667c16740 1 -- 172.21.15.76:0/2014049498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6250 msgr2=0x55d3771ac740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb667c16740 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6250 0x55d3771ac740 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fb6500bcad0 tx=0x7fb65002bcd0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb667c16740 1 -- 172.21.15.76:0/2014049498 shutdown_connections 2024-03-12T06:05:22.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb667c16740 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 0x55d3771ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb667c16740 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6b70 0x55d3771acc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb667c16740 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d3771bf2c0 0x55d3771c16b0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb667c16740 1 --2- 172.21.15.76:0/2014049498 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6250 0x55d3771ac740 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.695 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb6467fc640 0 client.17646 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:22.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb667c16740 1 -- 172.21.15.76:0/2014049498 >> 172.21.15.76:0/2014049498 conn(0x55d3771a0980 msgr2=0x55d3771a26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb667c16740 1 -- 172.21.15.76:0/2014049498 shutdown_connections 2024-03-12T06:05:22.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb667c16740 1 -- 172.21.15.76:0/2014049498 wait complete. 2024-03-12T06:05:22.696 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.676+0000 7fb667c16740 20 client.17646 trim_cache size 0 max 16384 2024-03-12T06:05:22.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:22 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2620602984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:05:22.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.687+0000 7fb667c16740 1 Processor -- start 2024-03-12T06:05:22.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.687+0000 7fb667c16740 1 -- start start 2024-03-12T06:05:22.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb667c16740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6250 0x55d3771aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb667c16740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d3771b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.703 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb667c16740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 0x55d3771a7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb667c16740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d377093130 con 0x55d3771a6b70 2024-03-12T06:05:22.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb667c16740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d377192a70 con 0x55d3771af750 2024-03-12T06:05:22.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb667c16740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3771bc2c0 con 0x55d3771a6250 2024-03-12T06:05:22.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb657fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d3771b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb657fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d3771b66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57900/0 (socket says 172.21.15.76:57900) 2024-03-12T06:05:22.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb657fff640 1 -- 172.21.15.76:0/1292967520 learned_addr learned my addr 172.21.15.76:0/1292967520 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.704 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb65f59e640 1 --2- 172.21.15.76:0/1292967520 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 0x55d3771a7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb657fff640 1 -- 172.21.15.76:0/1292967520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6250 msgr2=0x55d3771aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb65ed9d640 1 --2- 172.21.15.76:0/1292967520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6250 0x55d3771aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb657fff640 1 --2- 172.21.15.76:0/1292967520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6250 0x55d3771aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb657fff640 1 -- 172.21.15.76:0/1292967520 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 msgr2=0x55d3771a7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb657fff640 1 --2- 172.21.15.76:0/1292967520 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 0x55d3771a7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb657fff640 1 -- 172.21.15.76:0/1292967520 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3770a8c20 con 0x55d3771a6b70 2024-03-12T06:05:22.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb657fff640 1 --2- 172.21.15.76:0/1292967520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d3771b66b0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x55d3771a1b10 tx=0x7fb648009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae7fb4f7ca9b753 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.705 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb654ff9640 1 -- 172.21.15.76:0/1292967520 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb648018070 con 0x55d3771a6b70 2024-03-12T06:05:22.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb654ff9640 1 -- 172.21.15.76:0/1292967520 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb648019040 con 0x55d3771a6b70 2024-03-12T06:05:22.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.688+0000 7fb654ff9640 1 -- 172.21.15.76:0/1292967520 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb648022960 con 0x55d3771a6b70 2024-03-12T06:05:22.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.689+0000 7fb667c16740 1 -- 172.21.15.76:0/1292967520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 msgr2=0x55d3771b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.689+0000 7fb667c16740 1 --2- 172.21.15.76:0/1292967520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d3771b66b0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x55d3771a1b10 tx=0x7fb648009510 comp rx=0 tx=0).stop 2024-03-12T06:05:22.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.689+0000 7fb667c16740 1 -- 172.21.15.76:0/1292967520 shutdown_connections 2024-03-12T06:05:22.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.689+0000 7fb667c16740 1 --2- 172.21.15.76:0/1292967520 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 0x55d3771a7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.689+0000 7fb667c16740 1 --2- 172.21.15.76:0/1292967520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d3771b66b0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.706 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.689+0000 7fb667c16740 1 --2- 172.21.15.76:0/1292967520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6250 0x55d3771aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.689+0000 7fb667c16740 1 -- 172.21.15.76:0/1292967520 >> 172.21.15.76:0/1292967520 conn(0x55d3771a0980 msgr2=0x55d37719ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.689+0000 7fb667c16740 1 -- 172.21.15.76:0/1292967520 shutdown_connections 2024-03-12T06:05:22.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.689+0000 7fb667c16740 1 -- 172.21.15.76:0/1292967520 wait complete. 2024-03-12T06:05:22.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.689+0000 7fb667c16740 1 -- start start 2024-03-12T06:05:22.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.689+0000 7fb667c16740 1 Processor -- start 2024-03-12T06:05:22.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb667c16740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6250 0x55d377213cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.707 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb667c16740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d377214200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb667c16740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 0x55d37720e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb667c16740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d37720e690 con 0x55d3771a6b70 2024-03-12T06:05:22.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb667c16740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d37720e800 con 0x55d3771af750 2024-03-12T06:05:22.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb667c16740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d37720e9c0 con 0x55d3771a6250 2024-03-12T06:05:22.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb667c16740 10 client.0 _ll_register_callbacks cb 0x7fb6500b89d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:22.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb657fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d377214200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb657fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d377214200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57904/0 (socket says 172.21.15.76:57904) 2024-03-12T06:05:22.708 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb657fff640 1 -- 172.21.15.76:0/1534256562 learned_addr learned my addr 172.21.15.76:0/1534256562 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:22.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb65ed9d640 1 --2- 172.21.15.76:0/1534256562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6250 0x55d377213cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb65f59e640 1 --2- 172.21.15.76:0/1534256562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 0x55d37720e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb657fff640 1 -- 172.21.15.76:0/1534256562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6250 msgr2=0x55d377213cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb657fff640 1 --2- 172.21.15.76:0/1534256562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6250 0x55d377213cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb657fff640 1 -- 172.21.15.76:0/1534256562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 msgr2=0x55d37720e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:22.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb657fff640 1 --2- 172.21.15.76:0/1534256562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 0x55d37720e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb657fff640 1 -- 172.21.15.76:0/1534256562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6480099d0 con 0x55d3771a6b70 2024-03-12T06:05:22.709 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb65f59e640 1 --2- 172.21.15.76:0/1534256562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 0x55d37720e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:22.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb657fff640 1 --2- 172.21.15.76:0/1534256562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d377214200 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fb648021000 tx=0x7fb6480236e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3105bdfa1c1b890 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:22.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb6467fc640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:22.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb6467fc640 1 -- 172.21.15.76:0/1534256562 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb648018070 con 0x55d3771a6b70 2024-03-12T06:05:22.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb6467fc640 1 -- 172.21.15.76:0/1534256562 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb648023d50 con 0x55d3771a6b70 2024-03-12T06:05:22.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.690+0000 7fb667c16740 1 -- 172.21.15.76:0/1534256562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb6500bd4d0 con 0x55d3771a6b70 2024-03-12T06:05:22.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.691+0000 7fb6467fc640 1 -- 172.21.15.76:0/1534256562 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb64802c440 con 0x55d3771a6b70 2024-03-12T06:05:22.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.691+0000 7fb667c16740 10 client.17652 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:22.710 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.691+0000 7fb667c16740 1 -- 172.21.15.76:0/1534256562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d3770ce8c0 con 0x55d3771a6b70 2024-03-12T06:05:22.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.691+0000 7fb6467fc640 1 -- 172.21.15.76:0/1534256562 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6500bd4d0 con 0x55d3771a6b70 2024-03-12T06:05:22.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.691+0000 7fb6467fc640 1 -- 172.21.15.76:0/1534256562 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb648020020 con 0x55d3771a6b70 2024-03-12T06:05:22.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.691+0000 7fb667c16740 10 client.17652 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:22.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.691+0000 7fb667c16740 10 client.17652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:05:22.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.691+0000 7fb667c16740 20 client.17652 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:22.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.691+0000 7fb667c16740 1 --2- 172.21.15.76:0/1534256562 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d37721bb40 0x55d37721df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:22.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.691+0000 7fb667c16740 4 client.17652 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:05:22.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.691+0000 7fb667c16740 1 -- 172.21.15.76:0/1534256562 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d376e99890 con 0x55d37721bb40 2024-03-12T06:05:22.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.691+0000 7fb65ed9d640 1 --2- 172.21.15.76:0/1534256562 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d37721bb40 0x55d37721df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:22.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb6467fc640 10 client.17652 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:22.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb65ed9d640 1 --2- 172.21.15.76:0/1534256562 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d37721bb40 0x55d37721df60 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bde8ef7397e06386 server_cookie=b0286f3d9c9f6229 in_seq=0 out_seq=0 2024-03-12T06:05:22.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb6467fc640 1 -- 172.21.15.76:0/1534256562 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d376e99890 con 0x55d37721bb40 2024-03-12T06:05:22.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb6467fc640 10 client.17652 handle_command_reply: tid=0 2024-03-12T06:05:22.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb667c16740 1 client.17652 shutdown 2024-03-12T06:05:22.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb667c16740 1 -- 172.21.15.76:0/1534256562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 msgr2=0x55d377214200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb667c16740 1 --2- 172.21.15.76:0/1534256562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d377214200 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fb648021000 tx=0x7fb6480236e0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb667c16740 1 -- 172.21.15.76:0/1534256562 shutdown_connections 2024-03-12T06:05:22.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb667c16740 1 --2- 172.21.15.76:0/1534256562 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55d3771af750 0x55d37720e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb667c16740 1 --2- 172.21.15.76:0/1534256562 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55d37721bb40 0x55d37721df60 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb667c16740 1 --2- 172.21.15.76:0/1534256562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55d3771a6b70 0x55d377214200 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb667c16740 1 --2- 172.21.15.76:0/1534256562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3771a6250 0x55d377213cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb6467fc640 0 client.17652 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:22.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb667c16740 1 -- 172.21.15.76:0/1534256562 >> 172.21.15.76:0/1534256562 conn(0x55d3771a0980 msgr2=0x55d3771a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb667c16740 1 -- 172.21.15.76:0/1534256562 shutdown_connections 2024-03-12T06:05:22.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb667c16740 1 -- 172.21.15.76:0/1534256562 wait complete. 2024-03-12T06:05:22.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb667c16740 20 client.17652 trim_cache size 0 max 16384 2024-03-12T06:05:22.714 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:05:22.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb665049640 1 -- 172.21.15.76:0/4042184896 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb64c0a1ba0 msgr2=0x7fb64c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb665049640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb64c0a1ba0 0x7fb64c0a4060 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb658005cc0 tx=0x7fb65800b8c0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb665049640 1 -- 172.21.15.76:0/4042184896 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6601022e0 msgr2=0x7fb6601977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:22.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.692+0000 7fb665049640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6601022e0 0x7fb6601977a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fb650005940 tx=0x7fb650004040 comp rx=0 tx=0).stop 2024-03-12T06:05:22.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.693+0000 7fb665049640 1 -- 172.21.15.76:0/4042184896 shutdown_connections 2024-03-12T06:05:22.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.693+0000 7fb665049640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb64c0a1ba0 0x7fb64c0a4060 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.693+0000 7fb665049640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb66010b700 0x7fb66019c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.693+0000 7fb665049640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb660102c00 0x7fb660197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.693+0000 7fb665049640 1 --2- 172.21.15.76:0/4042184896 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb6601022e0 0x7fb6601977a0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.693+0000 7fb665049640 1 -- 172.21.15.76:0/4042184896 >> 172.21.15.76:0/4042184896 conn(0x7fb6600fbae0 msgr2=0x7fb6600fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.693+0000 7fb665049640 1 -- 172.21.15.76:0/4042184896 shutdown_connections 2024-03-12T06:05:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:22.693+0000 7fb665049640 1 -- 172.21.15.76:0/4042184896 wait complete. 2024-03-12T06:05:22.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:22 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2620602984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:05:23.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:23 smithi082 ceph-mon[36382]: pgmap v297: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:05:23.717 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-12T06:05:23.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:05:23.718 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-12T06:05:23.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:23 smithi088 ceph-mon[36585]: pgmap v297: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:05:23.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:23 smithi076 ceph-mon[32095]: pgmap v297: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:05:23.834 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.827+0000 7ff633e02640 1 -- 172.21.15.76:0/2312159255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff62c106450 msgr2=0x7ff62c10e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.827+0000 7ff633e02640 1 --2- 172.21.15.76:0/2312159255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff62c106450 0x7ff62c10e020 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7ff624009cd0 tx=0x7ff62402c0f0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.827+0000 7ff633e02640 1 -- 172.21.15.76:0/2312159255 shutdown_connections 2024-03-12T06:05:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.827+0000 7ff633e02640 1 --2- 172.21.15.76:0/2312159255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff62c106450 0x7ff62c10e020 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.827+0000 7ff633e02640 1 --2- 172.21.15.76:0/2312159255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff62c101d30 0x7ff62c105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.827+0000 7ff633e02640 1 --2- 172.21.15.76:0/2312159255 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff62c101380 0x7ff62c101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.827+0000 7ff633e02640 1 -- 172.21.15.76:0/2312159255 >> 172.21.15.76:0/2312159255 conn(0x7ff62c0fcc20 msgr2=0x7ff62c0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.827+0000 7ff633e02640 1 -- 172.21.15.76:0/2312159255 shutdown_connections 2024-03-12T06:05:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.827+0000 7ff633e02640 1 -- 172.21.15.76:0/2312159255 wait complete. 2024-03-12T06:05:23.835 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.828+0000 7ff633e02640 1 Processor -- start 2024-03-12T06:05:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.828+0000 7ff633e02640 1 -- start start 2024-03-12T06:05:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.828+0000 7ff633e02640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff62c101380 0x7ff62c19d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.828+0000 7ff633e02640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff62c106450 0x7ff62c19d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.828+0000 7ff633e02640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff62c1975f0 0x7ff62c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.828+0000 7ff633e02640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c197fe0 con 0x7ff62c1975f0 2024-03-12T06:05:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.828+0000 7ff633e02640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c198150 con 0x7ff62c101380 2024-03-12T06:05:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.828+0000 7ff633e02640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c198310 con 0x7ff62c106450 2024-03-12T06:05:23.836 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.828+0000 7ff63239f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff62c1975f0 0x7ff62c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.828+0000 7ff63239f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff62c1975f0 0x7ff62c197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57928/0 (socket says 172.21.15.76:57928) 2024-03-12T06:05:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.828+0000 7ff63239f640 1 -- 172.21.15.76:0/3379467163 learned_addr learned my addr 172.21.15.76:0/3379467163 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.828+0000 7ff631b9e640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff62c101380 0x7ff62c19d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.829+0000 7ff63239f640 1 -- 172.21.15.76:0/3379467163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff62c106450 msgr2=0x7ff62c19d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.829+0000 7ff63139d640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff62c106450 0x7ff62c19d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.829+0000 7ff63239f640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff62c106450 0x7ff62c19d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.837 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.829+0000 7ff63239f640 1 -- 172.21.15.76:0/3379467163 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff62c101380 msgr2=0x7ff62c19d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.829+0000 7ff63239f640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff62c101380 0x7ff62c19d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.829+0000 7ff63239f640 1 -- 172.21.15.76:0/3379467163 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff624009900 con 0x7ff62c1975f0 2024-03-12T06:05:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.829+0000 7ff63139d640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff62c106450 0x7ff62c19d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:05:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.829+0000 7ff631b9e640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff62c101380 0x7ff62c19d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:05:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.829+0000 7ff63239f640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff62c1975f0 0x7ff62c197aa0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7ff624037040 tx=0x7ff624034d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.829+0000 7ff622ffd640 1 -- 172.21.15.76:0/3379467163 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff624032ac0 con 0x7ff62c1975f0 2024-03-12T06:05:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.829+0000 7ff633e02640 1 -- 172.21.15.76:0/3379467163 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62c1a4300 con 0x7ff62c1975f0 2024-03-12T06:05:23.838 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.829+0000 7ff622ffd640 1 -- 172.21.15.76:0/3379467163 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff624003e50 con 0x7ff62c1975f0 2024-03-12T06:05:23.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.829+0000 7ff622ffd640 1 -- 172.21.15.76:0/3379467163 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6240406b0 con 0x7ff62c1975f0 2024-03-12T06:05:23.839 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.829+0000 7ff633e02640 1 -- 172.21.15.76:0/3379467163 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62c1a4760 con 0x7ff62c1975f0 2024-03-12T06:05:23.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.831+0000 7ff622ffd640 1 -- 172.21.15.76:0/3379467163 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff62400a040 con 0x7ff62c1975f0 2024-03-12T06:05:23.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff6369ce740 1 Processor -- start 2024-03-12T06:05:23.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff6369ce740 1 -- start start 2024-03-12T06:05:23.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff6369ce740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533100 0x55a7505334e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff622ffd640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff6180a1ba0 0x7ff6180a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff6369ce740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533ab0 0x55a7505406f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.846 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff6369ce740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750540cc0 0x55a7505430d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff6369ce740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7504a1f90 con 0x55a750533100 2024-03-12T06:05:23.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff6369ce740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a75049f4c0 con 0x55a750540cc0 2024-03-12T06:05:23.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff6369ce740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a75049aca0 con 0x55a750533ab0 2024-03-12T06:05:23.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff63139d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533100 0x55a7505334e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff631b9e640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff6180a1ba0 0x7ff6180a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff63139d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533100 0x55a7505334e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57936/0 (socket says 172.21.15.76:57936) 2024-03-12T06:05:23.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff63139d640 1 -- 172.21.15.76:0/1487483247 learned_addr learned my addr 172.21.15.76:0/1487483247 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:23.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff622ffd640 1 -- 172.21.15.76:0/3379467163 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7ff6240f07b0 con 0x7ff62c1975f0 2024-03-12T06:05:23.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff63239f640 1 --2- 172.21.15.76:0/1487483247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750540cc0 0x55a7505430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.847 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.832+0000 7ff63139d640 1 --2- 172.21.15.76:0/1487483247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533ab0 0x55a7505406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.833+0000 7ff63139d640 1 -- 172.21.15.76:0/1487483247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533ab0 msgr2=0x55a7505406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.833+0000 7ff63139d640 1 --2- 172.21.15.76:0/1487483247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533ab0 0x55a7505406f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.833+0000 7ff63139d640 1 -- 172.21.15.76:0/1487483247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750540cc0 msgr2=0x55a7505430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.833+0000 7ff63139d640 1 --2- 172.21.15.76:0/1487483247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750540cc0 0x55a7505430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.833+0000 7ff63139d640 1 -- 172.21.15.76:0/1487483247 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a75045d0b0 con 0x55a750533100 2024-03-12T06:05:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.833+0000 7ff631b9e640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff6180a1ba0 0x7ff6180a4060 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff62800baa0 tx=0x7ff628005d90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.833+0000 7ff63139d640 1 --2- 172.21.15.76:0/1487483247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533100 0x55a7505334e0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00aee0 tx=0x7ff61c00a3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8060c0149e4452a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.833+0000 7ff617fff640 1 -- 172.21.15.76:0/1487483247 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c00ed80 con 0x55a750533100 2024-03-12T06:05:23.848 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.833+0000 7ff617fff640 1 -- 172.21.15.76:0/1487483247 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff61c00c070 con 0x55a750533100 2024-03-12T06:05:23.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.833+0000 7ff617fff640 1 -- 172.21.15.76:0/1487483247 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c013600 con 0x55a750533100 2024-03-12T06:05:23.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.834+0000 7ff6369ce740 1 -- 172.21.15.76:0/1487483247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533100 msgr2=0x55a7505334e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:23.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.834+0000 7ff6369ce740 1 --2- 172.21.15.76:0/1487483247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533100 0x55a7505334e0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00aee0 tx=0x7ff61c00a3b0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.834+0000 7ff6369ce740 1 -- 172.21.15.76:0/1487483247 shutdown_connections 2024-03-12T06:05:23.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.834+0000 7ff6369ce740 1 --2- 172.21.15.76:0/1487483247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750540cc0 0x55a7505430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.834+0000 7ff6369ce740 1 --2- 172.21.15.76:0/1487483247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533ab0 0x55a7505406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.834+0000 7ff6369ce740 1 --2- 172.21.15.76:0/1487483247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533100 0x55a7505334e0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.834+0000 7ff6369ce740 1 -- 172.21.15.76:0/1487483247 >> 172.21.15.76:0/1487483247 conn(0x55a75052d840 msgr2=0x55a75052dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:23.849 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.834+0000 7ff6369ce740 1 -- 172.21.15.76:0/1487483247 shutdown_connections 2024-03-12T06:05:23.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.834+0000 7ff6369ce740 1 -- 172.21.15.76:0/1487483247 wait complete. 2024-03-12T06:05:23.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.834+0000 7ff6369ce740 1 -- start start 2024-03-12T06:05:23.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.834+0000 7ff6369ce740 1 Processor -- start 2024-03-12T06:05:23.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff6369ce740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533100 0x55a75053dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff6369ce740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533ab0 0x55a75053e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff6369ce740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750540cc0 0x55a75053e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff6369ce740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a75043ff40 con 0x55a750533ab0 2024-03-12T06:05:23.850 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff6369ce740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a75016e4b0 con 0x55a750540cc0 2024-03-12T06:05:23.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff6369ce740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a750420670 con 0x55a750533100 2024-03-12T06:05:23.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff6369ce740 10 client.0 _ll_register_callbacks cb 0x55a75048df70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:23.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff63139d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533ab0 0x55a75053e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff63139d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533ab0 0x55a75053e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57952/0 (socket says 172.21.15.76:57952) 2024-03-12T06:05:23.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff63139d640 1 -- 172.21.15.76:0/3287098850 learned_addr learned my addr 172.21.15.76:0/3287098850 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:23.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff63239f640 1 --2- 172.21.15.76:0/3287098850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750540cc0 0x55a75053e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff631b9e640 1 --2- 172.21.15.76:0/3287098850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533100 0x55a75053dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff63139d640 1 -- 172.21.15.76:0/3287098850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533100 msgr2=0x55a75053dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:23.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff63139d640 1 --2- 172.21.15.76:0/3287098850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533100 0x55a75053dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff63139d640 1 -- 172.21.15.76:0/3287098850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750540cc0 msgr2=0x55a75053e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:23.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff63139d640 1 --2- 172.21.15.76:0/3287098850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750540cc0 0x55a75053e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff63139d640 1 -- 172.21.15.76:0/3287098850 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61c00aae0 con 0x55a750533ab0 2024-03-12T06:05:23.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff63139d640 1 --2- 172.21.15.76:0/3287098850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533ab0 0x55a75053e1f0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00aeb0 tx=0x7ff61c016ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd68cba7608cdad server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:23.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff6157fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:23.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff6157fa640 1 -- 172.21.15.76:0/3287098850 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c016980 con 0x55a750533ab0 2024-03-12T06:05:23.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff6157fa640 1 -- 172.21.15.76:0/3287098850 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7ff61c016ae0 con 0x55a750533ab0 2024-03-12T06:05:23.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.835+0000 7ff6369ce740 1 -- 172.21.15.76:0/3287098850 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a750474f40 con 0x55a750533ab0 2024-03-12T06:05:23.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.836+0000 7ff6157fa640 1 -- 172.21.15.76:0/3287098850 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff61c01f430 con 0x55a750533ab0 2024-03-12T06:05:23.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.836+0000 7ff6369ce740 10 client.17664 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:23.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.836+0000 7ff6369ce740 1 -- 172.21.15.76:0/3287098850 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a750499320 con 0x55a750533ab0 2024-03-12T06:05:23.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.836+0000 7ff6157fa640 1 -- 172.21.15.76:0/3287098850 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a750474f40 con 0x55a750533ab0 2024-03-12T06:05:23.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.836+0000 7ff6157fa640 1 -- 172.21.15.76:0/3287098850 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff61c01e660 con 0x55a750533ab0 2024-03-12T06:05:23.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.836+0000 7ff6369ce740 10 client.17664 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:23.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.836+0000 7ff6369ce740 10 client.17664 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:05:23.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.836+0000 7ff6369ce740 20 client.17664 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:23.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.836+0000 7ff6369ce740 1 --2- 172.21.15.76:0/3287098850 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a750551200 0x55a7505535f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.836+0000 7ff6369ce740 4 client.17664 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:05:23.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.836+0000 7ff6369ce740 1 -- 172.21.15.76:0/3287098850 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a750491310 con 0x55a750551200 2024-03-12T06:05:23.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.836+0000 7ff631b9e640 1 --2- 172.21.15.76:0/3287098850 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a750551200 0x55a7505535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.837+0000 7ff6157fa640 10 client.17664 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:23.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.837+0000 7ff631b9e640 1 --2- 172.21.15.76:0/3287098850 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a750551200 0x55a7505535f0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6931a8fa51d47a69 server_cookie=d9f3eba0d218ef98 in_seq=0 out_seq=0 2024-03-12T06:05:23.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.837+0000 7ff6157fa640 1 -- 172.21.15.76:0/3287098850 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a750491310 con 0x55a750551200 2024-03-12T06:05:23.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.837+0000 7ff6157fa640 10 client.17664 handle_command_reply: tid=0 2024-03-12T06:05:23.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.837+0000 7ff6369ce740 1 client.17664 shutdown 2024-03-12T06:05:23.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.837+0000 7ff6369ce740 1 -- 172.21.15.76:0/3287098850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533ab0 msgr2=0x55a75053e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:23.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.837+0000 7ff6369ce740 1 --2- 172.21.15.76:0/3287098850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533ab0 0x55a75053e1f0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00aeb0 tx=0x7ff61c016ec0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.837+0000 7ff6369ce740 1 -- 172.21.15.76:0/3287098850 shutdown_connections 2024-03-12T06:05:23.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.837+0000 7ff6369ce740 1 --2- 172.21.15.76:0/3287098850 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750540cc0 0x55a75053e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.837+0000 7ff6369ce740 1 --2- 172.21.15.76:0/3287098850 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a750551200 0x55a7505535f0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.837+0000 7ff6369ce740 1 --2- 172.21.15.76:0/3287098850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533ab0 0x55a75053e1f0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.838+0000 7ff6369ce740 1 --2- 172.21.15.76:0/3287098850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533100 0x55a75053dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.838+0000 7ff6157fa640 0 client.17664 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:23.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.838+0000 7ff6369ce740 1 -- 172.21.15.76:0/3287098850 >> 172.21.15.76:0/3287098850 conn(0x55a75052d840 msgr2=0x55a75052f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:23.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.838+0000 7ff6369ce740 1 -- 172.21.15.76:0/3287098850 shutdown_connections 2024-03-12T06:05:23.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.838+0000 7ff6369ce740 1 -- 172.21.15.76:0/3287098850 wait complete. 2024-03-12T06:05:23.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.838+0000 7ff6369ce740 20 client.17664 trim_cache size 0 max 16384 2024-03-12T06:05:23.862 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff6369ce740 1 Processor -- start 2024-03-12T06:05:23.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff6369ce740 1 -- start start 2024-03-12T06:05:23.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff6369ce740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533100 0x55a7505403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff6369ce740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533ab0 0x55a75053e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff6369ce740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750540cc0 0x55a750538f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff6369ce740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7505340c0 con 0x55a750533100 2024-03-12T06:05:23.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff6369ce740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a750532410 con 0x55a750540cc0 2024-03-12T06:05:23.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff6369ce740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a75052d060 con 0x55a750533ab0 2024-03-12T06:05:23.863 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff631b9e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533100 0x55a7505403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff631b9e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533100 0x55a7505403e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57966/0 (socket says 172.21.15.76:57966) 2024-03-12T06:05:23.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff631b9e640 1 -- 172.21.15.76:0/3901798932 learned_addr learned my addr 172.21.15.76:0/3901798932 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:23.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff63139d640 1 --2- 172.21.15.76:0/3901798932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533ab0 0x55a75053e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff631b9e640 1 -- 172.21.15.76:0/3901798932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533ab0 msgr2=0x55a75053e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:23.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff631b9e640 1 --2- 172.21.15.76:0/3901798932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533ab0 0x55a75053e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff631b9e640 1 -- 172.21.15.76:0/3901798932 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750540cc0 msgr2=0x55a750538f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:23.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff631b9e640 1 --2- 172.21.15.76:0/3901798932 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750540cc0 0x55a750538f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff631b9e640 1 -- 172.21.15.76:0/3901798932 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a750435c00 con 0x55a750533100 2024-03-12T06:05:23.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff63139d640 1 --2- 172.21.15.76:0/3901798932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533ab0 0x55a75053e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:05:23.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.853+0000 7ff631b9e640 1 --2- 172.21.15.76:0/3901798932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533100 0x55a7505403e0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x55a750553f00 tx=0x7ff628013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a68959066ded30 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:23.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.854+0000 7ff6177fe640 1 -- 172.21.15.76:0/3901798932 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff628025520 con 0x55a750533100 2024-03-12T06:05:23.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.854+0000 7ff6177fe640 1 -- 172.21.15.76:0/3901798932 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff628025680 con 0x55a750533100 2024-03-12T06:05:23.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.854+0000 7ff6177fe640 1 -- 172.21.15.76:0/3901798932 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62801c680 con 0x55a750533100 2024-03-12T06:05:23.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.854+0000 7ff6369ce740 1 -- 172.21.15.76:0/3901798932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533100 msgr2=0x55a7505403e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:23.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.854+0000 7ff6369ce740 1 --2- 172.21.15.76:0/3901798932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533100 0x55a7505403e0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x55a750553f00 tx=0x7ff628013000 comp rx=0 tx=0).stop 2024-03-12T06:05:23.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.854+0000 7ff6369ce740 1 -- 172.21.15.76:0/3901798932 shutdown_connections 2024-03-12T06:05:23.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.854+0000 7ff6369ce740 1 --2- 172.21.15.76:0/3901798932 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750540cc0 0x55a750538f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.854+0000 7ff6369ce740 1 --2- 172.21.15.76:0/3901798932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533ab0 0x55a75053e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.854+0000 7ff6369ce740 1 --2- 172.21.15.76:0/3901798932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750533100 0x55a7505403e0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.854+0000 7ff6369ce740 1 -- 172.21.15.76:0/3901798932 >> 172.21.15.76:0/3901798932 conn(0x55a75052d840 msgr2=0x55a75052dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:23.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.854+0000 7ff6369ce740 1 -- 172.21.15.76:0/3901798932 shutdown_connections 2024-03-12T06:05:23.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.854+0000 7ff6369ce740 1 -- 172.21.15.76:0/3901798932 wait complete. 2024-03-12T06:05:23.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff6369ce740 1 -- start start 2024-03-12T06:05:23.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff6369ce740 1 Processor -- start 2024-03-12T06:05:23.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff6369ce740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533100 0x55a750546a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff6369ce740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750533ab0 0x55a750546fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff6369ce740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750540cc0 0x55a7505a76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff6369ce740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7505477e0 con 0x55a750540cc0 2024-03-12T06:05:23.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff6369ce740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a750547950 con 0x55a750533ab0 2024-03-12T06:05:23.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff6369ce740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a7505a7ca0 con 0x55a750533100 2024-03-12T06:05:23.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff6369ce740 10 client.0 _ll_register_callbacks cb 0x7ff61c00a210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:23.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff63239f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750540cc0 0x55a7505a76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff63239f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750540cc0 0x55a7505a76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:57968/0 (socket says 172.21.15.76:57968) 2024-03-12T06:05:23.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff63239f640 1 -- 172.21.15.76:0/186471760 learned_addr learned my addr 172.21.15.76:0/186471760 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:23.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff631b9e640 1 --2- 172.21.15.76:0/186471760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533100 0x55a750546a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff63139d640 1 --2- 172.21.15.76:0/186471760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750533ab0 0x55a750546fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff63239f640 1 -- 172.21.15.76:0/186471760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533100 msgr2=0x55a750546a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:23.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff63239f640 1 --2- 172.21.15.76:0/186471760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533100 0x55a750546a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff63239f640 1 -- 172.21.15.76:0/186471760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750533ab0 msgr2=0x55a750546fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:23.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff63239f640 1 --2- 172.21.15.76:0/186471760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750533ab0 0x55a750546fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff63239f640 1 -- 172.21.15.76:0/186471760 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62800da80 con 0x55a750540cc0 2024-03-12T06:05:23.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.855+0000 7ff63139d640 1 --2- 172.21.15.76:0/186471760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750533ab0 0x55a750546fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:05:23.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff63239f640 1 --2- 172.21.15.76:0/186471760 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750540cc0 0x55a7505a76e0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7ff6240ee0c0 tx=0x7ff6240bcf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61868da00b89494 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:23.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6157fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:23.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6157fa640 1 -- 172.21.15.76:0/186471760 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6240c1640 con 0x55a750540cc0 2024-03-12T06:05:23.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6369ce740 1 -- 172.21.15.76:0/186471760 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a7504c0900 con 0x55a750540cc0 2024-03-12T06:05:23.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6157fa640 1 -- 172.21.15.76:0/186471760 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7ff6240c5c90 con 0x55a750540cc0 2024-03-12T06:05:23.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6157fa640 1 -- 172.21.15.76:0/186471760 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6240bb7f0 con 0x55a750540cc0 2024-03-12T06:05:23.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6369ce740 10 client.17670 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:23.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6369ce740 1 -- 172.21.15.76:0/186471760 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a75045b910 con 0x55a750540cc0 2024-03-12T06:05:23.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6157fa640 1 -- 172.21.15.76:0/186471760 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a7504c0900 con 0x55a750540cc0 2024-03-12T06:05:23.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6157fa640 1 -- 172.21.15.76:0/186471760 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff62409c130 con 0x55a750540cc0 2024-03-12T06:05:23.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6369ce740 10 client.17670 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:23.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6369ce740 10 client.17670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:05:23.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6369ce740 20 client.17670 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:23.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6369ce740 1 --2- 172.21.15.76:0/186471760 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a7505a9380 0x55a7505ab770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:23.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6369ce740 4 client.17670 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-12T06:05:23.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.856+0000 7ff6369ce740 1 -- 172.21.15.76:0/186471760 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a75019a860 con 0x55a7505a9380 2024-03-12T06:05:23.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.857+0000 7ff631b9e640 1 --2- 172.21.15.76:0/186471760 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a7505a9380 0x55a7505ab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:23.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.857+0000 7ff631b9e640 1 --2- 172.21.15.76:0/186471760 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a7505a9380 0x55a7505ab770 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=599034492e721c37 server_cookie=fa6ff978310a995b in_seq=0 out_seq=0 2024-03-12T06:05:23.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.857+0000 7ff6157fa640 10 client.17670 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:23.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.858+0000 7ff6157fa640 1 -- 172.21.15.76:0/186471760 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a75019a860 con 0x55a7505a9380 2024-03-12T06:05:23.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.858+0000 7ff6157fa640 10 client.17670 handle_command_reply: tid=0 2024-03-12T06:05:23.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.858+0000 7ff6369ce740 1 client.17670 shutdown 2024-03-12T06:05:23.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.858+0000 7ff6369ce740 1 -- 172.21.15.76:0/186471760 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750540cc0 msgr2=0x55a7505a76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:23.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.858+0000 7ff6369ce740 1 --2- 172.21.15.76:0/186471760 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750540cc0 0x55a7505a76e0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7ff6240ee0c0 tx=0x7ff6240bcf10 comp rx=0 tx=0).stop 2024-03-12T06:05:23.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.858+0000 7ff6369ce740 1 -- 172.21.15.76:0/186471760 shutdown_connections 2024-03-12T06:05:23.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.858+0000 7ff6369ce740 1 --2- 172.21.15.76:0/186471760 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x55a7505a9380 0x55a7505ab770 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.858+0000 7ff6369ce740 1 --2- 172.21.15.76:0/186471760 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55a750540cc0 0x55a7505a76e0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.858+0000 7ff6369ce740 1 --2- 172.21.15.76:0/186471760 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55a750533ab0 0x55a750546fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.858+0000 7ff6157fa640 0 client.17670 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:23.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.858+0000 7ff6369ce740 1 --2- 172.21.15.76:0/186471760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a750533100 0x55a750546a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.858+0000 7ff6369ce740 1 -- 172.21.15.76:0/186471760 >> 172.21.15.76:0/186471760 conn(0x55a75052d840 msgr2=0x55a750538f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:23.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff6369ce740 1 -- 172.21.15.76:0/186471760 shutdown_connections 2024-03-12T06:05:23.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff6369ce740 1 -- 172.21.15.76:0/186471760 wait complete. 2024-03-12T06:05:23.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff6369ce740 20 client.17670 trim_cache size 0 max 16384 2024-03-12T06:05:23.873 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:05:23.873 INFO:teuthology.orchestra.run.smithi076.stdout: "return_code": 0, 2024-03-12T06:05:23.874 INFO:teuthology.orchestra.run.smithi076.stdout: "scrub_tag": "b715f94a-9851-4cf8-bd78-b88217769d58", 2024-03-12T06:05:23.874 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": "asynchronous" 2024-03-12T06:05:23.874 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:05:23.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff633e02640 1 -- 172.21.15.76:0/3379467163 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff6180a1ba0 msgr2=0x7ff6180a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:23.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff633e02640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff6180a1ba0 0x7ff6180a4060 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff62800baa0 tx=0x7ff628005d90 comp rx=0 tx=0).stop 2024-03-12T06:05:23.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff633e02640 1 -- 172.21.15.76:0/3379467163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff62c1975f0 msgr2=0x7ff62c197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:23.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff633e02640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff62c1975f0 0x7ff62c197aa0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7ff624037040 tx=0x7ff624034d30 comp rx=0 tx=0).stop 2024-03-12T06:05:23.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff633e02640 1 -- 172.21.15.76:0/3379467163 shutdown_connections 2024-03-12T06:05:23.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff633e02640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff6180a1ba0 0x7ff6180a4060 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff633e02640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff62c1975f0 0x7ff62c197aa0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff633e02640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff62c106450 0x7ff62c19d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff633e02640 1 --2- 172.21.15.76:0/3379467163 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff62c101380 0x7ff62c19d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:23.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff633e02640 1 -- 172.21.15.76:0/3379467163 >> 172.21.15.76:0/3379467163 conn(0x7ff62c0fcc20 msgr2=0x7ff62c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:23.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff633e02640 1 -- 172.21.15.76:0/3379467163 shutdown_connections 2024-03-12T06:05:23.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:23.859+0000 7ff633e02640 1 -- 172.21.15.76:0/3379467163 wait complete. 2024-03-12T06:05:23.877 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:05:23.877 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:05:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.042+0000 7f7c72cdf640 1 -- 172.21.15.76:0/1167843376 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c6c100c10 msgr2=0x7f7c6c104c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.042+0000 7f7c72cdf640 1 --2- 172.21.15.76:0/1167843376 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c6c100c10 0x7f7c6c104c50 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009860 tx=0x7f7c5c02bb70 comp rx=0 tx=0).stop 2024-03-12T06:05:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.042+0000 7f7c72cdf640 1 -- 172.21.15.76:0/1167843376 shutdown_connections 2024-03-12T06:05:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.043+0000 7f7c72cdf640 1 --2- 172.21.15.76:0/1167843376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c6c105330 0x7f7c6c068450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.043+0000 7f7c72cdf640 1 --2- 172.21.15.76:0/1167843376 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c6c100c10 0x7f7c6c104c50 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.049 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.043+0000 7f7c72cdf640 1 --2- 172.21.15.76:0/1167843376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c6c100260 0x7f7c6c100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.043+0000 7f7c72cdf640 1 -- 172.21.15.76:0/1167843376 >> 172.21.15.76:0/1167843376 conn(0x7f7c6c077180 msgr2=0x7f7c6c077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.043+0000 7f7c72cdf640 1 -- 172.21.15.76:0/1167843376 shutdown_connections 2024-03-12T06:05:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.043+0000 7f7c72cdf640 1 -- 172.21.15.76:0/1167843376 wait complete. 2024-03-12T06:05:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.043+0000 7f7c72cdf640 1 Processor -- start 2024-03-12T06:05:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.043+0000 7f7c72cdf640 1 -- start start 2024-03-12T06:05:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c72cdf640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c6c100260 0x7f7c6c06c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c72cdf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c6c105330 0x7f7c6c06c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c72cdf640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c6c06cba0 0x7f7c6c070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c72cdf640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c0710b0 con 0x7f7c6c100260 2024-03-12T06:05:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c72cdf640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c071220 con 0x7f7c6c06cba0 2024-03-12T06:05:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c72cdf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c0713e0 con 0x7f7c6c105330 2024-03-12T06:05:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c7127c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c6c06cba0 0x7f7c6c070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c7127c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c6c06cba0 0x7f7c6c070b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.76:45032/0 (socket says 172.21.15.76:45032) 2024-03-12T06:05:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c7127c640 1 -- 172.21.15.76:0/3902709707 learned_addr learned my addr 172.21.15.76:0/3902709707 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c70a7b640 1 --2- 172.21.15.76:0/3902709707 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c6c100260 0x7f7c6c06c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c63fff640 1 --2- 172.21.15.76:0/3902709707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c6c105330 0x7f7c6c06c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c7127c640 1 -- 172.21.15.76:0/3902709707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c6c105330 msgr2=0x7f7c6c06c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c7127c640 1 --2- 172.21.15.76:0/3902709707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c6c105330 0x7f7c6c06c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c7127c640 1 -- 172.21.15.76:0/3902709707 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c6c100260 msgr2=0x7f7c6c06c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c7127c640 1 --2- 172.21.15.76:0/3902709707 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c6c100260 0x7f7c6c06c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.044+0000 7f7c7127c640 1 -- 172.21.15.76:0/3902709707 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c009490 con 0x7f7c6c06cba0 2024-03-12T06:05:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.045+0000 7f7c7127c640 1 --2- 172.21.15.76:0/3902709707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c6c06cba0 0x7f7c6c070b70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400eaa0 tx=0x7f7c6400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.045+0000 7f7c61ffb640 1 -- 172.21.15.76:0/3902709707 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c64004d20 con 0x7f7c6c06cba0 2024-03-12T06:05:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.045+0000 7f7c72cdf640 1 -- 172.21.15.76:0/3902709707 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6c0716c0 con 0x7f7c6c06cba0 2024-03-12T06:05:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.045+0000 7f7c61ffb640 1 -- 172.21.15.76:0/3902709707 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c64004e80 con 0x7f7c6c06cba0 2024-03-12T06:05:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.045+0000 7f7c61ffb640 1 -- 172.21.15.76:0/3902709707 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c64005a10 con 0x7f7c6c06cba0 2024-03-12T06:05:24.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.045+0000 7f7c72cdf640 1 -- 172.21.15.76:0/3902709707 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6c0719c0 con 0x7f7c6c06cba0 2024-03-12T06:05:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.047+0000 7f7c61ffb640 1 -- 172.21.15.76:0/3902709707 <== mon.2 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f7c64005020 con 0x7f7c6c06cba0 2024-03-12T06:05:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c75991740 1 Processor -- start 2024-03-12T06:05:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c61ffb640 1 --2- 172.21.15.76:0/3902709707 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7c540a1ba0 0x7f7c540a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c75991740 1 -- start start 2024-03-12T06:05:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c63fff640 1 --2- 172.21.15.76:0/3902709707 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7c540a1ba0 0x7f7c540a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c61ffb640 1 -- 172.21.15.76:0/3902709707 <== mon.2 v2:172.21.15.82:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7c640cd990 con 0x7f7c6c06cba0 2024-03-12T06:05:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c75991740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf3732b0 0x556fbf373690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c75991740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf373bd0 0x556fbf37c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c75991740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf37c7b0 0x556fbf37eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c75991740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fbf2e2280 con 0x556fbf3732b0 2024-03-12T06:05:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c75991740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fbf2df7b0 con 0x556fbf373bd0 2024-03-12T06:05:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c75991740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fbf2daf90 con 0x556fbf37c7b0 2024-03-12T06:05:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c63fff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf373bd0 0x556fbf37c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c70a7b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf3732b0 0x556fbf373690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c7127c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf37c7b0 0x556fbf37eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c7127c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf37c7b0 0x556fbf37eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.76:36084/0 (socket says 172.21.15.76:36084) 2024-03-12T06:05:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.048+0000 7f7c7127c640 1 -- 172.21.15.76:0/206240911 learned_addr learned my addr 172.21.15.76:0/206240911 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.049+0000 7f7c63fff640 1 --2- 172.21.15.76:0/3902709707 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7c540a1ba0 0x7f7c540a4060 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009830 tx=0x7f7c5c034d50 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:24.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.049+0000 7f7c63fff640 1 -- 172.21.15.76:0/206240911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf37c7b0 msgr2=0x556fbf37eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.049+0000 7f7c63fff640 1 --2- 172.21.15.76:0/206240911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf37c7b0 0x556fbf37eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.049+0000 7f7c63fff640 1 -- 172.21.15.76:0/206240911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf3732b0 msgr2=0x556fbf373690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.049+0000 7f7c63fff640 1 --2- 172.21.15.76:0/206240911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf3732b0 0x556fbf373690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.049+0000 7f7c63fff640 1 -- 172.21.15.76:0/206240911 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fbf299570 con 0x556fbf373bd0 2024-03-12T06:05:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.049+0000 7f7c63fff640 1 --2- 172.21.15.76:0/206240911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf373bd0 0x556fbf37c270 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c02c0b0 tx=0x7f7c5c046000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82bd812fc3e4f7e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.049+0000 7f7c52ffd640 1 -- 172.21.15.76:0/206240911 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c034440 con 0x556fbf373bd0 2024-03-12T06:05:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.049+0000 7f7c52ffd640 1 -- 172.21.15.76:0/206240911 <== mon.2 v2:172.21.15.82:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c5c02c510 con 0x556fbf373bd0 2024-03-12T06:05:24.058 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.049+0000 7f7c52ffd640 1 -- 172.21.15.76:0/206240911 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0037f0 con 0x556fbf373bd0 2024-03-12T06:05:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.050+0000 7f7c75991740 1 -- 172.21.15.76:0/206240911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf373bd0 msgr2=0x556fbf37c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.050+0000 7f7c75991740 1 --2- 172.21.15.76:0/206240911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf373bd0 0x556fbf37c270 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c02c0b0 tx=0x7f7c5c046000 comp rx=0 tx=0).stop 2024-03-12T06:05:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.050+0000 7f7c75991740 1 -- 172.21.15.76:0/206240911 shutdown_connections 2024-03-12T06:05:24.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.050+0000 7f7c75991740 1 --2- 172.21.15.76:0/206240911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf37c7b0 0x556fbf37eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.050+0000 7f7c75991740 1 --2- 172.21.15.76:0/206240911 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf373bd0 0x556fbf37c270 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.050+0000 7f7c75991740 1 --2- 172.21.15.76:0/206240911 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf3732b0 0x556fbf373690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.050+0000 7f7c75991740 1 -- 172.21.15.76:0/206240911 >> 172.21.15.76:0/206240911 conn(0x556fbf36d990 msgr2=0x556fbf36dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.050+0000 7f7c75991740 1 -- 172.21.15.76:0/206240911 shutdown_connections 2024-03-12T06:05:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.050+0000 7f7c75991740 1 -- 172.21.15.76:0/206240911 wait complete. 2024-03-12T06:05:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.050+0000 7f7c75991740 1 -- start start 2024-03-12T06:05:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.050+0000 7f7c75991740 1 Processor -- start 2024-03-12T06:05:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c75991740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf3732b0 0x556fbf37c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c75991740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf373bd0 0x556fbf3795d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c75991740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf37c7b0 0x556fbf379cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c75991740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fbefae430 con 0x556fbf3732b0 2024-03-12T06:05:24.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c75991740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fbef6ae00 con 0x556fbf37c7b0 2024-03-12T06:05:24.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c75991740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fbf25ffd0 con 0x556fbf373bd0 2024-03-12T06:05:24.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c75991740 10 client.0 _ll_register_callbacks cb 0x556fbf2b5fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:24.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c70a7b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf3732b0 0x556fbf37c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c70a7b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf3732b0 0x556fbf37c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58006/0 (socket says 172.21.15.76:58006) 2024-03-12T06:05:24.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c70a7b640 1 -- 172.21.15.76:0/1014300000 learned_addr learned my addr 172.21.15.76:0/1014300000 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:24.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c7127c640 1 --2- 172.21.15.76:0/1014300000 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf37c7b0 0x556fbf379cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.065 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c63fff640 1 --2- 172.21.15.76:0/1014300000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf373bd0 0x556fbf3795d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c70a7b640 1 -- 172.21.15.76:0/1014300000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf373bd0 msgr2=0x556fbf3795d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:24.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c70a7b640 1 --2- 172.21.15.76:0/1014300000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf373bd0 0x556fbf3795d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c70a7b640 1 -- 172.21.15.76:0/1014300000 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf37c7b0 msgr2=0x556fbf379cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:24.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c70a7b640 1 --2- 172.21.15.76:0/1014300000 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf37c7b0 0x556fbf379cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.051+0000 7f7c70a7b640 1 -- 172.21.15.76:0/1014300000 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6c068d10 con 0x556fbf3732b0 2024-03-12T06:05:24.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.052+0000 7f7c70a7b640 1 --2- 172.21.15.76:0/1014300000 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf3732b0 0x556fbf37c240 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7c68012840 tx=0x7f7c68012c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8119339f36eaae3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:24.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.052+0000 7f7c4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:24.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.052+0000 7f7c4bfff640 1 -- 172.21.15.76:0/1014300000 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c68017070 con 0x556fbf3732b0 2024-03-12T06:05:24.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.052+0000 7f7c4bfff640 1 -- 172.21.15.76:0/1014300000 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7c6800f920 con 0x556fbf3732b0 2024-03-12T06:05:24.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.052+0000 7f7c75991740 1 -- 172.21.15.76:0/1014300000 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556fbefda7e0 con 0x556fbf3732b0 2024-03-12T06:05:24.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.052+0000 7f7c4bfff640 1 -- 172.21.15.76:0/1014300000 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c680122e0 con 0x556fbf3732b0 2024-03-12T06:05:24.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.052+0000 7f7c75991740 10 client.17682 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:24.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.052+0000 7f7c75991740 1 -- 172.21.15.76:0/1014300000 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556fbf2d9610 con 0x556fbf3732b0 2024-03-12T06:05:24.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.052+0000 7f7c4bfff640 1 -- 172.21.15.76:0/1014300000 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c68028080 con 0x556fbf3732b0 2024-03-12T06:05:24.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.053+0000 7f7c4bfff640 1 -- 172.21.15.76:0/1014300000 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7c68022030 con 0x556fbf3732b0 2024-03-12T06:05:24.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.053+0000 7f7c75991740 10 client.17682 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:24.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.053+0000 7f7c75991740 10 client.17682 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:05:24.067 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.053+0000 7f7c75991740 20 client.17682 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:24.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.053+0000 7f7c75991740 1 --2- 172.21.15.76:0/1014300000 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556fbf38ca00 0x556fbf38edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.053+0000 7f7c75991740 4 client.17682 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:05:24.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.053+0000 7f7c75991740 1 -- 172.21.15.76:0/1014300000 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556fbf017720 con 0x556fbf38ca00 2024-03-12T06:05:24.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.053+0000 7f7c63fff640 1 --2- 172.21.15.76:0/1014300000 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556fbf38ca00 0x556fbf38edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.054+0000 7f7c63fff640 1 --2- 172.21.15.76:0/1014300000 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556fbf38ca00 0x556fbf38edf0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbe4d26ce7d66f06 server_cookie=dad87d2d0aa5a6fa in_seq=0 out_seq=0 2024-03-12T06:05:24.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.054+0000 7f7c4bfff640 10 client.17682 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:24.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.054+0000 7f7c4bfff640 1 -- 172.21.15.76:0/1014300000 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556fbf017720 con 0x556fbf38ca00 2024-03-12T06:05:24.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.054+0000 7f7c4bfff640 10 client.17682 handle_command_reply: tid=0 2024-03-12T06:05:24.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.054+0000 7f7c75991740 1 client.17682 shutdown 2024-03-12T06:05:24.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.055+0000 7f7c75991740 1 -- 172.21.15.76:0/1014300000 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf3732b0 msgr2=0x556fbf37c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:24.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.055+0000 7f7c75991740 1 --2- 172.21.15.76:0/1014300000 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf3732b0 0x556fbf37c240 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7c68012840 tx=0x7f7c68012c40 comp rx=0 tx=0).stop 2024-03-12T06:05:24.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.055+0000 7f7c75991740 1 -- 172.21.15.76:0/1014300000 shutdown_connections 2024-03-12T06:05:24.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.055+0000 7f7c75991740 1 --2- 172.21.15.76:0/1014300000 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf37c7b0 0x556fbf379cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.055+0000 7f7c75991740 1 --2- 172.21.15.76:0/1014300000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf373bd0 0x556fbf3795d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.055+0000 7f7c75991740 1 --2- 172.21.15.76:0/1014300000 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556fbf38ca00 0x556fbf38edf0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.055+0000 7f7c75991740 1 --2- 172.21.15.76:0/1014300000 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf3732b0 0x556fbf37c240 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.055+0000 7f7c75991740 1 -- 172.21.15.76:0/1014300000 >> 172.21.15.76:0/1014300000 conn(0x556fbf36d990 msgr2=0x556fbf36f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:24.069 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.055+0000 7f7c4bfff640 0 client.17682 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:24.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.055+0000 7f7c75991740 1 -- 172.21.15.76:0/1014300000 shutdown_connections 2024-03-12T06:05:24.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.055+0000 7f7c75991740 1 -- 172.21.15.76:0/1014300000 wait complete. 2024-03-12T06:05:24.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.055+0000 7f7c75991740 20 client.17682 trim_cache size 0 max 16384 2024-03-12T06:05:24.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.076+0000 7f7c75991740 1 Processor -- start 2024-03-12T06:05:24.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.076+0000 7f7c75991740 1 -- start start 2024-03-12T06:05:24.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.076+0000 7f7c75991740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf3732b0 0x556fbf3749b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.076+0000 7f7c75991740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf373bd0 0x556fbf393720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.076+0000 7f7c75991740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf37c7b0 0x556fbf37bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.076+0000 7f7c75991740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fbf36e9e0 con 0x556fbf373bd0 2024-03-12T06:05:24.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.076+0000 7f7c75991740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fbf3692b0 con 0x556fbf37c7b0 2024-03-12T06:05:24.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.076+0000 7f7c75991740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fbf377b20 con 0x556fbf3732b0 2024-03-12T06:05:24.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.076+0000 7f7c63fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf373bd0 0x556fbf393720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.076+0000 7f7c63fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf373bd0 0x556fbf393720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58016/0 (socket says 172.21.15.76:58016) 2024-03-12T06:05:24.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.076+0000 7f7c63fff640 1 -- 172.21.15.76:0/3750298843 learned_addr learned my addr 172.21.15.76:0/3750298843 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:24.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.076+0000 7f7c7127c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf37c7b0 0x556fbf37bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.076+0000 7f7c70a7b640 1 --2- 172.21.15.76:0/3750298843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf3732b0 0x556fbf3749b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.077+0000 7f7c63fff640 1 -- 172.21.15.76:0/3750298843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf3732b0 msgr2=0x556fbf3749b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:24.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.077+0000 7f7c63fff640 1 --2- 172.21.15.76:0/3750298843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf3732b0 0x556fbf3749b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.077+0000 7f7c63fff640 1 -- 172.21.15.76:0/3750298843 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf37c7b0 msgr2=0x556fbf37bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:24.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.077+0000 7f7c63fff640 1 --2- 172.21.15.76:0/3750298843 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf37c7b0 0x556fbf37bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.077+0000 7f7c63fff640 1 -- 172.21.15.76:0/3750298843 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fbf275ac0 con 0x556fbf373bd0 2024-03-12T06:05:24.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.077+0000 7f7c63fff640 1 --2- 172.21.15.76:0/3750298843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf373bd0 0x556fbf393720 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c02c0b0 tx=0x7f7c5c046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5adbcc88bd349e66 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:24.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.077+0000 7f7c527fc640 1 -- 172.21.15.76:0/3750298843 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c05bd70 con 0x556fbf373bd0 2024-03-12T06:05:24.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.077+0000 7f7c527fc640 1 -- 172.21.15.76:0/3750298843 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c5c004490 con 0x556fbf373bd0 2024-03-12T06:05:24.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.077+0000 7f7c527fc640 1 -- 172.21.15.76:0/3750298843 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c05b930 con 0x556fbf373bd0 2024-03-12T06:05:24.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.078+0000 7f7c75991740 1 -- 172.21.15.76:0/3750298843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf373bd0 msgr2=0x556fbf393720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:24.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.078+0000 7f7c75991740 1 --2- 172.21.15.76:0/3750298843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf373bd0 0x556fbf393720 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c02c0b0 tx=0x7f7c5c046000 comp rx=0 tx=0).stop 2024-03-12T06:05:24.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.078+0000 7f7c75991740 1 -- 172.21.15.76:0/3750298843 shutdown_connections 2024-03-12T06:05:24.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.078+0000 7f7c75991740 1 --2- 172.21.15.76:0/3750298843 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf37c7b0 0x556fbf37bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.078+0000 7f7c75991740 1 --2- 172.21.15.76:0/3750298843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf373bd0 0x556fbf393720 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.078+0000 7f7c75991740 1 --2- 172.21.15.76:0/3750298843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf3732b0 0x556fbf3749b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.078+0000 7f7c75991740 1 -- 172.21.15.76:0/3750298843 >> 172.21.15.76:0/3750298843 conn(0x556fbf36d990 msgr2=0x556fbf36dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:24.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.078+0000 7f7c75991740 1 -- 172.21.15.76:0/3750298843 shutdown_connections 2024-03-12T06:05:24.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.078+0000 7f7c75991740 1 -- 172.21.15.76:0/3750298843 wait complete. 2024-03-12T06:05:24.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.078+0000 7f7c75991740 1 -- start start 2024-03-12T06:05:24.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.079+0000 7f7c75991740 1 Processor -- start 2024-03-12T06:05:24.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.079+0000 7f7c75991740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf3732b0 0x556fbf381fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.079+0000 7f7c75991740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf373bd0 0x556fbf382520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.079+0000 7f7c75991740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf37c7b0 0x556fbf382a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.079+0000 7f7c75991740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fbf383ee0 con 0x556fbf37c7b0 2024-03-12T06:05:24.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.079+0000 7f7c75991740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fbf3e2eb0 con 0x556fbf373bd0 2024-03-12T06:05:24.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.079+0000 7f7c75991740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fbf3e3040 con 0x556fbf3732b0 2024-03-12T06:05:24.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.079+0000 7f7c75991740 10 client.0 _ll_register_callbacks cb 0x7f7c68005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:24.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.079+0000 7f7c7127c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf37c7b0 0x556fbf382a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.079+0000 7f7c7127c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf37c7b0 0x556fbf382a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:58026/0 (socket says 172.21.15.76:58026) 2024-03-12T06:05:24.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.079+0000 7f7c7127c640 1 -- 172.21.15.76:0/3227646843 learned_addr learned my addr 172.21.15.76:0/3227646843 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:24.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.079+0000 7f7c63fff640 1 --2- 172.21.15.76:0/3227646843 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf373bd0 0x556fbf382520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.079+0000 7f7c70a7b640 1 --2- 172.21.15.76:0/3227646843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf3732b0 0x556fbf381fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.080+0000 7f7c63fff640 1 -- 172.21.15.76:0/3227646843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf3732b0 msgr2=0x556fbf381fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:24.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.080+0000 7f7c63fff640 1 --2- 172.21.15.76:0/3227646843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf3732b0 0x556fbf381fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.080+0000 7f7c63fff640 1 -- 172.21.15.76:0/3227646843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf37c7b0 msgr2=0x556fbf382a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:24.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.080+0000 7f7c63fff640 1 --2- 172.21.15.76:0/3227646843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf37c7b0 0x556fbf382a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.080+0000 7f7c63fff640 1 -- 172.21.15.76:0/3227646843 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556fbf299570 con 0x556fbf373bd0 2024-03-12T06:05:24.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.080+0000 7f7c63fff640 1 --2- 172.21.15.76:0/3227646843 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf373bd0 0x556fbf382520 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x556fbf3895b0 tx=0x7f7c5c0041f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b0d1ceec79969f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:24.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.080+0000 7f7c4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-03-12T06:05:24.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.080+0000 7f7c4bfff640 1 -- 172.21.15.76:0/3227646843 <== mon.2 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c02ca90 con 0x556fbf373bd0 2024-03-12T06:05:24.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.080+0000 7f7c4bfff640 1 -- 172.21.15.76:0/3227646843 <== mon.2 v2:172.21.15.82:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7c5c059070 con 0x556fbf373bd0 2024-03-12T06:05:24.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.080+0000 7f7c75991740 1 -- 172.21.15.76:0/3227646843 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c5c02c510 con 0x556fbf373bd0 2024-03-12T06:05:24.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.080+0000 7f7c4bfff640 1 -- 172.21.15.76:0/3227646843 <== mon.2 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c5c034440 con 0x556fbf373bd0 2024-03-12T06:05:24.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.080+0000 7f7c75991740 10 client.26657 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:24.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.080+0000 7f7c75991740 1 -- 172.21.15.76:0/3227646843 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556fbf29b6a0 con 0x556fbf373bd0 2024-03-12T06:05:24.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.080+0000 7f7c4bfff640 1 -- 172.21.15.76:0/3227646843 <== mon.2 v2:172.21.15.82:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c5c02c510 con 0x556fbf373bd0 2024-03-12T06:05:24.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.081+0000 7f7c4bfff640 1 -- 172.21.15.76:0/3227646843 <== mon.2 v2:172.21.15.82:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7c5c062030 con 0x556fbf373bd0 2024-03-12T06:05:24.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.081+0000 7f7c75991740 10 client.26657 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:24.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.081+0000 7f7c75991740 10 client.26657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:05:24.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.081+0000 7f7c75991740 20 client.26657 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:24.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.081+0000 7f7c75991740 1 --2- 172.21.15.76:0/3227646843 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556fbf3e4450 0x556fbf3e6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:24.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.081+0000 7f7c75991740 4 client.26657 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:05:24.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.081+0000 7f7c75991740 1 -- 172.21.15.76:0/3227646843 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556fbf066890 con 0x556fbf3e4450 2024-03-12T06:05:24.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.081+0000 7f7c70a7b640 1 --2- 172.21.15.76:0/3227646843 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556fbf3e4450 0x556fbf3e6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:24.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.081+0000 7f7c4bfff640 10 client.26657 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:24.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.081+0000 7f7c70a7b640 1 --2- 172.21.15.76:0/3227646843 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556fbf3e4450 0x556fbf3e6870 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbe8198a362c6189 server_cookie=169854867b499b35 in_seq=0 out_seq=0 2024-03-12T06:05:24.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c4bfff640 1 -- 172.21.15.76:0/3227646843 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x556fbf066890 con 0x556fbf3e4450 2024-03-12T06:05:24.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c4bfff640 10 client.26657 handle_command_reply: tid=0 2024-03-12T06:05:24.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c75991740 1 client.26657 shutdown 2024-03-12T06:05:24.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c75991740 1 -- 172.21.15.76:0/3227646843 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf373bd0 msgr2=0x556fbf382520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:24.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c75991740 1 --2- 172.21.15.76:0/3227646843 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf373bd0 0x556fbf382520 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x556fbf3895b0 tx=0x7f7c5c0041f0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c75991740 1 -- 172.21.15.76:0/3227646843 shutdown_connections 2024-03-12T06:05:24.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c75991740 1 --2- 172.21.15.76:0/3227646843 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x556fbf3e4450 0x556fbf3e6870 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c75991740 1 --2- 172.21.15.76:0/3227646843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x556fbf37c7b0 0x556fbf382a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c4bfff640 0 client.26657 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:24.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c75991740 1 --2- 172.21.15.76:0/3227646843 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x556fbf373bd0 0x556fbf382520 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c75991740 1 --2- 172.21.15.76:0/3227646843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556fbf3732b0 0x556fbf381fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c75991740 1 -- 172.21.15.76:0/3227646843 >> 172.21.15.76:0/3227646843 conn(0x556fbf36d990 msgr2=0x556fbf377d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:24.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c75991740 1 -- 172.21.15.76:0/3227646843 shutdown_connections 2024-03-12T06:05:24.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c75991740 1 -- 172.21.15.76:0/3227646843 wait complete. 2024-03-12T06:05:24.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.082+0000 7f7c75991740 20 client.26657 trim_cache size 0 max 16384 2024-03-12T06:05:24.097 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:05:24.097 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-12T06:05:24.097 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": { 2024-03-12T06:05:24.097 INFO:teuthology.orchestra.run.smithi076.stdout: "b715f94a-9851-4cf8-bd78-b88217769d58": { 2024-03-12T06:05:24.097 INFO:teuthology.orchestra.run.smithi076.stdout: "path": "/", 2024-03-12T06:05:24.097 INFO:teuthology.orchestra.run.smithi076.stdout: "tag": "b715f94a-9851-4cf8-bd78-b88217769d58", 2024-03-12T06:05:24.097 INFO:teuthology.orchestra.run.smithi076.stdout: "options": "recursive,force" 2024-03-12T06:05:24.097 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:24.098 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:24.098 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:05:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.083+0000 7f7c72cdf640 1 -- 172.21.15.76:0/3902709707 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7c540a1ba0 msgr2=0x7f7c540a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.083+0000 7f7c72cdf640 1 --2- 172.21.15.76:0/3902709707 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7c540a1ba0 0x7f7c540a4060 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c009830 tx=0x7f7c5c034d50 comp rx=0 tx=0).stop 2024-03-12T06:05:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.083+0000 7f7c72cdf640 1 -- 172.21.15.76:0/3902709707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c6c06cba0 msgr2=0x7f7c6c070b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.083+0000 7f7c72cdf640 1 --2- 172.21.15.76:0/3902709707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c6c06cba0 0x7f7c6c070b70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400eaa0 tx=0x7f7c6400eea0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.083+0000 7f7c72cdf640 1 -- 172.21.15.76:0/3902709707 shutdown_connections 2024-03-12T06:05:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.083+0000 7f7c72cdf640 1 --2- 172.21.15.76:0/3902709707 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f7c540a1ba0 0x7f7c540a4060 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.083+0000 7f7c72cdf640 1 --2- 172.21.15.76:0/3902709707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c6c06cba0 0x7f7c6c070b70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.083+0000 7f7c72cdf640 1 --2- 172.21.15.76:0/3902709707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c6c105330 0x7f7c6c06c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.083+0000 7f7c72cdf640 1 --2- 172.21.15.76:0/3902709707 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c6c100260 0x7f7c6c06c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:24.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.083+0000 7f7c72cdf640 1 -- 172.21.15.76:0/3902709707 >> 172.21.15.76:0/3902709707 conn(0x7f7c6c077180 msgr2=0x7f7c6c103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:24.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.083+0000 7f7c72cdf640 1 -- 172.21.15.76:0/3902709707 shutdown_connections 2024-03-12T06:05:24.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:24.083+0000 7f7c72cdf640 1 -- 172.21.15.76:0/3902709707 wait complete. 2024-03-12T06:05:24.101 INFO:tasks.cephfs.filesystem:scrub status for tag:b715f94a-9851-4cf8-bd78-b88217769d58 - {'path': '/', 'tag': 'b715f94a-9851-4cf8-bd78-b88217769d58', 'options': 'recursive,force'} 2024-03-12T06:05:24.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:24 smithi076 ceph-mon[32095]: scrub queued for path: / 2024-03-12T06:05:24.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:24 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:05:24.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:24 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:05:24.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:24 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:05:24.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:24 smithi076 ceph-mon[32095]: scrub summary: active paths [/] 2024-03-12T06:05:24.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:24 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:05:24.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:24 smithi082 ceph-mon[36382]: scrub queued for path: / 2024-03-12T06:05:24.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:24 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:05:24.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:24 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:05:24.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:24 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:05:24.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:24 smithi082 ceph-mon[36382]: scrub summary: active paths [/] 2024-03-12T06:05:24.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:24 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:05:24.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:24 smithi088 ceph-mon[36585]: scrub queued for path: / 2024-03-12T06:05:24.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:24 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:05:24.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:24 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:05:24.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:24 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:05:24.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:24 smithi088 ceph-mon[36585]: scrub summary: active paths [/] 2024-03-12T06:05:24.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:24 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:05:25.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:25 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:05:25.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:25 smithi082 ceph-mon[36382]: scrub summary: idle+waiting paths [/] 2024-03-12T06:05:25.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:25 smithi082 ceph-mon[36382]: pgmap v298: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:25.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:25 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:05:25.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:25 smithi088 ceph-mon[36585]: scrub summary: idle+waiting paths [/] 2024-03-12T06:05:25.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:25 smithi088 ceph-mon[36585]: pgmap v298: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:25.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:25 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:05:25.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:25 smithi076 ceph-mon[32095]: scrub summary: idle+waiting paths [/] 2024-03-12T06:05:25.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:25 smithi076 ceph-mon[32095]: pgmap v298: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:27.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:27 smithi082 ceph-mon[36382]: pgmap v299: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:27.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:27 smithi088 ceph-mon[36585]: pgmap v299: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:27.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:27 smithi076 ceph-mon[32095]: pgmap v299: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:29.498 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:29 smithi082 ceph-mon[36382]: pgmap v300: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:05:29.538 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:29 smithi088 ceph-mon[36585]: pgmap v300: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:05:29.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:29 smithi076 ceph-mon[32095]: pgmap v300: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:05:31.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:31 smithi082 ceph-mon[36382]: pgmap v301: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:05:31.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:31 smithi088 ceph-mon[36585]: pgmap v301: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:05:31.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:31 smithi076 ceph-mon[32095]: pgmap v301: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:05:33.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:33 smithi082 ceph-mon[36382]: pgmap v302: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:05:33.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:33 smithi088 ceph-mon[36585]: pgmap v302: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:05:33.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:33 smithi076 ceph-mon[32095]: pgmap v302: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:05:35.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:35 smithi082 ceph-mon[36382]: pgmap v303: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:05:35.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:35 smithi088 ceph-mon[36585]: pgmap v303: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:05:35.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:35 smithi076 ceph-mon[32095]: pgmap v303: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:05:37.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:37 smithi082 ceph-mon[36382]: pgmap v304: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:05:37.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:37 smithi088 ceph-mon[36585]: pgmap v304: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:05:37.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:37 smithi076 ceph-mon[32095]: pgmap v304: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-12T06:05:38.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:38 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:05:38.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:38 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:05:38.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:38 smithi082 ceph-mon[36382]: scrub summary: idle 2024-03-12T06:05:38.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:38 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:05:38.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:38 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:05:38.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:38 smithi088 ceph-mon[36585]: scrub summary: idle 2024-03-12T06:05:38.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:38 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:05:38.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:38 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:05:38.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:38 smithi076 ceph-mon[32095]: scrub summary: idle 2024-03-12T06:05:39.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:39 smithi082 ceph-mon[36382]: pgmap v305: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:05:39.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:39 smithi088 ceph-mon[36585]: pgmap v305: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:05:39.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:39 smithi076 ceph-mon[32095]: pgmap v305: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-12T06:05:41.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:41 smithi082 ceph-mon[36382]: pgmap v306: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:41.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:41 smithi088 ceph-mon[36585]: pgmap v306: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:41.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:41 smithi076 ceph-mon[32095]: pgmap v306: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:43.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:43 smithi082 ceph-mon[36382]: pgmap v307: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:05:43.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:43 smithi088 ceph-mon[36585]: pgmap v307: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:05:43.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:43 smithi076 ceph-mon[32095]: pgmap v307: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:05:45.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:45 smithi082 ceph-mon[36382]: pgmap v308: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:45.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:45 smithi088 ceph-mon[36585]: pgmap v308: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:45.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:45 smithi076 ceph-mon[32095]: pgmap v308: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:47.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:47 smithi082 ceph-mon[36382]: pgmap v309: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:47.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:47 smithi088 ceph-mon[36585]: pgmap v309: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:47.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:47 smithi076 ceph-mon[32095]: pgmap v309: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:49.107 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-12T06:05:49.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.220+0000 7ff1367c9640 1 -- 172.21.15.76:0/4172234284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff13010db20 msgr2=0x7ff13010ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:49.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.220+0000 7ff1367c9640 1 --2- 172.21.15.76:0/4172234284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff13010db20 0x7ff13010ff10 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7ff128009ca0 tx=0x7ff12802c0f0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.220+0000 7ff1367c9640 1 -- 172.21.15.76:0/4172234284 shutdown_connections 2024-03-12T06:05:49.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.220+0000 7ff1367c9640 1 --2- 172.21.15.76:0/4172234284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff13010db20 0x7ff13010ff10 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.220+0000 7ff1367c9640 1 --2- 172.21.15.76:0/4172234284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff130069340 0x7ff13010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.220+0000 7ff1367c9640 1 --2- 172.21.15.76:0/4172234284 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff130068a20 0x7ff130068e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.220+0000 7ff1367c9640 1 -- 172.21.15.76:0/4172234284 >> 172.21.15.76:0/4172234284 conn(0x7ff130077170 msgr2=0x7ff130077580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:49.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.220+0000 7ff1367c9640 1 -- 172.21.15.76:0/4172234284 shutdown_connections 2024-03-12T06:05:49.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.220+0000 7ff1367c9640 1 -- 172.21.15.76:0/4172234284 wait complete. 2024-03-12T06:05:49.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.221+0000 7ff1367c9640 1 Processor -- start 2024-03-12T06:05:49.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.221+0000 7ff1367c9640 1 -- start start 2024-03-12T06:05:49.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.221+0000 7ff1367c9640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff130068a20 0x7ff13019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:49.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.221+0000 7ff1367c9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff130069340 0x7ff13019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:49.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.221+0000 7ff1367c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff13010db20 0x7ff1301997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:49.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.221+0000 7ff1367c9640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff130199f30 con 0x7ff130069340 2024-03-12T06:05:49.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.221+0000 7ff1367c9640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13019a0a0 con 0x7ff130068a20 2024-03-12T06:05:49.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.221+0000 7ff1367c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13019a260 con 0x7ff13010db20 2024-03-12T06:05:49.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff12f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff130069340 0x7ff13019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:49.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff12f7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff130069340 0x7ff13019fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52574/0 (socket says 172.21.15.76:52574) 2024-03-12T06:05:49.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff12f7fe640 1 -- 172.21.15.76:0/216689549 learned_addr learned my addr 172.21.15.76:0/216689549 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:49.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff12ffff640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff130068a20 0x7ff13019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:49.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff134d66640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff13010db20 0x7ff1301997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:49.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff12f7fe640 1 -- 172.21.15.76:0/216689549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff13010db20 msgr2=0x7ff1301997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:49.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff12f7fe640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff13010db20 0x7ff1301997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff12f7fe640 1 -- 172.21.15.76:0/216689549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff130068a20 msgr2=0x7ff13019f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:49.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff12f7fe640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff130068a20 0x7ff13019f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff12f7fe640 1 -- 172.21.15.76:0/216689549 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff128009900 con 0x7ff130069340 2024-03-12T06:05:49.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff134d66640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff13010db20 0x7ff1301997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:05:49.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff12f7fe640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff130069340 0x7ff13019fbd0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ff12400d950 tx=0x7ff12400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:49.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff12d7fa640 1 -- 172.21.15.76:0/216689549 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff124009a30 con 0x7ff130069340 2024-03-12T06:05:49.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff12d7fa640 1 -- 172.21.15.76:0/216689549 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff124009b90 con 0x7ff130069340 2024-03-12T06:05:49.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff1367c9640 1 -- 172.21.15.76:0/216689549 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13019a540 con 0x7ff130069340 2024-03-12T06:05:49.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff12d7fa640 1 -- 172.21.15.76:0/216689549 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff124004a00 con 0x7ff130069340 2024-03-12T06:05:49.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.222+0000 7ff1367c9640 1 -- 172.21.15.76:0/216689549 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1301a6610 con 0x7ff130069340 2024-03-12T06:05:49.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.224+0000 7ff1367c9640 1 -- 172.21.15.76:0/216689549 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff130100650 con 0x7ff130069340 2024-03-12T06:05:49.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.225+0000 7ff12d7fa640 1 -- 172.21.15.76:0/216689549 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff124009d00 con 0x7ff130069340 2024-03-12T06:05:49.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.225+0000 7ff12d7fa640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff1180a1c10 0x7ff1180a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:49.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.226+0000 7ff12ffff640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff1180a1c10 0x7ff1180a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:49.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.226+0000 7ff12d7fa640 1 -- 172.21.15.76:0/216689549 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7ff1240cd120 con 0x7ff130069340 2024-03-12T06:05:49.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.226+0000 7ff12ffff640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff1180a1c10 0x7ff1180a40d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff12000bbe0 tx=0x7ff12000b390 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:49.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.230+0000 7ff12d7fa640 1 -- 172.21.15.76:0/216689549 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff124095ff0 con 0x7ff130069340 2024-03-12T06:05:49.432 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.426+0000 7ff1367c9640 1 -- 172.21.15.76:0/216689549 --> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7ff130113440 con 0x7ff1180a1c10 2024-03-12T06:05:49.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.433+0000 7ff12d7fa640 1 -- 172.21.15.76:0/216689549 <== mgr.15465 v2:172.21.15.76:6800/513615067 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7ff130113440 con 0x7ff1180a1c10 2024-03-12T06:05:49.439 INFO:teuthology.orchestra.run.smithi076.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-12T00:00:00", "created": "2024-03-12T05:55:48", "first": "2024-03-12T05:56:00", "last": "2024-03-12T06:05:00", "last_pruned": "2024-03-12T06:04:00", "created_count": 10, "pruned_count": 3, "active": true} 2024-03-12T06:05:49.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.435+0000 7ff1367c9640 1 -- 172.21.15.76:0/216689549 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff1180a1c10 msgr2=0x7ff1180a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:49.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.435+0000 7ff1367c9640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff1180a1c10 0x7ff1180a40d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff12000bbe0 tx=0x7ff12000b390 comp rx=0 tx=0).stop 2024-03-12T06:05:49.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.435+0000 7ff1367c9640 1 -- 172.21.15.76:0/216689549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff130069340 msgr2=0x7ff13019fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:49.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.435+0000 7ff1367c9640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff130069340 0x7ff13019fbd0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ff12400d950 tx=0x7ff12400de10 comp rx=0 tx=0).stop 2024-03-12T06:05:49.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.435+0000 7ff1367c9640 1 -- 172.21.15.76:0/216689549 shutdown_connections 2024-03-12T06:05:49.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.435+0000 7ff1367c9640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff1180a1c10 0x7ff1180a40d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.435+0000 7ff1367c9640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff13010db20 0x7ff1301997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.435+0000 7ff1367c9640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff130069340 0x7ff13019fbd0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.435+0000 7ff1367c9640 1 --2- 172.21.15.76:0/216689549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff130068a20 0x7ff13019f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.435+0000 7ff1367c9640 1 -- 172.21.15.76:0/216689549 >> 172.21.15.76:0/216689549 conn(0x7ff130077170 msgr2=0x7ff13010ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:49.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.435+0000 7ff1367c9640 1 -- 172.21.15.76:0/216689549 shutdown_connections 2024-03-12T06:05:49.443 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.435+0000 7ff1367c9640 1 -- 172.21.15.76:0/216689549 wait complete. 2024-03-12T06:05:49.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:49 smithi076 ceph-mon[32095]: pgmap v310: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:05:49.452 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-12T06:05:49.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.549+0000 7f69b3ec3640 1 -- 172.21.15.76:0/3521215818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69ac1003f0 msgr2=0x7f69ac10dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:49.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.549+0000 7f69b3ec3640 1 --2- 172.21.15.76:0/3521215818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69ac1003f0 0x7f69ac10dff0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f699c009cd0 tx=0x7f699c02c180 comp rx=0 tx=0).stop 2024-03-12T06:05:49.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.549+0000 7f69b3ec3640 1 -- 172.21.15.76:0/3521215818 shutdown_connections 2024-03-12T06:05:49.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.549+0000 7f69b3ec3640 1 --2- 172.21.15.76:0/3521215818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69ac1003f0 0x7f69ac10dff0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.549+0000 7f69b3ec3640 1 --2- 172.21.15.76:0/3521215818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ac0ffa30 0x7f69ac0ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.558 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.549+0000 7f69b3ec3640 1 --2- 172.21.15.76:0/3521215818 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69ac107370 0x7f69ac107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.549+0000 7f69b3ec3640 1 -- 172.21.15.76:0/3521215818 >> 172.21.15.76:0/3521215818 conn(0x7f69ac075320 msgr2=0x7f69ac075730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:49.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.549+0000 7f69b3ec3640 1 -- 172.21.15.76:0/3521215818 shutdown_connections 2024-03-12T06:05:49.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.549+0000 7f69b3ec3640 1 -- 172.21.15.76:0/3521215818 wait complete. 2024-03-12T06:05:49.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b3ec3640 1 Processor -- start 2024-03-12T06:05:49.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b3ec3640 1 -- start start 2024-03-12T06:05:49.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b3ec3640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69ac0ffa30 0x7f69ac19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:49.559 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b3ec3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69ac1003f0 0x7f69ac19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:49.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b3ec3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ac107370 0x7f69ac1975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:49.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b3ec3640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac197c40 con 0x7f69ac1003f0 2024-03-12T06:05:49.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b3ec3640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac197db0 con 0x7f69ac0ffa30 2024-03-12T06:05:49.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b3ec3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac197f70 con 0x7f69ac107370 2024-03-12T06:05:49.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b145e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69ac1003f0 0x7f69ac19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:49.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b145e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69ac1003f0 0x7f69ac19d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52606/0 (socket says 172.21.15.76:52606) 2024-03-12T06:05:49.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b145e640 1 -- 172.21.15.76:0/1263691755 learned_addr learned my addr 172.21.15.76:0/1263691755 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:49.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b1c5f640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69ac0ffa30 0x7f69ac19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:49.560 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b145e640 1 -- 172.21.15.76:0/1263691755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ac107370 msgr2=0x7f69ac1975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:49.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b2460640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ac107370 0x7f69ac1975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:49.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b145e640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ac107370 0x7f69ac1975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b145e640 1 -- 172.21.15.76:0/1263691755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69ac0ffa30 msgr2=0x7f69ac19d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:49.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b145e640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69ac0ffa30 0x7f69ac19d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b145e640 1 -- 172.21.15.76:0/1263691755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c009900 con 0x7f69ac1003f0 2024-03-12T06:05:49.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b1c5f640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69ac0ffa30 0x7f69ac19d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:05:49.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69b145e640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69ac1003f0 0x7f69ac19d930 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f69a000d770 tx=0x7f69a000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:49.561 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.550+0000 7f69aaffd640 1 -- 172.21.15.76:0/1263691755 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0009c30 con 0x7f69ac1003f0 2024-03-12T06:05:49.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.551+0000 7f69aaffd640 1 -- 172.21.15.76:0/1263691755 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69a000de80 con 0x7f69ac1003f0 2024-03-12T06:05:49.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.551+0000 7f69b3ec3640 1 -- 172.21.15.76:0/1263691755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ac198250 con 0x7f69ac1003f0 2024-03-12T06:05:49.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.551+0000 7f69aaffd640 1 -- 172.21.15.76:0/1263691755 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0004e80 con 0x7f69ac1003f0 2024-03-12T06:05:49.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.551+0000 7f69b3ec3640 1 -- 172.21.15.76:0/1263691755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ac1a4490 con 0x7f69ac1003f0 2024-03-12T06:05:49.562 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.552+0000 7f69b3ec3640 1 -- 172.21.15.76:0/1263691755 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ac04f730 con 0x7f69ac1003f0 2024-03-12T06:05:49.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.553+0000 7f69aaffd640 1 -- 172.21.15.76:0/1263691755 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f69a0005020 con 0x7f69ac1003f0 2024-03-12T06:05:49.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.553+0000 7f69aaffd640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f69940a1b90 0x7f69940a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:49.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.553+0000 7f69b1c5f640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f69940a1b90 0x7f69940a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:49.563 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.554+0000 7f69aaffd640 1 -- 172.21.15.76:0/1263691755 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f69a00cd2f0 con 0x7f69ac1003f0 2024-03-12T06:05:49.564 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.554+0000 7f69b1c5f640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f69940a1b90 0x7f69940a4050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f69a4009b40 tx=0x7f69a4005e50 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:49.565 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.558+0000 7f69aaffd640 1 -- 172.21.15.76:0/1263691755 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f69a00961c0 con 0x7f69ac1003f0 2024-03-12T06:05:49.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:49 smithi082 ceph-mon[36382]: pgmap v310: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:05:49.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:49 smithi088 ceph-mon[36585]: pgmap v310: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:05:49.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.751+0000 7f69b3ec3640 1 -- 172.21.15.76:0/1263691755 --> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f69ac111310 con 0x7f69940a1b90 2024-03-12T06:05:49.763 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.758+0000 7f69aaffd640 1 -- 172.21.15.76:0/1263691755 <== mgr.15465 v2:172.21.15.76:6800/513615067 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f69ac111310 con 0x7f69940a1b90 2024-03-12T06:05:49.764 INFO:teuthology.orchestra.run.smithi076.stdout:/ 1m 6m3h 2024-03-12T06:05:49.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.760+0000 7f69b3ec3640 1 -- 172.21.15.76:0/1263691755 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f69940a1b90 msgr2=0x7f69940a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:49.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.760+0000 7f69b3ec3640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f69940a1b90 0x7f69940a4050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f69a4009b40 tx=0x7f69a4005e50 comp rx=0 tx=0).stop 2024-03-12T06:05:49.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.760+0000 7f69b3ec3640 1 -- 172.21.15.76:0/1263691755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69ac1003f0 msgr2=0x7f69ac19d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:49.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.760+0000 7f69b3ec3640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69ac1003f0 0x7f69ac19d930 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f69a000d770 tx=0x7f69a000dc30 comp rx=0 tx=0).stop 2024-03-12T06:05:49.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.760+0000 7f69b3ec3640 1 -- 172.21.15.76:0/1263691755 shutdown_connections 2024-03-12T06:05:49.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.760+0000 7f69b3ec3640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f69940a1b90 0x7f69940a4050 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.760+0000 7f69b3ec3640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69ac107370 0x7f69ac1975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.760+0000 7f69b3ec3640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69ac1003f0 0x7f69ac19d930 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.760+0000 7f69b3ec3640 1 --2- 172.21.15.76:0/1263691755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69ac0ffa30 0x7f69ac19d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:49.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.760+0000 7f69b3ec3640 1 -- 172.21.15.76:0/1263691755 >> 172.21.15.76:0/1263691755 conn(0x7f69ac075320 msgr2=0x7f69ac0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:49.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.760+0000 7f69b3ec3640 1 -- 172.21.15.76:0/1263691755 shutdown_connections 2024-03-12T06:05:49.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:49.760+0000 7f69b3ec3640 1 -- 172.21.15.76:0/1263691755 wait complete. 2024-03-12T06:05:49.776 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-12T06:05:49.787 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-12T06:05:50.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:50 smithi082 ceph-mon[36382]: from='client.17694 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T06:05:50.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:50 smithi082 ceph-mon[36382]: from='client.17700 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T06:05:50.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:50 smithi088 ceph-mon[36585]: from='client.17694 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T06:05:50.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:50 smithi088 ceph-mon[36585]: from='client.17700 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T06:05:50.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:50 smithi076 ceph-mon[32095]: from='client.17694 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T06:05:50.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:50 smithi076 ceph-mon[32095]: from='client.17700 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T06:05:51.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:51 smithi082 ceph-mon[36382]: pgmap v311: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:51.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:51 smithi088 ceph-mon[36585]: pgmap v311: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:51.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:51 smithi076 ceph-mon[32095]: pgmap v311: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:05:53.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:53 smithi082 ceph-mon[36382]: pgmap v312: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T06:05:53.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:53 smithi088 ceph-mon[36585]: pgmap v312: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T06:05:53.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:53 smithi076 ceph-mon[32095]: pgmap v312: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T06:05:54.102 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-12T06:05:54.103 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-12T06:05:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.210+0000 7f55e2be2640 1 -- 172.21.15.76:0/3404982910 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55dc0fffb0 msgr2=0x7f55dc10dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.210+0000 7f55e2be2640 1 --2- 172.21.15.76:0/3404982910 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55dc0fffb0 0x7f55dc10dfe0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f55cc009930 tx=0x7f55cc02bc30 comp rx=0 tx=0).stop 2024-03-12T06:05:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.210+0000 7f55e2be2640 1 -- 172.21.15.76:0/3404982910 shutdown_connections 2024-03-12T06:05:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.210+0000 7f55e2be2640 1 --2- 172.21.15.76:0/3404982910 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55dc0fffb0 0x7f55dc10dfe0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.210+0000 7f55e2be2640 1 --2- 172.21.15.76:0/3404982910 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55dc0ff5f0 0x7f55dc0ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.210+0000 7f55e2be2640 1 --2- 172.21.15.76:0/3404982910 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55dc106130 0x7f55dc106510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.218 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.210+0000 7f55e2be2640 1 -- 172.21.15.76:0/3404982910 >> 172.21.15.76:0/3404982910 conn(0x7f55dc076570 msgr2=0x7f55dc0fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.210+0000 7f55e2be2640 1 -- 172.21.15.76:0/3404982910 shutdown_connections 2024-03-12T06:05:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.210+0000 7f55e2be2640 1 -- 172.21.15.76:0/3404982910 wait complete. 2024-03-12T06:05:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e2be2640 1 Processor -- start 2024-03-12T06:05:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e2be2640 1 -- start start 2024-03-12T06:05:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e2be2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55dc0ff5f0 0x7f55dc19d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e2be2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55dc0fffb0 0x7f55dc19daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e2be2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55dc106130 0x7f55dc197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e2be2640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc197e00 con 0x7f55dc106130 2024-03-12T06:05:54.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e2be2640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc197f70 con 0x7f55dc0fffb0 2024-03-12T06:05:54.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e2be2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc198130 con 0x7f55dc0ff5f0 2024-03-12T06:05:54.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e117f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55dc106130 0x7f55dc197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e117f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55dc106130 0x7f55dc197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52764/0 (socket says 172.21.15.76:52764) 2024-03-12T06:05:54.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e117f640 1 -- 172.21.15.76:0/3247673582 learned_addr learned my addr 172.21.15.76:0/3247673582 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e097e640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55dc0ff5f0 0x7f55dc19d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e117f640 1 -- 172.21.15.76:0/3247673582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55dc0ff5f0 msgr2=0x7f55dc19d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.220 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e117f640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55dc0ff5f0 0x7f55dc19d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e117f640 1 -- 172.21.15.76:0/3247673582 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55dc0fffb0 msgr2=0x7f55dc19daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55dbfff640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55dc0fffb0 0x7f55dc19daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e117f640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55dc0fffb0 0x7f55dc19daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e117f640 1 -- 172.21.15.76:0/3247673582 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc009560 con 0x7f55dc106130 2024-03-12T06:05:54.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e097e640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55dc0ff5f0 0x7f55dc19d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:54.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.211+0000 7f55e117f640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55dc106130 0x7f55dc197670 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f55cc02c170 tx=0x7f55cc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.212+0000 7f55d9ffb640 1 -- 172.21.15.76:0/3247673582 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc004440 con 0x7f55dc106130 2024-03-12T06:05:54.221 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.212+0000 7f55d9ffb640 1 -- 172.21.15.76:0/3247673582 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55cc0052d0 con 0x7f55dc106130 2024-03-12T06:05:54.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.212+0000 7f55e2be2640 1 -- 172.21.15.76:0/3247673582 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55dc1983b0 con 0x7f55dc106130 2024-03-12T06:05:54.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.212+0000 7f55d9ffb640 1 -- 172.21.15.76:0/3247673582 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc003d70 con 0x7f55dc106130 2024-03-12T06:05:54.222 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.212+0000 7f55e2be2640 1 -- 172.21.15.76:0/3247673582 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55dc1a4530 con 0x7f55dc106130 2024-03-12T06:05:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.214+0000 7f55d9ffb640 1 -- 172.21.15.76:0/3247673582 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f55cc005440 con 0x7f55dc106130 2024-03-12T06:05:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.214+0000 7f55e57b2740 1 Processor -- start 2024-03-12T06:05:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.214+0000 7f55e57b2740 1 -- start start 2024-03-12T06:05:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55d9ffb640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f55c80a1c20 0x7f55c80a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.229 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55e57b2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 0x5560f0513690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55e57b2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f051c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55e57b2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 0x5560f051eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55e57b2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f0482280 con 0x5560f0513bd0 2024-03-12T06:05:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55e57b2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f047f7b0 con 0x5560f051c7b0 2024-03-12T06:05:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55e57b2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f047af90 con 0x5560f05132b0 2024-03-12T06:05:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55d9ffb640 1 -- 172.21.15.76:0/3247673582 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f55cc0f0010 con 0x7f55dc106130 2024-03-12T06:05:54.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55dbfff640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f55c80a1c20 0x7f55c80a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f051c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f051c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52780/0 (socket says 172.21.15.76:52780) 2024-03-12T06:05:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55dbfff640 1 -- 172.21.15.76:0/1131851340 learned_addr learned my addr 172.21.15.76:0/1131851340 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55e117f640 1 --2- 172.21.15.76:0/1131851340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 0x5560f051eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55e097e640 1 --2- 172.21.15.76:0/1131851340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 0x5560f0513690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55dbfff640 1 -- 172.21.15.76:0/1131851340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 msgr2=0x5560f0513690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55dbfff640 1 --2- 172.21.15.76:0/1131851340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 0x5560f0513690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55dbfff640 1 -- 172.21.15.76:0/1131851340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 msgr2=0x5560f051eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55dbfff640 1 --2- 172.21.15.76:0/1131851340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 0x5560f051eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55dbfff640 1 -- 172.21.15.76:0/1131851340 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560f0439570 con 0x5560f0513bd0 2024-03-12T06:05:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55dbfff640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f55c80a1c20 0x7f55c80a40e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f55c40126a0 tx=0x7f55c400d970 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55dbfff640 1 --2- 172.21.15.76:0/1131851340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f051c270 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f55c400f840 tx=0x7f55c4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7ae39607ff5117 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.215+0000 7f55d2ffd640 1 -- 172.21.15.76:0/1131851340 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c401c8e0 con 0x5560f0513bd0 2024-03-12T06:05:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.216+0000 7f55d2ffd640 1 -- 172.21.15.76:0/1131851340 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55c4018040 con 0x5560f0513bd0 2024-03-12T06:05:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.216+0000 7f55d2ffd640 1 -- 172.21.15.76:0/1131851340 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c401ecc0 con 0x5560f0513bd0 2024-03-12T06:05:54.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.216+0000 7f55e57b2740 1 -- 172.21.15.76:0/1131851340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 msgr2=0x5560f051c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.216+0000 7f55e57b2740 1 --2- 172.21.15.76:0/1131851340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f051c270 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f55c400f840 tx=0x7f55c4019000 comp rx=0 tx=0).stop 2024-03-12T06:05:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.216+0000 7f55e57b2740 1 -- 172.21.15.76:0/1131851340 shutdown_connections 2024-03-12T06:05:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.216+0000 7f55e57b2740 1 --2- 172.21.15.76:0/1131851340 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 0x5560f051eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.216+0000 7f55e57b2740 1 --2- 172.21.15.76:0/1131851340 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f051c270 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.216+0000 7f55e57b2740 1 --2- 172.21.15.76:0/1131851340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 0x5560f0513690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.216+0000 7f55e57b2740 1 -- 172.21.15.76:0/1131851340 >> 172.21.15.76:0/1131851340 conn(0x5560f050d990 msgr2=0x5560f050dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.216+0000 7f55e57b2740 1 -- 172.21.15.76:0/1131851340 shutdown_connections 2024-03-12T06:05:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.216+0000 7f55e57b2740 1 -- 172.21.15.76:0/1131851340 wait complete. 2024-03-12T06:05:54.233 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.216+0000 7f55e57b2740 1 -- start start 2024-03-12T06:05:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e57b2740 1 Processor -- start 2024-03-12T06:05:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e57b2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f05132b0 0x5560f051c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e57b2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f0513bd0 0x5560f0519570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e57b2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f051c7b0 0x5560f0519cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e57b2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f014e430 con 0x5560f051c7b0 2024-03-12T06:05:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e57b2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f010ae00 con 0x5560f05132b0 2024-03-12T06:05:54.234 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e57b2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f03fffd0 con 0x5560f0513bd0 2024-03-12T06:05:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e57b2740 10 client.0 _ll_register_callbacks cb 0x5560f0455fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e117f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f051c7b0 0x5560f0519cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e117f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f051c7b0 0x5560f0519cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52792/0 (socket says 172.21.15.76:52792) 2024-03-12T06:05:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e117f640 1 -- 172.21.15.76:0/3612013333 learned_addr learned my addr 172.21.15.76:0/3612013333 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55dbfff640 1 --2- 172.21.15.76:0/3612013333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f0513bd0 0x5560f0519570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e097e640 1 --2- 172.21.15.76:0/3612013333 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f05132b0 0x5560f051c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e117f640 1 -- 172.21.15.76:0/3612013333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f0513bd0 msgr2=0x5560f0519570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e117f640 1 --2- 172.21.15.76:0/3612013333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f0513bd0 0x5560f0519570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.235 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e117f640 1 -- 172.21.15.76:0/3612013333 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f05132b0 msgr2=0x5560f051c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e117f640 1 --2- 172.21.15.76:0/3612013333 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f05132b0 0x5560f051c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55e117f640 1 -- 172.21.15.76:0/3612013333 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c400f600 con 0x5560f051c7b0 2024-03-12T06:05:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.217+0000 7f55dbfff640 1 --2- 172.21.15.76:0/3612013333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f0513bd0 0x5560f0519570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55e117f640 1 --2- 172.21.15.76:0/3612013333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f051c7b0 0x5560f0519cb0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f55cc0bcce0 tx=0x7f55cc04ab10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1670ab75a44f0922 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55bbfff640 1 -- 172.21.15.76:0/3612013333 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc0bd450 con 0x5560f051c7b0 2024-03-12T06:05:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55bbfff640 1 -- 172.21.15.76:0/3612013333 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f55cc04a700 con 0x5560f051c7b0 2024-03-12T06:05:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55e57b2740 1 -- 172.21.15.76:0/3612013333 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5560f017a7e0 con 0x5560f051c7b0 2024-03-12T06:05:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55bbfff640 1 -- 172.21.15.76:0/3612013333 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55cc0bcd50 con 0x5560f051c7b0 2024-03-12T06:05:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55e57b2740 10 client.17712 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:54.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55e57b2740 1 -- 172.21.15.76:0/3612013333 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5560f0479610 con 0x5560f051c7b0 2024-03-12T06:05:54.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55bbfff640 1 -- 172.21.15.76:0/3612013333 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5560f017a7e0 con 0x5560f051c7b0 2024-03-12T06:05:54.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55bbfff640 1 -- 172.21.15.76:0/3612013333 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f55cc03c070 con 0x5560f051c7b0 2024-03-12T06:05:54.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55e57b2740 10 client.17712 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:54.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55e57b2740 10 client.17712 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:05:54.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55e57b2740 20 client.17712 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:54.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55e57b2740 1 --2- 172.21.15.76:0/3612013333 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5560f052c6d0 0x5560f052eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55e57b2740 4 client.17712 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:05:54.237 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.218+0000 7f55e57b2740 1 -- 172.21.15.76:0/3612013333 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5560f01b7720 con 0x5560f052c6d0 2024-03-12T06:05:54.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.219+0000 7f55e097e640 1 --2- 172.21.15.76:0/3612013333 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5560f052c6d0 0x5560f052eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.219+0000 7f55bbfff640 10 client.17712 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:54.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.219+0000 7f55e097e640 1 --2- 172.21.15.76:0/3612013333 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5560f052c6d0 0x5560f052eac0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52c68da987e40b00 server_cookie=9d7e061b8390438a in_seq=0 out_seq=0 2024-03-12T06:05:54.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55bbfff640 1 -- 172.21.15.76:0/3612013333 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5560f01b7720 con 0x5560f052c6d0 2024-03-12T06:05:54.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55bbfff640 10 client.17712 handle_command_reply: tid=0 2024-03-12T06:05:54.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55e57b2740 1 client.17712 shutdown 2024-03-12T06:05:54.238 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55e57b2740 1 -- 172.21.15.76:0/3612013333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f051c7b0 msgr2=0x5560f0519cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55e57b2740 1 --2- 172.21.15.76:0/3612013333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f051c7b0 0x5560f0519cb0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f55cc0bcce0 tx=0x7f55cc04ab10 comp rx=0 tx=0).stop 2024-03-12T06:05:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55e57b2740 1 -- 172.21.15.76:0/3612013333 shutdown_connections 2024-03-12T06:05:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55e57b2740 1 --2- 172.21.15.76:0/3612013333 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5560f052c6d0 0x5560f052eac0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55e57b2740 1 --2- 172.21.15.76:0/3612013333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f051c7b0 0x5560f0519cb0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55e57b2740 1 --2- 172.21.15.76:0/3612013333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f0513bd0 0x5560f0519570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55e57b2740 1 --2- 172.21.15.76:0/3612013333 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f05132b0 0x5560f051c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55bbfff640 0 client.17712 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:54.239 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55e57b2740 1 -- 172.21.15.76:0/3612013333 >> 172.21.15.76:0/3612013333 conn(0x5560f050d990 msgr2=0x5560f050f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55e57b2740 1 -- 172.21.15.76:0/3612013333 shutdown_connections 2024-03-12T06:05:54.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55e57b2740 1 -- 172.21.15.76:0/3612013333 wait complete. 2024-03-12T06:05:54.240 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.220+0000 7f55e57b2740 20 client.17712 trim_cache size 0 max 16384 2024-03-12T06:05:54.247 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55e57b2740 1 Processor -- start 2024-03-12T06:05:54.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55e57b2740 1 -- start start 2024-03-12T06:05:54.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55e57b2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 0x5560f051bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55e57b2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f051c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55e57b2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 0x5560f0517e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55e57b2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f0529430 con 0x5560f0513bd0 2024-03-12T06:05:54.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55e57b2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f0528560 con 0x5560f051c7b0 2024-03-12T06:05:54.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55e57b2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f050ec10 con 0x5560f05132b0 2024-03-12T06:05:54.248 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f051c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f051c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52806/0 (socket says 172.21.15.76:52806) 2024-03-12T06:05:54.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55dbfff640 1 -- 172.21.15.76:0/497785763 learned_addr learned my addr 172.21.15.76:0/497785763 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55e097e640 1 --2- 172.21.15.76:0/497785763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 0x5560f051bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55e117f640 1 --2- 172.21.15.76:0/497785763 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 0x5560f0517e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55dbfff640 1 -- 172.21.15.76:0/497785763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 msgr2=0x5560f051bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55dbfff640 1 --2- 172.21.15.76:0/497785763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 0x5560f051bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55dbfff640 1 -- 172.21.15.76:0/497785763 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 msgr2=0x5560f0517e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55dbfff640 1 --2- 172.21.15.76:0/497785763 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 0x5560f0517e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.249 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.237+0000 7f55dbfff640 1 -- 172.21.15.76:0/497785763 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560f0415ac0 con 0x5560f0513bd0 2024-03-12T06:05:54.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.238+0000 7f55dbfff640 1 --2- 172.21.15.76:0/497785763 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f051c1e0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f55c400f840 tx=0x7f55c4005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7978d5edf26865a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.238+0000 7f55d27fc640 1 -- 172.21.15.76:0/497785763 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c402c070 con 0x5560f0513bd0 2024-03-12T06:05:54.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.238+0000 7f55d27fc640 1 -- 172.21.15.76:0/497785763 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55c4005520 con 0x5560f0513bd0 2024-03-12T06:05:54.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.238+0000 7f55d27fc640 1 -- 172.21.15.76:0/497785763 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c4004660 con 0x5560f0513bd0 2024-03-12T06:05:54.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.238+0000 7f55e57b2740 1 -- 172.21.15.76:0/497785763 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 msgr2=0x5560f051c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.238+0000 7f55e57b2740 1 --2- 172.21.15.76:0/497785763 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f051c1e0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f55c400f840 tx=0x7f55c4005f30 comp rx=0 tx=0).stop 2024-03-12T06:05:54.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 -- 172.21.15.76:0/497785763 shutdown_connections 2024-03-12T06:05:54.250 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 --2- 172.21.15.76:0/497785763 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 0x5560f0517e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 --2- 172.21.15.76:0/497785763 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f051c1e0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 --2- 172.21.15.76:0/497785763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 0x5560f051bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 -- 172.21.15.76:0/497785763 >> 172.21.15.76:0/497785763 conn(0x5560f050d990 msgr2=0x5560f0514810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 -- 172.21.15.76:0/497785763 shutdown_connections 2024-03-12T06:05:54.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 -- 172.21.15.76:0/497785763 wait complete. 2024-03-12T06:05:54.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 -- start start 2024-03-12T06:05:54.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 Processor -- start 2024-03-12T06:05:54.251 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 0x5560f0523650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f0523b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 0x5560f0521630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f0521b70 con 0x5560f0513bd0 2024-03-12T06:05:54.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f0521ce0 con 0x5560f051c7b0 2024-03-12T06:05:54.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f0521e50 con 0x5560f05132b0 2024-03-12T06:05:54.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55e57b2740 10 client.0 _ll_register_callbacks cb 0x7f55c400de40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:54.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f0523b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f0523b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52810/0 (socket says 172.21.15.76:52810) 2024-03-12T06:05:54.252 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.239+0000 7f55dbfff640 1 -- 172.21.15.76:0/3272887289 learned_addr learned my addr 172.21.15.76:0/3272887289 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55e117f640 1 --2- 172.21.15.76:0/3272887289 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 0x5560f0521630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55e097e640 1 --2- 172.21.15.76:0/3272887289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 0x5560f0523650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55dbfff640 1 -- 172.21.15.76:0/3272887289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 msgr2=0x5560f0523650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55dbfff640 1 --2- 172.21.15.76:0/3272887289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 0x5560f0523650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55dbfff640 1 -- 172.21.15.76:0/3272887289 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 msgr2=0x5560f0521630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55dbfff640 1 --2- 172.21.15.76:0/3272887289 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 0x5560f0521630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55dbfff640 1 -- 172.21.15.76:0/3272887289 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560f0439570 con 0x5560f0513bd0 2024-03-12T06:05:54.253 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55dbfff640 1 --2- 172.21.15.76:0/3272887289 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f0523b90 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f55c4000c00 tx=0x7f55c4005790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d1e0f4d2cb3097 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:54.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55bbfff640 1 -- 172.21.15.76:0/3272887289 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c402c070 con 0x5560f0513bd0 2024-03-12T06:05:54.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55bbfff640 1 -- 172.21.15.76:0/3272887289 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f55c401cdc0 con 0x5560f0513bd0 2024-03-12T06:05:54.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55e57b2740 1 -- 172.21.15.76:0/3272887289 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f55c4018040 con 0x5560f0513bd0 2024-03-12T06:05:54.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55bbfff640 1 -- 172.21.15.76:0/3272887289 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55c401c830 con 0x5560f0513bd0 2024-03-12T06:05:54.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55e57b2740 10 client.17718 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:54.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55e57b2740 1 -- 172.21.15.76:0/3272887289 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5560f043b6a0 con 0x5560f0513bd0 2024-03-12T06:05:54.254 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.240+0000 7f55bbfff640 1 -- 172.21.15.76:0/3272887289 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55c4018040 con 0x5560f0513bd0 2024-03-12T06:05:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.241+0000 7f55bbfff640 1 -- 172.21.15.76:0/3272887289 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f55c40325e0 con 0x5560f0513bd0 2024-03-12T06:05:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.241+0000 7f55e57b2740 10 client.17718 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.241+0000 7f55e57b2740 10 client.17718 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:05:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.241+0000 7f55e57b2740 20 client.17718 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.241+0000 7f55e57b2740 1 --2- 172.21.15.76:0/3272887289 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5560f05848f0 0x5560f0586d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.241+0000 7f55e57b2740 4 client.17718 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-12T06:05:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.241+0000 7f55e57b2740 1 -- 172.21.15.76:0/3272887289 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5560f0206890 con 0x5560f05848f0 2024-03-12T06:05:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.241+0000 7f55e097e640 1 --2- 172.21.15.76:0/3272887289 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5560f05848f0 0x5560f0586d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.255 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.241+0000 7f55bbfff640 10 client.17718 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:54.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.241+0000 7f55e097e640 1 --2- 172.21.15.76:0/3272887289 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5560f05848f0 0x5560f0586d10 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65e796c07717c5eb server_cookie=8c366402dce8cd3c in_seq=0 out_seq=0 2024-03-12T06:05:54.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55bbfff640 1 -- 172.21.15.76:0/3272887289 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5560f0206890 con 0x5560f05848f0 2024-03-12T06:05:54.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55bbfff640 10 client.17718 handle_command_reply: tid=0 2024-03-12T06:05:54.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e57b2740 1 client.17718 shutdown 2024-03-12T06:05:54.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e57b2740 1 -- 172.21.15.76:0/3272887289 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 msgr2=0x5560f0523b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e57b2740 1 --2- 172.21.15.76:0/3272887289 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f0523b90 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f55c4000c00 tx=0x7f55c4005790 comp rx=0 tx=0).stop 2024-03-12T06:05:54.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e57b2740 1 -- 172.21.15.76:0/3272887289 shutdown_connections 2024-03-12T06:05:54.256 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e57b2740 1 --2- 172.21.15.76:0/3272887289 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x5560f051c7b0 0x5560f0521630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e57b2740 1 --2- 172.21.15.76:0/3272887289 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x5560f05848f0 0x5560f0586d10 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e57b2740 1 --2- 172.21.15.76:0/3272887289 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5560f0513bd0 0x5560f0523b90 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e57b2740 1 --2- 172.21.15.76:0/3272887289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5560f05132b0 0x5560f0523650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55bbfff640 0 client.17718 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:54.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e57b2740 1 -- 172.21.15.76:0/3272887289 >> 172.21.15.76:0/3272887289 conn(0x5560f050d990 msgr2=0x5560f0517d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.257 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e57b2740 1 -- 172.21.15.76:0/3272887289 shutdown_connections 2024-03-12T06:05:54.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e57b2740 1 -- 172.21.15.76:0/3272887289 wait complete. 2024-03-12T06:05:54.258 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e57b2740 20 client.17718 trim_cache size 0 max 16384 2024-03-12T06:05:54.258 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:05:54.258 INFO:teuthology.orchestra.run.smithi076.stdout: "status": "no active scrubs running", 2024-03-12T06:05:54.258 INFO:teuthology.orchestra.run.smithi076.stdout: "scrubs": {} 2024-03-12T06:05:54.259 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:05:54.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e2be2640 1 -- 172.21.15.76:0/3247673582 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f55c80a1c20 msgr2=0x7f55c80a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e2be2640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f55c80a1c20 0x7f55c80a40e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f55c40126a0 tx=0x7f55c400d970 comp rx=0 tx=0).stop 2024-03-12T06:05:54.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e2be2640 1 -- 172.21.15.76:0/3247673582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55dc106130 msgr2=0x7f55dc197670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e2be2640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55dc106130 0x7f55dc197670 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f55cc02c170 tx=0x7f55cc004040 comp rx=0 tx=0).stop 2024-03-12T06:05:54.260 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e2be2640 1 -- 172.21.15.76:0/3247673582 shutdown_connections 2024-03-12T06:05:54.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e2be2640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f55c80a1c20 0x7f55c80a40e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e2be2640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55dc106130 0x7f55dc197670 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e2be2640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55dc0fffb0 0x7f55dc19daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e2be2640 1 --2- 172.21.15.76:0/3247673582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55dc0ff5f0 0x7f55dc19d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.242+0000 7f55e2be2640 1 -- 172.21.15.76:0/3247673582 >> 172.21.15.76:0/3247673582 conn(0x7f55dc076570 msgr2=0x7f55dc1040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.243+0000 7f55e2be2640 1 -- 172.21.15.76:0/3247673582 shutdown_connections 2024-03-12T06:05:54.261 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.243+0000 7f55e2be2640 1 -- 172.21.15.76:0/3247673582 wait complete. 2024-03-12T06:05:54.261 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-12T06:05:54.262 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T06:05:54.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.360+0000 7fd21c496640 1 -- 172.21.15.76:0/117994150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd214100c50 msgr2=0x7fd214104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.360+0000 7fd21c496640 1 --2- 172.21.15.76:0/117994150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd214100c50 0x7fd214104c90 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fd210009930 tx=0x7fd21002bc30 comp rx=0 tx=0).stop 2024-03-12T06:05:54.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.360+0000 7fd21c496640 1 -- 172.21.15.76:0/117994150 shutdown_connections 2024-03-12T06:05:54.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.360+0000 7fd21c496640 1 --2- 172.21.15.76:0/117994150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd214105370 0x7fd21410cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.360+0000 7fd21c496640 1 --2- 172.21.15.76:0/117994150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd214100c50 0x7fd214104c90 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.360+0000 7fd21c496640 1 --2- 172.21.15.76:0/117994150 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2141002a0 0x7fd214100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.360+0000 7fd21c496640 1 -- 172.21.15.76:0/117994150 >> 172.21.15.76:0/117994150 conn(0x7fd214077200 msgr2=0x7fd214077610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.360+0000 7fd21c496640 1 -- 172.21.15.76:0/117994150 shutdown_connections 2024-03-12T06:05:54.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.360+0000 7fd21c496640 1 -- 172.21.15.76:0/117994150 wait complete. 2024-03-12T06:05:54.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.360+0000 7fd21c496640 1 Processor -- start 2024-03-12T06:05:54.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.360+0000 7fd21c496640 1 -- start start 2024-03-12T06:05:54.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21c496640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2141002a0 0x7fd21419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21c496640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd214105370 0x7fd21419d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21c496640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd214197650 0x7fd214197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21c496640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd214198040 con 0x7fd2141002a0 2024-03-12T06:05:54.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21c496640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2141981b0 con 0x7fd214105370 2024-03-12T06:05:54.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21c496640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd214198370 con 0x7fd214197650 2024-03-12T06:05:54.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21a232640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2141002a0 0x7fd21419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21a232640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2141002a0 0x7fd21419d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52832/0 (socket says 172.21.15.76:52832) 2024-03-12T06:05:54.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21a232640 1 -- 172.21.15.76:0/1991643300 learned_addr learned my addr 172.21.15.76:0/1991643300 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd219a31640 1 --2- 172.21.15.76:0/1991643300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd214105370 0x7fd21419d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21aa33640 1 --2- 172.21.15.76:0/1991643300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd214197650 0x7fd214197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21a232640 1 -- 172.21.15.76:0/1991643300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd214197650 msgr2=0x7fd214197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21a232640 1 --2- 172.21.15.76:0/1991643300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd214197650 0x7fd214197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21a232640 1 -- 172.21.15.76:0/1991643300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd214105370 msgr2=0x7fd21419d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21a232640 1 --2- 172.21.15.76:0/1991643300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd214105370 0x7fd21419d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21a232640 1 -- 172.21.15.76:0/1991643300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd20c009840 con 0x7fd2141002a0 2024-03-12T06:05:54.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21a232640 1 --2- 172.21.15.76:0/1991643300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2141002a0 0x7fd21419d450 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fd20800d950 tx=0x7fd20800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd2077fe640 1 -- 172.21.15.76:0/1991643300 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208009a30 con 0x7fd2141002a0 2024-03-12T06:05:54.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21c496640 1 -- 172.21.15.76:0/1991643300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd210009560 con 0x7fd2141002a0 2024-03-12T06:05:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd2077fe640 1 -- 172.21.15.76:0/1991643300 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd208009b90 con 0x7fd2141002a0 2024-03-12T06:05:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd2077fe640 1 -- 172.21.15.76:0/1991643300 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208004a00 con 0x7fd2141002a0 2024-03-12T06:05:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.361+0000 7fd21c496640 1 -- 172.21.15.76:0/1991643300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2141a4660 con 0x7fd2141002a0 2024-03-12T06:05:54.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.362+0000 7fd21c496640 1 -- 172.21.15.76:0/1991643300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd214068000 con 0x7fd2141002a0 2024-03-12T06:05:54.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.363+0000 7fd2077fe640 1 -- 172.21.15.76:0/1991643300 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd208009d00 con 0x7fd2141002a0 2024-03-12T06:05:54.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.364+0000 7fd2077fe640 1 --2- 172.21.15.76:0/1991643300 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd2000a1ba0 0x7fd2000a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.364+0000 7fd219a31640 1 --2- 172.21.15.76:0/1991643300 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd2000a1ba0 0x7fd2000a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.364+0000 7fd2077fe640 1 -- 172.21.15.76:0/1991643300 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fd2080cd240 con 0x7fd2141002a0 2024-03-12T06:05:54.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.365+0000 7fd219a31640 1 --2- 172.21.15.76:0/1991643300 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd2000a1ba0 0x7fd2000a4060 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd210000c00 tx=0x7fd210005fd0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.369+0000 7fd2077fe640 1 -- 172.21.15.76:0/1991643300 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd208096110 con 0x7fd2141002a0 2024-03-12T06:05:54.583 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.578+0000 7fd21c496640 1 -- 172.21.15.76:0/1991643300 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd214108660 con 0x7fd2141002a0 2024-03-12T06:05:54.584 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.579+0000 7fd2077fe640 1 -- 172.21.15.76:0/1991643300 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11328 (secure 0 0 0) 0x7fd2080997d0 con 0x7fd2141002a0 2024-03-12T06:05:54.584 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:05:54.584 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T06:01:33.842001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T06:05:54.584 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 28 2024-03-12T06:05:54.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.581+0000 7fd21c496640 1 -- 172.21.15.76:0/1991643300 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd2000a1ba0 msgr2=0x7fd2000a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.581+0000 7fd21c496640 1 --2- 172.21.15.76:0/1991643300 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd2000a1ba0 0x7fd2000a4060 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd210000c00 tx=0x7fd210005fd0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.581+0000 7fd21c496640 1 -- 172.21.15.76:0/1991643300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2141002a0 msgr2=0x7fd21419d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.581+0000 7fd21c496640 1 --2- 172.21.15.76:0/1991643300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2141002a0 0x7fd21419d450 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fd20800d950 tx=0x7fd20800de10 comp rx=0 tx=0).stop 2024-03-12T06:05:54.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.581+0000 7fd21c496640 1 -- 172.21.15.76:0/1991643300 shutdown_connections 2024-03-12T06:05:54.586 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.581+0000 7fd21c496640 1 --2- 172.21.15.76:0/1991643300 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd2000a1ba0 0x7fd2000a4060 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.581+0000 7fd21c496640 1 --2- 172.21.15.76:0/1991643300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd214197650 0x7fd214197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.581+0000 7fd21c496640 1 --2- 172.21.15.76:0/1991643300 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd214105370 0x7fd21419d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.581+0000 7fd21c496640 1 --2- 172.21.15.76:0/1991643300 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd2141002a0 0x7fd21419d450 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.581+0000 7fd21c496640 1 -- 172.21.15.76:0/1991643300 >> 172.21.15.76:0/1991643300 conn(0x7fd214077200 msgr2=0x7fd2140fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.581+0000 7fd21c496640 1 -- 172.21.15.76:0/1991643300 shutdown_connections 2024-03-12T06:05:54.587 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.581+0000 7fd21c496640 1 -- 172.21.15.76:0/1991643300 wait complete. 2024-03-12T06:05:54.596 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:05:54.597 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-12T06:05:54.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.695+0000 7fb7aa90f640 1 -- 172.21.15.76:0/2168961214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7a410b770 msgr2=0x7fb7a410dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.695+0000 7fb7aa90f640 1 --2- 172.21.15.76:0/2168961214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7a410b770 0x7fb7a410dba0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009ca0 tx=0x7fb79c02c180 comp rx=0 tx=0).stop 2024-03-12T06:05:54.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.696+0000 7fb7aa90f640 1 -- 172.21.15.76:0/2168961214 shutdown_connections 2024-03-12T06:05:54.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.696+0000 7fb7aa90f640 1 --2- 172.21.15.76:0/2168961214 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7a410b770 0x7fb7a410dba0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.696+0000 7fb7aa90f640 1 --2- 172.21.15.76:0/2168961214 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb7a4102cc0 0x7fb7a410b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.701 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.696+0000 7fb7aa90f640 1 --2- 172.21.15.76:0/2168961214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7a41023a0 0x7fb7a4102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.696+0000 7fb7aa90f640 1 -- 172.21.15.76:0/2168961214 >> 172.21.15.76:0/2168961214 conn(0x7fb7a4077230 msgr2=0x7fb7a4077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.696+0000 7fb7aa90f640 1 -- 172.21.15.76:0/2168961214 shutdown_connections 2024-03-12T06:05:54.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.696+0000 7fb7aa90f640 1 -- 172.21.15.76:0/2168961214 wait complete. 2024-03-12T06:05:54.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.696+0000 7fb7aa90f640 1 Processor -- start 2024-03-12T06:05:54.702 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.696+0000 7fb7aa90f640 1 -- start start 2024-03-12T06:05:54.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.696+0000 7fb7aa90f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7a41023a0 0x7fb7a419d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.696+0000 7fb7aa90f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7a4102cc0 0x7fb7a419dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.696+0000 7fb7aa90f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb7a410b770 0x7fb7a4197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7aa90f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4197e10 con 0x7fb7a4102cc0 2024-03-12T06:05:54.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7aa90f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4197f80 con 0x7fb7a410b770 2024-03-12T06:05:54.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7aa90f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4198140 con 0x7fb7a41023a0 2024-03-12T06:05:54.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7a4102cc0 0x7fb7a419dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7a4102cc0 0x7fb7a419dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52846/0 (socket says 172.21.15.76:52846) 2024-03-12T06:05:54.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a37fe640 1 -- 172.21.15.76:0/660467433 learned_addr learned my addr 172.21.15.76:0/660467433 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a3fff640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7a41023a0 0x7fb7a419d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a8eac640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb7a410b770 0x7fb7a4197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a37fe640 1 -- 172.21.15.76:0/660467433 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7a41023a0 msgr2=0x7fb7a419d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.712 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a37fe640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7a41023a0 0x7fb7a419d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a37fe640 1 -- 172.21.15.76:0/660467433 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb7a410b770 msgr2=0x7fb7a4197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a37fe640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb7a410b770 0x7fb7a4197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a37fe640 1 -- 172.21.15.76:0/660467433 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c009900 con 0x7fb7a4102cc0 2024-03-12T06:05:54.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a8eac640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb7a410b770 0x7fb7a4197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:05:54.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a3fff640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7a41023a0 0x7fb7a419d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:54.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a37fe640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7a4102cc0 0x7fb7a419dab0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fb79800d950 tx=0x7fb79800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a17fa640 1 -- 172.21.15.76:0/660467433 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798009c30 con 0x7fb7a4102cc0 2024-03-12T06:05:54.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7aa90f640 1 -- 172.21.15.76:0/660467433 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a4198420 con 0x7fb7a4102cc0 2024-03-12T06:05:54.713 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a17fa640 1 -- 172.21.15.76:0/660467433 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb798009d90 con 0x7fb7a4102cc0 2024-03-12T06:05:54.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7a17fa640 1 -- 172.21.15.76:0/660467433 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79800f7e0 con 0x7fb7a4102cc0 2024-03-12T06:05:54.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.697+0000 7fb7aa90f640 1 -- 172.21.15.76:0/660467433 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a41a44f0 con 0x7fb7a4102cc0 2024-03-12T06:05:54.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.699+0000 7fb7a17fa640 1 -- 172.21.15.76:0/660467433 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb798010ad0 con 0x7fb7a4102cc0 2024-03-12T06:05:54.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7ad5b7740 1 Processor -- start 2024-03-12T06:05:54.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7ad5b7740 1 -- start start 2024-03-12T06:05:54.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7ad5b7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdcf62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7a17fa640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb78c0a1ba0 0x7fb78c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.714 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7ad5b7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdcf6820 0x55ecfdd03620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7ad5b7740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdd03b60 0x55ecfdd05f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7ad5b7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfdcb1960 con 0x55ecfdcf5f00 2024-03-12T06:05:54.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7ad5b7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfdc65310 con 0x55ecfdcf6820 2024-03-12T06:05:54.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7ad5b7740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfdc62840 con 0x55ecfdd03b60 2024-03-12T06:05:54.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7a3fff640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb78c0a1ba0 0x7fb78c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7a8eac640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdcf62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7a8eac640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdcf62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52860/0 (socket says 172.21.15.76:52860) 2024-03-12T06:05:54.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7a8eac640 1 -- 172.21.15.76:0/2363386774 learned_addr learned my addr 172.21.15.76:0/2363386774 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.715 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7a17fa640 1 -- 172.21.15.76:0/660467433 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb7980cd540 con 0x7fb7a4102cc0 2024-03-12T06:05:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7a37fe640 1 --2- 172.21.15.76:0/2363386774 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdcf6820 0x55ecfdd03620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7a8eac640 1 --2- 172.21.15.76:0/2363386774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdd03b60 0x55ecfdd05f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7a8eac640 1 -- 172.21.15.76:0/2363386774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdd03b60 msgr2=0x55ecfdd05f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7a8eac640 1 --2- 172.21.15.76:0/2363386774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdd03b60 0x55ecfdd05f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7a8eac640 1 -- 172.21.15.76:0/2363386774 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdcf6820 msgr2=0x55ecfdd03620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7a8eac640 1 --2- 172.21.15.76:0/2363386774 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdcf6820 0x55ecfdd03620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7a8eac640 1 -- 172.21.15.76:0/2363386774 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecfdc20060 con 0x55ecfdcf5f00 2024-03-12T06:05:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.700+0000 7fb7a3fff640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb78c0a1ba0 0x7fb78c0a4060 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb794009930 tx=0x7fb794008040 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.701+0000 7fb7a8eac640 1 --2- 172.21.15.76:0/2363386774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdcf62e0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fb79c02eed0 tx=0x7fb79c009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705e3c4d341a90c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.716 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.701+0000 7fb78a7fc640 1 -- 172.21.15.76:0/2363386774 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c034440 con 0x55ecfdcf5f00 2024-03-12T06:05:54.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.701+0000 7fb78a7fc640 1 -- 172.21.15.76:0/2363386774 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb79c003970 con 0x55ecfdcf5f00 2024-03-12T06:05:54.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.701+0000 7fb78a7fc640 1 -- 172.21.15.76:0/2363386774 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c032cf0 con 0x55ecfdcf5f00 2024-03-12T06:05:54.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.701+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/2363386774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 msgr2=0x55ecfdcf62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.701+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/2363386774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdcf62e0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fb79c02eed0 tx=0x7fb79c009360 comp rx=0 tx=0).stop 2024-03-12T06:05:54.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.701+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/2363386774 shutdown_connections 2024-03-12T06:05:54.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.701+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/2363386774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdd03b60 0x55ecfdd05f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.701+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/2363386774 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdcf6820 0x55ecfdd03620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.701+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/2363386774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdcf62e0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.701+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/2363386774 >> 172.21.15.76:0/2363386774 conn(0x55ecfdcf0980 msgr2=0x55ecfdcf0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.701+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/2363386774 shutdown_connections 2024-03-12T06:05:54.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.701+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/2363386774 wait complete. 2024-03-12T06:05:54.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.702+0000 7fb7ad5b7740 1 -- start start 2024-03-12T06:05:54.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.702+0000 7fb7ad5b7740 1 Processor -- start 2024-03-12T06:05:54.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.702+0000 7fb7ad5b7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd00b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.718 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.702+0000 7fb7ad5b7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdcf6820 0x55ecfdd01070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.702+0000 7fb7ad5b7740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdd03b60 0x55ecfdd015b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.702+0000 7fb7ad5b7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfdc5e020 con 0x55ecfdcf5f00 2024-03-12T06:05:54.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.702+0000 7fb7ad5b7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd931430 con 0x55ecfdcf6820 2024-03-12T06:05:54.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.702+0000 7fb7ad5b7740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd8ede00 con 0x55ecfdd03b60 2024-03-12T06:05:54.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.702+0000 7fb7ad5b7740 10 client.0 _ll_register_callbacks cb 0x55ecfdc05cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:54.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.702+0000 7fb7a3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd00b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.702+0000 7fb7a3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd00b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52866/0 (socket says 172.21.15.76:52866) 2024-03-12T06:05:54.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.702+0000 7fb7a3fff640 1 -- 172.21.15.76:0/1182217499 learned_addr learned my addr 172.21.15.76:0/1182217499 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.719 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7a8eac640 1 --2- 172.21.15.76:0/1182217499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdd03b60 0x55ecfdd015b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7a37fe640 1 --2- 172.21.15.76:0/1182217499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdcf6820 0x55ecfdd01070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7a3fff640 1 -- 172.21.15.76:0/1182217499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdd03b60 msgr2=0x55ecfdd015b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7a3fff640 1 --2- 172.21.15.76:0/1182217499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdd03b60 0x55ecfdd015b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7a3fff640 1 -- 172.21.15.76:0/1182217499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdcf6820 msgr2=0x55ecfdd01070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7a3fff640 1 --2- 172.21.15.76:0/1182217499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdcf6820 0x55ecfdd01070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7a3fff640 1 -- 172.21.15.76:0/1182217499 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a410e820 con 0x55ecfdcf5f00 2024-03-12T06:05:54.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7a3fff640 1 --2- 172.21.15.76:0/1182217499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd00b30 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fb79400def0 tx=0x7fb7940047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f788abf27d829ba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7837fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:54.720 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7837fe640 1 -- 172.21.15.76:0/1182217499 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79401f400 con 0x55ecfdcf5f00 2024-03-12T06:05:54.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7837fe640 1 -- 172.21.15.76:0/1182217499 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb79401f560 con 0x55ecfdcf5f00 2024-03-12T06:05:54.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/1182217499 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ecfd95d7e0 con 0x55ecfdcf5f00 2024-03-12T06:05:54.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7837fe640 1 -- 172.21.15.76:0/1182217499 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb79401f720 con 0x55ecfdcf5f00 2024-03-12T06:05:54.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7ad5b7740 10 client.17736 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:54.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/1182217499 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ecfdc5c6a0 con 0x55ecfdcf5f00 2024-03-12T06:05:54.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7837fe640 1 -- 172.21.15.76:0/1182217499 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb79402c020 con 0x55ecfdcf5f00 2024-03-12T06:05:54.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7837fe640 1 -- 172.21.15.76:0/1182217499 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb794012ad0 con 0x55ecfdcf5f00 2024-03-12T06:05:54.721 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7ad5b7740 10 client.17736 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:54.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7ad5b7740 10 client.17736 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:05:54.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7ad5b7740 20 client.17736 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:54.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/1182217499 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55ecfdd14090 0x55ecfdd164a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7ad5b7740 4 client.17736 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:05:54.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.703+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/1182217499 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ecfd99a720 con 0x55ecfdd14090 2024-03-12T06:05:54.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.704+0000 7fb7a37fe640 1 --2- 172.21.15.76:0/1182217499 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55ecfdd14090 0x55ecfdd164a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.704+0000 7fb7a37fe640 1 --2- 172.21.15.76:0/1182217499 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55ecfdd14090 0x55ecfdd164a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4fd8337bb6d86cb2 server_cookie=63941dad94020b0a in_seq=0 out_seq=0 2024-03-12T06:05:54.722 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.704+0000 7fb7837fe640 10 client.17736 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:05:54.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7837fe640 1 -- 172.21.15.76:0/1182217499 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ecfd99a720 con 0x55ecfdd14090 2024-03-12T06:05:54.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7837fe640 10 client.17736 handle_command_reply: tid=0 2024-03-12T06:05:54.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7ad5b7740 1 client.17736 shutdown 2024-03-12T06:05:54.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/1182217499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 msgr2=0x55ecfdd00b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/1182217499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd00b30 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fb79400def0 tx=0x7fb7940047d0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/1182217499 shutdown_connections 2024-03-12T06:05:54.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/1182217499 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55ecfdd14090 0x55ecfdd164a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/1182217499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdd03b60 0x55ecfdd015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.723 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7837fe640 0 client.17736 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:05:54.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/1182217499 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdcf6820 0x55ecfdd01070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/1182217499 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd00b30 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/1182217499 >> 172.21.15.76:0/1182217499 conn(0x55ecfdcf0980 msgr2=0x55ecfdcf26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/1182217499 shutdown_connections 2024-03-12T06:05:54.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/1182217499 wait complete. 2024-03-12T06:05:54.724 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.705+0000 7fb7ad5b7740 20 client.17736 trim_cache size 0 max 16384 2024-03-12T06:05:54.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7ad5b7740 1 Processor -- start 2024-03-12T06:05:54.730 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7ad5b7740 1 -- start start 2024-03-12T06:05:54.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7ad5b7740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd03260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7ad5b7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf6820 0x55ecfdd01080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7ad5b7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdd03b60 0x55ecfdd015c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7ad5b7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfdbe3130 con 0x55ecfdcf6820 2024-03-12T06:05:54.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7ad5b7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd99a750 con 0x55ecfdd03b60 2024-03-12T06:05:54.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7ad5b7740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfdcf1ad0 con 0x55ecfdcf5f00 2024-03-12T06:05:54.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7a37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf6820 0x55ecfdd01080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.731 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7a37fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf6820 0x55ecfdd01080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52876/0 (socket says 172.21.15.76:52876) 2024-03-12T06:05:54.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7a37fe640 1 -- 172.21.15.76:0/1032716404 learned_addr learned my addr 172.21.15.76:0/1032716404 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7a3fff640 1 --2- 172.21.15.76:0/1032716404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd03260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7a8eac640 1 --2- 172.21.15.76:0/1032716404 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdd03b60 0x55ecfdd015c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7a37fe640 1 -- 172.21.15.76:0/1032716404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdcf5f00 msgr2=0x55ecfdd03260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7a37fe640 1 --2- 172.21.15.76:0/1032716404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd03260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7a37fe640 1 -- 172.21.15.76:0/1032716404 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdd03b60 msgr2=0x55ecfdd015c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7a37fe640 1 --2- 172.21.15.76:0/1032716404 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdd03b60 0x55ecfdd015c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.732 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7a37fe640 1 -- 172.21.15.76:0/1032716404 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecfdbf8c20 con 0x55ecfdcf6820 2024-03-12T06:05:54.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7a8eac640 1 --2- 172.21.15.76:0/1032716404 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdd03b60 0x55ecfdd015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:05:54.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.717+0000 7fb7a37fe640 1 --2- 172.21.15.76:0/1032716404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf6820 0x55ecfdd01080 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fb7980cacf0 tx=0x7fb79801e7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca21a529a56b28 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.718+0000 7fb789ffb640 1 -- 172.21.15.76:0/1032716404 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79809a170 con 0x55ecfdcf6820 2024-03-12T06:05:54.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.718+0000 7fb789ffb640 1 -- 172.21.15.76:0/1032716404 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb79809e2c0 con 0x55ecfdcf6820 2024-03-12T06:05:54.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.718+0000 7fb789ffb640 1 -- 172.21.15.76:0/1032716404 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79809fa50 con 0x55ecfdcf6820 2024-03-12T06:05:54.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.718+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/1032716404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf6820 msgr2=0x55ecfdd01080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.718+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/1032716404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf6820 0x55ecfdd01080 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fb7980cacf0 tx=0x7fb79801e7b0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.718+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/1032716404 shutdown_connections 2024-03-12T06:05:54.733 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.718+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/1032716404 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdd03b60 0x55ecfdd015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.718+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/1032716404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf6820 0x55ecfdd01080 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.718+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/1032716404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd03260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.718+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/1032716404 >> 172.21.15.76:0/1032716404 conn(0x55ecfdcf0980 msgr2=0x55ecfdcfbe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.718+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/1032716404 shutdown_connections 2024-03-12T06:05:54.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.718+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/1032716404 wait complete. 2024-03-12T06:05:54.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.718+0000 7fb7ad5b7740 1 -- start start 2024-03-12T06:05:54.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7ad5b7740 1 Processor -- start 2024-03-12T06:05:54.734 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7ad5b7740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd09b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7ad5b7740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdcf6820 0x55ecfdd0a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7ad5b7740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdd03b60 0x55ecfdd6a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7ad5b7740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfdd0a750 con 0x55ecfdcf5f00 2024-03-12T06:05:54.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7ad5b7740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfdd6ab10 con 0x55ecfdd03b60 2024-03-12T06:05:54.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7ad5b7740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfdd6ac50 con 0x55ecfdcf6820 2024-03-12T06:05:54.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7ad5b7740 10 client.0 _ll_register_callbacks cb 0x7fb7a406aa10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:54.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7a3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd09b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7a3fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd09b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52884/0 (socket says 172.21.15.76:52884) 2024-03-12T06:05:54.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7a3fff640 1 -- 172.21.15.76:0/3059972611 learned_addr learned my addr 172.21.15.76:0/3059972611 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7a37fe640 1 --2- 172.21.15.76:0/3059972611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdcf6820 0x55ecfdd0a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7a8eac640 1 --2- 172.21.15.76:0/3059972611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdd03b60 0x55ecfdd6a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7a3fff640 1 -- 172.21.15.76:0/3059972611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdcf6820 msgr2=0x55ecfdd0a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7a3fff640 1 --2- 172.21.15.76:0/3059972611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdcf6820 0x55ecfdd0a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7a3fff640 1 -- 172.21.15.76:0/3059972611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdd03b60 msgr2=0x55ecfdd6a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7a3fff640 1 --2- 172.21.15.76:0/3059972611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdd03b60 0x55ecfdd6a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7a3fff640 1 -- 172.21.15.76:0/3059972611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a41a44f0 con 0x55ecfdcf5f00 2024-03-12T06:05:54.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7a3fff640 1 --2- 172.21.15.76:0/3059972611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd09b10 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fb794012c20 tx=0x7fb794012c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b489196cd7a6e6e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.719+0000 7fb7837fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:54.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7837fe640 1 -- 172.21.15.76:0/3059972611 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb794005bd0 con 0x55ecfdcf5f00 2024-03-12T06:05:54.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7837fe640 1 -- 172.21.15.76:0/3059972611 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb79402d080 con 0x55ecfdcf5f00 2024-03-12T06:05:54.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/3059972611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb79c003970 con 0x55ecfdcf5f00 2024-03-12T06:05:54.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7837fe640 1 -- 172.21.15.76:0/3059972611 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb794005d30 con 0x55ecfdcf5f00 2024-03-12T06:05:54.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7ad5b7740 10 client.17742 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:54.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/3059972611 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ecfdc1e8c0 con 0x55ecfdcf5f00 2024-03-12T06:05:54.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7837fe640 1 -- 172.21.15.76:0/3059972611 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb79c003970 con 0x55ecfdcf5f00 2024-03-12T06:05:54.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7837fe640 1 -- 172.21.15.76:0/3059972611 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb794031430 con 0x55ecfdcf5f00 2024-03-12T06:05:54.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7ad5b7740 10 client.17742 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:54.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7ad5b7740 10 client.17742 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-12T06:05:54.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7ad5b7740 20 client.17742 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:54.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/3059972611 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55ecfdd6c360 0x55ecfdd6e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7ad5b7740 4 client.17742 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:05:54.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/3059972611 --> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ecfd9e9890 con 0x55ecfdd6c360 2024-03-12T06:05:54.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.720+0000 7fb7a37fe640 1 --2- 172.21.15.76:0/3059972611 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55ecfdd6c360 0x55ecfdd6e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7837fe640 10 client.17742 ms_handle_connect on v2:172.21.15.88:6834/3940245802 2024-03-12T06:05:54.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7a37fe640 1 --2- 172.21.15.76:0/3059972611 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55ecfdd6c360 0x55ecfdd6e780 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=17d8f291e8d72e51 server_cookie=f08d64a7636497cd in_seq=0 out_seq=0 2024-03-12T06:05:54.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7837fe640 1 -- 172.21.15.76:0/3059972611 <== mds.2 v2:172.21.15.88:6834/3940245802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ecfd9e9890 con 0x55ecfdd6c360 2024-03-12T06:05:54.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7837fe640 10 client.17742 handle_command_reply: tid=0 2024-03-12T06:05:54.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7ad5b7740 1 client.17742 shutdown 2024-03-12T06:05:54.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/3059972611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 msgr2=0x55ecfdd09b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/3059972611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd09b10 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fb794012c20 tx=0x7fb794012c50 comp rx=0 tx=0).stop 2024-03-12T06:05:54.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/3059972611 shutdown_connections 2024-03-12T06:05:54.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/3059972611 >> [v2:172.21.15.88:6834/3940245802,v1:172.21.15.88:6835/3940245802] conn(0x55ecfdd6c360 0x55ecfdd6e780 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/3059972611 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ecfdd03b60 0x55ecfdd6a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7837fe640 0 client.17742 ms_handle_reset on v2:172.21.15.88:6834/3940245802 2024-03-12T06:05:54.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/3059972611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ecfdcf6820 0x55ecfdd0a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7ad5b7740 1 --2- 172.21.15.76:0/3059972611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ecfdcf5f00 0x55ecfdd09b10 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/3059972611 >> 172.21.15.76:0/3059972611 conn(0x55ecfdcf0980 msgr2=0x55ecfdcf1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/3059972611 shutdown_connections 2024-03-12T06:05:54.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7ad5b7740 1 -- 172.21.15.76:0/3059972611 wait complete. 2024-03-12T06:05:54.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.721+0000 7fb7ad5b7740 20 client.17742 trim_cache size 0 max 16384 2024-03-12T06:05:54.741 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:05:54.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.722+0000 7fb7aa90f640 1 -- 172.21.15.76:0/660467433 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb78c0a1ba0 msgr2=0x7fb78c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.722+0000 7fb7aa90f640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb78c0a1ba0 0x7fb78c0a4060 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb794009930 tx=0x7fb794008040 comp rx=0 tx=0).stop 2024-03-12T06:05:54.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.722+0000 7fb7aa90f640 1 -- 172.21.15.76:0/660467433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7a4102cc0 msgr2=0x7fb7a419dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.722+0000 7fb7aa90f640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7a4102cc0 0x7fb7a419dab0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fb79800d950 tx=0x7fb79800de10 comp rx=0 tx=0).stop 2024-03-12T06:05:54.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.722+0000 7fb7aa90f640 1 -- 172.21.15.76:0/660467433 shutdown_connections 2024-03-12T06:05:54.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.722+0000 7fb7aa90f640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb78c0a1ba0 0x7fb78c0a4060 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.722+0000 7fb7aa90f640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb7a410b770 0x7fb7a4197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.722+0000 7fb7aa90f640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7a4102cc0 0x7fb7a419dab0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.722+0000 7fb7aa90f640 1 --2- 172.21.15.76:0/660467433 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7a41023a0 0x7fb7a419d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.722+0000 7fb7aa90f640 1 -- 172.21.15.76:0/660467433 >> 172.21.15.76:0/660467433 conn(0x7fb7a4077230 msgr2=0x7fb7a40fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.722+0000 7fb7aa90f640 1 -- 172.21.15.76:0/660467433 shutdown_connections 2024-03-12T06:05:54.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.722+0000 7fb7aa90f640 1 -- 172.21.15.76:0/660467433 wait complete. 2024-03-12T06:05:54.744 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:05:54.744 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-12T06:05:54.851 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.844+0000 7f01f2716640 1 -- 172.21.15.76:0/3139099017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01ec0fffb0 msgr2=0x7f01ec10e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.844+0000 7f01f2716640 1 --2- 172.21.15.76:0/3139099017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01ec0fffb0 0x7f01ec10e0e0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009ca0 tx=0x7f01e402c140 comp rx=0 tx=0).stop 2024-03-12T06:05:54.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.844+0000 7f01f2716640 1 -- 172.21.15.76:0/3139099017 shutdown_connections 2024-03-12T06:05:54.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.844+0000 7f01f2716640 1 --2- 172.21.15.76:0/3139099017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01ec0fffb0 0x7f01ec10e0e0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.844+0000 7f01f2716640 1 --2- 172.21.15.76:0/3139099017 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f01ec0ff5f0 0x7f01ec0ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.844+0000 7f01f2716640 1 --2- 172.21.15.76:0/3139099017 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01ec103100 0x7f01ec1034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.852 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.844+0000 7f01f2716640 1 -- 172.21.15.76:0/3139099017 >> 172.21.15.76:0/3139099017 conn(0x7f01ec076570 msgr2=0x7f01ec0fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.844+0000 7f01f2716640 1 -- 172.21.15.76:0/3139099017 shutdown_connections 2024-03-12T06:05:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.844+0000 7f01f2716640 1 -- 172.21.15.76:0/3139099017 wait complete. 2024-03-12T06:05:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.844+0000 7f01f2716640 1 Processor -- start 2024-03-12T06:05:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.844+0000 7f01f2716640 1 -- start start 2024-03-12T06:05:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f2716640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f01ec0ff5f0 0x7f01ec19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f2716640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01ec0fffb0 0x7f01ec19db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.853 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f2716640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01ec103100 0x7f01ec197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f2716640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec197ea0 con 0x7f01ec103100 2024-03-12T06:05:54.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f2716640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec198010 con 0x7f01ec0ff5f0 2024-03-12T06:05:54.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f2716640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec1981d0 con 0x7f01ec0fffb0 2024-03-12T06:05:54.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f0cb3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01ec103100 0x7f01ec197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f0cb3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01ec103100 0x7f01ec197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52892/0 (socket says 172.21.15.76:52892) 2024-03-12T06:05:54.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f0cb3640 1 -- 172.21.15.76:0/257646075 learned_addr learned my addr 172.21.15.76:0/257646075 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01ec0fffb0 0x7f01ec19db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01ebfff640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f01ec0ff5f0 0x7f01ec19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f0cb3640 1 -- 172.21.15.76:0/257646075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01ec0fffb0 msgr2=0x7f01ec19db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f0cb3640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01ec0fffb0 0x7f01ec19db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f0cb3640 1 -- 172.21.15.76:0/257646075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f01ec0ff5f0 msgr2=0x7f01ec19d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f0cb3640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f01ec0ff5f0 0x7f01ec19d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f0cb3640 1 -- 172.21.15.76:0/257646075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e4009900 con 0x7f01ec103100 2024-03-12T06:05:54.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01ebfff640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f01ec0ff5f0 0x7f01ec19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:54.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.845+0000 7f01f0cb3640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01ec103100 0x7f01ec197710 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f01e402c750 tx=0x7f01e4034d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.846+0000 7f01e97fa640 1 -- 172.21.15.76:0/257646075 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e4032ac0 con 0x7f01ec103100 2024-03-12T06:05:54.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.846+0000 7f01e97fa640 1 -- 172.21.15.76:0/257646075 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01e4032eb0 con 0x7f01ec103100 2024-03-12T06:05:54.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.846+0000 7f01f2716640 1 -- 172.21.15.76:0/257646075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ec198450 con 0x7f01ec103100 2024-03-12T06:05:54.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.846+0000 7f01e97fa640 1 -- 172.21.15.76:0/257646075 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e4040450 con 0x7f01ec103100 2024-03-12T06:05:54.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.846+0000 7f01f2716640 1 -- 172.21.15.76:0/257646075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ec1a45d0 con 0x7f01ec103100 2024-03-12T06:05:54.864 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01e97fa640 1 -- 172.21.15.76:0/257646075 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f01e400a040 con 0x7f01ec103100 2024-03-12T06:05:54.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01f53bd740 1 Processor -- start 2024-03-12T06:05:54.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01f53bd740 1 -- start start 2024-03-12T06:05:54.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01f53bd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 0x560818cfe630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01f53bd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 0x560818d07210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01f53bd740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818d07750 0x560818d09b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01f53bd740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560818cb9960 con 0x560818d07750 2024-03-12T06:05:54.865 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01f53bd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560818c6d310 con 0x560818cfe250 2024-03-12T06:05:54.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01f53bd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560818c6a840 con 0x560818cfeb70 2024-03-12T06:05:54.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01e97fa640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f01d40a1ba0 0x7f01d40a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01eb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818d07750 0x560818d09b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01eb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818d07750 0x560818d09b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52898/0 (socket says 172.21.15.76:52898) 2024-03-12T06:05:54.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01eb7fe640 1 -- 172.21.15.76:0/279675701 learned_addr learned my addr 172.21.15.76:0/279675701 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01ebfff640 1 --2- 172.21.15.76:0/279675701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 0x560818d07210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.866 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01f0cb3640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f01d40a1ba0 0x7f01d40a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.848+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/279675701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 0x560818cfe630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.849+0000 7f01e97fa640 1 -- 172.21.15.76:0/257646075 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f01e40f02e0 con 0x7f01ec103100 2024-03-12T06:05:54.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.849+0000 7f01eb7fe640 1 -- 172.21.15.76:0/279675701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 msgr2=0x560818d07210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.849+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/279675701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 0x560818d07210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.849+0000 7f01eb7fe640 1 -- 172.21.15.76:0/279675701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 msgr2=0x560818cfe630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.849+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/279675701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 0x560818cfe630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.849+0000 7f01eb7fe640 1 -- 172.21.15.76:0/279675701 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560818c28060 con 0x560818d07750 2024-03-12T06:05:54.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.849+0000 7f01ebfff640 1 --2- 172.21.15.76:0/279675701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 0x560818d07210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:05:54.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.849+0000 7f01f0cb3640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f01d40a1ba0 0x7f01d40a4060 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f01e40bcfb0 tx=0x7f01e40b8c80 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.849+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/279675701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818d07750 0x560818d09b40 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f01dc0149b0 tx=0x7f01dc00c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cdaa84abb546c86 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.849+0000 7f01d27fc640 1 -- 172.21.15.76:0/279675701 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc0142d0 con 0x560818d07750 2024-03-12T06:05:54.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.849+0000 7f01d27fc640 1 -- 172.21.15.76:0/279675701 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01dc01a030 con 0x560818d07750 2024-03-12T06:05:54.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.849+0000 7f01d27fc640 1 -- 172.21.15.76:0/279675701 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc00f760 con 0x560818d07750 2024-03-12T06:05:54.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.850+0000 7f01f53bd740 1 -- 172.21.15.76:0/279675701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818d07750 msgr2=0x560818d09b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.850+0000 7f01f53bd740 1 --2- 172.21.15.76:0/279675701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818d07750 0x560818d09b40 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f01dc0149b0 tx=0x7f01dc00c4b0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.850+0000 7f01f53bd740 1 -- 172.21.15.76:0/279675701 shutdown_connections 2024-03-12T06:05:54.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.850+0000 7f01f53bd740 1 --2- 172.21.15.76:0/279675701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818d07750 0x560818d09b40 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.850+0000 7f01f53bd740 1 --2- 172.21.15.76:0/279675701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 0x560818d07210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.850+0000 7f01f53bd740 1 --2- 172.21.15.76:0/279675701 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 0x560818cfe630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.850+0000 7f01f53bd740 1 -- 172.21.15.76:0/279675701 >> 172.21.15.76:0/279675701 conn(0x560818cf8980 msgr2=0x560818cf8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.850+0000 7f01f53bd740 1 -- 172.21.15.76:0/279675701 shutdown_connections 2024-03-12T06:05:54.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.850+0000 7f01f53bd740 1 -- 172.21.15.76:0/279675701 wait complete. 2024-03-12T06:05:54.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.850+0000 7f01f53bd740 1 -- start start 2024-03-12T06:05:54.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.850+0000 7f01f53bd740 1 Processor -- start 2024-03-12T06:05:54.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01f53bd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 0x560818d047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01f53bd740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818cfeb70 0x560818d04d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01f53bd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818d07750 0x560818d05250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01f53bd740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560818c66020 con 0x560818cfeb70 2024-03-12T06:05:54.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01f53bd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560818939430 con 0x560818cfe250 2024-03-12T06:05:54.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01f53bd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608188f5e00 con 0x560818d07750 2024-03-12T06:05:54.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01f53bd740 10 client.0 _ll_register_callbacks cb 0x560818c0dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:54.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01ebfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818cfeb70 0x560818d04d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01ebfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818cfeb70 0x560818d04d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52914/0 (socket says 172.21.15.76:52914) 2024-03-12T06:05:54.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01ebfff640 1 -- 172.21.15.76:0/3024991545 learned_addr learned my addr 172.21.15.76:0/3024991545 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/3024991545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818d07750 0x560818d05250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.871 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/3024991545 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 0x560818d047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01ebfff640 1 -- 172.21.15.76:0/3024991545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818d07750 msgr2=0x560818d05250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01ebfff640 1 --2- 172.21.15.76:0/3024991545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818d07750 0x560818d05250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01ebfff640 1 -- 172.21.15.76:0/3024991545 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 msgr2=0x560818d047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01ebfff640 1 --2- 172.21.15.76:0/3024991545 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 0x560818d047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01ebfff640 1 -- 172.21.15.76:0/3024991545 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01dc014660 con 0x560818cfeb70 2024-03-12T06:05:54.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/3024991545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818d07750 0x560818d05250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:54.872 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01ebfff640 1 --2- 172.21.15.76:0/3024991545 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818cfeb70 0x560818d04d10 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f01e000bfc0 tx=0x7f01e00128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8570209f731d6106 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:54.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01cb7fe640 1 -- 172.21.15.76:0/3024991545 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e0005220 con 0x560818cfeb70 2024-03-12T06:05:54.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01cb7fe640 1 -- 172.21.15.76:0/3024991545 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f01e0005380 con 0x560818cfeb70 2024-03-12T06:05:54.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01f53bd740 1 -- 172.21.15.76:0/3024991545 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5608189657e0 con 0x560818cfeb70 2024-03-12T06:05:54.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01cb7fe640 1 -- 172.21.15.76:0/3024991545 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01e0005540 con 0x560818cfeb70 2024-03-12T06:05:54.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01f53bd740 10 client.17754 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:54.873 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.851+0000 7f01f53bd740 1 -- 172.21.15.76:0/3024991545 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560818c646a0 con 0x560818cfeb70 2024-03-12T06:05:54.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.852+0000 7f01cb7fe640 1 -- 172.21.15.76:0/3024991545 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5608189657e0 con 0x560818cfeb70 2024-03-12T06:05:54.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.852+0000 7f01cb7fe640 1 -- 172.21.15.76:0/3024991545 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f01e0010bf0 con 0x560818cfeb70 2024-03-12T06:05:54.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.852+0000 7f01f53bd740 10 client.17754 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:54.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.852+0000 7f01f53bd740 10 client.17754 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:05:54.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.852+0000 7f01f53bd740 20 client.17754 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:54.874 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.852+0000 7f01f53bd740 1 --2- 172.21.15.76:0/3024991545 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560818d17c50 0x560818d1a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.852+0000 7f01f53bd740 4 client.17754 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:05:54.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.852+0000 7f01f53bd740 1 -- 172.21.15.76:0/3024991545 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5608189a2720 con 0x560818d17c50 2024-03-12T06:05:54.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.852+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/3024991545 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560818d17c50 0x560818d1a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.852+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/3024991545 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560818d17c50 0x560818d1a070 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=13a0a17e5f12bfda server_cookie=cd5915a5f1d7971c in_seq=0 out_seq=0 2024-03-12T06:05:54.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.852+0000 7f01cb7fe640 10 client.17754 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:05:54.875 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01cb7fe640 1 -- 172.21.15.76:0/3024991545 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5608189a2720 con 0x560818d17c50 2024-03-12T06:05:54.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01cb7fe640 10 client.17754 handle_command_reply: tid=0 2024-03-12T06:05:54.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01f53bd740 1 client.17754 shutdown 2024-03-12T06:05:54.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01f53bd740 1 -- 172.21.15.76:0/3024991545 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818cfeb70 msgr2=0x560818d04d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01f53bd740 1 --2- 172.21.15.76:0/3024991545 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818cfeb70 0x560818d04d10 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f01e000bfc0 tx=0x7f01e00128e0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01f53bd740 1 -- 172.21.15.76:0/3024991545 shutdown_connections 2024-03-12T06:05:54.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01f53bd740 1 --2- 172.21.15.76:0/3024991545 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560818d17c50 0x560818d1a070 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01f53bd740 1 --2- 172.21.15.76:0/3024991545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818d07750 0x560818d05250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01f53bd740 1 --2- 172.21.15.76:0/3024991545 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818cfeb70 0x560818d04d10 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01cb7fe640 0 client.17754 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:05:54.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01f53bd740 1 --2- 172.21.15.76:0/3024991545 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 0x560818d047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01f53bd740 1 -- 172.21.15.76:0/3024991545 >> 172.21.15.76:0/3024991545 conn(0x560818cf8980 msgr2=0x560818cfa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01f53bd740 1 -- 172.21.15.76:0/3024991545 shutdown_connections 2024-03-12T06:05:54.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01f53bd740 1 -- 172.21.15.76:0/3024991545 wait complete. 2024-03-12T06:05:54.877 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.853+0000 7f01f53bd740 20 client.17754 trim_cache size 0 max 16384 2024-03-12T06:05:54.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01f53bd740 1 Processor -- start 2024-03-12T06:05:54.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01f53bd740 1 -- start start 2024-03-12T06:05:54.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01f53bd740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818cfe250 0x560818d06f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01f53bd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 0x560818d05070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01f53bd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818d07750 0x560818d01a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01f53bd740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560818beb130 con 0x560818cfe250 2024-03-12T06:05:54.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01f53bd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560818d1f2c0 con 0x560818d07750 2024-03-12T06:05:54.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01f53bd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560818cf4080 con 0x560818cfeb70 2024-03-12T06:05:54.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01eb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818cfe250 0x560818d06f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01eb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818cfe250 0x560818d06f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52926/0 (socket says 172.21.15.76:52926) 2024-03-12T06:05:54.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01eb7fe640 1 -- 172.21.15.76:0/3667341801 learned_addr learned my addr 172.21.15.76:0/3667341801 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01ebfff640 1 --2- 172.21.15.76:0/3667341801 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 0x560818d05070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/3667341801 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818d07750 0x560818d01a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01eb7fe640 1 -- 172.21.15.76:0/3667341801 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 msgr2=0x560818d05070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/3667341801 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 0x560818d05070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01eb7fe640 1 -- 172.21.15.76:0/3667341801 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818d07750 msgr2=0x560818d01a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/3667341801 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818d07750 0x560818d01a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.865+0000 7f01eb7fe640 1 -- 172.21.15.76:0/3667341801 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560818c00c20 con 0x560818cfe250 2024-03-12T06:05:54.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.866+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/3667341801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818cfe250 0x560818d06f00 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f01dc0145f0 tx=0x7f01dc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfbbe6c13369e30 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.866+0000 7f01d1ffb640 1 -- 172.21.15.76:0/3667341801 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc02ad70 con 0x560818cfe250 2024-03-12T06:05:54.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.866+0000 7f01d1ffb640 1 -- 172.21.15.76:0/3667341801 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01dc018070 con 0x560818cfe250 2024-03-12T06:05:54.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.866+0000 7f01d1ffb640 1 -- 172.21.15.76:0/3667341801 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc02a7d0 con 0x560818cfe250 2024-03-12T06:05:54.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.866+0000 7f01f53bd740 1 -- 172.21.15.76:0/3667341801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818cfe250 msgr2=0x560818d06f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.866+0000 7f01f53bd740 1 --2- 172.21.15.76:0/3667341801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818cfe250 0x560818d06f00 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f01dc0145f0 tx=0x7f01dc005850 comp rx=0 tx=0).stop 2024-03-12T06:05:54.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.866+0000 7f01f53bd740 1 -- 172.21.15.76:0/3667341801 shutdown_connections 2024-03-12T06:05:54.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.866+0000 7f01f53bd740 1 --2- 172.21.15.76:0/3667341801 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818d07750 0x560818d01a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.866+0000 7f01f53bd740 1 --2- 172.21.15.76:0/3667341801 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 0x560818d05070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.866+0000 7f01f53bd740 1 --2- 172.21.15.76:0/3667341801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818cfe250 0x560818d06f00 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.866+0000 7f01f53bd740 1 -- 172.21.15.76:0/3667341801 >> 172.21.15.76:0/3667341801 conn(0x560818cf8980 msgr2=0x560818cffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.866+0000 7f01f53bd740 1 -- 172.21.15.76:0/3667341801 shutdown_connections 2024-03-12T06:05:54.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.866+0000 7f01f53bd740 1 -- 172.21.15.76:0/3667341801 wait complete. 2024-03-12T06:05:54.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01f53bd740 1 -- start start 2024-03-12T06:05:54.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01f53bd740 1 Processor -- start 2024-03-12T06:05:54.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01f53bd740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 0x560818d0d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01f53bd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 0x560818d0d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01f53bd740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818d07750 0x560818d6e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01f53bd740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560818d0e080 con 0x560818d07750 2024-03-12T06:05:54.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01f53bd740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560818d0e1f0 con 0x560818cfe250 2024-03-12T06:05:54.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01f53bd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560818d6e730 con 0x560818cfeb70 2024-03-12T06:05:54.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01f53bd740 10 client.0 _ll_register_callbacks cb 0x7f01dc00ca70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:54.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01eb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818d07750 0x560818d6e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01eb7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818d07750 0x560818d6e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52942/0 (socket says 172.21.15.76:52942) 2024-03-12T06:05:54.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01eb7fe640 1 -- 172.21.15.76:0/1739115152 learned_addr learned my addr 172.21.15.76:0/1739115152 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:54.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01ebfff640 1 --2- 172.21.15.76:0/1739115152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 0x560818d0d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/1739115152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 0x560818d0d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01eb7fe640 1 -- 172.21.15.76:0/1739115152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 msgr2=0x560818d0d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/1739115152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 0x560818d0d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01eb7fe640 1 -- 172.21.15.76:0/1739115152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 msgr2=0x560818d0d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:54.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/1739115152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 0x560818d0d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01eb7fe640 1 -- 172.21.15.76:0/1739115152 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560818c28060 con 0x560818d07750 2024-03-12T06:05:54.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.867+0000 7f01ebfff640 1 --2- 172.21.15.76:0/1739115152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 0x560818d0d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:54.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01eb7fe640 1 --2- 172.21.15.76:0/1739115152 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818d07750 0x560818d6e1a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f01dc005fd0 tx=0x7f01dc02a3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e579f797bf1a32c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:54.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:54.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01cb7fe640 1 -- 172.21.15.76:0/1739115152 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc004230 con 0x560818d07750 2024-03-12T06:05:54.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01cb7fe640 1 -- 172.21.15.76:0/1739115152 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f01dc000e40 con 0x560818d07750 2024-03-12T06:05:54.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01f53bd740 1 -- 172.21.15.76:0/1739115152 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f01dc01a030 con 0x560818d07750 2024-03-12T06:05:54.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01cb7fe640 1 -- 172.21.15.76:0/1739115152 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01dc004390 con 0x560818d07750 2024-03-12T06:05:54.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01f53bd740 10 client.17760 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:54.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01f53bd740 1 -- 172.21.15.76:0/1739115152 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560818c268c0 con 0x560818d07750 2024-03-12T06:05:54.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01cb7fe640 1 -- 172.21.15.76:0/1739115152 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01dc01a030 con 0x560818d07750 2024-03-12T06:05:54.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01cb7fe640 1 -- 172.21.15.76:0/1739115152 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f01dc00fa90 con 0x560818d07750 2024-03-12T06:05:54.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01f53bd740 10 client.17760 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:54.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01f53bd740 10 client.17760 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-12T06:05:54.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01f53bd740 20 client.17760 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:54.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01f53bd740 1 --2- 172.21.15.76:0/1739115152 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560818d6fd10 0x560818d72100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:54.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01f53bd740 4 client.17760 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:05:54.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01f53bd740 1 -- 172.21.15.76:0/1739115152 --> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5608189f1890 con 0x560818d6fd10 2024-03-12T06:05:54.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.868+0000 7f01ebfff640 1 --2- 172.21.15.76:0/1739115152 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560818d6fd10 0x560818d72100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:54.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01cb7fe640 10 client.17760 ms_handle_connect on v2:172.21.15.88:6838/1164228359 2024-03-12T06:05:54.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01ebfff640 1 --2- 172.21.15.76:0/1739115152 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560818d6fd10 0x560818d72100 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4b51c1231b7810f9 server_cookie=e378bcd6098bd353 in_seq=0 out_seq=0 2024-03-12T06:05:54.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01cb7fe640 1 -- 172.21.15.76:0/1739115152 <== mds.1 v2:172.21.15.88:6838/1164228359 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5608189f1890 con 0x560818d6fd10 2024-03-12T06:05:54.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01cb7fe640 10 client.17760 handle_command_reply: tid=0 2024-03-12T06:05:54.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01f53bd740 1 client.17760 shutdown 2024-03-12T06:05:54.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01f53bd740 1 -- 172.21.15.76:0/1739115152 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818d07750 msgr2=0x560818d6e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01f53bd740 1 --2- 172.21.15.76:0/1739115152 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818d07750 0x560818d6e1a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f01dc005fd0 tx=0x7f01dc02a3b0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01f53bd740 1 -- 172.21.15.76:0/1739115152 shutdown_connections 2024-03-12T06:05:54.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01f53bd740 1 --2- 172.21.15.76:0/1739115152 >> [v2:172.21.15.88:6838/1164228359,v1:172.21.15.88:6839/1164228359] conn(0x560818d6fd10 0x560818d72100 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01f53bd740 1 --2- 172.21.15.76:0/1739115152 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560818d07750 0x560818d6e1a0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01cb7fe640 0 client.17760 ms_handle_reset on v2:172.21.15.88:6838/1164228359 2024-03-12T06:05:54.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01f53bd740 1 --2- 172.21.15.76:0/1739115152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560818cfeb70 0x560818d0d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01f53bd740 1 --2- 172.21.15.76:0/1739115152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560818cfe250 0x560818d0d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01f53bd740 1 -- 172.21.15.76:0/1739115152 >> 172.21.15.76:0/1739115152 conn(0x560818cf8980 msgr2=0x560818cf72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01f53bd740 1 -- 172.21.15.76:0/1739115152 shutdown_connections 2024-03-12T06:05:54.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01f53bd740 1 -- 172.21.15.76:0/1739115152 wait complete. 2024-03-12T06:05:54.898 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.869+0000 7f01f53bd740 20 client.17760 trim_cache size 0 max 16384 2024-03-12T06:05:54.898 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:05:54.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.870+0000 7f01f2716640 1 -- 172.21.15.76:0/257646075 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f01d40a1ba0 msgr2=0x7f01d40a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.870+0000 7f01f2716640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f01d40a1ba0 0x7f01d40a4060 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f01e40bcfb0 tx=0x7f01e40b8c80 comp rx=0 tx=0).stop 2024-03-12T06:05:54.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.870+0000 7f01f2716640 1 -- 172.21.15.76:0/257646075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01ec103100 msgr2=0x7f01ec197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:54.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.870+0000 7f01f2716640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01ec103100 0x7f01ec197710 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f01e402c750 tx=0x7f01e4034d40 comp rx=0 tx=0).stop 2024-03-12T06:05:54.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.870+0000 7f01f2716640 1 -- 172.21.15.76:0/257646075 shutdown_connections 2024-03-12T06:05:54.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.870+0000 7f01f2716640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f01d40a1ba0 0x7f01d40a4060 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.870+0000 7f01f2716640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f01ec103100 0x7f01ec197710 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.870+0000 7f01f2716640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01ec0fffb0 0x7f01ec19db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.870+0000 7f01f2716640 1 --2- 172.21.15.76:0/257646075 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f01ec0ff5f0 0x7f01ec19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:54.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.870+0000 7f01f2716640 1 -- 172.21.15.76:0/257646075 >> 172.21.15.76:0/257646075 conn(0x7f01ec076570 msgr2=0x7f01ec105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:54.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.870+0000 7f01f2716640 1 -- 172.21.15.76:0/257646075 shutdown_connections 2024-03-12T06:05:54.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:54.870+0000 7f01f2716640 1 -- 172.21.15.76:0/257646075 wait complete. 2024-03-12T06:05:54.901 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:54 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1991643300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:05:54.902 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-12T06:05:54.902 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-12T06:05:54.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:54 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1991643300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:05:54.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:54 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1991643300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:05:55.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.001+0000 7f27a3c17640 1 -- 172.21.15.76:0/348711908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f279c106440 msgr2=0x7f279c10e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:55.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.001+0000 7f27a3c17640 1 --2- 172.21.15.76:0/348711908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f279c106440 0x7f279c10e010 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f2794009cd0 tx=0x7f279402c0f0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.001+0000 7f27a3c17640 1 -- 172.21.15.76:0/348711908 shutdown_connections 2024-03-12T06:05:55.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.001+0000 7f27a3c17640 1 --2- 172.21.15.76:0/348711908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f279c106440 0x7f279c10e010 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.008 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.001+0000 7f27a3c17640 1 --2- 172.21.15.76:0/348711908 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f279c101d20 0x7f279c105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.001+0000 7f27a3c17640 1 --2- 172.21.15.76:0/348711908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f279c101370 0x7f279c101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.001+0000 7f27a3c17640 1 -- 172.21.15.76:0/348711908 >> 172.21.15.76:0/348711908 conn(0x7f279c0fcbf0 msgr2=0x7f279c0ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:55.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.001+0000 7f27a3c17640 1 -- 172.21.15.76:0/348711908 shutdown_connections 2024-03-12T06:05:55.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.001+0000 7f27a3c17640 1 -- 172.21.15.76:0/348711908 wait complete. 2024-03-12T06:05:55.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.001+0000 7f27a3c17640 1 Processor -- start 2024-03-12T06:05:55.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a3c17640 1 -- start start 2024-03-12T06:05:55.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a3c17640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f279c101370 0x7f279c19d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.009 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a3c17640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f279c106440 0x7f279c19d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a3c17640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f279c197670 0x7f279c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a3c17640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279c198060 con 0x7f279c101370 2024-03-12T06:05:55.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a3c17640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279c1981d0 con 0x7f279c106440 2024-03-12T06:05:55.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a3c17640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279c198390 con 0x7f279c197670 2024-03-12T06:05:55.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a19b3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f279c101370 0x7f279c19d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a19b3640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f279c101370 0x7f279c19d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52952/0 (socket says 172.21.15.76:52952) 2024-03-12T06:05:55.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a19b3640 1 -- 172.21.15.76:0/2188522658 learned_addr learned my addr 172.21.15.76:0/2188522658 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:55.010 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a11b2640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f279c106440 0x7f279c19d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a21b4640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f279c197670 0x7f279c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a19b3640 1 -- 172.21.15.76:0/2188522658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f279c197670 msgr2=0x7f279c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:55.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a19b3640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f279c197670 0x7f279c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a19b3640 1 -- 172.21.15.76:0/2188522658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f279c106440 msgr2=0x7f279c19d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:55.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a19b3640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f279c106440 0x7f279c19d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a19b3640 1 -- 172.21.15.76:0/2188522658 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2794009900 con 0x7f279c101370 2024-03-12T06:05:55.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a11b2640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f279c106440 0x7f279c19d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:55.011 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a19b3640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f279c101370 0x7f279c19d470 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f278c00d770 tx=0x7f278c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:55.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f2792ffd640 1 -- 172.21.15.76:0/2188522658 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278c009c30 con 0x7f279c101370 2024-03-12T06:05:55.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f2792ffd640 1 -- 172.21.15.76:0/2188522658 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f278c00de80 con 0x7f279c101370 2024-03-12T06:05:55.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f27a3c17640 1 -- 172.21.15.76:0/2188522658 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f279c1a4330 con 0x7f279c101370 2024-03-12T06:05:55.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.002+0000 7f2792ffd640 1 -- 172.21.15.76:0/2188522658 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278c004e80 con 0x7f279c101370 2024-03-12T06:05:55.012 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.003+0000 7f27a3c17640 1 -- 172.21.15.76:0/2188522658 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279c1a47f0 con 0x7f279c101370 2024-03-12T06:05:55.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.004+0000 7f2792ffd640 1 -- 172.21.15.76:0/2188522658 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f278c005020 con 0x7f279c101370 2024-03-12T06:05:55.017 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.005+0000 7f27a67e0740 1 Processor -- start 2024-03-12T06:05:55.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.005+0000 7f2792ffd640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f27880a1ba0 0x7f27880a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.005+0000 7f27a67e0740 1 -- start start 2024-03-12T06:05:55.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.005+0000 7f27a67e0740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82250 0x560940c82630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a67e0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82b70 0x560940c8b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a67e0740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940c8db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a21b4640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f27880a1ba0 0x7f27880a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.018 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a67e0740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560940c3d960 con 0x560940c8b750 2024-03-12T06:05:55.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a67e0740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560940bf1310 con 0x560940c82250 2024-03-12T06:05:55.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a67e0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560940bee840 con 0x560940c82b70 2024-03-12T06:05:55.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f2792ffd640 1 -- 172.21.15.76:0/2188522658 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f278c0cd300 con 0x7f279c101370 2024-03-12T06:05:55.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a21b4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940c8db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a21b4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940c8db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52958/0 (socket says 172.21.15.76:52958) 2024-03-12T06:05:55.019 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a21b4640 1 -- 172.21.15.76:0/1287689506 learned_addr learned my addr 172.21.15.76:0/1287689506 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:55.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a11b2640 1 --2- 172.21.15.76:0/1287689506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82b70 0x560940c8b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a19b3640 1 --2- 172.21.15.76:0/1287689506 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82250 0x560940c82630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a21b4640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f27880a1ba0 0x7f27880a4060 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2794005cc0 tx=0x7f2794032550 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:55.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a21b4640 1 -- 172.21.15.76:0/1287689506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82b70 msgr2=0x560940c8b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:55.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a21b4640 1 --2- 172.21.15.76:0/1287689506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82b70 0x560940c8b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a21b4640 1 -- 172.21.15.76:0/1287689506 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82250 msgr2=0x560940c82630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:55.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a21b4640 1 --2- 172.21.15.76:0/1287689506 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82250 0x560940c82630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.020 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a21b4640 1 -- 172.21.15.76:0/1287689506 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560940bac060 con 0x560940c8b750 2024-03-12T06:05:55.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a11b2640 1 --2- 172.21.15.76:0/1287689506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82b70 0x560940c8b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T06:05:55.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f27a21b4640 1 --2- 172.21.15.76:0/1287689506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940c8db40 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f2794009cd0 tx=0x7f279404a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37fd0f3c2df319b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:55.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.006+0000 7f2787fff640 1 -- 172.21.15.76:0/1287689506 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27940347c0 con 0x560940c8b750 2024-03-12T06:05:55.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.007+0000 7f2787fff640 1 -- 172.21.15.76:0/1287689506 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2794003970 con 0x560940c8b750 2024-03-12T06:05:55.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.007+0000 7f2787fff640 1 -- 172.21.15.76:0/1287689506 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27940044b0 con 0x560940c8b750 2024-03-12T06:05:55.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.007+0000 7f27a67e0740 1 -- 172.21.15.76:0/1287689506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 msgr2=0x560940c8db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:55.021 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.007+0000 7f27a67e0740 1 --2- 172.21.15.76:0/1287689506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940c8db40 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f2794009cd0 tx=0x7f279404a000 comp rx=0 tx=0).stop 2024-03-12T06:05:55.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.007+0000 7f27a67e0740 1 -- 172.21.15.76:0/1287689506 shutdown_connections 2024-03-12T06:05:55.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.007+0000 7f27a67e0740 1 --2- 172.21.15.76:0/1287689506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940c8db40 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.007+0000 7f27a67e0740 1 --2- 172.21.15.76:0/1287689506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82b70 0x560940c8b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.007+0000 7f27a67e0740 1 --2- 172.21.15.76:0/1287689506 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82250 0x560940c82630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.007+0000 7f27a67e0740 1 -- 172.21.15.76:0/1287689506 >> 172.21.15.76:0/1287689506 conn(0x560940c7c980 msgr2=0x560940c7cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:55.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.007+0000 7f27a67e0740 1 -- 172.21.15.76:0/1287689506 shutdown_connections 2024-03-12T06:05:55.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.007+0000 7f27a67e0740 1 -- 172.21.15.76:0/1287689506 wait complete. 2024-03-12T06:05:55.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a67e0740 1 -- start start 2024-03-12T06:05:55.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a67e0740 1 Processor -- start 2024-03-12T06:05:55.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a67e0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82250 0x560940c88760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a67e0740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82b70 0x560940c88ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a67e0740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940c891e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a67e0740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560940bea020 con 0x560940c8b750 2024-03-12T06:05:55.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a67e0740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609408bd430 con 0x560940c82b70 2024-03-12T06:05:55.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a67e0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560940879e00 con 0x560940c82250 2024-03-12T06:05:55.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a67e0740 10 client.0 _ll_register_callbacks cb 0x560940b91cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:55.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a21b4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940c891e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a21b4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940c891e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52968/0 (socket says 172.21.15.76:52968) 2024-03-12T06:05:55.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a21b4640 1 -- 172.21.15.76:0/1051877578 learned_addr learned my addr 172.21.15.76:0/1051877578 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:55.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a11b2640 1 --2- 172.21.15.76:0/1051877578 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82b70 0x560940c88ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a19b3640 1 --2- 172.21.15.76:0/1051877578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82250 0x560940c88760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a21b4640 1 -- 172.21.15.76:0/1051877578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82250 msgr2=0x560940c88760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:55.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a21b4640 1 --2- 172.21.15.76:0/1051877578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82250 0x560940c88760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a21b4640 1 -- 172.21.15.76:0/1051877578 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82b70 msgr2=0x560940c88ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:55.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.008+0000 7f27a21b4640 1 --2- 172.21.15.76:0/1051877578 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82b70 0x560940c88ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.024 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27a21b4640 1 -- 172.21.15.76:0/1051877578 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279c10e820 con 0x560940c8b750 2024-03-12T06:05:55.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27a19b3640 1 --2- 172.21.15.76:0/1051877578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82250 0x560940c88760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:55.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27a21b4640 1 --2- 172.21.15.76:0/1051877578 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940c891e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f2794037040 tx=0x7f2794032c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de61a1e9be17003d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:55.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27857fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:55.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27857fa640 1 -- 172.21.15.76:0/1051877578 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27940041b0 con 0x560940c8b750 2024-03-12T06:05:55.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27857fa640 1 -- 172.21.15.76:0/1051877578 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f2794004310 con 0x560940c8b750 2024-03-12T06:05:55.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27a67e0740 1 -- 172.21.15.76:0/1051877578 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5609408e97e0 con 0x560940c8b750 2024-03-12T06:05:55.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27857fa640 1 -- 172.21.15.76:0/1051877578 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27940044d0 con 0x560940c8b750 2024-03-12T06:05:55.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27a67e0740 10 client.17772 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:55.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27a67e0740 1 -- 172.21.15.76:0/1051877578 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560940be86a0 con 0x560940c8b750 2024-03-12T06:05:55.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27857fa640 1 -- 172.21.15.76:0/1051877578 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2794066080 con 0x560940c8b750 2024-03-12T06:05:55.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27857fa640 1 -- 172.21.15.76:0/1051877578 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f279404b070 con 0x560940c8b750 2024-03-12T06:05:55.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27a67e0740 10 client.17772 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:55.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27a67e0740 10 client.17772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:05:55.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27a67e0740 20 client.17772 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:55.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27a67e0740 1 --2- 172.21.15.76:0/1051877578 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560940c9b830 0x560940c9dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27a67e0740 4 client.17772 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-12T06:05:55.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.009+0000 7f27a67e0740 1 -- 172.21.15.76:0/1051877578 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560940926720 con 0x560940c9b830 2024-03-12T06:05:55.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.010+0000 7f27a19b3640 1 --2- 172.21.15.76:0/1051877578 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560940c9b830 0x560940c9dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.010+0000 7f27a19b3640 1 --2- 172.21.15.76:0/1051877578 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560940c9b830 0x560940c9dc20 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b51066afbb65241 server_cookie=ffe6a59a7f6dc6c9 in_seq=0 out_seq=0 2024-03-12T06:05:55.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.010+0000 7f27857fa640 10 client.17772 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:55.029 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27857fa640 1 -- 172.21.15.76:0/1051877578 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560940926720 con 0x560940c9b830 2024-03-12T06:05:55.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27857fa640 10 client.17772 handle_command_reply: tid=0 2024-03-12T06:05:55.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27a67e0740 1 client.17772 shutdown 2024-03-12T06:05:55.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27a67e0740 1 -- 172.21.15.76:0/1051877578 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 msgr2=0x560940c891e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:55.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27a67e0740 1 --2- 172.21.15.76:0/1051877578 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940c891e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f2794037040 tx=0x7f2794032c30 comp rx=0 tx=0).stop 2024-03-12T06:05:55.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27a67e0740 1 -- 172.21.15.76:0/1051877578 shutdown_connections 2024-03-12T06:05:55.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27a67e0740 1 --2- 172.21.15.76:0/1051877578 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560940c9b830 0x560940c9dc20 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27a67e0740 1 --2- 172.21.15.76:0/1051877578 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940c891e0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27857fa640 0 client.17772 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:55.030 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27a67e0740 1 --2- 172.21.15.76:0/1051877578 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82b70 0x560940c88ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27a67e0740 1 --2- 172.21.15.76:0/1051877578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82250 0x560940c88760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27a67e0740 1 -- 172.21.15.76:0/1051877578 >> 172.21.15.76:0/1051877578 conn(0x560940c7c980 msgr2=0x560940c7e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:55.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27a67e0740 1 -- 172.21.15.76:0/1051877578 shutdown_connections 2024-03-12T06:05:55.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27a67e0740 1 -- 172.21.15.76:0/1051877578 wait complete. 2024-03-12T06:05:55.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.011+0000 7f27a67e0740 20 client.17772 trim_cache size 0 max 16384 2024-03-12T06:05:55.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a67e0740 1 Processor -- start 2024-03-12T06:05:55.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a67e0740 1 -- start start 2024-03-12T06:05:55.037 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a67e0740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82250 0x560940c8ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a67e0740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c82b70 0x560940c89000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a67e0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c8b750 0x560940c859e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a67e0740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560940b6f130 con 0x560940c82b70 2024-03-12T06:05:55.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a67e0740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560940c9b630 con 0x560940c82250 2024-03-12T06:05:55.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a67e0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560940c84380 con 0x560940c8b750 2024-03-12T06:05:55.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a11b2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c82b70 0x560940c89000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a11b2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c82b70 0x560940c89000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52982/0 (socket says 172.21.15.76:52982) 2024-03-12T06:05:55.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a11b2640 1 -- 172.21.15.76:0/860903568 learned_addr learned my addr 172.21.15.76:0/860903568 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:55.038 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a19b3640 1 --2- 172.21.15.76:0/860903568 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82250 0x560940c8ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a21b4640 1 --2- 172.21.15.76:0/860903568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c8b750 0x560940c859e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a11b2640 1 -- 172.21.15.76:0/860903568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c8b750 msgr2=0x560940c859e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:55.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a11b2640 1 --2- 172.21.15.76:0/860903568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c8b750 0x560940c859e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a11b2640 1 -- 172.21.15.76:0/860903568 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82250 msgr2=0x560940c8ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:55.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a11b2640 1 --2- 172.21.15.76:0/860903568 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82250 0x560940c8ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a11b2640 1 -- 172.21.15.76:0/860903568 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560940b84c20 con 0x560940c82b70 2024-03-12T06:05:55.039 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a19b3640 1 --2- 172.21.15.76:0/860903568 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82250 0x560940c8ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:55.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.023+0000 7f27a11b2640 1 --2- 172.21.15.76:0/860903568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c82b70 0x560940c89000 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f279c100330 tx=0x7f2798012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=305b7f8eccb9c13d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:55.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.024+0000 7f27877fe640 1 -- 172.21.15.76:0/860903568 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2798017070 con 0x560940c82b70 2024-03-12T06:05:55.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.024+0000 7f27877fe640 1 -- 172.21.15.76:0/860903568 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f279800bdc0 con 0x560940c82b70 2024-03-12T06:05:55.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.024+0000 7f27877fe640 1 -- 172.21.15.76:0/860903568 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2798010630 con 0x560940c82b70 2024-03-12T06:05:55.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.024+0000 7f27a67e0740 1 -- 172.21.15.76:0/860903568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c82b70 msgr2=0x560940c89000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:55.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.024+0000 7f27a67e0740 1 --2- 172.21.15.76:0/860903568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c82b70 0x560940c89000 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f279c100330 tx=0x7f2798012820 comp rx=0 tx=0).stop 2024-03-12T06:05:55.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.024+0000 7f27a67e0740 1 -- 172.21.15.76:0/860903568 shutdown_connections 2024-03-12T06:05:55.040 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.024+0000 7f27a67e0740 1 --2- 172.21.15.76:0/860903568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c8b750 0x560940c859e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.024+0000 7f27a67e0740 1 --2- 172.21.15.76:0/860903568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c82b70 0x560940c89000 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.024+0000 7f27a67e0740 1 --2- 172.21.15.76:0/860903568 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82250 0x560940c8ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.024+0000 7f27a67e0740 1 -- 172.21.15.76:0/860903568 >> 172.21.15.76:0/860903568 conn(0x560940c7c980 msgr2=0x560940ca2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:55.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.024+0000 7f27a67e0740 1 -- 172.21.15.76:0/860903568 shutdown_connections 2024-03-12T06:05:55.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.024+0000 7f27a67e0740 1 -- 172.21.15.76:0/860903568 wait complete. 2024-03-12T06:05:55.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.024+0000 7f27a67e0740 1 -- start start 2024-03-12T06:05:55.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a67e0740 1 Processor -- start 2024-03-12T06:05:55.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a67e0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82250 0x560940c91770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.041 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a67e0740 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82b70 0x560940c91cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a67e0740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940cf1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a67e0740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x560940cf2220 con 0x560940c8b750 2024-03-12T06:05:55.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a67e0740 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x560940cf2360 con 0x560940c82b70 2024-03-12T06:05:55.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a67e0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560940cf24d0 con 0x560940c82250 2024-03-12T06:05:55.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a67e0740 10 client.0 _ll_register_callbacks cb 0x7f2794037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-12T06:05:55.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a21b4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940cf1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.042 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a21b4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940cf1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:52994/0 (socket says 172.21.15.76:52994) 2024-03-12T06:05:55.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a21b4640 1 -- 172.21.15.76:0/738437524 learned_addr learned my addr 172.21.15.76:0/738437524 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:55.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a19b3640 1 --2- 172.21.15.76:0/738437524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82250 0x560940c91770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a11b2640 1 --2- 172.21.15.76:0/738437524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82b70 0x560940c91cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a21b4640 1 -- 172.21.15.76:0/738437524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82250 msgr2=0x560940c91770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:55.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a21b4640 1 --2- 172.21.15.76:0/738437524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82250 0x560940c91770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a21b4640 1 -- 172.21.15.76:0/738437524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82b70 msgr2=0x560940c91cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-12T06:05:55.043 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a21b4640 1 --2- 172.21.15.76:0/738437524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82b70 0x560940c91cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a21b4640 1 -- 172.21.15.76:0/738437524 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279800b920 con 0x560940c8b750 2024-03-12T06:05:55.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27a21b4640 1 --2- 172.21.15.76:0/738437524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940cf1ce0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f27940348f0 tx=0x7f2794009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0774ccbaecbab3c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:55.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.025+0000 7f27857fa640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-03-12T06:05:55.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27857fa640 1 -- 172.21.15.76:0/738437524 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794058840 con 0x560940c8b750 2024-03-12T06:05:55.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27857fa640 1 -- 172.21.15.76:0/738437524 <== mon.0 v2:172.21.15.76:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f2794003d90 con 0x560940c8b750 2024-03-12T06:05:55.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27a67e0740 1 -- 172.21.15.76:0/738437524 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2794003970 con 0x560940c8b750 2024-03-12T06:05:55.044 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27857fa640 1 -- 172.21.15.76:0/738437524 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f279404ddd0 con 0x560940c8b750 2024-03-12T06:05:55.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27a67e0740 10 client.17778 fetch_fsmap learned FSMap version 28 2024-03-12T06:05:55.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27a67e0740 1 -- 172.21.15.76:0/738437524 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560940baa8c0 con 0x560940c8b750 2024-03-12T06:05:55.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27857fa640 1 -- 172.21.15.76:0/738437524 <== mon.0 v2:172.21.15.76:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2794071020 con 0x560940c8b750 2024-03-12T06:05:55.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27857fa640 1 -- 172.21.15.76:0/738437524 <== mon.0 v2:172.21.15.76:3300/0 5 ==== fsmap(e 28) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f279404b040 con 0x560940c8b750 2024-03-12T06:05:55.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27a67e0740 10 client.17778 fetch_fsmap finished waiting for FSMap version 28 2024-03-12T06:05:55.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27a67e0740 10 client.17778 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-12T06:05:55.045 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27a67e0740 20 client.17778 populate_metadata read hostname 'smithi076' 2024-03-12T06:05:55.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27a67e0740 1 --2- 172.21.15.76:0/738437524 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560940cf3950 0x560940cf5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:55.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27a67e0740 4 client.17778 mds_command: new command op to 24518 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-12T06:05:55.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27a67e0740 1 -- 172.21.15.76:0/738437524 --> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560940975890 con 0x560940cf3950 2024-03-12T06:05:55.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.026+0000 7f27a19b3640 1 --2- 172.21.15.76:0/738437524 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560940cf3950 0x560940cf5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:55.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27857fa640 10 client.17778 ms_handle_connect on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:55.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27a19b3640 1 --2- 172.21.15.76:0/738437524 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560940cf3950 0x560940cf5d70 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b46d7c502e45a563 server_cookie=8f7b6864a019f945 in_seq=0 out_seq=0 2024-03-12T06:05:55.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27857fa640 1 -- 172.21.15.76:0/738437524 <== mds.0 v2:172.21.15.82:6838/2242595718 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560940975890 con 0x560940cf3950 2024-03-12T06:05:55.046 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27857fa640 10 client.17778 handle_command_reply: tid=0 2024-03-12T06:05:55.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27a67e0740 1 client.17778 shutdown 2024-03-12T06:05:55.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27a67e0740 1 -- 172.21.15.76:0/738437524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 msgr2=0x560940cf1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:55.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27a67e0740 1 --2- 172.21.15.76:0/738437524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940cf1ce0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f27940348f0 tx=0x7f2794009890 comp rx=0 tx=0).stop 2024-03-12T06:05:55.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27a67e0740 1 -- 172.21.15.76:0/738437524 shutdown_connections 2024-03-12T06:05:55.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27a67e0740 1 --2- 172.21.15.76:0/738437524 >> [v2:172.21.15.82:6838/2242595718,v1:172.21.15.82:6839/2242595718] conn(0x560940cf3950 0x560940cf5d70 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27a67e0740 1 --2- 172.21.15.76:0/738437524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x560940c8b750 0x560940cf1ce0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27857fa640 0 client.17778 ms_handle_reset on v2:172.21.15.82:6838/2242595718 2024-03-12T06:05:55.047 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27a67e0740 1 --2- 172.21.15.76:0/738437524 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x560940c82b70 0x560940c91cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27a67e0740 1 --2- 172.21.15.76:0/738437524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560940c82250 0x560940c91770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27a67e0740 1 -- 172.21.15.76:0/738437524 >> 172.21.15.76:0/738437524 conn(0x560940c7c980 msgr2=0x560940c7b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:55.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27a67e0740 1 -- 172.21.15.76:0/738437524 shutdown_connections 2024-03-12T06:05:55.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27a67e0740 1 -- 172.21.15.76:0/738437524 wait complete. 2024-03-12T06:05:55.048 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.027+0000 7f27a67e0740 20 client.17778 trim_cache size 0 max 16384 2024-03-12T06:05:55.048 INFO:teuthology.orchestra.run.smithi076.stdout:[] 2024-03-12T06:05:55.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.028+0000 7f27a3c17640 1 -- 172.21.15.76:0/2188522658 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f27880a1ba0 msgr2=0x7f27880a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:55.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.028+0000 7f27a3c17640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f27880a1ba0 0x7f27880a4060 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2794005cc0 tx=0x7f2794032550 comp rx=0 tx=0).stop 2024-03-12T06:05:55.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.028+0000 7f27a3c17640 1 -- 172.21.15.76:0/2188522658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f279c101370 msgr2=0x7f279c19d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:55.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.028+0000 7f27a3c17640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f279c101370 0x7f279c19d470 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f278c00d770 tx=0x7f278c00dc30 comp rx=0 tx=0).stop 2024-03-12T06:05:55.050 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.028+0000 7f27a3c17640 1 -- 172.21.15.76:0/2188522658 shutdown_connections 2024-03-12T06:05:55.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.028+0000 7f27a3c17640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f27880a1ba0 0x7f27880a4060 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.028+0000 7f27a3c17640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f279c197670 0x7f279c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.028+0000 7f27a3c17640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f279c106440 0x7f279c19d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.028+0000 7f27a3c17640 1 --2- 172.21.15.76:0/2188522658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f279c101370 0x7f279c19d470 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:55.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.028+0000 7f27a3c17640 1 -- 172.21.15.76:0/2188522658 >> 172.21.15.76:0/2188522658 conn(0x7f279c0fcbf0 msgr2=0x7f279c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:55.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.028+0000 7f27a3c17640 1 -- 172.21.15.76:0/2188522658 shutdown_connections 2024-03-12T06:05:55.051 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:55.028+0000 7f27a3c17640 1 -- 172.21.15.76:0/2188522658 wait complete. 2024-03-12T06:05:55.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:55 smithi082 ceph-mon[36382]: pgmap v313: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-12T06:05:55.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:55 smithi088 ceph-mon[36585]: pgmap v313: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-12T06:05:55.989 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:55 smithi076 ceph-mon[32095]: pgmap v313: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-12T06:05:56.053 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-12T06:05:56.053 INFO:tasks.fwd_scrub:done joining 2024-03-12T06:05:56.053 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-12T06:05:56.064 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T06:05:56.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.171+0000 7f5e160f2640 1 -- 172.21.15.76:0/2113265458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e100ff9f0 msgr2=0x7f5e100ffe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.171+0000 7f5e160f2640 1 --2- 172.21.15.76:0/2113265458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e100ff9f0 0x7f5e100ffe70 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f5df8009930 tx=0x7f5df802bc30 comp rx=0 tx=0).stop 2024-03-12T06:05:56.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.171+0000 7f5e160f2640 1 -- 172.21.15.76:0/2113265458 shutdown_connections 2024-03-12T06:05:56.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.171+0000 7f5e160f2640 1 --2- 172.21.15.76:0/2113265458 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5e101003b0 0x7f5e1010e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.171+0000 7f5e160f2640 1 --2- 172.21.15.76:0/2113265458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e100ff9f0 0x7f5e100ffe70 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.179 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.171+0000 7f5e160f2640 1 --2- 172.21.15.76:0/2113265458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e101062c0 0x7f5e101066a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.171+0000 7f5e160f2640 1 -- 172.21.15.76:0/2113265458 >> 172.21.15.76:0/2113265458 conn(0x7f5e10076540 msgr2=0x7f5e10076950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:56.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.171+0000 7f5e160f2640 1 -- 172.21.15.76:0/2113265458 shutdown_connections 2024-03-12T06:05:56.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.171+0000 7f5e160f2640 1 -- 172.21.15.76:0/2113265458 wait complete. 2024-03-12T06:05:56.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e160f2640 1 Processor -- start 2024-03-12T06:05:56.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e160f2640 1 -- start start 2024-03-12T06:05:56.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e160f2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5e100ff9f0 0x7f5e1019d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:56.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e160f2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e101003b0 0x7f5e1019db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:56.180 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e160f2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e101062c0 0x7f5e101976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:56.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e160f2640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10197e80 con 0x7f5e101062c0 2024-03-12T06:05:56.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e160f2640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10197ff0 con 0x7f5e100ff9f0 2024-03-12T06:05:56.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e160f2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e101981b0 con 0x7f5e101003b0 2024-03-12T06:05:56.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e0ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e101062c0 0x7f5e101976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:56.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e0ffff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e101062c0 0x7f5e101976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53012/0 (socket says 172.21.15.76:53012) 2024-03-12T06:05:56.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e0ffff640 1 -- 172.21.15.76:0/3692965861 learned_addr learned my addr 172.21.15.76:0/3692965861 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:56.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e0f7fe640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5e100ff9f0 0x7f5e1019d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:56.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e07fff640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e101003b0 0x7f5e1019db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:56.181 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e0ffff640 1 -- 172.21.15.76:0/3692965861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e101003b0 msgr2=0x7f5e1019db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e0ffff640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e101003b0 0x7f5e1019db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e0ffff640 1 -- 172.21.15.76:0/3692965861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5e100ff9f0 msgr2=0x7f5e1019d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e0ffff640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5e100ff9f0 0x7f5e1019d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e0ffff640 1 -- 172.21.15.76:0/3692965861 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df8009560 con 0x7f5e101062c0 2024-03-12T06:05:56.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.172+0000 7f5e07fff640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e101003b0 0x7f5e1019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:05:56.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.173+0000 7f5e0ffff640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e101062c0 0x7f5e101976f0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800eaa0 tx=0x7f5e0800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:56.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.173+0000 7f5e0d7fa640 1 -- 172.21.15.76:0/3692965861 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0800cdb0 con 0x7f5e101062c0 2024-03-12T06:05:56.182 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.173+0000 7f5e0d7fa640 1 -- 172.21.15.76:0/3692965861 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e08005020 con 0x7f5e101062c0 2024-03-12T06:05:56.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.173+0000 7f5e0d7fa640 1 -- 172.21.15.76:0/3692965861 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e080116b0 con 0x7f5e101062c0 2024-03-12T06:05:56.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.173+0000 7f5e160f2640 1 -- 172.21.15.76:0/3692965861 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e10198490 con 0x7f5e101062c0 2024-03-12T06:05:56.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.173+0000 7f5e160f2640 1 -- 172.21.15.76:0/3692965861 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e101a4560 con 0x7f5e101062c0 2024-03-12T06:05:56.183 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.174+0000 7f5e160f2640 1 -- 172.21.15.76:0/3692965861 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e10069440 con 0x7f5e101062c0 2024-03-12T06:05:56.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.175+0000 7f5e0d7fa640 1 -- 172.21.15.76:0/3692965861 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5e08005520 con 0x7f5e101062c0 2024-03-12T06:05:56.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.175+0000 7f5e0d7fa640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5dfc0a1c20 0x7f5dfc0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:56.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.176+0000 7f5e0f7fe640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5dfc0a1c20 0x7f5dfc0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:56.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.176+0000 7f5e0d7fa640 1 -- 172.21.15.76:0/3692965861 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5e080cd9c0 con 0x7f5e101062c0 2024-03-12T06:05:56.184 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.176+0000 7f5e0f7fe640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5dfc0a1c20 0x7f5dfc0a40e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5e00005d00 tx=0x7f5e0000b410 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:56.185 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.180+0000 7f5e0d7fa640 1 -- 172.21.15.76:0/3692965861 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5e08096890 con 0x7f5e101062c0 2024-03-12T06:05:56.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.353+0000 7f5e160f2640 1 -- 172.21.15.76:0/3692965861 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e10068450 con 0x7f5e101062c0 2024-03-12T06:05:56.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.355+0000 7f5e0d7fa640 1 -- 172.21.15.76:0/3692965861 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+23008 (secure 0 0 0) 0x7f5e08099f50 con 0x7f5e101062c0 2024-03-12T06:05:56.360 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:05:56.361 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":95,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T06:04:11.487705+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":94,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6800/1718825251":"2024-03-13T05:55:46.980746+0000","172.21.15.76:0/189327369":"2024-03-13T05:55:46.980746+0000","172.21.15.76:0/3998885619":"2024-03-13T05:55:46.980746+0000","172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/2299856451":"2024-03-13T05:55:46.980746+0000","172.21.15.76:6801/1718825251":"2024-03-13T05:55:46.980746+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1939795424":"2024-03-13T05:55:46.980746+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":4,"length":1},{"begin":13,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T06:05:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.357+0000 7f5e160f2640 1 -- 172.21.15.76:0/3692965861 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5dfc0a1c20 msgr2=0x7f5dfc0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.357+0000 7f5e160f2640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5dfc0a1c20 0x7f5dfc0a40e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5e00005d00 tx=0x7f5e0000b410 comp rx=0 tx=0).stop 2024-03-12T06:05:56.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.357+0000 7f5e160f2640 1 -- 172.21.15.76:0/3692965861 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e101062c0 msgr2=0x7f5e101976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.357+0000 7f5e160f2640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e101062c0 0x7f5e101976f0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800eaa0 tx=0x7f5e0800ef60 comp rx=0 tx=0).stop 2024-03-12T06:05:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.357+0000 7f5e160f2640 1 -- 172.21.15.76:0/3692965861 shutdown_connections 2024-03-12T06:05:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.357+0000 7f5e160f2640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5dfc0a1c20 0x7f5dfc0a40e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.357+0000 7f5e160f2640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5e101062c0 0x7f5e101976f0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.357+0000 7f5e160f2640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e101003b0 0x7f5e1019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.357+0000 7f5e160f2640 1 --2- 172.21.15.76:0/3692965861 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5e100ff9f0 0x7f5e1019d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.357+0000 7f5e160f2640 1 -- 172.21.15.76:0/3692965861 >> 172.21.15.76:0/3692965861 conn(0x7f5e10076540 msgr2=0x7f5e100fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:56.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.357+0000 7f5e160f2640 1 -- 172.21.15.76:0/3692965861 shutdown_connections 2024-03-12T06:05:56.364 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.357+0000 7f5e160f2640 1 -- 172.21.15.76:0/3692965861 wait complete. 2024-03-12T06:05:56.374 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:49:09.125536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:53:35.663129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:53:36.740826+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 13, 'snap_epoch': 94, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T06:05:56.374 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T06:05:56.481 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.474+0000 7f5b94f4f640 1 -- 172.21.15.76:0/3175820294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b90100c60 msgr2=0x7f5b90104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.474+0000 7f5b94f4f640 1 --2- 172.21.15.76:0/3175820294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b90100c60 0x7f5b90104db0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009930 tx=0x7f5b7802bc30 comp rx=0 tx=0).stop 2024-03-12T06:05:56.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.474+0000 7f5b94f4f640 1 -- 172.21.15.76:0/3175820294 shutdown_connections 2024-03-12T06:05:56.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.474+0000 7f5b94f4f640 1 --2- 172.21.15.76:0/3175820294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b90105490 0x7f5b9010d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.474+0000 7f5b94f4f640 1 --2- 172.21.15.76:0/3175820294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b90100c60 0x7f5b90104db0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.482 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.474+0000 7f5b94f4f640 1 --2- 172.21.15.76:0/3175820294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b901002b0 0x7f5b90100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.474+0000 7f5b94f4f640 1 -- 172.21.15.76:0/3175820294 >> 172.21.15.76:0/3175820294 conn(0x7f5b900771f0 msgr2=0x7f5b90077600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:56.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.474+0000 7f5b94f4f640 1 -- 172.21.15.76:0/3175820294 shutdown_connections 2024-03-12T06:05:56.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.474+0000 7f5b94f4f640 1 -- 172.21.15.76:0/3175820294 wait complete. 2024-03-12T06:05:56.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.475+0000 7f5b94f4f640 1 Processor -- start 2024-03-12T06:05:56.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.475+0000 7f5b94f4f640 1 -- start start 2024-03-12T06:05:56.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.475+0000 7f5b94f4f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b901002b0 0x7f5b9019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:56.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.475+0000 7f5b94f4f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b90105490 0x7f5b9019fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:56.483 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.475+0000 7f5b94f4f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b90199960 0x7f5b90199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:56.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.475+0000 7f5b94f4f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9019a350 con 0x7f5b90199960 2024-03-12T06:05:56.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.475+0000 7f5b94f4f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9019a4c0 con 0x7f5b901002b0 2024-03-12T06:05:56.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.475+0000 7f5b94f4f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9019a680 con 0x7f5b90105490 2024-03-12T06:05:56.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.475+0000 7f5b8ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b90199960 0x7f5b90199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:56.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.475+0000 7f5b8ed9d640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b90199960 0x7f5b90199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53038/0 (socket says 172.21.15.76:53038) 2024-03-12T06:05:56.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.475+0000 7f5b8ed9d640 1 -- 172.21.15.76:0/2035306009 learned_addr learned my addr 172.21.15.76:0/2035306009 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:56.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.475+0000 7f5b8e59c640 1 --2- 172.21.15.76:0/2035306009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b901002b0 0x7f5b9019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:56.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.476+0000 7f5b87fff640 1 --2- 172.21.15.76:0/2035306009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b90105490 0x7f5b9019fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:56.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.476+0000 7f5b8ed9d640 1 -- 172.21.15.76:0/2035306009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b90105490 msgr2=0x7f5b9019fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.484 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.476+0000 7f5b8ed9d640 1 --2- 172.21.15.76:0/2035306009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b90105490 0x7f5b9019fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.476+0000 7f5b8ed9d640 1 -- 172.21.15.76:0/2035306009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b901002b0 msgr2=0x7f5b9019f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.476+0000 7f5b8ed9d640 1 --2- 172.21.15.76:0/2035306009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b901002b0 0x7f5b9019f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.476+0000 7f5b8ed9d640 1 -- 172.21.15.76:0/2035306009 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b78009560 con 0x7f5b90199960 2024-03-12T06:05:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.476+0000 7f5b8ed9d640 1 --2- 172.21.15.76:0/2035306009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b90199960 0x7f5b90199e10 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f5b8800da40 tx=0x7f5b8800df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.476+0000 7f5b877fe640 1 -- 172.21.15.76:0/2035306009 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8800bdd0 con 0x7f5b90199960 2024-03-12T06:05:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.476+0000 7f5b877fe640 1 -- 172.21.15.76:0/2035306009 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b88009890 con 0x7f5b90199960 2024-03-12T06:05:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.476+0000 7f5b94f4f640 1 -- 172.21.15.76:0/2035306009 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b901a6670 con 0x7f5b90199960 2024-03-12T06:05:56.485 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.476+0000 7f5b877fe640 1 -- 172.21.15.76:0/2035306009 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b880106c0 con 0x7f5b90199960 2024-03-12T06:05:56.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.476+0000 7f5b94f4f640 1 -- 172.21.15.76:0/2035306009 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b901a6b30 con 0x7f5b90199960 2024-03-12T06:05:56.486 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.477+0000 7f5b94f4f640 1 -- 172.21.15.76:0/2035306009 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b90101780 con 0x7f5b90199960 2024-03-12T06:05:56.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.478+0000 7f5b877fe640 1 -- 172.21.15.76:0/2035306009 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f5b880041b0 con 0x7f5b90199960 2024-03-12T06:05:56.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.479+0000 7f5b877fe640 1 --2- 172.21.15.76:0/2035306009 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5b7c0a1ba0 0x7f5b7c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:56.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.479+0000 7f5b8e59c640 1 --2- 172.21.15.76:0/2035306009 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5b7c0a1ba0 0x7f5b7c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:56.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.479+0000 7f5b877fe640 1 -- 172.21.15.76:0/2035306009 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5b880cd460 con 0x7f5b90199960 2024-03-12T06:05:56.487 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.479+0000 7f5b8e59c640 1 --2- 172.21.15.76:0/2035306009 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5b7c0a1ba0 0x7f5b7c0a4060 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5b800058c0 tx=0x7f5b8000a250 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:56.488 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.483+0000 7f5b877fe640 1 -- 172.21.15.76:0/2035306009 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5b88096330 con 0x7f5b90199960 2024-03-12T06:05:56.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.658+0000 7f5b94f4f640 1 -- 172.21.15.76:0/2035306009 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5b90103490 con 0x7f5b90199960 2024-03-12T06:05:56.663 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.658+0000 7f5b877fe640 1 -- 172.21.15.76:0/2035306009 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5b880999f0 con 0x7f5b90199960 2024-03-12T06:05:56.663 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-12T06:05:56.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.660+0000 7f5b94f4f640 1 -- 172.21.15.76:0/2035306009 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5b7c0a1ba0 msgr2=0x7f5b7c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.660+0000 7f5b94f4f640 1 --2- 172.21.15.76:0/2035306009 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5b7c0a1ba0 0x7f5b7c0a4060 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5b800058c0 tx=0x7f5b8000a250 comp rx=0 tx=0).stop 2024-03-12T06:05:56.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.660+0000 7f5b94f4f640 1 -- 172.21.15.76:0/2035306009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b90199960 msgr2=0x7f5b90199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.665 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.660+0000 7f5b94f4f640 1 --2- 172.21.15.76:0/2035306009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b90199960 0x7f5b90199e10 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f5b8800da40 tx=0x7f5b8800df00 comp rx=0 tx=0).stop 2024-03-12T06:05:56.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.660+0000 7f5b94f4f640 1 -- 172.21.15.76:0/2035306009 shutdown_connections 2024-03-12T06:05:56.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.660+0000 7f5b94f4f640 1 --2- 172.21.15.76:0/2035306009 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f5b7c0a1ba0 0x7f5b7c0a4060 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.660+0000 7f5b94f4f640 1 --2- 172.21.15.76:0/2035306009 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5b90199960 0x7f5b90199e10 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.660+0000 7f5b94f4f640 1 --2- 172.21.15.76:0/2035306009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b90105490 0x7f5b9019fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.660+0000 7f5b94f4f640 1 --2- 172.21.15.76:0/2035306009 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5b901002b0 0x7f5b9019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.660+0000 7f5b94f4f640 1 -- 172.21.15.76:0/2035306009 >> 172.21.15.76:0/2035306009 conn(0x7f5b900771f0 msgr2=0x7f5b900ff330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:56.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.660+0000 7f5b94f4f640 1 -- 172.21.15.76:0/2035306009 shutdown_connections 2024-03-12T06:05:56.666 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.660+0000 7f5b94f4f640 1 -- 172.21.15.76:0/2035306009 wait complete. 2024-03-12T06:05:56.676 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T06:05:56.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.777+0000 7ff699eff640 1 -- 172.21.15.76:0/2032278467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff694106460 msgr2=0x7ff69410e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.777+0000 7ff699eff640 1 --2- 172.21.15.76:0/2032278467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff694106460 0x7ff69410e030 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7ff68c009cd0 tx=0x7ff68c02c0f0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.777+0000 7ff699eff640 1 -- 172.21.15.76:0/2032278467 shutdown_connections 2024-03-12T06:05:56.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.777+0000 7ff699eff640 1 --2- 172.21.15.76:0/2032278467 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff694106460 0x7ff69410e030 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.777+0000 7ff699eff640 1 --2- 172.21.15.76:0/2032278467 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff694101d40 0x7ff694105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.777+0000 7ff699eff640 1 --2- 172.21.15.76:0/2032278467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff694101390 0x7ff694101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.784 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.777+0000 7ff699eff640 1 -- 172.21.15.76:0/2032278467 >> 172.21.15.76:0/2032278467 conn(0x7ff6940fcc10 msgr2=0x7ff6940ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:56.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.777+0000 7ff699eff640 1 -- 172.21.15.76:0/2032278467 shutdown_connections 2024-03-12T06:05:56.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.777+0000 7ff699eff640 1 -- 172.21.15.76:0/2032278467 wait complete. 2024-03-12T06:05:56.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff699eff640 1 Processor -- start 2024-03-12T06:05:56.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff699eff640 1 -- start start 2024-03-12T06:05:56.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff699eff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff694101390 0x7ff694197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:56.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff699eff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff694106460 0x7ff694197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:56.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff699eff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff69419bff0 0x7ff69419c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:56.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff699eff640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69419c9e0 con 0x7ff69419bff0 2024-03-12T06:05:56.785 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff699eff640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69419cb50 con 0x7ff694106460 2024-03-12T06:05:56.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff699eff640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69419ccc0 con 0x7ff694101390 2024-03-12T06:05:56.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff693fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff69419bff0 0x7ff69419c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:56.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff693fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff69419bff0 0x7ff69419c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53052/0 (socket says 172.21.15.76:53052) 2024-03-12T06:05:56.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff693fff640 1 -- 172.21.15.76:0/3394628853 learned_addr learned my addr 172.21.15.76:0/3394628853 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:56.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff6937fe640 1 --2- 172.21.15.76:0/3394628853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff694101390 0x7ff694197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:56.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff692ffd640 1 --2- 172.21.15.76:0/3394628853 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff694106460 0x7ff694197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:56.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff693fff640 1 -- 172.21.15.76:0/3394628853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff694101390 msgr2=0x7ff694197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.786 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff693fff640 1 --2- 172.21.15.76:0/3394628853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff694101390 0x7ff694197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff693fff640 1 -- 172.21.15.76:0/3394628853 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff694106460 msgr2=0x7ff694197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff693fff640 1 --2- 172.21.15.76:0/3394628853 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff694106460 0x7ff694197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.778+0000 7ff693fff640 1 -- 172.21.15.76:0/3394628853 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68c009900 con 0x7ff69419bff0 2024-03-12T06:05:56.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.779+0000 7ff693fff640 1 --2- 172.21.15.76:0/3394628853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff69419bff0 0x7ff69419c4a0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0348f0 tx=0x7ff68c034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:56.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.779+0000 7ff690ff9640 1 -- 172.21.15.76:0/3394628853 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c0328c0 con 0x7ff69419bff0 2024-03-12T06:05:56.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.779+0000 7ff690ff9640 1 -- 172.21.15.76:0/3394628853 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff68c003e50 con 0x7ff69419bff0 2024-03-12T06:05:56.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.779+0000 7ff699eff640 1 -- 172.21.15.76:0/3394628853 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6941a42a0 con 0x7ff69419bff0 2024-03-12T06:05:56.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.779+0000 7ff690ff9640 1 -- 172.21.15.76:0/3394628853 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c040450 con 0x7ff69419bff0 2024-03-12T06:05:56.787 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.779+0000 7ff699eff640 1 -- 172.21.15.76:0/3394628853 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6941a46c0 con 0x7ff69419bff0 2024-03-12T06:05:56.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.780+0000 7ff699eff640 1 -- 172.21.15.76:0/3394628853 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff694068450 con 0x7ff69419bff0 2024-03-12T06:05:56.788 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.781+0000 7ff690ff9640 1 -- 172.21.15.76:0/3394628853 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff68c00a040 con 0x7ff69419bff0 2024-03-12T06:05:56.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.781+0000 7ff690ff9640 1 --2- 172.21.15.76:0/3394628853 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff67c0a1ba0 0x7ff67c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:56.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.781+0000 7ff6937fe640 1 --2- 172.21.15.76:0/3394628853 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff67c0a1ba0 0x7ff67c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:56.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.782+0000 7ff690ff9640 1 -- 172.21.15.76:0/3394628853 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7ff68c0f0620 con 0x7ff69419bff0 2024-03-12T06:05:56.789 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.782+0000 7ff6937fe640 1 --2- 172.21.15.76:0/3394628853 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff67c0a1ba0 0x7ff67c0a4060 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff684005fd0 tx=0x7ff684005850 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:56.790 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.786+0000 7ff690ff9640 1 -- 172.21.15.76:0/3394628853 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff68c0b9600 con 0x7ff69419bff0 2024-03-12T06:05:56.946 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:56 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3692965861' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T06:05:56.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.958+0000 7ff699eff640 1 -- 172.21.15.76:0/3394628853 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff694101d40 con 0x7ff69419bff0 2024-03-12T06:05:56.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.959+0000 7ff690ff9640 1 -- 172.21.15.76:0/3394628853 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff68c0bccc0 con 0x7ff69419bff0 2024-03-12T06:05:56.964 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-12T06:05:56.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.961+0000 7ff699eff640 1 -- 172.21.15.76:0/3394628853 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff67c0a1ba0 msgr2=0x7ff67c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.961+0000 7ff699eff640 1 --2- 172.21.15.76:0/3394628853 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff67c0a1ba0 0x7ff67c0a4060 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff684005fd0 tx=0x7ff684005850 comp rx=0 tx=0).stop 2024-03-12T06:05:56.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.961+0000 7ff699eff640 1 -- 172.21.15.76:0/3394628853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff69419bff0 msgr2=0x7ff69419c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:56.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.961+0000 7ff699eff640 1 --2- 172.21.15.76:0/3394628853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff69419bff0 0x7ff69419c4a0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0348f0 tx=0x7ff68c034cc0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.961+0000 7ff699eff640 1 -- 172.21.15.76:0/3394628853 shutdown_connections 2024-03-12T06:05:56.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.961+0000 7ff699eff640 1 --2- 172.21.15.76:0/3394628853 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff67c0a1ba0 0x7ff67c0a4060 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.961+0000 7ff699eff640 1 --2- 172.21.15.76:0/3394628853 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff69419bff0 0x7ff69419c4a0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.961+0000 7ff699eff640 1 --2- 172.21.15.76:0/3394628853 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff694106460 0x7ff694197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.961+0000 7ff699eff640 1 --2- 172.21.15.76:0/3394628853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff694101390 0x7ff694197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.961+0000 7ff699eff640 1 -- 172.21.15.76:0/3394628853 >> 172.21.15.76:0/3394628853 conn(0x7ff6940fcc10 msgr2=0x7ff6940fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.961+0000 7ff699eff640 1 -- 172.21.15.76:0/3394628853 shutdown_connections 2024-03-12T06:05:56.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:56.961+0000 7ff699eff640 1 -- 172.21.15.76:0/3394628853 wait complete. 2024-03-12T06:05:56.975 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:56 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3692965861' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T06:05:56.976 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T06:05:56.985 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:56 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3692965861' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T06:05:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.077+0000 7fde3eb65640 1 -- 172.21.15.76:0/3051100710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde38106150 msgr2=0x7fde38106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.077+0000 7fde3eb65640 1 --2- 172.21.15.76:0/3051100710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde38106150 0x7fde38106530 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009930 tx=0x7fde2c02bc30 comp rx=0 tx=0).stop 2024-03-12T06:05:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.077+0000 7fde3eb65640 1 -- 172.21.15.76:0/3051100710 shutdown_connections 2024-03-12T06:05:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.077+0000 7fde3eb65640 1 --2- 172.21.15.76:0/3051100710 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde380fffb0 0x7fde3810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.077+0000 7fde3eb65640 1 --2- 172.21.15.76:0/3051100710 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde380ff5f0 0x7fde380ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.077+0000 7fde3eb65640 1 --2- 172.21.15.76:0/3051100710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde38106150 0x7fde38106530 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.077+0000 7fde3eb65640 1 -- 172.21.15.76:0/3051100710 >> 172.21.15.76:0/3051100710 conn(0x7fde38076580 msgr2=0x7fde380fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.077+0000 7fde3eb65640 1 -- 172.21.15.76:0/3051100710 shutdown_connections 2024-03-12T06:05:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.077+0000 7fde3eb65640 1 -- 172.21.15.76:0/3051100710 wait complete. 2024-03-12T06:05:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3eb65640 1 Processor -- start 2024-03-12T06:05:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3eb65640 1 -- start start 2024-03-12T06:05:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3eb65640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde380ff5f0 0x7fde381007d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3eb65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde380fffb0 0x7fde38100d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:57.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3eb65640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde38106150 0x7fde38101250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3eb65640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde381a5ce0 con 0x7fde38106150 2024-03-12T06:05:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3eb65640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde381a5e50 con 0x7fde380ff5f0 2024-03-12T06:05:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3eb65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde381a6010 con 0x7fde380fffb0 2024-03-12T06:05:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3d102640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde38106150 0x7fde38101250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3d102640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde38106150 0x7fde38101250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53064/0 (socket says 172.21.15.76:53064) 2024-03-12T06:05:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3d102640 1 -- 172.21.15.76:0/14223536 learned_addr learned my addr 172.21.15.76:0/14223536 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde37fff640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde380fffb0 0x7fde38100d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3c901640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde380ff5f0 0x7fde381007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:57.086 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3d102640 1 -- 172.21.15.76:0/14223536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde380fffb0 msgr2=0x7fde38100d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3d102640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde380fffb0 0x7fde38100d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3d102640 1 -- 172.21.15.76:0/14223536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde380ff5f0 msgr2=0x7fde381007d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3d102640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde380ff5f0 0x7fde381007d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3d102640 1 -- 172.21.15.76:0/14223536 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c009560 con 0x7fde38106150 2024-03-12T06:05:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.078+0000 7fde3c901640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde380ff5f0 0x7fde381007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T06:05:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.079+0000 7fde3d102640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde38106150 0x7fde38101250 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fde3000b970 tx=0x7fde3000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.079+0000 7fde35ffb640 1 -- 172.21.15.76:0/14223536 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3000cb40 con 0x7fde38106150 2024-03-12T06:05:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.079+0000 7fde35ffb640 1 -- 172.21.15.76:0/14223536 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde3000cca0 con 0x7fde38106150 2024-03-12T06:05:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.079+0000 7fde3eb65640 1 -- 172.21.15.76:0/14223536 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde381a62f0 con 0x7fde38106150 2024-03-12T06:05:57.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.079+0000 7fde35ffb640 1 -- 172.21.15.76:0/14223536 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde300077d0 con 0x7fde38106150 2024-03-12T06:05:57.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.079+0000 7fde3eb65640 1 -- 172.21.15.76:0/14223536 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde381a67b0 con 0x7fde38106150 2024-03-12T06:05:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.080+0000 7fde3eb65640 1 -- 172.21.15.76:0/14223536 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde38101d00 con 0x7fde38106150 2024-03-12T06:05:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.081+0000 7fde35ffb640 1 -- 172.21.15.76:0/14223536 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fde30014440 con 0x7fde38106150 2024-03-12T06:05:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.081+0000 7fde35ffb640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fde280a1ba0 0x7fde280a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.081+0000 7fde3c901640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fde280a1ba0 0x7fde280a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.081+0000 7fde35ffb640 1 -- 172.21.15.76:0/14223536 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fde300cd070 con 0x7fde38106150 2024-03-12T06:05:57.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.082+0000 7fde3c901640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fde280a1ba0 0x7fde280a4060 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fde2c005ec0 tx=0x7fde2c000bc0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:57.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.085+0000 7fde35ffb640 1 -- 172.21.15.76:0/14223536 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fde3004fd20 con 0x7fde38106150 2024-03-12T06:05:57.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.261+0000 7fde3eb65640 1 -- 172.21.15.76:0/14223536 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fde38109680 con 0x7fde38106150 2024-03-12T06:05:57.266 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.261+0000 7fde35ffb640 1 -- 172.21.15.76:0/14223536 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7fde30019020 con 0x7fde38106150 2024-03-12T06:05:57.266 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-12T06:05:57.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.263+0000 7fde3eb65640 1 -- 172.21.15.76:0/14223536 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fde280a1ba0 msgr2=0x7fde280a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:57.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.263+0000 7fde3eb65640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fde280a1ba0 0x7fde280a4060 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fde2c005ec0 tx=0x7fde2c000bc0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.263+0000 7fde3eb65640 1 -- 172.21.15.76:0/14223536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde38106150 msgr2=0x7fde38101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:57.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.263+0000 7fde3eb65640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde38106150 0x7fde38101250 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fde3000b970 tx=0x7fde3000be30 comp rx=0 tx=0).stop 2024-03-12T06:05:57.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.263+0000 7fde3eb65640 1 -- 172.21.15.76:0/14223536 shutdown_connections 2024-03-12T06:05:57.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.263+0000 7fde3eb65640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fde280a1ba0 0x7fde280a4060 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.263+0000 7fde3eb65640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde38106150 0x7fde38101250 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.263+0000 7fde3eb65640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde380fffb0 0x7fde38100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.263+0000 7fde3eb65640 1 --2- 172.21.15.76:0/14223536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fde380ff5f0 0x7fde381007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.263+0000 7fde3eb65640 1 -- 172.21.15.76:0/14223536 >> 172.21.15.76:0/14223536 conn(0x7fde38076580 msgr2=0x7fde38104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:57.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.263+0000 7fde3eb65640 1 -- 172.21.15.76:0/14223536 shutdown_connections 2024-03-12T06:05:57.269 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.263+0000 7fde3eb65640 1 -- 172.21.15.76:0/14223536 wait complete. 2024-03-12T06:05:57.279 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-12T06:05:57.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.376+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3697257363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd08105350 msgr2=0x7efd0810cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:57.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.376+0000 7efd0dfe2640 1 --2- 172.21.15.76:0/3697257363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd08105350 0x7efd0810cf20 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7efd00009cd0 tx=0x7efd0002c0f0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.376+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3697257363 shutdown_connections 2024-03-12T06:05:57.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.376+0000 7efd0dfe2640 1 --2- 172.21.15.76:0/3697257363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd08105350 0x7efd0810cf20 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.383 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.376+0000 7efd0dfe2640 1 --2- 172.21.15.76:0/3697257363 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd08100c30 0x7efd08104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.376+0000 7efd0dfe2640 1 --2- 172.21.15.76:0/3697257363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd08100280 0x7efd08100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.376+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3697257363 >> 172.21.15.76:0/3697257363 conn(0x7efd080fbb00 msgr2=0x7efd080fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:57.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.376+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3697257363 shutdown_connections 2024-03-12T06:05:57.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.376+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3697257363 wait complete. 2024-03-12T06:05:57.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.376+0000 7efd0dfe2640 1 Processor -- start 2024-03-12T06:05:57.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd0dfe2640 1 -- start start 2024-03-12T06:05:57.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd0dfe2640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd08100280 0x7efd081976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:57.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd0dfe2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd08105350 0x7efd08197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd0dfe2640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd0819bfb0 0x7efd0819c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd0dfe2640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0819c9a0 con 0x7efd08100280 2024-03-12T06:05:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd0dfe2640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0819cb10 con 0x7efd0819bfb0 2024-03-12T06:05:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd0dfe2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0819cc80 con 0x7efd08105350 2024-03-12T06:05:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd077fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd08100280 0x7efd081976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd077fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd08100280 0x7efd081976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:53084/0 (socket says 172.21.15.76:53084) 2024-03-12T06:05:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd077fe640 1 -- 172.21.15.76:0/3738249257 learned_addr learned my addr 172.21.15.76:0/3738249257 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:05:57.385 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd06ffd640 1 --2- 172.21.15.76:0/3738249257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd08105350 0x7efd08197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd07fff640 1 --2- 172.21.15.76:0/3738249257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd0819bfb0 0x7efd0819c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd077fe640 1 -- 172.21.15.76:0/3738249257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd08105350 msgr2=0x7efd08197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd077fe640 1 --2- 172.21.15.76:0/3738249257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd08105350 0x7efd08197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd077fe640 1 -- 172.21.15.76:0/3738249257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd0819bfb0 msgr2=0x7efd0819c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd077fe640 1 --2- 172.21.15.76:0/3738249257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd0819bfb0 0x7efd0819c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd077fe640 1 -- 172.21.15.76:0/3738249257 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd00009900 con 0x7efd08100280 2024-03-12T06:05:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.377+0000 7efd077fe640 1 --2- 172.21.15.76:0/3738249257 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd08100280 0x7efd081976f0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7efcf800c920 tx=0x7efcf800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.378+0000 7efd04ff9640 1 -- 172.21.15.76:0/3738249257 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8007c70 con 0x7efd08100280 2024-03-12T06:05:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.378+0000 7efd04ff9640 1 -- 172.21.15.76:0/3738249257 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efcf8007dd0 con 0x7efd08100280 2024-03-12T06:05:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.378+0000 7efd04ff9640 1 -- 172.21.15.76:0/3738249257 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf8018860 con 0x7efd08100280 2024-03-12T06:05:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.378+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3738249257 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd081a4260 con 0x7efd08100280 2024-03-12T06:05:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.378+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3738249257 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd081a4710 con 0x7efd08100280 2024-03-12T06:05:57.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.379+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3738249257 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd08101750 con 0x7efd08100280 2024-03-12T06:05:57.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.380+0000 7efd04ff9640 1 -- 172.21.15.76:0/3738249257 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7efcf800fac0 con 0x7efd08100280 2024-03-12T06:05:57.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.380+0000 7efd04ff9640 1 --2- 172.21.15.76:0/3738249257 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efcf40a1ba0 0x7efcf40a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:05:57.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.380+0000 7efd04ff9640 1 -- 172.21.15.76:0/3738249257 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8513+0+0 (secure 0 0 0) 0x7efcf80cdba0 con 0x7efd08100280 2024-03-12T06:05:57.388 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.380+0000 7efd06ffd640 1 --2- 172.21.15.76:0/3738249257 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efcf40a1ba0 0x7efcf40a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:05:57.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.381+0000 7efd06ffd640 1 --2- 172.21.15.76:0/3738249257 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efcf40a1ba0 0x7efcf40a4060 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efd080ff1a0 tx=0x7efcf000b390 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:05:57.389 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.384+0000 7efd04ff9640 1 -- 172.21.15.76:0/3738249257 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efcf80cde60 con 0x7efd08100280 2024-03-12T06:05:57.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.602+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3738249257 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7efd08068450 con 0x7efd08100280 2024-03-12T06:05:57.615 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.610+0000 7efd04ff9640 1 -- 172.21.15.76:0/3738249257 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) v1 ==== 81+0+734999 (secure 0 0 0) 0x7efcf80cde60 con 0x7efd08100280 2024-03-12T06:05:57.617 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:05:57.617 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:05:57.617 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 26, 2024-03-12T06:05:57.617 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.617 INFO:teuthology.orchestra.run.smithi076.stdout: "active_gid": 15465, 2024-03-12T06:05:57.617 INFO:teuthology.orchestra.run.smithi076.stdout: "active_name": "x", 2024-03-12T06:05:57.617 INFO:teuthology.orchestra.run.smithi076.stdout: "active_addrs": { 2024-03-12T06:05:57.617 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-03-12T06:05:57.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:57.618 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-03-12T06:05:57.618 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6800", 2024-03-12T06:05:57.618 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 513615067 2024-03-12T06:05:57.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:57.618 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2024-03-12T06:05:57.618 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6801", 2024-03-12T06:05:57.619 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 513615067 2024-03-12T06:05:57.619 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.619 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-12T06:05:57.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.619 INFO:teuthology.orchestra.run.smithi076.stdout: "active_addr": "172.21.15.76:6801/513615067", 2024-03-12T06:05:57.619 INFO:teuthology.orchestra.run.smithi076.stdout: "active_change": "2024-03-12T05:55:46.981065+0000", 2024-03-12T06:05:57.619 INFO:teuthology.orchestra.run.smithi076.stdout: "active_mgr_features": 4540701547738038271, 2024-03-12T06:05:57.619 INFO:teuthology.orchestra.run.smithi076.stdout: "available": true, 2024-03-12T06:05:57.620 INFO:teuthology.orchestra.run.smithi076.stdout: "standbys": [ 2024-03-12T06:05:57.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:57.620 INFO:teuthology.orchestra.run.smithi076.stdout: "gid": 24943, 2024-03-12T06:05:57.620 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "z", 2024-03-12T06:05:57.620 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_features": 4540701547738038271, 2024-03-12T06:05:57.620 INFO:teuthology.orchestra.run.smithi076.stdout: "available_modules": [ 2024-03-12T06:05:57.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:57.620 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "alerts", 2024-03-12T06:05:57.621 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:57.621 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:57.621 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:57.621 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:57.621 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:57.621 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.621 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.621 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.621 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:57.622 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.622 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.622 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.622 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How frequently to reexamine health status", 2024-03-12T06:05:57.622 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.622 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.622 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.622 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:57.623 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:57.623 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.623 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.623 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.623 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.623 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.623 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.623 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.624 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.624 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.624 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.624 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.624 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.624 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.624 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.624 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.624 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.625 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.625 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.625 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:57.625 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:57.625 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.625 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.625 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.625 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.626 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.626 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.626 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.626 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.626 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.626 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.626 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.626 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:57.627 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:57.627 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.627 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.627 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.627 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:57.627 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.627 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.627 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.627 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.628 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.628 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.628 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.628 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.628 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.628 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.628 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.628 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.628 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.628 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.629 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:57.629 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:57.629 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.629 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.629 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.629 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.630 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.630 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.630 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.630 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.630 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.630 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.630 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.631 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_destination": { 2024-03-12T06:05:57.631 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_destination", 2024-03-12T06:05:57.631 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.631 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.631 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.631 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.631 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.631 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.632 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.632 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Email address to send alerts to", 2024-03-12T06:05:57.632 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.632 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.632 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.632 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_from_name": { 2024-03-12T06:05:57.632 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_from_name", 2024-03-12T06:05:57.633 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.633 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.633 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.633 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "Ceph", 2024-03-12T06:05:57.633 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.633 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.633 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.633 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Email From: name", 2024-03-12T06:05:57.633 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.634 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.634 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.634 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_host": { 2024-03-12T06:05:57.634 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_host", 2024-03-12T06:05:57.634 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.634 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.635 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.635 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.635 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.635 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.635 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.635 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP server", 2024-03-12T06:05:57.635 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.635 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.635 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.636 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_password": { 2024-03-12T06:05:57.636 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_password", 2024-03-12T06:05:57.636 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.636 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.636 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.636 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.636 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.637 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.637 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.637 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Password to authenticate with", 2024-03-12T06:05:57.637 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.637 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.637 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.638 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_port": { 2024-03-12T06:05:57.638 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_port", 2024-03-12T06:05:57.638 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.638 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.638 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.638 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "465", 2024-03-12T06:05:57.638 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.638 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.638 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.639 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP port", 2024-03-12T06:05:57.639 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.639 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.639 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.639 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_sender": { 2024-03-12T06:05:57.639 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_sender", 2024-03-12T06:05:57.639 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.639 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.640 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.640 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.640 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.640 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.640 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.640 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP envelope sender", 2024-03-12T06:05:57.640 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.640 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.641 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.641 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_ssl": { 2024-03-12T06:05:57.641 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_ssl", 2024-03-12T06:05:57.641 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.641 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.641 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.641 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.642 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.642 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.642 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.642 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-12T06:05:57.642 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.642 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.642 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.643 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_user": { 2024-03-12T06:05:57.643 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_user", 2024-03-12T06:05:57.643 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.643 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.643 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.643 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.643 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.643 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.643 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.644 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "User to authenticate as", 2024-03-12T06:05:57.644 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.644 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.644 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.644 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.644 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:57.645 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "balancer", 2024-03-12T06:05:57.645 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:57.645 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:57.645 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:57.645 INFO:teuthology.orchestra.run.smithi076.stdout: "active": { 2024-03-12T06:05:57.645 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "active", 2024-03-12T06:05:57.645 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.645 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.646 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.646 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.646 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.646 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.646 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.646 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "automatically balance PGs across cluster", 2024-03-12T06:05:57.646 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.646 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.647 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.647 INFO:teuthology.orchestra.run.smithi076.stdout: "begin_time": { 2024-03-12T06:05:57.647 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "begin_time", 2024-03-12T06:05:57.647 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.647 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.647 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.647 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0000", 2024-03-12T06:05:57.648 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.648 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.648 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.648 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "beginning time of day to automatically balance", 2024-03-12T06:05:57.648 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-12T06:05:57.648 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.648 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.648 INFO:teuthology.orchestra.run.smithi076.stdout: "begin_weekday": { 2024-03-12T06:05:57.649 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "begin_weekday", 2024-03-12T06:05:57.649 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:57.649 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.649 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.649 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:57.649 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0", 2024-03-12T06:05:57.649 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "6", 2024-03-12T06:05:57.649 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.649 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-12T06:05:57.650 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-12T06:05:57.650 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.650 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.650 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_max_iterations": { 2024-03-12T06:05:57.650 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_max_iterations", 2024-03-12T06:05:57.650 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:57.651 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.651 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.651 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "25", 2024-03-12T06:05:57.651 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-12T06:05:57.651 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "250", 2024-03-12T06:05:57.651 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.651 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-12T06:05:57.651 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.652 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.652 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.652 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_metrics": { 2024-03-12T06:05:57.652 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_metrics", 2024-03-12T06:05:57.652 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.652 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.652 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.653 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "pgs,objects,bytes", 2024-03-12T06:05:57.653 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.653 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.653 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.653 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-12T06:05:57.653 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-12T06:05:57.653 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.653 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.654 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_step": { 2024-03-12T06:05:57.654 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_step", 2024-03-12T06:05:57.654 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:57.654 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.654 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.654 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0.5", 2024-03-12T06:05:57.654 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0.001", 2024-03-12T06:05:57.655 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "0.999", 2024-03-12T06:05:57.655 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.655 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "aggressiveness of optimization", 2024-03-12T06:05:57.655 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-12T06:05:57.655 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.655 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.655 INFO:teuthology.orchestra.run.smithi076.stdout: "end_time": { 2024-03-12T06:05:57.656 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "end_time", 2024-03-12T06:05:57.656 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.656 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.656 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.656 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "2359", 2024-03-12T06:05:57.656 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.656 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.656 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.657 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "ending time of day to automatically balance", 2024-03-12T06:05:57.657 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-12T06:05:57.657 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.657 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.657 INFO:teuthology.orchestra.run.smithi076.stdout: "end_weekday": { 2024-03-12T06:05:57.657 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "end_weekday", 2024-03-12T06:05:57.657 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:57.658 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.658 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.658 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:57.658 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0", 2024-03-12T06:05:57.658 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "6", 2024-03-12T06:05:57.658 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.658 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-12T06:05:57.658 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-12T06:05:57.659 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.659 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.659 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:57.659 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:57.659 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.659 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.660 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.660 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.660 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.660 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.660 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.660 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.660 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.660 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.660 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.661 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.661 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.661 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.661 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.661 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.661 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.661 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.661 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:57.662 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:57.662 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.662 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.662 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.662 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.662 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.662 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.663 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.663 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.663 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.663 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.663 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.663 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:57.663 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:57.664 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.664 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.664 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.664 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:57.664 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.664 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.664 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.665 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.665 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.665 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.665 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.665 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.665 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.665 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.665 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.666 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.666 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.666 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.666 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:57.666 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:57.666 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.666 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.667 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.667 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.667 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.667 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.667 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.667 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.667 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.667 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.667 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.668 INFO:teuthology.orchestra.run.smithi076.stdout: "min_score": { 2024-03-12T06:05:57.668 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "min_score", 2024-03-12T06:05:57.668 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:57.668 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.668 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.668 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:57.668 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.668 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.669 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.669 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-12T06:05:57.669 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.669 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.669 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.669 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": { 2024-03-12T06:05:57.669 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mode", 2024-03-12T06:05:57.669 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.670 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.670 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.670 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "upmap", 2024-03-12T06:05:57.670 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.670 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.670 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.670 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-compat", 2024-03-12T06:05:57.670 INFO:teuthology.orchestra.run.smithi076.stdout: "none", 2024-03-12T06:05:57.671 INFO:teuthology.orchestra.run.smithi076.stdout: "read", 2024-03-12T06:05:57.671 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap", 2024-03-12T06:05:57.671 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap-read" 2024-03-12T06:05:57.671 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.671 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Balancer mode", 2024-03-12T06:05:57.671 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.671 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.671 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.671 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_ids": { 2024-03-12T06:05:57.672 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pool_ids", 2024-03-12T06:05:57.672 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.672 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.672 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.672 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.672 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.672 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.672 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.672 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-12T06:05:57.672 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.673 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.673 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.673 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:57.673 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:57.673 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.673 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.673 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.673 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:57.674 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.674 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.674 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.674 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-12T06:05:57.674 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.674 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.674 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.674 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap_max_deviation": { 2024-03-12T06:05:57.675 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "upmap_max_deviation", 2024-03-12T06:05:57.675 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.675 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.675 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.675 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-12T06:05:57.675 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-12T06:05:57.675 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.675 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.675 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-12T06:05:57.676 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-12T06:05:57.676 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.676 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.676 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap_max_optimizations": { 2024-03-12T06:05:57.676 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "upmap_max_optimizations", 2024-03-12T06:05:57.676 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:57.676 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.676 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.676 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:57.677 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.677 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.677 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.677 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-12T06:05:57.677 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.677 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.677 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.677 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.677 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:57.678 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cephadm", 2024-03-12T06:05:57.678 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:57.678 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:57.678 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:57.678 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_down_multiplier": { 2024-03-12T06:05:57.678 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_down_multiplier", 2024-03-12T06:05:57.678 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:57.679 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.679 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.679 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3.0", 2024-03-12T06:05:57.679 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.679 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.679 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.679 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-12T06:05:57.679 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.679 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.680 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.680 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_refresh_rate": { 2024-03-12T06:05:57.680 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_refresh_rate", 2024-03-12T06:05:57.680 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.680 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.680 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.680 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "20", 2024-03-12T06:05:57.681 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.681 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.681 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.681 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-12T06:05:57.681 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.681 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.681 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.681 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_starting_port": { 2024-03-12T06:05:57.682 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_starting_port", 2024-03-12T06:05:57.682 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.682 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.682 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.682 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "4721", 2024-03-12T06:05:57.682 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.682 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.683 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.683 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-12T06:05:57.683 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.683 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.683 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.683 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_ptrace": { 2024-03-12T06:05:57.683 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_ptrace", 2024-03-12T06:05:57.684 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.684 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.684 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.684 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.684 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.684 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.684 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.684 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-12T06:05:57.685 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-12T06:05:57.685 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.685 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.685 INFO:teuthology.orchestra.run.smithi076.stdout: "autotune_interval": { 2024-03-12T06:05:57.685 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "autotune_interval", 2024-03-12T06:05:57.685 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.685 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.685 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.686 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:57.686 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.686 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.686 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.686 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-12T06:05:57.686 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.686 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.686 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.687 INFO:teuthology.orchestra.run.smithi076.stdout: "autotune_memory_target_ratio": { 2024-03-12T06:05:57.687 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "autotune_memory_target_ratio", 2024-03-12T06:05:57.687 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:57.687 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.687 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.687 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0.7", 2024-03-12T06:05:57.687 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.687 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.688 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.688 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-12T06:05:57.688 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.688 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.688 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.688 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm_log_destination": { 2024-03-12T06:05:57.688 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cephadm_log_destination", 2024-03-12T06:05:57.688 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.689 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.689 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.689 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.689 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.689 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.689 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.689 INFO:teuthology.orchestra.run.smithi076.stdout: "file", 2024-03-12T06:05:57.689 INFO:teuthology.orchestra.run.smithi076.stdout: "file,syslog", 2024-03-12T06:05:57.689 INFO:teuthology.orchestra.run.smithi076.stdout: "syslog" 2024-03-12T06:05:57.690 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.690 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-12T06:05:57.690 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.690 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.690 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.690 INFO:teuthology.orchestra.run.smithi076.stdout: "cgroups_split": { 2024-03-12T06:05:57.691 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cgroups_split", 2024-03-12T06:05:57.691 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.691 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.691 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.691 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.691 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.691 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.692 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.692 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-12T06:05:57.692 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.692 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.692 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.692 INFO:teuthology.orchestra.run.smithi076.stdout: "config_checks_enabled": { 2024-03-12T06:05:57.692 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_checks_enabled", 2024-03-12T06:05:57.693 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.693 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.693 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.693 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.693 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.693 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.693 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.693 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-12T06:05:57.693 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.694 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.694 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.694 INFO:teuthology.orchestra.run.smithi076.stdout: "config_dashboard": { 2024-03-12T06:05:57.694 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_dashboard", 2024-03-12T06:05:57.694 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.694 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.694 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.694 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.695 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.695 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.695 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.695 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-12T06:05:57.695 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.695 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.695 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.695 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_alertmanager": { 2024-03-12T06:05:57.696 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_alertmanager", 2024-03-12T06:05:57.696 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.696 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.696 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.696 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-12T06:05:57.696 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.696 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.696 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.696 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-12T06:05:57.696 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.697 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.697 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.697 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_base": { 2024-03-12T06:05:57.697 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_base", 2024-03-12T06:05:57.697 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.697 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.697 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.698 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-12T06:05:57.698 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.698 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.698 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.698 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Container image name, without the tag", 2024-03-12T06:05:57.698 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.698 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.698 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.699 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_elasticsearch": { 2024-03-12T06:05:57.699 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_elasticsearch", 2024-03-12T06:05:57.699 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.699 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.699 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.699 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-12T06:05:57.699 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.699 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.700 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.700 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "elasticsearch container image", 2024-03-12T06:05:57.700 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.700 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.700 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.700 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_grafana": { 2024-03-12T06:05:57.700 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_grafana", 2024-03-12T06:05:57.701 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.701 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.701 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.701 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-12T06:05:57.701 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.701 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.701 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.701 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-12T06:05:57.702 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.702 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.702 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.702 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_haproxy": { 2024-03-12T06:05:57.702 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_haproxy", 2024-03-12T06:05:57.702 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.703 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.703 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.703 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-12T06:05:57.703 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.703 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.703 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.703 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "HAproxy container image", 2024-03-12T06:05:57.703 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.704 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.704 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.704 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_agent": { 2024-03-12T06:05:57.704 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_agent", 2024-03-12T06:05:57.704 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.704 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.705 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.705 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-12T06:05:57.705 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.705 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.705 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.705 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger agent container image", 2024-03-12T06:05:57.705 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.705 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.705 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.706 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_collector": { 2024-03-12T06:05:57.706 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_collector", 2024-03-12T06:05:57.706 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.706 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.706 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.706 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-12T06:05:57.706 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.707 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.707 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.707 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger collector container image", 2024-03-12T06:05:57.707 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.707 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.707 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.707 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_query": { 2024-03-12T06:05:57.708 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_query", 2024-03-12T06:05:57.708 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.708 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.708 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.708 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-12T06:05:57.708 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.708 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.709 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.709 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger query container image", 2024-03-12T06:05:57.709 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.709 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.709 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.709 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_keepalived": { 2024-03-12T06:05:57.709 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_keepalived", 2024-03-12T06:05:57.709 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.710 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.710 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.710 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-12T06:05:57.710 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.710 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.710 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.710 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Keepalived container image", 2024-03-12T06:05:57.710 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.711 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.711 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.711 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_loki": { 2024-03-12T06:05:57.711 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_loki", 2024-03-12T06:05:57.711 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.711 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.711 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.711 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-12T06:05:57.712 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.712 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.712 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.712 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Loki container image", 2024-03-12T06:05:57.712 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.712 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.712 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.712 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_node_exporter": { 2024-03-12T06:05:57.713 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_node_exporter", 2024-03-12T06:05:57.713 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.713 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.713 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.713 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-12T06:05:57.713 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.713 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.713 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.713 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-12T06:05:57.714 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.714 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.714 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.714 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_nvmeof": { 2024-03-12T06:05:57.714 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_nvmeof", 2024-03-12T06:05:57.714 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.714 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.714 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.715 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-12T06:05:57.715 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.715 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.715 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.715 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Nvme-of container image", 2024-03-12T06:05:57.715 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.715 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.715 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.715 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_prometheus": { 2024-03-12T06:05:57.716 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_prometheus", 2024-03-12T06:05:57.716 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.716 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.716 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.716 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-12T06:05:57.716 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.716 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.716 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.716 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-12T06:05:57.717 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.717 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.717 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.717 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_promtail": { 2024-03-12T06:05:57.717 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_promtail", 2024-03-12T06:05:57.717 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.717 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.717 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.718 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-12T06:05:57.718 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.718 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.718 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.718 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Promtail container image", 2024-03-12T06:05:57.718 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.718 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.718 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.719 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_snmp_gateway": { 2024-03-12T06:05:57.719 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_snmp_gateway", 2024-03-12T06:05:57.719 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.719 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.719 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.719 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-12T06:05:57.719 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.719 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.720 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.720 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SNMP Gateway container image", 2024-03-12T06:05:57.720 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.720 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.720 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.720 INFO:teuthology.orchestra.run.smithi076.stdout: "container_init": { 2024-03-12T06:05:57.720 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_init", 2024-03-12T06:05:57.720 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.721 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.721 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.721 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.721 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.721 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.721 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.721 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Run podman/docker with `--init`", 2024-03-12T06:05:57.721 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.722 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.722 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.722 INFO:teuthology.orchestra.run.smithi076.stdout: "daemon_cache_timeout": { 2024-03-12T06:05:57.722 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "daemon_cache_timeout", 2024-03-12T06:05:57.722 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.722 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.722 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.722 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:57.723 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.723 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.723 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.723 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-12T06:05:57.723 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.723 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.723 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.724 INFO:teuthology.orchestra.run.smithi076.stdout: "default_cephadm_command_timeout": { 2024-03-12T06:05:57.724 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "default_cephadm_command_timeout", 2024-03-12T06:05:57.724 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.724 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.724 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.724 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "900", 2024-03-12T06:05:57.724 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.724 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.725 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.725 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-12T06:05:57.725 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.725 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.725 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.725 INFO:teuthology.orchestra.run.smithi076.stdout: "default_registry": { 2024-03-12T06:05:57.726 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "default_registry", 2024-03-12T06:05:57.726 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.726 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.726 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.726 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io", 2024-03-12T06:05:57.726 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.726 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.726 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.727 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-12T06:05:57.727 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.727 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.727 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.727 INFO:teuthology.orchestra.run.smithi076.stdout: "device_cache_timeout": { 2024-03-12T06:05:57.727 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_cache_timeout", 2024-03-12T06:05:57.727 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.728 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.728 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.728 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1800", 2024-03-12T06:05:57.728 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.728 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.728 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.728 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache device inventory", 2024-03-12T06:05:57.728 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.728 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.729 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.729 INFO:teuthology.orchestra.run.smithi076.stdout: "device_enhanced_scan": { 2024-03-12T06:05:57.729 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_enhanced_scan", 2024-03-12T06:05:57.729 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.729 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.729 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.729 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.729 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.730 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.730 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.730 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-12T06:05:57.730 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.730 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.730 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.730 INFO:teuthology.orchestra.run.smithi076.stdout: "facts_cache_timeout": { 2024-03-12T06:05:57.731 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "facts_cache_timeout", 2024-03-12T06:05:57.731 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.731 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.731 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.731 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:57.731 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.731 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.731 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.732 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache host facts data", 2024-03-12T06:05:57.732 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.732 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.732 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.732 INFO:teuthology.orchestra.run.smithi076.stdout: "host_check_interval": { 2024-03-12T06:05:57.732 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "host_check_interval", 2024-03-12T06:05:57.732 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.733 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.733 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.733 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:57.733 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.733 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.733 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.733 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to perform a host check", 2024-03-12T06:05:57.734 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.734 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.734 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.734 INFO:teuthology.orchestra.run.smithi076.stdout: "hw_monitoring": { 2024-03-12T06:05:57.734 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "hw_monitoring", 2024-03-12T06:05:57.734 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.734 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.735 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.735 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.735 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.735 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.735 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.735 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-12T06:05:57.735 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.736 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.736 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.736 INFO:teuthology.orchestra.run.smithi076.stdout: "inventory_list_all": { 2024-03-12T06:05:57.736 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "inventory_list_all", 2024-03-12T06:05:57.736 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.736 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.736 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.737 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.737 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.737 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.737 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.737 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-12T06:05:57.737 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.737 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.738 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.738 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:57.738 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:57.738 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.738 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.738 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.738 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.738 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.739 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.739 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.739 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.739 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.739 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.739 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.739 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.739 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.740 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.740 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.740 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.740 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.740 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.740 INFO:teuthology.orchestra.run.smithi076.stdout: "log_refresh_metadata": { 2024-03-12T06:05:57.741 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_refresh_metadata", 2024-03-12T06:05:57.741 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.741 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.741 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.741 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.741 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.741 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.741 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.742 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-12T06:05:57.742 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.742 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.742 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.742 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:57.742 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:57.742 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.743 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.743 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.743 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.743 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.743 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.743 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.743 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-12T06:05:57.743 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.745 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.745 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.745 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:57.745 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:57.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.746 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.746 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.753 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:57.753 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.753 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.753 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.754 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.754 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.754 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.754 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.754 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.754 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.754 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.754 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.755 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.755 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.755 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.755 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:57.755 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:57.755 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.756 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.756 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.756 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.756 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.756 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.756 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.756 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.756 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.757 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.757 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.757 INFO:teuthology.orchestra.run.smithi076.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-12T06:05:57.757 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-12T06:05:57.757 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.757 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.757 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.758 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.758 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.758 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.758 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.758 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-12T06:05:57.758 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.758 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.758 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.759 INFO:teuthology.orchestra.run.smithi076.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-12T06:05:57.759 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-12T06:05:57.759 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.759 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.759 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.759 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "*", 2024-03-12T06:05:57.760 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.760 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.760 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.760 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-12T06:05:57.760 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.760 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.760 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.761 INFO:teuthology.orchestra.run.smithi076.stdout: "max_count_per_host": { 2024-03-12T06:05:57.761 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_count_per_host", 2024-03-12T06:05:57.761 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.761 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.761 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.761 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:57.761 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.761 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.762 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.762 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "max number of daemons per service per host", 2024-03-12T06:05:57.762 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.762 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.762 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.762 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd_draining_count": { 2024-03-12T06:05:57.763 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_osd_draining_count", 2024-03-12T06:05:57.763 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.763 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.763 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.763 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:57.763 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.763 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.763 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.763 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-12T06:05:57.764 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.764 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.764 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.764 INFO:teuthology.orchestra.run.smithi076.stdout: "migration_current": { 2024-03-12T06:05:57.764 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "migration_current", 2024-03-12T06:05:57.764 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.765 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.765 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.765 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.765 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.765 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.765 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.765 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "internal - do not modify", 2024-03-12T06:05:57.765 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.765 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.766 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.766 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": { 2024-03-12T06:05:57.766 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mode", 2024-03-12T06:05:57.766 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.766 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.766 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.766 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "root", 2024-03-12T06:05:57.766 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.767 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.767 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.767 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm-package", 2024-03-12T06:05:57.767 INFO:teuthology.orchestra.run.smithi076.stdout: "root" 2024-03-12T06:05:57.767 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.767 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "mode for remote execution of cephadm", 2024-03-12T06:05:57.767 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.767 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.767 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.768 INFO:teuthology.orchestra.run.smithi076.stdout: "oob_default_addr": { 2024-03-12T06:05:57.768 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "oob_default_addr", 2024-03-12T06:05:57.768 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.768 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.768 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.768 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "169.254.1.1", 2024-03-12T06:05:57.768 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.769 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.769 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.769 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-12T06:05:57.769 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.769 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.769 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.769 INFO:teuthology.orchestra.run.smithi076.stdout: "prometheus_alerts_path": { 2024-03-12T06:05:57.769 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prometheus_alerts_path", 2024-03-12T06:05:57.770 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.770 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.770 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.770 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-12T06:05:57.770 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.770 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.770 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.770 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-12T06:05:57.771 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.771 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.771 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.771 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_insecure": { 2024-03-12T06:05:57.771 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_insecure", 2024-03-12T06:05:57.771 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.771 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.772 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.772 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.772 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.772 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.772 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.772 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-12T06:05:57.772 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.772 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.773 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.773 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_password": { 2024-03-12T06:05:57.773 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_password", 2024-03-12T06:05:57.773 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.773 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.773 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.773 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.773 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.774 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.774 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.774 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-12T06:05:57.774 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.774 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.774 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.774 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_url": { 2024-03-12T06:05:57.775 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_url", 2024-03-12T06:05:57.775 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.775 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.775 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.775 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.775 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.775 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.775 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.775 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-12T06:05:57.776 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.776 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.776 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.776 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_username": { 2024-03-12T06:05:57.776 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_username", 2024-03-12T06:05:57.776 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.777 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.777 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.777 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.777 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.777 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.777 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.777 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-12T06:05:57.777 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.778 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.778 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.778 INFO:teuthology.orchestra.run.smithi076.stdout: "secure_monitoring_stack": { 2024-03-12T06:05:57.778 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "secure_monitoring_stack", 2024-03-12T06:05:57.778 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.778 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.778 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.779 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.779 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.779 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.779 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.779 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-12T06:05:57.779 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.779 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.779 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.780 INFO:teuthology.orchestra.run.smithi076.stdout: "service_discovery_port": { 2024-03-12T06:05:57.780 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "service_discovery_port", 2024-03-12T06:05:57.780 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.780 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.780 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.780 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8765", 2024-03-12T06:05:57.780 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.781 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.781 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.781 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "cephadm service discovery port", 2024-03-12T06:05:57.781 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.781 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.781 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.781 INFO:teuthology.orchestra.run.smithi076.stdout: "ssh_config_file": { 2024-03-12T06:05:57.781 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssh_config_file", 2024-03-12T06:05:57.782 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.782 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.782 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.782 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.782 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.782 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.782 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.783 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-12T06:05:57.783 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.783 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.783 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.783 INFO:teuthology.orchestra.run.smithi076.stdout: "use_agent": { 2024-03-12T06:05:57.783 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "use_agent", 2024-03-12T06:05:57.783 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.783 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.784 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.784 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.784 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.784 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.784 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.784 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-12T06:05:57.784 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.784 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.784 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.785 INFO:teuthology.orchestra.run.smithi076.stdout: "use_repo_digest": { 2024-03-12T06:05:57.785 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "use_repo_digest", 2024-03-12T06:05:57.785 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.785 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.785 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.785 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.785 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.785 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.786 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.786 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-12T06:05:57.786 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.786 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.786 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.786 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_failed_host_check": { 2024-03-12T06:05:57.786 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_failed_host_check", 2024-03-12T06:05:57.787 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.787 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.787 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.787 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.787 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.787 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.787 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.787 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if the host check fails", 2024-03-12T06:05:57.788 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.788 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.788 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.788 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_stray_daemons": { 2024-03-12T06:05:57.788 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_stray_daemons", 2024-03-12T06:05:57.788 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.788 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.789 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.789 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.789 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.789 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.789 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.789 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-12T06:05:57.789 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.789 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.789 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.790 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_stray_hosts": { 2024-03-12T06:05:57.790 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_stray_hosts", 2024-03-12T06:05:57.790 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.790 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.790 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.790 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.790 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.791 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.791 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.791 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-12T06:05:57.791 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.791 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.791 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.791 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.791 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:57.792 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crash", 2024-03-12T06:05:57.792 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:57.792 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:57.792 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:57.792 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:57.792 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:57.792 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.792 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.793 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.793 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.793 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.793 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.793 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.793 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.793 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.794 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.794 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.794 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.794 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.794 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.794 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.794 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.794 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.795 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.795 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:57.795 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:57.795 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.795 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.795 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.795 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.796 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.796 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.796 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.796 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.796 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.796 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.796 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.797 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:57.797 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:57.797 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.797 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.797 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.797 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:57.798 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.798 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.798 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.798 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.798 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.798 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.798 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.798 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.799 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.799 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.799 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.799 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.799 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.799 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.799 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:57.800 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:57.800 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.800 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.800 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.800 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.800 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.800 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.800 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.801 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.801 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.801 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.801 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.801 INFO:teuthology.orchestra.run.smithi076.stdout: "retain_interval": { 2024-03-12T06:05:57.801 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "retain_interval", 2024-03-12T06:05:57.801 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.802 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.802 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.802 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "31536000", 2024-03-12T06:05:57.802 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.802 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.802 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.802 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-12T06:05:57.802 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.803 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.803 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.803 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_recent_interval": { 2024-03-12T06:05:57.803 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_recent_interval", 2024-03-12T06:05:57.803 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.803 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.803 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.804 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1209600", 2024-03-12T06:05:57.804 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.804 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.804 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.804 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-12T06:05:57.804 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.804 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.804 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.805 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.805 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:57.805 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "dashboard", 2024-03-12T06:05:57.805 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:57.805 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:57.805 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:57.806 INFO:teuthology.orchestra.run.smithi076.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-12T06:05:57.806 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-12T06:05:57.806 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.806 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.806 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.806 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:57.806 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.806 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.807 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.807 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.807 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.807 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.807 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.807 INFO:teuthology.orchestra.run.smithi076.stdout: "ALERTMANAGER_API_HOST": { 2024-03-12T06:05:57.807 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-12T06:05:57.808 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.808 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.808 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.808 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.808 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.808 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.808 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.808 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.808 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.809 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.809 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.809 INFO:teuthology.orchestra.run.smithi076.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-12T06:05:57.809 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-12T06:05:57.809 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.809 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.809 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.809 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.810 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.810 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.810 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.810 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.810 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.810 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.810 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.810 INFO:teuthology.orchestra.run.smithi076.stdout: "AUDIT_API_ENABLED": { 2024-03-12T06:05:57.811 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "AUDIT_API_ENABLED", 2024-03-12T06:05:57.811 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.811 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.811 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.811 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.811 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.811 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.811 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.811 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.812 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.812 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.812 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.812 INFO:teuthology.orchestra.run.smithi076.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-12T06:05:57.812 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-12T06:05:57.812 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.812 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.812 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.812 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.813 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.813 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.813 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.813 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.813 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.813 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.813 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.814 INFO:teuthology.orchestra.run.smithi076.stdout: "ENABLE_BROWSABLE_API": { 2024-03-12T06:05:57.814 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-12T06:05:57.814 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.814 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.814 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.814 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.814 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.814 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.814 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.815 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.815 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.815 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.815 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.815 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-12T06:05:57.815 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-12T06:05:57.815 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.816 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.816 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.816 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.816 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.816 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.816 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.816 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.816 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.816 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.817 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.817 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-12T06:05:57.817 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-12T06:05:57.817 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.817 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.817 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.817 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.817 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.818 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.818 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.818 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.818 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.818 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.818 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.818 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-12T06:05:57.819 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-12T06:05:57.819 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.819 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.819 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.819 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.819 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.819 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.819 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.820 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.820 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.820 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.820 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.820 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-12T06:05:57.820 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-12T06:05:57.820 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.821 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.821 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.821 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.821 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.821 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.821 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.821 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.821 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.822 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.822 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.822 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-12T06:05:57.822 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-12T06:05:57.822 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.822 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.822 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.823 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.823 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.823 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.823 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.823 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.823 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.823 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.824 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.824 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-12T06:05:57.824 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-12T06:05:57.824 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.824 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.824 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.824 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.824 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.825 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.825 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.825 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.825 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.825 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.825 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.825 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-12T06:05:57.825 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-12T06:05:57.826 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.826 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.826 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.826 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.826 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.826 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.826 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.827 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.827 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.827 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.827 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.827 INFO:teuthology.orchestra.run.smithi076.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-12T06:05:57.827 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-12T06:05:57.827 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.827 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.828 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.828 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.828 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.828 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.828 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.828 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.828 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.828 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.829 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.829 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_PASSWORD": { 2024-03-12T06:05:57.829 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-12T06:05:57.829 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.829 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.829 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.830 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-12T06:05:57.830 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.830 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.830 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.830 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.830 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.830 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.831 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.831 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-12T06:05:57.831 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-12T06:05:57.831 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.831 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.831 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.832 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.832 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.832 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.832 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.832 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.832 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.832 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.833 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.833 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_URL": { 2024-03-12T06:05:57.833 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_URL", 2024-03-12T06:05:57.833 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.833 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.833 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.833 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.834 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.834 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.834 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.834 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.834 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.834 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.834 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.835 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_USERNAME": { 2024-03-12T06:05:57.835 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-12T06:05:57.835 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.835 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.835 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.835 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-12T06:05:57.836 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.836 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.836 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.836 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.836 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.837 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.837 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.837 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-12T06:05:57.837 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-12T06:05:57.837 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.837 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.837 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.838 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.838 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.838 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.838 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.838 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.838 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.838 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.838 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.839 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-12T06:05:57.839 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-12T06:05:57.839 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.839 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.839 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.839 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.839 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.840 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.840 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.840 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.840 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.840 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.840 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.840 INFO:teuthology.orchestra.run.smithi076.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-12T06:05:57.841 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-12T06:05:57.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.841 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.841 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.841 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.841 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.841 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.841 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.842 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.842 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.842 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.842 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.842 INFO:teuthology.orchestra.run.smithi076.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-12T06:05:57.842 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-12T06:05:57.843 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.843 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.843 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.843 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.843 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.843 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.843 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.843 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.844 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.844 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.844 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.844 INFO:teuthology.orchestra.run.smithi076.stdout: "PROMETHEUS_API_HOST": { 2024-03-12T06:05:57.844 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-12T06:05:57.844 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.844 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.845 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.845 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.845 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.845 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.845 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.845 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.845 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.845 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.846 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.846 INFO:teuthology.orchestra.run.smithi076.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-12T06:05:57.846 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-12T06:05:57.846 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.846 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.846 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.847 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.847 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.847 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.847 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.847 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.847 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.847 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.847 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.848 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-12T06:05:57.848 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-12T06:05:57.848 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.848 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.848 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.848 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.848 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.849 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.849 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.849 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.849 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.849 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.849 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.850 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-12T06:05:57.850 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-12T06:05:57.850 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.850 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.850 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.850 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.850 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.850 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.851 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.851 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.851 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.851 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.851 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.851 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-12T06:05:57.851 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-12T06:05:57.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.852 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.852 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.852 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.852 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.852 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.852 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.852 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.852 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.853 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.853 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.853 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-12T06:05:57.853 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-12T06:05:57.853 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.853 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.853 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.854 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.854 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.854 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.854 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.854 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.854 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.854 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.854 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.855 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-12T06:05:57.855 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-12T06:05:57.855 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.855 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.855 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.855 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.855 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.855 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.856 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.856 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.856 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.856 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.856 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.856 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-12T06:05:57.857 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-12T06:05:57.857 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.857 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.857 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.857 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.857 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.857 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.857 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.858 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.858 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.858 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.858 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.858 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-12T06:05:57.858 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-12T06:05:57.858 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.859 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.859 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.859 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.859 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.859 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.859 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.859 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.859 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.860 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.860 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.860 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_ENABLED": { 2024-03-12T06:05:57.860 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-12T06:05:57.860 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.860 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.860 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.860 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.861 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.861 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.861 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.861 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.861 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.861 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.861 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.861 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-12T06:05:57.861 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-12T06:05:57.862 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.862 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.862 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.862 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-12T06:05:57.862 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.862 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.862 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.862 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.863 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.863 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.863 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.863 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-12T06:05:57.863 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-12T06:05:57.863 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.863 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.864 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.864 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:57.864 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.864 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.864 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.864 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.864 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.864 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.864 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.865 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-12T06:05:57.865 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-12T06:05:57.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.865 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.865 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.865 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8", 2024-03-12T06:05:57.866 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.866 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.866 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.866 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.866 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.866 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.866 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.866 INFO:teuthology.orchestra.run.smithi076.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-12T06:05:57.867 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-12T06:05:57.867 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.867 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.867 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.867 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "45", 2024-03-12T06:05:57.867 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.867 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.867 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.867 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.868 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.868 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.868 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.868 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_ACCESS_KEY": { 2024-03-12T06:05:57.868 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-12T06:05:57.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.868 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.868 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.869 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.869 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.869 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.869 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.869 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.869 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.869 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.869 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.870 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-12T06:05:57.870 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-12T06:05:57.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.870 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.870 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.870 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-12T06:05:57.870 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.870 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.871 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.871 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.871 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.871 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.871 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.871 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_SECRET_KEY": { 2024-03-12T06:05:57.871 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-12T06:05:57.872 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.872 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.872 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.872 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.872 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.872 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.872 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.872 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.873 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.873 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.873 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.873 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_SSL_VERIFY": { 2024-03-12T06:05:57.873 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-12T06:05:57.873 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.873 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.874 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.874 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.874 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.874 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.874 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.874 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.874 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.874 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.875 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.875 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-12T06:05:57.875 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-12T06:05:57.875 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.875 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.875 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.875 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:57.876 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.876 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.876 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.876 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.876 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.876 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.876 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.877 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-12T06:05:57.877 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-12T06:05:57.877 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.877 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.877 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.877 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:57.877 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.878 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.878 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.878 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.878 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.878 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.878 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.878 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-12T06:05:57.879 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-12T06:05:57.879 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.879 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.879 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.879 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-12T06:05:57.879 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.879 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.879 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.879 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.880 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.880 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.880 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.880 INFO:teuthology.orchestra.run.smithi076.stdout: "cross_origin_url": { 2024-03-12T06:05:57.880 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cross_origin_url", 2024-03-12T06:05:57.880 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.881 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.881 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.881 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.881 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.881 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.881 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.881 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.881 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.881 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.882 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.882 INFO:teuthology.orchestra.run.smithi076.stdout: "crt_file": { 2024-03-12T06:05:57.882 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crt_file", 2024-03-12T06:05:57.882 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.882 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.882 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.882 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.883 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.883 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.883 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.883 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.883 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.883 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.883 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.884 INFO:teuthology.orchestra.run.smithi076.stdout: "debug": { 2024-03-12T06:05:57.884 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "debug", 2024-03-12T06:05:57.884 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.884 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.884 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.884 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.884 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.884 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.885 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.885 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable/disable debug options", 2024-03-12T06:05:57.885 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.885 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.885 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.885 INFO:teuthology.orchestra.run.smithi076.stdout: "jwt_token_ttl": { 2024-03-12T06:05:57.885 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "jwt_token_ttl", 2024-03-12T06:05:57.885 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.886 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.886 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.886 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "28800", 2024-03-12T06:05:57.886 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.886 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.886 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.886 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.886 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.887 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.887 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.887 INFO:teuthology.orchestra.run.smithi076.stdout: "key_file": { 2024-03-12T06:05:57.887 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "key_file", 2024-03-12T06:05:57.887 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.887 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.887 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.887 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.888 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.888 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.888 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.888 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.888 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.888 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.888 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.889 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:57.889 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:57.889 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.889 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.889 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.889 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.889 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.889 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.890 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.890 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.890 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.890 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.890 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.890 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.890 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.890 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.891 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.891 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.891 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.891 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.891 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:57.891 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:57.891 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.892 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.892 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.892 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.892 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.892 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.892 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.892 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.892 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.893 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.893 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.893 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:57.893 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:57.893 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.893 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.893 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.894 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:57.894 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.894 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.894 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.894 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.894 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.894 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.894 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.895 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.895 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.895 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.895 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.895 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.895 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.895 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.896 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:57.896 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:57.896 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.896 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.896 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.896 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.896 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.896 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.896 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.897 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.897 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.897 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.897 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.897 INFO:teuthology.orchestra.run.smithi076.stdout: "motd": { 2024-03-12T06:05:57.897 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "motd", 2024-03-12T06:05:57.897 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.897 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.897 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.898 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.898 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.898 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.898 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.898 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "The message of the day", 2024-03-12T06:05:57.898 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.898 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.899 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.899 INFO:teuthology.orchestra.run.smithi076.stdout: "redirect_resolve_ip_addr": { 2024-03-12T06:05:57.899 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "redirect_resolve_ip_addr", 2024-03-12T06:05:57.899 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.899 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.899 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.900 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.900 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.900 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.900 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.900 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.900 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.900 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.900 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.901 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-12T06:05:57.901 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-12T06:05:57.901 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.901 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.901 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.901 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "::", 2024-03-12T06:05:57.901 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.902 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.902 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.902 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.902 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.902 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.902 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.902 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-12T06:05:57.903 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-12T06:05:57.903 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.903 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.903 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.903 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8080", 2024-03-12T06:05:57.903 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.903 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.903 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.904 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.904 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.904 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.904 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.904 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl": { 2024-03-12T06:05:57.904 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl", 2024-03-12T06:05:57.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.904 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.905 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.905 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.905 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.905 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.905 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.905 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.905 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.905 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.906 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.906 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl_server_port": { 2024-03-12T06:05:57.906 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl_server_port", 2024-03-12T06:05:57.906 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.906 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.907 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.907 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8443", 2024-03-12T06:05:57.907 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.907 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.907 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.907 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.907 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.907 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.908 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.908 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_behaviour": { 2024-03-12T06:05:57.908 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_behaviour", 2024-03-12T06:05:57.908 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.908 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.908 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.908 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "redirect", 2024-03-12T06:05:57.909 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.909 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.909 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.909 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.909 INFO:teuthology.orchestra.run.smithi076.stdout: "redirect" 2024-03-12T06:05:57.909 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.909 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.909 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.909 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.910 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.910 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_error_status_code": { 2024-03-12T06:05:57.910 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_error_status_code", 2024-03-12T06:05:57.910 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.910 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.910 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.910 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "500", 2024-03-12T06:05:57.910 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "400", 2024-03-12T06:05:57.911 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "599", 2024-03-12T06:05:57.911 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.911 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.911 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.911 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.911 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.912 INFO:teuthology.orchestra.run.smithi076.stdout: "url_prefix": { 2024-03-12T06:05:57.912 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "url_prefix", 2024-03-12T06:05:57.912 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.912 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.912 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.912 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.912 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.912 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.913 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.913 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.913 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.913 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.913 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.913 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.913 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:57.914 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "devicehealth", 2024-03-12T06:05:57.914 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:57.914 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:57.914 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:57.914 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_monitoring": { 2024-03-12T06:05:57.914 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enable_monitoring", 2024-03-12T06:05:57.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.915 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.915 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.915 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.915 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.915 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.915 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.916 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "monitor device health metrics", 2024-03-12T06:05:57.916 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.916 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.916 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.916 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:57.916 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:57.916 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.917 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.917 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.917 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.917 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.917 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.917 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.917 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.917 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.917 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.918 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.918 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.918 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.918 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.918 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.918 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.918 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.919 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.919 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:57.919 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:57.919 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.919 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.919 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.919 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.920 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.920 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.920 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.920 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.920 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.920 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.920 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.921 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:57.921 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:57.921 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.921 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.921 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.921 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:57.921 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.921 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.922 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.922 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.922 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.922 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.922 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.922 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.922 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.923 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.923 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.923 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.923 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.923 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.923 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:57.923 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:57.924 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.924 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.924 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.924 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.924 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.924 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.924 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.925 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.925 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.925 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.925 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.925 INFO:teuthology.orchestra.run.smithi076.stdout: "mark_out_threshold": { 2024-03-12T06:05:57.925 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mark_out_threshold", 2024-03-12T06:05:57.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.926 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.926 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.926 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "2419200", 2024-03-12T06:05:57.926 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.926 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.926 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.926 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-12T06:05:57.926 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.927 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.927 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.927 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": { 2024-03-12T06:05:57.927 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pool_name", 2024-03-12T06:05:57.927 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.927 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.927 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.928 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "device_health_metrics", 2024-03-12T06:05:57.928 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.928 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.928 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.928 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-12T06:05:57.928 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.928 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.928 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.929 INFO:teuthology.orchestra.run.smithi076.stdout: "retention_period": { 2024-03-12T06:05:57.929 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "retention_period", 2024-03-12T06:05:57.929 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.929 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.929 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.929 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15552000", 2024-03-12T06:05:57.929 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.929 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.929 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.929 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long to retain device health metrics", 2024-03-12T06:05:57.930 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.930 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.930 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.930 INFO:teuthology.orchestra.run.smithi076.stdout: "scrape_frequency": { 2024-03-12T06:05:57.930 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "scrape_frequency", 2024-03-12T06:05:57.930 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.931 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.931 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.931 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "86400", 2024-03-12T06:05:57.931 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.931 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.931 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.931 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-12T06:05:57.931 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.931 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.932 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.932 INFO:teuthology.orchestra.run.smithi076.stdout: "self_heal": { 2024-03-12T06:05:57.932 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "self_heal", 2024-03-12T06:05:57.932 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.932 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.932 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.932 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:57.932 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.933 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.933 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.933 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-12T06:05:57.933 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.933 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.933 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.933 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:57.934 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:57.934 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.934 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.934 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.934 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:57.934 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.934 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.934 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.935 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to wake up and check device health", 2024-03-12T06:05:57.935 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.935 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.935 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.935 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_threshold": { 2024-03-12T06:05:57.935 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_threshold", 2024-03-12T06:05:57.935 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.935 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.936 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.936 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "7257600", 2024-03-12T06:05:57.936 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.936 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.936 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.936 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-12T06:05:57.936 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.936 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.936 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:57.937 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "diskprediction_local", 2024-03-12T06:05:57.937 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:57.937 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:57.937 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:57.937 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:57.938 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:57.938 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.938 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.938 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.938 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.938 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.938 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.939 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.939 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.939 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.939 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.939 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.939 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.939 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.939 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.939 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.940 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.940 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.940 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.940 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:57.940 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:57.940 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.940 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.941 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.941 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.941 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.941 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.941 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.941 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.941 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.941 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.942 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.942 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:57.942 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:57.942 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.942 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.942 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.942 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:57.943 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.943 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.943 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.943 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.943 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.943 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.943 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.943 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.944 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.944 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.944 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.944 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.944 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.944 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.944 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:57.945 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:57.945 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.945 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.945 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.945 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.945 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.945 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.945 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.946 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.946 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.946 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.946 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.946 INFO:teuthology.orchestra.run.smithi076.stdout: "predict_interval": { 2024-03-12T06:05:57.946 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "predict_interval", 2024-03-12T06:05:57.947 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.947 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.947 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.947 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "86400", 2024-03-12T06:05:57.947 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.947 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.947 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.947 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.948 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.948 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.948 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.948 INFO:teuthology.orchestra.run.smithi076.stdout: "predictor_model": { 2024-03-12T06:05:57.948 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "predictor_model", 2024-03-12T06:05:57.948 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.948 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.948 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.949 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "prophetstor", 2024-03-12T06:05:57.949 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.949 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.949 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.949 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.949 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.949 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.950 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.950 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:57.950 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:57.950 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.950 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.950 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.950 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:57.951 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.951 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.951 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.951 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.951 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.951 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.951 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.951 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "influx", 2024-03-12T06:05:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": false, 2024-03-12T06:05:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "influxdb python module not found", 2024-03-12T06:05:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:57.952 INFO:teuthology.orchestra.run.smithi076.stdout: "batch_size": { 2024-03-12T06:05:57.953 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "batch_size", 2024-03-12T06:05:57.953 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.953 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.953 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.953 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5000", 2024-03-12T06:05:57.953 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.953 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.953 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.954 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-12T06:05:57.954 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.954 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.954 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.954 INFO:teuthology.orchestra.run.smithi076.stdout: "database": { 2024-03-12T06:05:57.954 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "database", 2024-03-12T06:05:57.955 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.955 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.955 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.955 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "ceph", 2024-03-12T06:05:57.955 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.955 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.955 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.955 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-12T06:05:57.955 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.956 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.956 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.956 INFO:teuthology.orchestra.run.smithi076.stdout: "hostname": { 2024-03-12T06:05:57.956 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "hostname", 2024-03-12T06:05:57.956 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.956 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.956 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.957 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.957 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.957 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.957 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.957 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB server hostname", 2024-03-12T06:05:57.957 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.957 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.957 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.958 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:57.958 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:57.958 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:57.958 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.958 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.958 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "30", 2024-03-12T06:05:57.958 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "5", 2024-03-12T06:05:57.959 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.959 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.959 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-12T06:05:57.959 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.959 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.959 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.959 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:57.960 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:57.960 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.960 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.960 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.960 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.960 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.960 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.960 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.961 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.961 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.961 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.961 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.961 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.961 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.961 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.961 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.962 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.962 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.962 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.962 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:57.962 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:57.962 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.962 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.963 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.963 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.963 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.963 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.963 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.963 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.963 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.964 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.964 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.964 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:57.964 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:57.964 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.964 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.964 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.964 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:57.965 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.965 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.965 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.965 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.965 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.965 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.965 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.966 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.966 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.966 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.966 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.966 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.966 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.966 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.967 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:57.967 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:57.967 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.967 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.967 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.967 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.967 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.967 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.967 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.968 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.968 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.968 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.968 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.968 INFO:teuthology.orchestra.run.smithi076.stdout: "password": { 2024-03-12T06:05:57.968 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "password", 2024-03-12T06:05:57.968 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.969 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.969 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.969 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.969 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.969 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.969 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.969 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "password of InfluxDB server user", 2024-03-12T06:05:57.969 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.969 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.970 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.970 INFO:teuthology.orchestra.run.smithi076.stdout: "port": { 2024-03-12T06:05:57.970 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "port", 2024-03-12T06:05:57.970 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.970 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.970 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.970 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8086", 2024-03-12T06:05:57.971 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.971 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.971 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.971 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB server port", 2024-03-12T06:05:57.971 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.971 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.971 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.971 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl": { 2024-03-12T06:05:57.972 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl", 2024-03-12T06:05:57.972 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.972 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.972 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.972 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "false", 2024-03-12T06:05:57.972 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.972 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.972 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.972 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-12T06:05:57.973 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.973 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.973 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.973 INFO:teuthology.orchestra.run.smithi076.stdout: "threads": { 2024-03-12T06:05:57.973 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "threads", 2024-03-12T06:05:57.973 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.973 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.973 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.974 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-12T06:05:57.974 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-12T06:05:57.974 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "32", 2024-03-12T06:05:57.974 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.974 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-12T06:05:57.974 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.974 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.975 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.975 INFO:teuthology.orchestra.run.smithi076.stdout: "username": { 2024-03-12T06:05:57.975 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "username", 2024-03-12T06:05:57.975 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.975 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.975 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.975 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.975 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.976 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.976 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.976 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "username of InfluxDB server user", 2024-03-12T06:05:57.976 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.976 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.976 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.976 INFO:teuthology.orchestra.run.smithi076.stdout: "verify_ssl": { 2024-03-12T06:05:57.976 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "verify_ssl", 2024-03-12T06:05:57.977 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.977 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.977 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.977 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "true", 2024-03-12T06:05:57.977 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.977 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.977 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.977 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-12T06:05:57.977 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.977 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.978 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.978 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.978 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:57.978 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "insights", 2024-03-12T06:05:57.978 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:57.978 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:57.978 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:57.979 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:57.979 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:57.979 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.979 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.979 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.979 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.979 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.979 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.980 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.980 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.980 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.980 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.980 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.980 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.980 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.980 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.981 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.981 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.981 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.981 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.981 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:57.981 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:57.981 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.981 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.982 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.982 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.982 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.982 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.982 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.982 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.982 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.982 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.983 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.983 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:57.983 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:57.983 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.983 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.983 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.984 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:57.984 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.984 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.984 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.984 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.984 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.984 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.985 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.985 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.985 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.985 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.985 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.985 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.985 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.985 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.986 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:57.986 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:57.986 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.986 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.986 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.986 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.986 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.987 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.987 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.987 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.987 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.987 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.987 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.987 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.987 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:57.988 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "iostat", 2024-03-12T06:05:57.988 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:57.988 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:57.988 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:57.988 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:57.988 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:57.989 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.989 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.989 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.989 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:57.989 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.989 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.989 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.989 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.990 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.990 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.990 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.990 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.990 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.990 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.990 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.990 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.991 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.991 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.991 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:57.991 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:57.991 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.991 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.991 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.992 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.992 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.992 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.992 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.992 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.992 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.992 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.992 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.993 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:57.993 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:57.993 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:57.993 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.993 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.993 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:57.994 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.994 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.994 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:57.994 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:57.994 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:57.994 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:57.995 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:57.995 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:57.995 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:57.995 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:57.995 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.995 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.995 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.995 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.996 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:57.996 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:57.996 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:57.996 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.996 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:57.996 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:57.996 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.997 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.997 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.997 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:57.997 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.997 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:57.997 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:57.997 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.997 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:57.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:57.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:57.998 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "k8sevents", 2024-03-12T06:05:57.998 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:57.998 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:57.998 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:57.998 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_event_retention_days": { 2024-03-12T06:05:57.998 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ceph_event_retention_days", 2024-03-12T06:05:57.998 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:57.999 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:57.999 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:57.999 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "7", 2024-03-12T06:05:57.999 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:57.999 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:57.999 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:57.999 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-12T06:05:57.999 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:57.999 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.000 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.000 INFO:teuthology.orchestra.run.smithi076.stdout: "config_check_secs": { 2024-03-12T06:05:58.000 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_check_secs", 2024-03-12T06:05:58.000 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.000 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.000 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.000 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:58.000 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "10", 2024-03-12T06:05:58.000 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.001 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.001 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-12T06:05:58.001 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.001 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.001 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.001 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.001 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.001 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.002 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.002 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.002 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.002 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.002 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.002 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.002 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.002 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.002 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.003 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.003 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.003 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.003 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.003 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.003 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.003 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.003 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.004 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.004 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.004 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.004 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.004 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.004 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.004 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.004 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.004 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.004 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.005 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.005 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.005 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.005 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.005 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.005 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.006 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.006 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.006 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.006 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.006 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.006 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.006 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.007 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.007 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.007 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.007 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.007 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.007 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.007 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.007 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.008 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.008 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.008 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.008 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.008 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.008 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.008 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.008 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.009 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.009 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.009 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.009 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.009 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.009 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.009 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.009 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.010 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.010 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "localpool", 2024-03-12T06:05:58.010 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.010 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.010 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.010 INFO:teuthology.orchestra.run.smithi076.stdout: "failure_domain": { 2024-03-12T06:05:58.011 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "failure_domain", 2024-03-12T06:05:58.011 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.011 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.011 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.011 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "host", 2024-03-12T06:05:58.011 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.011 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.012 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.012 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "failure domain for any created local pool", 2024-03-12T06:05:58.012 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-12T06:05:58.012 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.012 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.012 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.013 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.013 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.013 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.013 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.013 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.013 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.013 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.013 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.013 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.014 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.014 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.014 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.014 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.014 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.014 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.014 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.015 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.015 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.015 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.015 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.015 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.015 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.015 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.015 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.016 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.016 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.016 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.016 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.016 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.016 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.016 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.016 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.017 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.017 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.017 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.017 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.017 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.017 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.017 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.017 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.018 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.018 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.018 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.018 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.018 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.018 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.018 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.019 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.019 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.019 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.019 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.019 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.019 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.019 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.020 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.020 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.020 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.020 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.020 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.020 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.020 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.020 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.021 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.021 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.021 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.021 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": { 2024-03-12T06:05:58.021 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "min_size", 2024-03-12T06:05:58.021 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.021 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.021 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.022 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.022 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.022 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.022 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.022 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default min_size for any created local pool", 2024-03-12T06:05:58.022 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-12T06:05:58.022 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.022 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.023 INFO:teuthology.orchestra.run.smithi076.stdout: "num_rep": { 2024-03-12T06:05:58.023 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "num_rep", 2024-03-12T06:05:58.023 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.023 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.023 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.023 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3", 2024-03-12T06:05:58.023 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.023 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.023 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.024 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default replica count for any created local pool", 2024-03-12T06:05:58.024 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.024 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.024 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.024 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": { 2024-03-12T06:05:58.024 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pg_num", 2024-03-12T06:05:58.024 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.024 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.025 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.025 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "128", 2024-03-12T06:05:58.025 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.025 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.025 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.025 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default pg_num for any created local pool", 2024-03-12T06:05:58.025 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.025 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.025 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.026 INFO:teuthology.orchestra.run.smithi076.stdout: "prefix": { 2024-03-12T06:05:58.026 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prefix", 2024-03-12T06:05:58.026 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.026 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.026 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.026 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.026 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.026 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.027 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.027 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "name prefix for any created local pool", 2024-03-12T06:05:58.027 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.027 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.027 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.027 INFO:teuthology.orchestra.run.smithi076.stdout: "subtree": { 2024-03-12T06:05:58.027 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "subtree", 2024-03-12T06:05:58.027 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.028 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.028 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.028 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "rack", 2024-03-12T06:05:58.028 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.028 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.028 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.028 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-12T06:05:58.028 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-12T06:05:58.028 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.028 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.029 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.029 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.029 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mds_autoscaler", 2024-03-12T06:05:58.029 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.029 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.029 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.029 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.029 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.030 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.030 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.030 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.030 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.030 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.030 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.030 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.030 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.031 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.031 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.031 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.031 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.031 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.031 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.031 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.031 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.031 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.031 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.032 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.032 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.032 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.032 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.032 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.032 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.032 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.032 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.033 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.033 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.033 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.033 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.033 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.033 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.033 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.033 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.034 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.034 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.034 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.034 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.034 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.034 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.034 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.034 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.034 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.034 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.035 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.035 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.035 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.035 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.035 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.035 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.035 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.035 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.036 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.036 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.036 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.036 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.036 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.036 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.036 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.036 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.036 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.036 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.037 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.037 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.037 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.037 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.037 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mirroring", 2024-03-12T06:05:58.037 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.038 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.038 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.038 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.038 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.038 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.038 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.038 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.038 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.038 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.038 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.039 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.039 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.039 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.039 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.039 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.039 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.039 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.039 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.039 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.040 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.040 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.040 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.040 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.040 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.040 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.040 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.040 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.040 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.041 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.041 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.041 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.041 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.041 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.041 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.041 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.041 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.042 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.042 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.042 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.042 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.042 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.042 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.042 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.042 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.042 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.043 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.043 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.043 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.043 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.043 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.043 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.043 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.043 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.043 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.043 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.044 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.044 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.044 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.044 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.044 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.044 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.044 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.044 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.044 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.045 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.045 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.045 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.045 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.045 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.045 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.045 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "nfs", 2024-03-12T06:05:58.046 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.046 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.046 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.046 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.046 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.046 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.046 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.046 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.046 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.047 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.047 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.047 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.047 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.047 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.047 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.047 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.047 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.047 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.048 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.048 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.048 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.048 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.048 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.048 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.048 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.048 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.049 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.049 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.049 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.049 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.049 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.049 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.049 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.049 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.050 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.050 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.050 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.050 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.050 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.050 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.050 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.051 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.051 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.051 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.051 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.051 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.051 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.051 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.051 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.052 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.052 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.052 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.052 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.052 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.052 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.052 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.053 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.053 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.053 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.053 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.053 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.053 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.053 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.054 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.054 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.054 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.054 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.054 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.054 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.054 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.054 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.055 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "orchestrator", 2024-03-12T06:05:58.055 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.055 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.055 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.055 INFO:teuthology.orchestra.run.smithi076.stdout: "fail_fs": { 2024-03-12T06:05:58.055 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "fail_fs", 2024-03-12T06:05:58.055 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.055 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.056 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.056 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.056 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.056 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.056 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.056 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-12T06:05:58.056 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.056 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.057 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.057 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.057 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.057 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.057 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.057 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.057 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.058 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.058 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.058 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.058 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.058 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.058 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.058 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.058 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.058 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.059 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.059 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.059 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.059 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.059 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.059 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.059 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.060 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.060 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.060 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.060 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.060 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.060 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.060 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.060 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.061 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.061 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.061 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.061 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.061 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.061 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.061 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.062 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.062 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.062 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.062 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.062 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.062 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.062 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.062 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.063 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.063 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.063 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.063 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.063 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.063 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.063 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.063 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.064 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.064 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.064 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.064 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.064 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.064 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.064 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.065 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.065 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.065 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.065 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.065 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.065 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.065 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator": { 2024-03-12T06:05:58.066 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "orchestrator", 2024-03-12T06:05:58.066 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.066 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.066 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.066 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.066 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.066 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.066 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.067 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm", 2024-03-12T06:05:58.067 INFO:teuthology.orchestra.run.smithi076.stdout: "rook", 2024-03-12T06:05:58.067 INFO:teuthology.orchestra.run.smithi076.stdout: "test_orchestrator" 2024-03-12T06:05:58.067 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.067 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Orchestrator backend", 2024-03-12T06:05:58.067 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.067 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.067 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.068 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.068 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.068 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "osd_perf_query", 2024-03-12T06:05:58.068 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.068 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.068 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.069 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.069 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.069 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.069 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.069 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.069 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.069 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.069 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.070 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.070 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.070 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.070 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.070 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.070 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.070 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.070 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.071 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.071 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.071 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.071 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.071 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.071 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.071 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.071 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.072 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.072 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.072 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.072 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.072 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.072 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.072 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.072 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.073 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.073 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.073 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.073 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.073 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.073 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.073 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.073 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.074 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.074 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.074 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.074 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.074 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.074 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.074 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.075 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.075 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.075 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.075 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.075 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.075 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.075 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.076 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.076 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.076 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.076 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.076 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.076 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.076 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.076 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.077 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.077 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.077 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.077 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.077 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.077 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.078 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "osd_support", 2024-03-12T06:05:58.078 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.078 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.078 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.078 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.078 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.078 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.078 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.079 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.079 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.079 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.079 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.079 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.079 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.079 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.079 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.080 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.080 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.080 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.080 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.080 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.080 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.080 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.080 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.081 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.081 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.081 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.081 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.081 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.081 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.081 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.081 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.082 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.082 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.082 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.082 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.082 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.082 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.082 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.083 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.083 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.083 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.083 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.083 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.083 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.083 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.083 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.084 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.084 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.084 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.084 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.084 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.084 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.084 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.084 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.085 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.085 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.085 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.085 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.085 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.085 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.085 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.086 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.086 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.086 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.086 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.086 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.086 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.086 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.086 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.087 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.087 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.087 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pg_autoscaler", 2024-03-12T06:05:58.087 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.087 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.087 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.087 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.088 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.088 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.088 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.088 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.088 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.088 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.088 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.089 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.089 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.089 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.089 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.089 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.089 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.089 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.089 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.090 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.090 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.090 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.090 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.090 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.090 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.091 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.091 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.091 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.091 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.091 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.091 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.091 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.091 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.092 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.092 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.092 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.092 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.092 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.092 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.092 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.092 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.093 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.093 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.093 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.093 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.093 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.093 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.093 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.093 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.093 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.094 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.094 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.094 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.094 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.094 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.094 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.094 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.094 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.095 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.095 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.095 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.095 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.095 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.095 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.095 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.095 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.095 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.096 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.096 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.096 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:58.096 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:58.096 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.096 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.096 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.097 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:58.097 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.097 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.097 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.097 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.097 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.097 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.097 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.098 INFO:teuthology.orchestra.run.smithi076.stdout: "threshold": { 2024-03-12T06:05:58.098 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "threshold", 2024-03-12T06:05:58.098 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:58.098 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.098 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.098 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3.0", 2024-03-12T06:05:58.098 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1.0", 2024-03-12T06:05:58.099 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.099 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.099 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "scaling threshold", 2024-03-12T06:05:58.099 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-12T06:05:58.099 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.099 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.099 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.099 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.100 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "progress", 2024-03-12T06:05:58.100 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.100 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.100 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.100 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_pg_recovery_event": { 2024-03-12T06:05:58.100 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_pg_recovery_event", 2024-03-12T06:05:58.101 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.101 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.101 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.101 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.101 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.101 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.101 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.101 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-12T06:05:58.102 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.102 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.102 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.102 INFO:teuthology.orchestra.run.smithi076.stdout: "enabled": { 2024-03-12T06:05:58.102 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enabled", 2024-03-12T06:05:58.102 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.102 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.103 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.103 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.103 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.103 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.103 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.103 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.103 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.103 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.104 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.104 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.104 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.104 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.104 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.104 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.104 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.105 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.105 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.105 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.105 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.105 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.105 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.105 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.105 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.106 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.106 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.106 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.106 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.106 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.106 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.106 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.107 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.107 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.107 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.107 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.107 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.107 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.107 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.108 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.108 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.108 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.108 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.108 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.108 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.108 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.109 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.109 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.109 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.109 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.109 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.109 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.109 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.109 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.110 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.110 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.110 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.110 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.110 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.110 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.110 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.111 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.111 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.111 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.111 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.111 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.111 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.111 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.112 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.112 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.112 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.112 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.112 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.112 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.112 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.113 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.113 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.113 INFO:teuthology.orchestra.run.smithi076.stdout: "max_completed_events": { 2024-03-12T06:05:58.113 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_completed_events", 2024-03-12T06:05:58.113 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.113 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.114 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.114 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "50", 2024-03-12T06:05:58.114 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.114 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.114 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.114 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "number of past completed events to remember", 2024-03-12T06:05:58.114 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.114 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.115 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.115 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:58.115 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:58.115 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.115 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.115 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.116 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-12T06:05:58.116 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.116 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.116 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.116 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long the module is going to sleep", 2024-03-12T06:05:58.116 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.116 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.116 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.116 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.117 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.117 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prometheus", 2024-03-12T06:05:58.117 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.117 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.117 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.117 INFO:teuthology.orchestra.run.smithi076.stdout: "cache": { 2024-03-12T06:05:58.118 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cache", 2024-03-12T06:05:58.118 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.118 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.118 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.118 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.118 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.118 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.118 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.119 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.119 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.119 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.119 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.119 INFO:teuthology.orchestra.run.smithi076.stdout: "exclude_perf_counters": { 2024-03-12T06:05:58.119 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "exclude_perf_counters", 2024-03-12T06:05:58.119 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.120 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.120 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.120 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.120 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.120 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.120 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.120 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-12T06:05:58.120 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-12T06:05:58.121 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.121 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.121 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.121 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.121 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.121 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.121 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.122 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.122 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.122 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.122 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.122 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.122 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.122 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.122 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.123 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.123 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.123 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.123 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.123 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.123 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.123 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.124 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.124 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.124 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.124 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.124 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.124 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.124 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.124 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.124 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.125 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.125 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.125 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.125 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.125 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.125 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.125 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.125 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.126 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.126 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.126 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.126 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.126 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.126 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.126 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.127 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.127 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.127 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.127 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.127 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.127 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.127 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.127 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.127 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.128 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.128 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.128 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.128 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.128 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.128 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.128 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.128 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.128 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.129 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.129 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.129 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.129 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.129 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_stats_pools": { 2024-03-12T06:05:58.129 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_stats_pools", 2024-03-12T06:05:58.129 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.129 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.129 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.130 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.130 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.130 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.130 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.130 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.130 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.130 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.130 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.131 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-12T06:05:58.131 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-12T06:05:58.131 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.131 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.131 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.131 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "300", 2024-03-12T06:05:58.131 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.131 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.131 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.132 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.132 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.132 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.132 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.132 INFO:teuthology.orchestra.run.smithi076.stdout: "scrape_interval": { 2024-03-12T06:05:58.132 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "scrape_interval", 2024-03-12T06:05:58.132 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:58.132 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.133 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.133 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15.0", 2024-03-12T06:05:58.133 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.133 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.133 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.133 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.133 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.133 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.133 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.134 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-12T06:05:58.134 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-12T06:05:58.134 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.134 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.134 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.134 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "::", 2024-03-12T06:05:58.134 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.134 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.135 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.135 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-12T06:05:58.135 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.135 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.135 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.135 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-12T06:05:58.135 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-12T06:05:58.135 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.136 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.136 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.136 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "9283", 2024-03-12T06:05:58.136 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.136 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.136 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.136 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-12T06:05:58.136 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.136 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.136 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.137 INFO:teuthology.orchestra.run.smithi076.stdout: "stale_cache_strategy": { 2024-03-12T06:05:58.137 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "stale_cache_strategy", 2024-03-12T06:05:58.137 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.137 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.137 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.137 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "log", 2024-03-12T06:05:58.137 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.137 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.138 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.138 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.138 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.138 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.138 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.138 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_behaviour": { 2024-03-12T06:05:58.138 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_behaviour", 2024-03-12T06:05:58.138 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.139 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.139 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.139 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "default", 2024-03-12T06:05:58.139 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.139 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.139 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.139 INFO:teuthology.orchestra.run.smithi076.stdout: "default", 2024-03-12T06:05:58.139 INFO:teuthology.orchestra.run.smithi076.stdout: "error" 2024-03-12T06:05:58.139 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.140 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.140 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.140 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.140 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.140 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_error_status_code": { 2024-03-12T06:05:58.140 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_error_status_code", 2024-03-12T06:05:58.140 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.140 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.141 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.141 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "500", 2024-03-12T06:05:58.141 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "400", 2024-03-12T06:05:58.141 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "599", 2024-03-12T06:05:58.141 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.141 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.141 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.141 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.142 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.142 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.142 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.142 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_support", 2024-03-12T06:05:58.142 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.142 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.142 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.143 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.143 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.143 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.143 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.143 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.143 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.143 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.143 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.144 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.144 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.144 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.144 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.144 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.144 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.144 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.144 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.144 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.145 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.145 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.145 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.145 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.145 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.145 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.145 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.145 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.146 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.146 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.146 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.146 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.146 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.146 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.146 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.146 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.147 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.147 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.147 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.147 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.147 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.147 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.147 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.147 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.148 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.148 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.148 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.148 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.148 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.148 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.148 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.148 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.149 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.149 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.149 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.149 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.149 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.149 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.149 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.149 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.150 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.150 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.150 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.150 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.150 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.150 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.150 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.151 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.151 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.151 INFO:teuthology.orchestra.run.smithi076.stdout: "max_concurrent_snap_create": { 2024-03-12T06:05:58.151 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_concurrent_snap_create", 2024-03-12T06:05:58.151 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.151 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.151 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.152 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:58.152 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.152 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.152 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.152 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.152 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.152 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.152 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.153 INFO:teuthology.orchestra.run.smithi076.stdout: "mirror_snapshot_schedule": { 2024-03-12T06:05:58.153 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mirror_snapshot_schedule", 2024-03-12T06:05:58.153 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.153 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.153 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.153 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.153 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.153 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.154 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.154 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.154 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.154 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.154 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.154 INFO:teuthology.orchestra.run.smithi076.stdout: "trash_purge_schedule": { 2024-03-12T06:05:58.154 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "trash_purge_schedule", 2024-03-12T06:05:58.155 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.155 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.155 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.155 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.155 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.155 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.155 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.155 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.155 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.157 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.158 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.158 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.158 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.158 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "restful", 2024-03-12T06:05:58.158 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.159 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.159 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.159 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_auth": { 2024-03-12T06:05:58.159 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enable_auth", 2024-03-12T06:05:58.159 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.159 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.159 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.159 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.159 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.160 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.160 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.160 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.160 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.160 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.160 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.160 INFO:teuthology.orchestra.run.smithi076.stdout: "key_file": { 2024-03-12T06:05:58.161 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "key_file", 2024-03-12T06:05:58.161 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.161 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.161 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.161 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.161 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.161 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.161 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.162 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.162 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.162 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.162 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.162 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.162 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.163 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.163 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.163 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.163 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.163 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.163 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.163 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.164 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.164 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.164 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.164 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.164 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.164 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.164 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.164 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.164 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.165 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.165 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.165 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.165 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.165 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.165 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.165 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.166 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.166 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.166 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.166 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.166 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.166 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.167 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.167 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.167 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.167 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.167 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.167 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.167 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.168 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.168 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.168 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.168 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.168 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.168 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.168 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.168 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.169 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.169 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.169 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.169 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.169 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.169 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.169 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.170 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.170 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.170 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.170 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.170 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.170 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.170 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.170 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.171 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.171 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.171 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.171 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.171 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.171 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-12T06:05:58.171 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-12T06:05:58.172 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.172 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.172 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.172 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.172 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.172 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.172 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.172 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.173 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.173 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.173 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.173 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-12T06:05:58.173 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-12T06:05:58.173 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.173 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.174 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.174 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.174 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.174 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.174 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.174 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.174 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.174 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.175 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.175 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.175 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.175 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rgw", 2024-03-12T06:05:58.175 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.175 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.176 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.176 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.176 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.176 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.176 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.176 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.176 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.176 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.177 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.177 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.177 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.177 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.177 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.177 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.177 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.177 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.177 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.178 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.178 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.178 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.178 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.178 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.178 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.178 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.179 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.179 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.179 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.179 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.179 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.179 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.179 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.180 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.180 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.180 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.180 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.180 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.180 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.181 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.181 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.181 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.181 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.181 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.181 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.181 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.181 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.182 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.182 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.182 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.182 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.182 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.182 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.182 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.183 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.183 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.183 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.183 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.183 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.183 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.184 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.184 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.184 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.184 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.184 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.184 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.184 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.185 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.185 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.185 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.185 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.185 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rook", 2024-03-12T06:05:58.186 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.186 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.186 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.186 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.186 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.186 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.186 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.186 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.187 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.187 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.187 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.187 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.187 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.187 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.187 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.187 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.188 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.188 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.188 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.188 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.188 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.188 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.188 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.189 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.189 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.189 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.189 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.189 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.189 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.189 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.190 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.190 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.190 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.190 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.190 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.190 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.190 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.191 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.191 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.191 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.191 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.191 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.191 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.191 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.192 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.192 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.192 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.192 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.192 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.192 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.192 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.193 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.193 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.193 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.193 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.193 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.193 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.193 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.194 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.194 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.194 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.194 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.194 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.194 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.194 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.194 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.195 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.195 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.195 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.195 INFO:teuthology.orchestra.run.smithi076.stdout: "storage_class": { 2024-03-12T06:05:58.195 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "storage_class", 2024-03-12T06:05:58.195 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.195 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.196 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.196 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "local", 2024-03-12T06:05:58.196 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.196 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.196 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.196 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-12T06:05:58.196 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.196 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.197 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.197 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.197 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.197 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "selftest", 2024-03-12T06:05:58.197 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.197 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.197 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.198 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.198 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.198 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.198 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.198 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.198 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.198 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.199 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.199 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.199 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.199 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.199 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.199 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.199 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.199 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.200 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.200 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.200 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.200 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.200 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.200 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.200 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.201 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.201 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.201 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.201 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.201 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.201 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.201 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.201 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.201 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.202 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.202 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.202 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.202 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.202 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.202 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.202 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.203 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.203 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.203 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.203 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.203 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.203 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.203 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.203 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.204 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.204 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.204 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.204 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.204 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.204 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.204 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.204 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.205 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.205 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.205 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.205 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.205 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.205 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.205 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.205 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.206 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.206 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.206 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.206 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.206 INFO:teuthology.orchestra.run.smithi076.stdout: "roption1": { 2024-03-12T06:05:58.206 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "roption1", 2024-03-12T06:05:58.206 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.207 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.207 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.207 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.207 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.207 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.207 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.207 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.207 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.207 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.208 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.208 INFO:teuthology.orchestra.run.smithi076.stdout: "roption2": { 2024-03-12T06:05:58.208 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "roption2", 2024-03-12T06:05:58.208 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.208 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.208 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.208 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "xyz", 2024-03-12T06:05:58.209 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.209 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.209 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.209 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.209 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.209 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.209 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.210 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption1": { 2024-03-12T06:05:58.210 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption1", 2024-03-12T06:05:58.210 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.210 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.210 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.210 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.210 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.211 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.211 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.211 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.211 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.211 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.211 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.211 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption2": { 2024-03-12T06:05:58.212 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption2", 2024-03-12T06:05:58.212 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.212 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.212 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.212 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.212 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.212 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.212 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.213 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.213 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.213 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.213 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.213 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption3": { 2024-03-12T06:05:58.213 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption3", 2024-03-12T06:05:58.213 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:58.214 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.214 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.214 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.214 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.214 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.214 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.214 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.214 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.215 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.215 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.215 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption4": { 2024-03-12T06:05:58.215 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption4", 2024-03-12T06:05:58.215 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.215 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.215 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.215 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.216 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.216 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.216 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.216 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.216 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.216 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.216 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.217 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption5": { 2024-03-12T06:05:58.217 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption5", 2024-03-12T06:05:58.217 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.217 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.217 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.217 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.217 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.217 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.218 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.218 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.218 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.218 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.218 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.218 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption6": { 2024-03-12T06:05:58.218 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption6", 2024-03-12T06:05:58.219 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.219 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.219 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.219 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.219 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.219 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.219 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.219 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.220 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.220 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.220 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.220 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption7": { 2024-03-12T06:05:58.220 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption7", 2024-03-12T06:05:58.220 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.220 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.220 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.221 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.221 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-12T06:05:58.221 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "42", 2024-03-12T06:05:58.221 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.221 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.221 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.221 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.221 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.222 INFO:teuthology.orchestra.run.smithi076.stdout: "testkey": { 2024-03-12T06:05:58.222 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testkey", 2024-03-12T06:05:58.222 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.222 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.222 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.222 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.222 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.223 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.223 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.223 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.223 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.223 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.223 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.223 INFO:teuthology.orchestra.run.smithi076.stdout: "testlkey": { 2024-03-12T06:05:58.224 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testlkey", 2024-03-12T06:05:58.224 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.224 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.224 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.224 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.224 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.224 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.224 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.225 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.225 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.225 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.225 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.225 INFO:teuthology.orchestra.run.smithi076.stdout: "testnewline": { 2024-03-12T06:05:58.225 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testnewline", 2024-03-12T06:05:58.225 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.226 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.226 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.226 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.226 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.226 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.226 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.226 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.227 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.227 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.227 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.227 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.227 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.227 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snap_schedule", 2024-03-12T06:05:58.228 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.228 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.228 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.228 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_m_granularity": { 2024-03-12T06:05:58.228 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_m_granularity", 2024-03-12T06:05:58.228 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.228 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.228 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.229 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.229 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.229 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.229 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.229 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow minute scheduled snapshots", 2024-03-12T06:05:58.229 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.229 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.229 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.230 INFO:teuthology.orchestra.run.smithi076.stdout: "dump_on_update": { 2024-03-12T06:05:58.230 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "dump_on_update", 2024-03-12T06:05:58.230 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.230 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.230 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.230 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.230 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.230 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.231 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.231 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "dump database to debug log on update", 2024-03-12T06:05:58.231 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.231 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.231 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.231 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.231 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.231 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.232 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.232 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.232 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.232 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.232 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.232 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.232 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.232 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.233 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.233 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.233 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.233 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.233 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.233 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.233 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.233 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.234 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.234 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.234 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.234 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.234 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.234 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.234 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.234 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.235 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.235 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.235 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.235 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.235 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.235 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.235 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.235 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.236 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.236 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.236 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.236 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.236 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.236 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.236 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.236 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.236 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.237 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.237 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.237 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.237 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.237 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.237 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.237 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.237 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.237 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.238 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.238 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.238 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.238 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.238 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.238 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.238 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.238 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.239 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.239 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.239 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.239 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.239 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.239 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.239 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.240 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "stats", 2024-03-12T06:05:58.240 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.240 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.240 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.240 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.240 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.240 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.240 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.240 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.240 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.241 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.241 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.241 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.241 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.241 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.241 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.241 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.241 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.241 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.241 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.242 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.242 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.242 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.242 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.242 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.242 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.242 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.243 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.243 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.243 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.243 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.243 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.243 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.243 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.243 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.244 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.244 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.244 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.244 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.244 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.244 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.244 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.245 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.245 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.245 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.245 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.245 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.245 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.245 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.245 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.245 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.246 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.246 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.246 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.246 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.246 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.246 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.246 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.247 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.247 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.247 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.247 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.247 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.247 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.247 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.247 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.247 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.248 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.248 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.248 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.248 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.248 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.248 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "status", 2024-03-12T06:05:58.249 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.249 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.249 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.249 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.249 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.249 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.249 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.249 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.249 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.250 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.250 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.250 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.250 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.250 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.250 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.250 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.250 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.251 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.251 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.251 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.251 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.251 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.251 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.251 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.252 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.252 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.252 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.252 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.252 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.252 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.252 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.252 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.253 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.253 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.253 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.253 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.253 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.253 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.254 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.254 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.254 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.254 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.254 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.254 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.254 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.254 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.254 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.255 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.255 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.255 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.255 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.255 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.255 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.255 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.255 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.255 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.256 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.256 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.256 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.256 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.256 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.256 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.256 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.257 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.257 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.257 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.257 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.257 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.257 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.257 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.257 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.258 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "telegraf", 2024-03-12T06:05:58.258 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.258 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.258 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.258 INFO:teuthology.orchestra.run.smithi076.stdout: "address": { 2024-03-12T06:05:58.258 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "address", 2024-03-12T06:05:58.258 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.259 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.259 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.259 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-12T06:05:58.259 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.259 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.259 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.259 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.259 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.260 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.260 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.260 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:58.260 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:58.260 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.260 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.260 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.260 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15", 2024-03-12T06:05:58.261 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.261 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.261 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.261 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.261 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.261 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.261 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.261 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.262 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.262 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.262 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.262 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.262 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.262 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.262 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.262 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.263 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.263 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.263 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.263 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.263 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.263 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.263 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.263 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.263 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.264 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.264 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.264 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.264 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.264 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.264 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.264 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.265 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.265 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.265 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.265 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.265 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.265 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.265 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.265 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.266 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.266 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.266 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.266 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.266 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.266 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.266 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.267 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.267 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.267 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.267 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.267 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.267 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.267 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.267 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.268 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.268 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.268 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.268 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.268 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.268 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.268 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.268 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.269 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.269 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.269 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.269 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.269 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.269 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.269 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.269 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.270 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.270 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.270 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.270 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.270 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "telemetry", 2024-03-12T06:05:58.270 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.271 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.271 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.271 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_basic": { 2024-03-12T06:05:58.271 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_basic", 2024-03-12T06:05:58.271 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.271 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.271 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.271 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.271 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.272 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.272 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.272 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-12T06:05:58.272 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.272 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.272 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.272 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_crash": { 2024-03-12T06:05:58.272 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_crash", 2024-03-12T06:05:58.273 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.273 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.273 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.273 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.273 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.273 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.273 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.274 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-12T06:05:58.274 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.274 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.274 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.274 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_device": { 2024-03-12T06:05:58.274 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_device", 2024-03-12T06:05:58.274 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.275 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.275 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.275 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.275 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.275 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.275 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.275 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-12T06:05:58.275 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.276 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.276 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.276 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_ident": { 2024-03-12T06:05:58.276 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_ident", 2024-03-12T06:05:58.276 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.276 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.276 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.277 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.277 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.277 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.277 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.277 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-12T06:05:58.277 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.277 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.277 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.278 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_perf": { 2024-03-12T06:05:58.278 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_perf", 2024-03-12T06:05:58.278 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.278 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.278 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.278 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.278 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.279 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.279 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.279 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-12T06:05:58.279 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.279 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.279 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.280 INFO:teuthology.orchestra.run.smithi076.stdout: "contact": { 2024-03-12T06:05:58.280 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "contact", 2024-03-12T06:05:58.280 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.280 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.280 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.280 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.280 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.280 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.281 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.281 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.281 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.281 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.281 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.281 INFO:teuthology.orchestra.run.smithi076.stdout: "description": { 2024-03-12T06:05:58.281 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "description", 2024-03-12T06:05:58.281 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.282 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.282 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.282 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.282 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.282 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.282 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.282 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.282 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.283 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.283 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.283 INFO:teuthology.orchestra.run.smithi076.stdout: "device_url": { 2024-03-12T06:05:58.283 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_url", 2024-03-12T06:05:58.283 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.283 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.283 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.284 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-12T06:05:58.284 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.284 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.284 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.284 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.284 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.284 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.284 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.285 INFO:teuthology.orchestra.run.smithi076.stdout: "enabled": { 2024-03-12T06:05:58.285 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enabled", 2024-03-12T06:05:58.285 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.285 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.285 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.285 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.285 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.285 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.285 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.286 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.286 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.286 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.286 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.286 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:58.286 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:58.286 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.286 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.287 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.287 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "24", 2024-03-12T06:05:58.287 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "8", 2024-03-12T06:05:58.287 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.287 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.287 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.287 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.287 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.288 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.288 INFO:teuthology.orchestra.run.smithi076.stdout: "last_opt_revision": { 2024-03-12T06:05:58.288 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "last_opt_revision", 2024-03-12T06:05:58.288 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.288 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.288 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.288 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1", 2024-03-12T06:05:58.288 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.288 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.289 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.289 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.289 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.289 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.289 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.289 INFO:teuthology.orchestra.run.smithi076.stdout: "leaderboard": { 2024-03-12T06:05:58.289 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "leaderboard", 2024-03-12T06:05:58.289 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.290 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.290 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.290 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.290 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.290 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.290 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.290 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.290 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.291 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.291 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.291 INFO:teuthology.orchestra.run.smithi076.stdout: "leaderboard_description": { 2024-03-12T06:05:58.291 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "leaderboard_description", 2024-03-12T06:05:58.291 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.291 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.291 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.291 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.292 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.292 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.292 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.292 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.292 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.292 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.292 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.293 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.293 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.293 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.293 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.293 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.293 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.293 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.293 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.293 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.294 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.294 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.294 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.294 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.294 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.294 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.294 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.295 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.295 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.295 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.295 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.295 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.295 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.295 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.296 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.296 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.296 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.296 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.296 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.296 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.296 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.296 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.297 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.297 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.297 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.297 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.297 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.297 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.297 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.298 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.298 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.298 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.298 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.298 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.298 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.298 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.298 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.299 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.299 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.299 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.299 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.299 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.299 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.299 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.299 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.300 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.300 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.300 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.300 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.300 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.300 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.300 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.300 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.300 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.301 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.301 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.301 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.301 INFO:teuthology.orchestra.run.smithi076.stdout: "organization": { 2024-03-12T06:05:58.301 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "organization", 2024-03-12T06:05:58.301 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.301 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.301 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.301 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.302 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.302 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.302 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.302 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.302 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.302 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.302 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.302 INFO:teuthology.orchestra.run.smithi076.stdout: "proxy": { 2024-03-12T06:05:58.303 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "proxy", 2024-03-12T06:05:58.303 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.303 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.303 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.303 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.303 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.303 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.303 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.303 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.304 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.304 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.304 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.304 INFO:teuthology.orchestra.run.smithi076.stdout: "url": { 2024-03-12T06:05:58.304 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "url", 2024-03-12T06:05:58.304 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.304 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.304 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.305 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-12T06:05:58.305 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.305 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.305 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.305 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.305 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.305 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.305 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.305 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.306 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.306 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "test_orchestrator", 2024-03-12T06:05:58.306 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.306 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.306 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.306 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.306 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.307 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.307 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.307 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.307 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.307 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.307 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.307 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.307 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.307 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.308 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.308 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.308 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.308 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.308 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.308 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.308 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.309 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.309 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.309 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.309 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.309 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.309 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.309 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.310 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.310 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.310 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.310 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.310 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.310 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.310 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.310 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.311 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.311 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.311 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.311 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.311 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.311 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.311 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.312 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.312 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.312 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.312 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.312 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.312 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.312 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.312 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.312 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.313 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.313 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.313 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.313 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.313 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.313 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.313 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.314 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.314 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.314 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.314 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.314 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.314 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.314 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.314 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.315 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.315 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.315 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.315 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.315 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "volumes", 2024-03-12T06:05:58.315 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.315 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.316 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.316 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.316 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.316 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.316 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.316 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.316 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.316 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.316 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.317 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.317 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.317 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.317 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.317 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.317 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.317 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.317 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.318 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.318 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.318 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.318 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.318 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.318 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.319 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.319 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.319 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.319 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.319 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.319 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.319 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.320 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.320 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.320 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.320 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.320 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.320 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.320 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.321 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.321 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.321 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.321 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.321 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.321 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.321 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.321 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.321 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.322 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.322 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.322 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.322 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.322 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.322 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.322 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.322 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.323 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.323 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.323 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.323 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.323 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.323 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.323 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.323 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.323 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.324 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.324 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.324 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.324 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.324 INFO:teuthology.orchestra.run.smithi076.stdout: "max_concurrent_clones": { 2024-03-12T06:05:58.324 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_concurrent_clones", 2024-03-12T06:05:58.324 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.325 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.325 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.325 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "4", 2024-03-12T06:05:58.325 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.325 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.325 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.325 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-12T06:05:58.325 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.325 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.325 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.326 INFO:teuthology.orchestra.run.smithi076.stdout: "periodic_async_work": { 2024-03-12T06:05:58.326 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "periodic_async_work", 2024-03-12T06:05:58.326 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.326 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.326 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.326 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.327 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.327 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.327 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.327 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Periodically check for async work", 2024-03-12T06:05:58.327 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.327 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.327 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.328 INFO:teuthology.orchestra.run.smithi076.stdout: "snapshot_clone_delay": { 2024-03-12T06:05:58.328 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snapshot_clone_delay", 2024-03-12T06:05:58.328 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.328 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.328 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.328 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:58.328 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.329 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.329 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.329 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-12T06:05:58.329 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.329 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.329 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.329 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.329 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.330 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix", 2024-03-12T06:05:58.330 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.330 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.330 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.330 INFO:teuthology.orchestra.run.smithi076.stdout: "discovery_interval": { 2024-03-12T06:05:58.330 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "discovery_interval", 2024-03-12T06:05:58.330 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:58.331 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.331 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.331 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "100", 2024-03-12T06:05:58.331 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.331 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.331 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.331 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.331 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.331 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.332 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.332 INFO:teuthology.orchestra.run.smithi076.stdout: "identifier": { 2024-03-12T06:05:58.332 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "identifier", 2024-03-12T06:05:58.332 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.332 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.332 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.332 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.332 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.333 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.333 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.333 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.333 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.333 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.333 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.333 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:58.333 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:58.334 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.334 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.334 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.334 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:58.334 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.334 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.334 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.334 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.335 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.335 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.335 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.335 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.335 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.335 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.335 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.335 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.336 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.336 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.336 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.336 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.336 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.336 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.336 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.336 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.336 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.337 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.337 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.337 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.337 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.337 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.337 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.337 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.338 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.338 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.338 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.338 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.338 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.338 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.338 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.338 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.339 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.339 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.339 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.339 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.339 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.339 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.339 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.339 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.340 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.340 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.340 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.340 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.340 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.340 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.340 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.340 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.340 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.341 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.341 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.341 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.341 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.341 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.341 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.341 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.341 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.342 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.342 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.342 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.342 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.342 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.342 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.342 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.342 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.342 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.342 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.343 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.343 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.343 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_host": { 2024-03-12T06:05:58.343 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_host", 2024-03-12T06:05:58.343 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.343 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.343 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.343 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.344 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.344 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.344 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.344 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.344 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.344 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.344 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.344 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_port": { 2024-03-12T06:05:58.344 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_port", 2024-03-12T06:05:58.345 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.345 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.345 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.345 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10051", 2024-03-12T06:05:58.345 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.345 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.345 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.345 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.345 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.346 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.346 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.346 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_sender": { 2024-03-12T06:05:58.346 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_sender", 2024-03-12T06:05:58.346 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.346 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.346 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.346 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-12T06:05:58.346 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.347 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.347 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.347 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.347 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.347 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.347 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.347 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.347 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.347 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-12T06:05:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "gid": 25001, 2024-03-12T06:05:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "y", 2024-03-12T06:05:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_features": 4540701547738038271, 2024-03-12T06:05:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "available_modules": [ 2024-03-12T06:05:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "alerts", 2024-03-12T06:05:58.348 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:58.349 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How frequently to reexamine health status", 2024-03-12T06:05:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.350 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.351 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.352 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.353 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.354 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.355 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.356 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.357 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.358 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_destination": { 2024-03-12T06:05:58.359 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_destination", 2024-03-12T06:05:58.359 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.359 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.359 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.359 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.359 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.359 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.359 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.359 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Email address to send alerts to", 2024-03-12T06:05:58.360 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.361 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.361 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.361 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_from_name": { 2024-03-12T06:05:58.361 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_from_name", 2024-03-12T06:05:58.362 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.362 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.362 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.362 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "Ceph", 2024-03-12T06:05:58.362 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.362 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.362 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.362 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Email From: name", 2024-03-12T06:05:58.362 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.363 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.363 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.363 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_host": { 2024-03-12T06:05:58.363 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_host", 2024-03-12T06:05:58.363 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.363 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.363 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.364 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.364 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.364 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.364 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.364 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP server", 2024-03-12T06:05:58.364 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.364 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.364 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.364 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_password": { 2024-03-12T06:05:58.365 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_password", 2024-03-12T06:05:58.365 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.365 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.365 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.365 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.365 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.365 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.365 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.365 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Password to authenticate with", 2024-03-12T06:05:58.366 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.366 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.366 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.366 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_port": { 2024-03-12T06:05:58.366 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_port", 2024-03-12T06:05:58.366 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.366 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.366 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.367 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "465", 2024-03-12T06:05:58.367 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.367 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.367 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.367 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP port", 2024-03-12T06:05:58.367 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.367 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.367 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.368 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_sender": { 2024-03-12T06:05:58.368 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_sender", 2024-03-12T06:05:58.368 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.368 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.368 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.368 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.368 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.368 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.368 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.369 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP envelope sender", 2024-03-12T06:05:58.369 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.369 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.369 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.369 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_ssl": { 2024-03-12T06:05:58.369 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_ssl", 2024-03-12T06:05:58.369 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.369 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.370 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.370 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.370 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.370 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.370 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.370 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-12T06:05:58.370 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.370 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.371 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.371 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_user": { 2024-03-12T06:05:58.371 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_user", 2024-03-12T06:05:58.371 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.371 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.371 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.372 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.372 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.372 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.372 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.372 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "User to authenticate as", 2024-03-12T06:05:58.372 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.372 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.373 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.373 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "balancer", 2024-03-12T06:05:58.373 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.373 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.373 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.374 INFO:teuthology.orchestra.run.smithi076.stdout: "active": { 2024-03-12T06:05:58.374 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "active", 2024-03-12T06:05:58.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.374 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.374 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.374 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.374 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.375 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.375 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.375 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "automatically balance PGs across cluster", 2024-03-12T06:05:58.375 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.375 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.375 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.375 INFO:teuthology.orchestra.run.smithi076.stdout: "begin_time": { 2024-03-12T06:05:58.376 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "begin_time", 2024-03-12T06:05:58.376 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.376 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.376 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.376 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0000", 2024-03-12T06:05:58.376 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.376 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.376 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.376 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "beginning time of day to automatically balance", 2024-03-12T06:05:58.377 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-12T06:05:58.377 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.377 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.377 INFO:teuthology.orchestra.run.smithi076.stdout: "begin_weekday": { 2024-03-12T06:05:58.377 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "begin_weekday", 2024-03-12T06:05:58.377 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:58.377 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.377 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.378 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:58.378 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0", 2024-03-12T06:05:58.378 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "6", 2024-03-12T06:05:58.378 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.378 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-12T06:05:58.378 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-12T06:05:58.378 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.378 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.379 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_max_iterations": { 2024-03-12T06:05:58.379 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_max_iterations", 2024-03-12T06:05:58.379 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:58.379 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.379 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.379 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "25", 2024-03-12T06:05:58.379 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-12T06:05:58.379 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "250", 2024-03-12T06:05:58.379 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.380 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-12T06:05:58.380 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.380 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.380 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.380 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_metrics": { 2024-03-12T06:05:58.380 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_metrics", 2024-03-12T06:05:58.380 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.381 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.381 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.381 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "pgs,objects,bytes", 2024-03-12T06:05:58.381 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.381 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.381 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.381 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-12T06:05:58.381 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-12T06:05:58.381 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.382 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.382 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_step": { 2024-03-12T06:05:58.382 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_step", 2024-03-12T06:05:58.382 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:58.382 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.382 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.382 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0.5", 2024-03-12T06:05:58.383 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0.001", 2024-03-12T06:05:58.383 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "0.999", 2024-03-12T06:05:58.383 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.383 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "aggressiveness of optimization", 2024-03-12T06:05:58.383 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-12T06:05:58.383 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.383 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.383 INFO:teuthology.orchestra.run.smithi076.stdout: "end_time": { 2024-03-12T06:05:58.384 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "end_time", 2024-03-12T06:05:58.384 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.384 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.384 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.384 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "2359", 2024-03-12T06:05:58.384 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.384 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.384 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.384 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "ending time of day to automatically balance", 2024-03-12T06:05:58.384 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-12T06:05:58.385 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.385 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.385 INFO:teuthology.orchestra.run.smithi076.stdout: "end_weekday": { 2024-03-12T06:05:58.385 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "end_weekday", 2024-03-12T06:05:58.385 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:58.385 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.385 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.385 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:58.386 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0", 2024-03-12T06:05:58.386 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "6", 2024-03-12T06:05:58.386 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.386 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-12T06:05:58.386 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-12T06:05:58.386 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.386 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.387 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.387 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.387 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.387 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.387 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.387 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.387 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.387 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.387 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.388 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.388 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.388 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.388 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.388 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.388 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.388 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.388 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.388 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.389 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.389 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.389 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.389 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.389 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.389 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.390 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.390 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.390 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.390 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.390 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.390 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.390 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.390 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.391 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.391 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.391 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.391 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.391 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.391 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.392 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.392 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.392 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.392 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.392 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.392 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.392 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.392 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.393 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.393 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.393 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.393 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.393 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.393 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.393 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.394 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.394 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.394 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.394 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.394 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.394 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.394 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.394 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.395 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.395 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.395 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.395 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.395 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.395 INFO:teuthology.orchestra.run.smithi076.stdout: "min_score": { 2024-03-12T06:05:58.395 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "min_score", 2024-03-12T06:05:58.396 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:58.396 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.396 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.396 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:58.396 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.396 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.396 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.396 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-12T06:05:58.396 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.397 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.397 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.397 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": { 2024-03-12T06:05:58.397 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mode", 2024-03-12T06:05:58.397 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.397 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.398 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.398 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "upmap", 2024-03-12T06:05:58.398 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.398 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.398 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.398 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-compat", 2024-03-12T06:05:58.398 INFO:teuthology.orchestra.run.smithi076.stdout: "none", 2024-03-12T06:05:58.398 INFO:teuthology.orchestra.run.smithi076.stdout: "read", 2024-03-12T06:05:58.399 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap", 2024-03-12T06:05:58.399 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap-read" 2024-03-12T06:05:58.399 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.399 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Balancer mode", 2024-03-12T06:05:58.399 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.399 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.399 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.400 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_ids": { 2024-03-12T06:05:58.400 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pool_ids", 2024-03-12T06:05:58.400 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.400 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.400 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.400 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.400 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.400 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.401 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.401 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-12T06:05:58.401 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.401 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.401 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.401 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:58.401 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:58.402 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.402 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.402 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.402 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:58.402 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.402 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.402 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.402 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-12T06:05:58.403 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.403 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.403 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.403 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap_max_deviation": { 2024-03-12T06:05:58.403 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "upmap_max_deviation", 2024-03-12T06:05:58.403 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.403 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.404 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.404 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-12T06:05:58.404 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-12T06:05:58.404 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.404 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.404 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-12T06:05:58.404 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-12T06:05:58.404 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.405 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.405 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap_max_optimizations": { 2024-03-12T06:05:58.405 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "upmap_max_optimizations", 2024-03-12T06:05:58.405 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:58.405 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.405 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.405 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:58.406 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.406 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.406 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.406 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-12T06:05:58.406 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.406 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.406 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.406 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.407 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.407 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cephadm", 2024-03-12T06:05:58.407 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.407 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.407 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.407 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_down_multiplier": { 2024-03-12T06:05:58.407 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_down_multiplier", 2024-03-12T06:05:58.407 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:58.408 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.408 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.408 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3.0", 2024-03-12T06:05:58.408 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.408 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.408 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.408 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-12T06:05:58.408 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.408 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.409 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.409 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_refresh_rate": { 2024-03-12T06:05:58.409 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_refresh_rate", 2024-03-12T06:05:58.409 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.409 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.409 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.409 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "20", 2024-03-12T06:05:58.409 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.410 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.410 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.410 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-12T06:05:58.410 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.410 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.410 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.410 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_starting_port": { 2024-03-12T06:05:58.410 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_starting_port", 2024-03-12T06:05:58.411 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.411 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.411 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.411 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "4721", 2024-03-12T06:05:58.411 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.411 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.411 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.412 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-12T06:05:58.412 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.412 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.412 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.412 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_ptrace": { 2024-03-12T06:05:58.412 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_ptrace", 2024-03-12T06:05:58.413 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.413 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.413 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.413 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.413 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.413 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.414 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.414 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-12T06:05:58.414 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-12T06:05:58.414 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.414 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.414 INFO:teuthology.orchestra.run.smithi076.stdout: "autotune_interval": { 2024-03-12T06:05:58.415 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "autotune_interval", 2024-03-12T06:05:58.415 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.415 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.415 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.415 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:58.415 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.415 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.415 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.416 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-12T06:05:58.416 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.416 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.416 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.416 INFO:teuthology.orchestra.run.smithi076.stdout: "autotune_memory_target_ratio": { 2024-03-12T06:05:58.416 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "autotune_memory_target_ratio", 2024-03-12T06:05:58.416 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:58.416 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.416 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.417 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0.7", 2024-03-12T06:05:58.417 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.417 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.417 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.417 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-12T06:05:58.417 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.417 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.417 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.418 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm_log_destination": { 2024-03-12T06:05:58.418 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cephadm_log_destination", 2024-03-12T06:05:58.418 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.418 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.418 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.418 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.418 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.418 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.419 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.419 INFO:teuthology.orchestra.run.smithi076.stdout: "file", 2024-03-12T06:05:58.419 INFO:teuthology.orchestra.run.smithi076.stdout: "file,syslog", 2024-03-12T06:05:58.419 INFO:teuthology.orchestra.run.smithi076.stdout: "syslog" 2024-03-12T06:05:58.419 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.419 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-12T06:05:58.419 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.419 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.419 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.420 INFO:teuthology.orchestra.run.smithi076.stdout: "cgroups_split": { 2024-03-12T06:05:58.420 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cgroups_split", 2024-03-12T06:05:58.420 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.420 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.420 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.420 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.420 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.420 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.420 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.421 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-12T06:05:58.421 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.421 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.421 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.421 INFO:teuthology.orchestra.run.smithi076.stdout: "config_checks_enabled": { 2024-03-12T06:05:58.421 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_checks_enabled", 2024-03-12T06:05:58.421 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.421 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.422 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.422 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.422 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.422 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.422 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.422 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-12T06:05:58.422 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.422 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.422 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.423 INFO:teuthology.orchestra.run.smithi076.stdout: "config_dashboard": { 2024-03-12T06:05:58.423 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_dashboard", 2024-03-12T06:05:58.423 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.423 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.423 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.423 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.423 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.423 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.424 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.424 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-12T06:05:58.424 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.424 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.424 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.424 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_alertmanager": { 2024-03-12T06:05:58.424 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_alertmanager", 2024-03-12T06:05:58.425 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.425 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.425 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.425 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-12T06:05:58.425 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.425 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.425 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.425 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-12T06:05:58.425 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.426 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.426 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.426 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_base": { 2024-03-12T06:05:58.426 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_base", 2024-03-12T06:05:58.426 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.426 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.426 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.426 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-12T06:05:58.427 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.427 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.427 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.427 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Container image name, without the tag", 2024-03-12T06:05:58.427 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.427 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.427 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.427 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_elasticsearch": { 2024-03-12T06:05:58.428 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_elasticsearch", 2024-03-12T06:05:58.428 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.428 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.428 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.428 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-12T06:05:58.428 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.428 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.428 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.428 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "elasticsearch container image", 2024-03-12T06:05:58.429 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.429 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.429 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.429 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_grafana": { 2024-03-12T06:05:58.429 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_grafana", 2024-03-12T06:05:58.429 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.429 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.429 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.430 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-12T06:05:58.430 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.430 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.430 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.430 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-12T06:05:58.430 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.430 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.430 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.431 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_haproxy": { 2024-03-12T06:05:58.431 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_haproxy", 2024-03-12T06:05:58.431 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.431 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.431 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.431 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-12T06:05:58.431 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.431 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.432 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.432 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "HAproxy container image", 2024-03-12T06:05:58.432 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.432 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.432 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.432 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_agent": { 2024-03-12T06:05:58.433 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_agent", 2024-03-12T06:05:58.433 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.433 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.433 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.433 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-12T06:05:58.433 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.433 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.433 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.434 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger agent container image", 2024-03-12T06:05:58.434 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.434 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.434 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.434 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_collector": { 2024-03-12T06:05:58.434 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_collector", 2024-03-12T06:05:58.434 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.435 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.435 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.435 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-12T06:05:58.435 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.435 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.435 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.435 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger collector container image", 2024-03-12T06:05:58.436 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.436 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.436 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.436 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_query": { 2024-03-12T06:05:58.436 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_query", 2024-03-12T06:05:58.436 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.436 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.437 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.437 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-12T06:05:58.437 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.437 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.437 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.437 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger query container image", 2024-03-12T06:05:58.437 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.437 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.438 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.438 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_keepalived": { 2024-03-12T06:05:58.438 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_keepalived", 2024-03-12T06:05:58.438 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.438 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.438 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.438 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-12T06:05:58.439 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.439 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.439 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.439 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Keepalived container image", 2024-03-12T06:05:58.439 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.439 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.439 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.440 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_loki": { 2024-03-12T06:05:58.440 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_loki", 2024-03-12T06:05:58.440 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.440 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.440 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.440 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-12T06:05:58.440 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.440 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.440 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.441 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Loki container image", 2024-03-12T06:05:58.441 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.441 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.441 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.441 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_node_exporter": { 2024-03-12T06:05:58.441 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_node_exporter", 2024-03-12T06:05:58.441 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.442 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.442 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.442 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-12T06:05:58.442 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.442 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.442 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.442 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-12T06:05:58.442 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.442 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.443 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.443 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_nvmeof": { 2024-03-12T06:05:58.443 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_nvmeof", 2024-03-12T06:05:58.443 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.443 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.443 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.444 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-12T06:05:58.444 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.444 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.444 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.444 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Nvme-of container image", 2024-03-12T06:05:58.444 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.444 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.444 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.445 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_prometheus": { 2024-03-12T06:05:58.445 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_prometheus", 2024-03-12T06:05:58.445 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.445 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.445 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.445 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-12T06:05:58.445 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.445 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.445 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.446 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-12T06:05:58.446 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.446 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.446 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.446 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_promtail": { 2024-03-12T06:05:58.446 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_promtail", 2024-03-12T06:05:58.446 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.447 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.447 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.447 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-12T06:05:58.447 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.447 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.447 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.447 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Promtail container image", 2024-03-12T06:05:58.447 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.448 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.448 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.448 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_snmp_gateway": { 2024-03-12T06:05:58.448 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_snmp_gateway", 2024-03-12T06:05:58.448 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.448 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.448 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.449 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-12T06:05:58.449 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.449 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.449 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.449 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SNMP Gateway container image", 2024-03-12T06:05:58.449 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.449 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.449 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.450 INFO:teuthology.orchestra.run.smithi076.stdout: "container_init": { 2024-03-12T06:05:58.450 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_init", 2024-03-12T06:05:58.450 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.450 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.450 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.450 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.450 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.451 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.451 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.451 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Run podman/docker with `--init`", 2024-03-12T06:05:58.451 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.451 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.451 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.451 INFO:teuthology.orchestra.run.smithi076.stdout: "daemon_cache_timeout": { 2024-03-12T06:05:58.451 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "daemon_cache_timeout", 2024-03-12T06:05:58.451 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.452 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.452 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.452 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:58.452 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.452 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.452 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.452 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-12T06:05:58.452 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.453 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.453 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.453 INFO:teuthology.orchestra.run.smithi076.stdout: "default_cephadm_command_timeout": { 2024-03-12T06:05:58.453 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "default_cephadm_command_timeout", 2024-03-12T06:05:58.453 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.453 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.453 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.453 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "900", 2024-03-12T06:05:58.454 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.454 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.454 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.454 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-12T06:05:58.454 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.454 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.454 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.455 INFO:teuthology.orchestra.run.smithi076.stdout: "default_registry": { 2024-03-12T06:05:58.455 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "default_registry", 2024-03-12T06:05:58.455 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.455 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.455 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.455 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io", 2024-03-12T06:05:58.455 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.455 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.455 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.456 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-12T06:05:58.456 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.456 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.456 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.456 INFO:teuthology.orchestra.run.smithi076.stdout: "device_cache_timeout": { 2024-03-12T06:05:58.456 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_cache_timeout", 2024-03-12T06:05:58.456 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.456 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.456 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.457 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1800", 2024-03-12T06:05:58.457 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.457 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.457 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.457 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache device inventory", 2024-03-12T06:05:58.457 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.457 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.457 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.458 INFO:teuthology.orchestra.run.smithi076.stdout: "device_enhanced_scan": { 2024-03-12T06:05:58.458 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_enhanced_scan", 2024-03-12T06:05:58.458 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.458 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.458 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.458 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.458 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.458 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.459 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.459 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-12T06:05:58.459 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.459 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.459 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.459 INFO:teuthology.orchestra.run.smithi076.stdout: "facts_cache_timeout": { 2024-03-12T06:05:58.459 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "facts_cache_timeout", 2024-03-12T06:05:58.460 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.460 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.460 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.460 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:58.460 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.460 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.460 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.461 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache host facts data", 2024-03-12T06:05:58.461 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.461 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.461 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.461 INFO:teuthology.orchestra.run.smithi076.stdout: "host_check_interval": { 2024-03-12T06:05:58.461 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "host_check_interval", 2024-03-12T06:05:58.461 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.462 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.462 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.462 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:58.462 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.462 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.462 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.462 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to perform a host check", 2024-03-12T06:05:58.463 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.463 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.463 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.463 INFO:teuthology.orchestra.run.smithi076.stdout: "hw_monitoring": { 2024-03-12T06:05:58.463 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "hw_monitoring", 2024-03-12T06:05:58.463 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.463 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.463 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.464 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.464 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.464 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.464 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.464 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-12T06:05:58.464 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.464 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.464 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.465 INFO:teuthology.orchestra.run.smithi076.stdout: "inventory_list_all": { 2024-03-12T06:05:58.465 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "inventory_list_all", 2024-03-12T06:05:58.465 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.465 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.465 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.465 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.466 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.466 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.466 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.466 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-12T06:05:58.466 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.466 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.466 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.467 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.467 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.467 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.467 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.467 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.467 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.467 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.467 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.468 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.468 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.468 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.468 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.468 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.468 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.468 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.468 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.469 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.469 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.469 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.469 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.469 INFO:teuthology.orchestra.run.smithi076.stdout: "log_refresh_metadata": { 2024-03-12T06:05:58.469 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_refresh_metadata", 2024-03-12T06:05:58.469 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.469 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.470 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.470 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.470 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.470 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.470 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.470 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-12T06:05:58.470 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.470 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.471 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.471 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.471 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.471 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.471 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.471 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.471 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.471 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.472 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.472 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.472 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-12T06:05:58.472 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.472 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.472 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.472 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.473 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.473 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.473 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.473 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.473 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.473 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.473 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.473 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.473 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.474 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.474 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.474 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.474 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.474 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.474 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.474 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.474 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.475 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.475 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.475 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.475 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.475 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.475 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.475 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.476 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.476 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.476 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.476 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.476 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.476 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.476 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.476 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.477 INFO:teuthology.orchestra.run.smithi076.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-12T06:05:58.477 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-12T06:05:58.477 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.477 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.477 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.477 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.477 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.477 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.478 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.478 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-12T06:05:58.478 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.478 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.478 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.478 INFO:teuthology.orchestra.run.smithi076.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-12T06:05:58.478 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-12T06:05:58.479 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.479 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.479 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.479 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "*", 2024-03-12T06:05:58.479 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.479 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.479 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.479 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-12T06:05:58.479 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.480 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.480 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.480 INFO:teuthology.orchestra.run.smithi076.stdout: "max_count_per_host": { 2024-03-12T06:05:58.480 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_count_per_host", 2024-03-12T06:05:58.480 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.480 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.481 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.481 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:58.481 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.481 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.481 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.481 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "max number of daemons per service per host", 2024-03-12T06:05:58.481 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.481 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.481 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.482 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd_draining_count": { 2024-03-12T06:05:58.482 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_osd_draining_count", 2024-03-12T06:05:58.482 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.482 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.482 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.482 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:58.482 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.483 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.483 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.483 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-12T06:05:58.483 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.483 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.483 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.483 INFO:teuthology.orchestra.run.smithi076.stdout: "migration_current": { 2024-03-12T06:05:58.483 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "migration_current", 2024-03-12T06:05:58.483 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.484 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.484 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.484 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.484 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.484 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.484 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.484 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "internal - do not modify", 2024-03-12T06:05:58.485 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.485 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.485 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.485 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": { 2024-03-12T06:05:58.485 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mode", 2024-03-12T06:05:58.485 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.485 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.485 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.485 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "root", 2024-03-12T06:05:58.486 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.486 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.486 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.486 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm-package", 2024-03-12T06:05:58.486 INFO:teuthology.orchestra.run.smithi076.stdout: "root" 2024-03-12T06:05:58.486 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.486 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "mode for remote execution of cephadm", 2024-03-12T06:05:58.486 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.487 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.487 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.487 INFO:teuthology.orchestra.run.smithi076.stdout: "oob_default_addr": { 2024-03-12T06:05:58.487 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "oob_default_addr", 2024-03-12T06:05:58.487 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.487 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.487 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.488 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "169.254.1.1", 2024-03-12T06:05:58.488 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.488 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.488 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.488 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-12T06:05:58.488 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.488 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.488 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.489 INFO:teuthology.orchestra.run.smithi076.stdout: "prometheus_alerts_path": { 2024-03-12T06:05:58.489 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prometheus_alerts_path", 2024-03-12T06:05:58.489 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.489 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.489 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.489 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-12T06:05:58.489 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.489 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.490 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.490 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-12T06:05:58.490 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.490 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.490 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.490 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_insecure": { 2024-03-12T06:05:58.490 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_insecure", 2024-03-12T06:05:58.491 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.491 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.491 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.491 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.491 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.491 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.491 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.491 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-12T06:05:58.491 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.492 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.492 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.492 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_password": { 2024-03-12T06:05:58.492 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_password", 2024-03-12T06:05:58.492 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.492 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.492 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.492 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.493 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.493 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.493 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.493 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-12T06:05:58.493 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.493 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.493 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.493 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_url": { 2024-03-12T06:05:58.494 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_url", 2024-03-12T06:05:58.494 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.494 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.494 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.494 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.494 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.494 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.494 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.495 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-12T06:05:58.495 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.495 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.495 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.495 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_username": { 2024-03-12T06:05:58.495 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_username", 2024-03-12T06:05:58.495 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.496 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.496 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.496 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.496 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.496 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.496 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.496 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-12T06:05:58.496 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.497 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.497 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.497 INFO:teuthology.orchestra.run.smithi076.stdout: "secure_monitoring_stack": { 2024-03-12T06:05:58.497 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "secure_monitoring_stack", 2024-03-12T06:05:58.497 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.497 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.497 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.497 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.498 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.498 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.498 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.498 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-12T06:05:58.498 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.498 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.498 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.499 INFO:teuthology.orchestra.run.smithi076.stdout: "service_discovery_port": { 2024-03-12T06:05:58.499 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "service_discovery_port", 2024-03-12T06:05:58.499 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.499 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.499 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.499 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8765", 2024-03-12T06:05:58.499 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.499 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.500 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.500 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "cephadm service discovery port", 2024-03-12T06:05:58.500 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.500 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.500 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.500 INFO:teuthology.orchestra.run.smithi076.stdout: "ssh_config_file": { 2024-03-12T06:05:58.501 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssh_config_file", 2024-03-12T06:05:58.501 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.501 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.501 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.501 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.501 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.501 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.501 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.501 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-12T06:05:58.502 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.502 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.502 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.502 INFO:teuthology.orchestra.run.smithi076.stdout: "use_agent": { 2024-03-12T06:05:58.502 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "use_agent", 2024-03-12T06:05:58.502 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.502 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.503 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.503 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.503 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.503 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.503 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.503 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-12T06:05:58.503 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.503 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.503 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.504 INFO:teuthology.orchestra.run.smithi076.stdout: "use_repo_digest": { 2024-03-12T06:05:58.504 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "use_repo_digest", 2024-03-12T06:05:58.504 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.504 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.504 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.504 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.504 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.505 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.505 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.505 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-12T06:05:58.505 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.505 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.505 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.505 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_failed_host_check": { 2024-03-12T06:05:58.506 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_failed_host_check", 2024-03-12T06:05:58.506 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.506 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.506 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.506 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.506 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.506 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.506 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.507 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if the host check fails", 2024-03-12T06:05:58.507 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.507 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.507 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.507 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_stray_daemons": { 2024-03-12T06:05:58.507 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_stray_daemons", 2024-03-12T06:05:58.507 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.508 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.508 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.508 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.508 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.508 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.508 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.508 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-12T06:05:58.509 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.509 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.509 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.509 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_stray_hosts": { 2024-03-12T06:05:58.509 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_stray_hosts", 2024-03-12T06:05:58.509 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.510 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.510 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.510 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.510 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.510 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.510 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.510 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-12T06:05:58.510 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.511 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.511 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.511 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.511 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.511 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crash", 2024-03-12T06:05:58.511 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.512 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.512 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.512 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.512 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.512 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.512 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.512 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.512 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.512 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.513 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.513 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.513 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.513 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.513 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.513 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.513 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.513 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.513 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.514 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.514 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.514 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.514 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.514 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.514 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.514 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.514 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.515 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.515 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.515 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.515 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.515 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.515 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.515 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.515 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.516 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.517 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.518 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "retain_interval": { 2024-03-12T06:05:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "retain_interval", 2024-03-12T06:05:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "31536000", 2024-03-12T06:05:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.520 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-12T06:05:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_recent_interval": { 2024-03-12T06:05:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_recent_interval", 2024-03-12T06:05:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.521 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1209600", 2024-03-12T06:05:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-12T06:05:58.522 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.523 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "dashboard", 2024-03-12T06:05:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-12T06:05:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-12T06:05:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.524 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "ALERTMANAGER_API_HOST": { 2024-03-12T06:05:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-12T06:05:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.526 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.527 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.527 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.527 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.527 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.527 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.527 INFO:teuthology.orchestra.run.smithi076.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-12T06:05:58.527 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-12T06:05:58.528 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.528 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.528 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.528 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.528 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.528 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.528 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.528 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.529 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.529 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.529 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.529 INFO:teuthology.orchestra.run.smithi076.stdout: "AUDIT_API_ENABLED": { 2024-03-12T06:05:58.529 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "AUDIT_API_ENABLED", 2024-03-12T06:05:58.529 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.530 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.530 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.530 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.530 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.530 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.530 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.530 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.531 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.531 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.531 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.531 INFO:teuthology.orchestra.run.smithi076.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-12T06:05:58.531 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-12T06:05:58.531 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.531 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.531 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.532 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.532 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.532 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.532 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.532 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.532 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.532 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.532 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.533 INFO:teuthology.orchestra.run.smithi076.stdout: "ENABLE_BROWSABLE_API": { 2024-03-12T06:05:58.533 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-12T06:05:58.533 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.533 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.533 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.533 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.533 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.533 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.533 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.534 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.534 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.534 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.534 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.534 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-12T06:05:58.534 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-12T06:05:58.534 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.534 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.535 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.535 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.535 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.535 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.535 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.535 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.535 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.535 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.535 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.535 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.536 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-12T06:05:58.536 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-12T06:05:58.536 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.536 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.536 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.536 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.536 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.536 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.537 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.537 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.537 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.537 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.537 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.537 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-12T06:05:58.537 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-12T06:05:58.538 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.538 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.538 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.538 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.538 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.538 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.538 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.538 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-12T06:05:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-12T06:05:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.539 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.540 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-12T06:05:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-12T06:05:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.541 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.542 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-12T06:05:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-12T06:05:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.543 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-12T06:05:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-12T06:05:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.544 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.545 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-12T06:05:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-12T06:05:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.546 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.547 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_PASSWORD": { 2024-03-12T06:05:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-12T06:05:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-12T06:05:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.548 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-12T06:05:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-12T06:05:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.549 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.550 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.551 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.551 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.551 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_URL": { 2024-03-12T06:05:58.551 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_URL", 2024-03-12T06:05:58.551 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.551 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.551 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.551 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.552 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.552 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.552 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.552 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.552 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.552 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.552 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.553 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_USERNAME": { 2024-03-12T06:05:58.553 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-12T06:05:58.553 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.553 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.553 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.553 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-12T06:05:58.553 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.553 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.554 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.554 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.554 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.554 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.554 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.554 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-12T06:05:58.554 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-12T06:05:58.554 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.555 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.555 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.555 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.555 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.555 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.555 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.555 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.555 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.556 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.556 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.556 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.556 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-12T06:05:58.556 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-12T06:05:58.556 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.556 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.556 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.557 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.557 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.557 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.557 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.557 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.557 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.557 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.558 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.558 INFO:teuthology.orchestra.run.smithi076.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-12T06:05:58.558 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-12T06:05:58.558 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.558 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.558 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.559 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.559 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.559 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.559 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.559 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.559 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.559 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.560 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.560 INFO:teuthology.orchestra.run.smithi076.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-12T06:05:58.560 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-12T06:05:58.560 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.560 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.560 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.560 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.561 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.561 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.561 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.561 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.561 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.561 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.561 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.562 INFO:teuthology.orchestra.run.smithi076.stdout: "PROMETHEUS_API_HOST": { 2024-03-12T06:05:58.562 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-12T06:05:58.562 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.562 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.562 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.562 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.562 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.563 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.563 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.563 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.563 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.563 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.563 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.563 INFO:teuthology.orchestra.run.smithi076.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-12T06:05:58.564 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-12T06:05:58.564 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.564 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.564 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.564 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.564 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.564 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.564 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.565 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.565 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.565 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.565 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.565 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-12T06:05:58.565 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-12T06:05:58.565 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.565 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.566 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.566 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.566 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.566 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.566 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.566 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.566 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.566 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.566 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.567 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-12T06:05:58.567 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-12T06:05:58.567 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.567 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.567 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.567 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.567 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.567 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.568 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.568 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.568 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.568 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.568 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.568 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-12T06:05:58.568 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-12T06:05:58.568 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.569 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.569 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.569 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.569 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.569 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.569 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.569 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.569 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.569 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.570 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.570 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-12T06:05:58.570 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-12T06:05:58.570 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.570 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.570 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.570 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.570 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.571 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.571 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.571 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.571 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.571 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.571 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.571 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-12T06:05:58.571 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-12T06:05:58.572 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.572 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.572 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.572 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.572 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.572 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.572 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.572 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.573 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.573 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.573 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.573 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-12T06:05:58.573 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-12T06:05:58.573 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.573 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.573 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.574 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.574 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.574 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.574 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.574 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.574 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.574 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.574 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.575 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-12T06:05:58.575 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-12T06:05:58.575 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.575 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.575 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.575 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.575 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.575 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.575 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.575 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.576 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.576 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.576 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.576 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_ENABLED": { 2024-03-12T06:05:58.576 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-12T06:05:58.576 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.576 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.576 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.577 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.577 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.577 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.577 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.577 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.577 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.577 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.577 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.578 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-12T06:05:58.578 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-12T06:05:58.578 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.578 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.578 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.578 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-12T06:05:58.578 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.578 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.578 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.578 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.579 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.579 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.579 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.579 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-12T06:05:58.579 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-12T06:05:58.579 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.579 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.579 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.580 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:58.580 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.580 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.580 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.580 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.580 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.580 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.580 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.581 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-12T06:05:58.581 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-12T06:05:58.581 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.581 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.581 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.581 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8", 2024-03-12T06:05:58.581 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.581 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.581 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.582 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.582 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.582 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.582 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.582 INFO:teuthology.orchestra.run.smithi076.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-12T06:05:58.582 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-12T06:05:58.582 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.583 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.583 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.583 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "45", 2024-03-12T06:05:58.583 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.583 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.583 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.583 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.583 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.584 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.584 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.584 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.584 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_ACCESS_KEY": { 2024-03-12T06:05:58.584 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-12T06:05:58.584 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.584 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.584 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.585 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.585 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.585 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.585 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.585 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.585 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.585 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.585 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.585 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.586 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-12T06:05:58.586 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-12T06:05:58.586 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.586 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.586 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.586 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-12T06:05:58.586 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.586 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.586 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.587 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.587 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.587 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.587 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.587 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_SECRET_KEY": { 2024-03-12T06:05:58.587 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-12T06:05:58.587 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.587 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.588 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.588 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.588 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.588 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.588 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.588 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.588 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.588 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.588 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.589 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_SSL_VERIFY": { 2024-03-12T06:05:58.589 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-12T06:05:58.589 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.589 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.589 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.589 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.589 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.589 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.590 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.590 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.590 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.590 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.590 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.590 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-12T06:05:58.590 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-12T06:05:58.590 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.591 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.591 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.591 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:58.591 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.591 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.591 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.591 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.591 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.592 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.592 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.592 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-12T06:05:58.592 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-12T06:05:58.592 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.592 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.592 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.593 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:58.593 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.593 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.593 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.593 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.593 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.593 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.593 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.594 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-12T06:05:58.594 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-12T06:05:58.594 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.594 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.594 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.594 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-12T06:05:58.595 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.595 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.595 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.595 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.595 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.595 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.595 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.596 INFO:teuthology.orchestra.run.smithi076.stdout: "cross_origin_url": { 2024-03-12T06:05:58.596 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cross_origin_url", 2024-03-12T06:05:58.596 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.596 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.596 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.596 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.596 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.596 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.597 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.597 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.597 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.597 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.597 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.597 INFO:teuthology.orchestra.run.smithi076.stdout: "crt_file": { 2024-03-12T06:05:58.597 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crt_file", 2024-03-12T06:05:58.597 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.598 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.598 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.598 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.598 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.598 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.598 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.598 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.598 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.599 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.599 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.599 INFO:teuthology.orchestra.run.smithi076.stdout: "debug": { 2024-03-12T06:05:58.599 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "debug", 2024-03-12T06:05:58.599 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.599 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.599 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.600 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.600 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.600 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.600 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.600 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable/disable debug options", 2024-03-12T06:05:58.600 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.600 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.600 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.601 INFO:teuthology.orchestra.run.smithi076.stdout: "jwt_token_ttl": { 2024-03-12T06:05:58.601 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "jwt_token_ttl", 2024-03-12T06:05:58.601 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.601 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.601 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.601 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "28800", 2024-03-12T06:05:58.601 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.602 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.602 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.602 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.602 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.602 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.602 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.603 INFO:teuthology.orchestra.run.smithi076.stdout: "key_file": { 2024-03-12T06:05:58.603 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "key_file", 2024-03-12T06:05:58.603 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.603 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.603 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.603 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.603 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.604 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.604 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.604 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.604 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.604 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.604 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.605 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.605 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.605 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.605 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.605 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.605 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.605 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.605 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.605 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.606 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.606 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.606 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.606 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.606 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.606 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.606 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.606 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.607 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.607 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.607 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.607 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.607 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.607 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.607 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.608 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.608 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.608 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.608 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.608 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.608 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.608 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.608 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.609 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.609 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.609 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.609 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.609 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.609 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.609 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.610 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.610 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.610 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.610 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.610 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.610 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.610 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.610 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.611 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.611 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.611 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.611 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.611 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.611 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.611 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.611 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.612 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.612 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.612 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.612 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.612 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.612 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.612 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.612 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.613 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.613 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.613 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.613 INFO:teuthology.orchestra.run.smithi076.stdout: "motd": { 2024-03-12T06:05:58.613 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "motd", 2024-03-12T06:05:58.613 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.613 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.614 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.614 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.614 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.614 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.614 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.614 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "The message of the day", 2024-03-12T06:05:58.614 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.614 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.614 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.615 INFO:teuthology.orchestra.run.smithi076.stdout: "redirect_resolve_ip_addr": { 2024-03-12T06:05:58.615 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "redirect_resolve_ip_addr", 2024-03-12T06:05:58.615 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.615 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.615 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.615 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.616 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.616 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.616 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.616 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.616 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.616 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.616 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.616 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-12T06:05:58.617 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-12T06:05:58.617 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.617 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.617 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.617 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "::", 2024-03-12T06:05:58.617 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.617 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.617 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.617 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.618 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.618 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.618 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.618 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-12T06:05:58.618 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-12T06:05:58.618 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.618 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.619 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.619 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8080", 2024-03-12T06:05:58.619 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.619 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.619 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.619 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.619 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.619 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.619 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.620 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl": { 2024-03-12T06:05:58.620 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl", 2024-03-12T06:05:58.620 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.620 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.620 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.620 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.620 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.620 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.620 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.621 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.621 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.621 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.621 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.621 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.621 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl_server_port": { 2024-03-12T06:05:58.621 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl_server_port", 2024-03-12T06:05:58.621 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.621 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.621 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.622 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8443", 2024-03-12T06:05:58.622 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.622 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.622 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.622 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.622 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.622 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.622 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.623 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_behaviour": { 2024-03-12T06:05:58.623 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_behaviour", 2024-03-12T06:05:58.623 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.623 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.623 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.623 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "redirect", 2024-03-12T06:05:58.623 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.624 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.624 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.624 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.624 INFO:teuthology.orchestra.run.smithi076.stdout: "redirect" 2024-03-12T06:05:58.624 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.624 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.624 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.624 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.625 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.625 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_error_status_code": { 2024-03-12T06:05:58.625 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_error_status_code", 2024-03-12T06:05:58.625 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.625 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.625 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.625 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "500", 2024-03-12T06:05:58.626 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "400", 2024-03-12T06:05:58.626 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "599", 2024-03-12T06:05:58.626 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.626 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.626 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.626 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.626 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.627 INFO:teuthology.orchestra.run.smithi076.stdout: "url_prefix": { 2024-03-12T06:05:58.627 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "url_prefix", 2024-03-12T06:05:58.627 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.627 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.627 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.627 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.627 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.627 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.628 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.628 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.628 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.628 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.628 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.628 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.628 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.629 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "devicehealth", 2024-03-12T06:05:58.629 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.629 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.629 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.629 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_monitoring": { 2024-03-12T06:05:58.629 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enable_monitoring", 2024-03-12T06:05:58.630 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.630 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.630 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.630 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.630 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.630 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.630 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.630 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "monitor device health metrics", 2024-03-12T06:05:58.630 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.631 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.631 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.631 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.631 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.631 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.631 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.631 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.631 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.632 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.632 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.632 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.632 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.632 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.632 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.632 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.633 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.633 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.633 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.633 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.633 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.633 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.633 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.634 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.634 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.634 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.634 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.634 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.634 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.635 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.635 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.635 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.635 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.635 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.635 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.635 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.636 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.636 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.636 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.636 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.636 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.636 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.636 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.637 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.637 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.637 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.637 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.637 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.637 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.637 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.637 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.638 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.638 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.638 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.638 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.638 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.638 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.638 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.638 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.639 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.639 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.639 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.639 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.639 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.639 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.639 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.639 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.639 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.640 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.640 INFO:teuthology.orchestra.run.smithi076.stdout: "mark_out_threshold": { 2024-03-12T06:05:58.640 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mark_out_threshold", 2024-03-12T06:05:58.640 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.640 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.640 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.640 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "2419200", 2024-03-12T06:05:58.640 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.641 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.641 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.641 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-12T06:05:58.641 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.641 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.641 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.641 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": { 2024-03-12T06:05:58.641 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pool_name", 2024-03-12T06:05:58.642 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.642 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.642 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.642 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "device_health_metrics", 2024-03-12T06:05:58.642 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.642 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.642 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.642 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-12T06:05:58.642 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.643 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.643 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.643 INFO:teuthology.orchestra.run.smithi076.stdout: "retention_period": { 2024-03-12T06:05:58.643 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "retention_period", 2024-03-12T06:05:58.643 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.643 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.643 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.644 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15552000", 2024-03-12T06:05:58.644 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.644 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.644 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.644 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long to retain device health metrics", 2024-03-12T06:05:58.644 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.644 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.644 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.645 INFO:teuthology.orchestra.run.smithi076.stdout: "scrape_frequency": { 2024-03-12T06:05:58.645 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "scrape_frequency", 2024-03-12T06:05:58.645 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.645 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.645 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.645 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "86400", 2024-03-12T06:05:58.645 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.645 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.645 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.646 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-12T06:05:58.646 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.646 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.646 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.646 INFO:teuthology.orchestra.run.smithi076.stdout: "self_heal": { 2024-03-12T06:05:58.646 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "self_heal", 2024-03-12T06:05:58.646 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.647 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.647 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.647 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.647 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.647 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.647 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.647 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-12T06:05:58.647 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.647 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.647 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.648 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:58.648 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:58.648 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.648 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.648 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.648 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:58.648 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.648 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.649 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.649 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to wake up and check device health", 2024-03-12T06:05:58.649 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.649 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.649 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.649 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_threshold": { 2024-03-12T06:05:58.649 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_threshold", 2024-03-12T06:05:58.649 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.649 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.650 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.650 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "7257600", 2024-03-12T06:05:58.650 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.650 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.650 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.650 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-12T06:05:58.650 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.650 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.651 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.651 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.651 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.651 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "diskprediction_local", 2024-03-12T06:05:58.651 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.651 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.651 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.652 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.652 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.652 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.652 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.652 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.652 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.652 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.652 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.652 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.653 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.653 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.653 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.653 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.653 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.653 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.653 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.653 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.654 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.654 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.654 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.654 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.654 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.654 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.654 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.655 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.655 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.655 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.655 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.655 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.655 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.655 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.655 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.655 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.656 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.656 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.656 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.656 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.656 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.656 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.656 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.656 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.657 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.657 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.657 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.657 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.657 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.657 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.657 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.657 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.657 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.658 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.658 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.658 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.658 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.658 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.658 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.658 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.658 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.658 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.659 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.659 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.659 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.659 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.659 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.659 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.659 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.659 INFO:teuthology.orchestra.run.smithi076.stdout: "predict_interval": { 2024-03-12T06:05:58.660 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "predict_interval", 2024-03-12T06:05:58.660 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.660 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.660 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.660 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "86400", 2024-03-12T06:05:58.660 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.660 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.660 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.661 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.661 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.661 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.661 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.661 INFO:teuthology.orchestra.run.smithi076.stdout: "predictor_model": { 2024-03-12T06:05:58.661 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "predictor_model", 2024-03-12T06:05:58.661 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.662 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.662 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.662 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "prophetstor", 2024-03-12T06:05:58.662 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.662 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.662 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.662 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.662 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.663 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.663 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.663 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:58.663 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:58.663 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.663 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.663 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.663 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:58.664 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.664 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.664 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.664 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.664 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.664 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.664 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.665 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.665 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.665 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "influx", 2024-03-12T06:05:58.665 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": false, 2024-03-12T06:05:58.665 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "influxdb python module not found", 2024-03-12T06:05:58.665 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.666 INFO:teuthology.orchestra.run.smithi076.stdout: "batch_size": { 2024-03-12T06:05:58.666 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "batch_size", 2024-03-12T06:05:58.666 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.666 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.666 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.666 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5000", 2024-03-12T06:05:58.667 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.667 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.667 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.667 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-12T06:05:58.667 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.667 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.667 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.667 INFO:teuthology.orchestra.run.smithi076.stdout: "database": { 2024-03-12T06:05:58.668 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "database", 2024-03-12T06:05:58.668 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.668 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.668 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.668 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "ceph", 2024-03-12T06:05:58.668 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.668 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.668 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.669 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-12T06:05:58.669 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.669 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.669 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.669 INFO:teuthology.orchestra.run.smithi076.stdout: "hostname": { 2024-03-12T06:05:58.669 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "hostname", 2024-03-12T06:05:58.669 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.669 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.670 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.670 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.670 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.670 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.670 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.670 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB server hostname", 2024-03-12T06:05:58.670 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.670 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.671 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.671 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:58.671 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:58.671 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.671 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.671 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.671 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "30", 2024-03-12T06:05:58.672 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "5", 2024-03-12T06:05:58.672 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.672 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.672 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-12T06:05:58.672 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.672 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.672 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.673 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.673 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.673 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.673 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.673 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.673 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.673 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.673 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.673 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.674 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.674 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.674 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.674 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.674 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.674 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.674 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.674 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.675 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.675 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.675 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.675 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.675 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.675 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.675 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.675 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.676 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.676 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.676 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.676 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.676 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.676 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.676 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.676 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.677 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.677 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.677 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.677 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.677 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.677 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.677 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.677 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.678 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.678 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.678 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.678 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.678 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.678 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.678 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.678 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.679 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.679 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.679 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.679 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.679 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.679 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.679 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.680 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.680 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.680 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.680 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.680 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.680 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.680 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.680 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.680 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: "password": { 2024-03-12T06:05:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "password", 2024-03-12T06:05:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.681 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "password of InfluxDB server user", 2024-03-12T06:05:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "port": { 2024-03-12T06:05:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "port", 2024-03-12T06:05:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8086", 2024-03-12T06:05:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.683 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB server port", 2024-03-12T06:05:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl": { 2024-03-12T06:05:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl", 2024-03-12T06:05:58.684 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.685 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.685 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.685 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "false", 2024-03-12T06:05:58.685 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.685 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.685 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.685 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-12T06:05:58.685 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.686 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.686 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.686 INFO:teuthology.orchestra.run.smithi076.stdout: "threads": { 2024-03-12T06:05:58.686 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "threads", 2024-03-12T06:05:58.686 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.686 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.687 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.687 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-12T06:05:58.687 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-12T06:05:58.687 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "32", 2024-03-12T06:05:58.687 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.687 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-12T06:05:58.687 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.687 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.688 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.688 INFO:teuthology.orchestra.run.smithi076.stdout: "username": { 2024-03-12T06:05:58.688 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "username", 2024-03-12T06:05:58.688 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.688 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.688 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.688 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "username of InfluxDB server user", 2024-03-12T06:05:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "verify_ssl": { 2024-03-12T06:05:58.689 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "verify_ssl", 2024-03-12T06:05:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "true", 2024-03-12T06:05:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.690 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-12T06:05:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.691 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "insights", 2024-03-12T06:05:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.692 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.693 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.695 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.696 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.696 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.696 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.696 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.696 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.696 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.696 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.697 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.697 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.697 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.697 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.697 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.697 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.697 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.697 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.698 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.698 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.698 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.698 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.698 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.698 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.698 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.698 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.699 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.699 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.699 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.699 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.699 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.699 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.699 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.699 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.700 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "iostat", 2024-03-12T06:05:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.701 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.702 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.703 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.704 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.705 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.706 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.707 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.708 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.709 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "k8sevents", 2024-03-12T06:05:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_event_retention_days": { 2024-03-12T06:05:58.710 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ceph_event_retention_days", 2024-03-12T06:05:58.711 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.711 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.711 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.711 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "7", 2024-03-12T06:05:58.711 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.711 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.711 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.711 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-12T06:05:58.712 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.712 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.712 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.712 INFO:teuthology.orchestra.run.smithi076.stdout: "config_check_secs": { 2024-03-12T06:05:58.712 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_check_secs", 2024-03-12T06:05:58.712 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.712 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.713 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.713 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:58.713 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "10", 2024-03-12T06:05:58.713 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.713 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.713 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-12T06:05:58.713 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.713 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.713 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.714 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.714 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.714 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.714 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.714 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.714 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.714 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.715 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.715 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.715 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.715 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.715 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.715 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.715 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.715 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.716 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.716 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.716 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.716 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.716 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.716 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.716 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.716 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.717 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.717 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.717 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.717 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.717 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.717 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.717 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.717 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.717 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.718 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.718 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.718 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.718 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.718 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.718 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.719 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.719 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.719 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.719 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.719 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.719 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.719 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.719 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.719 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.720 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.720 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.720 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.720 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.720 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.720 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.720 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.721 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.721 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.721 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.721 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.721 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.721 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.721 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.721 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.722 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.722 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.722 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.722 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.722 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.722 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.723 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "localpool", 2024-03-12T06:05:58.723 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.723 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.723 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.723 INFO:teuthology.orchestra.run.smithi076.stdout: "failure_domain": { 2024-03-12T06:05:58.723 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "failure_domain", 2024-03-12T06:05:58.723 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.723 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "host", 2024-03-12T06:05:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "failure domain for any created local pool", 2024-03-12T06:05:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-12T06:05:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.724 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.725 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.725 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.725 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.725 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.725 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.725 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.725 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.726 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.726 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.726 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.726 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.726 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.726 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.726 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.727 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.727 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.727 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.727 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.727 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.727 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.727 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.728 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.728 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.728 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.728 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.728 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.728 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.728 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.729 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.729 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.729 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.729 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.729 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.729 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.729 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.730 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.730 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.730 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.730 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.730 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.730 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.730 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.731 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.731 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.731 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.731 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.731 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.731 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.731 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.731 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.732 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.732 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.732 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.732 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.732 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.732 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.732 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.732 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.733 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.733 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.733 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.733 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.733 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.733 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.733 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.733 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.734 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": { 2024-03-12T06:05:58.734 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "min_size", 2024-03-12T06:05:58.734 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.734 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.734 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.734 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.734 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.735 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.735 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.735 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default min_size for any created local pool", 2024-03-12T06:05:58.735 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-12T06:05:58.735 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.735 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.735 INFO:teuthology.orchestra.run.smithi076.stdout: "num_rep": { 2024-03-12T06:05:58.735 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "num_rep", 2024-03-12T06:05:58.736 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.736 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.736 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.736 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3", 2024-03-12T06:05:58.736 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.736 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.736 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.736 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default replica count for any created local pool", 2024-03-12T06:05:58.736 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.736 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.737 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.737 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": { 2024-03-12T06:05:58.737 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pg_num", 2024-03-12T06:05:58.737 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.737 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.737 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.737 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "128", 2024-03-12T06:05:58.737 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.737 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.738 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.738 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default pg_num for any created local pool", 2024-03-12T06:05:58.738 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.738 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.738 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.738 INFO:teuthology.orchestra.run.smithi076.stdout: "prefix": { 2024-03-12T06:05:58.738 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prefix", 2024-03-12T06:05:58.739 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.739 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.739 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.739 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.739 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.739 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.739 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.739 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "name prefix for any created local pool", 2024-03-12T06:05:58.739 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.739 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.740 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.740 INFO:teuthology.orchestra.run.smithi076.stdout: "subtree": { 2024-03-12T06:05:58.740 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "subtree", 2024-03-12T06:05:58.740 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.740 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.740 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.740 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "rack", 2024-03-12T06:05:58.740 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.741 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.741 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.741 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-12T06:05:58.741 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-12T06:05:58.741 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.741 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.741 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.741 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.742 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mds_autoscaler", 2024-03-12T06:05:58.742 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.742 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.742 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.742 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.742 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.742 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.742 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.743 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.743 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.743 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.743 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.743 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.743 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.743 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.744 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.744 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.744 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.744 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.744 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.744 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.744 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.744 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.744 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.745 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.745 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.745 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.745 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.745 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.745 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.746 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.746 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.746 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.746 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.746 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.746 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.746 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.747 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.747 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.747 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.747 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.747 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.748 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.748 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.748 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.748 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.748 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.748 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.748 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.748 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.749 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.749 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.749 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.749 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.749 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.749 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.749 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.750 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.750 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.750 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.750 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.750 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.750 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.750 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.750 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.750 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.751 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.751 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.751 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.751 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.751 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.751 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mirroring", 2024-03-12T06:05:58.752 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.752 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.752 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.752 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.752 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.752 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.752 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.752 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.753 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.753 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.753 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.753 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.753 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.753 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.753 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.753 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.754 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.754 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.754 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.754 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.754 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.754 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.754 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.754 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.755 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.755 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.755 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.755 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.755 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.755 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.755 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.755 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.755 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.755 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.756 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.756 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.756 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.756 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.756 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.756 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.756 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.757 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.757 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.757 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.757 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.757 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.757 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.757 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.757 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.757 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.758 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.758 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.758 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.758 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.758 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.758 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.758 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.758 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.759 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.759 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.759 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.759 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.759 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.759 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.759 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.760 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.760 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.760 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.760 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.760 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.760 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.761 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "nfs", 2024-03-12T06:05:58.761 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.761 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.761 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.761 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.761 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.761 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.762 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.762 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.762 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.762 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.762 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.762 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.762 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.762 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.763 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.763 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.763 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.763 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.763 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.763 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.764 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.764 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.764 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.764 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.764 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.764 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.764 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.764 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.765 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.765 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.765 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.765 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.765 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.765 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.765 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.765 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.766 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.766 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.766 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.766 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.766 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.766 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.766 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.767 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.767 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.767 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.767 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.767 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.767 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.767 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.767 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.767 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.768 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.768 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.768 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.768 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.768 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.768 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.768 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.768 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.768 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.769 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.769 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.769 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.769 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.769 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.769 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.769 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.769 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.769 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.769 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.770 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "orchestrator", 2024-03-12T06:05:58.770 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.770 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.770 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.770 INFO:teuthology.orchestra.run.smithi076.stdout: "fail_fs": { 2024-03-12T06:05:58.770 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "fail_fs", 2024-03-12T06:05:58.771 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.771 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.771 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.771 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.771 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.771 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.771 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.771 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-12T06:05:58.771 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.772 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.772 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.772 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.772 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.772 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.772 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.772 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.772 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.773 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.773 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.773 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.773 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.773 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.773 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.773 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.773 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.773 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.774 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.774 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.774 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.774 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.774 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.774 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.775 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.775 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.775 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.775 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.775 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.775 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.775 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.775 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.776 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.776 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.776 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.776 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.776 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.776 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.776 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.777 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.777 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.777 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.777 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.777 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.777 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.777 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.777 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.778 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.778 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.778 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.778 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.778 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.778 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.778 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.778 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.778 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.779 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.779 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.779 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.779 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.779 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.779 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.779 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.779 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.780 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.780 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.780 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.780 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.780 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.780 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator": { 2024-03-12T06:05:58.780 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "orchestrator", 2024-03-12T06:05:58.781 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.781 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.781 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.781 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.781 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.781 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.781 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.781 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm", 2024-03-12T06:05:58.782 INFO:teuthology.orchestra.run.smithi076.stdout: "rook", 2024-03-12T06:05:58.782 INFO:teuthology.orchestra.run.smithi076.stdout: "test_orchestrator" 2024-03-12T06:05:58.782 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.782 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Orchestrator backend", 2024-03-12T06:05:58.782 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.782 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.782 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.783 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.783 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.783 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "osd_perf_query", 2024-03-12T06:05:58.783 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.783 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.783 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.783 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.784 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.784 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.784 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.784 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.784 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.784 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.784 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.784 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.784 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.785 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.785 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.785 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.785 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.785 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.785 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.785 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.785 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.785 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.786 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.786 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.786 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.786 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.786 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.786 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.786 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.786 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.787 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.787 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.787 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.787 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.787 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.787 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.787 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.788 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.788 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.788 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.788 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.788 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.788 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.788 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.788 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.788 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.789 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.789 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.789 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.789 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.789 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.789 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.789 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.789 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.790 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.790 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.790 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.790 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.790 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.790 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.790 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.791 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.791 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.791 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.791 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.791 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.791 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.791 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.791 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.792 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.792 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.792 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "osd_support", 2024-03-12T06:05:58.792 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.792 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.792 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.793 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.793 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.793 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.793 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.793 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.793 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.793 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.793 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.794 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.794 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.794 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.794 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.794 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.794 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.794 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.794 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.795 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.795 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.795 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.795 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.795 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.795 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.795 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.795 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.796 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.796 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.796 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.796 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.796 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.796 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.796 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.796 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.797 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.797 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.797 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.797 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.797 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.797 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.797 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.797 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.798 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.798 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.798 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.798 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.798 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.798 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.798 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.798 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.799 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.799 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.799 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.799 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.799 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.799 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.799 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.800 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.800 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.800 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.800 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.800 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.800 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.800 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.800 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.800 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.801 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.801 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.801 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pg_autoscaler", 2024-03-12T06:05:58.801 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.802 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.802 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.802 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.802 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.802 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.802 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.802 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.802 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.803 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.803 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.803 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.803 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.803 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.803 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.803 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.803 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.804 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.804 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.804 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.804 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.804 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.804 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.804 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.805 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.805 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.805 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.805 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.805 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.805 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.805 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.805 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.805 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.806 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.806 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.806 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.806 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.806 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.806 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.807 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.807 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.807 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.807 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.807 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.807 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.807 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.807 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.808 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.808 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.808 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.808 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.808 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.808 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.808 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.808 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.808 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.809 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.809 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.809 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.809 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.809 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.809 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.809 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.809 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.809 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.810 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.810 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.810 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.810 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.810 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:58.810 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:58.810 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.811 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.811 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.811 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:58.811 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.811 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.811 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.811 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.811 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.811 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.812 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.812 INFO:teuthology.orchestra.run.smithi076.stdout: "threshold": { 2024-03-12T06:05:58.812 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "threshold", 2024-03-12T06:05:58.812 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:58.812 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.812 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.812 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3.0", 2024-03-12T06:05:58.813 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1.0", 2024-03-12T06:05:58.813 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.813 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.813 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "scaling threshold", 2024-03-12T06:05:58.813 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-12T06:05:58.813 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.813 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.813 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.814 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.814 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "progress", 2024-03-12T06:05:58.814 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.814 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.814 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.814 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_pg_recovery_event": { 2024-03-12T06:05:58.815 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_pg_recovery_event", 2024-03-12T06:05:58.815 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.815 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.815 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.815 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.815 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.815 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.815 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.815 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-12T06:05:58.816 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.816 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.816 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.816 INFO:teuthology.orchestra.run.smithi076.stdout: "enabled": { 2024-03-12T06:05:58.816 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enabled", 2024-03-12T06:05:58.816 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.816 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.817 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.817 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.817 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.817 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.817 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.817 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.817 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.817 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.818 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.818 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.818 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.818 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.818 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.818 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.819 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.819 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.819 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.819 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.819 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.819 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.819 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.819 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.819 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.820 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.820 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.820 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.820 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.820 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.820 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.820 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.821 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.821 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.821 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.821 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.821 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.821 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.821 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.821 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.822 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.822 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.822 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.822 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.822 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.822 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.822 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.823 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.823 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.823 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.823 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.823 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.823 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.823 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.824 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.824 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.824 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.824 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.824 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.824 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.824 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.825 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.825 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.825 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.825 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.825 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.825 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.825 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.826 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.826 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.826 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.826 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.826 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.826 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.826 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.826 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.827 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.827 INFO:teuthology.orchestra.run.smithi076.stdout: "max_completed_events": { 2024-03-12T06:05:58.827 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_completed_events", 2024-03-12T06:05:58.827 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.827 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.827 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.827 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "50", 2024-03-12T06:05:58.827 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.828 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.828 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.828 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "number of past completed events to remember", 2024-03-12T06:05:58.828 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.828 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.828 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.828 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:58.828 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:58.828 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.829 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.829 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.829 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-12T06:05:58.829 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.829 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.829 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.829 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long the module is going to sleep", 2024-03-12T06:05:58.829 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.829 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.830 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.830 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.830 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.830 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prometheus", 2024-03-12T06:05:58.830 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.830 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.830 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.831 INFO:teuthology.orchestra.run.smithi076.stdout: "cache": { 2024-03-12T06:05:58.831 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cache", 2024-03-12T06:05:58.831 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.831 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.831 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.831 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.831 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.831 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.831 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.832 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.832 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.832 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.832 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.832 INFO:teuthology.orchestra.run.smithi076.stdout: "exclude_perf_counters": { 2024-03-12T06:05:58.832 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "exclude_perf_counters", 2024-03-12T06:05:58.832 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.832 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.833 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.833 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.833 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.833 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.833 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.833 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-12T06:05:58.833 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-12T06:05:58.833 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.834 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.834 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.834 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.834 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.834 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.834 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.834 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.834 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.835 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.835 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.835 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.835 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.835 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.835 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.835 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.835 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.835 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.836 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.836 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.836 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.836 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.836 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.836 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.837 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.837 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.837 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.837 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.837 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.837 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.837 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.837 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.837 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.838 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.838 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.838 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.838 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.838 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.838 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.838 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.838 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.839 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.839 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.839 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.839 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.839 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.839 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.839 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.839 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.840 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.840 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.840 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.840 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.840 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.840 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.840 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.841 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.841 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.841 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.841 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.841 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.841 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.841 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.842 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.842 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.842 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.842 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.842 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_stats_pools": { 2024-03-12T06:05:58.842 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_stats_pools", 2024-03-12T06:05:58.842 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.843 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.843 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.843 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.843 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.843 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.843 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.843 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.843 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.844 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.844 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.844 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-12T06:05:58.844 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-12T06:05:58.844 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.844 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.844 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.844 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "300", 2024-03-12T06:05:58.845 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.845 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.845 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.845 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.845 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.845 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.845 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.846 INFO:teuthology.orchestra.run.smithi076.stdout: "scrape_interval": { 2024-03-12T06:05:58.846 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "scrape_interval", 2024-03-12T06:05:58.846 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:58.846 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.846 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.846 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15.0", 2024-03-12T06:05:58.846 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.846 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.847 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.847 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.847 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.847 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.847 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.847 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-12T06:05:58.847 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-12T06:05:58.848 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.848 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.848 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.848 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "::", 2024-03-12T06:05:58.848 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.848 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.848 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.848 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-12T06:05:58.849 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.849 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.849 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.849 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-12T06:05:58.849 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-12T06:05:58.849 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.849 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.849 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.850 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "9283", 2024-03-12T06:05:58.850 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.850 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.850 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.850 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-12T06:05:58.850 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.850 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.850 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.851 INFO:teuthology.orchestra.run.smithi076.stdout: "stale_cache_strategy": { 2024-03-12T06:05:58.851 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "stale_cache_strategy", 2024-03-12T06:05:58.851 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.851 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.851 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.851 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "log", 2024-03-12T06:05:58.852 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.852 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.852 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.852 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.852 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.852 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.852 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.853 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_behaviour": { 2024-03-12T06:05:58.853 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_behaviour", 2024-03-12T06:05:58.853 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.853 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.853 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.853 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "default", 2024-03-12T06:05:58.853 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.853 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.853 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.854 INFO:teuthology.orchestra.run.smithi076.stdout: "default", 2024-03-12T06:05:58.854 INFO:teuthology.orchestra.run.smithi076.stdout: "error" 2024-03-12T06:05:58.854 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.854 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.854 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.854 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.854 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.855 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_error_status_code": { 2024-03-12T06:05:58.855 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_error_status_code", 2024-03-12T06:05:58.855 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.855 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.855 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.855 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "500", 2024-03-12T06:05:58.855 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "400", 2024-03-12T06:05:58.856 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "599", 2024-03-12T06:05:58.856 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.856 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.856 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.856 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.856 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.856 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.857 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.857 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_support", 2024-03-12T06:05:58.857 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.857 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.857 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.857 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.858 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.858 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.858 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.858 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.858 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.858 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.858 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.858 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.859 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.859 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.859 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.859 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.859 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.859 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.859 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.859 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.860 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.860 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.860 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.860 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.860 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.860 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.861 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.861 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.861 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.861 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.861 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.861 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.861 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.861 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.861 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.862 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.862 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.862 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.862 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.862 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.862 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.863 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.863 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.863 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.863 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.863 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.863 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.863 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.863 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.864 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.864 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.864 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.864 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.864 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.864 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.864 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.864 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.865 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.865 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.865 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.865 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.865 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.865 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.865 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.865 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.866 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.866 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.866 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.866 INFO:teuthology.orchestra.run.smithi076.stdout: "max_concurrent_snap_create": { 2024-03-12T06:05:58.866 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_concurrent_snap_create", 2024-03-12T06:05:58.866 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.866 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.867 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.867 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:58.867 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.867 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.867 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.867 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.867 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.867 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.868 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.868 INFO:teuthology.orchestra.run.smithi076.stdout: "mirror_snapshot_schedule": { 2024-03-12T06:05:58.868 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mirror_snapshot_schedule", 2024-03-12T06:05:58.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.868 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.868 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.868 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.868 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.869 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.869 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.869 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.869 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.869 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.869 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.869 INFO:teuthology.orchestra.run.smithi076.stdout: "trash_purge_schedule": { 2024-03-12T06:05:58.870 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "trash_purge_schedule", 2024-03-12T06:05:58.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.870 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.870 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.870 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.870 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.870 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.871 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.871 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.871 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.871 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.871 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.871 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.871 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.872 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "restful", 2024-03-12T06:05:58.872 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.872 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.872 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.872 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_auth": { 2024-03-12T06:05:58.872 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enable_auth", 2024-03-12T06:05:58.873 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.873 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.873 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.873 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.873 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.873 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.873 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.873 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.874 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.874 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.874 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.874 INFO:teuthology.orchestra.run.smithi076.stdout: "key_file": { 2024-03-12T06:05:58.874 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "key_file", 2024-03-12T06:05:58.874 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.874 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.875 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.875 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.875 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.875 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.875 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.875 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.875 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.875 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.875 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.876 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.876 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.876 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.876 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.876 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.876 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.877 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.877 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.877 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.877 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.877 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.877 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.877 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.877 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.878 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.878 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.878 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.878 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.878 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.878 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.879 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.879 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.879 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.879 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.879 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.879 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.879 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.879 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.880 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.880 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.880 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.880 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.880 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.880 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.880 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.881 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.881 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.881 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.881 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.881 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.881 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.881 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.881 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.882 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.882 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.882 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.882 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.882 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.882 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.882 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.882 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.883 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.883 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.883 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.883 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.883 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.883 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.883 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.884 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.884 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.884 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.884 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.884 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.884 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.884 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.885 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.885 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-12T06:05:58.885 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-12T06:05:58.885 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.885 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.885 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.886 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.886 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.886 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.886 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.886 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.886 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.886 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.886 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.887 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-12T06:05:58.887 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-12T06:05:58.887 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.887 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.887 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.887 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.887 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.888 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.888 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.888 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.888 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.888 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.888 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.888 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.888 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.889 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rgw", 2024-03-12T06:05:58.889 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.889 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.889 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.889 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.889 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.889 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.890 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.890 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.890 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.890 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.890 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.890 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.890 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.890 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.891 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.891 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.891 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.891 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.891 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.891 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.891 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.891 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.891 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.892 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.892 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.892 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.892 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.892 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.892 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.892 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.892 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.893 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.893 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.893 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.893 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.893 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.893 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.893 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.894 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.894 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.894 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.894 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.894 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.894 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.894 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.894 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.894 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.895 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.895 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.895 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.895 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.895 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.895 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.895 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.895 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.896 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.896 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.896 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.896 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.896 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.896 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.896 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.896 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.896 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.897 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.897 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.897 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.897 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.897 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.897 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.897 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.898 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rook", 2024-03-12T06:05:58.898 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.898 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.898 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.898 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.898 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.898 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.898 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.899 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.899 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.899 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.899 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.899 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.899 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.899 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.899 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.899 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.899 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.900 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.900 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.900 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.900 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.900 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.900 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.900 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.900 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.901 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.901 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.901 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.901 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.901 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.901 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.901 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.901 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.901 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.901 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.902 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.902 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.902 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.902 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.902 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.902 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.902 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.903 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.904 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.904 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.904 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.904 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.904 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.904 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.905 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.905 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.905 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.905 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.905 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.905 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.905 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.905 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.906 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.906 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.906 INFO:teuthology.orchestra.run.smithi076.stdout: "storage_class": { 2024-03-12T06:05:58.906 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "storage_class", 2024-03-12T06:05:58.906 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.906 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.906 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.907 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "local", 2024-03-12T06:05:58.907 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.907 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.907 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.907 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-12T06:05:58.907 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.907 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.907 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.907 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.907 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.908 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "selftest", 2024-03-12T06:05:58.908 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.908 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.908 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.908 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.908 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.908 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.909 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.909 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.909 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.909 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.909 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.909 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.909 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.909 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.910 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.910 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.910 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.910 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.910 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.910 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.910 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.910 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.911 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.911 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.911 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.911 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.911 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.911 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.911 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.912 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.912 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.912 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.912 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.912 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.912 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.912 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.913 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.913 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.913 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.913 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.913 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.913 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.913 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.913 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.914 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.914 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.914 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.914 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.914 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.914 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.914 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.914 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.914 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.915 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.915 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.915 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.915 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.915 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.915 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.915 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.915 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.916 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.916 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.916 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.916 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.916 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.916 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.916 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.917 INFO:teuthology.orchestra.run.smithi076.stdout: "roption1": { 2024-03-12T06:05:58.917 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "roption1", 2024-03-12T06:05:58.917 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.917 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.917 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.917 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.917 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.917 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.917 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.917 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.918 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.918 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.918 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.918 INFO:teuthology.orchestra.run.smithi076.stdout: "roption2": { 2024-03-12T06:05:58.918 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "roption2", 2024-03-12T06:05:58.918 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.918 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.919 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.919 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "xyz", 2024-03-12T06:05:58.919 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.919 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.919 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.919 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.919 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.919 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.919 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.920 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption1": { 2024-03-12T06:05:58.920 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption1", 2024-03-12T06:05:58.920 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.920 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.920 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.920 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.920 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.920 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.920 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.921 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.921 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.921 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.921 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.921 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption2": { 2024-03-12T06:05:58.921 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption2", 2024-03-12T06:05:58.921 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.921 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.922 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.922 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.922 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.922 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.922 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.922 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.922 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.923 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.923 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.923 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption3": { 2024-03-12T06:05:58.923 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption3", 2024-03-12T06:05:58.923 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:58.923 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.923 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.923 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.924 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.924 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.924 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.924 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.924 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.924 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.924 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.925 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption4": { 2024-03-12T06:05:58.925 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption4", 2024-03-12T06:05:58.925 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.925 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.925 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.925 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.925 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.925 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.925 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.926 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.926 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.926 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.926 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.926 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption5": { 2024-03-12T06:05:58.926 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption5", 2024-03-12T06:05:58.926 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.927 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.927 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.927 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.927 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.927 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.927 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.927 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.927 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.928 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.928 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.928 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption6": { 2024-03-12T06:05:58.928 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption6", 2024-03-12T06:05:58.928 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.928 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.928 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.928 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.929 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.929 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.929 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.929 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.929 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.929 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.929 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.930 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption7": { 2024-03-12T06:05:58.930 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption7", 2024-03-12T06:05:58.930 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:58.930 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.930 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.930 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.930 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-12T06:05:58.931 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "42", 2024-03-12T06:05:58.931 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.931 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.931 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.931 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.931 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.931 INFO:teuthology.orchestra.run.smithi076.stdout: "testkey": { 2024-03-12T06:05:58.931 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testkey", 2024-03-12T06:05:58.932 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.932 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.932 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.932 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.932 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.932 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.932 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.932 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.933 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.933 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.933 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.933 INFO:teuthology.orchestra.run.smithi076.stdout: "testlkey": { 2024-03-12T06:05:58.933 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testlkey", 2024-03-12T06:05:58.933 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.933 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.934 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.934 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.934 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.934 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.934 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.934 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.934 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.934 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.935 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.935 INFO:teuthology.orchestra.run.smithi076.stdout: "testnewline": { 2024-03-12T06:05:58.935 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testnewline", 2024-03-12T06:05:58.935 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.935 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.935 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.935 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.936 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.936 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.936 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.936 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.936 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.936 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.936 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.936 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.937 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snap_schedule", 2024-03-12T06:05:58.937 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.937 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.937 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.937 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_m_granularity": { 2024-03-12T06:05:58.937 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_m_granularity", 2024-03-12T06:05:58.938 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.938 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.938 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.938 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.938 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.938 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.938 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.938 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow minute scheduled snapshots", 2024-03-12T06:05:58.939 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.939 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.939 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.939 INFO:teuthology.orchestra.run.smithi076.stdout: "dump_on_update": { 2024-03-12T06:05:58.939 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "dump_on_update", 2024-03-12T06:05:58.939 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.939 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.939 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.940 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.940 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.940 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.940 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.940 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "dump database to debug log on update", 2024-03-12T06:05:58.940 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.940 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.940 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.941 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.941 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.941 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.941 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.941 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.941 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.941 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.941 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.941 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.942 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.942 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.942 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.942 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.942 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.942 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.942 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.942 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.943 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.943 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.943 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.943 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.943 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.943 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.943 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.943 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.944 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.944 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.944 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.944 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.944 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.944 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.944 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.944 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.945 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.945 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.945 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.945 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.945 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.945 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.945 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.945 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.945 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.946 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.946 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.946 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.946 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.946 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.946 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.946 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.947 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.947 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.947 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.947 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.947 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.947 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.947 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.948 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.948 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.948 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.948 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.948 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.948 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.948 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.948 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.949 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.949 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.949 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.949 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.949 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "stats", 2024-03-12T06:05:58.949 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.950 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.950 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.950 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.950 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.950 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.950 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.950 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.951 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.951 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.951 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.951 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.951 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.951 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.951 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.951 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.952 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.952 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.952 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.952 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.952 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.952 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.952 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.953 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.953 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.953 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.953 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.953 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.953 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.953 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.953 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.954 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.954 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.954 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.954 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.954 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.954 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.954 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.955 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.955 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.955 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.955 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.955 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.955 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.955 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.955 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.956 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.956 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.956 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.956 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.956 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.956 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.956 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.956 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.957 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.957 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.957 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.957 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.957 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.957 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.957 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.957 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.957 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.958 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.958 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.958 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.958 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.958 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.958 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.958 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.958 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.959 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "status", 2024-03-12T06:05:58.959 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.959 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.959 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.959 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.959 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.959 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.959 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.960 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.960 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.960 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.960 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.960 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.960 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.960 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.960 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.960 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.961 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.961 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.961 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.961 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.961 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.961 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.961 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.961 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.962 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.962 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.962 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.962 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.962 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.962 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.962 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.963 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.963 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.963 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.963 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.963 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.963 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.963 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.963 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.964 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.964 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.964 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.964 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.964 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.964 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.964 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.964 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.965 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.965 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.965 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.965 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.965 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.965 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.965 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.965 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.966 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.966 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.966 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.966 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.966 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.966 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.967 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.967 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.967 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.967 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.967 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.967 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.967 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.967 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.967 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.968 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.968 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "telegraf", 2024-03-12T06:05:58.968 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.968 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.968 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.968 INFO:teuthology.orchestra.run.smithi076.stdout: "address": { 2024-03-12T06:05:58.969 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "address", 2024-03-12T06:05:58.969 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.969 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.969 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.969 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-12T06:05:58.969 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.970 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.970 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.970 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.970 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.970 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.970 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.971 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:58.971 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:58.971 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:58.971 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.971 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.971 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15", 2024-03-12T06:05:58.971 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.972 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.972 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.972 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.972 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.972 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.972 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.973 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:58.973 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:58.973 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.973 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.973 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.973 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.973 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.973 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.973 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.974 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.974 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.974 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.974 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.974 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.974 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.974 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.974 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.975 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.975 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.975 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.975 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:58.975 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:58.975 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.975 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.975 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.976 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.976 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.976 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.976 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.976 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.976 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.976 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.976 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.977 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:58.977 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:58.977 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.977 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.977 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.977 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:58.977 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.978 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.978 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:58.978 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:58.978 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:58.978 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:58.978 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:58.978 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:58.979 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:58.979 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:58.979 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.979 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.979 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.979 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.979 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:58.979 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:58.980 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.980 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.980 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:58.980 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.980 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.980 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.980 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.981 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.981 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.981 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.981 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.981 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.981 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:58.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:58.982 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "telemetry", 2024-03-12T06:05:58.982 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.615+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3738249257 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efcf40a1ba0 msgr2=0x7efcf40a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:58.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.615+0000 7efd0dfe2640 1 --2- 172.21.15.76:0/3738249257 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efcf40a1ba0 0x7efcf40a4060 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efd080ff1a0 tx=0x7efcf000b390 comp rx=0 tx=0).stop 2024-03-12T06:05:58.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.615+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3738249257 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd08100280 msgr2=0x7efd081976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:05:58.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.615+0000 7efd0dfe2640 1 --2- 172.21.15.76:0/3738249257 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd08100280 0x7efd081976f0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7efcf800c920 tx=0x7efcf800cde0 comp rx=0 tx=0).stop 2024-03-12T06:05:58.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.615+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3738249257 shutdown_connections 2024-03-12T06:05:58.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.615+0000 7efd0dfe2640 1 --2- 172.21.15.76:0/3738249257 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efcf40a1ba0 0x7efcf40a4060 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:58.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.615+0000 7efd0dfe2640 1 --2- 172.21.15.76:0/3738249257 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd0819bfb0 0x7efd0819c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:58.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.615+0000 7efd0dfe2640 1 --2- 172.21.15.76:0/3738249257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd08105350 0x7efd08197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:58.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.615+0000 7efd0dfe2640 1 --2- 172.21.15.76:0/3738249257 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd08100280 0x7efd081976f0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:05:58.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.615+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3738249257 >> 172.21.15.76:0/3738249257 conn(0x7efd080fbb00 msgr2=0x7efd080fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:05:58.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.616+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3738249257 shutdown_connections 2024-03-12T06:05:58.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:05:57.616+0000 7efd0dfe2640 1 -- 172.21.15.76:0/3738249257 wait complete. 2024-03-12T06:05:58.986 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:58.986 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:58.986 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:58.986 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_basic": { 2024-03-12T06:05:58.987 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_basic", 2024-03-12T06:05:58.987 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.987 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.987 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.987 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.987 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.987 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.987 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.988 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-12T06:05:58.988 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.988 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.988 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.988 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_crash": { 2024-03-12T06:05:58.988 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_crash", 2024-03-12T06:05:58.988 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.988 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.988 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.989 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.989 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.989 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.989 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.989 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-12T06:05:58.989 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.989 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.989 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.990 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_device": { 2024-03-12T06:05:58.990 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_device", 2024-03-12T06:05:58.990 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.990 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.990 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.990 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:58.990 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.990 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.990 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.991 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-12T06:05:58.991 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.991 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.991 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.991 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_ident": { 2024-03-12T06:05:58.991 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_ident", 2024-03-12T06:05:58.991 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.991 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.992 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.992 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.992 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.992 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.992 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.992 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-12T06:05:58.992 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.992 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.993 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.993 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_perf": { 2024-03-12T06:05:58.993 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_perf", 2024-03-12T06:05:58.993 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.993 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.993 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.994 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:58.994 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.994 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.994 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.994 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-12T06:05:58.994 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.994 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.994 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.995 INFO:teuthology.orchestra.run.smithi076.stdout: "contact": { 2024-03-12T06:05:58.995 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "contact", 2024-03-12T06:05:58.995 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.995 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.995 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.995 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.995 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.995 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.995 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.996 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.996 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.996 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.996 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.996 INFO:teuthology.orchestra.run.smithi076.stdout: "description": { 2024-03-12T06:05:58.996 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "description", 2024-03-12T06:05:58.996 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.996 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.996 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.997 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:58.997 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.997 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.997 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.997 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.997 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.997 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.997 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.998 INFO:teuthology.orchestra.run.smithi076.stdout: "device_url": { 2024-03-12T06:05:58.998 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_url", 2024-03-12T06:05:58.998 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:58.998 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.998 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:58.998 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-12T06:05:58.998 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:58.998 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:58.998 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:58.999 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:58.999 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:58.999 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:58.999 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:58.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:58.999 INFO:teuthology.orchestra.run.smithi076.stdout: "enabled": { 2024-03-12T06:05:58.999 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enabled", 2024-03-12T06:05:58.999 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:58.999 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:58.999 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.000 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.000 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.000 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.000 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.000 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.000 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.000 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.000 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.001 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:59.001 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:59.001 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.001 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.001 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.001 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "24", 2024-03-12T06:05:59.001 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "8", 2024-03-12T06:05:59.001 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.002 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.002 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.002 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.002 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.002 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.002 INFO:teuthology.orchestra.run.smithi076.stdout: "last_opt_revision": { 2024-03-12T06:05:59.002 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "last_opt_revision", 2024-03-12T06:05:59.002 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.003 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.003 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.003 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1", 2024-03-12T06:05:59.003 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.003 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.003 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.003 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.003 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.003 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.004 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.004 INFO:teuthology.orchestra.run.smithi076.stdout: "leaderboard": { 2024-03-12T06:05:59.004 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "leaderboard", 2024-03-12T06:05:59.004 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.004 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.004 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.004 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.005 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.005 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.005 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.005 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.005 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.005 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.005 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.006 INFO:teuthology.orchestra.run.smithi076.stdout: "leaderboard_description": { 2024-03-12T06:05:59.006 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "leaderboard_description", 2024-03-12T06:05:59.006 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.006 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.006 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.006 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.006 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.006 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.006 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.007 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.007 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.007 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.007 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.007 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.007 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.007 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.008 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.008 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.008 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.008 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.008 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.008 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.008 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.008 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.008 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.009 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.009 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.009 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.009 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.009 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.009 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.009 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.009 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.010 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.010 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.010 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.010 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.010 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.010 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.010 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.010 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.011 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.011 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.011 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.011 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.011 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.011 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.011 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.011 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.012 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.012 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.012 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.012 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.012 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.012 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.012 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.012 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.013 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.013 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.013 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.013 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.013 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.013 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.013 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.013 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.013 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.014 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.014 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.014 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.014 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.014 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.014 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.015 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.015 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.015 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.015 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.015 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.015 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.015 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.015 INFO:teuthology.orchestra.run.smithi076.stdout: "organization": { 2024-03-12T06:05:59.016 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "organization", 2024-03-12T06:05:59.016 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.016 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.016 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.016 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.016 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.016 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.016 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.016 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.016 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.017 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.017 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.017 INFO:teuthology.orchestra.run.smithi076.stdout: "proxy": { 2024-03-12T06:05:59.017 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "proxy", 2024-03-12T06:05:59.017 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.017 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.017 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.017 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.018 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.018 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.018 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.018 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.018 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.018 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.018 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.018 INFO:teuthology.orchestra.run.smithi076.stdout: "url": { 2024-03-12T06:05:59.018 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "url", 2024-03-12T06:05:59.019 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.019 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.019 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.019 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-12T06:05:59.019 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.019 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.019 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.019 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.019 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.020 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.020 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.020 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.020 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.020 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "test_orchestrator", 2024-03-12T06:05:59.020 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.020 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.020 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.021 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.021 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.021 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.021 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.021 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.021 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.021 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.021 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.021 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.022 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.022 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.022 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.022 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.022 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.022 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.022 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.022 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.022 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.023 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.023 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.023 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.023 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.023 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.023 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.023 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.023 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.023 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.024 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.024 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.024 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.024 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.024 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.024 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.024 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.024 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.025 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.025 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.025 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.025 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.025 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.025 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.025 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.025 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.025 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.026 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.026 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.026 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.026 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.026 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.026 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.026 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.026 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.027 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.027 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.027 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.027 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.027 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.027 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.027 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.027 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.028 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.028 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.028 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.028 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.028 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.028 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.028 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.028 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.029 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "volumes", 2024-03-12T06:05:59.029 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.029 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.029 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.029 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.029 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.030 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.030 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.030 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.030 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.030 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.030 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.030 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.030 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.031 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.031 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.031 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.031 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.031 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.031 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.031 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.031 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.031 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.032 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.032 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.032 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.032 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.032 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.032 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.032 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.032 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.033 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.033 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.033 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.033 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.033 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.033 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.033 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.034 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.034 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.034 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.034 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.034 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.034 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.034 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.034 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.034 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.035 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.035 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.035 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.035 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.035 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.035 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.035 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.035 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.036 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.036 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.036 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.036 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.036 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.036 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.036 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.037 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.037 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.037 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.037 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.037 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.037 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.037 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.037 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.038 INFO:teuthology.orchestra.run.smithi076.stdout: "max_concurrent_clones": { 2024-03-12T06:05:59.038 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_concurrent_clones", 2024-03-12T06:05:59.038 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.038 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.038 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.038 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "4", 2024-03-12T06:05:59.038 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.039 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.039 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.039 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-12T06:05:59.039 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.039 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.039 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.039 INFO:teuthology.orchestra.run.smithi076.stdout: "periodic_async_work": { 2024-03-12T06:05:59.040 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "periodic_async_work", 2024-03-12T06:05:59.040 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.040 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.040 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.040 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.040 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.040 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.040 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.041 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Periodically check for async work", 2024-03-12T06:05:59.041 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.041 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.041 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.041 INFO:teuthology.orchestra.run.smithi076.stdout: "snapshot_clone_delay": { 2024-03-12T06:05:59.041 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snapshot_clone_delay", 2024-03-12T06:05:59.041 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.042 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.042 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.042 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:59.042 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.042 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.042 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.042 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-12T06:05:59.042 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.043 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.043 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.043 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.043 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.043 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix", 2024-03-12T06:05:59.043 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.044 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.044 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.044 INFO:teuthology.orchestra.run.smithi076.stdout: "discovery_interval": { 2024-03-12T06:05:59.044 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "discovery_interval", 2024-03-12T06:05:59.044 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:59.044 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.044 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.044 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "100", 2024-03-12T06:05:59.045 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.045 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.045 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.045 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.045 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.045 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.045 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.046 INFO:teuthology.orchestra.run.smithi076.stdout: "identifier": { 2024-03-12T06:05:59.046 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "identifier", 2024-03-12T06:05:59.046 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.046 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.046 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.046 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.046 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.047 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.047 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.047 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.047 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.047 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.047 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.047 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:59.047 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:59.048 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.048 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.048 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.048 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:59.048 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.048 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.048 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.048 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.048 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.049 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.050 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.050 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.050 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.050 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.050 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.050 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.051 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.051 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.051 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.051 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.051 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.051 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.051 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.051 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.052 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.052 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.052 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.052 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.052 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.052 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.052 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.053 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.053 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.053 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.053 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.053 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.053 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.053 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.054 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.054 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.054 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.054 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.054 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.054 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.054 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.054 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.055 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.055 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.055 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.055 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.055 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.055 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.055 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.055 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.055 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.055 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.056 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.056 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.056 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.056 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.056 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.056 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.056 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.057 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.057 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.057 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.057 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.057 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.057 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.057 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_host": { 2024-03-12T06:05:59.058 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_host", 2024-03-12T06:05:59.058 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.058 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.058 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.058 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.058 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.058 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.059 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.059 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.059 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.059 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.059 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.059 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_port": { 2024-03-12T06:05:59.059 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_port", 2024-03-12T06:05:59.059 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.060 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.060 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.060 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10051", 2024-03-12T06:05:59.060 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.060 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.060 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.060 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.060 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.061 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.061 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.061 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_sender": { 2024-03-12T06:05:59.061 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_sender", 2024-03-12T06:05:59.061 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.061 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.061 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.061 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-12T06:05:59.062 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.062 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.062 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.062 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.062 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.062 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.062 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.063 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.063 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.063 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.063 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-12T06:05:59.063 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.063 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.063 INFO:teuthology.orchestra.run.smithi076.stdout: "modules": [ 2024-03-12T06:05:59.063 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm", 2024-03-12T06:05:59.064 INFO:teuthology.orchestra.run.smithi076.stdout: "dashboard", 2024-03-12T06:05:59.064 INFO:teuthology.orchestra.run.smithi076.stdout: "iostat", 2024-03-12T06:05:59.064 INFO:teuthology.orchestra.run.smithi076.stdout: "nfs", 2024-03-12T06:05:59.064 INFO:teuthology.orchestra.run.smithi076.stdout: "restful", 2024-03-12T06:05:59.064 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_schedule" 2024-03-12T06:05:59.064 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.064 INFO:teuthology.orchestra.run.smithi076.stdout: "available_modules": [ 2024-03-12T06:05:59.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.065 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "alerts", 2024-03-12T06:05:59.065 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.065 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.065 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.065 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:59.065 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:59.065 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.066 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.066 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.066 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:59.066 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.066 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.066 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.066 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How frequently to reexamine health status", 2024-03-12T06:05:59.066 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.067 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.067 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.067 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.067 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.067 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.067 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.068 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.068 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.068 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.068 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.068 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.068 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.068 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.068 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.068 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.068 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.069 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.069 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.069 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.069 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.069 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.069 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.069 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.069 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.070 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.070 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.070 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.070 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.070 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.070 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.070 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.070 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.071 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.071 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.071 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.071 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.071 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.071 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.071 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.071 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.072 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.072 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.072 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.072 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.072 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.072 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.072 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.072 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.072 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.073 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.073 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.073 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.073 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.073 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.073 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.073 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.074 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.074 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.074 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.074 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.074 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.074 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.075 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.075 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.075 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.075 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.075 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.075 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.076 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_destination": { 2024-03-12T06:05:59.076 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_destination", 2024-03-12T06:05:59.076 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.076 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.076 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.076 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.076 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.076 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.077 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.077 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Email address to send alerts to", 2024-03-12T06:05:59.077 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.077 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.077 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.077 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_from_name": { 2024-03-12T06:05:59.077 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_from_name", 2024-03-12T06:05:59.077 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.078 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.078 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.078 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "Ceph", 2024-03-12T06:05:59.078 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.078 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.078 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.078 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Email From: name", 2024-03-12T06:05:59.079 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.079 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.079 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.079 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_host": { 2024-03-12T06:05:59.079 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_host", 2024-03-12T06:05:59.079 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.079 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.080 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.080 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.080 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.080 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.080 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.080 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP server", 2024-03-12T06:05:59.080 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.080 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.081 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.081 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_password": { 2024-03-12T06:05:59.081 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_password", 2024-03-12T06:05:59.081 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.081 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.081 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.081 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.081 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.082 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.082 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.082 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Password to authenticate with", 2024-03-12T06:05:59.082 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.082 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.082 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.083 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_port": { 2024-03-12T06:05:59.083 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_port", 2024-03-12T06:05:59.083 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.083 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.083 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.083 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "465", 2024-03-12T06:05:59.083 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.083 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.083 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.084 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP port", 2024-03-12T06:05:59.084 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.084 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.084 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.084 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_sender": { 2024-03-12T06:05:59.084 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_sender", 2024-03-12T06:05:59.084 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.084 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.085 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.085 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.085 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.085 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.085 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.085 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SMTP envelope sender", 2024-03-12T06:05:59.085 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.085 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.086 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.086 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_ssl": { 2024-03-12T06:05:59.086 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_ssl", 2024-03-12T06:05:59.086 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.086 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.086 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.087 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.087 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.087 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.087 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.087 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-12T06:05:59.087 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.087 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.088 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.088 INFO:teuthology.orchestra.run.smithi076.stdout: "smtp_user": { 2024-03-12T06:05:59.088 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "smtp_user", 2024-03-12T06:05:59.088 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.088 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.088 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.088 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.089 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.089 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.089 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.089 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "User to authenticate as", 2024-03-12T06:05:59.089 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.089 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.089 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.089 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.090 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.090 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "balancer", 2024-03-12T06:05:59.090 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.090 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.090 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.090 INFO:teuthology.orchestra.run.smithi076.stdout: "active": { 2024-03-12T06:05:59.090 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "active", 2024-03-12T06:05:59.091 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.091 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.091 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.091 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.091 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.091 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.091 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.091 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "automatically balance PGs across cluster", 2024-03-12T06:05:59.091 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.092 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.092 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.092 INFO:teuthology.orchestra.run.smithi076.stdout: "begin_time": { 2024-03-12T06:05:59.092 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "begin_time", 2024-03-12T06:05:59.092 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.092 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.092 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.092 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0000", 2024-03-12T06:05:59.093 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.093 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.093 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.093 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "beginning time of day to automatically balance", 2024-03-12T06:05:59.093 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-12T06:05:59.093 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.093 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.093 INFO:teuthology.orchestra.run.smithi076.stdout: "begin_weekday": { 2024-03-12T06:05:59.094 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "begin_weekday", 2024-03-12T06:05:59.094 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:59.094 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.094 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.094 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:59.094 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0", 2024-03-12T06:05:59.094 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "6", 2024-03-12T06:05:59.094 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.094 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-12T06:05:59.095 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-12T06:05:59.095 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.095 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.095 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_max_iterations": { 2024-03-12T06:05:59.095 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_max_iterations", 2024-03-12T06:05:59.095 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:59.095 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.095 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.096 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "25", 2024-03-12T06:05:59.096 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-12T06:05:59.096 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "250", 2024-03-12T06:05:59.096 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.096 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-12T06:05:59.096 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.096 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.096 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.096 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_metrics": { 2024-03-12T06:05:59.097 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_metrics", 2024-03-12T06:05:59.097 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.097 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.097 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.097 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "pgs,objects,bytes", 2024-03-12T06:05:59.097 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.097 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.097 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.097 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-12T06:05:59.098 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-12T06:05:59.098 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.098 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.098 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_compat_step": { 2024-03-12T06:05:59.098 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crush_compat_step", 2024-03-12T06:05:59.098 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:59.098 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.098 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.099 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0.5", 2024-03-12T06:05:59.099 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0.001", 2024-03-12T06:05:59.099 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "0.999", 2024-03-12T06:05:59.099 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.099 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "aggressiveness of optimization", 2024-03-12T06:05:59.099 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-12T06:05:59.099 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.099 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.100 INFO:teuthology.orchestra.run.smithi076.stdout: "end_time": { 2024-03-12T06:05:59.100 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "end_time", 2024-03-12T06:05:59.100 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.100 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.100 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.100 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "2359", 2024-03-12T06:05:59.100 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.100 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.101 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.101 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "ending time of day to automatically balance", 2024-03-12T06:05:59.101 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-12T06:05:59.101 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.101 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.101 INFO:teuthology.orchestra.run.smithi076.stdout: "end_weekday": { 2024-03-12T06:05:59.102 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "end_weekday", 2024-03-12T06:05:59.102 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:59.102 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.102 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.102 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:59.102 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "0", 2024-03-12T06:05:59.102 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "6", 2024-03-12T06:05:59.103 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.103 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-12T06:05:59.103 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-12T06:05:59.103 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.103 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.103 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.103 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.103 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.104 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.104 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.104 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.104 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.104 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.104 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.104 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.104 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.104 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.105 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.105 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.105 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.105 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.105 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.105 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.105 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.105 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.105 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.106 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.106 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.106 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.106 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.106 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.106 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.106 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.106 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.106 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.107 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.107 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.107 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.107 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.107 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.107 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.107 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.107 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.108 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.108 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.108 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.108 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.108 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.108 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.108 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.108 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.108 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.109 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.109 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.109 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.109 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.109 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.109 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.109 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.109 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.110 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.110 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.110 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.110 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.110 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.110 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.110 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.110 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.110 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.111 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.111 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.111 INFO:teuthology.orchestra.run.smithi076.stdout: "min_score": { 2024-03-12T06:05:59.111 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "min_score", 2024-03-12T06:05:59.111 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:59.111 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.111 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.112 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:59.112 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.112 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.112 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.112 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-12T06:05:59.112 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.112 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.112 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.113 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": { 2024-03-12T06:05:59.113 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mode", 2024-03-12T06:05:59.113 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.113 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.113 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.113 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "upmap", 2024-03-12T06:05:59.113 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.114 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.114 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.114 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-compat", 2024-03-12T06:05:59.114 INFO:teuthology.orchestra.run.smithi076.stdout: "none", 2024-03-12T06:05:59.114 INFO:teuthology.orchestra.run.smithi076.stdout: "read", 2024-03-12T06:05:59.114 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap", 2024-03-12T06:05:59.114 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap-read" 2024-03-12T06:05:59.114 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.114 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Balancer mode", 2024-03-12T06:05:59.115 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.115 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.115 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.115 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_ids": { 2024-03-12T06:05:59.115 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pool_ids", 2024-03-12T06:05:59.115 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.115 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.116 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.116 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.116 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.116 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.116 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.116 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-12T06:05:59.116 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.116 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.116 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.117 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:59.117 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:59.117 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.117 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.117 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.117 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:59.117 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.118 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.118 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.118 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-12T06:05:59.118 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.118 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.118 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.118 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap_max_deviation": { 2024-03-12T06:05:59.118 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "upmap_max_deviation", 2024-03-12T06:05:59.119 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.119 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.119 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.119 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-12T06:05:59.119 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-12T06:05:59.119 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.119 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.119 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-12T06:05:59.120 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-12T06:05:59.120 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.120 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.120 INFO:teuthology.orchestra.run.smithi076.stdout: "upmap_max_optimizations": { 2024-03-12T06:05:59.120 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "upmap_max_optimizations", 2024-03-12T06:05:59.120 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:59.120 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.120 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.121 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:59.121 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.121 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.121 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.121 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-12T06:05:59.121 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.121 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.121 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.121 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.122 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.122 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.122 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cephadm", 2024-03-12T06:05:59.122 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.122 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.122 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.122 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_down_multiplier": { 2024-03-12T06:05:59.123 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_down_multiplier", 2024-03-12T06:05:59.123 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:59.123 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.123 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.123 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3.0", 2024-03-12T06:05:59.123 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.123 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.124 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.124 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-12T06:05:59.124 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.124 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.124 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.124 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_refresh_rate": { 2024-03-12T06:05:59.124 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_refresh_rate", 2024-03-12T06:05:59.125 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.125 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.125 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.125 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "20", 2024-03-12T06:05:59.125 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.125 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.125 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.125 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-12T06:05:59.126 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.126 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.126 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.126 INFO:teuthology.orchestra.run.smithi076.stdout: "agent_starting_port": { 2024-03-12T06:05:59.126 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "agent_starting_port", 2024-03-12T06:05:59.126 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.126 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.127 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.127 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "4721", 2024-03-12T06:05:59.127 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.127 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.127 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.127 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-12T06:05:59.127 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.127 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.127 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.128 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_ptrace": { 2024-03-12T06:05:59.128 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_ptrace", 2024-03-12T06:05:59.128 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.128 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.128 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.128 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.128 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.128 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.129 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.129 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-12T06:05:59.129 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-12T06:05:59.129 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.129 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.129 INFO:teuthology.orchestra.run.smithi076.stdout: "autotune_interval": { 2024-03-12T06:05:59.129 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "autotune_interval", 2024-03-12T06:05:59.129 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.130 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.130 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.130 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:59.130 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.130 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.130 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.130 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-12T06:05:59.130 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.131 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.131 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.131 INFO:teuthology.orchestra.run.smithi076.stdout: "autotune_memory_target_ratio": { 2024-03-12T06:05:59.131 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "autotune_memory_target_ratio", 2024-03-12T06:05:59.131 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:59.131 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.132 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.132 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0.7", 2024-03-12T06:05:59.132 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.132 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.132 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.132 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-12T06:05:59.132 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.132 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.133 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.133 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm_log_destination": { 2024-03-12T06:05:59.133 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cephadm_log_destination", 2024-03-12T06:05:59.133 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.133 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.133 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.133 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.134 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.134 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.134 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.134 INFO:teuthology.orchestra.run.smithi076.stdout: "file", 2024-03-12T06:05:59.134 INFO:teuthology.orchestra.run.smithi076.stdout: "file,syslog", 2024-03-12T06:05:59.134 INFO:teuthology.orchestra.run.smithi076.stdout: "syslog" 2024-03-12T06:05:59.134 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.135 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-12T06:05:59.135 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.135 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.135 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.135 INFO:teuthology.orchestra.run.smithi076.stdout: "cgroups_split": { 2024-03-12T06:05:59.135 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cgroups_split", 2024-03-12T06:05:59.135 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.136 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.136 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.136 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.136 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.136 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.136 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.136 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-12T06:05:59.136 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.137 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.137 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.137 INFO:teuthology.orchestra.run.smithi076.stdout: "config_checks_enabled": { 2024-03-12T06:05:59.137 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_checks_enabled", 2024-03-12T06:05:59.137 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.137 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.137 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.138 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.138 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.138 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.138 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.138 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-12T06:05:59.138 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.138 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.138 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.139 INFO:teuthology.orchestra.run.smithi076.stdout: "config_dashboard": { 2024-03-12T06:05:59.139 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_dashboard", 2024-03-12T06:05:59.139 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.139 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.139 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.139 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.139 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.140 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.140 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.140 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-12T06:05:59.140 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.140 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.140 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.141 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_alertmanager": { 2024-03-12T06:05:59.141 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_alertmanager", 2024-03-12T06:05:59.141 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.141 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.141 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.141 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-12T06:05:59.141 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.141 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.142 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.142 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-12T06:05:59.142 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.142 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.142 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.142 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_base": { 2024-03-12T06:05:59.142 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_base", 2024-03-12T06:05:59.143 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.143 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.143 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.143 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-12T06:05:59.143 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.143 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.143 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.143 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Container image name, without the tag", 2024-03-12T06:05:59.144 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.144 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.144 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.144 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_elasticsearch": { 2024-03-12T06:05:59.144 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_elasticsearch", 2024-03-12T06:05:59.144 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.144 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.144 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.145 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-12T06:05:59.145 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.145 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.145 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.145 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "elasticsearch container image", 2024-03-12T06:05:59.145 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.145 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.145 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.146 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_grafana": { 2024-03-12T06:05:59.146 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_grafana", 2024-03-12T06:05:59.146 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.146 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.146 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.146 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-12T06:05:59.146 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.146 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.147 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.147 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-12T06:05:59.147 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.147 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.147 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.147 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_haproxy": { 2024-03-12T06:05:59.147 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_haproxy", 2024-03-12T06:05:59.148 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.148 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.148 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.148 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-12T06:05:59.148 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.148 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.148 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.149 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "HAproxy container image", 2024-03-12T06:05:59.149 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.149 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.149 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.149 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_agent": { 2024-03-12T06:05:59.149 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_agent", 2024-03-12T06:05:59.149 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.150 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.150 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.150 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-12T06:05:59.150 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.150 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.150 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.150 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger agent container image", 2024-03-12T06:05:59.150 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.151 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.151 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.151 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_collector": { 2024-03-12T06:05:59.151 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_collector", 2024-03-12T06:05:59.151 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.151 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.151 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.152 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-12T06:05:59.152 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.152 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.152 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.152 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger collector container image", 2024-03-12T06:05:59.152 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.152 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.153 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.153 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_jaeger_query": { 2024-03-12T06:05:59.153 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_jaeger_query", 2024-03-12T06:05:59.153 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.153 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.153 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.153 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-12T06:05:59.153 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.154 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.154 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.154 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Jaeger query container image", 2024-03-12T06:05:59.154 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.154 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.154 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.154 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_keepalived": { 2024-03-12T06:05:59.155 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_keepalived", 2024-03-12T06:05:59.155 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.155 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.155 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.155 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-12T06:05:59.155 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.155 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.155 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.155 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Keepalived container image", 2024-03-12T06:05:59.156 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.156 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.156 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.156 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_loki": { 2024-03-12T06:05:59.156 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_loki", 2024-03-12T06:05:59.156 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.156 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.157 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.157 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-12T06:05:59.157 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.157 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.157 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.157 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Loki container image", 2024-03-12T06:05:59.157 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.157 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.158 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.158 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_node_exporter": { 2024-03-12T06:05:59.158 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_node_exporter", 2024-03-12T06:05:59.158 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.158 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.158 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.158 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-12T06:05:59.158 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.159 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.159 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.159 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-12T06:05:59.159 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.159 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.159 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.159 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_nvmeof": { 2024-03-12T06:05:59.160 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_nvmeof", 2024-03-12T06:05:59.160 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.160 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.160 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.160 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-12T06:05:59.160 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.160 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.160 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.161 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Nvme-of container image", 2024-03-12T06:05:59.161 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.161 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.161 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.161 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_prometheus": { 2024-03-12T06:05:59.161 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_prometheus", 2024-03-12T06:05:59.162 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.162 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.162 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.162 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-12T06:05:59.162 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.162 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.162 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.162 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Prometheus container image", 2024-03-12T06:05:59.163 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.163 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.163 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.163 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_promtail": { 2024-03-12T06:05:59.163 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_promtail", 2024-03-12T06:05:59.163 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.163 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.164 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.164 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-12T06:05:59.164 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.164 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.164 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.164 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Promtail container image", 2024-03-12T06:05:59.164 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.164 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.164 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.165 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image_snmp_gateway": { 2024-03-12T06:05:59.165 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_image_snmp_gateway", 2024-03-12T06:05:59.165 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.165 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.165 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.165 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-12T06:05:59.165 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.166 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.166 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.166 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "SNMP Gateway container image", 2024-03-12T06:05:59.166 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.166 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.166 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.167 INFO:teuthology.orchestra.run.smithi076.stdout: "container_init": { 2024-03-12T06:05:59.167 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "container_init", 2024-03-12T06:05:59.167 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.167 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.167 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.167 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.167 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.168 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.168 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.168 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Run podman/docker with `--init`", 2024-03-12T06:05:59.168 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.168 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.168 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.168 INFO:teuthology.orchestra.run.smithi076.stdout: "daemon_cache_timeout": { 2024-03-12T06:05:59.169 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "daemon_cache_timeout", 2024-03-12T06:05:59.169 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.169 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.169 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.169 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:59.169 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.169 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.169 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.170 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-12T06:05:59.170 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.170 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.170 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.170 INFO:teuthology.orchestra.run.smithi076.stdout: "default_cephadm_command_timeout": { 2024-03-12T06:05:59.170 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "default_cephadm_command_timeout", 2024-03-12T06:05:59.171 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.171 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.171 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.171 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "900", 2024-03-12T06:05:59.171 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.171 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.171 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.171 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-12T06:05:59.172 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.172 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.172 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.172 INFO:teuthology.orchestra.run.smithi076.stdout: "default_registry": { 2024-03-12T06:05:59.172 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "default_registry", 2024-03-12T06:05:59.172 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.173 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.173 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.173 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "docker.io", 2024-03-12T06:05:59.173 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.173 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.173 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.173 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-12T06:05:59.173 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.173 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.174 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.174 INFO:teuthology.orchestra.run.smithi076.stdout: "device_cache_timeout": { 2024-03-12T06:05:59.174 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_cache_timeout", 2024-03-12T06:05:59.174 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.174 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.174 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.174 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1800", 2024-03-12T06:05:59.175 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.175 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.175 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.175 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache device inventory", 2024-03-12T06:05:59.175 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.175 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.175 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.176 INFO:teuthology.orchestra.run.smithi076.stdout: "device_enhanced_scan": { 2024-03-12T06:05:59.176 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_enhanced_scan", 2024-03-12T06:05:59.176 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.176 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.176 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.176 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.176 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.176 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.177 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.177 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-12T06:05:59.177 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.177 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.177 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.177 INFO:teuthology.orchestra.run.smithi076.stdout: "facts_cache_timeout": { 2024-03-12T06:05:59.177 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "facts_cache_timeout", 2024-03-12T06:05:59.178 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.178 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.178 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.178 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:59.178 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.178 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.178 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.178 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "seconds to cache host facts data", 2024-03-12T06:05:59.179 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.179 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.179 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.179 INFO:teuthology.orchestra.run.smithi076.stdout: "host_check_interval": { 2024-03-12T06:05:59.179 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "host_check_interval", 2024-03-12T06:05:59.179 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.179 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.180 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.180 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:59.180 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.180 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.180 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.180 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to perform a host check", 2024-03-12T06:05:59.180 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.180 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.180 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.181 INFO:teuthology.orchestra.run.smithi076.stdout: "hw_monitoring": { 2024-03-12T06:05:59.181 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "hw_monitoring", 2024-03-12T06:05:59.181 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.181 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.181 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.181 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.181 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.182 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.182 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.182 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-12T06:05:59.182 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.182 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.182 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.183 INFO:teuthology.orchestra.run.smithi076.stdout: "inventory_list_all": { 2024-03-12T06:05:59.183 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "inventory_list_all", 2024-03-12T06:05:59.183 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.183 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.183 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.183 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.183 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.183 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.184 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.184 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-12T06:05:59.184 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.184 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.184 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.184 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.184 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.185 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.185 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.185 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.185 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.185 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.185 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.185 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.185 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.185 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.186 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.186 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.186 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.186 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.186 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.186 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.186 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.186 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.186 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.187 INFO:teuthology.orchestra.run.smithi076.stdout: "log_refresh_metadata": { 2024-03-12T06:05:59.187 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_refresh_metadata", 2024-03-12T06:05:59.187 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.187 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.187 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.187 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.187 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.187 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.187 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.188 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-12T06:05:59.188 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.188 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.188 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.188 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.188 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.188 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.188 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.189 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.189 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.189 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.189 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.189 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.189 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-12T06:05:59.189 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.189 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.189 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.190 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.190 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.190 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.190 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.190 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.190 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.190 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.191 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.191 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.191 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.191 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.191 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.191 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.191 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.191 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.191 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.192 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.192 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.192 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.192 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.192 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.192 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.193 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.193 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.193 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.193 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.193 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.193 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.193 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.193 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.194 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.194 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.194 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.194 INFO:teuthology.orchestra.run.smithi076.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-12T06:05:59.194 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-12T06:05:59.194 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.195 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.195 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.195 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.195 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.195 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.195 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.195 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-12T06:05:59.195 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.196 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.196 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.196 INFO:teuthology.orchestra.run.smithi076.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-12T06:05:59.196 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-12T06:05:59.196 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.196 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.196 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.197 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "*", 2024-03-12T06:05:59.197 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.197 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.197 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.197 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-12T06:05:59.197 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.197 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.198 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.198 INFO:teuthology.orchestra.run.smithi076.stdout: "max_count_per_host": { 2024-03-12T06:05:59.198 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_count_per_host", 2024-03-12T06:05:59.198 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.198 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.198 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.198 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:59.199 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.199 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.199 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.199 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "max number of daemons per service per host", 2024-03-12T06:05:59.199 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.199 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.199 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.200 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd_draining_count": { 2024-03-12T06:05:59.200 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_osd_draining_count", 2024-03-12T06:05:59.200 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.200 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.200 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.200 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:59.200 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.200 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.200 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.201 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-12T06:05:59.201 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.201 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.201 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.201 INFO:teuthology.orchestra.run.smithi076.stdout: "migration_current": { 2024-03-12T06:05:59.201 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "migration_current", 2024-03-12T06:05:59.201 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.201 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.202 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.202 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.202 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.202 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.202 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.202 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "internal - do not modify", 2024-03-12T06:05:59.202 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.202 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.203 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.203 INFO:teuthology.orchestra.run.smithi076.stdout: "mode": { 2024-03-12T06:05:59.203 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mode", 2024-03-12T06:05:59.203 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.203 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.203 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.203 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "root", 2024-03-12T06:05:59.203 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.204 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.204 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.204 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm-package", 2024-03-12T06:05:59.204 INFO:teuthology.orchestra.run.smithi076.stdout: "root" 2024-03-12T06:05:59.204 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.204 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "mode for remote execution of cephadm", 2024-03-12T06:05:59.204 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.204 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.205 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.205 INFO:teuthology.orchestra.run.smithi076.stdout: "oob_default_addr": { 2024-03-12T06:05:59.205 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "oob_default_addr", 2024-03-12T06:05:59.205 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.205 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.205 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.205 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "169.254.1.1", 2024-03-12T06:05:59.205 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.206 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.206 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.206 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-12T06:05:59.206 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.206 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.206 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.206 INFO:teuthology.orchestra.run.smithi076.stdout: "prometheus_alerts_path": { 2024-03-12T06:05:59.207 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prometheus_alerts_path", 2024-03-12T06:05:59.207 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.207 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.207 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.207 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-12T06:05:59.207 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.207 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.207 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.208 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-12T06:05:59.208 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.208 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.208 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.208 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_insecure": { 2024-03-12T06:05:59.208 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_insecure", 2024-03-12T06:05:59.208 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.209 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.209 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.209 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.209 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.209 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.209 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.209 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-12T06:05:59.209 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.209 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.210 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.210 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_password": { 2024-03-12T06:05:59.210 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_password", 2024-03-12T06:05:59.210 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.210 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.210 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.210 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.210 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.211 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.211 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.211 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-12T06:05:59.211 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.211 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.211 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.211 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_url": { 2024-03-12T06:05:59.212 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_url", 2024-03-12T06:05:59.212 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.212 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.212 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.212 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.212 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.212 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.212 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.213 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-12T06:05:59.213 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.213 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.213 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.213 INFO:teuthology.orchestra.run.smithi076.stdout: "registry_username": { 2024-03-12T06:05:59.213 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "registry_username", 2024-03-12T06:05:59.213 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.214 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.214 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.214 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.214 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.214 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.214 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.214 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-12T06:05:59.214 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.215 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.215 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.215 INFO:teuthology.orchestra.run.smithi076.stdout: "secure_monitoring_stack": { 2024-03-12T06:05:59.215 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "secure_monitoring_stack", 2024-03-12T06:05:59.215 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.215 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.215 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.216 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.216 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.216 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.216 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.216 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-12T06:05:59.216 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.216 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.216 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.217 INFO:teuthology.orchestra.run.smithi076.stdout: "service_discovery_port": { 2024-03-12T06:05:59.217 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "service_discovery_port", 2024-03-12T06:05:59.217 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.217 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.217 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.217 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8765", 2024-03-12T06:05:59.217 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.218 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.218 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.218 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "cephadm service discovery port", 2024-03-12T06:05:59.218 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.218 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.218 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.218 INFO:teuthology.orchestra.run.smithi076.stdout: "ssh_config_file": { 2024-03-12T06:05:59.219 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssh_config_file", 2024-03-12T06:05:59.219 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.219 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.219 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.219 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.219 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.219 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.219 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.220 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-12T06:05:59.220 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.220 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.220 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.220 INFO:teuthology.orchestra.run.smithi076.stdout: "use_agent": { 2024-03-12T06:05:59.220 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "use_agent", 2024-03-12T06:05:59.220 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.221 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.221 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.221 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.221 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.221 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.221 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.221 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-12T06:05:59.222 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.222 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.222 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.222 INFO:teuthology.orchestra.run.smithi076.stdout: "use_repo_digest": { 2024-03-12T06:05:59.222 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "use_repo_digest", 2024-03-12T06:05:59.222 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.222 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.223 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.223 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.223 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.223 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.223 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.223 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-12T06:05:59.223 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.224 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.224 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.224 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_failed_host_check": { 2024-03-12T06:05:59.224 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_failed_host_check", 2024-03-12T06:05:59.224 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.224 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.224 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.225 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.225 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.225 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.225 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.225 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if the host check fails", 2024-03-12T06:05:59.225 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.225 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.225 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.226 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_stray_daemons": { 2024-03-12T06:05:59.226 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_stray_daemons", 2024-03-12T06:05:59.226 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.226 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.226 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.226 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.226 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.226 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.226 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.227 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-12T06:05:59.227 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.227 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.227 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.227 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_on_stray_hosts": { 2024-03-12T06:05:59.227 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_on_stray_hosts", 2024-03-12T06:05:59.227 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.227 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.228 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.228 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.228 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.228 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.228 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.228 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-12T06:05:59.228 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.228 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.228 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.229 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.229 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.229 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crash", 2024-03-12T06:05:59.229 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.229 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.229 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.229 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.230 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.230 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.230 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.230 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.230 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.230 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.230 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.230 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.230 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.231 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.231 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.231 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.231 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.231 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.231 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.231 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.231 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.232 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.232 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.232 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.232 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.232 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.232 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.232 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.233 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.233 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.233 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.233 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.233 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.233 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.233 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.233 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.234 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.234 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.234 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.234 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.234 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.234 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.235 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.235 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.235 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.235 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.235 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.235 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.235 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.235 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.236 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.236 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.236 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.236 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.236 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.236 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.236 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.237 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.237 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.237 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.237 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.237 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.237 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.237 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.237 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.238 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.238 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.238 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.238 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.238 INFO:teuthology.orchestra.run.smithi076.stdout: "retain_interval": { 2024-03-12T06:05:59.238 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "retain_interval", 2024-03-12T06:05:59.238 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.239 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.239 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.239 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "31536000", 2024-03-12T06:05:59.239 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.239 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.239 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.239 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-12T06:05:59.239 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.240 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.240 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.240 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_recent_interval": { 2024-03-12T06:05:59.240 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_recent_interval", 2024-03-12T06:05:59.240 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.240 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.240 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.241 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1209600", 2024-03-12T06:05:59.241 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.241 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.241 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.241 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-12T06:05:59.241 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.241 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.241 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.241 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.242 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.242 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "dashboard", 2024-03-12T06:05:59.242 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.242 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.242 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.242 INFO:teuthology.orchestra.run.smithi076.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-12T06:05:59.243 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-12T06:05:59.243 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.243 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.243 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.243 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:59.243 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.243 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.243 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.243 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.244 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.244 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.244 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.244 INFO:teuthology.orchestra.run.smithi076.stdout: "ALERTMANAGER_API_HOST": { 2024-03-12T06:05:59.244 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-12T06:05:59.244 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.244 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.245 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.245 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.245 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.245 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.245 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.245 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.245 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.245 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.245 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.246 INFO:teuthology.orchestra.run.smithi076.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-12T06:05:59.246 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-12T06:05:59.246 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.246 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.246 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.246 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.247 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.247 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.247 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.247 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.247 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.247 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.247 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.247 INFO:teuthology.orchestra.run.smithi076.stdout: "AUDIT_API_ENABLED": { 2024-03-12T06:05:59.248 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "AUDIT_API_ENABLED", 2024-03-12T06:05:59.248 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.248 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.248 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.248 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.248 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.248 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.249 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.249 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.249 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.249 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.249 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.249 INFO:teuthology.orchestra.run.smithi076.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-12T06:05:59.249 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-12T06:05:59.250 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.250 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.250 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.250 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.250 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.250 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.250 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.250 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.251 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.251 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.251 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.251 INFO:teuthology.orchestra.run.smithi076.stdout: "ENABLE_BROWSABLE_API": { 2024-03-12T06:05:59.251 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-12T06:05:59.251 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.251 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.252 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.252 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.252 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.252 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.252 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.252 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.252 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.252 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.253 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.253 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-12T06:05:59.253 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-12T06:05:59.253 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.253 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.253 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.253 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.254 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.254 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.254 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.254 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.254 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.254 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.254 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.255 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-12T06:05:59.255 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-12T06:05:59.255 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.255 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.255 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.255 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.255 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.256 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.256 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.256 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.256 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.256 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.256 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.257 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-12T06:05:59.257 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-12T06:05:59.257 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.257 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.257 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.257 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.257 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.257 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.257 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.258 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.258 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.258 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.258 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.258 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-12T06:05:59.258 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-12T06:05:59.258 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.259 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.259 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.259 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.259 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.259 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.259 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.259 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.259 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.259 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.260 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.260 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-12T06:05:59.260 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-12T06:05:59.260 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.260 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.260 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.260 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.260 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.261 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.261 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.261 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.261 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.261 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.261 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.261 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-12T06:05:59.262 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-12T06:05:59.262 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.262 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.262 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.262 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.262 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.262 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.263 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.263 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.263 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.263 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.263 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.263 INFO:teuthology.orchestra.run.smithi076.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-12T06:05:59.264 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-12T06:05:59.264 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.264 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.264 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.264 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.264 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.264 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.264 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.264 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.265 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.265 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.265 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.265 INFO:teuthology.orchestra.run.smithi076.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-12T06:05:59.265 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-12T06:05:59.265 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.265 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.265 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.266 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.266 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.266 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.266 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.266 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.266 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.266 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.266 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.267 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_PASSWORD": { 2024-03-12T06:05:59.267 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-12T06:05:59.267 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.267 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.267 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.267 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-12T06:05:59.267 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.268 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.268 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.268 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.268 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.268 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.268 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.268 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-12T06:05:59.268 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-12T06:05:59.269 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.269 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.269 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.269 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.269 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.269 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.269 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.269 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.269 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.269 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.270 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.270 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_URL": { 2024-03-12T06:05:59.270 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_URL", 2024-03-12T06:05:59.270 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.270 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.270 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.270 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.271 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.271 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.271 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.271 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.271 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.271 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.271 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.272 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_API_USERNAME": { 2024-03-12T06:05:59.272 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-12T06:05:59.272 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.272 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.272 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.272 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-12T06:05:59.272 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.273 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.273 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.273 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.273 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.273 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.273 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.273 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-12T06:05:59.273 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-12T06:05:59.274 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.274 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.274 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.274 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.274 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.274 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.274 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.274 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.274 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.275 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.275 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.275 INFO:teuthology.orchestra.run.smithi076.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-12T06:05:59.275 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-12T06:05:59.275 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.275 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.275 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.276 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.276 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.276 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.276 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.276 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.276 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.276 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.276 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.277 INFO:teuthology.orchestra.run.smithi076.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-12T06:05:59.277 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-12T06:05:59.277 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.277 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.277 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.277 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.277 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.277 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.278 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.278 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.278 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.278 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.278 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.278 INFO:teuthology.orchestra.run.smithi076.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-12T06:05:59.278 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-12T06:05:59.279 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.279 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.279 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.279 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.279 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.279 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.279 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.280 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.280 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.280 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.280 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.280 INFO:teuthology.orchestra.run.smithi076.stdout: "PROMETHEUS_API_HOST": { 2024-03-12T06:05:59.280 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-12T06:05:59.280 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.281 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.281 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.281 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.281 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.281 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.281 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.281 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.281 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.282 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.282 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.282 INFO:teuthology.orchestra.run.smithi076.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-12T06:05:59.282 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-12T06:05:59.282 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.282 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.283 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.283 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.283 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.283 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.283 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.283 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.283 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.283 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.284 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.284 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-12T06:05:59.284 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-12T06:05:59.284 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.284 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.284 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.284 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.285 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.285 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.285 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.285 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.285 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.285 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.285 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.286 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-12T06:05:59.286 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-12T06:05:59.286 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.286 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.286 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.286 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.286 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.286 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.287 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.287 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.287 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.287 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.287 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.287 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-12T06:05:59.288 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-12T06:05:59.288 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.288 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.288 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.288 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.288 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.288 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.288 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.289 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.289 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.289 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.289 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.289 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-12T06:05:59.289 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-12T06:05:59.290 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.290 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.290 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.290 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.290 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.290 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.291 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.291 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.291 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.291 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.291 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.291 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-12T06:05:59.291 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-12T06:05:59.292 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.292 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.292 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.292 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.292 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.292 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.292 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.292 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.293 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.293 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.293 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.293 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-12T06:05:59.293 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-12T06:05:59.293 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.293 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.293 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.294 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.294 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.294 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.294 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.294 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.294 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.294 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.294 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.295 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-12T06:05:59.295 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-12T06:05:59.295 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.295 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.295 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.295 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.295 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.295 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.295 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.296 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.296 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.296 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.296 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.296 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_ENABLED": { 2024-03-12T06:05:59.296 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-12T06:05:59.296 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.296 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.297 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.297 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.297 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.297 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.297 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.297 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.297 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.297 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.298 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.298 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-12T06:05:59.298 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-12T06:05:59.298 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.298 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.298 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.298 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-12T06:05:59.298 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.299 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.299 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.299 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.299 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.299 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.299 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.299 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-12T06:05:59.299 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-12T06:05:59.299 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.300 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.300 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.300 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:59.300 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.300 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.300 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.300 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.300 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.301 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.301 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.301 INFO:teuthology.orchestra.run.smithi076.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-12T06:05:59.301 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-12T06:05:59.301 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.301 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.301 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.302 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8", 2024-03-12T06:05:59.302 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.302 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.302 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.302 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.302 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.302 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.302 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.303 INFO:teuthology.orchestra.run.smithi076.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-12T06:05:59.303 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-12T06:05:59.303 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.303 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.303 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.303 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "45", 2024-03-12T06:05:59.303 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.304 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.304 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.304 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.304 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.304 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.304 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.304 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_ACCESS_KEY": { 2024-03-12T06:05:59.304 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-12T06:05:59.305 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.305 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.305 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.305 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.305 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.305 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.305 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.305 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.305 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.306 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.306 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.306 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-12T06:05:59.306 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-12T06:05:59.306 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.306 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.306 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.307 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "admin", 2024-03-12T06:05:59.307 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.307 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.307 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.307 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.307 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.307 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.307 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.308 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_SECRET_KEY": { 2024-03-12T06:05:59.308 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-12T06:05:59.308 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.308 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.308 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.308 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.308 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.308 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.308 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.308 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.309 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.309 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.309 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.309 INFO:teuthology.orchestra.run.smithi076.stdout: "RGW_API_SSL_VERIFY": { 2024-03-12T06:05:59.309 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-12T06:05:59.309 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.309 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.310 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.310 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.310 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.310 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.310 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.310 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.310 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.310 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.310 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.311 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-12T06:05:59.311 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-12T06:05:59.311 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.311 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.311 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.311 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:59.311 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.311 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.311 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.312 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.312 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.312 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.312 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.312 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-12T06:05:59.312 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-12T06:05:59.312 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.313 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.313 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.313 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:59.313 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.313 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.313 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.313 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.313 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.313 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.314 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.314 INFO:teuthology.orchestra.run.smithi076.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-12T06:05:59.314 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-12T06:05:59.314 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.314 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.314 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.314 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-12T06:05:59.315 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.315 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.315 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.315 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.315 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.315 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.315 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.316 INFO:teuthology.orchestra.run.smithi076.stdout: "cross_origin_url": { 2024-03-12T06:05:59.316 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cross_origin_url", 2024-03-12T06:05:59.316 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.316 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.316 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.316 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.316 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.316 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.317 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.317 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.317 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.317 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.317 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.317 INFO:teuthology.orchestra.run.smithi076.stdout: "crt_file": { 2024-03-12T06:05:59.318 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "crt_file", 2024-03-12T06:05:59.318 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.318 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.318 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.318 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.318 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.318 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.318 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.319 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.319 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.319 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.319 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.319 INFO:teuthology.orchestra.run.smithi076.stdout: "debug": { 2024-03-12T06:05:59.319 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "debug", 2024-03-12T06:05:59.319 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.320 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.320 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.320 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.320 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.320 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.320 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.320 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Enable/disable debug options", 2024-03-12T06:05:59.320 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.320 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.321 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.321 INFO:teuthology.orchestra.run.smithi076.stdout: "jwt_token_ttl": { 2024-03-12T06:05:59.321 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "jwt_token_ttl", 2024-03-12T06:05:59.321 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.321 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.322 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.322 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "28800", 2024-03-12T06:05:59.322 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.322 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.322 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.322 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.322 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.322 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.322 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.323 INFO:teuthology.orchestra.run.smithi076.stdout: "key_file": { 2024-03-12T06:05:59.323 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "key_file", 2024-03-12T06:05:59.323 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.323 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.323 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.323 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.323 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.323 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.324 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.324 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.324 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.324 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.324 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.324 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.324 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.325 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.325 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.325 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.325 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.325 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.325 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.325 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.325 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.325 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.326 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.326 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.326 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.326 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.326 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.326 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.326 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.326 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.327 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.327 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.327 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.327 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.327 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.327 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.327 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.328 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.328 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.328 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.328 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.328 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.328 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.328 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.328 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.329 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.329 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.329 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.329 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.329 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.329 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.329 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.329 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.330 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.330 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.330 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.330 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.330 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.330 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.330 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.331 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.331 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.331 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.331 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.331 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.331 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.331 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.332 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.332 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.332 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.332 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.332 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.332 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.332 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.333 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.333 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.333 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.333 INFO:teuthology.orchestra.run.smithi076.stdout: "motd": { 2024-03-12T06:05:59.333 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "motd", 2024-03-12T06:05:59.333 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.333 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.334 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.334 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.334 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.334 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.334 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.334 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "The message of the day", 2024-03-12T06:05:59.334 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.334 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.334 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.335 INFO:teuthology.orchestra.run.smithi076.stdout: "redirect_resolve_ip_addr": { 2024-03-12T06:05:59.335 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "redirect_resolve_ip_addr", 2024-03-12T06:05:59.335 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.335 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.335 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.335 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.336 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.336 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.336 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.336 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.336 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.336 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.336 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.336 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-12T06:05:59.337 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-12T06:05:59.337 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.337 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.337 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.337 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "::", 2024-03-12T06:05:59.337 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.337 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.337 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.337 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.338 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.338 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.338 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.338 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-12T06:05:59.338 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-12T06:05:59.338 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.338 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.338 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.339 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8080", 2024-03-12T06:05:59.339 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.339 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.339 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.339 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.339 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.339 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.339 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl": { 2024-03-12T06:05:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl", 2024-03-12T06:05:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.340 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl_server_port": { 2024-03-12T06:05:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl_server_port", 2024-03-12T06:05:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.341 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8443", 2024-03-12T06:05:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.342 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_behaviour": { 2024-03-12T06:05:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_behaviour", 2024-03-12T06:05:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.343 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "redirect", 2024-03-12T06:05:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "redirect" 2024-03-12T06:05:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.344 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_error_status_code": { 2024-03-12T06:05:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_error_status_code", 2024-03-12T06:05:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.345 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "500", 2024-03-12T06:05:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "400", 2024-03-12T06:05:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "599", 2024-03-12T06:05:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "url_prefix": { 2024-03-12T06:05:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "url_prefix", 2024-03-12T06:05:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.347 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.348 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "devicehealth", 2024-03-12T06:05:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_monitoring": { 2024-03-12T06:05:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enable_monitoring", 2024-03-12T06:05:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.349 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "monitor device health metrics", 2024-03-12T06:05:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.351 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.352 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.353 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.354 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.355 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.356 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.357 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.358 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.359 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.359 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.359 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.359 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.359 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.359 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.359 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.359 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: "mark_out_threshold": { 2024-03-12T06:05:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mark_out_threshold", 2024-03-12T06:05:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "2419200", 2024-03-12T06:05:59.360 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-12T06:05:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": { 2024-03-12T06:05:59.361 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pool_name", 2024-03-12T06:05:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "device_health_metrics", 2024-03-12T06:05:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-12T06:05:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "retention_period": { 2024-03-12T06:05:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "retention_period", 2024-03-12T06:05:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15552000", 2024-03-12T06:05:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long to retain device health metrics", 2024-03-12T06:05:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.364 INFO:teuthology.orchestra.run.smithi076.stdout: "scrape_frequency": { 2024-03-12T06:05:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "scrape_frequency", 2024-03-12T06:05:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "86400", 2024-03-12T06:05:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.365 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-12T06:05:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "self_heal": { 2024-03-12T06:05:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "self_heal", 2024-03-12T06:05:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.366 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-12T06:05:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.368 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how frequently to wake up and check device health", 2024-03-12T06:05:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "warn_threshold": { 2024-03-12T06:05:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "warn_threshold", 2024-03-12T06:05:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.369 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "7257600", 2024-03-12T06:05:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-12T06:05:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.370 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "diskprediction_local", 2024-03-12T06:05:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.371 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.372 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.373 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.374 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.375 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.376 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.377 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.378 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.379 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "predict_interval": { 2024-03-12T06:05:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "predict_interval", 2024-03-12T06:05:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "86400", 2024-03-12T06:05:59.380 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "predictor_model": { 2024-03-12T06:05:59.381 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "predictor_model", 2024-03-12T06:05:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "prophetstor", 2024-03-12T06:05:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.382 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.383 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "600", 2024-03-12T06:05:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.384 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "influx", 2024-03-12T06:05:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": false, 2024-03-12T06:05:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "influxdb python module not found", 2024-03-12T06:05:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "batch_size": { 2024-03-12T06:05:59.385 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "batch_size", 2024-03-12T06:05:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5000", 2024-03-12T06:05:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.386 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-12T06:05:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "database": { 2024-03-12T06:05:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "database", 2024-03-12T06:05:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.387 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "ceph", 2024-03-12T06:05:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-12T06:05:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.388 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "hostname": { 2024-03-12T06:05:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "hostname", 2024-03-12T06:05:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.389 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB server hostname", 2024-03-12T06:05:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:59.390 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "30", 2024-03-12T06:05:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "5", 2024-03-12T06:05:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.391 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-12T06:05:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.392 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.393 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.394 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.395 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.397 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.398 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.399 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.400 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "password": { 2024-03-12T06:05:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "password", 2024-03-12T06:05:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.401 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "password of InfluxDB server user", 2024-03-12T06:05:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.402 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "port": { 2024-03-12T06:05:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "port", 2024-03-12T06:05:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "8086", 2024-03-12T06:05:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.403 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "InfluxDB server port", 2024-03-12T06:05:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: "ssl": { 2024-03-12T06:05:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ssl", 2024-03-12T06:05:59.404 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "false", 2024-03-12T06:05:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-12T06:05:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.405 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: "threads": { 2024-03-12T06:05:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "threads", 2024-03-12T06:05:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-12T06:05:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-12T06:05:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "32", 2024-03-12T06:05:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-12T06:05:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "username": { 2024-03-12T06:05:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "username", 2024-03-12T06:05:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.408 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "username of InfluxDB server user", 2024-03-12T06:05:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "verify_ssl": { 2024-03-12T06:05:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "verify_ssl", 2024-03-12T06:05:59.409 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "true", 2024-03-12T06:05:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-12T06:05:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.410 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "insights", 2024-03-12T06:05:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.411 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.412 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.413 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.414 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.415 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.416 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.417 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.418 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.419 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "iostat", 2024-03-12T06:05:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.420 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.421 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.422 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.423 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.424 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.425 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.426 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.427 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.428 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.429 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "k8sevents", 2024-03-12T06:05:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.430 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_event_retention_days": { 2024-03-12T06:05:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "ceph_event_retention_days", 2024-03-12T06:05:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "7", 2024-03-12T06:05:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.431 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-12T06:05:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "config_check_secs": { 2024-03-12T06:05:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "config_check_secs", 2024-03-12T06:05:59.432 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "10", 2024-03-12T06:05:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-12T06:05:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.433 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.435 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.436 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.437 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.438 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.439 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.440 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.441 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.441 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.441 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.441 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.441 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.441 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.441 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.442 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.442 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.442 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.442 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.442 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.442 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.442 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.443 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "localpool", 2024-03-12T06:05:59.443 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.443 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.443 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.443 INFO:teuthology.orchestra.run.smithi076.stdout: "failure_domain": { 2024-03-12T06:05:59.443 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "failure_domain", 2024-03-12T06:05:59.443 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.443 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.443 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.444 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "host", 2024-03-12T06:05:59.444 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.444 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.444 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.444 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "failure domain for any created local pool", 2024-03-12T06:05:59.444 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-12T06:05:59.444 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.444 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.445 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.445 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.445 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.445 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.445 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.446 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.446 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.446 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.446 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.446 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.446 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.446 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.446 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.447 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.447 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.447 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.447 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.447 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.447 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.447 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.448 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.448 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.448 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.448 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.448 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.448 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.448 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.448 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.448 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.449 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.449 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.449 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.449 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.449 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.449 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.449 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.449 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.450 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.450 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.450 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.450 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.450 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.450 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.450 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.450 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.450 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.450 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.451 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.451 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.451 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.451 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.451 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.451 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.451 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.451 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.452 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.452 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.452 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.452 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.452 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.452 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.452 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.452 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.452 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.452 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.453 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.453 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": { 2024-03-12T06:05:59.453 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "min_size", 2024-03-12T06:05:59.453 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.453 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.453 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.453 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.453 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.454 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.454 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.454 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default min_size for any created local pool", 2024-03-12T06:05:59.454 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-12T06:05:59.454 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.454 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.454 INFO:teuthology.orchestra.run.smithi076.stdout: "num_rep": { 2024-03-12T06:05:59.454 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "num_rep", 2024-03-12T06:05:59.455 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.455 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.455 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.455 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3", 2024-03-12T06:05:59.455 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.455 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.455 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.455 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default replica count for any created local pool", 2024-03-12T06:05:59.456 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.456 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.456 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.456 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": { 2024-03-12T06:05:59.456 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pg_num", 2024-03-12T06:05:59.456 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.456 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.456 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.456 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "128", 2024-03-12T06:05:59.457 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.457 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.457 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.457 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "default pg_num for any created local pool", 2024-03-12T06:05:59.457 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.457 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.457 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.457 INFO:teuthology.orchestra.run.smithi076.stdout: "prefix": { 2024-03-12T06:05:59.458 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prefix", 2024-03-12T06:05:59.458 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.458 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.458 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.458 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.458 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.458 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.458 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.458 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "name prefix for any created local pool", 2024-03-12T06:05:59.459 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.459 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.459 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.459 INFO:teuthology.orchestra.run.smithi076.stdout: "subtree": { 2024-03-12T06:05:59.459 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "subtree", 2024-03-12T06:05:59.459 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.459 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.460 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.460 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "rack", 2024-03-12T06:05:59.460 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.460 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.460 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.460 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-12T06:05:59.460 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-12T06:05:59.460 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.461 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.461 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.461 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.461 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mds_autoscaler", 2024-03-12T06:05:59.461 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.461 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.462 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.462 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.462 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.462 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.462 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.462 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.462 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.462 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.463 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.463 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.463 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.463 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.463 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.463 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.463 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.464 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.464 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.464 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.464 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.464 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.464 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.464 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.465 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.465 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.465 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.465 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.465 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.465 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.465 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.465 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.466 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.466 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.466 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.466 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.466 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.466 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.466 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.467 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.467 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.467 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.467 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.467 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.467 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.467 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.468 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.468 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.468 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.468 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.468 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.468 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.468 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.468 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.469 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.469 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.469 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.469 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.469 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.469 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.470 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.470 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.470 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.470 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.470 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.470 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.470 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.470 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.471 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.471 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.471 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.471 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mirroring", 2024-03-12T06:05:59.471 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.471 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.471 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.472 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.472 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.472 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.472 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.472 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.472 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.472 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.472 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.472 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.473 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.473 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.473 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.473 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.473 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.473 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.473 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.473 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.473 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.473 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.474 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.474 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.474 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.474 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.474 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.474 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.474 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.474 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.475 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.475 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.475 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.475 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.475 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.475 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.475 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.476 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.476 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.476 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.476 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.476 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.476 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.476 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.476 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.476 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.476 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.477 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.477 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.477 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.477 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.477 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.477 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.477 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.477 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.478 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.478 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.478 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.478 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.478 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.478 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.478 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.478 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.479 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.479 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.479 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.479 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.479 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.479 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.479 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.479 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.480 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "nfs", 2024-03-12T06:05:59.480 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.480 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.480 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.480 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.480 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.480 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.480 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.481 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.481 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.481 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.481 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.481 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.481 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.481 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.481 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.481 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.481 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.482 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.482 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.482 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.482 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.482 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.482 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.482 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.482 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.483 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.483 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.483 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.483 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.483 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.483 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.483 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.483 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.483 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.484 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.484 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.484 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.484 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.484 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.484 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.484 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.484 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.485 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.485 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.485 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.485 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.485 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.485 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.485 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.485 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.486 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.486 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.486 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.486 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.486 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.486 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.486 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.486 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.487 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.487 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.487 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.487 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.487 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.487 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.487 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.488 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.488 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.488 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.488 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.488 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.488 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.489 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "orchestrator", 2024-03-12T06:05:59.489 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.489 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.489 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.489 INFO:teuthology.orchestra.run.smithi076.stdout: "fail_fs": { 2024-03-12T06:05:59.489 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "fail_fs", 2024-03-12T06:05:59.489 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.489 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.490 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.490 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.490 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.490 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.490 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.490 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-12T06:05:59.490 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.490 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.491 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.491 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.491 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.491 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.491 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.491 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.491 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.492 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.492 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.492 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.492 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.492 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.492 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.492 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.492 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.492 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.493 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.493 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.493 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.493 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.493 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.493 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.493 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.494 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.494 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.494 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.494 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.494 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.494 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.494 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.494 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.494 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.494 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.495 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.495 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.495 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.495 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.495 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.495 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.495 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.496 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.496 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.496 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.496 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.496 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.496 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.496 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.496 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.497 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.497 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.497 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.497 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.497 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.497 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.497 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.498 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.498 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.498 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.498 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.498 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.498 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.498 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.498 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.499 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.499 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.499 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.499 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.499 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator": { 2024-03-12T06:05:59.499 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "orchestrator", 2024-03-12T06:05:59.499 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.500 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.500 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.500 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.500 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.500 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.500 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.500 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm", 2024-03-12T06:05:59.500 INFO:teuthology.orchestra.run.smithi076.stdout: "rook", 2024-03-12T06:05:59.500 INFO:teuthology.orchestra.run.smithi076.stdout: "test_orchestrator" 2024-03-12T06:05:59.501 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.501 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Orchestrator backend", 2024-03-12T06:05:59.501 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.501 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.501 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.501 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.501 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.502 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "osd_perf_query", 2024-03-12T06:05:59.502 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.502 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.502 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.502 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.502 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.502 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.503 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.503 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.503 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.503 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.503 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.503 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.503 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.504 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.504 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.504 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.504 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.504 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.504 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.504 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.504 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.505 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.505 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.505 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.505 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.505 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.505 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.505 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.506 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.506 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.506 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.506 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.506 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.506 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.506 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.506 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.507 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.507 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.507 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.507 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.507 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.507 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.507 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.507 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.507 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.508 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.508 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.508 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.508 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.508 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.508 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.508 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.508 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.509 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.509 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.509 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.509 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.509 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.509 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.510 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.510 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.510 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.510 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.510 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.510 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.510 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.511 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.511 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.511 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.511 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.511 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.511 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "osd_support", 2024-03-12T06:05:59.511 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.512 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.512 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.512 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.512 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.512 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.512 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.512 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.512 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.513 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.513 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.513 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.513 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.513 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.513 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.513 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.513 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.513 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.514 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.514 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.514 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.514 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.514 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.514 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.514 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.514 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.515 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.515 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.515 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.515 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.515 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.515 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.515 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.515 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.515 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.515 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.516 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.516 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.516 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.516 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.516 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.516 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.516 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.517 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.517 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.517 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.517 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.517 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.517 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.517 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.517 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.518 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.518 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.518 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.518 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.518 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.518 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.518 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.518 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.519 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.519 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.519 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.519 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.519 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.519 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.519 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.519 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.519 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.519 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.520 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.520 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.520 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "pg_autoscaler", 2024-03-12T06:05:59.520 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.520 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.520 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.521 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.521 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.521 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.521 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.521 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.521 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.522 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.522 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.522 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.522 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.522 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.522 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.522 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.522 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.523 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.523 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.523 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.523 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.523 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.523 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.523 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.524 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.524 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.524 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.524 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.524 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.524 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.524 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.524 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.525 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.525 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.525 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.525 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.525 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.525 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.525 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.525 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.526 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.526 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.526 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.526 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.526 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.526 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.526 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.526 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.527 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.527 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.527 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.527 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.527 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.527 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.527 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.527 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.528 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.528 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.528 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.528 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.528 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.528 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.528 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.528 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.529 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.529 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.529 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.529 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.529 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.529 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:59.529 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:59.529 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.530 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.530 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.530 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:59.530 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.530 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.530 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.530 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.531 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.531 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.531 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.531 INFO:teuthology.orchestra.run.smithi076.stdout: "threshold": { 2024-03-12T06:05:59.531 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "threshold", 2024-03-12T06:05:59.531 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:59.531 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.531 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.532 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "3.0", 2024-03-12T06:05:59.532 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1.0", 2024-03-12T06:05:59.532 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.532 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.532 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "scaling threshold", 2024-03-12T06:05:59.532 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-12T06:05:59.532 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.532 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.533 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.533 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.533 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "progress", 2024-03-12T06:05:59.533 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.533 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.534 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.534 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_pg_recovery_event": { 2024-03-12T06:05:59.534 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_pg_recovery_event", 2024-03-12T06:05:59.534 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.534 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.534 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.534 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.534 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.535 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.535 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.535 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-12T06:05:59.535 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.535 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.535 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.535 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.535 INFO:teuthology.orchestra.run.smithi076.stdout: "enabled": { 2024-03-12T06:05:59.535 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enabled", 2024-03-12T06:05:59.536 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.536 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.536 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.536 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.536 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.536 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.536 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.536 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.537 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.537 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.537 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.537 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.537 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.537 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.537 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.537 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.538 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.538 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.538 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.538 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.538 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.538 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.538 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.538 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.539 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.539 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.539 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.539 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.539 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.539 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.539 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.540 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.540 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.540 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.540 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.540 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.540 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.541 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.541 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.541 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.541 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.541 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.541 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.541 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.542 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.542 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.542 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.542 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.542 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.542 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.542 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.543 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.543 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.543 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.543 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.543 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.543 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.543 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.543 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.544 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.544 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.544 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.544 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.544 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.544 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.544 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.544 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.545 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.545 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.545 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.545 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.545 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.545 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.545 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.545 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.546 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.546 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.546 INFO:teuthology.orchestra.run.smithi076.stdout: "max_completed_events": { 2024-03-12T06:05:59.546 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_completed_events", 2024-03-12T06:05:59.546 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.546 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.546 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.546 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "50", 2024-03-12T06:05:59.547 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.547 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.547 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.547 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "number of past completed events to remember", 2024-03-12T06:05:59.547 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.547 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.547 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.547 INFO:teuthology.orchestra.run.smithi076.stdout: "sleep_interval": { 2024-03-12T06:05:59.548 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "sleep_interval", 2024-03-12T06:05:59.548 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.548 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.548 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.548 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "5", 2024-03-12T06:05:59.548 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.548 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.548 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.548 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "how long the module is going to sleep", 2024-03-12T06:05:59.548 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.549 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.549 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.549 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.549 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.549 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "prometheus", 2024-03-12T06:05:59.549 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.549 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.550 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.550 INFO:teuthology.orchestra.run.smithi076.stdout: "cache": { 2024-03-12T06:05:59.550 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "cache", 2024-03-12T06:05:59.550 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.550 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.550 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.550 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.550 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.551 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.551 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.551 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.551 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.551 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.551 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.551 INFO:teuthology.orchestra.run.smithi076.stdout: "exclude_perf_counters": { 2024-03-12T06:05:59.551 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "exclude_perf_counters", 2024-03-12T06:05:59.552 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.552 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.552 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.552 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.552 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.552 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.552 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.552 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-12T06:05:59.553 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-12T06:05:59.553 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.553 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.553 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.553 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.553 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.553 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.553 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.554 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.554 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.554 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.554 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.554 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.554 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.554 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.554 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.554 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.555 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.555 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.555 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.555 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.555 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.555 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.555 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.555 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.556 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.556 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.556 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.556 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.556 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.556 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.556 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.556 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.556 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.557 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.557 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.557 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.557 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.557 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.557 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.557 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.558 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.558 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.558 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.558 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.558 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.558 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.558 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.559 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.559 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.559 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.559 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.559 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.559 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.559 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.560 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.560 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.560 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.560 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.560 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.560 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.560 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.561 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.561 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.561 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.561 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.561 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.561 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.561 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.561 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_stats_pools": { 2024-03-12T06:05:59.561 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_stats_pools", 2024-03-12T06:05:59.562 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.562 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.562 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.562 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.562 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.562 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.562 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.562 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.563 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.563 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.563 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.563 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-12T06:05:59.563 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-12T06:05:59.563 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.563 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.563 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.564 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "300", 2024-03-12T06:05:59.564 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.564 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.564 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.564 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.564 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.564 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.564 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.565 INFO:teuthology.orchestra.run.smithi076.stdout: "scrape_interval": { 2024-03-12T06:05:59.565 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "scrape_interval", 2024-03-12T06:05:59.565 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:59.565 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.565 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.565 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15.0", 2024-03-12T06:05:59.565 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.565 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.565 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.566 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.566 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.566 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.566 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.566 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-12T06:05:59.566 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-12T06:05:59.566 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.567 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.567 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.567 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "::", 2024-03-12T06:05:59.567 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.567 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.567 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.567 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-12T06:05:59.567 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.567 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.568 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.568 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-12T06:05:59.568 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-12T06:05:59.568 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.568 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.568 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.568 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "9283", 2024-03-12T06:05:59.568 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.569 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.569 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.569 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-12T06:05:59.569 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.569 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.569 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.569 INFO:teuthology.orchestra.run.smithi076.stdout: "stale_cache_strategy": { 2024-03-12T06:05:59.570 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "stale_cache_strategy", 2024-03-12T06:05:59.570 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.570 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.570 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.570 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "log", 2024-03-12T06:05:59.570 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.570 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.570 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.571 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.571 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.571 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.571 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.571 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_behaviour": { 2024-03-12T06:05:59.571 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_behaviour", 2024-03-12T06:05:59.572 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.572 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.572 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.572 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "default", 2024-03-12T06:05:59.572 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.572 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.572 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.572 INFO:teuthology.orchestra.run.smithi076.stdout: "default", 2024-03-12T06:05:59.573 INFO:teuthology.orchestra.run.smithi076.stdout: "error" 2024-03-12T06:05:59.573 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.573 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.573 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.573 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.573 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.573 INFO:teuthology.orchestra.run.smithi076.stdout: "standby_error_status_code": { 2024-03-12T06:05:59.573 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "standby_error_status_code", 2024-03-12T06:05:59.574 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.574 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.574 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.574 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "500", 2024-03-12T06:05:59.574 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "400", 2024-03-12T06:05:59.574 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "599", 2024-03-12T06:05:59.574 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.574 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.574 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.575 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.575 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.575 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.575 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.575 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.575 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_support", 2024-03-12T06:05:59.575 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.575 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.575 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.576 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.576 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.576 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.576 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.576 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.576 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.576 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.576 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.576 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.577 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.577 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.577 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.577 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.577 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.577 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.577 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.577 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.577 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.578 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.578 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.578 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.578 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.578 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.578 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.578 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.578 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.579 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.579 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.579 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.579 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.579 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.579 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.579 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.580 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.580 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.580 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.580 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.580 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.580 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.580 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.580 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.581 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.581 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.581 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.581 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.581 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.581 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.581 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.581 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.582 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.582 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.582 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.582 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.582 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.582 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.583 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.583 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.583 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.583 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.583 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.583 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.583 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.583 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.584 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.584 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.584 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.584 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.584 INFO:teuthology.orchestra.run.smithi076.stdout: "max_concurrent_snap_create": { 2024-03-12T06:05:59.584 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_concurrent_snap_create", 2024-03-12T06:05:59.584 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.584 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.584 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.585 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10", 2024-03-12T06:05:59.585 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.585 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.585 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.585 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.585 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.585 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.585 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.586 INFO:teuthology.orchestra.run.smithi076.stdout: "mirror_snapshot_schedule": { 2024-03-12T06:05:59.586 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mirror_snapshot_schedule", 2024-03-12T06:05:59.586 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.586 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.586 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.586 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.586 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.586 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.587 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.587 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.587 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.587 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.587 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.587 INFO:teuthology.orchestra.run.smithi076.stdout: "trash_purge_schedule": { 2024-03-12T06:05:59.587 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "trash_purge_schedule", 2024-03-12T06:05:59.587 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.588 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.588 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.588 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.588 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.588 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.588 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.588 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.588 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.589 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.589 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.589 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.589 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.589 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "restful", 2024-03-12T06:05:59.589 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.590 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.590 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.590 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_auth": { 2024-03-12T06:05:59.590 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enable_auth", 2024-03-12T06:05:59.590 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.590 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.590 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.591 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.591 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.591 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.591 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.591 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.591 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.591 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.591 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.593 INFO:teuthology.orchestra.run.smithi076.stdout: "key_file": { 2024-03-12T06:05:59.593 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "key_file", 2024-03-12T06:05:59.593 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.593 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.593 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.594 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.594 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.594 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.594 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.594 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.594 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.594 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.594 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.595 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.595 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.595 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.595 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.595 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.595 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.595 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.595 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.596 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.596 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.596 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.596 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.596 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.596 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.596 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.596 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.596 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.597 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.597 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.597 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.597 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.597 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.597 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.597 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.598 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.598 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.598 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.598 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.598 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.598 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.598 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.598 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.599 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.599 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.599 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.599 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.599 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.599 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.600 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.600 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.600 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.600 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.600 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.600 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.600 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.600 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.600 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.601 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.601 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.601 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.601 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.601 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.601 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.602 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: "server_addr": { 2024-03-12T06:05:59.603 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_addr", 2024-03-12T06:05:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.604 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: "server_port": { 2024-03-12T06:05:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "server_port", 2024-03-12T06:05:59.605 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.606 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rgw", 2024-03-12T06:05:59.607 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.608 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.608 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.608 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.608 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.608 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.608 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.608 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.609 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.609 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.609 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.609 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.609 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.609 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.609 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.610 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.610 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.610 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.610 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.610 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.610 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.610 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.611 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.611 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.611 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.611 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.611 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.611 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.611 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.612 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.612 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.612 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.612 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.612 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.612 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.612 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.613 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.613 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.613 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.613 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.613 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.613 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.613 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.614 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.614 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.614 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.614 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.614 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.614 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.614 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.614 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.615 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.615 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.615 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.615 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.615 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.615 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.616 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.616 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.616 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.616 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.616 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.616 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.616 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.616 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.617 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.617 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.617 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.617 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.617 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.617 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.618 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rook", 2024-03-12T06:05:59.618 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.618 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.618 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.618 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.618 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.618 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.618 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.619 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.619 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.619 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.619 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.619 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.619 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.619 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.619 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.620 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.620 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.620 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.620 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.620 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.620 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.620 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.620 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.621 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.621 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.621 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.621 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.621 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.621 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.621 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.621 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.622 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.622 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.622 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.622 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.622 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.622 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.622 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.623 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.623 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.623 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.623 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.623 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.623 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.623 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.623 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.623 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.624 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.624 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.624 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.624 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.624 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.624 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.624 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.624 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.625 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.625 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.625 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.625 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.625 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.625 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.625 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.625 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.626 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.626 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.626 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.626 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.626 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.626 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.626 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.627 INFO:teuthology.orchestra.run.smithi076.stdout: "storage_class": { 2024-03-12T06:05:59.627 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "storage_class", 2024-03-12T06:05:59.627 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.627 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.627 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.627 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "local", 2024-03-12T06:05:59.627 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.627 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.628 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.628 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-12T06:05:59.628 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.628 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.628 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.628 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.628 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.629 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "selftest", 2024-03-12T06:05:59.629 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.629 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.629 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.629 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.629 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.629 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.630 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.630 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.630 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.630 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.630 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.630 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.630 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.630 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.631 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.631 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.631 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.631 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.631 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.631 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.631 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.632 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.632 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.632 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.632 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.632 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.632 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.632 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.632 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.633 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.633 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.633 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.633 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.633 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.633 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.633 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.634 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.634 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.634 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.634 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.634 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.634 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.635 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.635 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.635 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.635 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.635 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.635 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.635 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.635 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.636 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.636 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.636 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.636 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.636 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.636 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.636 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.636 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.637 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.637 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.637 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.637 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.637 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.637 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.637 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.637 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.638 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.638 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.638 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.638 INFO:teuthology.orchestra.run.smithi076.stdout: "roption1": { 2024-03-12T06:05:59.638 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "roption1", 2024-03-12T06:05:59.638 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.638 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.639 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.639 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.639 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.639 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.639 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.639 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.639 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.639 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.639 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.640 INFO:teuthology.orchestra.run.smithi076.stdout: "roption2": { 2024-03-12T06:05:59.640 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "roption2", 2024-03-12T06:05:59.640 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.640 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.640 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.640 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "xyz", 2024-03-12T06:05:59.640 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.640 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.641 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.641 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.641 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.641 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.641 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.641 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption1": { 2024-03-12T06:05:59.641 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption1", 2024-03-12T06:05:59.641 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.642 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.642 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.642 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.642 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.642 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.642 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.642 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.642 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.643 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.643 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.643 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption2": { 2024-03-12T06:05:59.643 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption2", 2024-03-12T06:05:59.643 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.643 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.643 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.643 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.644 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.644 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.644 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.644 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.644 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.644 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.644 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.644 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption3": { 2024-03-12T06:05:59.644 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption3", 2024-03-12T06:05:59.645 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "float", 2024-03-12T06:05:59.645 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.645 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.645 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.645 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.645 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.645 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.645 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.646 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.646 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.646 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.646 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption4": { 2024-03-12T06:05:59.646 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption4", 2024-03-12T06:05:59.646 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.647 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.647 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.647 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.647 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.647 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.647 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.647 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.647 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.648 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.648 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.648 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption5": { 2024-03-12T06:05:59.648 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption5", 2024-03-12T06:05:59.648 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.648 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.648 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.649 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.649 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.649 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.649 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.649 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.649 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.650 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.650 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.650 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption6": { 2024-03-12T06:05:59.650 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption6", 2024-03-12T06:05:59.650 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.650 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.650 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.651 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.651 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.651 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.651 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.651 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.651 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.651 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.651 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.652 INFO:teuthology.orchestra.run.smithi076.stdout: "rwoption7": { 2024-03-12T06:05:59.652 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rwoption7", 2024-03-12T06:05:59.652 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.652 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.652 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.652 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.653 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "1", 2024-03-12T06:05:59.653 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "42", 2024-03-12T06:05:59.653 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.653 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.653 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.653 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.653 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.654 INFO:teuthology.orchestra.run.smithi076.stdout: "testkey": { 2024-03-12T06:05:59.654 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testkey", 2024-03-12T06:05:59.654 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.654 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.654 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.654 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.654 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.654 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.655 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.655 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.655 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.655 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.655 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.655 INFO:teuthology.orchestra.run.smithi076.stdout: "testlkey": { 2024-03-12T06:05:59.656 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testlkey", 2024-03-12T06:05:59.656 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.656 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.656 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.656 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.656 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.656 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.656 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.656 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.657 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.657 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.657 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.657 INFO:teuthology.orchestra.run.smithi076.stdout: "testnewline": { 2024-03-12T06:05:59.657 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "testnewline", 2024-03-12T06:05:59.657 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.657 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.657 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.658 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.658 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.658 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.658 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.658 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.658 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.658 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.658 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.658 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.659 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.659 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snap_schedule", 2024-03-12T06:05:59.659 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.659 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.659 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.659 INFO:teuthology.orchestra.run.smithi076.stdout: "allow_m_granularity": { 2024-03-12T06:05:59.660 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "allow_m_granularity", 2024-03-12T06:05:59.660 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.660 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.660 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.660 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.660 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.660 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.660 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.660 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "allow minute scheduled snapshots", 2024-03-12T06:05:59.661 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.661 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.661 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.661 INFO:teuthology.orchestra.run.smithi076.stdout: "dump_on_update": { 2024-03-12T06:05:59.661 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "dump_on_update", 2024-03-12T06:05:59.661 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.661 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.661 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.662 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.662 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.662 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.662 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.662 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "dump database to debug log on update", 2024-03-12T06:05:59.662 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.662 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.663 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.663 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.663 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.663 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.663 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.663 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.663 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.664 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.664 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.664 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.664 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.664 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.664 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.664 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.664 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.665 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.665 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.665 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.665 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.665 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.665 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.665 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.666 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.666 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.666 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.666 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.666 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.666 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.666 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.666 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.667 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.667 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.667 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.667 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.667 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.667 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.667 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.668 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.668 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.668 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.668 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.668 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.668 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.668 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.668 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.668 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.669 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.669 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.669 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.669 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.669 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.669 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.669 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.669 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.670 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.670 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.670 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.670 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.670 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.670 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.670 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.670 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.670 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.671 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.671 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.671 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.671 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.671 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.671 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.671 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "stats", 2024-03-12T06:05:59.671 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.672 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.672 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.672 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.672 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.672 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.672 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.672 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.672 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.672 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.673 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.673 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.673 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.673 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.673 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.673 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.673 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.673 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.674 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.674 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.674 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.674 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.674 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.674 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.674 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.676 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.676 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.676 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.676 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.676 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.676 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.676 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.676 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.676 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.677 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.677 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.677 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.677 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.677 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.677 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.677 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.677 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.677 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.678 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.678 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.678 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.678 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.678 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.678 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.679 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.679 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.679 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.679 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.679 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.679 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.679 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.679 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.680 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.680 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.680 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.680 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.680 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.680 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "status", 2024-03-12T06:05:59.680 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.681 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.681 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.681 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.681 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.681 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.681 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.681 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.681 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.681 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.682 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.682 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.682 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.682 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.682 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.682 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.682 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.682 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.682 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.683 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.683 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.683 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.683 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.683 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.683 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.683 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.683 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.683 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.684 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.684 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.684 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.684 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.684 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.684 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.684 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.684 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.685 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.685 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.685 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.685 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.685 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.685 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.685 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.685 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.686 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.686 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.686 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.686 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.686 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.686 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.686 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.686 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.687 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.687 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.687 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.687 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.687 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.687 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.687 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.688 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.688 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.688 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.688 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.688 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.688 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.688 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.688 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.688 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.689 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.689 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.689 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.689 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "telegraf", 2024-03-12T06:05:59.689 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.689 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.690 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.690 INFO:teuthology.orchestra.run.smithi076.stdout: "address": { 2024-03-12T06:05:59.690 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "address", 2024-03-12T06:05:59.690 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.690 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.690 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.690 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-12T06:05:59.691 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.691 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.691 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.691 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.691 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.691 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.691 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.691 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:59.692 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:59.692 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.692 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.692 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.692 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "15", 2024-03-12T06:05:59.692 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.692 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.692 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.692 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.693 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.693 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.693 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.693 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.693 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.693 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.693 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.694 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.694 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.694 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.694 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.694 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.694 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.694 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.695 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.695 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.695 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.695 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.695 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.695 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.695 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.695 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.695 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.696 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.696 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.696 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.696 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.696 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.696 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.696 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.696 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.697 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.697 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.697 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.697 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.697 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.697 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.697 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.697 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.698 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.698 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.698 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.698 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.698 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.698 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.698 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.698 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.698 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.699 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.699 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.699 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.699 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.699 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.699 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.699 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.699 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.699 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.700 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.700 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.700 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.700 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.700 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.700 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.700 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.700 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.701 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.701 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.701 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.701 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.701 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.701 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.702 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "telemetry", 2024-03-12T06:05:59.702 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.702 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.702 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.702 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_basic": { 2024-03-12T06:05:59.702 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_basic", 2024-03-12T06:05:59.702 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.702 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.703 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.703 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.703 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.703 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.703 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.703 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-12T06:05:59.703 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.703 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.703 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.704 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_crash": { 2024-03-12T06:05:59.704 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_crash", 2024-03-12T06:05:59.704 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.704 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.704 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.704 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.704 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.704 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.705 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.705 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-12T06:05:59.705 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.705 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.705 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.705 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_device": { 2024-03-12T06:05:59.706 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_device", 2024-03-12T06:05:59.706 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.706 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.706 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.706 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "True", 2024-03-12T06:05:59.706 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.706 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.706 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.707 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-12T06:05:59.707 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.707 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.707 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.707 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_ident": { 2024-03-12T06:05:59.707 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_ident", 2024-03-12T06:05:59.707 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.708 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.708 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.708 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.708 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.708 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.708 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.708 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-12T06:05:59.708 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.708 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.709 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.709 INFO:teuthology.orchestra.run.smithi076.stdout: "channel_perf": { 2024-03-12T06:05:59.709 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "channel_perf", 2024-03-12T06:05:59.709 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.709 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.709 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.709 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.709 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.709 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.710 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.710 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-12T06:05:59.710 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.710 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.710 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.710 INFO:teuthology.orchestra.run.smithi076.stdout: "contact": { 2024-03-12T06:05:59.710 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "contact", 2024-03-12T06:05:59.711 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.711 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.711 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.711 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.711 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.711 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.711 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.712 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.712 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.712 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.712 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.712 INFO:teuthology.orchestra.run.smithi076.stdout: "description": { 2024-03-12T06:05:59.712 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "description", 2024-03-12T06:05:59.713 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.713 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.713 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.713 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.713 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.713 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.713 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.713 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.714 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.714 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.714 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.714 INFO:teuthology.orchestra.run.smithi076.stdout: "device_url": { 2024-03-12T06:05:59.714 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "device_url", 2024-03-12T06:05:59.714 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.714 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.715 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.715 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-12T06:05:59.715 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.715 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.715 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.715 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.715 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.716 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.716 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.716 INFO:teuthology.orchestra.run.smithi076.stdout: "enabled": { 2024-03-12T06:05:59.716 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "enabled", 2024-03-12T06:05:59.716 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.716 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.716 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.717 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.717 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.717 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.717 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.717 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.717 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.717 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.717 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.718 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:59.718 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:59.718 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.718 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.718 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.718 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "24", 2024-03-12T06:05:59.718 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "8", 2024-03-12T06:05:59.719 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.719 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.719 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.719 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.719 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.719 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.719 INFO:teuthology.orchestra.run.smithi076.stdout: "last_opt_revision": { 2024-03-12T06:05:59.720 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "last_opt_revision", 2024-03-12T06:05:59.720 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.720 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.720 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.720 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "1", 2024-03-12T06:05:59.720 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.720 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.720 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.720 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.721 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.721 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.721 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.721 INFO:teuthology.orchestra.run.smithi076.stdout: "leaderboard": { 2024-03-12T06:05:59.721 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "leaderboard", 2024-03-12T06:05:59.721 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.721 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.721 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.722 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.722 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.722 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.722 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.722 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.722 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.722 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.722 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.723 INFO:teuthology.orchestra.run.smithi076.stdout: "leaderboard_description": { 2024-03-12T06:05:59.723 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "leaderboard_description", 2024-03-12T06:05:59.723 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.723 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.723 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.723 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.723 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.724 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.724 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.724 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.724 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.724 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.724 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.724 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.725 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.725 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.725 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.725 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.725 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.725 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.725 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.726 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.726 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.726 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.726 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.726 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.726 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.726 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.727 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.727 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.727 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.727 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.727 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.727 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.727 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.728 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.728 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.728 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.728 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.728 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.728 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.728 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.728 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.729 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.729 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.729 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.729 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.729 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.729 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.729 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.729 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.730 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.730 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.730 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.730 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.730 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.730 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.730 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.730 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.730 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.731 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.731 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.731 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.731 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.731 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.731 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.731 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.731 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.731 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.732 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.732 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.732 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.732 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.732 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.732 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.732 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.732 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.732 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.733 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.733 INFO:teuthology.orchestra.run.smithi076.stdout: "organization": { 2024-03-12T06:05:59.733 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "organization", 2024-03-12T06:05:59.733 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.733 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.733 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.733 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.733 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.734 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.734 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.734 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.734 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.734 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.734 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.734 INFO:teuthology.orchestra.run.smithi076.stdout: "proxy": { 2024-03-12T06:05:59.735 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "proxy", 2024-03-12T06:05:59.735 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.735 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.735 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.735 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.735 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.736 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.736 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.736 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.736 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.736 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.736 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.736 INFO:teuthology.orchestra.run.smithi076.stdout: "url": { 2024-03-12T06:05:59.736 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "url", 2024-03-12T06:05:59.737 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.737 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.737 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.737 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-12T06:05:59.737 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.737 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.737 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.737 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.738 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.738 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.738 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.738 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.738 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.739 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "test_orchestrator", 2024-03-12T06:05:59.739 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.739 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.739 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.739 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.739 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.739 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.739 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.740 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.740 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.740 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.740 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.740 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.740 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.740 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.741 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.741 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.741 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.741 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.741 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.741 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.741 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.741 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.742 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.742 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.742 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.742 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.742 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.742 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.743 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.743 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.743 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.743 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.743 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.743 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.743 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.743 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.744 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.744 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.744 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.744 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.744 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.744 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.745 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.745 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.745 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.745 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.745 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.745 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.745 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.745 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.746 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.746 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.746 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.746 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.746 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.746 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.746 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.747 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.747 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.747 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.747 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.747 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.747 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.747 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.747 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.748 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.748 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.748 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.748 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.748 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.748 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.749 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "volumes", 2024-03-12T06:05:59.749 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.749 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.749 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.749 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.749 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.749 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.750 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.750 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.750 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.750 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.750 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.750 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.750 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.750 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.751 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.751 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.751 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.751 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.751 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.751 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.751 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.751 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.751 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.752 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.752 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.752 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.752 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.752 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.752 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.752 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.752 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.753 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.753 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.753 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.753 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.753 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.753 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.754 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.754 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.754 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.754 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.754 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.754 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.754 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.754 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.754 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.755 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.755 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.755 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.755 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.755 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.755 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.755 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.755 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.755 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.756 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.756 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.756 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.756 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.756 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.756 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.756 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.756 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.757 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.757 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.757 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.758 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.758 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.758 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.758 INFO:teuthology.orchestra.run.smithi076.stdout: "max_concurrent_clones": { 2024-03-12T06:05:59.758 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "max_concurrent_clones", 2024-03-12T06:05:59.758 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.758 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.759 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.759 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "4", 2024-03-12T06:05:59.759 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.759 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.759 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.759 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-12T06:05:59.759 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.760 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.760 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.760 INFO:teuthology.orchestra.run.smithi076.stdout: "periodic_async_work": { 2024-03-12T06:05:59.760 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "periodic_async_work", 2024-03-12T06:05:59.760 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.760 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.760 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.760 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.760 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.761 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.761 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.761 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Periodically check for async work", 2024-03-12T06:05:59.761 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.761 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.761 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.761 INFO:teuthology.orchestra.run.smithi076.stdout: "snapshot_clone_delay": { 2024-03-12T06:05:59.761 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snapshot_clone_delay", 2024-03-12T06:05:59.762 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.762 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.762 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.762 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "0", 2024-03-12T06:05:59.762 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.762 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.762 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.762 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-12T06:05:59.763 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.763 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.763 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.763 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.763 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.763 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix", 2024-03-12T06:05:59.763 INFO:teuthology.orchestra.run.smithi076.stdout: "can_run": true, 2024-03-12T06:05:59.764 INFO:teuthology.orchestra.run.smithi076.stdout: "error_string": "", 2024-03-12T06:05:59.764 INFO:teuthology.orchestra.run.smithi076.stdout: "module_options": { 2024-03-12T06:05:59.764 INFO:teuthology.orchestra.run.smithi076.stdout: "discovery_interval": { 2024-03-12T06:05:59.764 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "discovery_interval", 2024-03-12T06:05:59.764 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "uint", 2024-03-12T06:05:59.764 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.764 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.764 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "100", 2024-03-12T06:05:59.764 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.765 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.765 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.765 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.765 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.765 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.765 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.765 INFO:teuthology.orchestra.run.smithi076.stdout: "identifier": { 2024-03-12T06:05:59.765 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "identifier", 2024-03-12T06:05:59.765 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.766 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.766 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.766 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.766 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.766 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.766 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.766 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.766 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.766 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.767 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.767 INFO:teuthology.orchestra.run.smithi076.stdout: "interval": { 2024-03-12T06:05:59.767 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "interval", 2024-03-12T06:05:59.767 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "secs", 2024-03-12T06:05:59.767 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.767 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.767 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "60", 2024-03-12T06:05:59.767 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.768 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.768 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.768 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.768 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.768 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.768 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.768 INFO:teuthology.orchestra.run.smithi076.stdout: "log_level": { 2024-03-12T06:05:59.768 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_level", 2024-03-12T06:05:59.768 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.769 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.769 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.769 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.769 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.769 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.769 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.769 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.769 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.770 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.770 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.770 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.770 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.770 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.770 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.770 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.770 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.770 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.771 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster": { 2024-03-12T06:05:59.771 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster", 2024-03-12T06:05:59.771 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.771 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.771 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.771 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.772 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.772 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.772 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.772 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.772 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.772 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.772 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_cluster_level": { 2024-03-12T06:05:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_cluster_level", 2024-03-12T06:05:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "info", 2024-03-12T06:05:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.773 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [ 2024-03-12T06:05:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "", 2024-03-12T06:05:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "critical", 2024-03-12T06:05:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "debug", 2024-03-12T06:05:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "error", 2024-03-12T06:05:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "info", 2024-03-12T06:05:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "warning" 2024-03-12T06:05:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.774 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": { 2024-03-12T06:05:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "log_to_file", 2024-03-12T06:05:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "bool", 2024-03-12T06:05:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2024-03-12T06:05:59.775 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "False", 2024-03-12T06:05:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_host": { 2024-03-12T06:05:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_host", 2024-03-12T06:05:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "", 2024-03-12T06:05:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.777 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_port": { 2024-03-12T06:05:59.778 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_port", 2024-03-12T06:05:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "int", 2024-03-12T06:05:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "10051", 2024-03-12T06:05:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "zabbix_sender": { 2024-03-12T06:05:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "zabbix_sender", 2024-03-12T06:05:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "str", 2024-03-12T06:05:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "level": "advanced", 2024-03-12T06:05:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 0, 2024-03-12T06:05:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-12T06:05:59.780 INFO:teuthology.orchestra.run.smithi076.stdout: "min": "", 2024-03-12T06:05:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "max": "", 2024-03-12T06:05:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "enum_allowed": [], 2024-03-12T06:05:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "desc": "", 2024-03-12T06:05:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "long_desc": "", 2024-03-12T06:05:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "tags": [], 2024-03-12T06:05:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: "see_also": [] 2024-03-12T06:05:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.781 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "services": { 2024-03-12T06:05:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "dashboard": "https://172.21.15.76:8443/" 2024-03-12T06:05:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "always_on_modules": { 2024-03-12T06:05:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "octopus": [ 2024-03-12T06:05:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "balancer", 2024-03-12T06:05:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "crash", 2024-03-12T06:05:59.782 INFO:teuthology.orchestra.run.smithi076.stdout: "devicehealth", 2024-03-12T06:05:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator", 2024-03-12T06:05:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscaler", 2024-03-12T06:05:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "progress", 2024-03-12T06:05:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_support", 2024-03-12T06:05:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "status", 2024-03-12T06:05:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "telemetry", 2024-03-12T06:05:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: "volumes" 2024-03-12T06:05:59.783 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "pacific": [ 2024-03-12T06:05:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "balancer", 2024-03-12T06:05:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "crash", 2024-03-12T06:05:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "devicehealth", 2024-03-12T06:05:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator", 2024-03-12T06:05:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscaler", 2024-03-12T06:05:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "progress", 2024-03-12T06:05:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_support", 2024-03-12T06:05:59.784 INFO:teuthology.orchestra.run.smithi076.stdout: "status", 2024-03-12T06:05:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "telemetry", 2024-03-12T06:05:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "volumes" 2024-03-12T06:05:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "quincy": [ 2024-03-12T06:05:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "balancer", 2024-03-12T06:05:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "crash", 2024-03-12T06:05:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "devicehealth", 2024-03-12T06:05:59.785 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator", 2024-03-12T06:05:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscaler", 2024-03-12T06:05:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "progress", 2024-03-12T06:05:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_support", 2024-03-12T06:05:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "status", 2024-03-12T06:05:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "telemetry", 2024-03-12T06:05:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "volumes" 2024-03-12T06:05:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "reef": [ 2024-03-12T06:05:59.786 INFO:teuthology.orchestra.run.smithi076.stdout: "balancer", 2024-03-12T06:05:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "crash", 2024-03-12T06:05:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "devicehealth", 2024-03-12T06:05:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator", 2024-03-12T06:05:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscaler", 2024-03-12T06:05:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "progress", 2024-03-12T06:05:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_support", 2024-03-12T06:05:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "status", 2024-03-12T06:05:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "telemetry", 2024-03-12T06:05:59.787 INFO:teuthology.orchestra.run.smithi076.stdout: "volumes" 2024-03-12T06:05:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2024-03-12T06:05:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "squid": [ 2024-03-12T06:05:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "balancer", 2024-03-12T06:05:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "crash", 2024-03-12T06:05:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "devicehealth", 2024-03-12T06:05:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "orchestrator", 2024-03-12T06:05:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscaler", 2024-03-12T06:05:59.788 INFO:teuthology.orchestra.run.smithi076.stdout: "progress", 2024-03-12T06:05:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_support", 2024-03-12T06:05:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "status", 2024-03-12T06:05:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "telemetry", 2024-03-12T06:05:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "volumes" 2024-03-12T06:05:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-12T06:05:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "last_failure_osd_epoch": 89, 2024-03-12T06:05:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: "active_clients": [ 2024-03-12T06:05:59.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "devicehealth", 2024-03-12T06:05:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-03-12T06:05:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-03-12T06:05:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:0", 2024-03-12T06:05:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 4190937605 2024-03-12T06:05:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.790 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-12T06:05:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "libcephsqlite", 2024-03-12T06:05:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-03-12T06:05:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-03-12T06:05:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:0", 2024-03-12T06:05:59.793 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2762103459 2024-03-12T06:05:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-12T06:05:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "rbd_support", 2024-03-12T06:05:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-03-12T06:05:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.794 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-03-12T06:05:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:0", 2024-03-12T06:05:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1249515779 2024-03-12T06:05:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-12T06:05:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snap_schedule", 2024-03-12T06:05:59.795 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-03-12T06:05:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-03-12T06:05:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:0", 2024-03-12T06:05:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1858430017 2024-03-12T06:05:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-12T06:05:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.796 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "snap_schedule", 2024-03-12T06:05:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-03-12T06:05:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-03-12T06:05:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:0", 2024-03-12T06:05:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3967554079 2024-03-12T06:05:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-12T06:05:59.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:05:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "volumes", 2024-03-12T06:05:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2024-03-12T06:05:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2024-03-12T06:05:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2024-03-12T06:05:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:0", 2024-03-12T06:05:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 2268309672 2024-03-12T06:05:59.798 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-12T06:05:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:05:59.799 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-03-12T06:05:59.799 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:05:59.806 DEBUG:tasks.check_counter:Getting stats from a 2024-03-12T06:05:59.806 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-12T06:05:59.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:57 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2035306009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T06:05:59.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:57 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3394628853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T06:05:59.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:57 smithi076 ceph-mon[32095]: pgmap v314: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-12T06:05:59.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:57 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/14223536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T06:05:59.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:57 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3738249257' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-12T06:05:59.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:05:59 smithi076 ceph-mon[32095]: pgmap v315: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T06:05:59.809 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:57 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2035306009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T06:05:59.809 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:57 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3394628853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T06:05:59.809 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:57 smithi088 ceph-mon[36585]: pgmap v314: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-12T06:05:59.809 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:57 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/14223536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T06:05:59.810 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:57 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3738249257' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-12T06:05:59.810 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:05:59 smithi088 ceph-mon[36585]: pgmap v315: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T06:05:59.810 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:57 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2035306009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T06:05:59.810 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:57 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3394628853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T06:05:59.810 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:57 smithi082 ceph-mon[36382]: pgmap v314: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-12T06:05:59.810 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:57 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/14223536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T06:05:59.811 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:57 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3738249257' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-12T06:05:59.811 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:05:59 smithi082 ceph-mon[36382]: pgmap v315: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T06:06:00.083 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T06:06:00.459 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:00 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:06:00.633 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:00 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:06:00.636 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:00 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T06:06:01.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:01 smithi076 ceph-mon[32095]: pgmap v316: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-12T06:06:01.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:01 smithi082 ceph-mon[36382]: pgmap v316: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-12T06:06:01.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:01 smithi088 ceph-mon[36585]: pgmap v316: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-12T06:06:02.423 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:06:02.423 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T06:06:02.423 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 0, 2024-03-12T06:06:02.424 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 2, 2024-03-12T06:06:02.424 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 0, 2024-03-12T06:06:02.424 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 0, 2024-03-12T06:06:02.424 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 4, 2024-03-12T06:06:02.424 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 0, 2024-03-12T06:06:02.424 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.001818809, 2024-03-12T06:06:02.424 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.000000000, 2024-03-12T06:06:02.424 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.000281540, 2024-03-12T06:06:02.425 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T06:06:02.425 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:02.425 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:02.425 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:02.425 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:02.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:02.425 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:02.425 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:02.426 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:02.426 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:02.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:02.426 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-12T06:06:02.426 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 0 2024-03-12T06:06:02.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:02.426 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T06:06:02.427 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 214, 2024-03-12T06:06:02.427 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 203, 2024-03-12T06:06:02.427 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 1561424, 2024-03-12T06:06:02.427 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 121664, 2024-03-12T06:06:02.427 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 2, 2024-03-12T06:06:02.427 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 1, 2024-03-12T06:06:02.427 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.042263005, 2024-03-12T06:06:02.428 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.013353588, 2024-03-12T06:06:02.428 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.033006492, 2024-03-12T06:06:02.428 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.004200100, 2024-03-12T06:06:02.428 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:02.428 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 202, 2024-03-12T06:06:02.428 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.010919120, 2024-03-12T06:06:02.428 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000054055 2024-03-12T06:06:02.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:02.429 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:02.429 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:02.429 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:02.429 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:02.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:02.430 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 1561424, 2024-03-12T06:06:02.430 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 121664 2024-03-12T06:06:02.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:02.430 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T06:06:02.430 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 6, 2024-03-12T06:06:02.430 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 157, 2024-03-12T06:06:02.430 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 3312, 2024-03-12T06:06:02.430 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 182432, 2024-03-12T06:06:02.431 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 5, 2024-03-12T06:06:02.431 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 3, 2024-03-12T06:06:02.431 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.014147728, 2024-03-12T06:06:02.431 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.009468287, 2024-03-12T06:06:02.431 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.002315987, 2024-03-12T06:06:02.431 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T06:06:02.431 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:02.431 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 155, 2024-03-12T06:06:02.432 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.012763626, 2024-03-12T06:06:02.432 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000082345 2024-03-12T06:06:02.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:02.432 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:02.432 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:02.432 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:02.432 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:02.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:02.432 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-12T06:06:02.433 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 182432 2024-03-12T06:06:02.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:02.433 INFO:teuthology.orchestra.run.smithi076.stdout: "cct": { 2024-03-12T06:06:02.433 INFO:teuthology.orchestra.run.smithi076.stdout: "total_workers": 0, 2024-03-12T06:06:02.433 INFO:teuthology.orchestra.run.smithi076.stdout: "unhealthy_workers": 0 2024-03-12T06:06:02.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:02.433 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool": { 2024-03-12T06:06:02.433 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_bytes": 0, 2024-03-12T06:06:02.433 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_items": 0, 2024-03-12T06:06:02.434 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T06:06:02.434 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_items": 0, 2024-03-12T06:06:02.434 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T06:06:02.434 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_items": 0, 2024-03-12T06:06:02.434 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T06:06:02.434 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T06:06:02.434 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T06:06:02.434 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T06:06:02.435 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T06:06:02.435 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_items": 0, 2024-03-12T06:06:02.435 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T06:06:02.435 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T06:06:02.435 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_bytes": 0, 2024-03-12T06:06:02.435 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_items": 0, 2024-03-12T06:06:02.435 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_bytes": 0, 2024-03-12T06:06:02.435 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_items": 0, 2024-03-12T06:06:02.435 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T06:06:02.436 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T06:06:02.436 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T06:06:02.436 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T06:06:02.436 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T06:06:02.436 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_items": 0, 2024-03-12T06:06:02.436 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_bytes": 0, 2024-03-12T06:06:02.436 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_items": 0, 2024-03-12T06:06:02.436 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T06:06:02.437 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T06:06:02.437 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_bytes": 0, 2024-03-12T06:06:02.437 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_items": 0, 2024-03-12T06:06:02.437 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_bytes": 0, 2024-03-12T06:06:02.437 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_items": 0, 2024-03-12T06:06:02.437 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T06:06:02.437 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_items": 0, 2024-03-12T06:06:02.438 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T06:06:02.438 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_items": 0, 2024-03-12T06:06:02.438 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_bytes": 150025, 2024-03-12T06:06:02.438 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_items": 32, 2024-03-12T06:06:02.438 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_bytes": 0, 2024-03-12T06:06:02.438 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_items": 0, 2024-03-12T06:06:02.438 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bytes": 0, 2024-03-12T06:06:02.438 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_items": 0, 2024-03-12T06:06:02.438 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_bytes": 0, 2024-03-12T06:06:02.439 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_items": 0, 2024-03-12T06:06:02.439 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_bytes": 0, 2024-03-12T06:06:02.439 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_items": 0, 2024-03-12T06:06:02.439 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_bytes": 0, 2024-03-12T06:06:02.439 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_items": 0, 2024-03-12T06:06:02.439 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T06:06:02.439 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_items": 0, 2024-03-12T06:06:02.439 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_bytes": 0, 2024-03-12T06:06:02.440 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_items": 0, 2024-03-12T06:06:02.440 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_bytes": 928, 2024-03-12T06:06:02.440 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_items": 2, 2024-03-12T06:06:02.440 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_bytes": 0, 2024-03-12T06:06:02.440 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_items": 0, 2024-03-12T06:06:02.440 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_bytes": 0, 2024-03-12T06:06:02.440 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_items": 0 2024-03-12T06:06:02.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:02.441 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T06:06:02.441 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-03-12T06:06:02.441 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-03-12T06:06:02.441 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-03-12T06:06:02.441 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 216, 2024-03-12T06:06:02.441 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 1531268, 2024-03-12T06:06:02.441 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:02.441 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 216, 2024-03-12T06:06:02.442 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-03-12T06:06:02.442 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-03-12T06:06:02.442 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 216, 2024-03-12T06:06:02.442 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 1531268, 2024-03-12T06:06:02.442 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-03-12T06:06:02.442 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:02.442 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:02.442 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:02.443 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:06:02.443 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:06:02.443 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:06:02.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:02.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:02.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:02.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:02.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi088 to 174.1M 2024-03-12T06:06:02.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: Unable to set osd_memory_target on smithi088 to 182634188: error parsing value: Value '182634188' is below minimum 939524096 2024-03-12T06:06:02.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi082 to 174.1M 2024-03-12T06:06:02.698 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:02 smithi082 ceph-mon[36382]: Unable to set osd_memory_target on smithi082 to 182611251: error parsing value: Value '182611251' is below minimum 939524096 2024-03-12T06:06:02.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:02.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:02.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:02.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:02.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi088 to 174.1M 2024-03-12T06:06:02.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: Unable to set osd_memory_target on smithi088 to 182634188: error parsing value: Value '182634188' is below minimum 939524096 2024-03-12T06:06:02.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.737 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi082 to 174.1M 2024-03-12T06:06:02.738 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:02 smithi088 ceph-mon[36585]: Unable to set osd_memory_target on smithi082 to 182611251: error parsing value: Value '182611251' is below minimum 939524096 2024-03-12T06:06:02.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:02.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:02.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:02.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:02.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi088 to 174.1M 2024-03-12T06:06:02.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: Unable to set osd_memory_target on smithi088 to 182634188: error parsing value: Value '182634188' is below minimum 939524096 2024-03-12T06:06:02.741 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.741 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.741 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.741 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:02.741 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi082 to 174.1M 2024-03-12T06:06:02.741 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:02 smithi076 ceph-mon[32095]: Unable to set osd_memory_target on smithi082 to 182611251: error parsing value: Value '182611251' is below minimum 939524096 2024-03-12T06:06:03.132 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-12T06:06:03.132 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-12T06:06:03.132 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-12T06:06:03.132 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-12T06:06:03.133 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-12T06:06:03.133 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-03-12T06:06:03.133 DEBUG:tasks.check_counter:Getting stats from d 2024-03-12T06:06:03.133 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-12T06:06:03.403 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T06:06:03.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:03 smithi076 ceph-mon[32095]: pgmap v317: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 27 op/s 2024-03-12T06:06:03.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:03 smithi082 ceph-mon[36382]: pgmap v317: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 27 op/s 2024-03-12T06:06:03.735 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:03 smithi088 ceph-mon[36585]: pgmap v317: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 27 op/s 2024-03-12T06:06:05.122 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:06:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T06:06:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 4, 2024-03-12T06:06:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 4, 2024-03-12T06:06:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 2992, 2024-03-12T06:06:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 448, 2024-03-12T06:06:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 4, 2024-03-12T06:06:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 0, 2024-03-12T06:06:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.001468509, 2024-03-12T06:06:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.000038360, 2024-03-12T06:06:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.000569001, 2024-03-12T06:06:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T06:06:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 3, 2024-03-12T06:06:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000899610, 2024-03-12T06:06:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000299870 2024-03-12T06:06:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-12T06:06:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 448 2024-03-12T06:06:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T06:06:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 2, 2024-03-12T06:06:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 159, 2024-03-12T06:06:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 320, 2024-03-12T06:06:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 182688, 2024-03-12T06:06:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 5, 2024-03-12T06:06:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 3, 2024-03-12T06:06:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.013041313, 2024-03-12T06:06:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.008865909, 2024-03-12T06:06:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.001771083, 2024-03-12T06:06:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T06:06:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 156, 2024-03-12T06:06:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.014801784, 2024-03-12T06:06:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000094883 2024-03-12T06:06:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-12T06:06:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 182688 2024-03-12T06:06:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T06:06:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 219, 2024-03-12T06:06:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 208, 2024-03-12T06:06:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 1562784, 2024-03-12T06:06:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 124784, 2024-03-12T06:06:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 2, 2024-03-12T06:06:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 1, 2024-03-12T06:06:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.041804809, 2024-03-12T06:06:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.013473688, 2024-03-12T06:06:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.033976740, 2024-03-12T06:06:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.004058883, 2024-03-12T06:06:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 207, 2024-03-12T06:06:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.009502386, 2024-03-12T06:06:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000045905 2024-03-12T06:06:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 1562784, 2024-03-12T06:06:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 124784 2024-03-12T06:06:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "cct": { 2024-03-12T06:06:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "total_workers": 0, 2024-03-12T06:06:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "unhealthy_workers": 0 2024-03-12T06:06:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool": { 2024-03-12T06:06:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_bytes": 0, 2024-03-12T06:06:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_items": 0, 2024-03-12T06:06:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T06:06:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_items": 0, 2024-03-12T06:06:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T06:06:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_items": 0, 2024-03-12T06:06:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T06:06:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T06:06:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T06:06:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T06:06:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T06:06:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_items": 0, 2024-03-12T06:06:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T06:06:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T06:06:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_bytes": 0, 2024-03-12T06:06:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_items": 0, 2024-03-12T06:06:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_bytes": 0, 2024-03-12T06:06:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_items": 0, 2024-03-12T06:06:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T06:06:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T06:06:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T06:06:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T06:06:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T06:06:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_items": 0, 2024-03-12T06:06:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_bytes": 0, 2024-03-12T06:06:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_items": 0, 2024-03-12T06:06:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T06:06:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T06:06:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_bytes": 0, 2024-03-12T06:06:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_items": 0, 2024-03-12T06:06:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_bytes": 0, 2024-03-12T06:06:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_items": 0, 2024-03-12T06:06:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T06:06:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_items": 0, 2024-03-12T06:06:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T06:06:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_items": 0, 2024-03-12T06:06:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_bytes": 148955, 2024-03-12T06:06:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_items": 30, 2024-03-12T06:06:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_bytes": 0, 2024-03-12T06:06:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_items": 0, 2024-03-12T06:06:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bytes": 0, 2024-03-12T06:06:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_items": 0, 2024-03-12T06:06:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_bytes": 0, 2024-03-12T06:06:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_items": 0, 2024-03-12T06:06:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_bytes": 0, 2024-03-12T06:06:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_items": 0, 2024-03-12T06:06:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_bytes": 0, 2024-03-12T06:06:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_items": 0, 2024-03-12T06:06:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T06:06:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_items": 0, 2024-03-12T06:06:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_bytes": 0, 2024-03-12T06:06:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_items": 0, 2024-03-12T06:06:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_bytes": 928, 2024-03-12T06:06:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_items": 2, 2024-03-12T06:06:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_bytes": 0, 2024-03-12T06:06:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_items": 0, 2024-03-12T06:06:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_bytes": 0, 2024-03-12T06:06:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_items": 0 2024-03-12T06:06:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T06:06:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-03-12T06:06:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-03-12T06:06:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-03-12T06:06:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 221, 2024-03-12T06:06:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 1531918, 2024-03-12T06:06:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 221, 2024-03-12T06:06:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-03-12T06:06:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-03-12T06:06:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 221, 2024-03-12T06:06:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 1531918, 2024-03-12T06:06:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-03-12T06:06:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:06:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:06:05.140 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:06:05.833 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-12T06:06:05.833 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-12T06:06:05.834 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-12T06:06:05.834 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-12T06:06:05.834 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-12T06:06:05.834 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-12T06:06:05.834 DEBUG:tasks.check_counter:Getting stats from g 2024-03-12T06:06:05.834 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-12T06:06:06.074 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T06:06:06.196 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:05 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:06.196 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:05 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:06.196 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:05 smithi076 ceph-mon[32095]: pgmap v318: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:06:06.196 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:05 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:06.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:05 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:06.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:05 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:06.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:05 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:06.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:05 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:06:06.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:05 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:06:06.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:05 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:06.197 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:05 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:06.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:05 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:06.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:05 smithi082 ceph-mon[36382]: pgmap v318: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:06:06.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:05 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:06.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:05 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:06.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:05 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:06.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:05 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:06.198 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:05 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:06:06.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:05 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:06:06.199 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:05 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:06.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:05 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:06.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:05 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:06.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:05 smithi088 ceph-mon[36585]: pgmap v318: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:06:06.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:05 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:06.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:05 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:06.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:05 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:06.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:05 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-12T06:06:06.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:05 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T06:06:06.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:05 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T06:06:06.237 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:05 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' 2024-03-12T06:06:07.196 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:06 smithi082 ceph-mon[36382]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-03-12T06:06:07.196 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:06 smithi082 ceph-mon[36382]: Unable to set osd_memory_target on smithi076 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-03-12T06:06:07.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:06 smithi088 ceph-mon[36585]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-03-12T06:06:07.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:06 smithi088 ceph-mon[36585]: Unable to set osd_memory_target on smithi076 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-03-12T06:06:07.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:06 smithi076 ceph-mon[32095]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-03-12T06:06:07.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:06 smithi076 ceph-mon[32095]: Unable to set osd_memory_target on smithi076 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-03-12T06:06:07.818 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:06:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T06:06:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 216, 2024-03-12T06:06:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 205, 2024-03-12T06:06:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 1561968, 2024-03-12T06:06:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 122912, 2024-03-12T06:06:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 4, 2024-03-12T06:06:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 0, 2024-03-12T06:06:07.819 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.044112623, 2024-03-12T06:06:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.013652841, 2024-03-12T06:06:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.034591034, 2024-03-12T06:06:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.003929068, 2024-03-12T06:06:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 204, 2024-03-12T06:06:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.010637177, 2024-03-12T06:06:07.820 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000052143 2024-03-12T06:06:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 1561968, 2024-03-12T06:06:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 122912 2024-03-12T06:06:07.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T06:06:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 2, 2024-03-12T06:06:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 156, 2024-03-12T06:06:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 320, 2024-03-12T06:06:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 182160, 2024-03-12T06:06:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 5, 2024-03-12T06:06:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 3, 2024-03-12T06:06:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.014806503, 2024-03-12T06:06:07.822 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.010483731, 2024-03-12T06:06:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.001831354, 2024-03-12T06:06:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T06:06:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 153, 2024-03-12T06:06:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.009989019, 2024-03-12T06:06:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000065287 2024-03-12T06:06:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:07.823 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-12T06:06:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 182160 2024-03-12T06:06:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T06:06:07.824 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 4, 2024-03-12T06:06:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 4, 2024-03-12T06:06:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 2992, 2024-03-12T06:06:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 448, 2024-03-12T06:06:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 2, 2024-03-12T06:06:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 1, 2024-03-12T06:06:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.000634025, 2024-03-12T06:06:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.000033117, 2024-03-12T06:06:07.825 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.000568683, 2024-03-12T06:06:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T06:06:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 3, 2024-03-12T06:06:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000665196, 2024-03-12T06:06:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000221732 2024-03-12T06:06:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:07.826 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-12T06:06:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 448 2024-03-12T06:06:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "cct": { 2024-03-12T06:06:07.827 INFO:teuthology.orchestra.run.smithi076.stdout: "total_workers": 0, 2024-03-12T06:06:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "unhealthy_workers": 0 2024-03-12T06:06:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool": { 2024-03-12T06:06:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_bytes": 0, 2024-03-12T06:06:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_items": 0, 2024-03-12T06:06:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T06:06:07.828 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_items": 0, 2024-03-12T06:06:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T06:06:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_items": 0, 2024-03-12T06:06:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T06:06:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T06:06:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T06:06:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T06:06:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T06:06:07.829 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_items": 0, 2024-03-12T06:06:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T06:06:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T06:06:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_bytes": 0, 2024-03-12T06:06:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_items": 0, 2024-03-12T06:06:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_bytes": 0, 2024-03-12T06:06:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_items": 0, 2024-03-12T06:06:07.830 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T06:06:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T06:06:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T06:06:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T06:06:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T06:06:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_items": 0, 2024-03-12T06:06:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_bytes": 0, 2024-03-12T06:06:07.831 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_items": 0, 2024-03-12T06:06:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T06:06:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T06:06:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_bytes": 0, 2024-03-12T06:06:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_items": 0, 2024-03-12T06:06:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_bytes": 0, 2024-03-12T06:06:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_items": 0, 2024-03-12T06:06:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T06:06:07.832 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_items": 0, 2024-03-12T06:06:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T06:06:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_items": 0, 2024-03-12T06:06:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_bytes": 148994, 2024-03-12T06:06:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_items": 31, 2024-03-12T06:06:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_bytes": 0, 2024-03-12T06:06:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_items": 0, 2024-03-12T06:06:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bytes": 0, 2024-03-12T06:06:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_items": 0, 2024-03-12T06:06:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_bytes": 0, 2024-03-12T06:06:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_items": 0, 2024-03-12T06:06:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_bytes": 0, 2024-03-12T06:06:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_items": 0, 2024-03-12T06:06:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_bytes": 0, 2024-03-12T06:06:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_items": 0, 2024-03-12T06:06:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T06:06:07.834 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_items": 0, 2024-03-12T06:06:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_bytes": 0, 2024-03-12T06:06:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_items": 0, 2024-03-12T06:06:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_bytes": 928, 2024-03-12T06:06:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_items": 2, 2024-03-12T06:06:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_bytes": 0, 2024-03-12T06:06:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_items": 0, 2024-03-12T06:06:07.835 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_bytes": 0, 2024-03-12T06:06:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_items": 0 2024-03-12T06:06:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T06:06:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-03-12T06:06:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-03-12T06:06:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-03-12T06:06:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 218, 2024-03-12T06:06:07.836 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 1531528, 2024-03-12T06:06:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 218, 2024-03-12T06:06:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-03-12T06:06:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-03-12T06:06:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 218, 2024-03-12T06:06:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 1531528, 2024-03-12T06:06:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-03-12T06:06:07.837 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:06:07.838 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:06:07.838 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:06:08.196 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:07 smithi082 ceph-mon[36382]: pgmap v319: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:06:08.236 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:07 smithi088 ceph-mon[36585]: pgmap v319: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:06:08.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:07 smithi076 ceph-mon[32095]: pgmap v319: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-12T06:06:08.604 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-12T06:06:08.604 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-12T06:06:08.604 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-12T06:06:08.604 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-12T06:06:08.604 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-12T06:06:08.604 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-03-12T06:06:08.604 DEBUG:tasks.check_counter:Getting stats from j 2024-03-12T06:06:08.605 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-12T06:06:08.848 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T06:06:09.196 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:08 smithi082 ceph-mon[36382]: osdmap e96: 12 total, 12 up, 12 in 2024-03-12T06:06:09.230 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:08 smithi076 ceph-mon[32095]: osdmap e96: 12 total, 12 up, 12 in 2024-03-12T06:06:09.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:08 smithi088 ceph-mon[36585]: osdmap e96: 12 total, 12 up, 12 in 2024-03-12T06:06:10.178 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:09 smithi076 ceph-mon[32095]: pgmap v321: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:06:10.196 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:09 smithi082 ceph-mon[36382]: pgmap v321: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:06:10.235 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:09 smithi088 ceph-mon[36585]: pgmap v321: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:06:10.613 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-03-12T06:06:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T06:06:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 4, 2024-03-12T06:06:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 4, 2024-03-12T06:06:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 2992, 2024-03-12T06:06:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 448, 2024-03-12T06:06:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 4, 2024-03-12T06:06:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 0, 2024-03-12T06:06:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.001328857, 2024-03-12T06:06:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.000031843, 2024-03-12T06:06:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.000475181, 2024-03-12T06:06:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T06:06:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 3, 2024-03-12T06:06:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000686808, 2024-03-12T06:06:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000228936 2024-03-12T06:06:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-12T06:06:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 448 2024-03-12T06:06:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T06:06:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 221, 2024-03-12T06:06:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 210, 2024-03-12T06:06:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 1563328, 2024-03-12T06:06:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 126032, 2024-03-12T06:06:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 2, 2024-03-12T06:06:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 1, 2024-03-12T06:06:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.042942015, 2024-03-12T06:06:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.013357695, 2024-03-12T06:06:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.034219080, 2024-03-12T06:06:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.004355610, 2024-03-12T06:06:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 209, 2024-03-12T06:06:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.011147234, 2024-03-12T06:06:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000053336 2024-03-12T06:06:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 1563328, 2024-03-12T06:06:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 126032 2024-03-12T06:06:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T06:06:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_messages": 2, 2024-03-12T06:06:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages": 161, 2024-03-12T06:06:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_bytes": 320, 2024-03-12T06:06:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_bytes": 183040, 2024-03-12T06:06:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_created_connections": 5, 2024-03-12T06:06:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_active_connections": 3, 2024-03-12T06:06:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_total_time": 0.013553260, 2024-03-12T06:06:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_send_time": 0.009483548, 2024-03-12T06:06:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_recv_time": 0.001839685, 2024-03-12T06:06:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T06:06:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 158, 2024-03-12T06:06:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.011867790, 2024-03-12T06:06:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000075112 2024-03-12T06:06:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-12T06:06:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "msgr_send_encrypted_bytes": 183040 2024-03-12T06:06:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "cct": { 2024-03-12T06:06:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "total_workers": 0, 2024-03-12T06:06:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "unhealthy_workers": 0 2024-03-12T06:06:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool": { 2024-03-12T06:06:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_bytes": 0, 2024-03-12T06:06:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "bloom_filter_items": 0, 2024-03-12T06:06:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T06:06:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_items": 0, 2024-03-12T06:06:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T06:06:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_data_items": 0, 2024-03-12T06:06:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T06:06:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T06:06:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T06:06:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T06:06:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T06:06:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_other_items": 0, 2024-03-12T06:06:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T06:06:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T06:06:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_bytes": 0, 2024-03-12T06:06:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_items": 0, 2024-03-12T06:06:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_bytes": 0, 2024-03-12T06:06:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blob_items": 0, 2024-03-12T06:06:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T06:06:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T06:06:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T06:06:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T06:06:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T06:06:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_items": 0, 2024-03-12T06:06:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_bytes": 0, 2024-03-12T06:06:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_txc_items": 0, 2024-03-12T06:06:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T06:06:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T06:06:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_bytes": 0, 2024-03-12T06:06:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_writing_items": 0, 2024-03-12T06:06:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_bytes": 0, 2024-03-12T06:06:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_items": 0, 2024-03-12T06:06:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T06:06:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_reader_items": 0, 2024-03-12T06:06:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T06:06:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_file_writer_items": 0, 2024-03-12T06:06:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_bytes": 148955, 2024-03-12T06:06:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_anon_items": 30, 2024-03-12T06:06:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_bytes": 0, 2024-03-12T06:06:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "buffer_meta_items": 0, 2024-03-12T06:06:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bytes": 0, 2024-03-12T06:06:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_items": 0, 2024-03-12T06:06:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_bytes": 0, 2024-03-12T06:06:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mapbl_items": 0, 2024-03-12T06:06:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_bytes": 0, 2024-03-12T06:06:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pglog_items": 0, 2024-03-12T06:06:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_bytes": 0, 2024-03-12T06:06:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_items": 0, 2024-03-12T06:06:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T06:06:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "osdmap_mapping_items": 0, 2024-03-12T06:06:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_bytes": 0, 2024-03-12T06:06:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "pgmap_items": 0, 2024-03-12T06:06:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_bytes": 928, 2024-03-12T06:06:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_co_items": 2, 2024-03-12T06:06:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_bytes": 0, 2024-03-12T06:06:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_1_items": 0, 2024-03-12T06:06:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_bytes": 0, 2024-03-12T06:06:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "unittest_2_items": 0 2024-03-12T06:06:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2024-03-12T06:06:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T06:06:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "val": 0, 2024-03-12T06:06:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "max": 104857600, 2024-03-12T06:06:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "get_started": 0, 2024-03-12T06:06:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "get": 223, 2024-03-12T06:06:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "get_sum": 1532178, 2024-03-12T06:06:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "get_or_fail_success": 223, 2024-03-12T06:06:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "take": 0, 2024-03-12T06:06:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "take_sum": 0, 2024-03-12T06:06:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "put": 223, 2024-03-12T06:06:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "put_sum": 1532178, 2024-03-12T06:06:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "wait": { 2024-03-12T06:06:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "avgcount": 0, 2024-03-12T06:06:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "sum": 0.000000000, 2024-03-12T06:06:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "avgtime": 0.000000000 2024-03-12T06:06:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:06:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: } 2024-03-12T06:06:10.633 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-03-12T06:06:11.348 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-12T06:06:11.348 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-12T06:06:11.348 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-12T06:06:11.348 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-12T06:06:11.348 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-12T06:06:11.348 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-12T06:06:11.348 DEBUG:tasks.check_counter:Getting stats from b 2024-03-12T06:06:11.349 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-12T06:06:11.597 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T06:06:11.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:11 smithi082 ceph-mon[36382]: pgmap v322: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:06:11.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:11 smithi088 ceph-mon[36585]: pgmap v322: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:06:11.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:11 smithi076 ceph-mon[32095]: pgmap v322: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-12T06:06:13.239 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-12T06:06:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T06:06:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 9803, 2024-03-12T06:06:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 6206, 2024-03-12T06:06:13.239 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 5873340, 2024-03-12T06:06:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 19399152, 2024-03-12T06:06:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 167, 2024-03-12T06:06:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 3, 2024-03-12T06:06:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.602210758, 2024-03-12T06:06:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.214528668, 2024-03-12T06:06:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.294376377, 2024-03-12T06:06:13.240 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.114734780, 2024-03-12T06:06:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 6208, 2024-03-12T06:06:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.384373405, 2024-03-12T06:06:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000061915 2024-03-12T06:06:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 8299, 2024-03-12T06:06:13.241 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000911277, 2024-03-12T06:06:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000109 2024-03-12T06:06:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 1684768, 2024-03-12T06:06:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 159440 2024-03-12T06:06:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T06:06:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 24526, 2024-03-12T06:06:13.242 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 44199, 2024-03-12T06:06:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 7555429, 2024-03-12T06:06:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 35938972, 2024-03-12T06:06:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 15, 2024-03-12T06:06:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 7, 2024-03-12T06:06:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 1.377949432, 2024-03-12T06:06:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.635767521, 2024-03-12T06:06:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 1.925413657, 2024-03-12T06:06:13.243 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.168408852, 2024-03-12T06:06:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 44193, 2024-03-12T06:06:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 1.242571505, 2024-03-12T06:06:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000028116 2024-03-12T06:06:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 36664, 2024-03-12T06:06:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.006527803, 2024-03-12T06:06:13.244 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000178 2024-03-12T06:06:13.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.245 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 99776, 2024-03-12T06:06:13.245 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 475872 2024-03-12T06:06:13.245 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.245 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T06:06:13.245 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 8886, 2024-03-12T06:06:13.245 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 6767, 2024-03-12T06:06:13.245 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 2597742, 2024-03-12T06:06:13.245 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 17353184, 2024-03-12T06:06:13.246 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 12, 2024-03-12T06:06:13.246 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 7, 2024-03-12T06:06:13.246 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.627305651, 2024-03-12T06:06:13.246 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.255785087, 2024-03-12T06:06:13.246 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.222917731, 2024-03-12T06:06:13.246 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.155223337, 2024-03-12T06:06:13.246 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:13.246 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 6766, 2024-03-12T06:06:13.246 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.452562769, 2024-03-12T06:06:13.247 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000066887 2024-03-12T06:06:13.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.247 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:13.247 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 6824, 2024-03-12T06:06:13.247 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000952350, 2024-03-12T06:06:13.247 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000139 2024-03-12T06:06:13.247 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.247 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 323696, 2024-03-12T06:06:13.248 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 96576 2024-03-12T06:06:13.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.248 INFO:teuthology.orchestra.run.smithi082.stdout: "cct": { 2024-03-12T06:06:13.248 INFO:teuthology.orchestra.run.smithi082.stdout: "total_workers": 1, 2024-03-12T06:06:13.248 INFO:teuthology.orchestra.run.smithi082.stdout: "unhealthy_workers": 0 2024-03-12T06:06:13.248 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.248 INFO:teuthology.orchestra.run.smithi082.stdout: "finisher-MDSRank": { 2024-03-12T06:06:13.248 INFO:teuthology.orchestra.run.smithi082.stdout: "queue_len": 0, 2024-03-12T06:06:13.248 INFO:teuthology.orchestra.run.smithi082.stdout: "complete_latency": { 2024-03-12T06:06:13.249 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 5780, 2024-03-12T06:06:13.249 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 12.368188179, 2024-03-12T06:06:13.249 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.002139824 2024-03-12T06:06:13.249 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:13.249 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.249 INFO:teuthology.orchestra.run.smithi082.stdout: "finisher-PurgeQueue": { 2024-03-12T06:06:13.249 INFO:teuthology.orchestra.run.smithi082.stdout: "queue_len": 0, 2024-03-12T06:06:13.249 INFO:teuthology.orchestra.run.smithi082.stdout: "complete_latency": { 2024-03-12T06:06:13.250 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 4148, 2024-03-12T06:06:13.250 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 1.125458744, 2024-03-12T06:06:13.250 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000271325 2024-03-12T06:06:13.250 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:13.250 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.250 INFO:teuthology.orchestra.run.smithi082.stdout: "mds": { 2024-03-12T06:06:13.251 INFO:teuthology.orchestra.run.smithi082.stdout: "request": 11148, 2024-03-12T06:06:13.251 INFO:teuthology.orchestra.run.smithi082.stdout: "reply": 11060, 2024-03-12T06:06:13.251 INFO:teuthology.orchestra.run.smithi082.stdout: "reply_latency": { 2024-03-12T06:06:13.251 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 11060, 2024-03-12T06:06:13.251 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 36.863631177, 2024-03-12T06:06:13.251 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.003333058 2024-03-12T06:06:13.252 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.252 INFO:teuthology.orchestra.run.smithi082.stdout: "slow_reply": 0, 2024-03-12T06:06:13.252 INFO:teuthology.orchestra.run.smithi082.stdout: "forward": 88, 2024-03-12T06:06:13.252 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_fetch_complete": 0, 2024-03-12T06:06:13.252 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_fetch_keys": 0, 2024-03-12T06:06:13.252 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_commit": 212, 2024-03-12T06:06:13.253 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_split": 24, 2024-03-12T06:06:13.253 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_merge": 2, 2024-03-12T06:06:13.253 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes": 2004, 2024-03-12T06:06:13.253 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_top": 962, 2024-03-12T06:06:13.253 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_bottom": 901, 2024-03-12T06:06:13.253 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_pin_tail": 141, 2024-03-12T06:06:13.253 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_pinned": 629, 2024-03-12T06:06:13.253 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_expired": 4404, 2024-03-12T06:06:13.253 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_with_caps": 55, 2024-03-12T06:06:13.254 INFO:teuthology.orchestra.run.smithi082.stdout: "caps": 55, 2024-03-12T06:06:13.254 INFO:teuthology.orchestra.run.smithi082.stdout: "subtrees": 83, 2024-03-12T06:06:13.254 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse": 12699, 2024-03-12T06:06:13.254 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_hit": 12010, 2024-03-12T06:06:13.254 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_forward": 87, 2024-03-12T06:06:13.254 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_discover": 0, 2024-03-12T06:06:13.254 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_dir_fetch": 0, 2024-03-12T06:06:13.254 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_remote_ino": 0, 2024-03-12T06:06:13.255 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_lock": 3, 2024-03-12T06:06:13.255 INFO:teuthology.orchestra.run.smithi082.stdout: "load_cent": 149, 2024-03-12T06:06:13.255 INFO:teuthology.orchestra.run.smithi082.stdout: "q": 0, 2024-03-12T06:06:13.255 INFO:teuthology.orchestra.run.smithi082.stdout: "exported": 98, 2024-03-12T06:06:13.255 INFO:teuthology.orchestra.run.smithi082.stdout: "exported_inodes": 4846, 2024-03-12T06:06:13.255 INFO:teuthology.orchestra.run.smithi082.stdout: "imported": 15, 2024-03-12T06:06:13.255 INFO:teuthology.orchestra.run.smithi082.stdout: "imported_inodes": 485, 2024-03-12T06:06:13.255 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_dir_fetch": 0, 2024-03-12T06:06:13.255 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_backtrace_fetch": 8215, 2024-03-12T06:06:13.256 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_peer_discover": 0, 2024-03-12T06:06:13.256 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rfiles": 3452, 2024-03-12T06:06:13.256 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rbytes": 26524023, 2024-03-12T06:06:13.256 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rsnaps": 7, 2024-03-12T06:06:13.256 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_backtrace_fetch": 8215, 2024-03-12T06:06:13.256 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_set_tag": 0, 2024-03-12T06:06:13.256 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T06:06:13.256 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_inotable_repaired": 0, 2024-03-12T06:06:13.256 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dir_inodes": 481, 2024-03-12T06:06:13.257 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dir_base_inodes": 20, 2024-03-12T06:06:13.257 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dirfrag_rstats": 461, 2024-03-12T06:06:13.257 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_file_inodes": 7734, 2024-03-12T06:06:13.257 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_inode_file_caps": 4, 2024-03-12T06:06:13.257 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_revoke": 119, 2024-03-12T06:06:13.257 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_grant": 9492, 2024-03-12T06:06:13.257 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_trunc": 8, 2024-03-12T06:06:13.257 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T06:06:13.257 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T06:06:13.258 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_caps": 7273, 2024-03-12T06:06:13.258 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_caps_dirty": 5967, 2024-03-12T06:06:13.258 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_cap_release": 460, 2024-03-12T06:06:13.258 INFO:teuthology.orchestra.run.smithi082.stdout: "process_request_cap_release": 4282 2024-03-12T06:06:13.258 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.258 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_cache": { 2024-03-12T06:06:13.258 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_update": 0, 2024-03-12T06:06:13.258 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_update_receipt": 0, 2024-03-12T06:06:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_try_discover": 0, 2024-03-12T06:06:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_send_discover": 0, 2024-03-12T06:06:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_handle_discover": 88, 2024-03-12T06:06:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays": 0, 2024-03-12T06:06:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays_delayed": 0, 2024-03-12T06:06:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays_enqueuing": 0, 2024-03-12T06:06:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_created": 2751, 2024-03-12T06:06:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_enqueued": 2749, 2024-03-12T06:06:13.259 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_reintegrated": 2, 2024-03-12T06:06:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_migrated": 0, 2024-03-12T06:06:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_processing": 0, 2024-03-12T06:06:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_enqueued": 0, 2024-03-12T06:06:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_prioritized": 0, 2024-03-12T06:06:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: "recovery_started": 0, 2024-03-12T06:06:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: "recovery_completed": 0, 2024-03-12T06:06:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_enqueue_scrub": 20, 2024-03-12T06:06:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_exportdir": 98, 2024-03-12T06:06:13.260 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_flush": 0, 2024-03-12T06:06:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_fragmentdir": 26, 2024-03-12T06:06:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_fragstats": 0, 2024-03-12T06:06:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_inodestats": 0 2024-03-12T06:06:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_log": { 2024-03-12T06:06:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: "evlrg": 0, 2024-03-12T06:06:13.261 INFO:teuthology.orchestra.run.smithi082.stdout: "evadd": 19897, 2024-03-12T06:06:13.262 INFO:teuthology.orchestra.run.smithi082.stdout: "evex": 0, 2024-03-12T06:06:13.262 INFO:teuthology.orchestra.run.smithi082.stdout: "evtrm": 0, 2024-03-12T06:06:13.262 INFO:teuthology.orchestra.run.smithi082.stdout: "ev": 19897, 2024-03-12T06:06:13.262 INFO:teuthology.orchestra.run.smithi082.stdout: "evexg": 0, 2024-03-12T06:06:13.262 INFO:teuthology.orchestra.run.smithi082.stdout: "evexd": 0, 2024-03-12T06:06:13.262 INFO:teuthology.orchestra.run.smithi082.stdout: "segadd": 25, 2024-03-12T06:06:13.263 INFO:teuthology.orchestra.run.smithi082.stdout: "segex": 0, 2024-03-12T06:06:13.263 INFO:teuthology.orchestra.run.smithi082.stdout: "segtrm": 0, 2024-03-12T06:06:13.263 INFO:teuthology.orchestra.run.smithi082.stdout: "seg": 25, 2024-03-12T06:06:13.263 INFO:teuthology.orchestra.run.smithi082.stdout: "segmjr": 3, 2024-03-12T06:06:13.263 INFO:teuthology.orchestra.run.smithi082.stdout: "segexg": 0, 2024-03-12T06:06:13.263 INFO:teuthology.orchestra.run.smithi082.stdout: "segexd": 0, 2024-03-12T06:06:13.263 INFO:teuthology.orchestra.run.smithi082.stdout: "expos": 4194304, 2024-03-12T06:06:13.263 INFO:teuthology.orchestra.run.smithi082.stdout: "wrpos": 50576201, 2024-03-12T06:06:13.264 INFO:teuthology.orchestra.run.smithi082.stdout: "rdpos": 0, 2024-03-12T06:06:13.264 INFO:teuthology.orchestra.run.smithi082.stdout: "jlat": { 2024-03-12T06:06:13.264 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 609, 2024-03-12T06:06:13.264 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 9.788314108, 2024-03-12T06:06:13.264 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.016072765 2024-03-12T06:06:13.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.264 INFO:teuthology.orchestra.run.smithi082.stdout: "replayed": 0 2024-03-12T06:06:13.264 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.265 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_mem": { 2024-03-12T06:06:13.265 INFO:teuthology.orchestra.run.smithi082.stdout: "ino": 1414, 2024-03-12T06:06:13.265 INFO:teuthology.orchestra.run.smithi082.stdout: "ino+": 7120, 2024-03-12T06:06:13.265 INFO:teuthology.orchestra.run.smithi082.stdout: "ino-": 5706, 2024-03-12T06:06:13.265 INFO:teuthology.orchestra.run.smithi082.stdout: "dir": 183, 2024-03-12T06:06:13.265 INFO:teuthology.orchestra.run.smithi082.stdout: "dir+": 294, 2024-03-12T06:06:13.265 INFO:teuthology.orchestra.run.smithi082.stdout: "dir-": 111, 2024-03-12T06:06:13.265 INFO:teuthology.orchestra.run.smithi082.stdout: "dn": 2004, 2024-03-12T06:06:13.265 INFO:teuthology.orchestra.run.smithi082.stdout: "dn+": 8823, 2024-03-12T06:06:13.266 INFO:teuthology.orchestra.run.smithi082.stdout: "dn-": 6819, 2024-03-12T06:06:13.266 INFO:teuthology.orchestra.run.smithi082.stdout: "cap": 55, 2024-03-12T06:06:13.266 INFO:teuthology.orchestra.run.smithi082.stdout: "cap+": 6256, 2024-03-12T06:06:13.266 INFO:teuthology.orchestra.run.smithi082.stdout: "cap-": 6201, 2024-03-12T06:06:13.266 INFO:teuthology.orchestra.run.smithi082.stdout: "rss": 93680, 2024-03-12T06:06:13.266 INFO:teuthology.orchestra.run.smithi082.stdout: "heap": 215420 2024-03-12T06:06:13.266 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.266 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_server": { 2024-03-12T06:06:13.267 INFO:teuthology.orchestra.run.smithi082.stdout: "dispatch_client_request": 11471, 2024-03-12T06:06:13.267 INFO:teuthology.orchestra.run.smithi082.stdout: "dispatch_server_request": 22, 2024-03-12T06:06:13.267 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_request": 11148, 2024-03-12T06:06:13.267 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_session": 131, 2024-03-12T06:06:13.267 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_peer_request": 44, 2024-03-12T06:06:13.267 INFO:teuthology.orchestra.run.smithi082.stdout: "req_create_latency": { 2024-03-12T06:06:13.267 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 6155, 2024-03-12T06:06:13.267 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 18.883839701, 2024-03-12T06:06:13.267 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.003068048 2024-03-12T06:06:13.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.268 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getattr_latency": { 2024-03-12T06:06:13.268 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 153, 2024-03-12T06:06:13.268 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.211974678, 2024-03-12T06:06:13.268 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.001385455 2024-03-12T06:06:13.268 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.269 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getfilelock_latency": { 2024-03-12T06:06:13.269 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.269 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.269 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.269 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.269 INFO:teuthology.orchestra.run.smithi082.stdout: "req_link_latency": { 2024-03-12T06:06:13.270 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 2, 2024-03-12T06:06:13.270 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.002258631, 2024-03-12T06:06:13.270 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.001129315 2024-03-12T06:06:13.270 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.270 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookup_latency": { 2024-03-12T06:06:13.270 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 1275, 2024-03-12T06:06:13.270 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 4.945701343, 2024-03-12T06:06:13.271 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.003878981 2024-03-12T06:06:13.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.271 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookuphash_latency": { 2024-03-12T06:06:13.271 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.271 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.271 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.271 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.272 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupino_latency": { 2024-03-12T06:06:13.272 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 1, 2024-03-12T06:06:13.272 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000191654, 2024-03-12T06:06:13.272 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000191654 2024-03-12T06:06:13.272 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.272 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupname_latency": { 2024-03-12T06:06:13.272 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.273 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.273 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.273 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.273 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupparent_latency": { 2024-03-12T06:06:13.273 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.273 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.274 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.274 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.274 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupsnap_latency": { 2024-03-12T06:06:13.274 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.274 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.274 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.275 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.275 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lssnap_latency": { 2024-03-12T06:06:13.275 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 11, 2024-03-12T06:06:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.001832359, 2024-03-12T06:06:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000166578 2024-03-12T06:06:13.276 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mkdir_latency": { 2024-03-12T06:06:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 84, 2024-03-12T06:06:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 3.181326808, 2024-03-12T06:06:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.037872938 2024-03-12T06:06:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mknod_latency": { 2024-03-12T06:06:13.277 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mksnap_latency": { 2024-03-12T06:06:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 11, 2024-03-12T06:06:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.164231455, 2024-03-12T06:06:13.278 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.014930132 2024-03-12T06:06:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: "req_open_latency": { 2024-03-12T06:06:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 438, 2024-03-12T06:06:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.100539155, 2024-03-12T06:06:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000229541 2024-03-12T06:06:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: "req_readdir_latency": { 2024-03-12T06:06:13.279 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 100, 2024-03-12T06:06:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.746044255, 2024-03-12T06:06:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.007460442 2024-03-12T06:06:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rename_latency": { 2024-03-12T06:06:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 81, 2024-03-12T06:06:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.150255555, 2024-03-12T06:06:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.001855006 2024-03-12T06:06:13.280 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: "req_renamesnap_latency": { 2024-03-12T06:06:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: "req_snapdiff_latency": { 2024-03-12T06:06:13.281 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmdir_latency": { 2024-03-12T06:06:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 22, 2024-03-12T06:06:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.793061082, 2024-03-12T06:06:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.036048231 2024-03-12T06:06:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.282 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmsnap_latency": { 2024-03-12T06:06:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 4, 2024-03-12T06:06:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.017066744, 2024-03-12T06:06:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.004266686 2024-03-12T06:06:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmxattr_latency": { 2024-03-12T06:06:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.283 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setattr_latency": { 2024-03-12T06:06:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 9, 2024-03-12T06:06:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.045885322, 2024-03-12T06:06:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.005098369 2024-03-12T06:06:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.284 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setdirlayout_latency": { 2024-03-12T06:06:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setfilelock_latency": { 2024-03-12T06:06:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 8, 2024-03-12T06:06:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.003410923, 2024-03-12T06:06:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000426365 2024-03-12T06:06:13.285 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setlayout_latency": { 2024-03-12T06:06:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setxattr_latency": { 2024-03-12T06:06:13.286 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 7, 2024-03-12T06:06:13.289 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.019275150, 2024-03-12T06:06:13.289 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.002753592 2024-03-12T06:06:13.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.290 INFO:teuthology.orchestra.run.smithi082.stdout: "req_symlink_latency": { 2024-03-12T06:06:13.290 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 7, 2024-03-12T06:06:13.290 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.004996946, 2024-03-12T06:06:13.290 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000713849 2024-03-12T06:06:13.290 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.290 INFO:teuthology.orchestra.run.smithi082.stdout: "req_unlink_latency": { 2024-03-12T06:06:13.290 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 2692, 2024-03-12T06:06:13.291 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 7.591739416, 2024-03-12T06:06:13.291 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.002820111 2024-03-12T06:06:13.291 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.291 INFO:teuthology.orchestra.run.smithi082.stdout: "cap_revoke_eviction": 0, 2024-03-12T06:06:13.291 INFO:teuthology.orchestra.run.smithi082.stdout: "cap_acquisition_throttle": 0, 2024-03-12T06:06:13.291 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getvxattr_latency": { 2024-03-12T06:06:13.291 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.291 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.292 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.292 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:13.292 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.292 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_sessions": { 2024-03-12T06:06:13.292 INFO:teuthology.orchestra.run.smithi082.stdout: "session_count": 2, 2024-03-12T06:06:13.292 INFO:teuthology.orchestra.run.smithi082.stdout: "session_add": 5, 2024-03-12T06:06:13.292 INFO:teuthology.orchestra.run.smithi082.stdout: "session_remove": 3, 2024-03-12T06:06:13.292 INFO:teuthology.orchestra.run.smithi082.stdout: "sessions_open": 2, 2024-03-12T06:06:13.293 INFO:teuthology.orchestra.run.smithi082.stdout: "sessions_stale": 0, 2024-03-12T06:06:13.293 INFO:teuthology.orchestra.run.smithi082.stdout: "total_load": 55, 2024-03-12T06:06:13.293 INFO:teuthology.orchestra.run.smithi082.stdout: "average_load": 27, 2024-03-12T06:06:13.293 INFO:teuthology.orchestra.run.smithi082.stdout: "avg_session_uptime": 625, 2024-03-12T06:06:13.293 INFO:teuthology.orchestra.run.smithi082.stdout: "mdthresh_evicted": 0 2024-03-12T06:06:13.293 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.293 INFO:teuthology.orchestra.run.smithi082.stdout: "mempool": { 2024-03-12T06:06:13.293 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_bytes": 0, 2024-03-12T06:06:13.294 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_items": 0, 2024-03-12T06:06:13.294 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T06:06:13.294 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_items": 0, 2024-03-12T06:06:13.294 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T06:06:13.294 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_items": 0, 2024-03-12T06:06:13.294 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T06:06:13.294 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T06:06:13.294 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T06:06:13.295 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T06:06:13.295 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T06:06:13.295 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_items": 0, 2024-03-12T06:06:13.295 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T06:06:13.295 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T06:06:13.295 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_bytes": 0, 2024-03-12T06:06:13.296 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_items": 0, 2024-03-12T06:06:13.296 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_bytes": 0, 2024-03-12T06:06:13.296 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_items": 0, 2024-03-12T06:06:13.296 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T06:06:13.296 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T06:06:13.296 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T06:06:13.296 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T06:06:13.296 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T06:06:13.296 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_items": 0, 2024-03-12T06:06:13.297 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_bytes": 0, 2024-03-12T06:06:13.297 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_items": 0, 2024-03-12T06:06:13.297 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T06:06:13.297 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T06:06:13.297 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_bytes": 0, 2024-03-12T06:06:13.297 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_items": 0, 2024-03-12T06:06:13.297 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_bytes": 0, 2024-03-12T06:06:13.297 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_items": 0, 2024-03-12T06:06:13.298 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T06:06:13.298 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_items": 0, 2024-03-12T06:06:13.298 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T06:06:13.298 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_items": 0, 2024-03-12T06:06:13.298 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_bytes": 356705, 2024-03-12T06:06:13.298 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_items": 1499, 2024-03-12T06:06:13.298 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_bytes": 1232, 2024-03-12T06:06:13.298 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_items": 14, 2024-03-12T06:06:13.298 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_bytes": 0, 2024-03-12T06:06:13.298 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_items": 0, 2024-03-12T06:06:13.299 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_bytes": 0, 2024-03-12T06:06:13.299 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_items": 0, 2024-03-12T06:06:13.299 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_bytes": 0, 2024-03-12T06:06:13.299 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_items": 0, 2024-03-12T06:06:13.299 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_bytes": 8152, 2024-03-12T06:06:13.299 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_items": 170, 2024-03-12T06:06:13.299 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T06:06:13.299 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_items": 0, 2024-03-12T06:06:13.299 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_bytes": 0, 2024-03-12T06:06:13.300 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_items": 0, 2024-03-12T06:06:13.300 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_bytes": 5055910, 2024-03-12T06:06:13.300 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_items": 64011, 2024-03-12T06:06:13.300 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_bytes": 0, 2024-03-12T06:06:13.300 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_items": 0, 2024-03-12T06:06:13.300 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_bytes": 0, 2024-03-12T06:06:13.300 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_items": 0 2024-03-12T06:06:13.300 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.300 INFO:teuthology.orchestra.run.smithi082.stdout: "objecter": { 2024-03-12T06:06:13.300 INFO:teuthology.orchestra.run.smithi082.stdout: "op_active": 0, 2024-03-12T06:06:13.301 INFO:teuthology.orchestra.run.smithi082.stdout: "op_laggy": 0, 2024-03-12T06:06:13.301 INFO:teuthology.orchestra.run.smithi082.stdout: "op_send": 12640, 2024-03-12T06:06:13.301 INFO:teuthology.orchestra.run.smithi082.stdout: "op_send_bytes": 49979665, 2024-03-12T06:06:13.301 INFO:teuthology.orchestra.run.smithi082.stdout: "op_resend": 31, 2024-03-12T06:06:13.301 INFO:teuthology.orchestra.run.smithi082.stdout: "op_reply": 12640, 2024-03-12T06:06:13.301 INFO:teuthology.orchestra.run.smithi082.stdout: "op_latency": { 2024-03-12T06:06:13.302 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 12640, 2024-03-12T06:06:13.302 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 32.087377479, 2024-03-12T06:06:13.302 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.002538558 2024-03-12T06:06:13.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.302 INFO:teuthology.orchestra.run.smithi082.stdout: "op_inflight": 0, 2024-03-12T06:06:13.302 INFO:teuthology.orchestra.run.smithi082.stdout: "oplen_avg": { 2024-03-12T06:06:13.302 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 12640, 2024-03-12T06:06:13.302 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 12939 2024-03-12T06:06:13.302 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.303 INFO:teuthology.orchestra.run.smithi082.stdout: "op": 12640, 2024-03-12T06:06:13.303 INFO:teuthology.orchestra.run.smithi082.stdout: "op_r": 8317, 2024-03-12T06:06:13.303 INFO:teuthology.orchestra.run.smithi082.stdout: "op_w": 4323, 2024-03-12T06:06:13.303 INFO:teuthology.orchestra.run.smithi082.stdout: "op_rmw": 0, 2024-03-12T06:06:13.303 INFO:teuthology.orchestra.run.smithi082.stdout: "op_pg": 0, 2024-03-12T06:06:13.303 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_stat": 0, 2024-03-12T06:06:13.303 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_create": 11, 2024-03-12T06:06:13.304 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_read": 82, 2024-03-12T06:06:13.304 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_write": 671, 2024-03-12T06:06:13.304 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_writefull": 52, 2024-03-12T06:06:13.304 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_writesame": 0, 2024-03-12T06:06:13.304 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_append": 0, 2024-03-12T06:06:13.304 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_zero": 0, 2024-03-12T06:06:13.304 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_truncate": 8, 2024-03-12T06:06:13.305 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_delete": 3304, 2024-03-12T06:06:13.305 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_mapext": 0, 2024-03-12T06:06:13.305 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_sparse_read": 0, 2024-03-12T06:06:13.305 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_clonerange": 0, 2024-03-12T06:06:13.305 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_getxattr": 8235, 2024-03-12T06:06:13.305 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_setxattr": 22, 2024-03-12T06:06:13.305 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_cmpxattr": 0, 2024-03-12T06:06:13.305 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_rmxattr": 0, 2024-03-12T06:06:13.306 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_resetxattrs": 0, 2024-03-12T06:06:13.306 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_call": 0, 2024-03-12T06:06:13.306 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_watch": 0, 2024-03-12T06:06:13.306 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_notify": 0, 2024-03-12T06:06:13.306 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T06:06:13.306 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_pgls": 0, 2024-03-12T06:06:13.307 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_pgls_filter": 0, 2024-03-12T06:06:13.307 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_other": 8, 2024-03-12T06:06:13.307 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_active": 0, 2024-03-12T06:06:13.307 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_send": 0, 2024-03-12T06:06:13.307 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_resend": 0, 2024-03-12T06:06:13.307 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_ping": 0, 2024-03-12T06:06:13.307 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_active": 0, 2024-03-12T06:06:13.307 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_send": 0, 2024-03-12T06:06:13.308 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_resend": 0, 2024-03-12T06:06:13.308 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_active": 0, 2024-03-12T06:06:13.308 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_send": 0, 2024-03-12T06:06:13.308 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_resend": 0, 2024-03-12T06:06:13.308 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_active": 0, 2024-03-12T06:06:13.308 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_send": 0, 2024-03-12T06:06:13.308 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_resend": 0, 2024-03-12T06:06:13.308 INFO:teuthology.orchestra.run.smithi082.stdout: "command_active": 0, 2024-03-12T06:06:13.309 INFO:teuthology.orchestra.run.smithi082.stdout: "command_send": 0, 2024-03-12T06:06:13.309 INFO:teuthology.orchestra.run.smithi082.stdout: "command_resend": 0, 2024-03-12T06:06:13.309 INFO:teuthology.orchestra.run.smithi082.stdout: "map_epoch": 96, 2024-03-12T06:06:13.309 INFO:teuthology.orchestra.run.smithi082.stdout: "map_full": 0, 2024-03-12T06:06:13.309 INFO:teuthology.orchestra.run.smithi082.stdout: "map_inc": 11, 2024-03-12T06:06:13.309 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_sessions": 12, 2024-03-12T06:06:13.310 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_session_open": 12, 2024-03-12T06:06:13.310 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_session_close": 0, 2024-03-12T06:06:13.310 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_laggy": 0, 2024-03-12T06:06:13.310 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_wr": 511, 2024-03-12T06:06:13.310 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_rd": 0, 2024-03-12T06:06:13.310 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_del": 35 2024-03-12T06:06:13.310 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.311 INFO:teuthology.orchestra.run.smithi082.stdout: "oft": { 2024-03-12T06:06:13.311 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_objs": 1, 2024-03-12T06:06:13.311 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_kv_pairs": 0, 2024-03-12T06:06:13.311 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_updates": 0, 2024-03-12T06:06:13.311 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_removes": 0 2024-03-12T06:06:13.311 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.312 INFO:teuthology.orchestra.run.smithi082.stdout: "osd-slow-ops": { 2024-03-12T06:06:13.312 INFO:teuthology.orchestra.run.smithi082.stdout: "slow_ops_count": 0 2024-03-12T06:06:13.312 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.312 INFO:teuthology.orchestra.run.smithi082.stdout: "purge_queue": { 2024-03-12T06:06:13.312 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_ops": 0, 2024-03-12T06:06:13.312 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_ops_high_water": 10, 2024-03-12T06:06:13.312 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing": 0, 2024-03-12T06:06:13.312 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_high_water": 10, 2024-03-12T06:06:13.313 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executed_ops": 2749, 2024-03-12T06:06:13.313 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executed": 2749, 2024-03-12T06:06:13.313 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_item_in_journal": 0 2024-03-12T06:06:13.313 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.313 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T06:06:13.313 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:13.313 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-12T06:06:13.313 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:13.314 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 43211, 2024-03-12T06:06:13.314 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 12379863, 2024-03-12T06:06:13.314 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:13.314 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 43211, 2024-03-12T06:06:13.314 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-12T06:06:13.314 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-12T06:06:13.314 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 43211, 2024-03-12T06:06:13.314 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 12379863, 2024-03-12T06:06:13.315 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:13.315 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.315 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.315 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.315 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:13.315 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.315 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-objecter_bytes": { 2024-03-12T06:06:13.315 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:13.315 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-12T06:06:13.316 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:13.316 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-12T06:06:13.316 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-12T06:06:13.316 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:13.316 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:13.316 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 12640, 2024-03-12T06:06:13.316 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 50258378, 2024-03-12T06:06:13.317 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 12640, 2024-03-12T06:06:13.317 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 50258378, 2024-03-12T06:06:13.317 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:13.317 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.317 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.317 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.317 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:13.317 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.318 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-objecter_ops": { 2024-03-12T06:06:13.318 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:13.318 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 1024, 2024-03-12T06:06:13.318 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:13.318 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-12T06:06:13.318 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-12T06:06:13.318 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:13.318 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:13.319 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 12640, 2024-03-12T06:06:13.319 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 12640, 2024-03-12T06:06:13.319 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 12640, 2024-03-12T06:06:13.319 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 12640, 2024-03-12T06:06:13.319 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:13.319 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.319 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.319 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.320 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:13.320 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.320 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-write_buf_throttle": { 2024-03-12T06:06:13.320 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:13.320 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 3758096384, 2024-03-12T06:06:13.320 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:13.320 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 2749, 2024-03-12T06:06:13.320 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 278713, 2024-03-12T06:06:13.321 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:13.321 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 2749, 2024-03-12T06:06:13.321 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-12T06:06:13.321 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-12T06:06:13.321 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 62, 2024-03-12T06:06:13.321 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 278713, 2024-03-12T06:06:13.321 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:13.321 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.322 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.322 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.322 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:13.322 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:13.322 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-write_buf_throttle-0x55dcf4912640": { 2024-03-12T06:06:13.322 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:13.322 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 3758096384, 2024-03-12T06:06:13.322 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:13.323 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 19897, 2024-03-12T06:06:13.323 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 46381897, 2024-03-12T06:06:13.323 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:13.323 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 19897, 2024-03-12T06:06:13.323 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-12T06:06:13.323 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-12T06:06:13.323 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 609, 2024-03-12T06:06:13.323 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 46381897, 2024-03-12T06:06:13.323 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:13.324 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:13.324 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:13.324 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:13.324 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:13.324 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:13.324 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-12T06:06:13.508 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:13 smithi082 ceph-mon[36382]: pgmap v323: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-12T06:06:13.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:13 smithi088 ceph-mon[36585]: pgmap v323: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-12T06:06:13.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:13 smithi076 ceph-mon[32095]: pgmap v323: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-12T06:06:13.995 INFO:tasks.check_counter:Daemon mds.b mds.exported=98 2024-03-12T06:06:13.996 INFO:tasks.check_counter:Daemon mds.b mds.imported=15 2024-03-12T06:06:13.996 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=7 2024-03-12T06:06:13.996 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-03-12T06:06:13.996 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-03-12T06:06:13.996 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=24 2024-03-12T06:06:13.996 DEBUG:tasks.check_counter:Getting stats from e 2024-03-12T06:06:13.996 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-12T06:06:14.245 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T06:06:15.460 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:15 smithi082 ceph-mon[36382]: pgmap v324: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-12T06:06:15.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:15 smithi088 ceph-mon[36585]: pgmap v324: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-12T06:06:15.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:15 smithi076 ceph-mon[32095]: pgmap v324: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-12T06:06:15.954 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-12T06:06:15.954 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T06:06:15.954 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 248, 2024-03-12T06:06:15.954 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 228, 2024-03-12T06:06:15.954 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 1615218, 2024-03-12T06:06:15.955 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 129389, 2024-03-12T06:06:15.955 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 5, 2024-03-12T06:06:15.955 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 1, 2024-03-12T06:06:15.955 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.047062223, 2024-03-12T06:06:15.955 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.014477142, 2024-03-12T06:06:15.955 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.035623455, 2024-03-12T06:06:15.955 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.005243788, 2024-03-12T06:06:15.955 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:15.956 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 228, 2024-03-12T06:06:15.956 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.012135189, 2024-03-12T06:06:15.956 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000053224 2024-03-12T06:06:15.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.956 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:15.956 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.956 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.956 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.957 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 1614384, 2024-03-12T06:06:15.957 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 129072 2024-03-12T06:06:15.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.957 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T06:06:15.957 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 647, 2024-03-12T06:06:15.957 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 806, 2024-03-12T06:06:15.957 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 159012, 2024-03-12T06:06:15.957 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 641848, 2024-03-12T06:06:15.957 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 7, 2024-03-12T06:06:15.958 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 5, 2024-03-12T06:06:15.958 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.071129559, 2024-03-12T06:06:15.958 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.025185573, 2024-03-12T06:06:15.958 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.014314850, 2024-03-12T06:06:15.958 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.021173785, 2024-03-12T06:06:15.958 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:15.958 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 803, 2024-03-12T06:06:15.958 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.041380039, 2024-03-12T06:06:15.959 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000051531 2024-03-12T06:06:15.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.959 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:15.959 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.959 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.959 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.959 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-12T06:06:15.960 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 441248 2024-03-12T06:06:15.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.960 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T06:06:15.960 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 1535, 2024-03-12T06:06:15.960 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 1537, 2024-03-12T06:06:15.960 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 4320722, 2024-03-12T06:06:15.960 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 477443, 2024-03-12T06:06:15.960 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 4, 2024-03-12T06:06:15.961 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 3, 2024-03-12T06:06:15.961 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.156517344, 2024-03-12T06:06:15.961 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.055413740, 2024-03-12T06:06:15.961 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.040509650, 2024-03-12T06:06:15.961 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.046142474, 2024-03-12T06:06:15.961 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:15.961 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 1535, 2024-03-12T06:06:15.961 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.062598523, 2024-03-12T06:06:15.962 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000040780 2024-03-12T06:06:15.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.962 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:15.962 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.962 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.962 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.962 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-12T06:06:15.963 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 0 2024-03-12T06:06:15.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.963 INFO:teuthology.orchestra.run.smithi082.stdout: "cct": { 2024-03-12T06:06:15.963 INFO:teuthology.orchestra.run.smithi082.stdout: "total_workers": 1, 2024-03-12T06:06:15.963 INFO:teuthology.orchestra.run.smithi082.stdout: "unhealthy_workers": 0 2024-03-12T06:06:15.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.963 INFO:teuthology.orchestra.run.smithi082.stdout: "finisher-MDSRank": { 2024-03-12T06:06:15.963 INFO:teuthology.orchestra.run.smithi082.stdout: "queue_len": 0, 2024-03-12T06:06:15.964 INFO:teuthology.orchestra.run.smithi082.stdout: "complete_latency": { 2024-03-12T06:06:15.964 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 3658, 2024-03-12T06:06:15.964 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.111676900, 2024-03-12T06:06:15.964 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000030529 2024-03-12T06:06:15.964 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:15.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.964 INFO:teuthology.orchestra.run.smithi082.stdout: "finisher-PurgeQueue": { 2024-03-12T06:06:15.964 INFO:teuthology.orchestra.run.smithi082.stdout: "queue_len": 0, 2024-03-12T06:06:15.964 INFO:teuthology.orchestra.run.smithi082.stdout: "complete_latency": { 2024-03-12T06:06:15.965 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.965 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.965 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.965 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:15.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.965 INFO:teuthology.orchestra.run.smithi082.stdout: "mds": { 2024-03-12T06:06:15.965 INFO:teuthology.orchestra.run.smithi082.stdout: "request": 0, 2024-03-12T06:06:15.965 INFO:teuthology.orchestra.run.smithi082.stdout: "reply": 0, 2024-03-12T06:06:15.966 INFO:teuthology.orchestra.run.smithi082.stdout: "reply_latency": { 2024-03-12T06:06:15.966 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.966 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.966 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.966 INFO:teuthology.orchestra.run.smithi082.stdout: "slow_reply": 0, 2024-03-12T06:06:15.966 INFO:teuthology.orchestra.run.smithi082.stdout: "forward": 0, 2024-03-12T06:06:15.966 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_fetch_complete": 0, 2024-03-12T06:06:15.967 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_fetch_keys": 0, 2024-03-12T06:06:15.967 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_commit": 0, 2024-03-12T06:06:15.967 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_split": 0, 2024-03-12T06:06:15.967 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_merge": 0, 2024-03-12T06:06:15.967 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes": 2439, 2024-03-12T06:06:15.967 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_top": 2, 2024-03-12T06:06:15.967 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_bottom": 2437, 2024-03-12T06:06:15.967 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_pin_tail": 0, 2024-03-12T06:06:15.967 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_pinned": 2437, 2024-03-12T06:06:15.968 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_expired": 0, 2024-03-12T06:06:15.968 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_with_caps": 0, 2024-03-12T06:06:15.968 INFO:teuthology.orchestra.run.smithi082.stdout: "caps": 0, 2024-03-12T06:06:15.969 INFO:teuthology.orchestra.run.smithi082.stdout: "subtrees": 71, 2024-03-12T06:06:15.969 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse": 0, 2024-03-12T06:06:15.969 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_hit": 0, 2024-03-12T06:06:15.969 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_forward": 0, 2024-03-12T06:06:15.969 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_discover": 0, 2024-03-12T06:06:15.969 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_dir_fetch": 0, 2024-03-12T06:06:15.970 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_remote_ino": 0, 2024-03-12T06:06:15.970 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_lock": 0, 2024-03-12T06:06:15.970 INFO:teuthology.orchestra.run.smithi082.stdout: "load_cent": 0, 2024-03-12T06:06:15.970 INFO:teuthology.orchestra.run.smithi082.stdout: "q": 0, 2024-03-12T06:06:15.970 INFO:teuthology.orchestra.run.smithi082.stdout: "exported": 0, 2024-03-12T06:06:15.970 INFO:teuthology.orchestra.run.smithi082.stdout: "exported_inodes": 0, 2024-03-12T06:06:15.970 INFO:teuthology.orchestra.run.smithi082.stdout: "imported": 0, 2024-03-12T06:06:15.971 INFO:teuthology.orchestra.run.smithi082.stdout: "imported_inodes": 0, 2024-03-12T06:06:15.971 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_dir_fetch": 0, 2024-03-12T06:06:15.971 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_backtrace_fetch": 0, 2024-03-12T06:06:15.971 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_peer_discover": 0, 2024-03-12T06:06:15.971 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rfiles": 0, 2024-03-12T06:06:15.971 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rbytes": 0, 2024-03-12T06:06:15.971 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rsnaps": 0, 2024-03-12T06:06:15.971 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_backtrace_fetch": 0, 2024-03-12T06:06:15.972 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_set_tag": 0, 2024-03-12T06:06:15.972 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T06:06:15.972 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_inotable_repaired": 0, 2024-03-12T06:06:15.972 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dir_inodes": 0, 2024-03-12T06:06:15.972 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dir_base_inodes": 0, 2024-03-12T06:06:15.972 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dirfrag_rstats": 0, 2024-03-12T06:06:15.972 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_file_inodes": 0, 2024-03-12T06:06:15.972 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_inode_file_caps": 0, 2024-03-12T06:06:15.973 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_revoke": 0, 2024-03-12T06:06:15.973 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_grant": 0, 2024-03-12T06:06:15.973 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_trunc": 0, 2024-03-12T06:06:15.973 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T06:06:15.973 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T06:06:15.973 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_caps": 0, 2024-03-12T06:06:15.973 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_caps_dirty": 0, 2024-03-12T06:06:15.973 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_cap_release": 0, 2024-03-12T06:06:15.973 INFO:teuthology.orchestra.run.smithi082.stdout: "process_request_cap_release": 0 2024-03-12T06:06:15.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.974 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_cache": { 2024-03-12T06:06:15.974 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_update": 0, 2024-03-12T06:06:15.974 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_update_receipt": 0, 2024-03-12T06:06:15.974 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_try_discover": 0, 2024-03-12T06:06:15.974 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_send_discover": 0, 2024-03-12T06:06:15.974 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_handle_discover": 0, 2024-03-12T06:06:15.974 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays": 0, 2024-03-12T06:06:15.975 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays_delayed": 0, 2024-03-12T06:06:15.975 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays_enqueuing": 0, 2024-03-12T06:06:15.975 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_created": 0, 2024-03-12T06:06:15.975 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_enqueued": 0, 2024-03-12T06:06:15.975 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_reintegrated": 0, 2024-03-12T06:06:15.975 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_migrated": 0, 2024-03-12T06:06:15.975 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_processing": 0, 2024-03-12T06:06:15.975 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_enqueued": 0, 2024-03-12T06:06:15.976 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_prioritized": 0, 2024-03-12T06:06:15.976 INFO:teuthology.orchestra.run.smithi082.stdout: "recovery_started": 0, 2024-03-12T06:06:15.976 INFO:teuthology.orchestra.run.smithi082.stdout: "recovery_completed": 0, 2024-03-12T06:06:15.976 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_enqueue_scrub": 0, 2024-03-12T06:06:15.976 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_exportdir": 0, 2024-03-12T06:06:15.976 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_flush": 0, 2024-03-12T06:06:15.976 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_fragmentdir": 0, 2024-03-12T06:06:15.976 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_fragstats": 0, 2024-03-12T06:06:15.977 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_inodestats": 0 2024-03-12T06:06:15.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.977 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_log": { 2024-03-12T06:06:15.977 INFO:teuthology.orchestra.run.smithi082.stdout: "evlrg": 0, 2024-03-12T06:06:15.977 INFO:teuthology.orchestra.run.smithi082.stdout: "evadd": 0, 2024-03-12T06:06:15.977 INFO:teuthology.orchestra.run.smithi082.stdout: "evex": 0, 2024-03-12T06:06:15.977 INFO:teuthology.orchestra.run.smithi082.stdout: "evtrm": 0, 2024-03-12T06:06:15.977 INFO:teuthology.orchestra.run.smithi082.stdout: "ev": 1783, 2024-03-12T06:06:15.977 INFO:teuthology.orchestra.run.smithi082.stdout: "evexg": 0, 2024-03-12T06:06:15.978 INFO:teuthology.orchestra.run.smithi082.stdout: "evexd": 0, 2024-03-12T06:06:15.978 INFO:teuthology.orchestra.run.smithi082.stdout: "segadd": 0, 2024-03-12T06:06:15.978 INFO:teuthology.orchestra.run.smithi082.stdout: "segex": 0, 2024-03-12T06:06:15.978 INFO:teuthology.orchestra.run.smithi082.stdout: "segtrm": 0, 2024-03-12T06:06:15.978 INFO:teuthology.orchestra.run.smithi082.stdout: "seg": 3, 2024-03-12T06:06:15.978 INFO:teuthology.orchestra.run.smithi082.stdout: "segmjr": 2, 2024-03-12T06:06:15.978 INFO:teuthology.orchestra.run.smithi082.stdout: "segexg": 0, 2024-03-12T06:06:15.978 INFO:teuthology.orchestra.run.smithi082.stdout: "segexd": 0, 2024-03-12T06:06:15.979 INFO:teuthology.orchestra.run.smithi082.stdout: "expos": 4194304, 2024-03-12T06:06:15.979 INFO:teuthology.orchestra.run.smithi082.stdout: "wrpos": 8057499, 2024-03-12T06:06:15.979 INFO:teuthology.orchestra.run.smithi082.stdout: "rdpos": 8056925, 2024-03-12T06:06:15.979 INFO:teuthology.orchestra.run.smithi082.stdout: "jlat": { 2024-03-12T06:06:15.979 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.979 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.979 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.980 INFO:teuthology.orchestra.run.smithi082.stdout: "replayed": 1783 2024-03-12T06:06:15.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.980 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_mem": { 2024-03-12T06:06:15.980 INFO:teuthology.orchestra.run.smithi082.stdout: "ino": 1941, 2024-03-12T06:06:15.980 INFO:teuthology.orchestra.run.smithi082.stdout: "ino+": 2091, 2024-03-12T06:06:15.980 INFO:teuthology.orchestra.run.smithi082.stdout: "ino-": 150, 2024-03-12T06:06:15.980 INFO:teuthology.orchestra.run.smithi082.stdout: "dir": 118, 2024-03-12T06:06:15.980 INFO:teuthology.orchestra.run.smithi082.stdout: "dir+": 118, 2024-03-12T06:06:15.981 INFO:teuthology.orchestra.run.smithi082.stdout: "dir-": 0, 2024-03-12T06:06:15.981 INFO:teuthology.orchestra.run.smithi082.stdout: "dn": 2439, 2024-03-12T06:06:15.981 INFO:teuthology.orchestra.run.smithi082.stdout: "dn+": 2439, 2024-03-12T06:06:15.981 INFO:teuthology.orchestra.run.smithi082.stdout: "dn-": 0, 2024-03-12T06:06:15.981 INFO:teuthology.orchestra.run.smithi082.stdout: "cap": 0, 2024-03-12T06:06:15.981 INFO:teuthology.orchestra.run.smithi082.stdout: "cap+": 0, 2024-03-12T06:06:15.982 INFO:teuthology.orchestra.run.smithi082.stdout: "cap-": 0, 2024-03-12T06:06:15.982 INFO:teuthology.orchestra.run.smithi082.stdout: "rss": 54860, 2024-03-12T06:06:15.982 INFO:teuthology.orchestra.run.smithi082.stdout: "heap": 215420 2024-03-12T06:06:15.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.982 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_server": { 2024-03-12T06:06:15.982 INFO:teuthology.orchestra.run.smithi082.stdout: "dispatch_client_request": 0, 2024-03-12T06:06:15.982 INFO:teuthology.orchestra.run.smithi082.stdout: "dispatch_server_request": 0, 2024-03-12T06:06:15.982 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_request": 0, 2024-03-12T06:06:15.983 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_session": 0, 2024-03-12T06:06:15.983 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_peer_request": 0, 2024-03-12T06:06:15.983 INFO:teuthology.orchestra.run.smithi082.stdout: "req_create_latency": { 2024-03-12T06:06:15.983 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.983 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.983 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.983 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getattr_latency": { 2024-03-12T06:06:15.984 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.984 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.984 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.984 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getfilelock_latency": { 2024-03-12T06:06:15.984 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.984 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.985 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.985 INFO:teuthology.orchestra.run.smithi082.stdout: "req_link_latency": { 2024-03-12T06:06:15.985 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.985 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.985 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.985 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookup_latency": { 2024-03-12T06:06:15.986 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.986 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.986 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.986 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookuphash_latency": { 2024-03-12T06:06:15.986 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.986 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.987 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.987 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupino_latency": { 2024-03-12T06:06:15.987 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.987 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.987 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.987 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupname_latency": { 2024-03-12T06:06:15.987 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.988 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.988 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.988 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupparent_latency": { 2024-03-12T06:06:15.988 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.988 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.988 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.989 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupsnap_latency": { 2024-03-12T06:06:15.989 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.989 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.989 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.989 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lssnap_latency": { 2024-03-12T06:06:15.989 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.989 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.989 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.990 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mkdir_latency": { 2024-03-12T06:06:15.990 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.990 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.991 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.991 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mknod_latency": { 2024-03-12T06:06:15.991 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.991 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.991 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.992 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mksnap_latency": { 2024-03-12T06:06:15.992 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.992 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.992 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.992 INFO:teuthology.orchestra.run.smithi082.stdout: "req_open_latency": { 2024-03-12T06:06:15.992 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.992 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.992 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.993 INFO:teuthology.orchestra.run.smithi082.stdout: "req_readdir_latency": { 2024-03-12T06:06:15.993 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.993 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.993 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.993 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rename_latency": { 2024-03-12T06:06:15.993 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.993 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.994 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.994 INFO:teuthology.orchestra.run.smithi082.stdout: "req_renamesnap_latency": { 2024-03-12T06:06:15.994 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.994 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.994 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.994 INFO:teuthology.orchestra.run.smithi082.stdout: "req_snapdiff_latency": { 2024-03-12T06:06:15.995 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.995 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.995 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.995 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.995 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmdir_latency": { 2024-03-12T06:06:15.995 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.995 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.995 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.995 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.996 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmsnap_latency": { 2024-03-12T06:06:15.996 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.996 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.996 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.996 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmxattr_latency": { 2024-03-12T06:06:15.996 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.996 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.996 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.997 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setattr_latency": { 2024-03-12T06:06:15.997 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.997 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.997 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.997 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setdirlayout_latency": { 2024-03-12T06:06:15.997 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.997 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.998 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.998 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setfilelock_latency": { 2024-03-12T06:06:15.998 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.998 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.998 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.998 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setlayout_latency": { 2024-03-12T06:06:15.998 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.999 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.999 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.999 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setxattr_latency": { 2024-03-12T06:06:15.999 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:15.999 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:15.999 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:15.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:15.999 INFO:teuthology.orchestra.run.smithi082.stdout: "req_symlink_latency": { 2024-03-12T06:06:16.000 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:16.000 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:16.000 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:16.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.000 INFO:teuthology.orchestra.run.smithi082.stdout: "req_unlink_latency": { 2024-03-12T06:06:16.000 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:16.000 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:16.000 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:16.001 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.001 INFO:teuthology.orchestra.run.smithi082.stdout: "cap_revoke_eviction": 0, 2024-03-12T06:06:16.001 INFO:teuthology.orchestra.run.smithi082.stdout: "cap_acquisition_throttle": 0, 2024-03-12T06:06:16.001 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getvxattr_latency": { 2024-03-12T06:06:16.001 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:16.001 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:16.001 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:16.001 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:16.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.002 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_sessions": { 2024-03-12T06:06:16.002 INFO:teuthology.orchestra.run.smithi082.stdout: "session_count": 1, 2024-03-12T06:06:16.002 INFO:teuthology.orchestra.run.smithi082.stdout: "session_add": 2, 2024-03-12T06:06:16.002 INFO:teuthology.orchestra.run.smithi082.stdout: "session_remove": 1, 2024-03-12T06:06:16.002 INFO:teuthology.orchestra.run.smithi082.stdout: "sessions_open": 2, 2024-03-12T06:06:16.002 INFO:teuthology.orchestra.run.smithi082.stdout: "sessions_stale": 0, 2024-03-12T06:06:16.002 INFO:teuthology.orchestra.run.smithi082.stdout: "total_load": 0, 2024-03-12T06:06:16.003 INFO:teuthology.orchestra.run.smithi082.stdout: "average_load": 0, 2024-03-12T06:06:16.003 INFO:teuthology.orchestra.run.smithi082.stdout: "avg_session_uptime": 4253, 2024-03-12T06:06:16.003 INFO:teuthology.orchestra.run.smithi082.stdout: "mdthresh_evicted": 0 2024-03-12T06:06:16.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.003 INFO:teuthology.orchestra.run.smithi082.stdout: "mempool": { 2024-03-12T06:06:16.003 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_bytes": 0, 2024-03-12T06:06:16.003 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_items": 0, 2024-03-12T06:06:16.003 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T06:06:16.003 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_items": 0, 2024-03-12T06:06:16.004 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T06:06:16.004 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_items": 0, 2024-03-12T06:06:16.004 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T06:06:16.004 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T06:06:16.004 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T06:06:16.004 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T06:06:16.004 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T06:06:16.004 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_items": 0, 2024-03-12T06:06:16.005 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T06:06:16.005 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T06:06:16.005 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_bytes": 0, 2024-03-12T06:06:16.005 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_items": 0, 2024-03-12T06:06:16.005 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_bytes": 0, 2024-03-12T06:06:16.005 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_items": 0, 2024-03-12T06:06:16.005 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T06:06:16.005 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T06:06:16.006 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T06:06:16.006 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T06:06:16.006 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T06:06:16.006 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_items": 0, 2024-03-12T06:06:16.006 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_bytes": 0, 2024-03-12T06:06:16.006 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_items": 0, 2024-03-12T06:06:16.006 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T06:06:16.006 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T06:06:16.007 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_bytes": 0, 2024-03-12T06:06:16.007 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_items": 0, 2024-03-12T06:06:16.007 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_bytes": 0, 2024-03-12T06:06:16.007 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_items": 0, 2024-03-12T06:06:16.007 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T06:06:16.007 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_items": 0, 2024-03-12T06:06:16.007 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T06:06:16.007 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_items": 0, 2024-03-12T06:06:16.007 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_bytes": 262384, 2024-03-12T06:06:16.008 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_items": 2023, 2024-03-12T06:06:16.008 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_bytes": 0, 2024-03-12T06:06:16.008 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_items": 0, 2024-03-12T06:06:16.008 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_bytes": 0, 2024-03-12T06:06:16.008 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_items": 0, 2024-03-12T06:06:16.008 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_bytes": 0, 2024-03-12T06:06:16.008 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_items": 0, 2024-03-12T06:06:16.008 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_bytes": 0, 2024-03-12T06:06:16.008 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_items": 0, 2024-03-12T06:06:16.009 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_bytes": 8152, 2024-03-12T06:06:16.009 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_items": 170, 2024-03-12T06:06:16.009 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T06:06:16.009 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_items": 0, 2024-03-12T06:06:16.009 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_bytes": 0, 2024-03-12T06:06:16.009 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_items": 0, 2024-03-12T06:06:16.009 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_bytes": 6229867, 2024-03-12T06:06:16.009 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_items": 60969, 2024-03-12T06:06:16.010 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_bytes": 0, 2024-03-12T06:06:16.010 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_items": 0, 2024-03-12T06:06:16.010 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_bytes": 0, 2024-03-12T06:06:16.010 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_items": 0 2024-03-12T06:06:16.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.010 INFO:teuthology.orchestra.run.smithi082.stdout: "objecter": { 2024-03-12T06:06:16.010 INFO:teuthology.orchestra.run.smithi082.stdout: "op_active": 0, 2024-03-12T06:06:16.010 INFO:teuthology.orchestra.run.smithi082.stdout: "op_laggy": 0, 2024-03-12T06:06:16.011 INFO:teuthology.orchestra.run.smithi082.stdout: "op_send": 2181, 2024-03-12T06:06:16.011 INFO:teuthology.orchestra.run.smithi082.stdout: "op_send_bytes": 21, 2024-03-12T06:06:16.011 INFO:teuthology.orchestra.run.smithi082.stdout: "op_resend": 3, 2024-03-12T06:06:16.011 INFO:teuthology.orchestra.run.smithi082.stdout: "op_reply": 2181, 2024-03-12T06:06:16.011 INFO:teuthology.orchestra.run.smithi082.stdout: "op_latency": { 2024-03-12T06:06:16.011 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 2181, 2024-03-12T06:06:16.011 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 1.055979501, 2024-03-12T06:06:16.011 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000484172 2024-03-12T06:06:16.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.012 INFO:teuthology.orchestra.run.smithi082.stdout: "op_inflight": 0, 2024-03-12T06:06:16.012 INFO:teuthology.orchestra.run.smithi082.stdout: "oplen_avg": { 2024-03-12T06:06:16.012 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 2181, 2024-03-12T06:06:16.012 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 2182 2024-03-12T06:06:16.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.012 INFO:teuthology.orchestra.run.smithi082.stdout: "op": 2181, 2024-03-12T06:06:16.012 INFO:teuthology.orchestra.run.smithi082.stdout: "op_r": 2181, 2024-03-12T06:06:16.012 INFO:teuthology.orchestra.run.smithi082.stdout: "op_w": 0, 2024-03-12T06:06:16.013 INFO:teuthology.orchestra.run.smithi082.stdout: "op_rmw": 0, 2024-03-12T06:06:16.013 INFO:teuthology.orchestra.run.smithi082.stdout: "op_pg": 0, 2024-03-12T06:06:16.013 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_stat": 1396, 2024-03-12T06:06:16.013 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_create": 0, 2024-03-12T06:06:16.013 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_read": 783, 2024-03-12T06:06:16.013 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_write": 0, 2024-03-12T06:06:16.013 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_writefull": 0, 2024-03-12T06:06:16.013 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_writesame": 0, 2024-03-12T06:06:16.014 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_append": 0, 2024-03-12T06:06:16.014 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_zero": 0, 2024-03-12T06:06:16.014 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_truncate": 0, 2024-03-12T06:06:16.014 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_delete": 0, 2024-03-12T06:06:16.014 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_mapext": 0, 2024-03-12T06:06:16.014 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_sparse_read": 0, 2024-03-12T06:06:16.014 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_clonerange": 0, 2024-03-12T06:06:16.014 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_getxattr": 1, 2024-03-12T06:06:16.014 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_setxattr": 0, 2024-03-12T06:06:16.015 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_cmpxattr": 0, 2024-03-12T06:06:16.015 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_rmxattr": 0, 2024-03-12T06:06:16.015 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_resetxattrs": 0, 2024-03-12T06:06:16.015 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_call": 0, 2024-03-12T06:06:16.015 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_watch": 0, 2024-03-12T06:06:16.015 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_notify": 0, 2024-03-12T06:06:16.015 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T06:06:16.015 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_pgls": 0, 2024-03-12T06:06:16.016 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_pgls_filter": 0, 2024-03-12T06:06:16.016 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_other": 0, 2024-03-12T06:06:16.016 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_active": 0, 2024-03-12T06:06:16.016 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_send": 0, 2024-03-12T06:06:16.016 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_resend": 0, 2024-03-12T06:06:16.016 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_ping": 0, 2024-03-12T06:06:16.016 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_active": 0, 2024-03-12T06:06:16.016 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_send": 0, 2024-03-12T06:06:16.017 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_resend": 0, 2024-03-12T06:06:16.017 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_active": 0, 2024-03-12T06:06:16.017 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_send": 0, 2024-03-12T06:06:16.017 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_resend": 0, 2024-03-12T06:06:16.017 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_active": 0, 2024-03-12T06:06:16.017 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_send": 0, 2024-03-12T06:06:16.017 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_resend": 0, 2024-03-12T06:06:16.017 INFO:teuthology.orchestra.run.smithi082.stdout: "command_active": 0, 2024-03-12T06:06:16.017 INFO:teuthology.orchestra.run.smithi082.stdout: "command_send": 0, 2024-03-12T06:06:16.018 INFO:teuthology.orchestra.run.smithi082.stdout: "command_resend": 0, 2024-03-12T06:06:16.018 INFO:teuthology.orchestra.run.smithi082.stdout: "map_epoch": 96, 2024-03-12T06:06:16.018 INFO:teuthology.orchestra.run.smithi082.stdout: "map_full": 0, 2024-03-12T06:06:16.018 INFO:teuthology.orchestra.run.smithi082.stdout: "map_inc": 10, 2024-03-12T06:06:16.018 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_sessions": 5, 2024-03-12T06:06:16.018 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_session_open": 5, 2024-03-12T06:06:16.018 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_session_close": 0, 2024-03-12T06:06:16.018 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_laggy": 0, 2024-03-12T06:06:16.019 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_wr": 0, 2024-03-12T06:06:16.019 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_rd": 2, 2024-03-12T06:06:16.019 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_del": 0 2024-03-12T06:06:16.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.019 INFO:teuthology.orchestra.run.smithi082.stdout: "oft": { 2024-03-12T06:06:16.019 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_objs": 0, 2024-03-12T06:06:16.019 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_kv_pairs": 0, 2024-03-12T06:06:16.019 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_updates": 0, 2024-03-12T06:06:16.019 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_removes": 0 2024-03-12T06:06:16.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.020 INFO:teuthology.orchestra.run.smithi082.stdout: "osd-slow-ops": { 2024-03-12T06:06:16.020 INFO:teuthology.orchestra.run.smithi082.stdout: "slow_ops_count": 0 2024-03-12T06:06:16.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.020 INFO:teuthology.orchestra.run.smithi082.stdout: "purge_queue": { 2024-03-12T06:06:16.020 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_ops": 0, 2024-03-12T06:06:16.020 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_ops_high_water": 0, 2024-03-12T06:06:16.020 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing": 0, 2024-03-12T06:06:16.021 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_high_water": 0, 2024-03-12T06:06:16.021 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executed_ops": 0, 2024-03-12T06:06:16.021 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executed": 0, 2024-03-12T06:06:16.021 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_item_in_journal": 0 2024-03-12T06:06:16.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.021 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T06:06:16.021 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:16.021 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-12T06:06:16.022 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:16.022 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 2426, 2024-03-12T06:06:16.022 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 5861217, 2024-03-12T06:06:16.022 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:16.022 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 2426, 2024-03-12T06:06:16.022 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-12T06:06:16.022 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-12T06:06:16.022 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 2426, 2024-03-12T06:06:16.022 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 5861217, 2024-03-12T06:06:16.023 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:16.023 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:16.023 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:16.023 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:16.023 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:16.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.023 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-objecter_bytes": { 2024-03-12T06:06:16.023 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:16.023 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-12T06:06:16.024 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:16.024 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-12T06:06:16.024 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-12T06:06:16.024 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:16.024 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:16.024 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 2181, 2024-03-12T06:06:16.024 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 3863200, 2024-03-12T06:06:16.024 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 2181, 2024-03-12T06:06:16.025 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 3863200, 2024-03-12T06:06:16.025 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:16.025 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:16.025 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:16.025 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:16.025 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:16.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.025 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-objecter_ops": { 2024-03-12T06:06:16.025 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:16.026 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 1024, 2024-03-12T06:06:16.026 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:16.026 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-12T06:06:16.026 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-12T06:06:16.026 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:16.026 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:16.026 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 2181, 2024-03-12T06:06:16.026 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 2181, 2024-03-12T06:06:16.027 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 2181, 2024-03-12T06:06:16.027 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 2181, 2024-03-12T06:06:16.027 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:16.027 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:16.027 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:16.027 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:16.027 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:16.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.028 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-write_buf_throttle": { 2024-03-12T06:06:16.028 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:16.028 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 3758096384, 2024-03-12T06:06:16.028 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:16.028 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-12T06:06:16.028 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-12T06:06:16.028 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:16.028 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:16.028 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-12T06:06:16.029 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-12T06:06:16.029 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 0, 2024-03-12T06:06:16.029 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 0, 2024-03-12T06:06:16.029 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:16.029 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:16.029 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:16.029 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:16.030 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:16.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:16.030 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-write_buf_throttle-0x55eb8f4ca960": { 2024-03-12T06:06:16.030 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:16.030 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 3758096384, 2024-03-12T06:06:16.030 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:16.030 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-12T06:06:16.030 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-12T06:06:16.031 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:16.031 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:16.031 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-12T06:06:16.031 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-12T06:06:16.031 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 0, 2024-03-12T06:06:16.031 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 0, 2024-03-12T06:06:16.031 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:16.031 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:16.032 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:16.032 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:16.032 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:16.032 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:16.032 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-12T06:06:16.771 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-12T06:06:16.771 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-12T06:06:16.771 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-03-12T06:06:16.772 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-12T06:06:16.772 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-12T06:06:16.772 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-12T06:06:16.772 DEBUG:tasks.check_counter:Getting stats from h 2024-03-12T06:06:16.772 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-12T06:06:17.009 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T06:06:17.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:17 smithi082 ceph-mon[36382]: osdmap e97: 12 total, 12 up, 12 in 2024-03-12T06:06:17.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:17 smithi082 ceph-mon[36382]: pgmap v326: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s 2024-03-12T06:06:17.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:17 smithi088 ceph-mon[36585]: osdmap e97: 12 total, 12 up, 12 in 2024-03-12T06:06:17.986 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:17 smithi088 ceph-mon[36585]: pgmap v326: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s 2024-03-12T06:06:17.989 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:17 smithi076 ceph-mon[32095]: osdmap e97: 12 total, 12 up, 12 in 2024-03-12T06:06:17.989 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:17 smithi076 ceph-mon[32095]: pgmap v326: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 221 B/s wr, 6 op/s 2024-03-12T06:06:18.743 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-12T06:06:18.743 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T06:06:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 1890, 2024-03-12T06:06:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 1867, 2024-03-12T06:06:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 19135011, 2024-03-12T06:06:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 638517, 2024-03-12T06:06:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 6, 2024-03-12T06:06:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 2, 2024-03-12T06:06:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.195514251, 2024-03-12T06:06:18.744 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.072222204, 2024-03-12T06:06:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.074155353, 2024-03-12T06:06:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.044892282, 2024-03-12T06:06:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 1867, 2024-03-12T06:06:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.070903656, 2024-03-12T06:06:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000037977 2024-03-12T06:06:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:18.745 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 1624816, 2024-03-12T06:06:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 128784 2024-03-12T06:06:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T06:06:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 72, 2024-03-12T06:06:18.746 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 230, 2024-03-12T06:06:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 16800197, 2024-03-12T06:06:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 464508, 2024-03-12T06:06:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 7, 2024-03-12T06:06:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 5, 2024-03-12T06:06:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.027786601, 2024-03-12T06:06:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.007667344, 2024-03-12T06:06:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.016219909, 2024-03-12T06:06:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.001588185, 2024-03-12T06:06:18.747 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 227, 2024-03-12T06:06:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.016413809, 2024-03-12T06:06:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000072307 2024-03-12T06:06:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.748 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-12T06:06:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 442672 2024-03-12T06:06:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T06:06:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 692, 2024-03-12T06:06:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 694, 2024-03-12T06:06:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 12753938, 2024-03-12T06:06:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 215219, 2024-03-12T06:06:18.749 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 5, 2024-03-12T06:06:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 4, 2024-03-12T06:06:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.064391624, 2024-03-12T06:06:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.016603495, 2024-03-12T06:06:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.020823374, 2024-03-12T06:06:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.021741956, 2024-03-12T06:06:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 692, 2024-03-12T06:06:18.750 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.019255740, 2024-03-12T06:06:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000027826 2024-03-12T06:06:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-12T06:06:18.751 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 0 2024-03-12T06:06:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "cct": { 2024-03-12T06:06:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "total_workers": 1, 2024-03-12T06:06:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "unhealthy_workers": 0 2024-03-12T06:06:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "finisher-MDSRank": { 2024-03-12T06:06:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "queue_len": 0, 2024-03-12T06:06:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "complete_latency": { 2024-03-12T06:06:18.752 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 4000, 2024-03-12T06:06:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.097969388, 2024-03-12T06:06:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000024492 2024-03-12T06:06:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "finisher-PurgeQueue": { 2024-03-12T06:06:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "queue_len": 0, 2024-03-12T06:06:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "complete_latency": { 2024-03-12T06:06:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.753 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: "mds": { 2024-03-12T06:06:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: "request": 0, 2024-03-12T06:06:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: "reply": 0, 2024-03-12T06:06:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: "reply_latency": { 2024-03-12T06:06:18.754 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "slow_reply": 0, 2024-03-12T06:06:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "forward": 0, 2024-03-12T06:06:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_fetch_complete": 0, 2024-03-12T06:06:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_fetch_keys": 0, 2024-03-12T06:06:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_commit": 0, 2024-03-12T06:06:18.755 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_split": 0, 2024-03-12T06:06:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_merge": 0, 2024-03-12T06:06:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes": 3375, 2024-03-12T06:06:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_top": 0, 2024-03-12T06:06:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_bottom": 3375, 2024-03-12T06:06:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_pin_tail": 0, 2024-03-12T06:06:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_pinned": 3375, 2024-03-12T06:06:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_expired": 0, 2024-03-12T06:06:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "inodes_with_caps": 0, 2024-03-12T06:06:18.756 INFO:teuthology.orchestra.run.smithi082.stdout: "caps": 0, 2024-03-12T06:06:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "subtrees": 37, 2024-03-12T06:06:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse": 0, 2024-03-12T06:06:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_hit": 0, 2024-03-12T06:06:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_forward": 0, 2024-03-12T06:06:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_discover": 0, 2024-03-12T06:06:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_dir_fetch": 0, 2024-03-12T06:06:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_remote_ino": 0, 2024-03-12T06:06:18.757 INFO:teuthology.orchestra.run.smithi082.stdout: "traverse_lock": 0, 2024-03-12T06:06:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "load_cent": 0, 2024-03-12T06:06:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "q": 0, 2024-03-12T06:06:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "exported": 0, 2024-03-12T06:06:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "exported_inodes": 0, 2024-03-12T06:06:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "imported": 0, 2024-03-12T06:06:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "imported_inodes": 0, 2024-03-12T06:06:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_dir_fetch": 0, 2024-03-12T06:06:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_backtrace_fetch": 0, 2024-03-12T06:06:18.758 INFO:teuthology.orchestra.run.smithi082.stdout: "openino_peer_discover": 0, 2024-03-12T06:06:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rfiles": 3452, 2024-03-12T06:06:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rbytes": 26524023, 2024-03-12T06:06:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "root_rsnaps": 7, 2024-03-12T06:06:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_backtrace_fetch": 0, 2024-03-12T06:06:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_set_tag": 0, 2024-03-12T06:06:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T06:06:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_inotable_repaired": 0, 2024-03-12T06:06:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dir_inodes": 0, 2024-03-12T06:06:18.759 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dir_base_inodes": 0, 2024-03-12T06:06:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_dirfrag_rstats": 0, 2024-03-12T06:06:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "scrub_file_inodes": 0, 2024-03-12T06:06:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_inode_file_caps": 0, 2024-03-12T06:06:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_revoke": 0, 2024-03-12T06:06:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_grant": 0, 2024-03-12T06:06:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_trunc": 0, 2024-03-12T06:06:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T06:06:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T06:06:18.760 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_caps": 0, 2024-03-12T06:06:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_caps_dirty": 0, 2024-03-12T06:06:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_cap_release": 0, 2024-03-12T06:06:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: "process_request_cap_release": 0 2024-03-12T06:06:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_cache": { 2024-03-12T06:06:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_update": 0, 2024-03-12T06:06:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_update_receipt": 0, 2024-03-12T06:06:18.761 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_try_discover": 0, 2024-03-12T06:06:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_send_discover": 0, 2024-03-12T06:06:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "dir_handle_discover": 0, 2024-03-12T06:06:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays": 0, 2024-03-12T06:06:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays_delayed": 0, 2024-03-12T06:06:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "num_strays_enqueuing": 0, 2024-03-12T06:06:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_created": 0, 2024-03-12T06:06:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_enqueued": 0, 2024-03-12T06:06:18.762 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_reintegrated": 0, 2024-03-12T06:06:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "strays_migrated": 0, 2024-03-12T06:06:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_processing": 0, 2024-03-12T06:06:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_enqueued": 0, 2024-03-12T06:06:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "num_recovering_prioritized": 0, 2024-03-12T06:06:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "recovery_started": 0, 2024-03-12T06:06:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "recovery_completed": 0, 2024-03-12T06:06:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_enqueue_scrub": 0, 2024-03-12T06:06:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_exportdir": 0, 2024-03-12T06:06:18.763 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_flush": 0, 2024-03-12T06:06:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_fragmentdir": 0, 2024-03-12T06:06:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_fragstats": 0, 2024-03-12T06:06:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: "ireq_inodestats": 0 2024-03-12T06:06:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_log": { 2024-03-12T06:06:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: "evlrg": 0, 2024-03-12T06:06:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: "evadd": 0, 2024-03-12T06:06:18.764 INFO:teuthology.orchestra.run.smithi082.stdout: "evex": 0, 2024-03-12T06:06:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "evtrm": 0, 2024-03-12T06:06:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "ev": 19897, 2024-03-12T06:06:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "evexg": 0, 2024-03-12T06:06:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "evexd": 0, 2024-03-12T06:06:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "segadd": 0, 2024-03-12T06:06:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "segex": 0, 2024-03-12T06:06:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "segtrm": 0, 2024-03-12T06:06:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "seg": 25, 2024-03-12T06:06:18.765 INFO:teuthology.orchestra.run.smithi082.stdout: "segmjr": 3, 2024-03-12T06:06:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "segexg": 0, 2024-03-12T06:06:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "segexd": 0, 2024-03-12T06:06:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "expos": 4194304, 2024-03-12T06:06:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "wrpos": 50576201, 2024-03-12T06:06:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "rdpos": 50576089, 2024-03-12T06:06:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "jlat": { 2024-03-12T06:06:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.766 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: "replayed": 19897 2024-03-12T06:06:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_mem": { 2024-03-12T06:06:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: "ino": 503, 2024-03-12T06:06:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: "ino+": 7004, 2024-03-12T06:06:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: "ino-": 6501, 2024-03-12T06:06:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: "dir": 59, 2024-03-12T06:06:18.767 INFO:teuthology.orchestra.run.smithi082.stdout: "dir+": 305, 2024-03-12T06:06:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "dir-": 246, 2024-03-12T06:06:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "dn": 3375, 2024-03-12T06:06:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "dn+": 8267, 2024-03-12T06:06:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "dn-": 4892, 2024-03-12T06:06:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "cap": 0, 2024-03-12T06:06:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "cap+": 0, 2024-03-12T06:06:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "cap-": 0, 2024-03-12T06:06:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "rss": 66804, 2024-03-12T06:06:18.768 INFO:teuthology.orchestra.run.smithi082.stdout: "heap": 215420 2024-03-12T06:06:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_server": { 2024-03-12T06:06:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: "dispatch_client_request": 0, 2024-03-12T06:06:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: "dispatch_server_request": 0, 2024-03-12T06:06:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_request": 0, 2024-03-12T06:06:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_client_session": 0, 2024-03-12T06:06:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: "handle_peer_request": 0, 2024-03-12T06:06:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: "req_create_latency": { 2024-03-12T06:06:18.769 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getattr_latency": { 2024-03-12T06:06:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.770 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getfilelock_latency": { 2024-03-12T06:06:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: "req_link_latency": { 2024-03-12T06:06:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.771 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookup_latency": { 2024-03-12T06:06:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookuphash_latency": { 2024-03-12T06:06:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.772 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupino_latency": { 2024-03-12T06:06:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupname_latency": { 2024-03-12T06:06:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.773 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupparent_latency": { 2024-03-12T06:06:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lookupsnap_latency": { 2024-03-12T06:06:18.774 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: "req_lssnap_latency": { 2024-03-12T06:06:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.775 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mkdir_latency": { 2024-03-12T06:06:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.776 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mknod_latency": { 2024-03-12T06:06:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "req_mksnap_latency": { 2024-03-12T06:06:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.777 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: "req_open_latency": { 2024-03-12T06:06:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: "req_readdir_latency": { 2024-03-12T06:06:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.778 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rename_latency": { 2024-03-12T06:06:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.779 INFO:teuthology.orchestra.run.smithi082.stdout: "req_renamesnap_latency": { 2024-03-12T06:06:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "req_snapdiff_latency": { 2024-03-12T06:06:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.780 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmdir_latency": { 2024-03-12T06:06:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmsnap_latency": { 2024-03-12T06:06:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.781 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: "req_rmxattr_latency": { 2024-03-12T06:06:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setattr_latency": { 2024-03-12T06:06:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.782 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setdirlayout_latency": { 2024-03-12T06:06:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.783 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setfilelock_latency": { 2024-03-12T06:06:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setlayout_latency": { 2024-03-12T06:06:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.784 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: "req_setxattr_latency": { 2024-03-12T06:06:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: "req_symlink_latency": { 2024-03-12T06:06:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.785 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "req_unlink_latency": { 2024-03-12T06:06:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "cap_revoke_eviction": 0, 2024-03-12T06:06:18.786 INFO:teuthology.orchestra.run.smithi082.stdout: "cap_acquisition_throttle": 0, 2024-03-12T06:06:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "req_getvxattr_latency": { 2024-03-12T06:06:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_sessions": { 2024-03-12T06:06:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "session_count": 2, 2024-03-12T06:06:18.787 INFO:teuthology.orchestra.run.smithi082.stdout: "session_add": 5, 2024-03-12T06:06:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "session_remove": 3, 2024-03-12T06:06:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "sessions_open": 4, 2024-03-12T06:06:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "sessions_stale": 0, 2024-03-12T06:06:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "total_load": 0, 2024-03-12T06:06:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "average_load": 0, 2024-03-12T06:06:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "avg_session_uptime": 4256, 2024-03-12T06:06:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "mdthresh_evicted": 0 2024-03-12T06:06:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.788 INFO:teuthology.orchestra.run.smithi082.stdout: "mempool": { 2024-03-12T06:06:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_bytes": 0, 2024-03-12T06:06:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_items": 0, 2024-03-12T06:06:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T06:06:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_items": 0, 2024-03-12T06:06:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T06:06:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_items": 0, 2024-03-12T06:06:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T06:06:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T06:06:18.789 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T06:06:18.790 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T06:06:18.790 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T06:06:18.790 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_items": 0, 2024-03-12T06:06:18.790 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T06:06:18.790 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T06:06:18.790 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_bytes": 0, 2024-03-12T06:06:18.790 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_items": 0, 2024-03-12T06:06:18.790 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_bytes": 0, 2024-03-12T06:06:18.790 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_items": 0, 2024-03-12T06:06:18.791 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T06:06:18.791 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T06:06:18.791 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T06:06:18.791 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T06:06:18.791 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T06:06:18.791 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_items": 0, 2024-03-12T06:06:18.791 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_bytes": 0, 2024-03-12T06:06:18.791 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_items": 0, 2024-03-12T06:06:18.792 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T06:06:18.792 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T06:06:18.792 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_bytes": 0, 2024-03-12T06:06:18.792 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_items": 0, 2024-03-12T06:06:18.792 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_bytes": 0, 2024-03-12T06:06:18.792 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_items": 0, 2024-03-12T06:06:18.792 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T06:06:18.792 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_items": 0, 2024-03-12T06:06:18.792 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T06:06:18.793 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_items": 0, 2024-03-12T06:06:18.793 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_bytes": 225150, 2024-03-12T06:06:18.793 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_items": 570, 2024-03-12T06:06:18.793 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_bytes": 0, 2024-03-12T06:06:18.793 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_items": 0, 2024-03-12T06:06:18.793 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_bytes": 0, 2024-03-12T06:06:18.793 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_items": 0, 2024-03-12T06:06:18.793 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_bytes": 0, 2024-03-12T06:06:18.793 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_items": 0, 2024-03-12T06:06:18.794 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_bytes": 0, 2024-03-12T06:06:18.794 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_items": 0, 2024-03-12T06:06:18.794 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_bytes": 8152, 2024-03-12T06:06:18.794 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_items": 170, 2024-03-12T06:06:18.794 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T06:06:18.794 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_items": 0, 2024-03-12T06:06:18.794 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_bytes": 0, 2024-03-12T06:06:18.794 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_items": 0, 2024-03-12T06:06:18.794 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_bytes": 3625470, 2024-03-12T06:06:18.795 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_items": 25797, 2024-03-12T06:06:18.795 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_bytes": 0, 2024-03-12T06:06:18.795 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_items": 0, 2024-03-12T06:06:18.795 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_bytes": 0, 2024-03-12T06:06:18.795 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_items": 0 2024-03-12T06:06:18.795 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.795 INFO:teuthology.orchestra.run.smithi082.stdout: "objecter": { 2024-03-12T06:06:18.795 INFO:teuthology.orchestra.run.smithi082.stdout: "op_active": 0, 2024-03-12T06:06:18.796 INFO:teuthology.orchestra.run.smithi082.stdout: "op_laggy": 0, 2024-03-12T06:06:18.796 INFO:teuthology.orchestra.run.smithi082.stdout: "op_send": 2401, 2024-03-12T06:06:18.796 INFO:teuthology.orchestra.run.smithi082.stdout: "op_send_bytes": 26, 2024-03-12T06:06:18.796 INFO:teuthology.orchestra.run.smithi082.stdout: "op_resend": 4, 2024-03-12T06:06:18.796 INFO:teuthology.orchestra.run.smithi082.stdout: "op_reply": 2401, 2024-03-12T06:06:18.796 INFO:teuthology.orchestra.run.smithi082.stdout: "op_latency": { 2024-03-12T06:06:18.796 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 2401, 2024-03-12T06:06:18.796 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 1.158977490, 2024-03-12T06:06:18.796 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000482706 2024-03-12T06:06:18.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.797 INFO:teuthology.orchestra.run.smithi082.stdout: "op_inflight": 0, 2024-03-12T06:06:18.797 INFO:teuthology.orchestra.run.smithi082.stdout: "oplen_avg": { 2024-03-12T06:06:18.797 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 2401, 2024-03-12T06:06:18.797 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 2402 2024-03-12T06:06:18.797 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.797 INFO:teuthology.orchestra.run.smithi082.stdout: "op": 2401, 2024-03-12T06:06:18.797 INFO:teuthology.orchestra.run.smithi082.stdout: "op_r": 2401, 2024-03-12T06:06:18.797 INFO:teuthology.orchestra.run.smithi082.stdout: "op_w": 0, 2024-03-12T06:06:18.798 INFO:teuthology.orchestra.run.smithi082.stdout: "op_rmw": 0, 2024-03-12T06:06:18.798 INFO:teuthology.orchestra.run.smithi082.stdout: "op_pg": 0, 2024-03-12T06:06:18.798 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_stat": 1522, 2024-03-12T06:06:18.798 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_create": 0, 2024-03-12T06:06:18.798 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_read": 876, 2024-03-12T06:06:18.798 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_write": 0, 2024-03-12T06:06:18.798 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_writefull": 0, 2024-03-12T06:06:18.798 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_writesame": 0, 2024-03-12T06:06:18.799 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_append": 0, 2024-03-12T06:06:18.799 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_zero": 0, 2024-03-12T06:06:18.799 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_truncate": 0, 2024-03-12T06:06:18.799 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_delete": 0, 2024-03-12T06:06:18.799 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_mapext": 0, 2024-03-12T06:06:18.799 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_sparse_read": 0, 2024-03-12T06:06:18.799 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_clonerange": 0, 2024-03-12T06:06:18.799 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_getxattr": 2, 2024-03-12T06:06:18.799 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_setxattr": 0, 2024-03-12T06:06:18.800 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_cmpxattr": 0, 2024-03-12T06:06:18.800 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_rmxattr": 0, 2024-03-12T06:06:18.800 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_resetxattrs": 0, 2024-03-12T06:06:18.800 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_call": 0, 2024-03-12T06:06:18.800 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_watch": 0, 2024-03-12T06:06:18.800 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_notify": 0, 2024-03-12T06:06:18.800 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T06:06:18.800 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_pgls": 0, 2024-03-12T06:06:18.801 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_pgls_filter": 0, 2024-03-12T06:06:18.801 INFO:teuthology.orchestra.run.smithi082.stdout: "osdop_other": 0, 2024-03-12T06:06:18.801 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_active": 0, 2024-03-12T06:06:18.801 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_send": 0, 2024-03-12T06:06:18.801 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_resend": 0, 2024-03-12T06:06:18.801 INFO:teuthology.orchestra.run.smithi082.stdout: "linger_ping": 0, 2024-03-12T06:06:18.801 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_active": 0, 2024-03-12T06:06:18.801 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_send": 0, 2024-03-12T06:06:18.801 INFO:teuthology.orchestra.run.smithi082.stdout: "poolop_resend": 0, 2024-03-12T06:06:18.802 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_active": 0, 2024-03-12T06:06:18.802 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_send": 0, 2024-03-12T06:06:18.802 INFO:teuthology.orchestra.run.smithi082.stdout: "poolstat_resend": 0, 2024-03-12T06:06:18.802 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_active": 0, 2024-03-12T06:06:18.802 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_send": 0, 2024-03-12T06:06:18.802 INFO:teuthology.orchestra.run.smithi082.stdout: "statfs_resend": 0, 2024-03-12T06:06:18.802 INFO:teuthology.orchestra.run.smithi082.stdout: "command_active": 0, 2024-03-12T06:06:18.802 INFO:teuthology.orchestra.run.smithi082.stdout: "command_send": 0, 2024-03-12T06:06:18.803 INFO:teuthology.orchestra.run.smithi082.stdout: "command_resend": 0, 2024-03-12T06:06:18.803 INFO:teuthology.orchestra.run.smithi082.stdout: "map_epoch": 97, 2024-03-12T06:06:18.803 INFO:teuthology.orchestra.run.smithi082.stdout: "map_full": 0, 2024-03-12T06:06:18.803 INFO:teuthology.orchestra.run.smithi082.stdout: "map_inc": 11, 2024-03-12T06:06:18.803 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_sessions": 7, 2024-03-12T06:06:18.803 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_session_open": 7, 2024-03-12T06:06:18.803 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_session_close": 0, 2024-03-12T06:06:18.803 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_laggy": 0, 2024-03-12T06:06:18.803 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_wr": 0, 2024-03-12T06:06:18.804 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_rd": 2, 2024-03-12T06:06:18.804 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_del": 0 2024-03-12T06:06:18.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.804 INFO:teuthology.orchestra.run.smithi082.stdout: "oft": { 2024-03-12T06:06:18.804 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_objs": 0, 2024-03-12T06:06:18.804 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_kv_pairs": 0, 2024-03-12T06:06:18.804 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_updates": 0, 2024-03-12T06:06:18.804 INFO:teuthology.orchestra.run.smithi082.stdout: "omap_total_removes": 0 2024-03-12T06:06:18.804 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.805 INFO:teuthology.orchestra.run.smithi082.stdout: "osd-slow-ops": { 2024-03-12T06:06:18.805 INFO:teuthology.orchestra.run.smithi082.stdout: "slow_ops_count": 0 2024-03-12T06:06:18.805 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.805 INFO:teuthology.orchestra.run.smithi082.stdout: "purge_queue": { 2024-03-12T06:06:18.805 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_ops": 0, 2024-03-12T06:06:18.805 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_ops_high_water": 0, 2024-03-12T06:06:18.805 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing": 0, 2024-03-12T06:06:18.805 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executing_high_water": 0, 2024-03-12T06:06:18.805 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executed_ops": 0, 2024-03-12T06:06:18.806 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_executed": 0, 2024-03-12T06:06:18.806 INFO:teuthology.orchestra.run.smithi082.stdout: "pq_item_in_journal": 0 2024-03-12T06:06:18.806 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.806 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T06:06:18.806 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:18.806 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-12T06:06:18.806 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:18.806 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 2650, 2024-03-12T06:06:18.806 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 48435066, 2024-03-12T06:06:18.807 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:18.807 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 2650, 2024-03-12T06:06:18.807 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-12T06:06:18.807 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-12T06:06:18.807 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 2650, 2024-03-12T06:06:18.807 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 48435066, 2024-03-12T06:06:18.807 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:18.807 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.807 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.808 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.808 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:18.808 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.808 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-objecter_bytes": { 2024-03-12T06:06:18.808 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:18.808 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-12T06:06:18.808 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:18.809 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-12T06:06:18.809 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-12T06:06:18.809 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:18.809 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:18.809 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 2401, 2024-03-12T06:06:18.809 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 46390700, 2024-03-12T06:06:18.809 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 2401, 2024-03-12T06:06:18.809 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 46390700, 2024-03-12T06:06:18.809 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:18.810 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.810 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.810 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.810 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:18.810 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.810 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-objecter_ops": { 2024-03-12T06:06:18.810 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:18.810 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 1024, 2024-03-12T06:06:18.811 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:18.811 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-12T06:06:18.811 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-12T06:06:18.811 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:18.811 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:18.811 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 2401, 2024-03-12T06:06:18.811 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 2401, 2024-03-12T06:06:18.811 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 2401, 2024-03-12T06:06:18.811 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 2401, 2024-03-12T06:06:18.812 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:18.812 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.812 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.812 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.812 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:18.812 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.812 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-write_buf_throttle": { 2024-03-12T06:06:18.812 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:18.813 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 3758096384, 2024-03-12T06:06:18.813 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:18.813 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-12T06:06:18.813 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-12T06:06:18.813 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:18.813 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:18.813 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-12T06:06:18.813 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-12T06:06:18.813 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 0, 2024-03-12T06:06:18.814 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 0, 2024-03-12T06:06:18.814 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:18.814 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.814 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.814 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.814 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:18.814 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:18.814 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-write_buf_throttle-0x55f077d46960": { 2024-03-12T06:06:18.815 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:18.815 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 3758096384, 2024-03-12T06:06:18.815 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:18.815 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 0, 2024-03-12T06:06:18.815 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 0, 2024-03-12T06:06:18.815 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:18.815 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:18.815 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-12T06:06:18.815 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-12T06:06:18.816 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 0, 2024-03-12T06:06:18.816 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 0, 2024-03-12T06:06:18.816 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:18.816 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:18.816 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:18.816 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:18.816 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:18.816 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:18.817 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-12T06:06:19.518 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:19 smithi082 ceph-mon[36382]: pgmap v327: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:06:19.546 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-12T06:06:19.546 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-12T06:06:19.547 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=7 2024-03-12T06:06:19.547 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-12T06:06:19.547 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-12T06:06:19.547 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-03-12T06:06:19.547 DEBUG:tasks.check_counter:Getting stats from k 2024-03-12T06:06:19.548 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-12T06:06:19.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:19 smithi088 ceph-mon[36585]: pgmap v327: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:06:19.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:19 smithi076 ceph-mon[32095]: pgmap v327: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:06:19.787 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.b/config 2024-03-12T06:06:21.493 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-12T06:06:21.612 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T06:06:21.612 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 4, 2024-03-12T06:06:21.612 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 4, 2024-03-12T06:06:21.612 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 2992, 2024-03-12T06:06:21.612 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 448, 2024-03-12T06:06:21.612 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 4, 2024-03-12T06:06:21.612 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 0, 2024-03-12T06:06:21.613 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.001664723, 2024-03-12T06:06:21.613 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.000041933, 2024-03-12T06:06:21.613 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.000707111, 2024-03-12T06:06:21.613 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T06:06:21.613 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:21.613 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 3, 2024-03-12T06:06:21.613 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000753455, 2024-03-12T06:06:21.613 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000251151 2024-03-12T06:06:21.613 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:21.614 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:21.614 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:21.614 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:21.614 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:21.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:21.614 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-12T06:06:21.614 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 448 2024-03-12T06:06:21.614 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:21.614 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T06:06:21.615 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 2, 2024-03-12T06:06:21.615 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 163, 2024-03-12T06:06:21.615 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 320, 2024-03-12T06:06:21.615 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 183392, 2024-03-12T06:06:21.615 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 5, 2024-03-12T06:06:21.615 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 3, 2024-03-12T06:06:21.615 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.011350176, 2024-03-12T06:06:21.615 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.007436217, 2024-03-12T06:06:21.616 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.001630277, 2024-03-12T06:06:21.616 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T06:06:21.616 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:21.616 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 160, 2024-03-12T06:06:21.616 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.016168947, 2024-03-12T06:06:21.616 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000101055 2024-03-12T06:06:21.616 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:21.616 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:21.616 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:21.617 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:21.617 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:21.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:21.617 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-12T06:06:21.617 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 183392 2024-03-12T06:06:21.617 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:21.617 INFO:teuthology.orchestra.run.smithi082.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T06:06:21.617 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_messages": 225, 2024-03-12T06:06:21.617 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages": 214, 2024-03-12T06:06:21.618 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_bytes": 1564416, 2024-03-12T06:06:21.618 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_bytes": 128528, 2024-03-12T06:06:21.618 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_created_connections": 2, 2024-03-12T06:06:21.618 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_active_connections": 1, 2024-03-12T06:06:21.618 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_total_time": 0.046595131, 2024-03-12T06:06:21.618 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_send_time": 0.014148438, 2024-03-12T06:06:21.618 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_recv_time": 0.037323728, 2024-03-12T06:06:21.618 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_running_fast_dispatch_time": 0.005373762, 2024-03-12T06:06:21.619 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:21.619 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 213, 2024-03-12T06:06:21.619 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.010314242, 2024-03-12T06:06:21.619 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000048423 2024-03-12T06:06:21.619 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:21.619 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:21.619 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:21.619 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:21.619 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:21.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:21.620 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_recv_encrypted_bytes": 1564416, 2024-03-12T06:06:21.620 INFO:teuthology.orchestra.run.smithi082.stdout: "msgr_send_encrypted_bytes": 128528 2024-03-12T06:06:21.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:21.620 INFO:teuthology.orchestra.run.smithi082.stdout: "cct": { 2024-03-12T06:06:21.620 INFO:teuthology.orchestra.run.smithi082.stdout: "total_workers": 0, 2024-03-12T06:06:21.620 INFO:teuthology.orchestra.run.smithi082.stdout: "unhealthy_workers": 0 2024-03-12T06:06:21.620 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:21.621 INFO:teuthology.orchestra.run.smithi082.stdout: "mempool": { 2024-03-12T06:06:21.621 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_bytes": 0, 2024-03-12T06:06:21.621 INFO:teuthology.orchestra.run.smithi082.stdout: "bloom_filter_items": 0, 2024-03-12T06:06:21.621 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T06:06:21.621 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_alloc_items": 0, 2024-03-12T06:06:21.621 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T06:06:21.621 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_data_items": 0, 2024-03-12T06:06:21.621 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T06:06:21.622 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T06:06:21.622 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T06:06:21.622 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T06:06:21.622 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T06:06:21.622 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_other_items": 0, 2024-03-12T06:06:21.622 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T06:06:21.622 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T06:06:21.622 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_bytes": 0, 2024-03-12T06:06:21.623 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_extent_items": 0, 2024-03-12T06:06:21.623 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_bytes": 0, 2024-03-12T06:06:21.623 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_blob_items": 0, 2024-03-12T06:06:21.623 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T06:06:21.623 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T06:06:21.623 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T06:06:21.623 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T06:06:21.623 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T06:06:21.623 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_fsck_items": 0, 2024-03-12T06:06:21.624 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_bytes": 0, 2024-03-12T06:06:21.624 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_txc_items": 0, 2024-03-12T06:06:21.624 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T06:06:21.624 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T06:06:21.624 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_bytes": 0, 2024-03-12T06:06:21.624 INFO:teuthology.orchestra.run.smithi082.stdout: "bluestore_writing_items": 0, 2024-03-12T06:06:21.624 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_bytes": 0, 2024-03-12T06:06:21.624 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_items": 0, 2024-03-12T06:06:21.625 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T06:06:21.625 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_reader_items": 0, 2024-03-12T06:06:21.625 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T06:06:21.625 INFO:teuthology.orchestra.run.smithi082.stdout: "bluefs_file_writer_items": 0, 2024-03-12T06:06:21.625 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_bytes": 148984, 2024-03-12T06:06:21.625 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_anon_items": 30, 2024-03-12T06:06:21.625 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_bytes": 0, 2024-03-12T06:06:21.625 INFO:teuthology.orchestra.run.smithi082.stdout: "buffer_meta_items": 0, 2024-03-12T06:06:21.625 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_bytes": 0, 2024-03-12T06:06:21.626 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_items": 0, 2024-03-12T06:06:21.626 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_bytes": 0, 2024-03-12T06:06:21.626 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_mapbl_items": 0, 2024-03-12T06:06:21.626 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_bytes": 0, 2024-03-12T06:06:21.626 INFO:teuthology.orchestra.run.smithi082.stdout: "osd_pglog_items": 0, 2024-03-12T06:06:21.626 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_bytes": 0, 2024-03-12T06:06:21.626 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_items": 0, 2024-03-12T06:06:21.627 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T06:06:21.627 INFO:teuthology.orchestra.run.smithi082.stdout: "osdmap_mapping_items": 0, 2024-03-12T06:06:21.627 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_bytes": 0, 2024-03-12T06:06:21.627 INFO:teuthology.orchestra.run.smithi082.stdout: "pgmap_items": 0, 2024-03-12T06:06:21.627 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_bytes": 928, 2024-03-12T06:06:21.627 INFO:teuthology.orchestra.run.smithi082.stdout: "mds_co_items": 2, 2024-03-12T06:06:21.627 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_bytes": 0, 2024-03-12T06:06:21.628 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_1_items": 0, 2024-03-12T06:06:21.628 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_bytes": 0, 2024-03-12T06:06:21.628 INFO:teuthology.orchestra.run.smithi082.stdout: "unittest_2_items": 0 2024-03-12T06:06:21.628 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-12T06:06:21.628 INFO:teuthology.orchestra.run.smithi082.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T06:06:21.628 INFO:teuthology.orchestra.run.smithi082.stdout: "val": 0, 2024-03-12T06:06:21.628 INFO:teuthology.orchestra.run.smithi082.stdout: "max": 104857600, 2024-03-12T06:06:21.628 INFO:teuthology.orchestra.run.smithi082.stdout: "get_started": 0, 2024-03-12T06:06:21.628 INFO:teuthology.orchestra.run.smithi082.stdout: "get": 227, 2024-03-12T06:06:21.629 INFO:teuthology.orchestra.run.smithi082.stdout: "get_sum": 1532698, 2024-03-12T06:06:21.629 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:21.629 INFO:teuthology.orchestra.run.smithi082.stdout: "get_or_fail_success": 227, 2024-03-12T06:06:21.629 INFO:teuthology.orchestra.run.smithi082.stdout: "take": 0, 2024-03-12T06:06:21.629 INFO:teuthology.orchestra.run.smithi082.stdout: "take_sum": 0, 2024-03-12T06:06:21.629 INFO:teuthology.orchestra.run.smithi082.stdout: "put": 227, 2024-03-12T06:06:21.629 INFO:teuthology.orchestra.run.smithi082.stdout: "put_sum": 1532698, 2024-03-12T06:06:21.629 INFO:teuthology.orchestra.run.smithi082.stdout: "wait": { 2024-03-12T06:06:21.629 INFO:teuthology.orchestra.run.smithi082.stdout: "avgcount": 0, 2024-03-12T06:06:21.630 INFO:teuthology.orchestra.run.smithi082.stdout: "sum": 0.000000000, 2024-03-12T06:06:21.630 INFO:teuthology.orchestra.run.smithi082.stdout: "avgtime": 0.000000000 2024-03-12T06:06:21.630 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:21.630 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-12T06:06:21.630 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-12T06:06:21.631 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:21 smithi082 ceph-mon[36382]: pgmap v328: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:06:21.736 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:21 smithi088 ceph-mon[36585]: pgmap v328: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:06:21.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:21 smithi076 ceph-mon[32095]: pgmap v328: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-12T06:06:22.279 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-12T06:06:22.279 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-12T06:06:22.279 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-12T06:06:22.279 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-12T06:06:22.279 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-12T06:06:22.279 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-03-12T06:06:22.280 DEBUG:tasks.check_counter:Getting stats from c 2024-03-12T06:06:22.280 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-12T06:06:22.527 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T06:06:23.665 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:23 smithi088 ceph-mon[36585]: pgmap v329: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:06:23.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:23 smithi082 ceph-mon[36382]: pgmap v329: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:06:23.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:23 smithi076 ceph-mon[32095]: pgmap v329: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:06:24.213 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-03-12T06:06:24.214 INFO:teuthology.orchestra.run.smithi088.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T06:06:24.214 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_messages": 2225, 2024-03-12T06:06:24.214 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages": 2225, 2024-03-12T06:06:24.214 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_bytes": 6341533, 2024-03-12T06:06:24.214 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_bytes": 691186, 2024-03-12T06:06:24.214 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_created_connections": 6, 2024-03-12T06:06:24.214 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_active_connections": 2, 2024-03-12T06:06:24.215 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_total_time": 0.195290056, 2024-03-12T06:06:24.215 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_send_time": 0.064481441, 2024-03-12T06:06:24.215 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_recv_time": 0.050401388, 2024-03-12T06:06:24.215 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_fast_dispatch_time": 0.064651992, 2024-03-12T06:06:24.215 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:24.215 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 2224, 2024-03-12T06:06:24.215 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.087439548, 2024-03-12T06:06:24.215 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000039316 2024-03-12T06:06:24.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.216 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:24.216 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.216 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.216 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.216 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-12T06:06:24.216 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_encrypted_bytes": 448 2024-03-12T06:06:24.216 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.217 INFO:teuthology.orchestra.run.smithi088.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T06:06:24.217 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_messages": 4, 2024-03-12T06:06:24.217 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages": 164, 2024-03-12T06:06:24.217 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_bytes": 876, 2024-03-12T06:06:24.217 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_bytes": 445308, 2024-03-12T06:06:24.217 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_created_connections": 6, 2024-03-12T06:06:24.217 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_active_connections": 4, 2024-03-12T06:06:24.217 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_total_time": 0.010870567, 2024-03-12T06:06:24.218 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_send_time": 0.006210010, 2024-03-12T06:06:24.218 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_recv_time": 0.001857671, 2024-03-12T06:06:24.218 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_fast_dispatch_time": 0.000079218, 2024-03-12T06:06:24.218 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:24.218 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 161, 2024-03-12T06:06:24.218 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.012885987, 2024-03-12T06:06:24.218 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000080037 2024-03-12T06:06:24.218 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.219 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:24.219 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.219 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.219 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.219 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-12T06:06:24.219 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_encrypted_bytes": 444624 2024-03-12T06:06:24.219 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.219 INFO:teuthology.orchestra.run.smithi088.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T06:06:24.220 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_messages": 250, 2024-03-12T06:06:24.220 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages": 229, 2024-03-12T06:06:24.220 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_bytes": 1622812, 2024-03-12T06:06:24.220 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_bytes": 129895, 2024-03-12T06:06:24.221 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_created_connections": 3, 2024-03-12T06:06:24.221 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_active_connections": 2, 2024-03-12T06:06:24.221 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_total_time": 0.047981115, 2024-03-12T06:06:24.221 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_send_time": 0.014658570, 2024-03-12T06:06:24.221 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_recv_time": 0.037082969, 2024-03-12T06:06:24.221 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_fast_dispatch_time": 0.005464815, 2024-03-12T06:06:24.221 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:24.221 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 228, 2024-03-12T06:06:24.222 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.013362656, 2024-03-12T06:06:24.222 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000058608 2024-03-12T06:06:24.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.222 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:24.222 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.222 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.222 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.222 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.222 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_encrypted_bytes": 1622544, 2024-03-12T06:06:24.223 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_encrypted_bytes": 129584 2024-03-12T06:06:24.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.223 INFO:teuthology.orchestra.run.smithi088.stdout: "cct": { 2024-03-12T06:06:24.223 INFO:teuthology.orchestra.run.smithi088.stdout: "total_workers": 1, 2024-03-12T06:06:24.223 INFO:teuthology.orchestra.run.smithi088.stdout: "unhealthy_workers": 0 2024-03-12T06:06:24.223 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.223 INFO:teuthology.orchestra.run.smithi088.stdout: "finisher-MDSRank": { 2024-03-12T06:06:24.223 INFO:teuthology.orchestra.run.smithi088.stdout: "queue_len": 0, 2024-03-12T06:06:24.223 INFO:teuthology.orchestra.run.smithi088.stdout: "complete_latency": { 2024-03-12T06:06:24.224 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 3765, 2024-03-12T06:06:24.224 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.112221423, 2024-03-12T06:06:24.224 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000029806 2024-03-12T06:06:24.224 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:24.224 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.224 INFO:teuthology.orchestra.run.smithi088.stdout: "finisher-PurgeQueue": { 2024-03-12T06:06:24.224 INFO:teuthology.orchestra.run.smithi088.stdout: "queue_len": 0, 2024-03-12T06:06:24.224 INFO:teuthology.orchestra.run.smithi088.stdout: "complete_latency": { 2024-03-12T06:06:24.224 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.225 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.225 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.225 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:24.225 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.225 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": { 2024-03-12T06:06:24.225 INFO:teuthology.orchestra.run.smithi088.stdout: "request": 0, 2024-03-12T06:06:24.225 INFO:teuthology.orchestra.run.smithi088.stdout: "reply": 0, 2024-03-12T06:06:24.225 INFO:teuthology.orchestra.run.smithi088.stdout: "reply_latency": { 2024-03-12T06:06:24.225 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.226 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.226 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.226 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.226 INFO:teuthology.orchestra.run.smithi088.stdout: "slow_reply": 0, 2024-03-12T06:06:24.226 INFO:teuthology.orchestra.run.smithi088.stdout: "forward": 0, 2024-03-12T06:06:24.226 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_fetch_complete": 0, 2024-03-12T06:06:24.226 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_fetch_keys": 0, 2024-03-12T06:06:24.226 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_commit": 0, 2024-03-12T06:06:24.227 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_split": 0, 2024-03-12T06:06:24.227 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_merge": 0, 2024-03-12T06:06:24.227 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes": 2936, 2024-03-12T06:06:24.227 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_top": 2, 2024-03-12T06:06:24.227 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_bottom": 2934, 2024-03-12T06:06:24.227 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_pin_tail": 0, 2024-03-12T06:06:24.227 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_pinned": 2932, 2024-03-12T06:06:24.227 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_expired": 0, 2024-03-12T06:06:24.227 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_with_caps": 0, 2024-03-12T06:06:24.228 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": 0, 2024-03-12T06:06:24.228 INFO:teuthology.orchestra.run.smithi088.stdout: "subtrees": 68, 2024-03-12T06:06:24.228 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse": 0, 2024-03-12T06:06:24.228 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_hit": 0, 2024-03-12T06:06:24.228 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_forward": 0, 2024-03-12T06:06:24.228 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_discover": 0, 2024-03-12T06:06:24.228 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_dir_fetch": 0, 2024-03-12T06:06:24.228 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_remote_ino": 0, 2024-03-12T06:06:24.229 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_lock": 0, 2024-03-12T06:06:24.229 INFO:teuthology.orchestra.run.smithi088.stdout: "load_cent": 0, 2024-03-12T06:06:24.229 INFO:teuthology.orchestra.run.smithi088.stdout: "q": 0, 2024-03-12T06:06:24.229 INFO:teuthology.orchestra.run.smithi088.stdout: "exported": 0, 2024-03-12T06:06:24.229 INFO:teuthology.orchestra.run.smithi088.stdout: "exported_inodes": 0, 2024-03-12T06:06:24.229 INFO:teuthology.orchestra.run.smithi088.stdout: "imported": 0, 2024-03-12T06:06:24.229 INFO:teuthology.orchestra.run.smithi088.stdout: "imported_inodes": 0, 2024-03-12T06:06:24.229 INFO:teuthology.orchestra.run.smithi088.stdout: "openino_dir_fetch": 0, 2024-03-12T06:06:24.230 INFO:teuthology.orchestra.run.smithi088.stdout: "openino_backtrace_fetch": 0, 2024-03-12T06:06:24.230 INFO:teuthology.orchestra.run.smithi088.stdout: "openino_peer_discover": 0, 2024-03-12T06:06:24.230 INFO:teuthology.orchestra.run.smithi088.stdout: "root_rfiles": 0, 2024-03-12T06:06:24.230 INFO:teuthology.orchestra.run.smithi088.stdout: "root_rbytes": 0, 2024-03-12T06:06:24.230 INFO:teuthology.orchestra.run.smithi088.stdout: "root_rsnaps": 0, 2024-03-12T06:06:24.230 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_backtrace_fetch": 0, 2024-03-12T06:06:24.230 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_set_tag": 0, 2024-03-12T06:06:24.230 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T06:06:24.231 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_inotable_repaired": 0, 2024-03-12T06:06:24.231 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_dir_inodes": 0, 2024-03-12T06:06:24.231 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_dir_base_inodes": 0, 2024-03-12T06:06:24.231 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_dirfrag_rstats": 0, 2024-03-12T06:06:24.231 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_file_inodes": 0, 2024-03-12T06:06:24.231 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_inode_file_caps": 0, 2024-03-12T06:06:24.231 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_revoke": 0, 2024-03-12T06:06:24.231 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_grant": 0, 2024-03-12T06:06:24.232 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_trunc": 0, 2024-03-12T06:06:24.232 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T06:06:24.232 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T06:06:24.232 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_caps": 0, 2024-03-12T06:06:24.232 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_caps_dirty": 0, 2024-03-12T06:06:24.232 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_cap_release": 0, 2024-03-12T06:06:24.232 INFO:teuthology.orchestra.run.smithi088.stdout: "process_request_cap_release": 0 2024-03-12T06:06:24.232 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.232 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_cache": { 2024-03-12T06:06:24.233 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_update": 0, 2024-03-12T06:06:24.233 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_update_receipt": 0, 2024-03-12T06:06:24.233 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_try_discover": 0, 2024-03-12T06:06:24.233 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_send_discover": 0, 2024-03-12T06:06:24.233 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_handle_discover": 0, 2024-03-12T06:06:24.233 INFO:teuthology.orchestra.run.smithi088.stdout: "num_strays": 0, 2024-03-12T06:06:24.233 INFO:teuthology.orchestra.run.smithi088.stdout: "num_strays_delayed": 0, 2024-03-12T06:06:24.233 INFO:teuthology.orchestra.run.smithi088.stdout: "num_strays_enqueuing": 0, 2024-03-12T06:06:24.234 INFO:teuthology.orchestra.run.smithi088.stdout: "strays_created": 0, 2024-03-12T06:06:24.234 INFO:teuthology.orchestra.run.smithi088.stdout: "strays_enqueued": 0, 2024-03-12T06:06:24.234 INFO:teuthology.orchestra.run.smithi088.stdout: "strays_reintegrated": 0, 2024-03-12T06:06:24.234 INFO:teuthology.orchestra.run.smithi088.stdout: "strays_migrated": 0, 2024-03-12T06:06:24.234 INFO:teuthology.orchestra.run.smithi088.stdout: "num_recovering_processing": 0, 2024-03-12T06:06:24.234 INFO:teuthology.orchestra.run.smithi088.stdout: "num_recovering_enqueued": 0, 2024-03-12T06:06:24.234 INFO:teuthology.orchestra.run.smithi088.stdout: "num_recovering_prioritized": 0, 2024-03-12T06:06:24.234 INFO:teuthology.orchestra.run.smithi088.stdout: "recovery_started": 0, 2024-03-12T06:06:24.235 INFO:teuthology.orchestra.run.smithi088.stdout: "recovery_completed": 0, 2024-03-12T06:06:24.235 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_enqueue_scrub": 0, 2024-03-12T06:06:24.235 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_exportdir": 0, 2024-03-12T06:06:24.235 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_flush": 0, 2024-03-12T06:06:24.235 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_fragmentdir": 0, 2024-03-12T06:06:24.235 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_fragstats": 0, 2024-03-12T06:06:24.235 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_inodestats": 0 2024-03-12T06:06:24.235 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.236 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_log": { 2024-03-12T06:06:24.236 INFO:teuthology.orchestra.run.smithi088.stdout: "evlrg": 0, 2024-03-12T06:06:24.236 INFO:teuthology.orchestra.run.smithi088.stdout: "evadd": 0, 2024-03-12T06:06:24.236 INFO:teuthology.orchestra.run.smithi088.stdout: "evex": 0, 2024-03-12T06:06:24.236 INFO:teuthology.orchestra.run.smithi088.stdout: "evtrm": 0, 2024-03-12T06:06:24.236 INFO:teuthology.orchestra.run.smithi088.stdout: "ev": 2330, 2024-03-12T06:06:24.236 INFO:teuthology.orchestra.run.smithi088.stdout: "evexg": 0, 2024-03-12T06:06:24.236 INFO:teuthology.orchestra.run.smithi088.stdout: "evexd": 0, 2024-03-12T06:06:24.237 INFO:teuthology.orchestra.run.smithi088.stdout: "segadd": 0, 2024-03-12T06:06:24.237 INFO:teuthology.orchestra.run.smithi088.stdout: "segex": 0, 2024-03-12T06:06:24.237 INFO:teuthology.orchestra.run.smithi088.stdout: "segtrm": 0, 2024-03-12T06:06:24.237 INFO:teuthology.orchestra.run.smithi088.stdout: "seg": 4, 2024-03-12T06:06:24.237 INFO:teuthology.orchestra.run.smithi088.stdout: "segmjr": 2, 2024-03-12T06:06:24.237 INFO:teuthology.orchestra.run.smithi088.stdout: "segexg": 0, 2024-03-12T06:06:24.237 INFO:teuthology.orchestra.run.smithi088.stdout: "segexd": 0, 2024-03-12T06:06:24.237 INFO:teuthology.orchestra.run.smithi088.stdout: "expos": 4194304, 2024-03-12T06:06:24.238 INFO:teuthology.orchestra.run.smithi088.stdout: "wrpos": 9898980, 2024-03-12T06:06:24.238 INFO:teuthology.orchestra.run.smithi088.stdout: "rdpos": 9884866, 2024-03-12T06:06:24.238 INFO:teuthology.orchestra.run.smithi088.stdout: "jlat": { 2024-03-12T06:06:24.238 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.238 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.238 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.238 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.238 INFO:teuthology.orchestra.run.smithi088.stdout: "replayed": 2330 2024-03-12T06:06:24.239 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.239 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_mem": { 2024-03-12T06:06:24.239 INFO:teuthology.orchestra.run.smithi088.stdout: "ino": 1969, 2024-03-12T06:06:24.239 INFO:teuthology.orchestra.run.smithi088.stdout: "ino+": 2682, 2024-03-12T06:06:24.239 INFO:teuthology.orchestra.run.smithi088.stdout: "ino-": 713, 2024-03-12T06:06:24.239 INFO:teuthology.orchestra.run.smithi088.stdout: "dir": 171, 2024-03-12T06:06:24.239 INFO:teuthology.orchestra.run.smithi088.stdout: "dir+": 171, 2024-03-12T06:06:24.239 INFO:teuthology.orchestra.run.smithi088.stdout: "dir-": 0, 2024-03-12T06:06:24.239 INFO:teuthology.orchestra.run.smithi088.stdout: "dn": 2936, 2024-03-12T06:06:24.240 INFO:teuthology.orchestra.run.smithi088.stdout: "dn+": 3421, 2024-03-12T06:06:24.240 INFO:teuthology.orchestra.run.smithi088.stdout: "dn-": 485, 2024-03-12T06:06:24.240 INFO:teuthology.orchestra.run.smithi088.stdout: "cap": 0, 2024-03-12T06:06:24.240 INFO:teuthology.orchestra.run.smithi088.stdout: "cap+": 0, 2024-03-12T06:06:24.240 INFO:teuthology.orchestra.run.smithi088.stdout: "cap-": 0, 2024-03-12T06:06:24.240 INFO:teuthology.orchestra.run.smithi088.stdout: "rss": 56340, 2024-03-12T06:06:24.240 INFO:teuthology.orchestra.run.smithi088.stdout: "heap": 215420 2024-03-12T06:06:24.240 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.241 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_server": { 2024-03-12T06:06:24.241 INFO:teuthology.orchestra.run.smithi088.stdout: "dispatch_client_request": 0, 2024-03-12T06:06:24.241 INFO:teuthology.orchestra.run.smithi088.stdout: "dispatch_server_request": 0, 2024-03-12T06:06:24.241 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_request": 0, 2024-03-12T06:06:24.241 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_session": 0, 2024-03-12T06:06:24.241 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_peer_request": 0, 2024-03-12T06:06:24.241 INFO:teuthology.orchestra.run.smithi088.stdout: "req_create_latency": { 2024-03-12T06:06:24.241 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.241 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.242 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.242 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.242 INFO:teuthology.orchestra.run.smithi088.stdout: "req_getattr_latency": { 2024-03-12T06:06:24.242 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.242 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.242 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.242 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.242 INFO:teuthology.orchestra.run.smithi088.stdout: "req_getfilelock_latency": { 2024-03-12T06:06:24.243 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.243 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.243 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.243 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.243 INFO:teuthology.orchestra.run.smithi088.stdout: "req_link_latency": { 2024-03-12T06:06:24.243 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.243 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.243 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.244 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookup_latency": { 2024-03-12T06:06:24.244 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.244 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.244 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.244 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.244 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookuphash_latency": { 2024-03-12T06:06:24.244 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.244 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.245 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.245 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookupino_latency": { 2024-03-12T06:06:24.245 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.245 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.245 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.245 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.245 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookupname_latency": { 2024-03-12T06:06:24.245 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.246 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.246 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.246 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookupparent_latency": { 2024-03-12T06:06:24.246 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.246 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.246 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.246 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.247 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookupsnap_latency": { 2024-03-12T06:06:24.247 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.247 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.247 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.247 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.247 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lssnap_latency": { 2024-03-12T06:06:24.247 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.247 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.247 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.248 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.248 INFO:teuthology.orchestra.run.smithi088.stdout: "req_mkdir_latency": { 2024-03-12T06:06:24.248 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.248 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.249 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.249 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.249 INFO:teuthology.orchestra.run.smithi088.stdout: "req_mknod_latency": { 2024-03-12T06:06:24.249 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.249 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.250 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.250 INFO:teuthology.orchestra.run.smithi088.stdout: "req_mksnap_latency": { 2024-03-12T06:06:24.250 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.250 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.250 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.250 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.250 INFO:teuthology.orchestra.run.smithi088.stdout: "req_open_latency": { 2024-03-12T06:06:24.251 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.251 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.251 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.251 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.251 INFO:teuthology.orchestra.run.smithi088.stdout: "req_readdir_latency": { 2024-03-12T06:06:24.251 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.251 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.251 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.252 INFO:teuthology.orchestra.run.smithi088.stdout: "req_rename_latency": { 2024-03-12T06:06:24.252 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.252 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.252 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.252 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.253 INFO:teuthology.orchestra.run.smithi088.stdout: "req_renamesnap_latency": { 2024-03-12T06:06:24.253 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.253 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.253 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.253 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.253 INFO:teuthology.orchestra.run.smithi088.stdout: "req_snapdiff_latency": { 2024-03-12T06:06:24.253 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.254 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.254 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.254 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.254 INFO:teuthology.orchestra.run.smithi088.stdout: "req_rmdir_latency": { 2024-03-12T06:06:24.254 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.254 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.254 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.255 INFO:teuthology.orchestra.run.smithi088.stdout: "req_rmsnap_latency": { 2024-03-12T06:06:24.255 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.255 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.255 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.255 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.255 INFO:teuthology.orchestra.run.smithi088.stdout: "req_rmxattr_latency": { 2024-03-12T06:06:24.255 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.256 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.256 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.256 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setattr_latency": { 2024-03-12T06:06:24.256 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.256 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.256 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.256 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.256 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setdirlayout_latency": { 2024-03-12T06:06:24.257 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.257 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.257 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.257 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setfilelock_latency": { 2024-03-12T06:06:24.257 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.257 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.257 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.257 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.257 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setlayout_latency": { 2024-03-12T06:06:24.258 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.258 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.258 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.258 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.258 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setxattr_latency": { 2024-03-12T06:06:24.258 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.258 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.258 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.259 INFO:teuthology.orchestra.run.smithi088.stdout: "req_symlink_latency": { 2024-03-12T06:06:24.259 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.259 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.259 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.259 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.259 INFO:teuthology.orchestra.run.smithi088.stdout: "req_unlink_latency": { 2024-03-12T06:06:24.259 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.259 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.260 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.260 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.260 INFO:teuthology.orchestra.run.smithi088.stdout: "cap_revoke_eviction": 0, 2024-03-12T06:06:24.260 INFO:teuthology.orchestra.run.smithi088.stdout: "cap_acquisition_throttle": 0, 2024-03-12T06:06:24.260 INFO:teuthology.orchestra.run.smithi088.stdout: "req_getvxattr_latency": { 2024-03-12T06:06:24.260 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.260 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.260 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.260 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:24.261 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.261 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_sessions": { 2024-03-12T06:06:24.261 INFO:teuthology.orchestra.run.smithi088.stdout: "session_count": 2, 2024-03-12T06:06:24.261 INFO:teuthology.orchestra.run.smithi088.stdout: "session_add": 4, 2024-03-12T06:06:24.261 INFO:teuthology.orchestra.run.smithi088.stdout: "session_remove": 2, 2024-03-12T06:06:24.261 INFO:teuthology.orchestra.run.smithi088.stdout: "sessions_open": 3, 2024-03-12T06:06:24.261 INFO:teuthology.orchestra.run.smithi088.stdout: "sessions_stale": 0, 2024-03-12T06:06:24.261 INFO:teuthology.orchestra.run.smithi088.stdout: "total_load": 0, 2024-03-12T06:06:24.262 INFO:teuthology.orchestra.run.smithi088.stdout: "average_load": 0, 2024-03-12T06:06:24.262 INFO:teuthology.orchestra.run.smithi088.stdout: "avg_session_uptime": 3317, 2024-03-12T06:06:24.262 INFO:teuthology.orchestra.run.smithi088.stdout: "mdthresh_evicted": 0 2024-03-12T06:06:24.262 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.262 INFO:teuthology.orchestra.run.smithi088.stdout: "mempool": { 2024-03-12T06:06:24.262 INFO:teuthology.orchestra.run.smithi088.stdout: "bloom_filter_bytes": 0, 2024-03-12T06:06:24.262 INFO:teuthology.orchestra.run.smithi088.stdout: "bloom_filter_items": 0, 2024-03-12T06:06:24.262 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T06:06:24.262 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_alloc_items": 0, 2024-03-12T06:06:24.263 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T06:06:24.263 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_data_items": 0, 2024-03-12T06:06:24.263 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T06:06:24.263 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T06:06:24.263 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T06:06:24.263 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T06:06:24.263 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T06:06:24.263 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_other_items": 0, 2024-03-12T06:06:24.263 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T06:06:24.264 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T06:06:24.264 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_extent_bytes": 0, 2024-03-12T06:06:24.264 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_extent_items": 0, 2024-03-12T06:06:24.264 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_blob_bytes": 0, 2024-03-12T06:06:24.264 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_blob_items": 0, 2024-03-12T06:06:24.264 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T06:06:24.264 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T06:06:24.264 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T06:06:24.264 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T06:06:24.265 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T06:06:24.265 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_items": 0, 2024-03-12T06:06:24.265 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_txc_bytes": 0, 2024-03-12T06:06:24.265 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_txc_items": 0, 2024-03-12T06:06:24.265 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T06:06:24.265 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T06:06:24.265 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_bytes": 0, 2024-03-12T06:06:24.265 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_items": 0, 2024-03-12T06:06:24.266 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_bytes": 0, 2024-03-12T06:06:24.266 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_items": 0, 2024-03-12T06:06:24.266 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T06:06:24.266 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_reader_items": 0, 2024-03-12T06:06:24.266 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T06:06:24.266 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_writer_items": 0, 2024-03-12T06:06:24.266 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_anon_bytes": 254008, 2024-03-12T06:06:24.266 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_anon_items": 2011, 2024-03-12T06:06:24.266 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_meta_bytes": 0, 2024-03-12T06:06:24.267 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_meta_items": 0, 2024-03-12T06:06:24.267 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_bytes": 0, 2024-03-12T06:06:24.267 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_items": 0, 2024-03-12T06:06:24.267 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mapbl_bytes": 0, 2024-03-12T06:06:24.267 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mapbl_items": 0, 2024-03-12T06:06:24.267 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pglog_bytes": 0, 2024-03-12T06:06:24.267 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pglog_items": 0, 2024-03-12T06:06:24.267 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_bytes": 8032, 2024-03-12T06:06:24.267 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_items": 167, 2024-03-12T06:06:24.268 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T06:06:24.268 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_mapping_items": 0, 2024-03-12T06:06:24.268 INFO:teuthology.orchestra.run.smithi088.stdout: "pgmap_bytes": 0, 2024-03-12T06:06:24.268 INFO:teuthology.orchestra.run.smithi088.stdout: "pgmap_items": 0, 2024-03-12T06:06:24.268 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_co_bytes": 6692140, 2024-03-12T06:06:24.268 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_co_items": 57273, 2024-03-12T06:06:24.268 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_1_bytes": 0, 2024-03-12T06:06:24.268 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_1_items": 0, 2024-03-12T06:06:24.268 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_2_bytes": 0, 2024-03-12T06:06:24.269 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_2_items": 0 2024-03-12T06:06:24.269 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.269 INFO:teuthology.orchestra.run.smithi088.stdout: "objecter": { 2024-03-12T06:06:24.269 INFO:teuthology.orchestra.run.smithi088.stdout: "op_active": 0, 2024-03-12T06:06:24.269 INFO:teuthology.orchestra.run.smithi088.stdout: "op_laggy": 0, 2024-03-12T06:06:24.269 INFO:teuthology.orchestra.run.smithi088.stdout: "op_send": 2224, 2024-03-12T06:06:24.269 INFO:teuthology.orchestra.run.smithi088.stdout: "op_send_bytes": 21, 2024-03-12T06:06:24.269 INFO:teuthology.orchestra.run.smithi088.stdout: "op_resend": 3, 2024-03-12T06:06:24.269 INFO:teuthology.orchestra.run.smithi088.stdout: "op_reply": 2224, 2024-03-12T06:06:24.270 INFO:teuthology.orchestra.run.smithi088.stdout: "op_latency": { 2024-03-12T06:06:24.270 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 2224, 2024-03-12T06:06:24.270 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.959982166, 2024-03-12T06:06:24.270 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000431646 2024-03-12T06:06:24.270 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.270 INFO:teuthology.orchestra.run.smithi088.stdout: "op_inflight": 0, 2024-03-12T06:06:24.270 INFO:teuthology.orchestra.run.smithi088.stdout: "oplen_avg": { 2024-03-12T06:06:24.270 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 2224, 2024-03-12T06:06:24.270 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 2225 2024-03-12T06:06:24.271 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.271 INFO:teuthology.orchestra.run.smithi088.stdout: "op": 2224, 2024-03-12T06:06:24.271 INFO:teuthology.orchestra.run.smithi088.stdout: "op_r": 2224, 2024-03-12T06:06:24.271 INFO:teuthology.orchestra.run.smithi088.stdout: "op_w": 0, 2024-03-12T06:06:24.271 INFO:teuthology.orchestra.run.smithi088.stdout: "op_rmw": 0, 2024-03-12T06:06:24.271 INFO:teuthology.orchestra.run.smithi088.stdout: "op_pg": 0, 2024-03-12T06:06:24.271 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_stat": 1434, 2024-03-12T06:06:24.271 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_create": 0, 2024-03-12T06:06:24.271 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_read": 788, 2024-03-12T06:06:24.272 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_write": 0, 2024-03-12T06:06:24.272 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_writefull": 0, 2024-03-12T06:06:24.272 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_writesame": 0, 2024-03-12T06:06:24.272 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_append": 0, 2024-03-12T06:06:24.272 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_zero": 0, 2024-03-12T06:06:24.272 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_truncate": 0, 2024-03-12T06:06:24.272 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_delete": 0, 2024-03-12T06:06:24.272 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_mapext": 0, 2024-03-12T06:06:24.273 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_sparse_read": 0, 2024-03-12T06:06:24.273 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_clonerange": 0, 2024-03-12T06:06:24.273 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_getxattr": 1, 2024-03-12T06:06:24.273 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_setxattr": 0, 2024-03-12T06:06:24.273 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_cmpxattr": 0, 2024-03-12T06:06:24.273 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_rmxattr": 0, 2024-03-12T06:06:24.273 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_resetxattrs": 0, 2024-03-12T06:06:24.273 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_call": 0, 2024-03-12T06:06:24.273 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_watch": 0, 2024-03-12T06:06:24.274 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_notify": 0, 2024-03-12T06:06:24.274 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T06:06:24.274 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_pgls": 0, 2024-03-12T06:06:24.274 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_pgls_filter": 0, 2024-03-12T06:06:24.274 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_other": 0, 2024-03-12T06:06:24.274 INFO:teuthology.orchestra.run.smithi088.stdout: "linger_active": 0, 2024-03-12T06:06:24.274 INFO:teuthology.orchestra.run.smithi088.stdout: "linger_send": 0, 2024-03-12T06:06:24.274 INFO:teuthology.orchestra.run.smithi088.stdout: "linger_resend": 0, 2024-03-12T06:06:24.275 INFO:teuthology.orchestra.run.smithi088.stdout: "linger_ping": 0, 2024-03-12T06:06:24.275 INFO:teuthology.orchestra.run.smithi088.stdout: "poolop_active": 0, 2024-03-12T06:06:24.275 INFO:teuthology.orchestra.run.smithi088.stdout: "poolop_send": 0, 2024-03-12T06:06:24.275 INFO:teuthology.orchestra.run.smithi088.stdout: "poolop_resend": 0, 2024-03-12T06:06:24.275 INFO:teuthology.orchestra.run.smithi088.stdout: "poolstat_active": 0, 2024-03-12T06:06:24.275 INFO:teuthology.orchestra.run.smithi088.stdout: "poolstat_send": 0, 2024-03-12T06:06:24.275 INFO:teuthology.orchestra.run.smithi088.stdout: "poolstat_resend": 0, 2024-03-12T06:06:24.275 INFO:teuthology.orchestra.run.smithi088.stdout: "statfs_active": 0, 2024-03-12T06:06:24.275 INFO:teuthology.orchestra.run.smithi088.stdout: "statfs_send": 0, 2024-03-12T06:06:24.275 INFO:teuthology.orchestra.run.smithi088.stdout: "statfs_resend": 0, 2024-03-12T06:06:24.276 INFO:teuthology.orchestra.run.smithi088.stdout: "command_active": 0, 2024-03-12T06:06:24.276 INFO:teuthology.orchestra.run.smithi088.stdout: "command_send": 0, 2024-03-12T06:06:24.276 INFO:teuthology.orchestra.run.smithi088.stdout: "command_resend": 0, 2024-03-12T06:06:24.276 INFO:teuthology.orchestra.run.smithi088.stdout: "map_epoch": 97, 2024-03-12T06:06:24.276 INFO:teuthology.orchestra.run.smithi088.stdout: "map_full": 0, 2024-03-12T06:06:24.276 INFO:teuthology.orchestra.run.smithi088.stdout: "map_inc": 11, 2024-03-12T06:06:24.276 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_sessions": 4, 2024-03-12T06:06:24.276 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_session_open": 4, 2024-03-12T06:06:24.276 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_session_close": 0, 2024-03-12T06:06:24.277 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_laggy": 0, 2024-03-12T06:06:24.277 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_wr": 0, 2024-03-12T06:06:24.277 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_rd": 2, 2024-03-12T06:06:24.277 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_del": 0 2024-03-12T06:06:24.277 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.277 INFO:teuthology.orchestra.run.smithi088.stdout: "oft": { 2024-03-12T06:06:24.277 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_total_objs": 0, 2024-03-12T06:06:24.277 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_total_kv_pairs": 0, 2024-03-12T06:06:24.278 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_total_updates": 0, 2024-03-12T06:06:24.278 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_total_removes": 0 2024-03-12T06:06:24.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.278 INFO:teuthology.orchestra.run.smithi088.stdout: "osd-slow-ops": { 2024-03-12T06:06:24.278 INFO:teuthology.orchestra.run.smithi088.stdout: "slow_ops_count": 0 2024-03-12T06:06:24.278 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.278 INFO:teuthology.orchestra.run.smithi088.stdout: "purge_queue": { 2024-03-12T06:06:24.278 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executing_ops": 0, 2024-03-12T06:06:24.279 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executing_ops_high_water": 0, 2024-03-12T06:06:24.279 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executing": 0, 2024-03-12T06:06:24.279 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executing_high_water": 0, 2024-03-12T06:06:24.279 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executed_ops": 0, 2024-03-12T06:06:24.279 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executed": 0, 2024-03-12T06:06:24.279 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_item_in_journal": 0 2024-03-12T06:06:24.279 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.279 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T06:06:24.280 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:24.280 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 104857600, 2024-03-12T06:06:24.280 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:24.280 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 2475, 2024-03-12T06:06:24.280 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 7726781, 2024-03-12T06:06:24.280 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:24.280 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 2475, 2024-03-12T06:06:24.280 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 0, 2024-03-12T06:06:24.280 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 0, 2024-03-12T06:06:24.281 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 2475, 2024-03-12T06:06:24.281 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 7726781, 2024-03-12T06:06:24.281 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:24.281 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.281 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.281 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.281 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:24.281 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.281 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-objecter_bytes": { 2024-03-12T06:06:24.282 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:24.282 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 104857600, 2024-03-12T06:06:24.282 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:24.282 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 0, 2024-03-12T06:06:24.282 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 0, 2024-03-12T06:06:24.282 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:24.282 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:24.282 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 2224, 2024-03-12T06:06:24.283 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 5710465, 2024-03-12T06:06:24.283 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 2224, 2024-03-12T06:06:24.283 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 5710465, 2024-03-12T06:06:24.283 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:24.283 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.283 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.283 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.283 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:24.283 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.284 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-objecter_ops": { 2024-03-12T06:06:24.284 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:24.284 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 1024, 2024-03-12T06:06:24.284 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:24.284 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 0, 2024-03-12T06:06:24.284 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 0, 2024-03-12T06:06:24.284 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:24.284 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:24.284 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 2224, 2024-03-12T06:06:24.285 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 2224, 2024-03-12T06:06:24.285 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 2224, 2024-03-12T06:06:24.285 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 2224, 2024-03-12T06:06:24.285 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:24.285 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.285 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.285 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.285 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:24.285 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.286 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-write_buf_throttle": { 2024-03-12T06:06:24.286 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:24.286 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 3758096384, 2024-03-12T06:06:24.286 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:24.286 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 0, 2024-03-12T06:06:24.286 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 0, 2024-03-12T06:06:24.286 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:24.286 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:24.287 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 0, 2024-03-12T06:06:24.287 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 0, 2024-03-12T06:06:24.287 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 0, 2024-03-12T06:06:24.287 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 0, 2024-03-12T06:06:24.287 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:24.287 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.287 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.287 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.287 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:24.287 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:24.288 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-write_buf_throttle-0x557b2b024960": { 2024-03-12T06:06:24.288 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:24.288 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 3758096384, 2024-03-12T06:06:24.288 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:24.288 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 0, 2024-03-12T06:06:24.288 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 0, 2024-03-12T06:06:24.288 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:24.288 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:24.289 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 0, 2024-03-12T06:06:24.289 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 0, 2024-03-12T06:06:24.289 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 0, 2024-03-12T06:06:24.289 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 0, 2024-03-12T06:06:24.289 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:24.289 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:24.289 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:24.289 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:24.289 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:24.290 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:24.290 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-03-12T06:06:24.855 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-12T06:06:24.856 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-12T06:06:24.856 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-03-12T06:06:24.856 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-12T06:06:24.856 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-12T06:06:24.856 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-12T06:06:24.856 DEBUG:tasks.check_counter:Getting stats from f 2024-03-12T06:06:24.856 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-12T06:06:25.101 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T06:06:25.481 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:25 smithi088 ceph-mon[36585]: pgmap v330: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:06:25.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:25 smithi082 ceph-mon[36382]: pgmap v330: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:06:25.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:25 smithi076 ceph-mon[32095]: pgmap v330: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:06:26.763 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-03-12T06:06:26.763 INFO:teuthology.orchestra.run.smithi088.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T06:06:26.763 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_messages": 8109, 2024-03-12T06:06:26.764 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages": 10229, 2024-03-12T06:06:26.764 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_bytes": 2140402, 2024-03-12T06:06:26.764 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_bytes": 7678928, 2024-03-12T06:06:26.764 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_created_connections": 52, 2024-03-12T06:06:26.764 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_active_connections": 3, 2024-03-12T06:06:26.764 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_total_time": 0.538746864, 2024-03-12T06:06:26.764 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_send_time": 0.175977513, 2024-03-12T06:06:26.765 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_recv_time": 0.180643879, 2024-03-12T06:06:26.765 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_fast_dispatch_time": 0.161066242, 2024-03-12T06:06:26.765 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:26.765 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 10227, 2024-03-12T06:06:26.765 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.390341735, 2024-03-12T06:06:26.765 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000038167 2024-03-12T06:06:26.765 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.766 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:26.766 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 3070, 2024-03-12T06:06:26.766 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000674129, 2024-03-12T06:06:26.766 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000219 2024-03-12T06:06:26.766 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.766 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-12T06:06:26.767 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_encrypted_bytes": 0 2024-03-12T06:06:26.767 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.767 INFO:teuthology.orchestra.run.smithi088.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T06:06:26.767 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_messages": 9500, 2024-03-12T06:06:26.767 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages": 9483, 2024-03-12T06:06:26.767 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_bytes": 3966375, 2024-03-12T06:06:26.768 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_bytes": 3226272, 2024-03-12T06:06:26.768 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_created_connections": 12, 2024-03-12T06:06:26.768 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_active_connections": 7, 2024-03-12T06:06:26.768 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_total_time": 0.857978792, 2024-03-12T06:06:26.768 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_send_time": 0.424745600, 2024-03-12T06:06:26.769 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_recv_time": 0.181963431, 2024-03-12T06:06:26.769 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_fast_dispatch_time": 0.213540211, 2024-03-12T06:06:26.769 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:26.769 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 9480, 2024-03-12T06:06:26.769 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.545409739, 2024-03-12T06:06:26.769 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000057532 2024-03-12T06:06:26.769 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.770 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:26.770 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 11, 2024-03-12T06:06:26.773 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000001697, 2024-03-12T06:06:26.773 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000154 2024-03-12T06:06:26.774 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.774 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_encrypted_bytes": 1634272, 2024-03-12T06:06:26.774 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_encrypted_bytes": 150416 2024-03-12T06:06:26.774 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.774 INFO:teuthology.orchestra.run.smithi088.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T06:06:26.775 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_messages": 8959, 2024-03-12T06:06:26.775 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages": 10562, 2024-03-12T06:06:26.775 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_bytes": 4822801, 2024-03-12T06:06:26.775 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_bytes": 3872638, 2024-03-12T06:06:26.776 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_created_connections": 9, 2024-03-12T06:06:26.776 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_active_connections": 5, 2024-03-12T06:06:26.776 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_total_time": 0.603084450, 2024-03-12T06:06:26.776 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_send_time": 0.201997790, 2024-03-12T06:06:26.776 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_recv_time": 0.237608804, 2024-03-12T06:06:26.776 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_fast_dispatch_time": 0.153056745, 2024-03-12T06:06:26.776 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:26.777 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 10562, 2024-03-12T06:06:26.777 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.598329995, 2024-03-12T06:06:26.777 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000056649 2024-03-12T06:06:26.777 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.777 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:26.777 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 6241, 2024-03-12T06:06:26.777 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.001255754, 2024-03-12T06:06:26.778 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000201 2024-03-12T06:06:26.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.778 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-12T06:06:26.778 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_encrypted_bytes": 444144 2024-03-12T06:06:26.778 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.778 INFO:teuthology.orchestra.run.smithi088.stdout: "cct": { 2024-03-12T06:06:26.779 INFO:teuthology.orchestra.run.smithi088.stdout: "total_workers": 1, 2024-03-12T06:06:26.779 INFO:teuthology.orchestra.run.smithi088.stdout: "unhealthy_workers": 0 2024-03-12T06:06:26.779 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.779 INFO:teuthology.orchestra.run.smithi088.stdout: "finisher-MDSRank": { 2024-03-12T06:06:26.779 INFO:teuthology.orchestra.run.smithi088.stdout: "queue_len": 0, 2024-03-12T06:06:26.779 INFO:teuthology.orchestra.run.smithi088.stdout: "complete_latency": { 2024-03-12T06:06:26.780 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 13976, 2024-03-12T06:06:26.780 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 3.580695020, 2024-03-12T06:06:26.780 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000256203 2024-03-12T06:06:26.780 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:26.780 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.780 INFO:teuthology.orchestra.run.smithi088.stdout: "finisher-PurgeQueue": { 2024-03-12T06:06:26.781 INFO:teuthology.orchestra.run.smithi088.stdout: "queue_len": 0, 2024-03-12T06:06:26.781 INFO:teuthology.orchestra.run.smithi088.stdout: "complete_latency": { 2024-03-12T06:06:26.781 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 266, 2024-03-12T06:06:26.781 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.018305553, 2024-03-12T06:06:26.781 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000068817 2024-03-12T06:06:26.781 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:26.781 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.782 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": { 2024-03-12T06:06:26.782 INFO:teuthology.orchestra.run.smithi088.stdout: "request": 714, 2024-03-12T06:06:26.782 INFO:teuthology.orchestra.run.smithi088.stdout: "reply": 707, 2024-03-12T06:06:26.782 INFO:teuthology.orchestra.run.smithi088.stdout: "reply_latency": { 2024-03-12T06:06:26.783 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 707, 2024-03-12T06:06:26.783 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.625233866, 2024-03-12T06:06:26.783 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000884347 2024-03-12T06:06:26.783 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.783 INFO:teuthology.orchestra.run.smithi088.stdout: "slow_reply": 0, 2024-03-12T06:06:26.783 INFO:teuthology.orchestra.run.smithi088.stdout: "forward": 7, 2024-03-12T06:06:26.784 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_fetch_complete": 0, 2024-03-12T06:06:26.784 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_fetch_keys": 0, 2024-03-12T06:06:26.784 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_commit": 11, 2024-03-12T06:06:26.784 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_split": 0, 2024-03-12T06:06:26.784 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_merge": 0, 2024-03-12T06:06:26.785 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes": 2375, 2024-03-12T06:06:26.785 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_top": 56, 2024-03-12T06:06:26.785 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_bottom": 2319, 2024-03-12T06:06:26.785 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_pin_tail": 0, 2024-03-12T06:06:26.785 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_pinned": 2296, 2024-03-12T06:06:26.786 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_expired": 0, 2024-03-12T06:06:26.786 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_with_caps": 303, 2024-03-12T06:06:26.786 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": 303, 2024-03-12T06:06:26.786 INFO:teuthology.orchestra.run.smithi088.stdout: "subtrees": 72, 2024-03-12T06:06:26.787 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse": 1164, 2024-03-12T06:06:26.787 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_hit": 757, 2024-03-12T06:06:26.787 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_forward": 6, 2024-03-12T06:06:26.787 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_discover": 21, 2024-03-12T06:06:26.788 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_dir_fetch": 0, 2024-03-12T06:06:26.788 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_remote_ino": 0, 2024-03-12T06:06:26.788 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_lock": 0, 2024-03-12T06:06:26.788 INFO:teuthology.orchestra.run.smithi088.stdout: "load_cent": 0, 2024-03-12T06:06:26.788 INFO:teuthology.orchestra.run.smithi088.stdout: "q": 0, 2024-03-12T06:06:26.789 INFO:teuthology.orchestra.run.smithi088.stdout: "exported": 0, 2024-03-12T06:06:26.789 INFO:teuthology.orchestra.run.smithi088.stdout: "exported_inodes": 0, 2024-03-12T06:06:26.789 INFO:teuthology.orchestra.run.smithi088.stdout: "imported": 69, 2024-03-12T06:06:26.789 INFO:teuthology.orchestra.run.smithi088.stdout: "imported_inodes": 2098, 2024-03-12T06:06:26.790 INFO:teuthology.orchestra.run.smithi088.stdout: "openino_dir_fetch": 0, 2024-03-12T06:06:26.790 INFO:teuthology.orchestra.run.smithi088.stdout: "openino_backtrace_fetch": 21855, 2024-03-12T06:06:26.790 INFO:teuthology.orchestra.run.smithi088.stdout: "openino_peer_discover": 0, 2024-03-12T06:06:26.790 INFO:teuthology.orchestra.run.smithi088.stdout: "root_rfiles": 3452, 2024-03-12T06:06:26.790 INFO:teuthology.orchestra.run.smithi088.stdout: "root_rbytes": 26524023, 2024-03-12T06:06:26.791 INFO:teuthology.orchestra.run.smithi088.stdout: "root_rsnaps": 7, 2024-03-12T06:06:26.791 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_backtrace_fetch": 21855, 2024-03-12T06:06:26.791 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_set_tag": 0, 2024-03-12T06:06:26.791 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T06:06:26.792 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_inotable_repaired": 0, 2024-03-12T06:06:26.792 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_dir_inodes": 570, 2024-03-12T06:06:26.792 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_dir_base_inodes": 0, 2024-03-12T06:06:26.792 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_dirfrag_rstats": 570, 2024-03-12T06:06:26.792 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_file_inodes": 21285, 2024-03-12T06:06:26.792 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_inode_file_caps": 0, 2024-03-12T06:06:26.793 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_revoke": 10, 2024-03-12T06:06:26.793 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_grant": 355, 2024-03-12T06:06:26.793 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_trunc": 0, 2024-03-12T06:06:26.793 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T06:06:26.793 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T06:06:26.794 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_caps": 646, 2024-03-12T06:06:26.794 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_caps_dirty": 628, 2024-03-12T06:06:26.794 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_cap_release": 17, 2024-03-12T06:06:26.794 INFO:teuthology.orchestra.run.smithi088.stdout: "process_request_cap_release": 340 2024-03-12T06:06:26.794 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.795 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_cache": { 2024-03-12T06:06:26.795 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_update": 0, 2024-03-12T06:06:26.795 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_update_receipt": 0, 2024-03-12T06:06:26.795 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_try_discover": 0, 2024-03-12T06:06:26.795 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_send_discover": 21, 2024-03-12T06:06:26.795 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_handle_discover": 50, 2024-03-12T06:06:26.796 INFO:teuthology.orchestra.run.smithi088.stdout: "num_strays": 0, 2024-03-12T06:06:26.796 INFO:teuthology.orchestra.run.smithi088.stdout: "num_strays_delayed": 0, 2024-03-12T06:06:26.796 INFO:teuthology.orchestra.run.smithi088.stdout: "num_strays_enqueuing": 0, 2024-03-12T06:06:26.796 INFO:teuthology.orchestra.run.smithi088.stdout: "strays_created": 150, 2024-03-12T06:06:26.796 INFO:teuthology.orchestra.run.smithi088.stdout: "strays_enqueued": 150, 2024-03-12T06:06:26.797 INFO:teuthology.orchestra.run.smithi088.stdout: "strays_reintegrated": 0, 2024-03-12T06:06:26.797 INFO:teuthology.orchestra.run.smithi088.stdout: "strays_migrated": 0, 2024-03-12T06:06:26.797 INFO:teuthology.orchestra.run.smithi088.stdout: "num_recovering_processing": 0, 2024-03-12T06:06:26.797 INFO:teuthology.orchestra.run.smithi088.stdout: "num_recovering_enqueued": 0, 2024-03-12T06:06:26.798 INFO:teuthology.orchestra.run.smithi088.stdout: "num_recovering_prioritized": 0, 2024-03-12T06:06:26.798 INFO:teuthology.orchestra.run.smithi088.stdout: "recovery_started": 0, 2024-03-12T06:06:26.798 INFO:teuthology.orchestra.run.smithi088.stdout: "recovery_completed": 0, 2024-03-12T06:06:26.798 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_enqueue_scrub": 0, 2024-03-12T06:06:26.798 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_exportdir": 0, 2024-03-12T06:06:26.798 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_flush": 0, 2024-03-12T06:06:26.799 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_fragmentdir": 0, 2024-03-12T06:06:26.799 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_fragstats": 0, 2024-03-12T06:06:26.799 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_inodestats": 0 2024-03-12T06:06:26.799 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.800 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_log": { 2024-03-12T06:06:26.800 INFO:teuthology.orchestra.run.smithi088.stdout: "evlrg": 0, 2024-03-12T06:06:26.800 INFO:teuthology.orchestra.run.smithi088.stdout: "evadd": 1783, 2024-03-12T06:06:26.800 INFO:teuthology.orchestra.run.smithi088.stdout: "evex": 0, 2024-03-12T06:06:26.800 INFO:teuthology.orchestra.run.smithi088.stdout: "evtrm": 0, 2024-03-12T06:06:26.800 INFO:teuthology.orchestra.run.smithi088.stdout: "ev": 1783, 2024-03-12T06:06:26.801 INFO:teuthology.orchestra.run.smithi088.stdout: "evexg": 0, 2024-03-12T06:06:26.801 INFO:teuthology.orchestra.run.smithi088.stdout: "evexd": 0, 2024-03-12T06:06:26.801 INFO:teuthology.orchestra.run.smithi088.stdout: "segadd": 3, 2024-03-12T06:06:26.801 INFO:teuthology.orchestra.run.smithi088.stdout: "segex": 0, 2024-03-12T06:06:26.801 INFO:teuthology.orchestra.run.smithi088.stdout: "segtrm": 0, 2024-03-12T06:06:26.802 INFO:teuthology.orchestra.run.smithi088.stdout: "seg": 3, 2024-03-12T06:06:26.802 INFO:teuthology.orchestra.run.smithi088.stdout: "segmjr": 2, 2024-03-12T06:06:26.802 INFO:teuthology.orchestra.run.smithi088.stdout: "segexg": 0, 2024-03-12T06:06:26.802 INFO:teuthology.orchestra.run.smithi088.stdout: "segexd": 0, 2024-03-12T06:06:26.802 INFO:teuthology.orchestra.run.smithi088.stdout: "expos": 4194304, 2024-03-12T06:06:26.803 INFO:teuthology.orchestra.run.smithi088.stdout: "wrpos": 8057499, 2024-03-12T06:06:26.803 INFO:teuthology.orchestra.run.smithi088.stdout: "rdpos": 0, 2024-03-12T06:06:26.803 INFO:teuthology.orchestra.run.smithi088.stdout: "jlat": { 2024-03-12T06:06:26.803 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 116, 2024-03-12T06:06:26.803 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.627343003, 2024-03-12T06:06:26.804 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.005408129 2024-03-12T06:06:26.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.804 INFO:teuthology.orchestra.run.smithi088.stdout: "replayed": 0 2024-03-12T06:06:26.804 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.804 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_mem": { 2024-03-12T06:06:26.805 INFO:teuthology.orchestra.run.smithi088.stdout: "ino": 1950, 2024-03-12T06:06:26.805 INFO:teuthology.orchestra.run.smithi088.stdout: "ino+": 2100, 2024-03-12T06:06:26.805 INFO:teuthology.orchestra.run.smithi088.stdout: "ino-": 150, 2024-03-12T06:06:26.805 INFO:teuthology.orchestra.run.smithi088.stdout: "dir": 127, 2024-03-12T06:06:26.805 INFO:teuthology.orchestra.run.smithi088.stdout: "dir+": 128, 2024-03-12T06:06:26.806 INFO:teuthology.orchestra.run.smithi088.stdout: "dir-": 1, 2024-03-12T06:06:26.806 INFO:teuthology.orchestra.run.smithi088.stdout: "dn": 2375, 2024-03-12T06:06:26.806 INFO:teuthology.orchestra.run.smithi088.stdout: "dn+": 2542, 2024-03-12T06:06:26.806 INFO:teuthology.orchestra.run.smithi088.stdout: "dn-": 167, 2024-03-12T06:06:26.807 INFO:teuthology.orchestra.run.smithi088.stdout: "cap": 303, 2024-03-12T06:06:26.807 INFO:teuthology.orchestra.run.smithi088.stdout: "cap+": 1613, 2024-03-12T06:06:26.807 INFO:teuthology.orchestra.run.smithi088.stdout: "cap-": 1310, 2024-03-12T06:06:26.807 INFO:teuthology.orchestra.run.smithi088.stdout: "rss": 69536, 2024-03-12T06:06:26.807 INFO:teuthology.orchestra.run.smithi088.stdout: "heap": 207228 2024-03-12T06:06:26.808 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.808 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_server": { 2024-03-12T06:06:26.808 INFO:teuthology.orchestra.run.smithi088.stdout: "dispatch_client_request": 793, 2024-03-12T06:06:26.808 INFO:teuthology.orchestra.run.smithi088.stdout: "dispatch_server_request": 0, 2024-03-12T06:06:26.808 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_request": 714, 2024-03-12T06:06:26.808 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_session": 13, 2024-03-12T06:06:26.809 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_peer_request": 21, 2024-03-12T06:06:26.809 INFO:teuthology.orchestra.run.smithi088.stdout: "req_create_latency": { 2024-03-12T06:06:26.809 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 176, 2024-03-12T06:06:26.809 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.150422014, 2024-03-12T06:06:26.810 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000854670 2024-03-12T06:06:26.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.810 INFO:teuthology.orchestra.run.smithi088.stdout: "req_getattr_latency": { 2024-03-12T06:06:26.810 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 1, 2024-03-12T06:06:26.810 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.001212489, 2024-03-12T06:06:26.811 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.001212489 2024-03-12T06:06:26.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.811 INFO:teuthology.orchestra.run.smithi088.stdout: "req_getfilelock_latency": { 2024-03-12T06:06:26.811 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.811 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.812 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.812 INFO:teuthology.orchestra.run.smithi088.stdout: "req_link_latency": { 2024-03-12T06:06:26.812 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.812 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.813 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.813 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookup_latency": { 2024-03-12T06:06:26.813 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 359, 2024-03-12T06:06:26.813 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.114874402, 2024-03-12T06:06:26.814 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000319984 2024-03-12T06:06:26.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.814 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookuphash_latency": { 2024-03-12T06:06:26.814 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.814 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.815 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.815 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookupino_latency": { 2024-03-12T06:06:26.815 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.815 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.816 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.816 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookupname_latency": { 2024-03-12T06:06:26.816 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.816 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.816 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.817 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookupparent_latency": { 2024-03-12T06:06:26.817 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.817 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.817 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.818 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookupsnap_latency": { 2024-03-12T06:06:26.818 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.818 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.818 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.819 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lssnap_latency": { 2024-03-12T06:06:26.819 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.819 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.819 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.821 INFO:teuthology.orchestra.run.smithi088.stdout: "req_mkdir_latency": { 2024-03-12T06:06:26.821 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 1, 2024-03-12T06:06:26.821 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.002141319, 2024-03-12T06:06:26.821 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.002141319 2024-03-12T06:06:26.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.822 INFO:teuthology.orchestra.run.smithi088.stdout: "req_mknod_latency": { 2024-03-12T06:06:26.822 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.822 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.822 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.823 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.823 INFO:teuthology.orchestra.run.smithi088.stdout: "req_mksnap_latency": { 2024-03-12T06:06:26.823 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.823 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.823 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.824 INFO:teuthology.orchestra.run.smithi088.stdout: "req_open_latency": { 2024-03-12T06:06:26.824 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 3, 2024-03-12T06:06:26.824 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.004490781, 2024-03-12T06:06:26.824 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.001496927 2024-03-12T06:06:26.824 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.825 INFO:teuthology.orchestra.run.smithi088.stdout: "req_readdir_latency": { 2024-03-12T06:06:26.825 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 17, 2024-03-12T06:06:26.825 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.017451455, 2024-03-12T06:06:26.825 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.001026556 2024-03-12T06:06:26.825 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.825 INFO:teuthology.orchestra.run.smithi088.stdout: "req_rename_latency": { 2024-03-12T06:06:26.825 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.826 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.826 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.826 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.826 INFO:teuthology.orchestra.run.smithi088.stdout: "req_renamesnap_latency": { 2024-03-12T06:06:26.827 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.827 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.827 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.827 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.827 INFO:teuthology.orchestra.run.smithi088.stdout: "req_snapdiff_latency": { 2024-03-12T06:06:26.828 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.828 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.828 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.828 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.828 INFO:teuthology.orchestra.run.smithi088.stdout: "req_rmdir_latency": { 2024-03-12T06:06:26.829 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.829 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.829 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.829 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.829 INFO:teuthology.orchestra.run.smithi088.stdout: "req_rmsnap_latency": { 2024-03-12T06:06:26.830 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.830 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.830 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.830 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.830 INFO:teuthology.orchestra.run.smithi088.stdout: "req_rmxattr_latency": { 2024-03-12T06:06:26.831 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.831 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.831 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.831 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.831 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setattr_latency": { 2024-03-12T06:06:26.832 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.832 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.832 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.832 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.832 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setdirlayout_latency": { 2024-03-12T06:06:26.832 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.833 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.833 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.833 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.833 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setfilelock_latency": { 2024-03-12T06:06:26.833 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.833 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.834 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.834 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.834 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setlayout_latency": { 2024-03-12T06:06:26.834 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.834 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.835 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.835 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.835 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setxattr_latency": { 2024-03-12T06:06:26.835 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.835 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.836 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.836 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.836 INFO:teuthology.orchestra.run.smithi088.stdout: "req_symlink_latency": { 2024-03-12T06:06:26.836 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.836 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.837 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.837 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.837 INFO:teuthology.orchestra.run.smithi088.stdout: "req_unlink_latency": { 2024-03-12T06:06:26.837 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 150, 2024-03-12T06:06:26.837 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.334641406, 2024-03-12T06:06:26.837 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.002230942 2024-03-12T06:06:26.838 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.838 INFO:teuthology.orchestra.run.smithi088.stdout: "cap_revoke_eviction": 0, 2024-03-12T06:06:26.838 INFO:teuthology.orchestra.run.smithi088.stdout: "cap_acquisition_throttle": 0, 2024-03-12T06:06:26.838 INFO:teuthology.orchestra.run.smithi088.stdout: "req_getvxattr_latency": { 2024-03-12T06:06:26.838 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.839 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.839 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.839 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:26.839 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.839 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_sessions": { 2024-03-12T06:06:26.840 INFO:teuthology.orchestra.run.smithi088.stdout: "session_count": 1, 2024-03-12T06:06:26.840 INFO:teuthology.orchestra.run.smithi088.stdout: "session_add": 2, 2024-03-12T06:06:26.840 INFO:teuthology.orchestra.run.smithi088.stdout: "session_remove": 1, 2024-03-12T06:06:26.840 INFO:teuthology.orchestra.run.smithi088.stdout: "sessions_open": 1, 2024-03-12T06:06:26.840 INFO:teuthology.orchestra.run.smithi088.stdout: "sessions_stale": 0, 2024-03-12T06:06:26.841 INFO:teuthology.orchestra.run.smithi088.stdout: "total_load": 626, 2024-03-12T06:06:26.841 INFO:teuthology.orchestra.run.smithi088.stdout: "average_load": 626, 2024-03-12T06:06:26.841 INFO:teuthology.orchestra.run.smithi088.stdout: "avg_session_uptime": 653, 2024-03-12T06:06:26.841 INFO:teuthology.orchestra.run.smithi088.stdout: "mdthresh_evicted": 0 2024-03-12T06:06:26.841 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.841 INFO:teuthology.orchestra.run.smithi088.stdout: "mempool": { 2024-03-12T06:06:26.842 INFO:teuthology.orchestra.run.smithi088.stdout: "bloom_filter_bytes": 0, 2024-03-12T06:06:26.842 INFO:teuthology.orchestra.run.smithi088.stdout: "bloom_filter_items": 0, 2024-03-12T06:06:26.842 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T06:06:26.842 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_alloc_items": 0, 2024-03-12T06:06:26.842 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T06:06:26.842 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_data_items": 0, 2024-03-12T06:06:26.843 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T06:06:26.843 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T06:06:26.843 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T06:06:26.843 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T06:06:26.843 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T06:06:26.844 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_other_items": 0, 2024-03-12T06:06:26.844 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T06:06:26.844 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T06:06:26.844 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_extent_bytes": 0, 2024-03-12T06:06:26.844 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_extent_items": 0, 2024-03-12T06:06:26.845 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_blob_bytes": 0, 2024-03-12T06:06:26.845 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_blob_items": 0, 2024-03-12T06:06:26.845 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T06:06:26.845 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T06:06:26.845 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T06:06:26.846 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T06:06:26.846 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T06:06:26.846 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_items": 0, 2024-03-12T06:06:26.846 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_txc_bytes": 0, 2024-03-12T06:06:26.846 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_txc_items": 0, 2024-03-12T06:06:26.847 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T06:06:26.847 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T06:06:26.847 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_bytes": 0, 2024-03-12T06:06:26.847 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_items": 0, 2024-03-12T06:06:26.847 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_bytes": 0, 2024-03-12T06:06:26.847 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_items": 0, 2024-03-12T06:06:26.848 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T06:06:26.848 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_reader_items": 0, 2024-03-12T06:06:26.848 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T06:06:26.848 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_writer_items": 0, 2024-03-12T06:06:26.848 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_anon_bytes": 367569, 2024-03-12T06:06:26.849 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_anon_items": 1950, 2024-03-12T06:06:26.849 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_meta_bytes": 880, 2024-03-12T06:06:26.849 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_meta_items": 10, 2024-03-12T06:06:26.849 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_bytes": 0, 2024-03-12T06:06:26.849 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_items": 0, 2024-03-12T06:06:26.850 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mapbl_bytes": 0, 2024-03-12T06:06:26.850 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mapbl_items": 0, 2024-03-12T06:06:26.850 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pglog_bytes": 0, 2024-03-12T06:06:26.850 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pglog_items": 0, 2024-03-12T06:06:26.850 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_bytes": 8032, 2024-03-12T06:06:26.851 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_items": 167, 2024-03-12T06:06:26.851 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T06:06:26.851 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_mapping_items": 0, 2024-03-12T06:06:26.851 INFO:teuthology.orchestra.run.smithi088.stdout: "pgmap_bytes": 0, 2024-03-12T06:06:26.851 INFO:teuthology.orchestra.run.smithi088.stdout: "pgmap_items": 0, 2024-03-12T06:06:26.851 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_co_bytes": 6450641, 2024-03-12T06:06:26.851 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_co_items": 74342, 2024-03-12T06:06:26.852 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_1_bytes": 0, 2024-03-12T06:06:26.852 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_1_items": 0, 2024-03-12T06:06:26.852 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_2_bytes": 0, 2024-03-12T06:06:26.852 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_2_items": 0 2024-03-12T06:06:26.852 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.853 INFO:teuthology.orchestra.run.smithi088.stdout: "objecter": { 2024-03-12T06:06:26.853 INFO:teuthology.orchestra.run.smithi088.stdout: "op_active": 0, 2024-03-12T06:06:26.853 INFO:teuthology.orchestra.run.smithi088.stdout: "op_laggy": 0, 2024-03-12T06:06:26.853 INFO:teuthology.orchestra.run.smithi088.stdout: "op_send": 22204, 2024-03-12T06:06:26.853 INFO:teuthology.orchestra.run.smithi088.stdout: "op_send_bytes": 4087077, 2024-03-12T06:06:26.853 INFO:teuthology.orchestra.run.smithi088.stdout: "op_resend": 33, 2024-03-12T06:06:26.854 INFO:teuthology.orchestra.run.smithi088.stdout: "op_reply": 22204, 2024-03-12T06:06:26.854 INFO:teuthology.orchestra.run.smithi088.stdout: "op_latency": { 2024-03-12T06:06:26.854 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 22204, 2024-03-12T06:06:26.854 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 10.522804325, 2024-03-12T06:06:26.854 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000473914 2024-03-12T06:06:26.854 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.855 INFO:teuthology.orchestra.run.smithi088.stdout: "op_inflight": 0, 2024-03-12T06:06:26.855 INFO:teuthology.orchestra.run.smithi088.stdout: "oplen_avg": { 2024-03-12T06:06:26.855 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 22204, 2024-03-12T06:06:26.855 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 22250 2024-03-12T06:06:26.855 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.856 INFO:teuthology.orchestra.run.smithi088.stdout: "op": 22204, 2024-03-12T06:06:26.856 INFO:teuthology.orchestra.run.smithi088.stdout: "op_r": 21860, 2024-03-12T06:06:26.856 INFO:teuthology.orchestra.run.smithi088.stdout: "op_w": 344, 2024-03-12T06:06:26.856 INFO:teuthology.orchestra.run.smithi088.stdout: "op_rmw": 0, 2024-03-12T06:06:26.856 INFO:teuthology.orchestra.run.smithi088.stdout: "op_pg": 0, 2024-03-12T06:06:26.856 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_stat": 0, 2024-03-12T06:06:26.857 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_create": 10, 2024-03-12T06:06:26.857 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_read": 5, 2024-03-12T06:06:26.857 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_write": 121, 2024-03-12T06:06:26.857 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_writefull": 17, 2024-03-12T06:06:26.857 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_writesame": 0, 2024-03-12T06:06:26.858 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_append": 0, 2024-03-12T06:06:26.858 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_zero": 0, 2024-03-12T06:06:26.858 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_truncate": 0, 2024-03-12T06:06:26.858 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_delete": 162, 2024-03-12T06:06:26.858 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_mapext": 0, 2024-03-12T06:06:26.859 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_sparse_read": 0, 2024-03-12T06:06:26.859 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_clonerange": 0, 2024-03-12T06:06:26.859 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_getxattr": 21855, 2024-03-12T06:06:26.859 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_setxattr": 20, 2024-03-12T06:06:26.859 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_cmpxattr": 0, 2024-03-12T06:06:26.860 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_rmxattr": 0, 2024-03-12T06:06:26.860 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_resetxattrs": 0, 2024-03-12T06:06:26.860 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_call": 0, 2024-03-12T06:06:26.860 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_watch": 0, 2024-03-12T06:06:26.860 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_notify": 0, 2024-03-12T06:06:26.861 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T06:06:26.861 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_pgls": 0, 2024-03-12T06:06:26.861 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_pgls_filter": 0, 2024-03-12T06:06:26.861 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_other": 0, 2024-03-12T06:06:26.861 INFO:teuthology.orchestra.run.smithi088.stdout: "linger_active": 0, 2024-03-12T06:06:26.862 INFO:teuthology.orchestra.run.smithi088.stdout: "linger_send": 0, 2024-03-12T06:06:26.862 INFO:teuthology.orchestra.run.smithi088.stdout: "linger_resend": 0, 2024-03-12T06:06:26.862 INFO:teuthology.orchestra.run.smithi088.stdout: "linger_ping": 0, 2024-03-12T06:06:26.862 INFO:teuthology.orchestra.run.smithi088.stdout: "poolop_active": 0, 2024-03-12T06:06:26.862 INFO:teuthology.orchestra.run.smithi088.stdout: "poolop_send": 0, 2024-03-12T06:06:26.862 INFO:teuthology.orchestra.run.smithi088.stdout: "poolop_resend": 0, 2024-03-12T06:06:26.863 INFO:teuthology.orchestra.run.smithi088.stdout: "poolstat_active": 0, 2024-03-12T06:06:26.863 INFO:teuthology.orchestra.run.smithi088.stdout: "poolstat_send": 0, 2024-03-12T06:06:26.863 INFO:teuthology.orchestra.run.smithi088.stdout: "poolstat_resend": 0, 2024-03-12T06:06:26.863 INFO:teuthology.orchestra.run.smithi088.stdout: "statfs_active": 0, 2024-03-12T06:06:26.863 INFO:teuthology.orchestra.run.smithi088.stdout: "statfs_send": 0, 2024-03-12T06:06:26.863 INFO:teuthology.orchestra.run.smithi088.stdout: "statfs_resend": 0, 2024-03-12T06:06:26.864 INFO:teuthology.orchestra.run.smithi088.stdout: "command_active": 0, 2024-03-12T06:06:26.864 INFO:teuthology.orchestra.run.smithi088.stdout: "command_send": 0, 2024-03-12T06:06:26.864 INFO:teuthology.orchestra.run.smithi088.stdout: "command_resend": 0, 2024-03-12T06:06:26.864 INFO:teuthology.orchestra.run.smithi088.stdout: "map_epoch": 97, 2024-03-12T06:06:26.864 INFO:teuthology.orchestra.run.smithi088.stdout: "map_full": 0, 2024-03-12T06:06:26.865 INFO:teuthology.orchestra.run.smithi088.stdout: "map_inc": 11, 2024-03-12T06:06:26.865 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_sessions": 12, 2024-03-12T06:06:26.865 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_session_open": 12, 2024-03-12T06:06:26.865 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_session_close": 0, 2024-03-12T06:06:26.865 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_laggy": 0, 2024-03-12T06:06:26.865 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_wr": 49, 2024-03-12T06:06:26.866 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_rd": 0, 2024-03-12T06:06:26.866 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_del": 11 2024-03-12T06:06:26.866 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.866 INFO:teuthology.orchestra.run.smithi088.stdout: "oft": { 2024-03-12T06:06:26.866 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_total_objs": 1, 2024-03-12T06:06:26.866 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_total_kv_pairs": 0, 2024-03-12T06:06:26.867 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_total_updates": 0, 2024-03-12T06:06:26.867 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_total_removes": 0 2024-03-12T06:06:26.867 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.867 INFO:teuthology.orchestra.run.smithi088.stdout: "osd-slow-ops": { 2024-03-12T06:06:26.867 INFO:teuthology.orchestra.run.smithi088.stdout: "slow_ops_count": 0 2024-03-12T06:06:26.867 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.868 INFO:teuthology.orchestra.run.smithi088.stdout: "purge_queue": { 2024-03-12T06:06:26.868 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executing_ops": 0, 2024-03-12T06:06:26.868 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executing_ops_high_water": 10, 2024-03-12T06:06:26.868 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executing": 0, 2024-03-12T06:06:26.868 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executing_high_water": 10, 2024-03-12T06:06:26.868 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executed_ops": 150, 2024-03-12T06:06:26.868 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executed": 150, 2024-03-12T06:06:26.869 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_item_in_journal": 0 2024-03-12T06:06:26.869 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.869 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T06:06:26.869 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:26.869 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 104857600, 2024-03-12T06:06:26.869 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:26.870 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 26564, 2024-03-12T06:06:26.870 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 8542269, 2024-03-12T06:06:26.870 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:26.870 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 26564, 2024-03-12T06:06:26.871 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 0, 2024-03-12T06:06:26.871 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 0, 2024-03-12T06:06:26.871 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 26564, 2024-03-12T06:06:26.871 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 8542269, 2024-03-12T06:06:26.871 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:26.871 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.872 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.872 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.872 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:26.872 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.872 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-objecter_bytes": { 2024-03-12T06:06:26.872 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:26.872 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 104857600, 2024-03-12T06:06:26.873 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:26.873 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 0, 2024-03-12T06:06:26.873 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 0, 2024-03-12T06:06:26.873 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:26.873 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:26.873 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 22204, 2024-03-12T06:06:26.873 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 4102227, 2024-03-12T06:06:26.874 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 22204, 2024-03-12T06:06:26.874 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 4102227, 2024-03-12T06:06:26.874 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:26.874 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.874 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.875 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.875 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:26.875 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.875 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-objecter_ops": { 2024-03-12T06:06:26.875 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:26.875 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 1024, 2024-03-12T06:06:26.876 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:26.876 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 0, 2024-03-12T06:06:26.876 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 0, 2024-03-12T06:06:26.876 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:26.876 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:26.876 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 22204, 2024-03-12T06:06:26.877 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 22204, 2024-03-12T06:06:26.877 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 22204, 2024-03-12T06:06:26.877 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 22204, 2024-03-12T06:06:26.877 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:26.877 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.878 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.878 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.878 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:26.878 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.878 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-write_buf_throttle": { 2024-03-12T06:06:26.878 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:26.879 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 3758096384, 2024-03-12T06:06:26.879 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:26.879 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 150, 2024-03-12T06:06:26.879 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 15150, 2024-03-12T06:06:26.879 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:26.879 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 150, 2024-03-12T06:06:26.880 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 0, 2024-03-12T06:06:26.880 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 0, 2024-03-12T06:06:26.880 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 5, 2024-03-12T06:06:26.880 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 15150, 2024-03-12T06:06:26.880 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:26.880 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.881 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.881 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.881 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:26.881 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:26.881 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-write_buf_throttle-0x55bcedf26640": { 2024-03-12T06:06:26.881 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:26.882 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 3758096384, 2024-03-12T06:06:26.882 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:26.882 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 1783, 2024-03-12T06:06:26.882 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 3863195, 2024-03-12T06:06:26.882 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:26.882 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 1783, 2024-03-12T06:06:26.882 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 0, 2024-03-12T06:06:26.883 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 0, 2024-03-12T06:06:26.883 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 116, 2024-03-12T06:06:26.883 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 3863195, 2024-03-12T06:06:26.883 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:26.883 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:26.884 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:26.884 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:26.884 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:26.884 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:26.884 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-03-12T06:06:27.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:27 smithi088 ceph-mon[36585]: pgmap v331: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-12T06:06:27.587 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-12T06:06:27.587 INFO:tasks.check_counter:Daemon mds.f mds.imported=69 2024-03-12T06:06:27.587 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=7 2024-03-12T06:06:27.588 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-12T06:06:27.588 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-12T06:06:27.588 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-12T06:06:27.588 DEBUG:tasks.check_counter:Getting stats from i 2024-03-12T06:06:27.588 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-12T06:06:27.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:27 smithi082 ceph-mon[36382]: pgmap v331: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-12T06:06:27.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:27 smithi076 ceph-mon[32095]: pgmap v331: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-12T06:06:27.826 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T06:06:29.472 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:29 smithi088 ceph-mon[36585]: pgmap v332: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:06:29.507 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-03-12T06:06:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T06:06:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_messages": 10760, 2024-03-12T06:06:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages": 17519, 2024-03-12T06:06:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_bytes": 6441807, 2024-03-12T06:06:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_bytes": 13673534, 2024-03-12T06:06:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_created_connections": 9, 2024-03-12T06:06:29.508 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_active_connections": 5, 2024-03-12T06:06:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_total_time": 0.811387002, 2024-03-12T06:06:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_send_time": 0.334333932, 2024-03-12T06:06:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_recv_time": 0.306559606, 2024-03-12T06:06:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_fast_dispatch_time": 0.162978138, 2024-03-12T06:06:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 17518, 2024-03-12T06:06:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.480860488, 2024-03-12T06:06:29.509 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000027449 2024-03-12T06:06:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 11885, 2024-03-12T06:06:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.002136038, 2024-03-12T06:06:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000179 2024-03-12T06:06:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.510 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_encrypted_bytes": 1634608, 2024-03-12T06:06:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_encrypted_bytes": 154288 2024-03-12T06:06:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T06:06:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_messages": 11112, 2024-03-12T06:06:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages": 11272, 2024-03-12T06:06:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_bytes": 2949157, 2024-03-12T06:06:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_bytes": 4695472, 2024-03-12T06:06:29.511 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_created_connections": 49, 2024-03-12T06:06:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_active_connections": 7, 2024-03-12T06:06:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_total_time": 0.801845386, 2024-03-12T06:06:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_send_time": 0.336840062, 2024-03-12T06:06:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_recv_time": 0.179721868, 2024-03-12T06:06:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_fast_dispatch_time": 0.235081501, 2024-03-12T06:06:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 11269, 2024-03-12T06:06:29.512 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.549641008, 2024-03-12T06:06:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000048774 2024-03-12T06:06:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 677, 2024-03-12T06:06:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000124047, 2024-03-12T06:06:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000183 2024-03-12T06:06:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.513 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_encrypted_bytes": 7584, 2024-03-12T06:06:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_encrypted_bytes": 146992 2024-03-12T06:06:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T06:06:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_messages": 10339, 2024-03-12T06:06:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages": 10468, 2024-03-12T06:06:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_bytes": 2659964, 2024-03-12T06:06:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_bytes": 3869299, 2024-03-12T06:06:29.514 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_created_connections": 17, 2024-03-12T06:06:29.515 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_active_connections": 5, 2024-03-12T06:06:29.515 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_total_time": 0.644728126, 2024-03-12T06:06:29.515 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_send_time": 0.218591431, 2024-03-12T06:06:29.515 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_recv_time": 0.153255106, 2024-03-12T06:06:29.515 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_fast_dispatch_time": 0.222324796, 2024-03-12T06:06:29.515 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:29.515 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 10467, 2024-03-12T06:06:29.515 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.478908249, 2024-03-12T06:06:29.515 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000045754 2024-03-12T06:06:29.516 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.516 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:29.516 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 8, 2024-03-12T06:06:29.516 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000775, 2024-03-12T06:06:29.516 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000096 2024-03-12T06:06:29.516 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.516 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-12T06:06:29.516 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_encrypted_bytes": 317696 2024-03-12T06:06:29.517 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.517 INFO:teuthology.orchestra.run.smithi088.stdout: "cct": { 2024-03-12T06:06:29.517 INFO:teuthology.orchestra.run.smithi088.stdout: "total_workers": 1, 2024-03-12T06:06:29.517 INFO:teuthology.orchestra.run.smithi088.stdout: "unhealthy_workers": 0 2024-03-12T06:06:29.517 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.517 INFO:teuthology.orchestra.run.smithi088.stdout: "finisher-MDSRank": { 2024-03-12T06:06:29.517 INFO:teuthology.orchestra.run.smithi088.stdout: "queue_len": 0, 2024-03-12T06:06:29.517 INFO:teuthology.orchestra.run.smithi088.stdout: "complete_latency": { 2024-03-12T06:06:29.517 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 18299, 2024-03-12T06:06:29.518 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 4.415421335, 2024-03-12T06:06:29.518 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000241293 2024-03-12T06:06:29.518 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:29.518 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.518 INFO:teuthology.orchestra.run.smithi088.stdout: "finisher-PurgeQueue": { 2024-03-12T06:06:29.518 INFO:teuthology.orchestra.run.smithi088.stdout: "queue_len": 0, 2024-03-12T06:06:29.518 INFO:teuthology.orchestra.run.smithi088.stdout: "complete_latency": { 2024-03-12T06:06:29.519 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 518, 2024-03-12T06:06:29.519 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.036458158, 2024-03-12T06:06:29.519 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000070382 2024-03-12T06:06:29.519 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:29.519 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.519 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": { 2024-03-12T06:06:29.519 INFO:teuthology.orchestra.run.smithi088.stdout: "request": 974, 2024-03-12T06:06:29.519 INFO:teuthology.orchestra.run.smithi088.stdout: "reply": 963, 2024-03-12T06:06:29.519 INFO:teuthology.orchestra.run.smithi088.stdout: "reply_latency": { 2024-03-12T06:06:29.520 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 963, 2024-03-12T06:06:29.520 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.960598109, 2024-03-12T06:06:29.520 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000997505 2024-03-12T06:06:29.520 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.520 INFO:teuthology.orchestra.run.smithi088.stdout: "slow_reply": 0, 2024-03-12T06:06:29.520 INFO:teuthology.orchestra.run.smithi088.stdout: "forward": 11, 2024-03-12T06:06:29.520 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_fetch_complete": 0, 2024-03-12T06:06:29.520 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_fetch_keys": 0, 2024-03-12T06:06:29.521 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_commit": 11, 2024-03-12T06:06:29.521 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_split": 0, 2024-03-12T06:06:29.521 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_merge": 0, 2024-03-12T06:06:29.521 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes": 2624, 2024-03-12T06:06:29.521 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_top": 0, 2024-03-12T06:06:29.522 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_bottom": 0, 2024-03-12T06:06:29.522 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_pin_tail": 2624, 2024-03-12T06:06:29.522 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_pinned": 2624, 2024-03-12T06:06:29.522 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_expired": 491, 2024-03-12T06:06:29.522 INFO:teuthology.orchestra.run.smithi088.stdout: "inodes_with_caps": 45, 2024-03-12T06:06:29.522 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": 46, 2024-03-12T06:06:29.522 INFO:teuthology.orchestra.run.smithi088.stdout: "subtrees": 68, 2024-03-12T06:06:29.523 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse": 1531, 2024-03-12T06:06:29.523 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_hit": 1279, 2024-03-12T06:06:29.523 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_forward": 5, 2024-03-12T06:06:29.523 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_discover": 17, 2024-03-12T06:06:29.523 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_dir_fetch": 0, 2024-03-12T06:06:29.523 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_remote_ino": 0, 2024-03-12T06:06:29.523 INFO:teuthology.orchestra.run.smithi088.stdout: "traverse_lock": 0, 2024-03-12T06:06:29.523 INFO:teuthology.orchestra.run.smithi088.stdout: "load_cent": 0, 2024-03-12T06:06:29.524 INFO:teuthology.orchestra.run.smithi088.stdout: "q": 0, 2024-03-12T06:06:29.524 INFO:teuthology.orchestra.run.smithi088.stdout: "exported": 15, 2024-03-12T06:06:29.524 INFO:teuthology.orchestra.run.smithi088.stdout: "exported_inodes": 485, 2024-03-12T06:06:29.524 INFO:teuthology.orchestra.run.smithi088.stdout: "imported": 29, 2024-03-12T06:06:29.524 INFO:teuthology.orchestra.run.smithi088.stdout: "imported_inodes": 2748, 2024-03-12T06:06:29.524 INFO:teuthology.orchestra.run.smithi088.stdout: "openino_dir_fetch": 0, 2024-03-12T06:06:29.524 INFO:teuthology.orchestra.run.smithi088.stdout: "openino_backtrace_fetch": 27187, 2024-03-12T06:06:29.524 INFO:teuthology.orchestra.run.smithi088.stdout: "openino_peer_discover": 1, 2024-03-12T06:06:29.525 INFO:teuthology.orchestra.run.smithi088.stdout: "root_rfiles": 3452, 2024-03-12T06:06:29.525 INFO:teuthology.orchestra.run.smithi088.stdout: "root_rbytes": 26524023, 2024-03-12T06:06:29.525 INFO:teuthology.orchestra.run.smithi088.stdout: "root_rsnaps": 7, 2024-03-12T06:06:29.525 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_backtrace_fetch": 27187, 2024-03-12T06:06:29.525 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_set_tag": 0, 2024-03-12T06:06:29.525 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_backtrace_repaired": 0, 2024-03-12T06:06:29.525 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_inotable_repaired": 0, 2024-03-12T06:06:29.525 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_dir_inodes": 180, 2024-03-12T06:06:29.526 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_dir_base_inodes": 0, 2024-03-12T06:06:29.526 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_dirfrag_rstats": 180, 2024-03-12T06:06:29.526 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_file_inodes": 27007, 2024-03-12T06:06:29.526 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_inode_file_caps": 0, 2024-03-12T06:06:29.526 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_revoke": 59, 2024-03-12T06:06:29.526 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_grant": 637, 2024-03-12T06:06:29.526 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_trunc": 0, 2024-03-12T06:06:29.526 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-12T06:06:29.527 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-12T06:06:29.527 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_caps": 563, 2024-03-12T06:06:29.527 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_caps_dirty": 502, 2024-03-12T06:06:29.527 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_cap_release": 21, 2024-03-12T06:06:29.527 INFO:teuthology.orchestra.run.smithi088.stdout: "process_request_cap_release": 636 2024-03-12T06:06:29.527 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.527 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_cache": { 2024-03-12T06:06:29.527 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_update": 0, 2024-03-12T06:06:29.527 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_update_receipt": 0, 2024-03-12T06:06:29.528 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_try_discover": 0, 2024-03-12T06:06:29.528 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_send_discover": 117, 2024-03-12T06:06:29.528 INFO:teuthology.orchestra.run.smithi088.stdout: "dir_handle_discover": 0, 2024-03-12T06:06:29.528 INFO:teuthology.orchestra.run.smithi088.stdout: "num_strays": 0, 2024-03-12T06:06:29.528 INFO:teuthology.orchestra.run.smithi088.stdout: "num_strays_delayed": 0, 2024-03-12T06:06:29.528 INFO:teuthology.orchestra.run.smithi088.stdout: "num_strays_enqueuing": 0, 2024-03-12T06:06:29.528 INFO:teuthology.orchestra.run.smithi088.stdout: "strays_created": 316, 2024-03-12T06:06:29.528 INFO:teuthology.orchestra.run.smithi088.stdout: "strays_enqueued": 316, 2024-03-12T06:06:29.529 INFO:teuthology.orchestra.run.smithi088.stdout: "strays_reintegrated": 0, 2024-03-12T06:06:29.529 INFO:teuthology.orchestra.run.smithi088.stdout: "strays_migrated": 0, 2024-03-12T06:06:29.529 INFO:teuthology.orchestra.run.smithi088.stdout: "num_recovering_processing": 0, 2024-03-12T06:06:29.529 INFO:teuthology.orchestra.run.smithi088.stdout: "num_recovering_enqueued": 0, 2024-03-12T06:06:29.529 INFO:teuthology.orchestra.run.smithi088.stdout: "num_recovering_prioritized": 0, 2024-03-12T06:06:29.529 INFO:teuthology.orchestra.run.smithi088.stdout: "recovery_started": 0, 2024-03-12T06:06:29.529 INFO:teuthology.orchestra.run.smithi088.stdout: "recovery_completed": 0, 2024-03-12T06:06:29.529 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_enqueue_scrub": 0, 2024-03-12T06:06:29.529 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_exportdir": 15, 2024-03-12T06:06:29.530 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_flush": 0, 2024-03-12T06:06:29.530 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_fragmentdir": 0, 2024-03-12T06:06:29.530 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_fragstats": 0, 2024-03-12T06:06:29.530 INFO:teuthology.orchestra.run.smithi088.stdout: "ireq_inodestats": 0 2024-03-12T06:06:29.530 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.530 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_log": { 2024-03-12T06:06:29.530 INFO:teuthology.orchestra.run.smithi088.stdout: "evlrg": 1, 2024-03-12T06:06:29.531 INFO:teuthology.orchestra.run.smithi088.stdout: "evadd": 2330, 2024-03-12T06:06:29.531 INFO:teuthology.orchestra.run.smithi088.stdout: "evex": 0, 2024-03-12T06:06:29.531 INFO:teuthology.orchestra.run.smithi088.stdout: "evtrm": 0, 2024-03-12T06:06:29.531 INFO:teuthology.orchestra.run.smithi088.stdout: "ev": 2330, 2024-03-12T06:06:29.531 INFO:teuthology.orchestra.run.smithi088.stdout: "evexg": 0, 2024-03-12T06:06:29.531 INFO:teuthology.orchestra.run.smithi088.stdout: "evexd": 0, 2024-03-12T06:06:29.531 INFO:teuthology.orchestra.run.smithi088.stdout: "segadd": 4, 2024-03-12T06:06:29.531 INFO:teuthology.orchestra.run.smithi088.stdout: "segex": 0, 2024-03-12T06:06:29.532 INFO:teuthology.orchestra.run.smithi088.stdout: "segtrm": 0, 2024-03-12T06:06:29.532 INFO:teuthology.orchestra.run.smithi088.stdout: "seg": 4, 2024-03-12T06:06:29.532 INFO:teuthology.orchestra.run.smithi088.stdout: "segmjr": 2, 2024-03-12T06:06:29.532 INFO:teuthology.orchestra.run.smithi088.stdout: "segexg": 0, 2024-03-12T06:06:29.532 INFO:teuthology.orchestra.run.smithi088.stdout: "segexd": 0, 2024-03-12T06:06:29.532 INFO:teuthology.orchestra.run.smithi088.stdout: "expos": 4194304, 2024-03-12T06:06:29.532 INFO:teuthology.orchestra.run.smithi088.stdout: "wrpos": 9898980, 2024-03-12T06:06:29.532 INFO:teuthology.orchestra.run.smithi088.stdout: "rdpos": 0, 2024-03-12T06:06:29.533 INFO:teuthology.orchestra.run.smithi088.stdout: "jlat": { 2024-03-12T06:06:29.533 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 66, 2024-03-12T06:06:29.533 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.669837386, 2024-03-12T06:06:29.533 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.010149051 2024-03-12T06:06:29.533 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.533 INFO:teuthology.orchestra.run.smithi088.stdout: "replayed": 0 2024-03-12T06:06:29.533 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.533 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_mem": { 2024-03-12T06:06:29.534 INFO:teuthology.orchestra.run.smithi088.stdout: "ino": 1977, 2024-03-12T06:06:29.534 INFO:teuthology.orchestra.run.smithi088.stdout: "ino+": 2690, 2024-03-12T06:06:29.534 INFO:teuthology.orchestra.run.smithi088.stdout: "ino-": 713, 2024-03-12T06:06:29.534 INFO:teuthology.orchestra.run.smithi088.stdout: "dir": 179, 2024-03-12T06:06:29.534 INFO:teuthology.orchestra.run.smithi088.stdout: "dir+": 180, 2024-03-12T06:06:29.534 INFO:teuthology.orchestra.run.smithi088.stdout: "dir-": 1, 2024-03-12T06:06:29.534 INFO:teuthology.orchestra.run.smithi088.stdout: "dn": 2624, 2024-03-12T06:06:29.535 INFO:teuthology.orchestra.run.smithi088.stdout: "dn+": 3431, 2024-03-12T06:06:29.535 INFO:teuthology.orchestra.run.smithi088.stdout: "dn-": 807, 2024-03-12T06:06:29.535 INFO:teuthology.orchestra.run.smithi088.stdout: "cap": 46, 2024-03-12T06:06:29.535 INFO:teuthology.orchestra.run.smithi088.stdout: "cap+": 2329, 2024-03-12T06:06:29.535 INFO:teuthology.orchestra.run.smithi088.stdout: "cap-": 2283, 2024-03-12T06:06:29.535 INFO:teuthology.orchestra.run.smithi088.stdout: "rss": 69268, 2024-03-12T06:06:29.535 INFO:teuthology.orchestra.run.smithi088.stdout: "heap": 207228 2024-03-12T06:06:29.535 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.535 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_server": { 2024-03-12T06:06:29.536 INFO:teuthology.orchestra.run.smithi088.stdout: "dispatch_client_request": 1624, 2024-03-12T06:06:29.536 INFO:teuthology.orchestra.run.smithi088.stdout: "dispatch_server_request": 0, 2024-03-12T06:06:29.536 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_request": 974, 2024-03-12T06:06:29.536 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_client_session": 26, 2024-03-12T06:06:29.536 INFO:teuthology.orchestra.run.smithi088.stdout: "handle_peer_request": 1, 2024-03-12T06:06:29.536 INFO:teuthology.orchestra.run.smithi088.stdout: "req_create_latency": { 2024-03-12T06:06:29.536 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 319, 2024-03-12T06:06:29.536 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.189684546, 2024-03-12T06:06:29.536 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000594622 2024-03-12T06:06:29.537 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.537 INFO:teuthology.orchestra.run.smithi088.stdout: "req_getattr_latency": { 2024-03-12T06:06:29.537 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 5, 2024-03-12T06:06:29.537 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.001047696, 2024-03-12T06:06:29.537 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000209539 2024-03-12T06:06:29.537 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.537 INFO:teuthology.orchestra.run.smithi088.stdout: "req_getfilelock_latency": { 2024-03-12T06:06:29.538 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.538 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.538 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.538 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.538 INFO:teuthology.orchestra.run.smithi088.stdout: "req_link_latency": { 2024-03-12T06:06:29.538 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.538 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.539 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.539 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookup_latency": { 2024-03-12T06:06:29.539 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 322, 2024-03-12T06:06:29.539 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.101497506, 2024-03-12T06:06:29.539 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000315209 2024-03-12T06:06:29.539 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.539 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookuphash_latency": { 2024-03-12T06:06:29.540 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.540 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.540 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.540 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookupino_latency": { 2024-03-12T06:06:29.540 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 1, 2024-03-12T06:06:29.540 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000083950, 2024-03-12T06:06:29.540 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000083950 2024-03-12T06:06:29.540 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.541 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookupname_latency": { 2024-03-12T06:06:29.541 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.541 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.541 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.541 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.541 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookupparent_latency": { 2024-03-12T06:06:29.541 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.541 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.542 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.542 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lookupsnap_latency": { 2024-03-12T06:06:29.542 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.542 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.542 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.542 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.543 INFO:teuthology.orchestra.run.smithi088.stdout: "req_lssnap_latency": { 2024-03-12T06:06:29.543 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.543 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.544 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.544 INFO:teuthology.orchestra.run.smithi088.stdout: "req_mkdir_latency": { 2024-03-12T06:06:29.544 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.544 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.544 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.544 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.544 INFO:teuthology.orchestra.run.smithi088.stdout: "req_mknod_latency": { 2024-03-12T06:06:29.545 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.545 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.545 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.545 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.545 INFO:teuthology.orchestra.run.smithi088.stdout: "req_mksnap_latency": { 2024-03-12T06:06:29.545 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.545 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.546 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.546 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.546 INFO:teuthology.orchestra.run.smithi088.stdout: "req_open_latency": { 2024-03-12T06:06:29.546 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.546 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.546 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.547 INFO:teuthology.orchestra.run.smithi088.stdout: "req_readdir_latency": { 2024-03-12T06:06:29.547 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.547 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.547 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.547 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.547 INFO:teuthology.orchestra.run.smithi088.stdout: "req_rename_latency": { 2024-03-12T06:06:29.547 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.548 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.548 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.548 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.548 INFO:teuthology.orchestra.run.smithi088.stdout: "req_renamesnap_latency": { 2024-03-12T06:06:29.548 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.549 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.549 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.549 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.549 INFO:teuthology.orchestra.run.smithi088.stdout: "req_snapdiff_latency": { 2024-03-12T06:06:29.550 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.550 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.550 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.550 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.551 INFO:teuthology.orchestra.run.smithi088.stdout: "req_rmdir_latency": { 2024-03-12T06:06:29.551 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.551 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.551 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.551 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.551 INFO:teuthology.orchestra.run.smithi088.stdout: "req_rmsnap_latency": { 2024-03-12T06:06:29.552 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.552 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.552 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.552 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.553 INFO:teuthology.orchestra.run.smithi088.stdout: "req_rmxattr_latency": { 2024-03-12T06:06:29.553 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.553 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.553 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.553 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.553 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setattr_latency": { 2024-03-12T06:06:29.553 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.554 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.554 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.554 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setdirlayout_latency": { 2024-03-12T06:06:29.554 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.554 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.554 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.554 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.555 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setfilelock_latency": { 2024-03-12T06:06:29.555 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.555 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.555 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.555 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.555 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setlayout_latency": { 2024-03-12T06:06:29.555 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.555 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.556 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.556 INFO:teuthology.orchestra.run.smithi088.stdout: "req_setxattr_latency": { 2024-03-12T06:06:29.556 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.556 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.556 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.556 INFO:teuthology.orchestra.run.smithi088.stdout: "req_symlink_latency": { 2024-03-12T06:06:29.556 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.557 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.557 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.557 INFO:teuthology.orchestra.run.smithi088.stdout: "req_unlink_latency": { 2024-03-12T06:06:29.557 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 316, 2024-03-12T06:06:29.557 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.668284411, 2024-03-12T06:06:29.557 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.002114824 2024-03-12T06:06:29.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.557 INFO:teuthology.orchestra.run.smithi088.stdout: "cap_revoke_eviction": 0, 2024-03-12T06:06:29.558 INFO:teuthology.orchestra.run.smithi088.stdout: "cap_acquisition_throttle": 0, 2024-03-12T06:06:29.558 INFO:teuthology.orchestra.run.smithi088.stdout: "req_getvxattr_latency": { 2024-03-12T06:06:29.558 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.558 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.558 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.558 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:29.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.558 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_sessions": { 2024-03-12T06:06:29.559 INFO:teuthology.orchestra.run.smithi088.stdout: "session_count": 2, 2024-03-12T06:06:29.559 INFO:teuthology.orchestra.run.smithi088.stdout: "session_add": 4, 2024-03-12T06:06:29.559 INFO:teuthology.orchestra.run.smithi088.stdout: "session_remove": 2, 2024-03-12T06:06:29.559 INFO:teuthology.orchestra.run.smithi088.stdout: "sessions_open": 2, 2024-03-12T06:06:29.559 INFO:teuthology.orchestra.run.smithi088.stdout: "sessions_stale": 0, 2024-03-12T06:06:29.559 INFO:teuthology.orchestra.run.smithi088.stdout: "total_load": 904, 2024-03-12T06:06:29.559 INFO:teuthology.orchestra.run.smithi088.stdout: "average_load": 452, 2024-03-12T06:06:29.559 INFO:teuthology.orchestra.run.smithi088.stdout: "avg_session_uptime": 633, 2024-03-12T06:06:29.560 INFO:teuthology.orchestra.run.smithi088.stdout: "mdthresh_evicted": 0 2024-03-12T06:06:29.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.560 INFO:teuthology.orchestra.run.smithi088.stdout: "mempool": { 2024-03-12T06:06:29.560 INFO:teuthology.orchestra.run.smithi088.stdout: "bloom_filter_bytes": 0, 2024-03-12T06:06:29.560 INFO:teuthology.orchestra.run.smithi088.stdout: "bloom_filter_items": 0, 2024-03-12T06:06:29.560 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T06:06:29.560 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_alloc_items": 0, 2024-03-12T06:06:29.560 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T06:06:29.561 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_data_items": 0, 2024-03-12T06:06:29.561 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T06:06:29.561 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T06:06:29.561 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T06:06:29.561 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T06:06:29.561 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T06:06:29.561 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_other_items": 0, 2024-03-12T06:06:29.561 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T06:06:29.562 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T06:06:29.562 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_extent_bytes": 0, 2024-03-12T06:06:29.562 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_extent_items": 0, 2024-03-12T06:06:29.562 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_blob_bytes": 0, 2024-03-12T06:06:29.562 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_blob_items": 0, 2024-03-12T06:06:29.562 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T06:06:29.562 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T06:06:29.562 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T06:06:29.563 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T06:06:29.563 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T06:06:29.563 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_items": 0, 2024-03-12T06:06:29.563 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_txc_bytes": 0, 2024-03-12T06:06:29.563 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_txc_items": 0, 2024-03-12T06:06:29.563 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T06:06:29.563 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T06:06:29.563 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_bytes": 0, 2024-03-12T06:06:29.564 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_items": 0, 2024-03-12T06:06:29.564 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_bytes": 0, 2024-03-12T06:06:29.564 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_items": 0, 2024-03-12T06:06:29.564 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T06:06:29.564 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_reader_items": 0, 2024-03-12T06:06:29.564 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T06:06:29.564 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_writer_items": 0, 2024-03-12T06:06:29.564 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_anon_bytes": 373438, 2024-03-12T06:06:29.564 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_anon_items": 1886, 2024-03-12T06:06:29.565 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_meta_bytes": 880, 2024-03-12T06:06:29.565 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_meta_items": 10, 2024-03-12T06:06:29.565 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_bytes": 0, 2024-03-12T06:06:29.565 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_items": 0, 2024-03-12T06:06:29.565 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mapbl_bytes": 0, 2024-03-12T06:06:29.565 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mapbl_items": 0, 2024-03-12T06:06:29.565 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pglog_bytes": 0, 2024-03-12T06:06:29.565 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pglog_items": 0, 2024-03-12T06:06:29.566 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_bytes": 8032, 2024-03-12T06:06:29.566 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_items": 167, 2024-03-12T06:06:29.566 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T06:06:29.566 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_mapping_items": 0, 2024-03-12T06:06:29.566 INFO:teuthology.orchestra.run.smithi088.stdout: "pgmap_bytes": 0, 2024-03-12T06:06:29.566 INFO:teuthology.orchestra.run.smithi088.stdout: "pgmap_items": 0, 2024-03-12T06:06:29.566 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_co_bytes": 6671645, 2024-03-12T06:06:29.566 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_co_items": 71335, 2024-03-12T06:06:29.566 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_1_bytes": 0, 2024-03-12T06:06:29.567 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_1_items": 0, 2024-03-12T06:06:29.567 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_2_bytes": 0, 2024-03-12T06:06:29.567 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_2_items": 0 2024-03-12T06:06:29.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.567 INFO:teuthology.orchestra.run.smithi088.stdout: "objecter": { 2024-03-12T06:06:29.567 INFO:teuthology.orchestra.run.smithi088.stdout: "op_active": 0, 2024-03-12T06:06:29.567 INFO:teuthology.orchestra.run.smithi088.stdout: "op_laggy": 0, 2024-03-12T06:06:29.567 INFO:teuthology.orchestra.run.smithi088.stdout: "op_send": 27659, 2024-03-12T06:06:29.567 INFO:teuthology.orchestra.run.smithi088.stdout: "op_send_bytes": 5989725, 2024-03-12T06:06:29.568 INFO:teuthology.orchestra.run.smithi088.stdout: "op_resend": 28, 2024-03-12T06:06:29.568 INFO:teuthology.orchestra.run.smithi088.stdout: "op_reply": 27659, 2024-03-12T06:06:29.568 INFO:teuthology.orchestra.run.smithi088.stdout: "op_latency": { 2024-03-12T06:06:29.568 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 27659, 2024-03-12T06:06:29.568 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 12.569766277, 2024-03-12T06:06:29.568 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000454454 2024-03-12T06:06:29.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.568 INFO:teuthology.orchestra.run.smithi088.stdout: "op_inflight": 0, 2024-03-12T06:06:29.569 INFO:teuthology.orchestra.run.smithi088.stdout: "oplen_avg": { 2024-03-12T06:06:29.569 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 27659, 2024-03-12T06:06:29.569 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 27703 2024-03-12T06:06:29.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.569 INFO:teuthology.orchestra.run.smithi088.stdout: "op": 27659, 2024-03-12T06:06:29.569 INFO:teuthology.orchestra.run.smithi088.stdout: "op_r": 27196, 2024-03-12T06:06:29.569 INFO:teuthology.orchestra.run.smithi088.stdout: "op_w": 463, 2024-03-12T06:06:29.570 INFO:teuthology.orchestra.run.smithi088.stdout: "op_rmw": 0, 2024-03-12T06:06:29.570 INFO:teuthology.orchestra.run.smithi088.stdout: "op_pg": 0, 2024-03-12T06:06:29.570 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_stat": 0, 2024-03-12T06:06:29.570 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_create": 10, 2024-03-12T06:06:29.570 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_read": 9, 2024-03-12T06:06:29.570 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_write": 75, 2024-03-12T06:06:29.570 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_writefull": 17, 2024-03-12T06:06:29.570 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_writesame": 0, 2024-03-12T06:06:29.571 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_append": 0, 2024-03-12T06:06:29.571 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_zero": 0, 2024-03-12T06:06:29.571 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_truncate": 0, 2024-03-12T06:06:29.571 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_delete": 329, 2024-03-12T06:06:29.571 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_mapext": 0, 2024-03-12T06:06:29.571 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_sparse_read": 0, 2024-03-12T06:06:29.571 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_clonerange": 0, 2024-03-12T06:06:29.571 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_getxattr": 27187, 2024-03-12T06:06:29.571 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_setxattr": 20, 2024-03-12T06:06:29.572 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_cmpxattr": 0, 2024-03-12T06:06:29.572 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_rmxattr": 0, 2024-03-12T06:06:29.572 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_resetxattrs": 0, 2024-03-12T06:06:29.572 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_call": 0, 2024-03-12T06:06:29.572 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_watch": 0, 2024-03-12T06:06:29.572 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_notify": 0, 2024-03-12T06:06:29.572 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_src_cmpxattr": 0, 2024-03-12T06:06:29.572 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_pgls": 0, 2024-03-12T06:06:29.573 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_pgls_filter": 0, 2024-03-12T06:06:29.573 INFO:teuthology.orchestra.run.smithi088.stdout: "osdop_other": 0, 2024-03-12T06:06:29.573 INFO:teuthology.orchestra.run.smithi088.stdout: "linger_active": 0, 2024-03-12T06:06:29.573 INFO:teuthology.orchestra.run.smithi088.stdout: "linger_send": 0, 2024-03-12T06:06:29.573 INFO:teuthology.orchestra.run.smithi088.stdout: "linger_resend": 0, 2024-03-12T06:06:29.573 INFO:teuthology.orchestra.run.smithi088.stdout: "linger_ping": 0, 2024-03-12T06:06:29.573 INFO:teuthology.orchestra.run.smithi088.stdout: "poolop_active": 0, 2024-03-12T06:06:29.573 INFO:teuthology.orchestra.run.smithi088.stdout: "poolop_send": 0, 2024-03-12T06:06:29.573 INFO:teuthology.orchestra.run.smithi088.stdout: "poolop_resend": 0, 2024-03-12T06:06:29.574 INFO:teuthology.orchestra.run.smithi088.stdout: "poolstat_active": 0, 2024-03-12T06:06:29.574 INFO:teuthology.orchestra.run.smithi088.stdout: "poolstat_send": 0, 2024-03-12T06:06:29.574 INFO:teuthology.orchestra.run.smithi088.stdout: "poolstat_resend": 0, 2024-03-12T06:06:29.574 INFO:teuthology.orchestra.run.smithi088.stdout: "statfs_active": 0, 2024-03-12T06:06:29.574 INFO:teuthology.orchestra.run.smithi088.stdout: "statfs_send": 0, 2024-03-12T06:06:29.574 INFO:teuthology.orchestra.run.smithi088.stdout: "statfs_resend": 0, 2024-03-12T06:06:29.575 INFO:teuthology.orchestra.run.smithi088.stdout: "command_active": 0, 2024-03-12T06:06:29.575 INFO:teuthology.orchestra.run.smithi088.stdout: "command_send": 0, 2024-03-12T06:06:29.575 INFO:teuthology.orchestra.run.smithi088.stdout: "command_resend": 0, 2024-03-12T06:06:29.575 INFO:teuthology.orchestra.run.smithi088.stdout: "map_epoch": 97, 2024-03-12T06:06:29.575 INFO:teuthology.orchestra.run.smithi088.stdout: "map_full": 0, 2024-03-12T06:06:29.575 INFO:teuthology.orchestra.run.smithi088.stdout: "map_inc": 11, 2024-03-12T06:06:29.575 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_sessions": 12, 2024-03-12T06:06:29.575 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_session_open": 12, 2024-03-12T06:06:29.575 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_session_close": 0, 2024-03-12T06:06:29.576 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_laggy": 0, 2024-03-12T06:06:29.576 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_wr": 46, 2024-03-12T06:06:29.576 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_rd": 0, 2024-03-12T06:06:29.576 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_del": 10 2024-03-12T06:06:29.576 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.576 INFO:teuthology.orchestra.run.smithi088.stdout: "oft": { 2024-03-12T06:06:29.576 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_total_objs": 1, 2024-03-12T06:06:29.576 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_total_kv_pairs": 0, 2024-03-12T06:06:29.577 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_total_updates": 0, 2024-03-12T06:06:29.577 INFO:teuthology.orchestra.run.smithi088.stdout: "omap_total_removes": 0 2024-03-12T06:06:29.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.577 INFO:teuthology.orchestra.run.smithi088.stdout: "osd-slow-ops": { 2024-03-12T06:06:29.577 INFO:teuthology.orchestra.run.smithi088.stdout: "slow_ops_count": 0 2024-03-12T06:06:29.577 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.577 INFO:teuthology.orchestra.run.smithi088.stdout: "purge_queue": { 2024-03-12T06:06:29.577 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executing_ops": 0, 2024-03-12T06:06:29.577 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executing_ops_high_water": 10, 2024-03-12T06:06:29.578 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executing": 0, 2024-03-12T06:06:29.578 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executing_high_water": 10, 2024-03-12T06:06:29.578 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executed_ops": 316, 2024-03-12T06:06:29.578 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_executed": 316, 2024-03-12T06:06:29.578 INFO:teuthology.orchestra.run.smithi088.stdout: "pq_item_in_journal": 0 2024-03-12T06:06:29.578 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.578 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T06:06:29.579 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:29.579 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 104857600, 2024-03-12T06:06:29.579 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:29.579 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 32207, 2024-03-12T06:06:29.579 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 9157418, 2024-03-12T06:06:29.579 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:29.579 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 32207, 2024-03-12T06:06:29.579 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 0, 2024-03-12T06:06:29.580 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 0, 2024-03-12T06:06:29.580 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 32207, 2024-03-12T06:06:29.580 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 9157418, 2024-03-12T06:06:29.580 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:29.580 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.580 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.580 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.580 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:29.580 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.581 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-objecter_bytes": { 2024-03-12T06:06:29.581 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:29.581 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 104857600, 2024-03-12T06:06:29.581 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:29.581 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 0, 2024-03-12T06:06:29.581 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 0, 2024-03-12T06:06:29.582 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:29.582 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:29.582 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 27659, 2024-03-12T06:06:29.582 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 6021641, 2024-03-12T06:06:29.582 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 27659, 2024-03-12T06:06:29.582 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 6021641, 2024-03-12T06:06:29.583 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:29.583 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.583 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.583 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.583 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:29.583 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.583 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-objecter_ops": { 2024-03-12T06:06:29.583 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:29.584 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 1024, 2024-03-12T06:06:29.584 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:29.584 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 0, 2024-03-12T06:06:29.584 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 0, 2024-03-12T06:06:29.584 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:29.584 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 0, 2024-03-12T06:06:29.584 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 27659, 2024-03-12T06:06:29.584 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 27659, 2024-03-12T06:06:29.585 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 27659, 2024-03-12T06:06:29.585 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 27659, 2024-03-12T06:06:29.585 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:29.585 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.585 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.585 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.585 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:29.585 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.585 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-write_buf_throttle": { 2024-03-12T06:06:29.586 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:29.586 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 3758096384, 2024-03-12T06:06:29.586 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:29.586 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 316, 2024-03-12T06:06:29.586 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 31916, 2024-03-12T06:06:29.586 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:29.586 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 316, 2024-03-12T06:06:29.586 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 0, 2024-03-12T06:06:29.587 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 0, 2024-03-12T06:06:29.587 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 9, 2024-03-12T06:06:29.587 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 31916, 2024-03-12T06:06:29.587 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:29.587 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.587 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.587 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.587 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:29.587 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:29.588 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-write_buf_throttle-0x5570e2e36640": { 2024-03-12T06:06:29.588 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:29.588 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 3758096384, 2024-03-12T06:06:29.588 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:29.588 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 2330, 2024-03-12T06:06:29.588 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 5704676, 2024-03-12T06:06:29.588 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:29.589 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 2330, 2024-03-12T06:06:29.589 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 0, 2024-03-12T06:06:29.589 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 0, 2024-03-12T06:06:29.589 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 66, 2024-03-12T06:06:29.589 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 5704676, 2024-03-12T06:06:29.589 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:29.589 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:29.589 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:29.589 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:29.590 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:29.590 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:29.590 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-03-12T06:06:29.696 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:29 smithi082 ceph-mon[36382]: pgmap v332: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:06:29.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:29 smithi076 ceph-mon[32095]: pgmap v332: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:06:30.271 INFO:tasks.check_counter:Daemon mds.i mds.exported=15 2024-03-12T06:06:30.271 INFO:tasks.check_counter:Daemon mds.i mds.imported=29 2024-03-12T06:06:30.271 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=7 2024-03-12T06:06:30.272 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-12T06:06:30.272 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-12T06:06:30.272 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-03-12T06:06:30.272 DEBUG:tasks.check_counter:Getting stats from l 2024-03-12T06:06:30.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 9f336de8-e033-11ee-95c8-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-12T06:06:30.512 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/mon.c/config 2024-03-12T06:06:31.451 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:31 smithi088 ceph-mon[36585]: pgmap v333: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:06:31.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:31 smithi076 ceph-mon[32095]: pgmap v333: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:06:31.697 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:31 smithi082 ceph-mon[36382]: pgmap v333: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:06:31.963 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-03-12T06:06:31.963 INFO:teuthology.orchestra.run.smithi088.stdout: "AsyncMessenger::Worker-0": { 2024-03-12T06:06:31.963 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_messages": 0, 2024-03-12T06:06:31.963 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages": 2, 2024-03-12T06:06:31.964 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_bytes": 0, 2024-03-12T06:06:31.964 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_bytes": 0, 2024-03-12T06:06:31.964 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_created_connections": 4, 2024-03-12T06:06:31.964 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_active_connections": 0, 2024-03-12T06:06:31.964 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_total_time": 0.001106337, 2024-03-12T06:06:31.964 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_send_time": 0.000000000, 2024-03-12T06:06:31.964 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_recv_time": 0.000218477, 2024-03-12T06:06:31.965 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T06:06:31.965 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:31.965 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:31.965 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:31.965 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:31.965 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:31.965 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:31.965 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:31.965 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:31.966 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:31.966 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:31.966 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-12T06:06:31.966 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_encrypted_bytes": 0 2024-03-12T06:06:31.966 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:31.966 INFO:teuthology.orchestra.run.smithi088.stdout: "AsyncMessenger::Worker-1": { 2024-03-12T06:06:31.966 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_messages": 2, 2024-03-12T06:06:31.966 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages": 166, 2024-03-12T06:06:31.967 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_bytes": 320, 2024-03-12T06:06:31.967 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_bytes": 183920, 2024-03-12T06:06:31.967 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_created_connections": 5, 2024-03-12T06:06:31.967 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_active_connections": 3, 2024-03-12T06:06:31.967 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_total_time": 0.011986371, 2024-03-12T06:06:31.967 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_send_time": 0.008117939, 2024-03-12T06:06:31.967 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_recv_time": 0.001250906, 2024-03-12T06:06:31.967 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-12T06:06:31.968 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:31.968 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 163, 2024-03-12T06:06:31.968 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.017800876, 2024-03-12T06:06:31.968 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000109207 2024-03-12T06:06:31.968 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:31.968 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:31.968 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:31.968 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:31.968 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:31.969 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:31.969 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-12T06:06:31.969 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_encrypted_bytes": 183920 2024-03-12T06:06:31.969 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:31.969 INFO:teuthology.orchestra.run.smithi088.stdout: "AsyncMessenger::Worker-2": { 2024-03-12T06:06:31.969 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_messages": 233, 2024-03-12T06:06:31.969 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages": 220, 2024-03-12T06:06:31.969 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_bytes": 1568560, 2024-03-12T06:06:31.969 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_bytes": 131008, 2024-03-12T06:06:31.970 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_created_connections": 2, 2024-03-12T06:06:31.970 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_active_connections": 1, 2024-03-12T06:06:31.970 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_total_time": 0.046426748, 2024-03-12T06:06:31.970 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_send_time": 0.014996430, 2024-03-12T06:06:31.970 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_recv_time": 0.034853803, 2024-03-12T06:06:31.970 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_running_fast_dispatch_time": 0.005447719, 2024-03-12T06:06:31.971 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_messages_queue_lat": { 2024-03-12T06:06:31.971 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 220, 2024-03-12T06:06:31.971 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.012600813, 2024-03-12T06:06:31.971 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000057276 2024-03-12T06:06:31.971 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:31.972 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_handle_ack_lat": { 2024-03-12T06:06:31.972 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:31.972 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:31.972 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:31.972 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:31.972 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_recv_encrypted_bytes": 1568560, 2024-03-12T06:06:31.972 INFO:teuthology.orchestra.run.smithi088.stdout: "msgr_send_encrypted_bytes": 131008 2024-03-12T06:06:31.972 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:31.973 INFO:teuthology.orchestra.run.smithi088.stdout: "cct": { 2024-03-12T06:06:31.973 INFO:teuthology.orchestra.run.smithi088.stdout: "total_workers": 0, 2024-03-12T06:06:31.973 INFO:teuthology.orchestra.run.smithi088.stdout: "unhealthy_workers": 0 2024-03-12T06:06:31.973 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:31.973 INFO:teuthology.orchestra.run.smithi088.stdout: "mempool": { 2024-03-12T06:06:31.973 INFO:teuthology.orchestra.run.smithi088.stdout: "bloom_filter_bytes": 0, 2024-03-12T06:06:31.973 INFO:teuthology.orchestra.run.smithi088.stdout: "bloom_filter_items": 0, 2024-03-12T06:06:31.974 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_alloc_bytes": 0, 2024-03-12T06:06:31.974 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_alloc_items": 0, 2024-03-12T06:06:31.974 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_data_bytes": 0, 2024-03-12T06:06:31.974 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_data_items": 0, 2024-03-12T06:06:31.974 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-12T06:06:31.975 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_onode_items": 0, 2024-03-12T06:06:31.975 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-12T06:06:31.975 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_meta_items": 0, 2024-03-12T06:06:31.975 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_other_bytes": 0, 2024-03-12T06:06:31.975 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_other_items": 0, 2024-03-12T06:06:31.975 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-12T06:06:31.975 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_cache_buffer_items": 0, 2024-03-12T06:06:31.975 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_extent_bytes": 0, 2024-03-12T06:06:31.975 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_extent_items": 0, 2024-03-12T06:06:31.976 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_blob_bytes": 0, 2024-03-12T06:06:31.976 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_blob_items": 0, 2024-03-12T06:06:31.976 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-12T06:06:31.976 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_shared_blob_items": 0, 2024-03-12T06:06:31.976 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-12T06:06:31.976 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_inline_bl_items": 0, 2024-03-12T06:06:31.976 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_bytes": 0, 2024-03-12T06:06:31.976 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_fsck_items": 0, 2024-03-12T06:06:31.976 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_txc_bytes": 0, 2024-03-12T06:06:31.977 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_txc_items": 0, 2024-03-12T06:06:31.977 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-12T06:06:31.977 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_deferred_items": 0, 2024-03-12T06:06:31.977 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_bytes": 0, 2024-03-12T06:06:31.977 INFO:teuthology.orchestra.run.smithi088.stdout: "bluestore_writing_items": 0, 2024-03-12T06:06:31.977 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_bytes": 0, 2024-03-12T06:06:31.977 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_items": 0, 2024-03-12T06:06:31.977 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_reader_bytes": 0, 2024-03-12T06:06:31.978 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_reader_items": 0, 2024-03-12T06:06:31.978 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_writer_bytes": 0, 2024-03-12T06:06:31.978 INFO:teuthology.orchestra.run.smithi088.stdout: "bluefs_file_writer_items": 0, 2024-03-12T06:06:31.978 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_anon_bytes": 148967, 2024-03-12T06:06:31.978 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_anon_items": 29, 2024-03-12T06:06:31.978 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_meta_bytes": 0, 2024-03-12T06:06:31.979 INFO:teuthology.orchestra.run.smithi088.stdout: "buffer_meta_items": 0, 2024-03-12T06:06:31.979 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_bytes": 0, 2024-03-12T06:06:31.979 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_items": 0, 2024-03-12T06:06:31.979 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mapbl_bytes": 0, 2024-03-12T06:06:31.979 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_mapbl_items": 0, 2024-03-12T06:06:31.979 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pglog_bytes": 0, 2024-03-12T06:06:31.979 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_pglog_items": 0, 2024-03-12T06:06:31.979 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_bytes": 0, 2024-03-12T06:06:31.980 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_items": 0, 2024-03-12T06:06:31.980 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_mapping_bytes": 0, 2024-03-12T06:06:31.980 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_mapping_items": 0, 2024-03-12T06:06:31.980 INFO:teuthology.orchestra.run.smithi088.stdout: "pgmap_bytes": 0, 2024-03-12T06:06:31.980 INFO:teuthology.orchestra.run.smithi088.stdout: "pgmap_items": 0, 2024-03-12T06:06:31.980 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_co_bytes": 928, 2024-03-12T06:06:31.980 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_co_items": 2, 2024-03-12T06:06:31.980 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_1_bytes": 0, 2024-03-12T06:06:31.981 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_1_items": 0, 2024-03-12T06:06:31.981 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_2_bytes": 0, 2024-03-12T06:06:31.981 INFO:teuthology.orchestra.run.smithi088.stdout: "unittest_2_items": 0 2024-03-12T06:06:31.981 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-03-12T06:06:31.981 INFO:teuthology.orchestra.run.smithi088.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-12T06:06:31.981 INFO:teuthology.orchestra.run.smithi088.stdout: "val": 0, 2024-03-12T06:06:31.981 INFO:teuthology.orchestra.run.smithi088.stdout: "max": 104857600, 2024-03-12T06:06:31.981 INFO:teuthology.orchestra.run.smithi088.stdout: "get_started": 0, 2024-03-12T06:06:31.982 INFO:teuthology.orchestra.run.smithi088.stdout: "get": 231, 2024-03-12T06:06:31.982 INFO:teuthology.orchestra.run.smithi088.stdout: "get_sum": 1533282, 2024-03-12T06:06:31.982 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_fail": 0, 2024-03-12T06:06:31.982 INFO:teuthology.orchestra.run.smithi088.stdout: "get_or_fail_success": 231, 2024-03-12T06:06:31.982 INFO:teuthology.orchestra.run.smithi088.stdout: "take": 0, 2024-03-12T06:06:31.982 INFO:teuthology.orchestra.run.smithi088.stdout: "take_sum": 0, 2024-03-12T06:06:31.982 INFO:teuthology.orchestra.run.smithi088.stdout: "put": 231, 2024-03-12T06:06:31.983 INFO:teuthology.orchestra.run.smithi088.stdout: "put_sum": 1533282, 2024-03-12T06:06:31.983 INFO:teuthology.orchestra.run.smithi088.stdout: "wait": { 2024-03-12T06:06:31.983 INFO:teuthology.orchestra.run.smithi088.stdout: "avgcount": 0, 2024-03-12T06:06:31.983 INFO:teuthology.orchestra.run.smithi088.stdout: "sum": 0.000000000, 2024-03-12T06:06:31.983 INFO:teuthology.orchestra.run.smithi088.stdout: "avgtime": 0.000000000 2024-03-12T06:06:31.983 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:31.983 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-03-12T06:06:31.983 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-03-12T06:06:32.629 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-12T06:06:32.630 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-12T06:06:32.630 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-12T06:06:32.630 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-12T06:06:32.630 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-12T06:06:32.630 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-03-12T06:06:32.630 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-12T06:06:32.642 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-12T06:06:32.642 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T06:06:32.642 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-12T06:06:32.666 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-12T06:06:32.666 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-12T06:06:32.725 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T06:06:32.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.877+0000 7f6212eec640 1 -- 172.21.15.76:0/226983376 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620c100c30 msgr2=0x7f620c104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:32.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.877+0000 7f6212eec640 1 --2- 172.21.15.76:0/226983376 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620c100c30 0x7f620c104c70 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f61f4009930 tx=0x7f61f402bc30 comp rx=0 tx=0).stop 2024-03-12T06:06:32.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 -- 172.21.15.76:0/226983376 shutdown_connections 2024-03-12T06:06:32.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 --2- 172.21.15.76:0/226983376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f620c105350 0x7f620c10cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:32.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 --2- 172.21.15.76:0/226983376 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620c100c30 0x7f620c104c70 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:32.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 --2- 172.21.15.76:0/226983376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f620c100280 0x7f620c100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:32.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 -- 172.21.15.76:0/226983376 >> 172.21.15.76:0/226983376 conn(0x7f620c0fbb00 msgr2=0x7f620c0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:32.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 -- 172.21.15.76:0/226983376 shutdown_connections 2024-03-12T06:06:32.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 -- 172.21.15.76:0/226983376 wait complete. 2024-03-12T06:06:32.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 Processor -- start 2024-03-12T06:06:32.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 -- start start 2024-03-12T06:06:32.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f620c100280 0x7f620c19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:32.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f620c105350 0x7f620c19d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:32.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620c197630 0x7f620c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:32.886 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c198020 con 0x7f620c197630 2024-03-12T06:06:32.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c198190 con 0x7f620c105350 2024-03-12T06:06:32.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.878+0000 7f6212eec640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c198350 con 0x7f620c100280 2024-03-12T06:06:32.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6211489640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620c197630 0x7f620c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:32.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6211489640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620c197630 0x7f620c197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47032/0 (socket says 172.21.15.76:47032) 2024-03-12T06:06:32.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6211489640 1 -- 172.21.15.76:0/857582891 learned_addr learned my addr 172.21.15.76:0/857582891 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:06:32.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f620bfff640 1 --2- 172.21.15.76:0/857582891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f620c105350 0x7f620c19d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:32.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6210c88640 1 --2- 172.21.15.76:0/857582891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f620c100280 0x7f620c19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:32.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6211489640 1 -- 172.21.15.76:0/857582891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f620c100280 msgr2=0x7f620c19d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:32.887 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6211489640 1 --2- 172.21.15.76:0/857582891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f620c100280 0x7f620c19d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:32.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6211489640 1 -- 172.21.15.76:0/857582891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f620c105350 msgr2=0x7f620c19d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:32.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6211489640 1 --2- 172.21.15.76:0/857582891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f620c105350 0x7f620c19d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:32.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6211489640 1 -- 172.21.15.76:0/857582891 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f4009560 con 0x7f620c197630 2024-03-12T06:06:32.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6211489640 1 --2- 172.21.15.76:0/857582891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620c197630 0x7f620c197ae0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00d750 tx=0x7f61fc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:32.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6209ffb640 1 -- 172.21.15.76:0/857582891 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc009bd0 con 0x7f620c197630 2024-03-12T06:06:32.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6212eec640 1 -- 172.21.15.76:0/857582891 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620c1a42f0 con 0x7f620c197630 2024-03-12T06:06:32.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6209ffb640 1 -- 172.21.15.76:0/857582891 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61fc00de60 con 0x7f620c197630 2024-03-12T06:06:32.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6209ffb640 1 -- 172.21.15.76:0/857582891 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc004e70 con 0x7f620c197630 2024-03-12T06:06:32.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.879+0000 7f6212eec640 1 -- 172.21.15.76:0/857582891 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620c1a47d0 con 0x7f620c197630 2024-03-12T06:06:32.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.880+0000 7f6212eec640 1 -- 172.21.15.76:0/857582891 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620c101750 con 0x7f620c197630 2024-03-12T06:06:32.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.881+0000 7f6209ffb640 1 -- 172.21.15.76:0/857582891 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f61fc005020 con 0x7f620c197630 2024-03-12T06:06:32.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.882+0000 7f6209ffb640 1 --2- 172.21.15.76:0/857582891 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f61f80a1ba0 0x7f61f80a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:32.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.882+0000 7f6210c88640 1 --2- 172.21.15.76:0/857582891 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f61f80a1ba0 0x7f61f80a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:32.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.882+0000 7f6209ffb640 1 -- 172.21.15.76:0/857582891 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f61fc0cd0e0 con 0x7f620c197630 2024-03-12T06:06:32.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.882+0000 7f6210c88640 1 --2- 172.21.15.76:0/857582891 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f61f80a1ba0 0x7f61f80a4060 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6204005fd0 tx=0x7f6204009500 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:32.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:32.886+0000 7f6209ffb640 1 -- 172.21.15.76:0/857582891 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f61fc099b80 con 0x7f620c197630 2024-03-12T06:06:33.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.061+0000 7f6212eec640 1 -- 172.21.15.76:0/857582891 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f620c100c30 con 0x7f620c197630 2024-03-12T06:06:33.068 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.063+0000 7f6209ffb640 1 -- 172.21.15.76:0/857582891 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+23008 (secure 0 0 0) 0x7f61fc020090 con 0x7f620c197630 2024-03-12T06:06:33.068 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:06:33.068 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":97,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T06:06:16.517813+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":16,"snap_epoch":96,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6800/1718825251":"2024-03-13T05:55:46.980746+0000","172.21.15.76:0/189327369":"2024-03-13T05:55:46.980746+0000","172.21.15.76:0/3998885619":"2024-03-13T05:55:46.980746+0000","172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/2299856451":"2024-03-13T05:55:46.980746+0000","172.21.15.76:6801/1718825251":"2024-03-13T05:55:46.980746+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1939795424":"2024-03-13T05:55:46.980746+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":16,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T06:06:33.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.065+0000 7f6212eec640 1 -- 172.21.15.76:0/857582891 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f61f80a1ba0 msgr2=0x7f61f80a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.070 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.065+0000 7f6212eec640 1 --2- 172.21.15.76:0/857582891 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f61f80a1ba0 0x7f61f80a4060 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6204005fd0 tx=0x7f6204009500 comp rx=0 tx=0).stop 2024-03-12T06:06:33.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.065+0000 7f6212eec640 1 -- 172.21.15.76:0/857582891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620c197630 msgr2=0x7f620c197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.065+0000 7f6212eec640 1 --2- 172.21.15.76:0/857582891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620c197630 0x7f620c197ae0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00d750 tx=0x7f61fc00dc10 comp rx=0 tx=0).stop 2024-03-12T06:06:33.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.065+0000 7f6212eec640 1 -- 172.21.15.76:0/857582891 shutdown_connections 2024-03-12T06:06:33.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.065+0000 7f6212eec640 1 --2- 172.21.15.76:0/857582891 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f61f80a1ba0 0x7f61f80a4060 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.065+0000 7f6212eec640 1 --2- 172.21.15.76:0/857582891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f620c197630 0x7f620c197ae0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.065+0000 7f6212eec640 1 --2- 172.21.15.76:0/857582891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f620c105350 0x7f620c19d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.065+0000 7f6212eec640 1 --2- 172.21.15.76:0/857582891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f620c100280 0x7f620c19d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.071 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.065+0000 7f6212eec640 1 -- 172.21.15.76:0/857582891 >> 172.21.15.76:0/857582891 conn(0x7f620c0fbb00 msgr2=0x7f620c0fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:33.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.065+0000 7f6212eec640 1 -- 172.21.15.76:0/857582891 shutdown_connections 2024-03-12T06:06:33.072 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.065+0000 7f6212eec640 1 -- 172.21.15.76:0/857582891 wait complete. 2024-03-12T06:06:33.082 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T05:49:09.125536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-12T05:53:35.663129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-12T05:53:36.740826+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 16, 'snap_epoch': 96, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T06:06:33.082 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T06:06:33.188 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.183+0000 7fb3291f4640 1 -- 172.21.15.76:0/2352978965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb324102d50 msgr2=0x7fb3241031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.183+0000 7fb3291f4640 1 --2- 172.21.15.76:0/2352978965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb324102d50 0x7fb3241031b0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fb30c009930 tx=0x7fb30c02bc30 comp rx=0 tx=0).stop 2024-03-12T06:06:33.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.183+0000 7fb3291f4640 1 -- 172.21.15.76:0/2352978965 shutdown_connections 2024-03-12T06:06:33.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.183+0000 7fb3291f4640 1 --2- 172.21.15.76:0/2352978965 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3241036f0 0x7fb32410e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.183+0000 7fb3291f4640 1 --2- 172.21.15.76:0/2352978965 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb324102d50 0x7fb3241031b0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.183+0000 7fb3291f4640 1 --2- 172.21.15.76:0/2352978965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324101470 0x7fb324101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.183+0000 7fb3291f4640 1 -- 172.21.15.76:0/2352978965 >> 172.21.15.76:0/2352978965 conn(0x7fb3240fcc10 msgr2=0x7fb3240ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:33.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.183+0000 7fb3291f4640 1 -- 172.21.15.76:0/2352978965 shutdown_connections 2024-03-12T06:06:33.189 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.183+0000 7fb3291f4640 1 -- 172.21.15.76:0/2352978965 wait complete. 2024-03-12T06:06:33.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.184+0000 7fb3291f4640 1 Processor -- start 2024-03-12T06:06:33.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.184+0000 7fb3291f4640 1 -- start start 2024-03-12T06:06:33.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.184+0000 7fb3291f4640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb324101470 0x7fb32419d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:33.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.184+0000 7fb3291f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324102d50 0x7fb32419dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:33.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.184+0000 7fb3291f4640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3241036f0 0x7fb324197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:33.192 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.184+0000 7fb3291f4640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb324197df0 con 0x7fb3241036f0 2024-03-12T06:06:33.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.184+0000 7fb3291f4640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb324197f60 con 0x7fb324101470 2024-03-12T06:06:33.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.184+0000 7fb3291f4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb324198120 con 0x7fb324102d50 2024-03-12T06:06:33.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.184+0000 7fb32359e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3241036f0 0x7fb324197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:33.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.184+0000 7fb32359e640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3241036f0 0x7fb324197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47042/0 (socket says 172.21.15.76:47042) 2024-03-12T06:06:33.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.184+0000 7fb32359e640 1 -- 172.21.15.76:0/1503549606 learned_addr learned my addr 172.21.15.76:0/1503549606 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:06:33.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.184+0000 7fb31a59c640 1 --2- 172.21.15.76:0/1503549606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324102d50 0x7fb32419dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:33.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.185+0000 7fb322d9d640 1 --2- 172.21.15.76:0/1503549606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb324101470 0x7fb32419d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:33.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.185+0000 7fb32359e640 1 -- 172.21.15.76:0/1503549606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324102d50 msgr2=0x7fb32419dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.193 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.185+0000 7fb32359e640 1 --2- 172.21.15.76:0/1503549606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324102d50 0x7fb32419dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.185+0000 7fb32359e640 1 -- 172.21.15.76:0/1503549606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb324101470 msgr2=0x7fb32419d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.185+0000 7fb32359e640 1 --2- 172.21.15.76:0/1503549606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb324101470 0x7fb32419d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.185+0000 7fb32359e640 1 -- 172.21.15.76:0/1503549606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30c009560 con 0x7fb3241036f0 2024-03-12T06:06:33.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.185+0000 7fb32359e640 1 --2- 172.21.15.76:0/1503549606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3241036f0 0x7fb324197660 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00cce0 tx=0x7fb31c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:33.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.185+0000 7fb320d99640 1 -- 172.21.15.76:0/1503549606 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c00f450 con 0x7fb3241036f0 2024-03-12T06:06:33.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.185+0000 7fb320d99640 1 -- 172.21.15.76:0/1503549606 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb31c00fa90 con 0x7fb3241036f0 2024-03-12T06:06:33.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.185+0000 7fb3291f4640 1 -- 172.21.15.76:0/1503549606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb324198400 con 0x7fb3241036f0 2024-03-12T06:06:33.194 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.185+0000 7fb320d99640 1 -- 172.21.15.76:0/1503549606 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c018b90 con 0x7fb3241036f0 2024-03-12T06:06:33.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.185+0000 7fb3291f4640 1 -- 172.21.15.76:0/1503549606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3241a45a0 con 0x7fb3241036f0 2024-03-12T06:06:33.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.186+0000 7fb3291f4640 1 -- 172.21.15.76:0/1503549606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb324102860 con 0x7fb3241036f0 2024-03-12T06:06:33.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.187+0000 7fb320d99640 1 -- 172.21.15.76:0/1503549606 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fb31c00fc00 con 0x7fb3241036f0 2024-03-12T06:06:33.195 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.188+0000 7fb320d99640 1 --2- 172.21.15.76:0/1503549606 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb3100a1ba0 0x7fb3100a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:33.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.188+0000 7fb322d9d640 1 --2- 172.21.15.76:0/1503549606 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb3100a1ba0 0x7fb3100a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:33.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.188+0000 7fb320d99640 1 -- 172.21.15.76:0/1503549606 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb31c0ce090 con 0x7fb3241036f0 2024-03-12T06:06:33.196 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.188+0000 7fb322d9d640 1 --2- 172.21.15.76:0/1503549606 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb3100a1ba0 0x7fb3100a4060 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb31400a810 tx=0x7fb314008040 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:33.197 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.192+0000 7fb320d99640 1 -- 172.21.15.76:0/1503549606 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb31c096f60 con 0x7fb3241036f0 2024-03-12T06:06:33.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.367+0000 7fb3291f4640 1 -- 172.21.15.76:0/1503549606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb324109740 con 0x7fb3241036f0 2024-03-12T06:06:33.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.367+0000 7fb320d99640 1 -- 172.21.15.76:0/1503549606 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb31c09a620 con 0x7fb3241036f0 2024-03-12T06:06:33.373 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2024-03-12T06:06:33.374 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.369+0000 7fb3291f4640 1 -- 172.21.15.76:0/1503549606 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb3100a1ba0 msgr2=0x7fb3100a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.369+0000 7fb3291f4640 1 --2- 172.21.15.76:0/1503549606 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb3100a1ba0 0x7fb3100a4060 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb31400a810 tx=0x7fb314008040 comp rx=0 tx=0).stop 2024-03-12T06:06:33.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.369+0000 7fb3291f4640 1 -- 172.21.15.76:0/1503549606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3241036f0 msgr2=0x7fb324197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.369+0000 7fb3291f4640 1 --2- 172.21.15.76:0/1503549606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3241036f0 0x7fb324197660 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00cce0 tx=0x7fb31c00a620 comp rx=0 tx=0).stop 2024-03-12T06:06:33.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.369+0000 7fb3291f4640 1 -- 172.21.15.76:0/1503549606 shutdown_connections 2024-03-12T06:06:33.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.369+0000 7fb3291f4640 1 --2- 172.21.15.76:0/1503549606 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fb3100a1ba0 0x7fb3100a4060 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.369+0000 7fb3291f4640 1 --2- 172.21.15.76:0/1503549606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3241036f0 0x7fb324197660 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.369+0000 7fb3291f4640 1 --2- 172.21.15.76:0/1503549606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb324102d50 0x7fb32419dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.369+0000 7fb3291f4640 1 --2- 172.21.15.76:0/1503549606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb324101470 0x7fb32419d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.369+0000 7fb3291f4640 1 -- 172.21.15.76:0/1503549606 >> 172.21.15.76:0/1503549606 conn(0x7fb3240fcc10 msgr2=0x7fb3240fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:33.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.369+0000 7fb3291f4640 1 -- 172.21.15.76:0/1503549606 shutdown_connections 2024-03-12T06:06:33.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.369+0000 7fb3291f4640 1 -- 172.21.15.76:0/1503549606 wait complete. 2024-03-12T06:06:33.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:33 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/857582891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T06:06:33.385 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-12T06:06:33.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:33 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/857582891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T06:06:33.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:33 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/857582891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T06:06:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.486+0000 7ff077640640 1 -- 172.21.15.76:0/4194222150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff070100c40 msgr2=0x7ff070104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.486+0000 7ff077640640 1 --2- 172.21.15.76:0/4194222150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff070100c40 0x7ff070104c80 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7ff060009930 tx=0x7ff06002bc30 comp rx=0 tx=0).stop 2024-03-12T06:06:33.495 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.486+0000 7ff077640640 1 -- 172.21.15.76:0/4194222150 shutdown_connections 2024-03-12T06:06:33.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.486+0000 7ff077640640 1 --2- 172.21.15.76:0/4194222150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff070105360 0x7ff07010cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.486+0000 7ff077640640 1 --2- 172.21.15.76:0/4194222150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff070100c40 0x7ff070104c80 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.486+0000 7ff077640640 1 --2- 172.21.15.76:0/4194222150 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff070100290 0x7ff070100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.486+0000 7ff077640640 1 -- 172.21.15.76:0/4194222150 >> 172.21.15.76:0/4194222150 conn(0x7ff070077230 msgr2=0x7ff070077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:33.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.486+0000 7ff077640640 1 -- 172.21.15.76:0/4194222150 shutdown_connections 2024-03-12T06:06:33.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.486+0000 7ff077640640 1 -- 172.21.15.76:0/4194222150 wait complete. 2024-03-12T06:06:33.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.486+0000 7ff077640640 1 Processor -- start 2024-03-12T06:06:33.496 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.486+0000 7ff077640640 1 -- start start 2024-03-12T06:06:33.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.486+0000 7ff077640640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff070100290 0x7ff070198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:33.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff077640640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff070105360 0x7ff070198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:33.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff077640640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0701990f0 0x7ff0701a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:33.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff077640640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0701a3d70 con 0x7ff070100290 2024-03-12T06:06:33.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff077640640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0701a3ee0 con 0x7ff070105360 2024-03-12T06:06:33.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff077640640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0701a40a0 con 0x7ff0701990f0 2024-03-12T06:06:33.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff0753dc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff070100290 0x7ff070198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:33.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff0753dc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff070100290 0x7ff070198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47050/0 (socket says 172.21.15.76:47050) 2024-03-12T06:06:33.497 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff0753dc640 1 -- 172.21.15.76:0/1592314066 learned_addr learned my addr 172.21.15.76:0/1592314066 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:06:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff074bdb640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff070105360 0x7ff070198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff075bdd640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0701990f0 0x7ff0701a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff0753dc640 1 -- 172.21.15.76:0/1592314066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0701990f0 msgr2=0x7ff0701a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff0753dc640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0701990f0 0x7ff0701a36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff0753dc640 1 -- 172.21.15.76:0/1592314066 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff070105360 msgr2=0x7ff070198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff0753dc640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff070105360 0x7ff070198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff0753dc640 1 -- 172.21.15.76:0/1592314066 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff060009560 con 0x7ff070100290 2024-03-12T06:06:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff074bdb640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff070105360 0x7ff070198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:06:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff075bdd640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0701990f0 0x7ff0701a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:06:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff0753dc640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff070100290 0x7ff070198620 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00cce0 tx=0x7ff06c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff0667fc640 1 -- 172.21.15.76:0/1592314066 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c004b20 con 0x7ff070100290 2024-03-12T06:06:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff0667fc640 1 -- 172.21.15.76:0/1592314066 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff06c004c80 con 0x7ff070100290 2024-03-12T06:06:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff077640640 1 -- 172.21.15.76:0/1592314066 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0701a4380 con 0x7ff070100290 2024-03-12T06:06:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff0667fc640 1 -- 172.21.15.76:0/1592314066 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c00f780 con 0x7ff070100290 2024-03-12T06:06:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.487+0000 7ff077640640 1 -- 172.21.15.76:0/1592314066 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0701a4860 con 0x7ff070100290 2024-03-12T06:06:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.488+0000 7ff077640640 1 -- 172.21.15.76:0/1592314066 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff070068000 con 0x7ff070100290 2024-03-12T06:06:33.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.489+0000 7ff0667fc640 1 -- 172.21.15.76:0/1592314066 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7ff06c00f8e0 con 0x7ff070100290 2024-03-12T06:06:33.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.490+0000 7ff0667fc640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff05c0a1ba0 0x7ff05c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:33.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.490+0000 7ff074bdb640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff05c0a1ba0 0x7ff05c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:33.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.490+0000 7ff0667fc640 1 -- 172.21.15.76:0/1592314066 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7ff06c0cd950 con 0x7ff070100290 2024-03-12T06:06:33.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.490+0000 7ff074bdb640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff05c0a1ba0 0x7ff05c0a4060 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff060009900 tx=0x7ff060005fd0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:33.500 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.494+0000 7ff0667fc640 1 -- 172.21.15.76:0/1592314066 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff06c096820 con 0x7ff070100290 2024-03-12T06:06:33.673 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.668+0000 7ff077640640 1 -- 172.21.15.76:0/1592314066 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff070108590 con 0x7ff070100290 2024-03-12T06:06:33.674 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.669+0000 7ff0667fc640 1 -- 172.21.15.76:0/1592314066 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff06c099ee0 con 0x7ff070100290 2024-03-12T06:06:33.674 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2024-03-12T06:06:33.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.671+0000 7ff077640640 1 -- 172.21.15.76:0/1592314066 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff05c0a1ba0 msgr2=0x7ff05c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.671+0000 7ff077640640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff05c0a1ba0 0x7ff05c0a4060 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff060009900 tx=0x7ff060005fd0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.671+0000 7ff077640640 1 -- 172.21.15.76:0/1592314066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff070100290 msgr2=0x7ff070198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.671+0000 7ff077640640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff070100290 0x7ff070198620 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00cce0 tx=0x7ff06c00a620 comp rx=0 tx=0).stop 2024-03-12T06:06:33.676 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.671+0000 7ff077640640 1 -- 172.21.15.76:0/1592314066 shutdown_connections 2024-03-12T06:06:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.671+0000 7ff077640640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7ff05c0a1ba0 0x7ff05c0a4060 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.671+0000 7ff077640640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0701990f0 0x7ff0701a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.671+0000 7ff077640640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff070105360 0x7ff070198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.671+0000 7ff077640640 1 --2- 172.21.15.76:0/1592314066 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff070100290 0x7ff070198620 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.671+0000 7ff077640640 1 -- 172.21.15.76:0/1592314066 >> 172.21.15.76:0/1592314066 conn(0x7ff070077230 msgr2=0x7ff0700fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.671+0000 7ff077640640 1 -- 172.21.15.76:0/1592314066 shutdown_connections 2024-03-12T06:06:33.677 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.671+0000 7ff077640640 1 -- 172.21.15.76:0/1592314066 wait complete. 2024-03-12T06:06:33.687 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-12T06:06:33.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.787+0000 7f334fc1c640 1 -- 172.21.15.76:0/2239214830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3348105370 msgr2=0x7f334810cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.794 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.787+0000 7f334fc1c640 1 --2- 172.21.15.76:0/2239214830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3348105370 0x7f334810cf40 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f3340009cd0 tx=0x7f334002c0f0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.787+0000 7f334fc1c640 1 -- 172.21.15.76:0/2239214830 shutdown_connections 2024-03-12T06:06:33.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.787+0000 7f334fc1c640 1 --2- 172.21.15.76:0/2239214830 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3348105370 0x7f334810cf40 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.787+0000 7f334fc1c640 1 --2- 172.21.15.76:0/2239214830 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3348100ca0 0x7f3348104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.787+0000 7f334fc1c640 1 --2- 172.21.15.76:0/2239214830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33481002f0 0x7f33481006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.787+0000 7f334fc1c640 1 -- 172.21.15.76:0/2239214830 >> 172.21.15.76:0/2239214830 conn(0x7f3348077250 msgr2=0x7f3348077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:33.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.787+0000 7f334fc1c640 1 -- 172.21.15.76:0/2239214830 shutdown_connections 2024-03-12T06:06:33.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.787+0000 7f334fc1c640 1 -- 172.21.15.76:0/2239214830 wait complete. 2024-03-12T06:06:33.795 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.787+0000 7f334fc1c640 1 Processor -- start 2024-03-12T06:06:33.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.787+0000 7f334fc1c640 1 -- start start 2024-03-12T06:06:33.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334fc1c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33481002f0 0x7f33481985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:33.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334fc1c640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3348105370 0x7f3348198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:33.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334fc1c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3348199070 0x7f33481a3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:33.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334fc1c640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33481a3cf0 con 0x7f3348199070 2024-03-12T06:06:33.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334fc1c640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33481a3e60 con 0x7f3348105370 2024-03-12T06:06:33.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334fc1c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33481a4020 con 0x7f33481002f0 2024-03-12T06:06:33.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334e1b9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3348199070 0x7f33481a3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:33.796 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334e1b9640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3348199070 0x7f33481a3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47078/0 (socket says 172.21.15.76:47078) 2024-03-12T06:06:33.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334e1b9640 1 -- 172.21.15.76:0/1234450727 learned_addr learned my addr 172.21.15.76:0/1234450727 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:06:33.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334d9b8640 1 --2- 172.21.15.76:0/1234450727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33481002f0 0x7f33481985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:33.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334d1b7640 1 --2- 172.21.15.76:0/1234450727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3348105370 0x7f3348198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:33.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334e1b9640 1 -- 172.21.15.76:0/1234450727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33481002f0 msgr2=0x7f33481985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334e1b9640 1 --2- 172.21.15.76:0/1234450727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33481002f0 0x7f33481985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334e1b9640 1 -- 172.21.15.76:0/1234450727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3348105370 msgr2=0x7f3348198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334e1b9640 1 --2- 172.21.15.76:0/1234450727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3348105370 0x7f3348198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.797 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334e1b9640 1 -- 172.21.15.76:0/1234450727 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3340009900 con 0x7f3348199070 2024-03-12T06:06:33.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334e1b9640 1 --2- 172.21.15.76:0/1234450727 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3348199070 0x7f33481a3620 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f3340004af0 tx=0x7f3340032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:33.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f333effd640 1 -- 172.21.15.76:0/1234450727 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33400328b0 con 0x7f3348199070 2024-03-12T06:06:33.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f333effd640 1 -- 172.21.15.76:0/1234450727 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3340003e50 con 0x7f3348199070 2024-03-12T06:06:33.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f333effd640 1 -- 172.21.15.76:0/1234450727 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3340004740 con 0x7f3348199070 2024-03-12T06:06:33.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.788+0000 7f334fc1c640 1 -- 172.21.15.76:0/1234450727 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33481a42a0 con 0x7f3348199070 2024-03-12T06:06:33.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.789+0000 7f334fc1c640 1 -- 172.21.15.76:0/1234450727 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33481a47a0 con 0x7f3348199070 2024-03-12T06:06:33.798 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.789+0000 7f334fc1c640 1 -- 172.21.15.76:0/1234450727 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3348068450 con 0x7f3348199070 2024-03-12T06:06:33.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.791+0000 7f333effd640 1 -- 172.21.15.76:0/1234450727 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f334000a040 con 0x7f3348199070 2024-03-12T06:06:33.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.791+0000 7f333effd640 1 --2- 172.21.15.76:0/1234450727 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f33340a1c20 0x7f33340a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:33.799 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.791+0000 7f334d9b8640 1 --2- 172.21.15.76:0/1234450727 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f33340a1c20 0x7f33340a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:33.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.791+0000 7f333effd640 1 -- 172.21.15.76:0/1234450727 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f33400f0620 con 0x7f3348199070 2024-03-12T06:06:33.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.792+0000 7f334d9b8640 1 --2- 172.21.15.76:0/1234450727 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f33340a1c20 0x7f33340a40e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3344005fd0 tx=0x7f334400b390 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:33.800 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.795+0000 7f333effd640 1 -- 172.21.15.76:0/1234450727 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f33400b9680 con 0x7f3348199070 2024-03-12T06:06:33.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.967+0000 7f334fc1c640 1 -- 172.21.15.76:0/1234450727 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f33481085a0 con 0x7f3348199070 2024-03-12T06:06:33.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.967+0000 7f333effd640 1 -- 172.21.15.76:0/1234450727 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3340044020 con 0x7f3348199070 2024-03-12T06:06:33.972 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 64 2024-03-12T06:06:33.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.969+0000 7f334fc1c640 1 -- 172.21.15.76:0/1234450727 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f33340a1c20 msgr2=0x7f33340a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.969+0000 7f334fc1c640 1 --2- 172.21.15.76:0/1234450727 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f33340a1c20 0x7f33340a40e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3344005fd0 tx=0x7f334400b390 comp rx=0 tx=0).stop 2024-03-12T06:06:33.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.969+0000 7f334fc1c640 1 -- 172.21.15.76:0/1234450727 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3348199070 msgr2=0x7f33481a3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:33.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.969+0000 7f334fc1c640 1 --2- 172.21.15.76:0/1234450727 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3348199070 0x7f33481a3620 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f3340004af0 tx=0x7f3340032570 comp rx=0 tx=0).stop 2024-03-12T06:06:33.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.969+0000 7f334fc1c640 1 -- 172.21.15.76:0/1234450727 shutdown_connections 2024-03-12T06:06:33.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.969+0000 7f334fc1c640 1 --2- 172.21.15.76:0/1234450727 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f33340a1c20 0x7f33340a40e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.969+0000 7f334fc1c640 1 --2- 172.21.15.76:0/1234450727 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3348199070 0x7f33481a3620 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.969+0000 7f334fc1c640 1 --2- 172.21.15.76:0/1234450727 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3348105370 0x7f3348198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.969+0000 7f334fc1c640 1 --2- 172.21.15.76:0/1234450727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33481002f0 0x7f33481985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:33.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.969+0000 7f334fc1c640 1 -- 172.21.15.76:0/1234450727 >> 172.21.15.76:0/1234450727 conn(0x7f3348077250 msgr2=0x7f33480fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:33.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.969+0000 7f334fc1c640 1 -- 172.21.15.76:0/1234450727 shutdown_connections 2024-03-12T06:06:33.976 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:33.969+0000 7f334fc1c640 1 -- 172.21.15.76:0/1234450727 wait complete. 2024-03-12T06:06:33.985 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-12T06:06:34.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.083+0000 7fd97b319640 1 -- 172.21.15.76:0/1042140386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9740ff5f0 msgr2=0x7fd9740ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.083+0000 7fd97b319640 1 --2- 172.21.15.76:0/1042140386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9740ff5f0 0x7fd9740ffa70 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fd970009930 tx=0x7fd97002bc30 comp rx=0 tx=0).stop 2024-03-12T06:06:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.083+0000 7fd97b319640 1 -- 172.21.15.76:0/1042140386 shutdown_connections 2024-03-12T06:06:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.083+0000 7fd97b319640 1 --2- 172.21.15.76:0/1042140386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9740fffb0 0x7fd97410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.083+0000 7fd97b319640 1 --2- 172.21.15.76:0/1042140386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd9740ff5f0 0x7fd9740ffa70 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.083+0000 7fd97b319640 1 --2- 172.21.15.76:0/1042140386 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd974106150 0x7fd974106530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.083+0000 7fd97b319640 1 -- 172.21.15.76:0/1042140386 >> 172.21.15.76:0/1042140386 conn(0x7fd974076580 msgr2=0x7fd9740fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.083+0000 7fd97b319640 1 -- 172.21.15.76:0/1042140386 shutdown_connections 2024-03-12T06:06:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.083+0000 7fd97b319640 1 -- 172.21.15.76:0/1042140386 wait complete. 2024-03-12T06:06:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd97b319640 1 Processor -- start 2024-03-12T06:06:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd97b319640 1 -- start start 2024-03-12T06:06:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd97b319640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd9740ff5f0 0x7fd97419d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd97b319640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9740fffb0 0x7fd97419dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd97b319640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd974106150 0x7fd9741976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd97b319640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd974197e40 con 0x7fd974106150 2024-03-12T06:06:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd97b319640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd974197fb0 con 0x7fd9740ff5f0 2024-03-12T06:06:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd97b319640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd974198170 con 0x7fd9740fffb0 2024-03-12T06:06:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd9798b6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd974106150 0x7fd9741976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd9798b6640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd974106150 0x7fd9741976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47092/0 (socket says 172.21.15.76:47092) 2024-03-12T06:06:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd9798b6640 1 -- 172.21.15.76:0/3344731102 learned_addr learned my addr 172.21.15.76:0/3344731102 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:06:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd9790b5640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd9740ff5f0 0x7fd97419d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd9788b4640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9740fffb0 0x7fd97419dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd9798b6640 1 -- 172.21.15.76:0/3344731102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9740fffb0 msgr2=0x7fd97419dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd9798b6640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9740fffb0 0x7fd97419dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd9798b6640 1 -- 172.21.15.76:0/3344731102 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd9740ff5f0 msgr2=0x7fd97419d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:34.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd9798b6640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd9740ff5f0 0x7fd97419d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd9798b6640 1 -- 172.21.15.76:0/3344731102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd970009560 con 0x7fd974106150 2024-03-12T06:06:34.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd9790b5640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd9740ff5f0 0x7fd97419d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:06:34.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd9798b6640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd974106150 0x7fd9741976b0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fd96c00eaa0 tx=0x7fd96c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:34.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd96a7fc640 1 -- 172.21.15.76:0/3344731102 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c00cdb0 con 0x7fd974106150 2024-03-12T06:06:34.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.084+0000 7fd96a7fc640 1 -- 172.21.15.76:0/3344731102 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd96c005020 con 0x7fd974106150 2024-03-12T06:06:34.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.085+0000 7fd97b319640 1 -- 172.21.15.76:0/3344731102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd974198450 con 0x7fd974106150 2024-03-12T06:06:34.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.085+0000 7fd96a7fc640 1 -- 172.21.15.76:0/3344731102 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c0116b0 con 0x7fd974106150 2024-03-12T06:06:34.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.085+0000 7fd97b319640 1 -- 172.21.15.76:0/3344731102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9741a4520 con 0x7fd974106150 2024-03-12T06:06:34.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.085+0000 7fd97b319640 1 -- 172.21.15.76:0/3344731102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd974102910 con 0x7fd974106150 2024-03-12T06:06:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.087+0000 7fd96a7fc640 1 -- 172.21.15.76:0/3344731102 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fd96c005520 con 0x7fd974106150 2024-03-12T06:06:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.087+0000 7fd96a7fc640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd9600a1c20 0x7fd9600a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.088+0000 7fd9790b5640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd9600a1c20 0x7fd9600a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.088+0000 7fd96a7fc640 1 -- 172.21.15.76:0/3344731102 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fd96c0cddf0 con 0x7fd974106150 2024-03-12T06:06:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.088+0000 7fd9790b5640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd9600a1c20 0x7fd9600a40e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd964005fd0 tx=0x7fd96400b5f0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:34.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.091+0000 7fd96a7fc640 1 -- 172.21.15.76:0/3344731102 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd96c096d40 con 0x7fd974106150 2024-03-12T06:06:34.126 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:34 smithi082 ceph-mon[36382]: pgmap v334: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:06:34.126 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:34 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1503549606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T06:06:34.127 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:34 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1592314066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T06:06:34.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:34 smithi076 ceph-mon[32095]: pgmap v334: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:06:34.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:34 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1503549606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T06:06:34.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:34 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1592314066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T06:06:34.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:34 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1234450727' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T06:06:34.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.280+0000 7fd97b319640 1 -- 172.21.15.76:0/3344731102 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd974198b90 con 0x7fd974106150 2024-03-12T06:06:34.285 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.281+0000 7fd96a7fc640 1 -- 172.21.15.76:0/3344731102 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v97) v1 ==== 81+0+942 (secure 0 0 0) 0x7fd96c01f070 con 0x7fd974106150 2024-03-12T06:06:34.286 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6800/1718825251 2024-03-13T05:55:46.980746+0000 2024-03-12T06:06:34.286 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/189327369 2024-03-13T05:55:46.980746+0000 2024-03-12T06:06:34.286 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/3998885619 2024-03-13T05:55:46.980746+0000 2024-03-12T06:06:34.286 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6801/306877437 2024-03-13T05:46:40.781750+0000 2024-03-12T06:06:34.286 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/2299856451 2024-03-13T05:55:46.980746+0000 2024-03-12T06:06:34.287 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6801/1718825251 2024-03-13T05:55:46.980746+0000 2024-03-12T06:06:34.287 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6800/306877437 2024-03-13T05:46:40.781750+0000 2024-03-12T06:06:34.287 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/1321850785 2024-03-13T05:46:40.781750+0000 2024-03-12T06:06:34.287 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/241684208 2024-03-13T05:46:40.781750+0000 2024-03-12T06:06:34.287 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6800/4265639592 2024-03-13T05:46:12.898913+0000 2024-03-12T06:06:34.287 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/1776556257 2024-03-13T05:46:12.898913+0000 2024-03-12T06:06:34.287 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/1939795424 2024-03-13T05:55:46.980746+0000 2024-03-12T06:06:34.287 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:6801/4265639592 2024-03-13T05:46:12.898913+0000 2024-03-12T06:06:34.287 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/3181569618 2024-03-13T05:46:12.898913+0000 2024-03-12T06:06:34.288 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/2728891083 2024-03-13T05:46:12.898913+0000 2024-03-12T06:06:34.288 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76:0/3468109661 2024-03-13T05:46:40.781750+0000 2024-03-12T06:06:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:listed 16 entries 2024-03-12T06:06:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.282+0000 7fd97b319640 1 -- 172.21.15.76:0/3344731102 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd9600a1c20 msgr2=0x7fd9600a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.282+0000 7fd97b319640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd9600a1c20 0x7fd9600a40e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd964005fd0 tx=0x7fd96400b5f0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.282+0000 7fd97b319640 1 -- 172.21.15.76:0/3344731102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd974106150 msgr2=0x7fd9741976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.282+0000 7fd97b319640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd974106150 0x7fd9741976b0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fd96c00eaa0 tx=0x7fd96c00ef60 comp rx=0 tx=0).stop 2024-03-12T06:06:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.283+0000 7fd97b319640 1 -- 172.21.15.76:0/3344731102 shutdown_connections 2024-03-12T06:06:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.283+0000 7fd97b319640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fd9600a1c20 0x7fd9600a40e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.283+0000 7fd97b319640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd974106150 0x7fd9741976b0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.283+0000 7fd97b319640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9740fffb0 0x7fd97419dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.283+0000 7fd97b319640 1 --2- 172.21.15.76:0/3344731102 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd9740ff5f0 0x7fd97419d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.283+0000 7fd97b319640 1 -- 172.21.15.76:0/3344731102 >> 172.21.15.76:0/3344731102 conn(0x7fd974076580 msgr2=0x7fd9741041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:34.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.283+0000 7fd97b319640 1 -- 172.21.15.76:0/3344731102 shutdown_connections 2024-03-12T06:06:34.290 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.283+0000 7fd97b319640 1 -- 172.21.15.76:0/3344731102 wait complete. 2024-03-12T06:06:34.298 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-12T06:06:34.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:06:34.299 DEBUG:teuthology.orchestra.run.smithi076:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-12T06:06:34.386 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi076.front.sepia.ceph.com 2024-03-12T06:06:34.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:06:34.387 DEBUG:teuthology.orchestra.run.smithi076:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-12T06:06:34.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:06:34.412 DEBUG:teuthology.orchestra.run.smithi076:> ip netns list 2024-03-12T06:06:34.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:34 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1234450727' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T06:06:34.467 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-12T06:06:34.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:06:34.468 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-12T06:06:34.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:34 smithi088 ceph-mon[36585]: pgmap v334: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-12T06:06:34.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:34 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1503549606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T06:06:34.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:34 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1592314066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-12T06:06:34.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:34 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1234450727' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-12T06:06:34.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T06:06:34.536 DEBUG:teuthology.orchestra.run.smithi076:> sudo ip link delete ceph-brx 2024-03-12T06:06:34.650 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-12T06:06:34.661 INFO:tasks.cephadm:Teardown begin 2024-03-12T06:06:34.661 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-12T06:06:34.662 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T06:06:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.759+0000 7f076a80f640 1 -- 172.21.15.76:0/2328321127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07641002a0 msgr2=0x7f0764100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.759+0000 7f076a80f640 1 --2- 172.21.15.76:0/2328321127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07641002a0 0x7f0764100680 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f074c009930 tx=0x7f074c02bc30 comp rx=0 tx=0).stop 2024-03-12T06:06:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.759+0000 7f076a80f640 1 -- 172.21.15.76:0/2328321127 shutdown_connections 2024-03-12T06:06:34.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.759+0000 7f076a80f640 1 --2- 172.21.15.76:0/2328321127 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0764105370 0x7f076410cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.759+0000 7f076a80f640 1 --2- 172.21.15.76:0/2328321127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0764100c50 0x7f0764104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.759+0000 7f076a80f640 1 --2- 172.21.15.76:0/2328321127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07641002a0 0x7f0764100680 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.760+0000 7f076a80f640 1 -- 172.21.15.76:0/2328321127 >> 172.21.15.76:0/2328321127 conn(0x7f0764077200 msgr2=0x7f0764077610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.760+0000 7f076a80f640 1 -- 172.21.15.76:0/2328321127 shutdown_connections 2024-03-12T06:06:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.760+0000 7f076a80f640 1 -- 172.21.15.76:0/2328321127 wait complete. 2024-03-12T06:06:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.760+0000 7f076a80f640 1 Processor -- start 2024-03-12T06:06:34.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.760+0000 7f076a80f640 1 -- start start 2024-03-12T06:06:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.760+0000 7f076a80f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07641002a0 0x7f076419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.760+0000 7f076a80f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0764105370 0x7f076419d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.760+0000 7f076a80f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0764197600 0x7f0764197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.760+0000 7f076a80f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0764197ff0 con 0x7f07641002a0 2024-03-12T06:06:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.760+0000 7f076a80f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0764198160 con 0x7f0764197600 2024-03-12T06:06:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.760+0000 7f076a80f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0764198320 con 0x7f0764105370 2024-03-12T06:06:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f0763fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07641002a0 0x7f076419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f0763fff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07641002a0 0x7f076419d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47100/0 (socket says 172.21.15.76:47100) 2024-03-12T06:06:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f0763fff640 1 -- 172.21.15.76:0/2160922705 learned_addr learned my addr 172.21.15.76:0/2160922705 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:06:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f07637fe640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0764105370 0x7f076419d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f0768dac640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0764197600 0x7f0764197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f0763fff640 1 -- 172.21.15.76:0/2160922705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0764105370 msgr2=0x7f076419d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f0763fff640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0764105370 0x7f076419d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.770 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f0763fff640 1 -- 172.21.15.76:0/2160922705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0764197600 msgr2=0x7f0764197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:34.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f0763fff640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0764197600 0x7f0764197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f0763fff640 1 -- 172.21.15.76:0/2160922705 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c009560 con 0x7f07641002a0 2024-03-12T06:06:34.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f0768dac640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0764197600 0x7f0764197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:06:34.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f0763fff640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07641002a0 0x7f076419d450 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f074c009230 tx=0x7f074c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:34.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f07617fa640 1 -- 172.21.15.76:0/2160922705 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c004440 con 0x7f07641002a0 2024-03-12T06:06:34.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f07617fa640 1 -- 172.21.15.76:0/2160922705 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f074c0052d0 con 0x7f07641002a0 2024-03-12T06:06:34.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f076a80f640 1 -- 172.21.15.76:0/2160922705 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07641a4310 con 0x7f07641002a0 2024-03-12T06:06:34.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f07617fa640 1 -- 172.21.15.76:0/2160922705 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c003d00 con 0x7f07641002a0 2024-03-12T06:06:34.771 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.761+0000 7f076a80f640 1 -- 172.21.15.76:0/2160922705 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07641a47c0 con 0x7f07641002a0 2024-03-12T06:06:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.762+0000 7f076a80f640 1 -- 172.21.15.76:0/2160922705 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0764068000 con 0x7f07641002a0 2024-03-12T06:06:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.763+0000 7f07617fa640 1 -- 172.21.15.76:0/2160922705 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f074c005440 con 0x7f07641002a0 2024-03-12T06:06:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.764+0000 7f07617fa640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f07500a1ba0 0x7f07500a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.764+0000 7f07637fe640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f07500a1ba0 0x7f07500a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.764+0000 7f07617fa640 1 -- 172.21.15.76:0/2160922705 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f074c0ef890 con 0x7f07641002a0 2024-03-12T06:06:34.772 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.764+0000 7f07637fe640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f07500a1ba0 0x7f07500a4060 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f07640fea00 tx=0x7f075400a2d0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:34.773 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.768+0000 7f07617fa640 1 -- 172.21.15.76:0/2160922705 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f074c0b8870 con 0x7f07641002a0 2024-03-12T06:06:34.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.978+0000 7f076a80f640 1 -- 172.21.15.76:0/2160922705 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0764100ce0 con 0x7f07641002a0 2024-03-12T06:06:34.984 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.979+0000 7f07617fa640 1 -- 172.21.15.76:0/2160922705 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11328 (secure 0 0 0) 0x7f074c003e60 con 0x7f07641002a0 2024-03-12T06:06:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:06:34.985 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T06:01:33.842001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T06:06:34.985 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 28 2024-03-12T06:06:34.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.981+0000 7f076a80f640 1 -- 172.21.15.76:0/2160922705 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f07500a1ba0 msgr2=0x7f07500a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:34.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.981+0000 7f076a80f640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f07500a1ba0 0x7f07500a4060 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f07640fea00 tx=0x7f075400a2d0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.981+0000 7f076a80f640 1 -- 172.21.15.76:0/2160922705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07641002a0 msgr2=0x7f076419d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.981+0000 7f076a80f640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07641002a0 0x7f076419d450 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f074c009230 tx=0x7f074c004040 comp rx=0 tx=0).stop 2024-03-12T06:06:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.981+0000 7f076a80f640 1 -- 172.21.15.76:0/2160922705 shutdown_connections 2024-03-12T06:06:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.981+0000 7f076a80f640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f07500a1ba0 0x7f07500a4060 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.981+0000 7f076a80f640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0764197600 0x7f0764197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.981+0000 7f076a80f640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0764105370 0x7f076419d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.981+0000 7f076a80f640 1 --2- 172.21.15.76:0/2160922705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07641002a0 0x7f076419d450 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.981+0000 7f076a80f640 1 -- 172.21.15.76:0/2160922705 >> 172.21.15.76:0/2160922705 conn(0x7f0764077200 msgr2=0x7f0764106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:34.987 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.981+0000 7f076a80f640 1 -- 172.21.15.76:0/2160922705 shutdown_connections 2024-03-12T06:06:34.988 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:34.981+0000 7f076a80f640 1 -- 172.21.15.76:0/2160922705 wait complete. 2024-03-12T06:06:34.997 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T06:06:35.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.095+0000 7fdd31447640 1 -- 172.21.15.76:0/395326702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd2c1064f0 msgr2=0x7fdd2c10e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.095+0000 7fdd31447640 1 --2- 172.21.15.76:0/395326702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd2c1064f0 0x7fdd2c10e070 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009cd0 tx=0x7fdd2402c0f0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.095+0000 7fdd31447640 1 -- 172.21.15.76:0/395326702 shutdown_connections 2024-03-12T06:06:35.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.095+0000 7fdd31447640 1 --2- 172.21.15.76:0/395326702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd2c1064f0 0x7fdd2c10e070 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.095+0000 7fdd31447640 1 --2- 172.21.15.76:0/395326702 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdd2c100230 0x7fdd2c105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.095+0000 7fdd31447640 1 --2- 172.21.15.76:0/395326702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd2c0ff880 0x7fdd2c0ffc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.095+0000 7fdd31447640 1 -- 172.21.15.76:0/395326702 >> 172.21.15.76:0/395326702 conn(0x7fdd2c076590 msgr2=0x7fdd2c0fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:35.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.095+0000 7fdd31447640 1 -- 172.21.15.76:0/395326702 shutdown_connections 2024-03-12T06:06:35.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.095+0000 7fdd31447640 1 -- 172.21.15.76:0/395326702 wait complete. 2024-03-12T06:06:35.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.095+0000 7fdd31447640 1 Processor -- start 2024-03-12T06:06:35.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.095+0000 7fdd31447640 1 -- start start 2024-03-12T06:06:35.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd31447640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdd2c0ff880 0x7fdd2c19d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:35.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd31447640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd2c1064f0 0x7fdd2c19d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:35.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd31447640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd2c197670 0x7fdd2c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:35.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd31447640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c198060 con 0x7fdd2c1064f0 2024-03-12T06:06:35.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd31447640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c1981d0 con 0x7fdd2c0ff880 2024-03-12T06:06:35.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd31447640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c198390 con 0x7fdd2c197670 2024-03-12T06:06:35.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd2a7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd2c1064f0 0x7fdd2c19d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:35.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd2a7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd2c1064f0 0x7fdd2c19d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47112/0 (socket says 172.21.15.76:47112) 2024-03-12T06:06:35.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd2a7fc640 1 -- 172.21.15.76:0/2751795358 learned_addr learned my addr 172.21.15.76:0/2751795358 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:06:35.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd2b7fe640 1 --2- 172.21.15.76:0/2751795358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd2c197670 0x7fdd2c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:35.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd2affd640 1 --2- 172.21.15.76:0/2751795358 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdd2c0ff880 0x7fdd2c19d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:35.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd2a7fc640 1 -- 172.21.15.76:0/2751795358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd2c197670 msgr2=0x7fdd2c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd2a7fc640 1 --2- 172.21.15.76:0/2751795358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd2c197670 0x7fdd2c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd2a7fc640 1 -- 172.21.15.76:0/2751795358 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdd2c0ff880 msgr2=0x7fdd2c19d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd2a7fc640 1 --2- 172.21.15.76:0/2751795358 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdd2c0ff880 0x7fdd2c19d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd2a7fc640 1 -- 172.21.15.76:0/2751795358 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd24009900 con 0x7fdd2c1064f0 2024-03-12T06:06:35.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd2a7fc640 1 --2- 172.21.15.76:0/2751795358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd2c1064f0 0x7fdd2c19d9b0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000d750 tx=0x7fdd2000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:35.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd13fff640 1 -- 172.21.15.76:0/2751795358 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd20009c30 con 0x7fdd2c1064f0 2024-03-12T06:06:35.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd31447640 1 -- 172.21.15.76:0/2751795358 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2c1a4330 con 0x7fdd2c1064f0 2024-03-12T06:06:35.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd13fff640 1 -- 172.21.15.76:0/2751795358 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd2000de60 con 0x7fdd2c1064f0 2024-03-12T06:06:35.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.096+0000 7fdd13fff640 1 -- 172.21.15.76:0/2751795358 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd20004e70 con 0x7fdd2c1064f0 2024-03-12T06:06:35.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.097+0000 7fdd31447640 1 -- 172.21.15.76:0/2751795358 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2c1a47f0 con 0x7fdd2c1064f0 2024-03-12T06:06:35.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.098+0000 7fdd31447640 1 -- 172.21.15.76:0/2751795358 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2c068450 con 0x7fdd2c1064f0 2024-03-12T06:06:35.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.099+0000 7fdd13fff640 1 -- 172.21.15.76:0/2751795358 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fdd20005020 con 0x7fdd2c1064f0 2024-03-12T06:06:35.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.099+0000 7fdd13fff640 1 --2- 172.21.15.76:0/2751795358 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdd140a1ba0 0x7fdd140a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:35.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.099+0000 7fdd2affd640 1 --2- 172.21.15.76:0/2751795358 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdd140a1ba0 0x7fdd140a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:35.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.099+0000 7fdd13fff640 1 -- 172.21.15.76:0/2751795358 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fdd200cd060 con 0x7fdd2c1064f0 2024-03-12T06:06:35.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.100+0000 7fdd2affd640 1 --2- 172.21.15.76:0/2751795358 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdd140a1ba0 0x7fdd140a4060 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c009bd0 tx=0x7fdd1c005f90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:35.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.103+0000 7fdd13fff640 1 -- 172.21.15.76:0/2751795358 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdd20099b10 con 0x7fdd2c1064f0 2024-03-12T06:06:35.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.314+0000 7fdd31447640 1 -- 172.21.15.76:0/2751795358 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd2c198b50 con 0x7fdd2c1064f0 2024-03-12T06:06:35.319 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.314+0000 7fdd13fff640 1 -- 172.21.15.76:0/2751795358 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11328 (secure 0 0 0) 0x7fdd20020090 con 0x7fdd2c1064f0 2024-03-12T06:06:35.320 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:06:35.320 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14568,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4172787847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4172787847},{"type":"v1","addr":"172.21.15.76:6835","nonce":4172787847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2194036306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2194036306},{"type":"v1","addr":"172.21.15.76:6837","nonce":2194036306}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14652,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/1304487620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":1304487620},{"type":"v1","addr":"172.21.15.76:6839","nonce":1304487620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14670,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6841/291567235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6840","nonce":291567235},{"type":"v1","addr":"172.21.15.76:6841","nonce":291567235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/3297889286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3297889286},{"type":"v1","addr":"172.21.15.88:6833","nonce":3297889286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/2359119459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":2359119459},{"type":"v1","addr":"172.21.15.82:6833","nonce":2359119459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T05:53:37.535927+0000","modified":"2024-03-12T06:01:33.842001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24518,"mds_1":24481,"mds_2":24451},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/2138041661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":2138041661},{"type":"v1","addr":"172.21.15.82:6835","nonce":2138041661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":19,"state":"up:active","state_seq":8,"addr":"172.21.15.88:6835/3940245802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3940245802},{"type":"v1","addr":"172.21.15.88:6835","nonce":3940245802}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6837/1311843051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1311843051},{"type":"v1","addr":"172.21.15.88:6837","nonce":1311843051}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/2523422510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":2523422510},{"type":"v1","addr":"172.21.15.82:6837","nonce":2523422510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.88:6839/1164228359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1164228359},{"type":"v1","addr":"172.21.15.88:6839","nonce":1164228359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24518":{"gid":24518,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.82:6839/2242595718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2242595718},{"type":"v1","addr":"172.21.15.82:6839","nonce":2242595718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-12T06:06:35.320 INFO:teuthology.orchestra.run.smithi076.stderr:dumped fsmap epoch 28 2024-03-12T06:06:35.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.316+0000 7fdd31447640 1 -- 172.21.15.76:0/2751795358 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdd140a1ba0 msgr2=0x7fdd140a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.321 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.316+0000 7fdd31447640 1 --2- 172.21.15.76:0/2751795358 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdd140a1ba0 0x7fdd140a4060 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c009bd0 tx=0x7fdd1c005f90 comp rx=0 tx=0).stop 2024-03-12T06:06:35.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.316+0000 7fdd31447640 1 -- 172.21.15.76:0/2751795358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd2c1064f0 msgr2=0x7fdd2c19d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.316+0000 7fdd31447640 1 --2- 172.21.15.76:0/2751795358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd2c1064f0 0x7fdd2c19d9b0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000d750 tx=0x7fdd2000dc10 comp rx=0 tx=0).stop 2024-03-12T06:06:35.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.316+0000 7fdd31447640 1 -- 172.21.15.76:0/2751795358 shutdown_connections 2024-03-12T06:06:35.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.316+0000 7fdd31447640 1 --2- 172.21.15.76:0/2751795358 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fdd140a1ba0 0x7fdd140a4060 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.316+0000 7fdd31447640 1 --2- 172.21.15.76:0/2751795358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd2c197670 0x7fdd2c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.316+0000 7fdd31447640 1 --2- 172.21.15.76:0/2751795358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdd2c1064f0 0x7fdd2c19d9b0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.322 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.316+0000 7fdd31447640 1 --2- 172.21.15.76:0/2751795358 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fdd2c0ff880 0x7fdd2c19d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.316+0000 7fdd31447640 1 -- 172.21.15.76:0/2751795358 >> 172.21.15.76:0/2751795358 conn(0x7fdd2c076590 msgr2=0x7fdd2c1045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:35.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.316+0000 7fdd31447640 1 -- 172.21.15.76:0/2751795358 shutdown_connections 2024-03-12T06:06:35.323 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.316+0000 7fdd31447640 1 -- 172.21.15.76:0/2751795358 wait complete. 2024-03-12T06:06:35.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:35 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3344731102' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T06:06:35.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:35 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2160922705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:06:35.333 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T06:06:35.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.430+0000 7fba7bfff640 1 -- 172.21.15.76:0/3236684874 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba7c100060 msgr2=0x7fba7c10c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.430+0000 7fba7bfff640 1 --2- 172.21.15.76:0/3236684874 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba7c100060 0x7fba7c10c570 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fba6c00ade0 tx=0x7fba6c02cfd0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.435 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.430+0000 7fba7bfff640 1 -- 172.21.15.76:0/3236684874 shutdown_connections 2024-03-12T06:06:35.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.430+0000 7fba7bfff640 1 --2- 172.21.15.76:0/3236684874 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba7c100060 0x7fba7c10c570 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.430+0000 7fba7bfff640 1 --2- 172.21.15.76:0/3236684874 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba7c0ff6a0 0x7fba7c0ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.430+0000 7fba7bfff640 1 --2- 172.21.15.76:0/3236684874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba7c1056b0 0x7fba7c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.430+0000 7fba7bfff640 1 -- 172.21.15.76:0/3236684874 >> 172.21.15.76:0/3236684874 conn(0x7fba7c076460 msgr2=0x7fba7c0fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:35.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.430+0000 7fba7bfff640 1 -- 172.21.15.76:0/3236684874 shutdown_connections 2024-03-12T06:06:35.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.430+0000 7fba7bfff640 1 -- 172.21.15.76:0/3236684874 wait complete. 2024-03-12T06:06:35.436 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.430+0000 7fba7bfff640 1 Processor -- start 2024-03-12T06:06:35.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7bfff640 1 -- start start 2024-03-12T06:06:35.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba7c0ff6a0 0x7fba7c1954a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:35.438 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba7c100060 0x7fba7c1959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:35.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba7c1056b0 0x7fba7c199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:35.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7bfff640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c19a310 con 0x7fba7c1056b0 2024-03-12T06:06:35.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7bfff640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c19a480 con 0x7fba7c100060 2024-03-12T06:06:35.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7bfff640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c19a640 con 0x7fba7c0ff6a0 2024-03-12T06:06:35.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba7c1056b0 0x7fba7c199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:35.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7b7fe640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba7c1056b0 0x7fba7c199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47134/0 (socket says 172.21.15.76:47134) 2024-03-12T06:06:35.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7b7fe640 1 -- 172.21.15.76:0/1087716155 learned_addr learned my addr 172.21.15.76:0/1087716155 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:06:35.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7a7fc640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba7c100060 0x7fba7c1959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:35.439 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7affd640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba7c0ff6a0 0x7fba7c1954a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:35.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7b7fe640 1 -- 172.21.15.76:0/1087716155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba7c0ff6a0 msgr2=0x7fba7c1954a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7b7fe640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba7c0ff6a0 0x7fba7c1954a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7b7fe640 1 -- 172.21.15.76:0/1087716155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba7c100060 msgr2=0x7fba7c1959e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7b7fe640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba7c100060 0x7fba7c1959e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7b7fe640 1 -- 172.21.15.76:0/1087716155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c00aa10 con 0x7fba7c1056b0 2024-03-12T06:06:35.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.431+0000 7fba7a7fc640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba7c100060 0x7fba7c1959e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:06:35.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.432+0000 7fba7b7fe640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba7c1056b0 0x7fba7c199d10 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fba6c00adb0 tx=0x7fba6c035670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:35.440 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.432+0000 7fba80a00640 1 -- 172.21.15.76:0/1087716155 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c033590 con 0x7fba7c1056b0 2024-03-12T06:06:35.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.432+0000 7fba7bfff640 1 -- 172.21.15.76:0/1087716155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7c19a8c0 con 0x7fba7c1056b0 2024-03-12T06:06:35.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.432+0000 7fba80a00640 1 -- 172.21.15.76:0/1087716155 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba6c003e50 con 0x7fba7c1056b0 2024-03-12T06:06:35.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.432+0000 7fba80a00640 1 -- 172.21.15.76:0/1087716155 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c0044e0 con 0x7fba7c1056b0 2024-03-12T06:06:35.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.432+0000 7fba7bfff640 1 -- 172.21.15.76:0/1087716155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7c19ab90 con 0x7fba7c1056b0 2024-03-12T06:06:35.441 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.433+0000 7fba7bfff640 1 -- 172.21.15.76:0/1087716155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba7c04ef50 con 0x7fba7c1056b0 2024-03-12T06:06:35.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.434+0000 7fba80a00640 1 -- 172.21.15.76:0/1087716155 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7fba6c033780 con 0x7fba7c1056b0 2024-03-12T06:06:35.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.434+0000 7fba80a00640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fba680a1b90 0x7fba680a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:35.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.434+0000 7fba7affd640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fba680a1b90 0x7fba680a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:35.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.434+0000 7fba80a00640 1 -- 172.21.15.76:0/1087716155 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fba6c0f14d0 con 0x7fba7c1056b0 2024-03-12T06:06:35.442 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.435+0000 7fba7affd640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fba680a1b90 0x7fba680a4050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fba74005fd0 tx=0x7fba7400b510 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:35.444 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.439+0000 7fba80a00640 1 -- 172.21.15.76:0/1087716155 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fba6c0ba4b0 con 0x7fba7c1056b0 2024-03-12T06:06:35.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:35 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3344731102' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T06:06:35.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:35 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2160922705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:06:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:35 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3344731102' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T06:06:35.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:35 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2160922705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:06:35.616 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.611+0000 7fba7bfff640 1 -- 172.21.15.76:0/1087716155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba7c107c10 con 0x7fba7c1056b0 2024-03-12T06:06:35.617 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.612+0000 7fba80a00640 1 -- 172.21.15.76:0/1087716155 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+23008 (secure 0 0 0) 0x7fba6c0bdb70 con 0x7fba7c1056b0 2024-03-12T06:06:35.618 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-12T06:06:35.618 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":97,"fsid":"9f336de8-e033-11ee-95c8-87774f69a715","created":"2024-03-12T05:45:46.455987+0000","modified":"2024-03-12T06:06:16.517813+0000","last_up_change":"2024-03-12T05:52:48.871356+0000","last_in_change":"2024-03-12T05:52:28.498480+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T05:49:09.125536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-12T05:53:35.663129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-12T05:53:36.740826+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":16,"snap_epoch":96,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5faf2b7d-916a-4077-8da3-06bba2a002ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6803","nonce":1797183805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6805","nonce":1797183805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6809","nonce":1797183805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":1797183805},{"type":"v1","addr":"172.21.15.76:6807","nonce":1797183805}]},"public_addr":"172.21.15.76:6803/1797183805","cluster_addr":"172.21.15.76:6805/1797183805","heartbeat_back_addr":"172.21.15.76:6809/1797183805","heartbeat_front_addr":"172.21.15.76:6807/1797183805","state":["exists","up"]},{"osd":1,"uuid":"7a64f23f-579e-4114-8250-228b150a753b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6801","nonce":1086375488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6803","nonce":1086375488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6807","nonce":1086375488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1086375488},{"type":"v1","addr":"172.21.15.82:6805","nonce":1086375488}]},"public_addr":"172.21.15.82:6801/1086375488","cluster_addr":"172.21.15.82:6803/1086375488","heartbeat_back_addr":"172.21.15.82:6807/1086375488","heartbeat_front_addr":"172.21.15.82:6805/1086375488","state":["exists","up"]},{"osd":2,"uuid":"621a410c-75b7-42c3-aa04-f4a3e188586b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6801","nonce":1437946838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6803","nonce":1437946838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6807","nonce":1437946838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":1437946838},{"type":"v1","addr":"172.21.15.88:6805","nonce":1437946838}]},"public_addr":"172.21.15.88:6801/1437946838","cluster_addr":"172.21.15.88:6803/1437946838","heartbeat_back_addr":"172.21.15.88:6807/1437946838","heartbeat_front_addr":"172.21.15.88:6805/1437946838","state":["exists","up"]},{"osd":3,"uuid":"e843f131-6b3e-4148-90d2-0e44ea1b5511","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6811","nonce":159124618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6813","nonce":159124618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6817","nonce":159124618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":159124618},{"type":"v1","addr":"172.21.15.76:6815","nonce":159124618}]},"public_addr":"172.21.15.76:6811/159124618","cluster_addr":"172.21.15.76:6813/159124618","heartbeat_back_addr":"172.21.15.76:6817/159124618","heartbeat_front_addr":"172.21.15.76:6815/159124618","state":["exists","up"]},{"osd":4,"uuid":"f20ff3d5-82fd-4d8c-aac9-a3b538e12dff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6809","nonce":3036883628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6811","nonce":3036883628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6815","nonce":3036883628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3036883628},{"type":"v1","addr":"172.21.15.82:6813","nonce":3036883628}]},"public_addr":"172.21.15.82:6809/3036883628","cluster_addr":"172.21.15.82:6811/3036883628","heartbeat_back_addr":"172.21.15.82:6815/3036883628","heartbeat_front_addr":"172.21.15.82:6813/3036883628","state":["exists","up"]},{"osd":5,"uuid":"c9499144-0f35-47f2-89bc-ce4af1aa1b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6809","nonce":2894543264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6811","nonce":2894543264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6815","nonce":2894543264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2894543264},{"type":"v1","addr":"172.21.15.88:6813","nonce":2894543264}]},"public_addr":"172.21.15.88:6809/2894543264","cluster_addr":"172.21.15.88:6811/2894543264","heartbeat_back_addr":"172.21.15.88:6815/2894543264","heartbeat_front_addr":"172.21.15.88:6813/2894543264","state":["exists","up"]},{"osd":6,"uuid":"1dc5e379-2d9e-4ffc-9bfc-4b94e819e4d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6819","nonce":4101985935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6821","nonce":4101985935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6825","nonce":4101985935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":4101985935},{"type":"v1","addr":"172.21.15.76:6823","nonce":4101985935}]},"public_addr":"172.21.15.76:6819/4101985935","cluster_addr":"172.21.15.76:6821/4101985935","heartbeat_back_addr":"172.21.15.76:6825/4101985935","heartbeat_front_addr":"172.21.15.76:6823/4101985935","state":["exists","up"]},{"osd":7,"uuid":"233f8b14-0d82-43d6-96f7-bc9a4ed524a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6817","nonce":88004522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6819","nonce":88004522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6823","nonce":88004522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":88004522},{"type":"v1","addr":"172.21.15.82:6821","nonce":88004522}]},"public_addr":"172.21.15.82:6817/88004522","cluster_addr":"172.21.15.82:6819/88004522","heartbeat_back_addr":"172.21.15.82:6823/88004522","heartbeat_front_addr":"172.21.15.82:6821/88004522","state":["exists","up"]},{"osd":8,"uuid":"959d820e-13bb-4a2b-a9fc-11122a0be243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6817","nonce":529149247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6819","nonce":529149247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6823","nonce":529149247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":529149247},{"type":"v1","addr":"172.21.15.88:6821","nonce":529149247}]},"public_addr":"172.21.15.88:6817/529149247","cluster_addr":"172.21.15.88:6819/529149247","heartbeat_back_addr":"172.21.15.88:6823/529149247","heartbeat_front_addr":"172.21.15.88:6821/529149247","state":["exists","up"]},{"osd":9,"uuid":"bcb44da2-47f9-4ff3-bd96-f026626faea2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6827","nonce":3035286191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6829","nonce":3035286191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6833","nonce":3035286191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3035286191},{"type":"v1","addr":"172.21.15.76:6831","nonce":3035286191}]},"public_addr":"172.21.15.76:6827/3035286191","cluster_addr":"172.21.15.76:6829/3035286191","heartbeat_back_addr":"172.21.15.76:6833/3035286191","heartbeat_front_addr":"172.21.15.76:6831/3035286191","state":["exists","up"]},{"osd":10,"uuid":"c565e2eb-11bf-4889-b19e-11473f9ed6eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6825","nonce":2430204327}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6827","nonce":2430204327}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6831","nonce":2430204327}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2430204327},{"type":"v1","addr":"172.21.15.82:6829","nonce":2430204327}]},"public_addr":"172.21.15.82:6825/2430204327","cluster_addr":"172.21.15.82:6827/2430204327","heartbeat_back_addr":"172.21.15.82:6831/2430204327","heartbeat_front_addr":"172.21.15.82:6829/2430204327","state":["exists","up"]},{"osd":11,"uuid":"cd7c7012-e16e-4a96-b9e0-fb8d7a5af114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6825","nonce":3057841649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6827","nonce":3057841649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6831","nonce":3057841649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3057841649},{"type":"v1","addr":"172.21.15.88:6829","nonce":3057841649}]},"public_addr":"172.21.15.88:6825/3057841649","cluster_addr":"172.21.15.88:6827/3057841649","heartbeat_back_addr":"172.21.15.88:6831/3057841649","heartbeat_front_addr":"172.21.15.88:6829/3057841649","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:17.455447+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:48:41.385198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:04.934852+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:29.067375+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:49:53.675794+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:17.306517+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:50:41.545523+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:06.114027+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:29.856377+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:51:56.112341+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:21.331752+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T05:52:46.019836+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6800/1718825251":"2024-03-13T05:55:46.980746+0000","172.21.15.76:0/189327369":"2024-03-13T05:55:46.980746+0000","172.21.15.76:0/3998885619":"2024-03-13T05:55:46.980746+0000","172.21.15.76:6801/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/2299856451":"2024-03-13T05:55:46.980746+0000","172.21.15.76:6801/1718825251":"2024-03-13T05:55:46.980746+0000","172.21.15.76:6800/306877437":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/1321850785":"2024-03-13T05:46:40.781750+0000","172.21.15.76:0/241684208":"2024-03-13T05:46:40.781750+0000","172.21.15.76:6800/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1776556257":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/1939795424":"2024-03-13T05:55:46.980746+0000","172.21.15.76:6801/4265639592":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3181569618":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/2728891083":"2024-03-13T05:46:12.898913+0000","172.21.15.76:0/3468109661":"2024-03-13T05:46:40.781750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":16,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T06:06:35.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.614+0000 7fba7bfff640 1 -- 172.21.15.76:0/1087716155 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fba680a1b90 msgr2=0x7fba680a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.614+0000 7fba7bfff640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fba680a1b90 0x7fba680a4050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fba74005fd0 tx=0x7fba7400b510 comp rx=0 tx=0).stop 2024-03-12T06:06:35.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.614+0000 7fba7bfff640 1 -- 172.21.15.76:0/1087716155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba7c1056b0 msgr2=0x7fba7c199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.614+0000 7fba7bfff640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba7c1056b0 0x7fba7c199d10 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fba6c00adb0 tx=0x7fba6c035670 comp rx=0 tx=0).stop 2024-03-12T06:06:35.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.614+0000 7fba7bfff640 1 -- 172.21.15.76:0/1087716155 shutdown_connections 2024-03-12T06:06:35.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.614+0000 7fba7bfff640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7fba680a1b90 0x7fba680a4050 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.620 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.614+0000 7fba7bfff640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba7c1056b0 0x7fba7c199d10 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.614+0000 7fba7bfff640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fba7c100060 0x7fba7c1959e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.614+0000 7fba7bfff640 1 --2- 172.21.15.76:0/1087716155 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba7c0ff6a0 0x7fba7c1954a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.614+0000 7fba7bfff640 1 -- 172.21.15.76:0/1087716155 >> 172.21.15.76:0/1087716155 conn(0x7fba7c076460 msgr2=0x7fba7c1025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:35.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.615+0000 7fba7bfff640 1 -- 172.21.15.76:0/1087716155 shutdown_connections 2024-03-12T06:06:35.621 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.615+0000 7fba7bfff640 1 -- 172.21.15.76:0/1087716155 wait complete. 2024-03-12T06:06:35.631 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-12T06:06:35.735 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.728+0000 7f356206f640 1 -- 172.21.15.76:0/1568805331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f355c100c30 msgr2=0x7f355c104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.728+0000 7f356206f640 1 --2- 172.21.15.76:0/1568805331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f355c100c30 0x7f355c104c70 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f3550009930 tx=0x7f355002bc30 comp rx=0 tx=0).stop 2024-03-12T06:06:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.728+0000 7f356206f640 1 -- 172.21.15.76:0/1568805331 shutdown_connections 2024-03-12T06:06:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.728+0000 7f356206f640 1 --2- 172.21.15.76:0/1568805331 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f355c105350 0x7f355c10cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.728+0000 7f356206f640 1 --2- 172.21.15.76:0/1568805331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f355c100c30 0x7f355c104c70 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.728+0000 7f356206f640 1 --2- 172.21.15.76:0/1568805331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f355c100280 0x7f355c100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.728+0000 7f356206f640 1 -- 172.21.15.76:0/1568805331 >> 172.21.15.76:0/1568805331 conn(0x7f355c0771a0 msgr2=0x7f355c0775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.728+0000 7f356206f640 1 -- 172.21.15.76:0/1568805331 shutdown_connections 2024-03-12T06:06:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.728+0000 7f356206f640 1 -- 172.21.15.76:0/1568805331 wait complete. 2024-03-12T06:06:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.729+0000 7f356206f640 1 Processor -- start 2024-03-12T06:06:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.729+0000 7f356206f640 1 -- start start 2024-03-12T06:06:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.729+0000 7f356206f640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f355c100280 0x7f355c19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.729+0000 7f356206f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f355c105350 0x7f355c19d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.729+0000 7f356206f640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f355c197630 0x7f355c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.729+0000 7f356206f640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c198020 con 0x7f355c197630 2024-03-12T06:06:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.729+0000 7f356206f640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c198190 con 0x7f355c100280 2024-03-12T06:06:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.729+0000 7f356206f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c198350 con 0x7f355c105350 2024-03-12T06:06:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.729+0000 7f355bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f355c197630 0x7f355c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f355bfff640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f355c197630 0x7f355c197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47158/0 (socket says 172.21.15.76:47158) 2024-03-12T06:06:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f355bfff640 1 -- 172.21.15.76:0/2271841887 learned_addr learned my addr 172.21.15.76:0/2271841887 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:06:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f355affd640 1 --2- 172.21.15.76:0/2271841887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f355c105350 0x7f355c19d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f355b7fe640 1 --2- 172.21.15.76:0/2271841887 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f355c100280 0x7f355c19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f355bfff640 1 -- 172.21.15.76:0/2271841887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f355c105350 msgr2=0x7f355c19d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f355bfff640 1 --2- 172.21.15.76:0/2271841887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f355c105350 0x7f355c19d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f355bfff640 1 -- 172.21.15.76:0/2271841887 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f355c100280 msgr2=0x7f355c19d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f355bfff640 1 --2- 172.21.15.76:0/2271841887 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f355c100280 0x7f355c19d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f355bfff640 1 -- 172.21.15.76:0/2271841887 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3550009560 con 0x7f355c197630 2024-03-12T06:06:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f355bfff640 1 --2- 172.21.15.76:0/2271841887 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f355c197630 0x7f355c197ae0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f355400eaa0 tx=0x7f355400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f3558ff9640 1 -- 172.21.15.76:0/2271841887 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355400cdb0 con 0x7f355c197630 2024-03-12T06:06:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f3558ff9640 1 -- 172.21.15.76:0/2271841887 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3554005020 con 0x7f355c197630 2024-03-12T06:06:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f3558ff9640 1 -- 172.21.15.76:0/2271841887 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35540116b0 con 0x7f355c197630 2024-03-12T06:06:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f356206f640 1 -- 172.21.15.76:0/2271841887 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355c1a42f0 con 0x7f355c197630 2024-03-12T06:06:35.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.730+0000 7f356206f640 1 -- 172.21.15.76:0/2271841887 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c1a4780 con 0x7f355c197630 2024-03-12T06:06:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.731+0000 7f356206f640 1 -- 172.21.15.76:0/2271841887 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355c068450 con 0x7f355c197630 2024-03-12T06:06:35.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.732+0000 7f3558ff9640 1 -- 172.21.15.76:0/2271841887 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f3554005520 con 0x7f355c197630 2024-03-12T06:06:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.733+0000 7f3558ff9640 1 --2- 172.21.15.76:0/2271841887 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f35440a1ba0 0x7f35440a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.733+0000 7f355b7fe640 1 --2- 172.21.15.76:0/2271841887 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f35440a1ba0 0x7f35440a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.733+0000 7f3558ff9640 1 -- 172.21.15.76:0/2271841887 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f35540cd950 con 0x7f355c197630 2024-03-12T06:06:35.741 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.733+0000 7f355b7fe640 1 --2- 172.21.15.76:0/2271841887 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f35440a1ba0 0x7f35440a4060 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f354c009bd0 tx=0x7f354c005f90 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.737+0000 7f3558ff9640 1 -- 172.21.15.76:0/2271841887 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f35540968d0 con 0x7f355c197630 2024-03-12T06:06:35.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.946+0000 7f356206f640 1 -- 172.21.15.76:0/2271841887 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f355c100c30 con 0x7f355c197630 2024-03-12T06:06:35.953 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.948+0000 7f3558ff9640 1 -- 172.21.15.76:0/2271841887 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3554099f90 con 0x7f355c197630 2024-03-12T06:06:35.953 INFO:teuthology.orchestra.run.smithi076.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-12T06:06:35.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.951+0000 7f356206f640 1 -- 172.21.15.76:0/2271841887 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f35440a1ba0 msgr2=0x7f35440a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.951+0000 7f356206f640 1 --2- 172.21.15.76:0/2271841887 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f35440a1ba0 0x7f35440a4060 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f354c009bd0 tx=0x7f354c005f90 comp rx=0 tx=0).stop 2024-03-12T06:06:35.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.951+0000 7f356206f640 1 -- 172.21.15.76:0/2271841887 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f355c197630 msgr2=0x7f355c197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:35.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.951+0000 7f356206f640 1 --2- 172.21.15.76:0/2271841887 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f355c197630 0x7f355c197ae0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f355400eaa0 tx=0x7f355400ef60 comp rx=0 tx=0).stop 2024-03-12T06:06:35.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.951+0000 7f356206f640 1 -- 172.21.15.76:0/2271841887 shutdown_connections 2024-03-12T06:06:35.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.951+0000 7f356206f640 1 --2- 172.21.15.76:0/2271841887 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f35440a1ba0 0x7f35440a4060 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.951+0000 7f356206f640 1 --2- 172.21.15.76:0/2271841887 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f355c197630 0x7f355c197ae0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.951+0000 7f356206f640 1 --2- 172.21.15.76:0/2271841887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f355c105350 0x7f355c19d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.951+0000 7f356206f640 1 --2- 172.21.15.76:0/2271841887 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f355c100280 0x7f355c19d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:35.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.951+0000 7f356206f640 1 -- 172.21.15.76:0/2271841887 >> 172.21.15.76:0/2271841887 conn(0x7f355c0771a0 msgr2=0x7f355c0fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:35.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.952+0000 7f356206f640 1 -- 172.21.15.76:0/2271841887 shutdown_connections 2024-03-12T06:06:35.958 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:35.952+0000 7f356206f640 1 -- 172.21.15.76:0/2271841887 wait complete. 2024-03-12T06:06:35.968 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-12T06:06:36.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.068+0000 7efd51a49640 1 -- 172.21.15.76:0/2940299343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd4c10b770 msgr2=0x7efd4c10dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:36.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.068+0000 7efd51a49640 1 --2- 172.21.15.76:0/2940299343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd4c10b770 0x7efd4c10dba0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7efd44009ca0 tx=0x7efd4402c180 comp rx=0 tx=0).stop 2024-03-12T06:06:36.076 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.068+0000 7efd51a49640 1 -- 172.21.15.76:0/2940299343 shutdown_connections 2024-03-12T06:06:36.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.068+0000 7efd51a49640 1 --2- 172.21.15.76:0/2940299343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd4c10b770 0x7efd4c10dba0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:36.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.068+0000 7efd51a49640 1 --2- 172.21.15.76:0/2940299343 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd4c102cc0 0x7efd4c10b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:36.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.068+0000 7efd51a49640 1 --2- 172.21.15.76:0/2940299343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd4c1023a0 0x7efd4c102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:36.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.068+0000 7efd51a49640 1 -- 172.21.15.76:0/2940299343 >> 172.21.15.76:0/2940299343 conn(0x7efd4c0771a0 msgr2=0x7efd4c0775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:36.077 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.068+0000 7efd51a49640 1 -- 172.21.15.76:0/2940299343 shutdown_connections 2024-03-12T06:06:36.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.068+0000 7efd51a49640 1 -- 172.21.15.76:0/2940299343 wait complete. 2024-03-12T06:06:36.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.069+0000 7efd51a49640 1 Processor -- start 2024-03-12T06:06:36.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.069+0000 7efd51a49640 1 -- start start 2024-03-12T06:06:36.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.069+0000 7efd51a49640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd4c1023a0 0x7efd4c19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:36.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.069+0000 7efd51a49640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd4c102cc0 0x7efd4c19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:36.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.069+0000 7efd51a49640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd4c10b770 0x7efd4c1975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:36.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.069+0000 7efd51a49640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4c197bf0 con 0x7efd4c102cc0 2024-03-12T06:06:36.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.069+0000 7efd51a49640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4c197d60 con 0x7efd4c1023a0 2024-03-12T06:06:36.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.069+0000 7efd51a49640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4c197f20 con 0x7efd4c10b770 2024-03-12T06:06:36.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.069+0000 7efd4a7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd4c102cc0 0x7efd4c19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:36.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.069+0000 7efd4a7fc640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd4c102cc0 0x7efd4c19d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47170/0 (socket says 172.21.15.76:47170) 2024-03-12T06:06:36.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.069+0000 7efd4a7fc640 1 -- 172.21.15.76:0/2995607039 learned_addr learned my addr 172.21.15.76:0/2995607039 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:06:36.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.069+0000 7efd4b7fe640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd4c10b770 0x7efd4c1975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:36.079 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.069+0000 7efd4affd640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd4c1023a0 0x7efd4c19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:36.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.070+0000 7efd4a7fc640 1 -- 172.21.15.76:0/2995607039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd4c10b770 msgr2=0x7efd4c1975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:36.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.070+0000 7efd4a7fc640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd4c10b770 0x7efd4c1975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:36.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.070+0000 7efd4a7fc640 1 -- 172.21.15.76:0/2995607039 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd4c1023a0 msgr2=0x7efd4c19d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:36.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.070+0000 7efd4a7fc640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd4c1023a0 0x7efd4c19d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:36.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.070+0000 7efd4a7fc640 1 -- 172.21.15.76:0/2995607039 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd44009900 con 0x7efd4c102cc0 2024-03-12T06:06:36.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.070+0000 7efd4b7fe640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd4c10b770 0x7efd4c1975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T06:06:36.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.070+0000 7efd4a7fc640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd4c102cc0 0x7efd4c19d930 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7efd4000c890 tx=0x7efd4000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:36.080 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.070+0000 7efd50a47640 1 -- 172.21.15.76:0/2995607039 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd40004220 con 0x7efd4c102cc0 2024-03-12T06:06:36.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.070+0000 7efd51a49640 1 -- 172.21.15.76:0/2995607039 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd4c198200 con 0x7efd4c102cc0 2024-03-12T06:06:36.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.070+0000 7efd50a47640 1 -- 172.21.15.76:0/2995607039 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd40007e90 con 0x7efd4c102cc0 2024-03-12T06:06:36.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.070+0000 7efd50a47640 1 -- 172.21.15.76:0/2995607039 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd40004a00 con 0x7efd4c102cc0 2024-03-12T06:06:36.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.070+0000 7efd51a49640 1 -- 172.21.15.76:0/2995607039 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd4c1a4490 con 0x7efd4c102cc0 2024-03-12T06:06:36.081 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.071+0000 7efd51a49640 1 -- 172.21.15.76:0/2995607039 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd4c06a040 con 0x7efd4c102cc0 2024-03-12T06:06:36.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.072+0000 7efd50a47640 1 -- 172.21.15.76:0/2995607039 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7efd400079b0 con 0x7efd4c102cc0 2024-03-12T06:06:36.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.073+0000 7efd50a47640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efd340a1ba0 0x7efd340a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:36.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.073+0000 7efd4affd640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efd340a1ba0 0x7efd340a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:36.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.073+0000 7efd50a47640 1 -- 172.21.15.76:0/2995607039 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8727+0+0 (secure 0 0 0) 0x7efd400cd620 con 0x7efd4c102cc0 2024-03-12T06:06:36.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.073+0000 7efd4affd640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efd340a1ba0 0x7efd340a4060 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efd3c005fd0 tx=0x7efd3c005e20 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:36.083 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.077+0000 7efd50a47640 1 -- 172.21.15.76:0/2995607039 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efd40096420 con 0x7efd4c102cc0 2024-03-12T06:06:36.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a[32091]: 2024-03-12T06:06:35.947+0000 7f96e26d6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-12T06:06:36.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:36 smithi076 ceph-mon[32095]: pgmap v335: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:06:36.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:36 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2751795358' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:06:36.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:36 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/1087716155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T06:06:36.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:36 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2271841887' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-12T06:06:36.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:36 smithi076 ceph-mon[32095]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-12T06:06:36.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:36 smithi076 ceph-mon[32095]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-12T06:06:36.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:36 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2271841887' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-12T06:06:36.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:36 smithi076 ceph-mon[32095]: osdmap e98: 12 total, 12 up, 12 in 2024-03-12T06:06:36.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:36 smithi076 ceph-mon[32095]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-12T06:06:36.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.292+0000 7efd51a49640 1 -- 172.21.15.76:0/2995607039 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efd4c1068c0 con 0x7efd4c102cc0 2024-03-12T06:06:36.446 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-h[66229]: ignoring --setuser ceph since I am not root 2024-03-12T06:06:36.447 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-h[66229]: ignoring --setgroup ceph since I am not root 2024-03-12T06:06:36.447 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-mds[66233]: -- 172.21.15.82:0/122561310 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560988a021a0 con 0x5609889d9000 2024-03-12T06:06:36.447 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-h[66229]: starting mds.h at 2024-03-12T06:06:36.447 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 06:06:35 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b[66548]: ignoring --setuser ceph since I am not root 2024-03-12T06:06:36.448 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 06:06:35 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b[66548]: ignoring --setgroup ceph since I am not root 2024-03-12T06:06:36.448 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 06:06:35 smithi082 ceph-mds[66552]: -- 172.21.15.82:0/855498580 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b0b03801a0 con 0x55b0b0356c00 2024-03-12T06:06:36.448 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 06:06:35 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b[66548]: starting mds.b at 2024-03-12T06:06:36.448 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-e[65918]: ignoring --setuser ceph since I am not root 2024-03-12T06:06:36.448 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-e[65918]: ignoring --setgroup ceph since I am not root 2024-03-12T06:06:36.448 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-mds[65922]: -- 172.21.15.82:0/4127110934 <== mon.0 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556a52aac1a0 con 0x556a52a82c00 2024-03-12T06:06:36.449 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-e[65918]: starting mds.e at 2024-03-12T06:06:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-mon[36382]: pgmap v335: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:06:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2751795358' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:06:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/1087716155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T06:06:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2271841887' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-12T06:06:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-mon[36382]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-12T06:06:36.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-mon[36382]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-12T06:06:36.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2271841887' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-12T06:06:36.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-mon[36382]: osdmap e98: 12 total, 12 up, 12 in 2024-03-12T06:06:36.450 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:36 smithi082 ceph-mon[36382]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-12T06:06:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:36 smithi088 ceph-mon[36585]: pgmap v335: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-12T06:06:36.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:36 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2751795358' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T06:06:36.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:36 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/1087716155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T06:06:36.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:36 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2271841887' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-12T06:06:36.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:36 smithi088 ceph-mon[36585]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-12T06:06:36.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:36 smithi088 ceph-mon[36585]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-12T06:06:36.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:36 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2271841887' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-12T06:06:36.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:36 smithi088 ceph-mon[36585]: osdmap e98: 12 total, 12 up, 12 in 2024-03-12T06:06:36.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:36 smithi088 ceph-mon[36585]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-12T06:06:36.488 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 06:06:35 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-c[68008]: ignoring --setuser ceph since I am not root 2024-03-12T06:06:36.488 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 06:06:35 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-c[68008]: ignoring --setgroup ceph since I am not root 2024-03-12T06:06:36.488 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 06:06:35 smithi088 ceph-mds[68012]: -- 172.21.15.88:0/980867305 <== mon.2 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b4ea55c1a0 con 0x55b4ea532800 2024-03-12T06:06:36.488 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 06:06:35 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-c[68008]: starting mds.c at 2024-03-12T06:06:36.488 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 06:06:35 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-f[67702]: ignoring --setuser ceph since I am not root 2024-03-12T06:06:36.489 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 06:06:35 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-f[67702]: ignoring --setgroup ceph since I am not root 2024-03-12T06:06:36.489 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 06:06:35 smithi088 ceph-mds[67706]: -- 172.21.15.88:0/2270586588 <== mon.0 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562a1c24c1a0 con 0x562a1c222c00 2024-03-12T06:06:36.489 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 06:06:35 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-f[67702]: starting mds.f at 2024-03-12T06:06:36.489 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 06:06:35 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-i[68318]: ignoring --setuser ceph since I am not root 2024-03-12T06:06:36.489 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 06:06:35 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-i[68318]: ignoring --setgroup ceph since I am not root 2024-03-12T06:06:36.489 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 06:06:35 smithi088 ceph-mds[68322]: -- 172.21.15.88:0/3276511216 <== mon.1 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dba7d101a0 con 0x55dba7ce6c00 2024-03-12T06:06:36.490 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 06:06:35 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-i[68318]: starting mds.i at 2024-03-12T06:06:36.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.950+0000 7efd50a47640 1 -- 172.21.15.76:0/2995607039 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) v1 ==== 104+0+0 (secure 0 0 0) 0x7efd40099ae0 con 0x7efd4c102cc0 2024-03-12T06:06:36.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.954+0000 7efd51a49640 1 -- 172.21.15.76:0/2995607039 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efd340a1ba0 msgr2=0x7efd340a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:36.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.954+0000 7efd51a49640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efd340a1ba0 0x7efd340a4060 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efd3c005fd0 tx=0x7efd3c005e20 comp rx=0 tx=0).stop 2024-03-12T06:06:36.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.954+0000 7efd51a49640 1 -- 172.21.15.76:0/2995607039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd4c102cc0 msgr2=0x7efd4c19d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:36.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.954+0000 7efd51a49640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd4c102cc0 0x7efd4c19d930 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7efd4000c890 tx=0x7efd4000cd50 comp rx=0 tx=0).stop 2024-03-12T06:06:36.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.954+0000 7efd51a49640 1 -- 172.21.15.76:0/2995607039 shutdown_connections 2024-03-12T06:06:36.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.954+0000 7efd51a49640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7efd340a1ba0 0x7efd340a4060 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:36.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.954+0000 7efd51a49640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd4c10b770 0x7efd4c1975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:36.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.954+0000 7efd51a49640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efd4c102cc0 0x7efd4c19d930 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:36.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.954+0000 7efd51a49640 1 --2- 172.21.15.76:0/2995607039 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efd4c1023a0 0x7efd4c19d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:36.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.954+0000 7efd51a49640 1 -- 172.21.15.76:0/2995607039 >> 172.21.15.76:0/2995607039 conn(0x7efd4c0771a0 msgr2=0x7efd4c0fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:36.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.954+0000 7efd51a49640 1 -- 172.21.15.76:0/2995607039 shutdown_connections 2024-03-12T06:06:36.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:36.954+0000 7efd51a49640 1 -- 172.21.15.76:0/2995607039 wait complete. 2024-03-12T06:06:36.974 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-12T06:06:37.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.080+0000 7f5548a39640 1 -- 172.21.15.76:0/1199943424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55440ffcf0 msgr2=0x7f55441000d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:37.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.080+0000 7f5548a39640 1 --2- 172.21.15.76:0/1199943424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55440ffcf0 0x7f55441000d0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f552c009930 tx=0x7f552c02bc30 comp rx=0 tx=0).stop 2024-03-12T06:06:37.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.080+0000 7f5548a39640 1 -- 172.21.15.76:0/1199943424 shutdown_connections 2024-03-12T06:06:37.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.080+0000 7f5548a39640 1 --2- 172.21.15.76:0/1199943424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55441064b0 0x7f554410e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:37.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.080+0000 7f5548a39640 1 --2- 172.21.15.76:0/1199943424 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f55441006a0 0x7f5544105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:37.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.080+0000 7f5548a39640 1 --2- 172.21.15.76:0/1199943424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55440ffcf0 0x7f55441000d0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:37.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.080+0000 7f5548a39640 1 -- 172.21.15.76:0/1199943424 >> 172.21.15.76:0/1199943424 conn(0x7f55440752d0 msgr2=0x7f55440756e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:37.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.080+0000 7f5548a39640 1 -- 172.21.15.76:0/1199943424 shutdown_connections 2024-03-12T06:06:37.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.080+0000 7f5548a39640 1 -- 172.21.15.76:0/1199943424 wait complete. 2024-03-12T06:06:37.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5548a39640 1 Processor -- start 2024-03-12T06:06:37.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5548a39640 1 -- start start 2024-03-12T06:06:37.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5548a39640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55440ffcf0 0x7f554419d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:37.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5548a39640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55441064b0 0x7f554419db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:37.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5548a39640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5544197790 0x7f5544197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:37.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5548a39640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5544198240 con 0x7f55441064b0 2024-03-12T06:06:37.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5548a39640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55441983b0 con 0x7f5544197790 2024-03-12T06:06:37.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5548a39640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5544198520 con 0x7f55440ffcf0 2024-03-12T06:06:37.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5541d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55441064b0 0x7f554419db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:37.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5541d9b640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55441064b0 0x7f554419db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47200/0 (socket says 172.21.15.76:47200) 2024-03-12T06:06:37.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5541d9b640 1 -- 172.21.15.76:0/4286636623 learned_addr learned my addr 172.21.15.76:0/4286636623 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:06:37.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f554259c640 1 --2- 172.21.15.76:0/4286636623 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55440ffcf0 0x7f554419d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:37.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5542d9d640 1 --2- 172.21.15.76:0/4286636623 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5544197790 0x7f5544197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:37.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5541d9b640 1 -- 172.21.15.76:0/4286636623 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55440ffcf0 msgr2=0x7f554419d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:37.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5541d9b640 1 --2- 172.21.15.76:0/4286636623 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55440ffcf0 0x7f554419d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:37.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5541d9b640 1 -- 172.21.15.76:0/4286636623 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5544197790 msgr2=0x7f5544197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:37.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5541d9b640 1 --2- 172.21.15.76:0/4286636623 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5544197790 0x7f5544197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:37.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5541d9b640 1 -- 172.21.15.76:0/4286636623 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f552c009560 con 0x7f55441064b0 2024-03-12T06:06:37.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.081+0000 7f5541d9b640 1 --2- 172.21.15.76:0/4286636623 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55441064b0 0x7f554419db60 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f553400b450 tx=0x7f553400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:37.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.082+0000 7f553b7fe640 1 -- 172.21.15.76:0/4286636623 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5534011890 con 0x7f55441064b0 2024-03-12T06:06:37.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.082+0000 7f553b7fe640 1 -- 172.21.15.76:0/4286636623 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55340103e0 con 0x7f55441064b0 2024-03-12T06:06:37.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.082+0000 7f5548a39640 1 -- 172.21.15.76:0/4286636623 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55441a44d0 con 0x7f55441064b0 2024-03-12T06:06:37.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.082+0000 7f553b7fe640 1 -- 172.21.15.76:0/4286636623 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553400f580 con 0x7f55441064b0 2024-03-12T06:06:37.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.082+0000 7f5548a39640 1 -- 172.21.15.76:0/4286636623 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55441a49b0 con 0x7f55441064b0 2024-03-12T06:06:37.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.083+0000 7f5548a39640 1 -- 172.21.15.76:0/4286636623 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55440693b0 con 0x7f55441064b0 2024-03-12T06:06:37.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.084+0000 7f553b7fe640 1 -- 172.21.15.76:0/4286636623 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f55340108e0 con 0x7f55441064b0 2024-03-12T06:06:37.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.084+0000 7f553b7fe640 1 --2- 172.21.15.76:0/4286636623 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f55300a1ba0 0x7f55300a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:37.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.084+0000 7f554259c640 1 --2- 172.21.15.76:0/4286636623 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f55300a1ba0 0x7f55300a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:37.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.084+0000 7f553b7fe640 1 -- 172.21.15.76:0/4286636623 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8727+0+0 (secure 0 0 0) 0x7f55340cd580 con 0x7f55441064b0 2024-03-12T06:06:37.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.085+0000 7f554259c640 1 --2- 172.21.15.76:0/4286636623 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f55300a1ba0 0x7f55300a4060 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f552c005ab0 tx=0x7f552c005fd0 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:37.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.089+0000 7f553b7fe640 1 -- 172.21.15.76:0/4286636623 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5534096490 con 0x7f55441064b0 2024-03-12T06:06:37.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2995607039' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T06:06:37.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-12T06:06:37.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-12T06:06:37.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: Cluster is now healthy 2024-03-12T06:06:37.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/2995607039' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-12T06:06:37.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.88:6834/184259728,v1:172.21.15.88:6837/184259728] up:boot 2024-03-12T06:06:37.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.88:6836/578733110,v1:172.21.15.88:6839/578733110] up:boot 2024-03-12T06:06:37.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.82:6837/1616837330,v1:172.21.15.82:6839/1616837330] up:boot 2024-03-12T06:06:37.241 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.88:6835/2950710119,v1:172.21.15.88:6838/2950710119] up:boot 2024-03-12T06:06:37.241 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.82:6834/1812934855,v1:172.21.15.82:6835/1812934855] up:boot 2024-03-12T06:06:37.241 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: mds.? [v2:172.21.15.82:6836/2809609039,v1:172.21.15.82:6838/2809609039] up:boot 2024-03-12T06:06:37.241 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: fsmap 12 up:standby 2024-03-12T06:06:37.241 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T06:06:37.241 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T06:06:37.241 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T06:06:37.241 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T06:06:37.241 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T06:06:37.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:37 smithi076 ceph-mon[32095]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T06:06:37.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:37.263+0000 7f5548a39640 1 -- 172.21.15.76:0/4286636623 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5544109660 con 0x7f55441064b0 2024-03-12T06:06:37.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2995607039' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T06:06:37.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-12T06:06:37.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-12T06:06:37.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: Cluster is now healthy 2024-03-12T06:06:37.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/2995607039' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-12T06:06:37.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.88:6834/184259728,v1:172.21.15.88:6837/184259728] up:boot 2024-03-12T06:06:37.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.88:6836/578733110,v1:172.21.15.88:6839/578733110] up:boot 2024-03-12T06:06:37.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.82:6837/1616837330,v1:172.21.15.82:6839/1616837330] up:boot 2024-03-12T06:06:37.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.88:6835/2950710119,v1:172.21.15.88:6838/2950710119] up:boot 2024-03-12T06:06:37.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.82:6834/1812934855,v1:172.21.15.82:6835/1812934855] up:boot 2024-03-12T06:06:37.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: mds.? [v2:172.21.15.82:6836/2809609039,v1:172.21.15.82:6838/2809609039] up:boot 2024-03-12T06:06:37.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: fsmap 12 up:standby 2024-03-12T06:06:37.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T06:06:37.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T06:06:37.448 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T06:06:37.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T06:06:37.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T06:06:37.449 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:37 smithi082 ceph-mon[36382]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T06:06:37.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2995607039' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T06:06:37.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-12T06:06:37.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-12T06:06:37.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: Cluster is now healthy 2024-03-12T06:06:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/2995607039' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-12T06:06:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.88:6834/184259728,v1:172.21.15.88:6837/184259728] up:boot 2024-03-12T06:06:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.88:6836/578733110,v1:172.21.15.88:6839/578733110] up:boot 2024-03-12T06:06:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.82:6837/1616837330,v1:172.21.15.82:6839/1616837330] up:boot 2024-03-12T06:06:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.88:6835/2950710119,v1:172.21.15.88:6838/2950710119] up:boot 2024-03-12T06:06:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.82:6834/1812934855,v1:172.21.15.82:6835/1812934855] up:boot 2024-03-12T06:06:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: mds.? [v2:172.21.15.82:6836/2809609039,v1:172.21.15.82:6838/2809609039] up:boot 2024-03-12T06:06:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: fsmap 12 up:standby 2024-03-12T06:06:37.487 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-12T06:06:37.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-12T06:06:37.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-12T06:06:37.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-12T06:06:37.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-12T06:06:37.488 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:37 smithi088 ceph-mon[36585]: from='mgr.15465 172.21.15.76:0/1858954767' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-12T06:06:38.138 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.132+0000 7f553b7fe640 1 -- 172.21.15.76:0/4286636623 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v99) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5534099b50 con 0x7f55441064b0 2024-03-12T06:06:38.138 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cephfs_metadata' removed 2024-03-12T06:06:38.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.136+0000 7f5548a39640 1 -- 172.21.15.76:0/4286636623 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f55300a1ba0 msgr2=0x7f55300a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:38.141 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.136+0000 7f5548a39640 1 --2- 172.21.15.76:0/4286636623 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f55300a1ba0 0x7f55300a4060 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f552c005ab0 tx=0x7f552c005fd0 comp rx=0 tx=0).stop 2024-03-12T06:06:38.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.136+0000 7f5548a39640 1 -- 172.21.15.76:0/4286636623 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55441064b0 msgr2=0x7f554419db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:38.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.136+0000 7f5548a39640 1 --2- 172.21.15.76:0/4286636623 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55441064b0 0x7f554419db60 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f553400b450 tx=0x7f553400b910 comp rx=0 tx=0).stop 2024-03-12T06:06:38.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.137+0000 7f5548a39640 1 -- 172.21.15.76:0/4286636623 shutdown_connections 2024-03-12T06:06:38.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.137+0000 7f5548a39640 1 --2- 172.21.15.76:0/4286636623 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f55300a1ba0 0x7f55300a4060 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:38.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.137+0000 7f5548a39640 1 --2- 172.21.15.76:0/4286636623 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5544197790 0x7f5544197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:38.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.137+0000 7f5548a39640 1 --2- 172.21.15.76:0/4286636623 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f55441064b0 0x7f554419db60 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:38.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.137+0000 7f5548a39640 1 --2- 172.21.15.76:0/4286636623 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55440ffcf0 0x7f554419d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:38.142 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.137+0000 7f5548a39640 1 -- 172.21.15.76:0/4286636623 >> 172.21.15.76:0/4286636623 conn(0x7f55440752d0 msgr2=0x7f55440fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:38.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.137+0000 7f5548a39640 1 -- 172.21.15.76:0/4286636623 shutdown_connections 2024-03-12T06:06:38.143 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.137+0000 7f5548a39640 1 -- 172.21.15.76:0/4286636623 wait complete. 2024-03-12T06:06:38.156 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-12T06:06:38.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 -- 172.21.15.76:0/2710715834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58f4100c40 msgr2=0x7f58f4104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:38.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 --2- 172.21.15.76:0/2710715834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58f4100c40 0x7f58f4104c80 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009930 tx=0x7f58e402bc30 comp rx=0 tx=0).stop 2024-03-12T06:06:38.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 -- 172.21.15.76:0/2710715834 shutdown_connections 2024-03-12T06:06:38.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 --2- 172.21.15.76:0/2710715834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58f4105360 0x7f58f410cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:38.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 --2- 172.21.15.76:0/2710715834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58f4100c40 0x7f58f4104c80 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:38.270 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 --2- 172.21.15.76:0/2710715834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58f4100290 0x7f58f4100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:38.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 -- 172.21.15.76:0/2710715834 >> 172.21.15.76:0/2710715834 conn(0x7f58f4077230 msgr2=0x7f58f4077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:38.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 -- 172.21.15.76:0/2710715834 shutdown_connections 2024-03-12T06:06:38.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 -- 172.21.15.76:0/2710715834 wait complete. 2024-03-12T06:06:38.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 Processor -- start 2024-03-12T06:06:38.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 -- start start 2024-03-12T06:06:38.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58f4100290 0x7f58f4198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:38.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58f4105360 0x7f58f4198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:38.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58f4199120 0x7f58f41a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:38.271 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f41a3d70 con 0x7f58f4199120 2024-03-12T06:06:38.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f41a3ee0 con 0x7f58f4105360 2024-03-12T06:06:38.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.263+0000 7f58fc6df640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f41a40a0 con 0x7f58f4100290 2024-03-12T06:06:38.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58fac7c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58f4199120 0x7f58f41a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:38.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58fac7c640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58f4199120 0x7f58f41a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:47210/0 (socket says 172.21.15.76:47210) 2024-03-12T06:06:38.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58fa47b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58f4100290 0x7f58f4198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:38.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58fac7c640 1 -- 172.21.15.76:0/3854020993 learned_addr learned my addr 172.21.15.76:0/3854020993 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-03-12T06:06:38.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58f9c7a640 1 --2- 172.21.15.76:0/3854020993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58f4105360 0x7f58f4198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:38.272 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58fac7c640 1 -- 172.21.15.76:0/3854020993 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58f4100290 msgr2=0x7f58f4198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:38.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58fac7c640 1 --2- 172.21.15.76:0/3854020993 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58f4100290 0x7f58f4198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:38.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58fac7c640 1 -- 172.21.15.76:0/3854020993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58f4105360 msgr2=0x7f58f4198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:38.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58fac7c640 1 --2- 172.21.15.76:0/3854020993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58f4105360 0x7f58f4198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:38.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58fac7c640 1 -- 172.21.15.76:0/3854020993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e4009560 con 0x7f58f4199120 2024-03-12T06:06:38.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58fac7c640 1 --2- 172.21.15.76:0/3854020993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58f4199120 0x7f58f41a3680 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f58ec00e960 tx=0x7f58ec00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:38.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58eb7fe640 1 -- 172.21.15.76:0/3854020993 <== mon.0 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec004d20 con 0x7f58f4199120 2024-03-12T06:06:38.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58eb7fe640 1 -- 172.21.15.76:0/3854020993 <== mon.0 v2:172.21.15.76:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58ec004e80 con 0x7f58f4199120 2024-03-12T06:06:38.273 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58fc6df640 1 -- 172.21.15.76:0/3854020993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f41a4380 con 0x7f58f4199120 2024-03-12T06:06:38.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58eb7fe640 1 -- 172.21.15.76:0/3854020993 <== mon.0 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec0116d0 con 0x7f58f4199120 2024-03-12T06:06:38.274 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.264+0000 7f58fc6df640 1 -- 172.21.15.76:0/3854020993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f41a4860 con 0x7f58f4199120 2024-03-12T06:06:38.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.265+0000 7f58fc6df640 1 -- 172.21.15.76:0/3854020993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f4068000 con 0x7f58f4199120 2024-03-12T06:06:38.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.266+0000 7f58eb7fe640 1 -- 172.21.15.76:0/3854020993 <== mon.0 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 26) v1 ==== 136410+0+0 (secure 0 0 0) 0x7f58ec005340 con 0x7f58f4199120 2024-03-12T06:06:38.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.267+0000 7f58eb7fe640 1 --2- 172.21.15.76:0/3854020993 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f58e00a1ba0 0x7f58e00a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T06:06:38.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.267+0000 7f58fa47b640 1 --2- 172.21.15.76:0/3854020993 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f58e00a1ba0 0x7f58e00a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T06:06:38.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.267+0000 7f58eb7fe640 1 -- 172.21.15.76:0/3854020993 <== mon.0 v2:172.21.15.76:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8231+0+0 (secure 0 0 0) 0x7f58ec0cdaf0 con 0x7f58f4199120 2024-03-12T06:06:38.275 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.267+0000 7f58fa47b640 1 --2- 172.21.15.76:0/3854020993 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f58e00a1ba0 0x7f58e00a4060 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f58f000bbe0 tx=0x7f58f000b390 comp rx=0 tx=0).ready entity=mgr.15465 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T06:06:38.276 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.271+0000 7f58eb7fe640 1 -- 172.21.15.76:0/3854020993 <== mon.0 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f58ec096b90 con 0x7f58f4199120 2024-03-12T06:06:38.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:38 smithi082 ceph-mon[36382]: pgmap v337: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:06:38.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:38 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4286636623' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T06:06:38.451 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:38.446+0000 7f58fc6df640 1 -- 172.21.15.76:0/3854020993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f58f4108590 con 0x7f58f4199120 2024-03-12T06:06:38.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:38 smithi088 ceph-mon[36585]: pgmap v337: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:06:38.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:38 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4286636623' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T06:06:38.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:38 smithi076 ceph-mon[32095]: pgmap v337: 97 pgs: 97 active+clean; 80 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-12T06:06:38.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:38 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4286636623' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T06:06:39.144 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:39.138+0000 7f58eb7fe640 1 -- 172.21.15.76:0/3854020993 <== mon.0 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v100) v1 ==== 169+0+0 (secure 0 0 0) 0x7f58ec09a250 con 0x7f58f4199120 2024-03-12T06:06:39.144 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'cephfs_data' removed 2024-03-12T06:06:39.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:39.142+0000 7f58fc6df640 1 -- 172.21.15.76:0/3854020993 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f58e00a1ba0 msgr2=0x7f58e00a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:39.147 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:39.142+0000 7f58fc6df640 1 --2- 172.21.15.76:0/3854020993 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f58e00a1ba0 0x7f58e00a4060 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f58f000bbe0 tx=0x7f58f000b390 comp rx=0 tx=0).stop 2024-03-12T06:06:39.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:39.142+0000 7f58fc6df640 1 -- 172.21.15.76:0/3854020993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58f4199120 msgr2=0x7f58f41a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T06:06:39.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:39.142+0000 7f58fc6df640 1 --2- 172.21.15.76:0/3854020993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58f4199120 0x7f58f41a3680 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f58ec00e960 tx=0x7f58ec00ee20 comp rx=0 tx=0).stop 2024-03-12T06:06:39.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:39.142+0000 7f58fc6df640 1 -- 172.21.15.76:0/3854020993 shutdown_connections 2024-03-12T06:06:39.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:39.142+0000 7f58fc6df640 1 --2- 172.21.15.76:0/3854020993 >> [v2:172.21.15.76:6800/513615067,v1:172.21.15.76:6801/513615067] conn(0x7f58e00a1ba0 0x7f58e00a4060 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:39.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:39.142+0000 7f58fc6df640 1 --2- 172.21.15.76:0/3854020993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f58f4199120 0x7f58f41a3680 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:39.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:39.142+0000 7f58fc6df640 1 --2- 172.21.15.76:0/3854020993 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f58f4105360 0x7f58f4198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:39.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:39.142+0000 7f58fc6df640 1 --2- 172.21.15.76:0/3854020993 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58f4100290 0x7f58f4198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T06:06:39.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:39.142+0000 7f58fc6df640 1 -- 172.21.15.76:0/3854020993 >> 172.21.15.76:0/3854020993 conn(0x7f58f4077230 msgr2=0x7f58f40fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T06:06:39.148 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:39.142+0000 7f58fc6df640 1 -- 172.21.15.76:0/3854020993 shutdown_connections 2024-03-12T06:06:39.149 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-12T06:06:39.142+0000 7f58fc6df640 1 -- 172.21.15.76:0/3854020993 wait complete. 2024-03-12T06:06:39.166 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T06:06:39.196 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T06:06:39.229 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T06:06:39.263 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-12T06:06:39.264 DEBUG:teuthology.orchestra.run.smithi076:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T06:06:39.279 DEBUG:teuthology.orchestra.run.smithi082:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T06:06:39.295 DEBUG:teuthology.orchestra.run.smithi088:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T06:06:39.320 INFO:tasks.cephadm:Stopping all daemons... 2024-03-12T06:06:39.321 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-12T06:06:39.321 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mon.a 2024-03-12T06:06:39.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:39 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/4286636623' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T06:06:39.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:39 smithi082 ceph-mon[36382]: osdmap e99: 12 total, 12 up, 12 in 2024-03-12T06:06:39.447 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:39 smithi082 ceph-mon[36382]: from='client.? 172.21.15.76:0/3854020993' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T06:06:39.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:39 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/4286636623' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T06:06:39.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:39 smithi088 ceph-mon[36585]: osdmap e99: 12 total, 12 up, 12 in 2024-03-12T06:06:39.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:39 smithi088 ceph-mon[36585]: from='client.? 172.21.15.76:0/3854020993' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T06:06:39.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:39 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/4286636623' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T06:06:39.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:39 smithi076 ceph-mon[32095]: osdmap e99: 12 total, 12 up, 12 in 2024-03-12T06:06:39.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:39 smithi076 ceph-mon[32095]: from='client.? 172.21.15.76:0/3854020993' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T06:06:39.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:39 smithi076 systemd[1]: Stopping Ceph mon.a for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:06:39.989 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:39 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a[32091]: 2024-03-12T06:06:39.631+0000 7f96e7ee1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:06:39.990 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:39 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a[32091]: 2024-03-12T06:06:39.631+0000 7f96e7ee1700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-12T06:06:39.990 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:39 smithi076 podman[104843]: 2024-03-12 06:06:39.754083438 +0000 UTC m=+0.253948410 container died 9c347f27958bdc05fa1da6cd36fd0914a3ba9813bd5064b4b60d8b0d04bfed02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-12T06:06:40.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:40 smithi076 podman[104843]: 2024-03-12 06:06:40.140390608 +0000 UTC m=+0.640255584 container cleanup 9c347f27958bdc05fa1da6cd36fd0914a3ba9813bd5064b4b60d8b0d04bfed02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397) 2024-03-12T06:06:40.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 12 06:06:40 smithi076 bash[104843]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-a 2024-03-12T06:06:40.492 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mon.a.service' 2024-03-12T06:06:40.545 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:06:40.545 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-12T06:06:40.545 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-12T06:06:40.545 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mon.b 2024-03-12T06:06:40.851 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:40 smithi082 systemd[1]: Stopping Ceph mon.b for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:06:40.852 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:40 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-b[36378]: 2024-03-12T06:06:40.845+0000 7fad8fe9f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:06:40.852 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:40 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-b[36378]: 2024-03-12T06:06:40.845+0000 7fad8fe9f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-12T06:06:41.190 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:40 smithi082 podman[72016]: 2024-03-12 06:06:40.958130991 +0000 UTC m=+0.226187623 container died e81f0cd76ab4f45a2c3869d06239667eb606c6f0773baf490432b0a2be5b4826 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-12T06:06:41.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:41 smithi082 podman[72016]: 2024-03-12 06:06:41.337046009 +0000 UTC m=+0.605102632 container cleanup e81f0cd76ab4f45a2c3869d06239667eb606c6f0773baf490432b0a2be5b4826 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-12T06:06:41.446 INFO:journalctl@ceph.mon.b.smithi082.stdout:Mar 12 06:06:41 smithi082 bash[72016]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-b 2024-03-12T06:06:41.764 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mon.b.service' 2024-03-12T06:06:41.810 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:06:41.811 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-12T06:06:41.811 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-12T06:06:41.811 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mon.c 2024-03-12T06:06:42.136 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:41 smithi088 systemd[1]: Stopping Ceph mon.c for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:06:42.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:42 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-c[36567]: 2024-03-12T06:06:42.130+0000 7f3a245ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:06:42.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:42 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-c[36567]: 2024-03-12T06:06:42.130+0000 7f3a245ce700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-12T06:06:42.486 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:42 smithi088 podman[73698]: 2024-03-12 06:06:42.202907812 +0000 UTC m=+0.194437938 container died 9ca03031be8eabda09b54d41d2a60fa443f1867a08ba32b52415c3bcb6fc9105 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-12T06:06:42.829 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:42 smithi088 podman[73698]: 2024-03-12 06:06:42.58203969 +0000 UTC m=+0.573569809 container cleanup 9ca03031be8eabda09b54d41d2a60fa443f1867a08ba32b52415c3bcb6fc9105 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-94a5397) 2024-03-12T06:06:42.829 INFO:journalctl@ceph.mon.c.smithi088.stdout:Mar 12 06:06:42 smithi088 bash[73698]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mon-c 2024-03-12T06:06:42.940 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mon.c.service' 2024-03-12T06:06:42.991 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:06:42.991 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-12T06:06:42.992 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-12T06:06:42.992 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mgr.x 2024-03-12T06:06:43.336 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 06:06:43 smithi076 systemd[1]: Stopping Ceph mgr.x for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:06:43.609 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 06:06:43 smithi076 podman[104963]: 2024-03-12 06:06:43.334907642 +0000 UTC m=+0.152345386 container died 04d3f8c04590debd9ea085e82bc185eddc5ca97569d443fe9401f32c39e175a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-12T06:06:43.986 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 06:06:43 smithi076 podman[104963]: 2024-03-12 06:06:43.688628072 +0000 UTC m=+0.506065824 container cleanup 04d3f8c04590debd9ea085e82bc185eddc5ca97569d443fe9401f32c39e175a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5) 2024-03-12T06:06:43.986 INFO:journalctl@ceph.mgr.x.smithi076.stdout:Mar 12 06:06:43 smithi076 bash[104963]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-x 2024-03-12T06:06:44.107 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-12T06:06:44.144 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:06:44.145 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-12T06:06:44.145 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-12T06:06:44.145 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mgr.y 2024-03-12T06:06:44.447 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 06:06:44 smithi082 systemd[1]: Stopping Ceph mgr.y for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:06:44.729 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 06:06:44 smithi082 podman[72140]: 2024-03-12 06:06:44.453898155 +0000 UTC m=+0.118114019 container died 60fa6851228615ba2ec618fdcfd123a583a03cd05ade0ada32358ffee74e2c43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True) 2024-03-12T06:06:45.199 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 06:06:44 smithi082 podman[72140]: 2024-03-12 06:06:44.824368515 +0000 UTC m=+0.488584376 container cleanup 60fa6851228615ba2ec618fdcfd123a583a03cd05ade0ada32358ffee74e2c43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397) 2024-03-12T06:06:45.199 INFO:journalctl@ceph.mgr.y.smithi082.stdout:Mar 12 06:06:44 smithi082 bash[72140]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-y 2024-03-12T06:06:45.388 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-12T06:06:45.424 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:06:45.424 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-12T06:06:45.425 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-12T06:06:45.425 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mgr.z 2024-03-12T06:06:45.720 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 06:06:45 smithi088 systemd[1]: Stopping Ceph mgr.z for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:06:45.986 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 06:06:45 smithi088 podman[73819]: 2024-03-12 06:06:45.71556442 +0000 UTC m=+0.103072960 container died 89c69d18a0a82cae00bac66580d89f934463b0e73b4397d1f9e230acd919bec3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-12T06:06:46.325 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 06:06:46 smithi088 podman[73819]: 2024-03-12 06:06:46.078070125 +0000 UTC m=+0.465578662 container cleanup 89c69d18a0a82cae00bac66580d89f934463b0e73b4397d1f9e230acd919bec3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-12T06:06:46.325 INFO:journalctl@ceph.mgr.z.smithi088.stdout:Mar 12 06:06:46 smithi088 bash[73819]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mgr-z 2024-03-12T06:06:46.496 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mgr.z.service' 2024-03-12T06:06:46.532 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:06:46.532 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-12T06:06:46.533 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-12T06:06:46.533 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.0 2024-03-12T06:06:46.989 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 06:06:46 smithi076 systemd[1]: Stopping Ceph osd.0 for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:06:46.990 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 06:06:46 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-0[46549]: 2024-03-12T06:06:46.801+0000 7f367750d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:06:46.990 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 06:06:46 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-0[46549]: 2024-03-12T06:06:46.801+0000 7f367750d700 -1 osd.0 100 *** Got signal Terminated *** 2024-03-12T06:06:46.990 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 06:06:46 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-0[46549]: 2024-03-12T06:06:46.801+0000 7f367750d700 -1 osd.0 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T06:06:52.475 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 06:06:52 smithi076 podman[105085]: 2024-03-12 06:06:52.186829903 +0000 UTC m=+5.468303377 container died 0c7ed207da593fd56e1b99f75a415cdbe69b0a9175edce0270517b68dff345e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-12T06:06:52.739 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 06:06:52 smithi076 podman[105085]: 2024-03-12 06:06:52.586391714 +0000 UTC m=+5.867865189 container cleanup 0c7ed207da593fd56e1b99f75a415cdbe69b0a9175edce0270517b68dff345e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-12T06:06:52.739 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 06:06:52 smithi076 bash[105085]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-0 2024-03-12T06:06:53.151 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 06:06:52 smithi076 podman[105102]: 2024-03-12 06:06:52.821444133 +0000 UTC m=+0.624476314 container remove 0c7ed207da593fd56e1b99f75a415cdbe69b0a9175edce0270517b68dff345e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-0, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-12T06:06:53.429 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 06:06:53 smithi076 podman[105194]: 2024-03-12 06:06:53.049643388 +0000 UTC m=+0.018056332 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T06:06:53.429 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 06:06:53 smithi076 podman[105194]: 2024-03-12 06:06:53.155755674 +0000 UTC m=+0.124168600 container create dd4c8cee435aa87a92715c9935ecfc52d8baeac5f120c241845261a141289ce9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T06:06:53.430 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 06:06:53 smithi076 podman[105194]: 2024-03-12 06:06:53.35864371 +0000 UTC m=+0.327056637 container init dd4c8cee435aa87a92715c9935ecfc52d8baeac5f120c241845261a141289ce9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-12T06:06:53.430 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 06:06:53 smithi076 podman[105194]: 2024-03-12 06:06:53.36185537 +0000 UTC m=+0.330268306 container start dd4c8cee435aa87a92715c9935ecfc52d8baeac5f120c241845261a141289ce9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-12T06:06:53.430 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 06:06:53 smithi076 podman[105194]: 2024-03-12 06:06:53.425157267 +0000 UTC m=+0.393570203 container attach dd4c8cee435aa87a92715c9935ecfc52d8baeac5f120c241845261a141289ce9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-12T06:06:53.740 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 12 06:06:53 smithi076 podman[105194]: 2024-03-12 06:06:53.659699672 +0000 UTC m=+0.628112600 container died dd4c8cee435aa87a92715c9935ecfc52d8baeac5f120c241845261a141289ce9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T06:06:54.284 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.0.service' 2024-03-12T06:06:54.327 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:06:54.328 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-12T06:06:54.328 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-12T06:06:54.328 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.1 2024-03-12T06:06:54.615 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:06:54 smithi082 systemd[1]: Stopping Ceph osd.1 for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:06:54.946 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:06:54 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1[40981]: 2024-03-12T06:06:54.610+0000 7f98e3f79700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:06:54.946 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:06:54 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1[40981]: 2024-03-12T06:06:54.610+0000 7f98e3f79700 -1 osd.1 100 *** Got signal Terminated *** 2024-03-12T06:06:54.946 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:06:54 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1[40981]: 2024-03-12T06:06:54.610+0000 7f98e3f79700 -1 osd.1 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T06:07:00.282 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:07:00 smithi082 podman[72263]: 2024-03-12 06:07:00.015702373 +0000 UTC m=+5.510262904 container died 85f29769d3410d9dab1c01f51d688c8b7b0639bc96ccac086fdc94718ad5fbe0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397) 2024-03-12T06:07:00.640 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:07:00 smithi082 podman[72263]: 2024-03-12 06:07:00.368282088 +0000 UTC m=+5.862842596 container cleanup 85f29769d3410d9dab1c01f51d688c8b7b0639bc96ccac086fdc94718ad5fbe0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397) 2024-03-12T06:07:00.640 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:07:00 smithi082 bash[72263]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1 2024-03-12T06:07:00.946 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:07:00 smithi082 podman[72278]: 2024-03-12 06:07:00.636150173 +0000 UTC m=+0.615358734 container remove 85f29769d3410d9dab1c01f51d688c8b7b0639bc96ccac086fdc94718ad5fbe0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-12T06:07:01.236 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:07:00 smithi082 podman[72374]: 2024-03-12 06:07:00.877164661 +0000 UTC m=+0.017468266 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T06:07:01.236 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:07:01 smithi082 podman[72374]: 2024-03-12 06:07:01.003686198 +0000 UTC m=+0.143989793 container create e6ef5ca34d5881f3e7b7e0d890a735e6350a512b77fdea811bf411c3008ff1ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-03-12T06:07:01.530 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:07:01 smithi082 podman[72374]: 2024-03-12 06:07:01.231935016 +0000 UTC m=+0.372238631 container init e6ef5ca34d5881f3e7b7e0d890a735e6350a512b77fdea811bf411c3008ff1ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-12T06:07:01.531 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:07:01 smithi082 podman[72374]: 2024-03-12 06:07:01.23558932 +0000 UTC m=+0.375892933 container start e6ef5ca34d5881f3e7b7e0d890a735e6350a512b77fdea811bf411c3008ff1ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-12T06:07:01.531 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:07:01 smithi082 podman[72374]: 2024-03-12 06:07:01.331254727 +0000 UTC m=+0.471558343 container attach e6ef5ca34d5881f3e7b7e0d890a735e6350a512b77fdea811bf411c3008ff1ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-12T06:07:01.829 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:07:01 smithi082 podman[72374]: 2024-03-12 06:07:01.525964221 +0000 UTC m=+0.666267840 container died e6ef5ca34d5881f3e7b7e0d890a735e6350a512b77fdea811bf411c3008ff1ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T06:07:02.071 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.1.service' 2024-03-12T06:07:02.097 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:07:02 smithi082 podman[72399]: 2024-03-12 06:07:02.051208573 +0000 UTC m=+0.515019157 container remove e6ef5ca34d5881f3e7b7e0d890a735e6350a512b77fdea811bf411c3008ff1ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-12T06:07:02.098 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:07:02 smithi082 systemd[1]: ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-12T06:07:02.098 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:07:02 smithi082 systemd[1]: Stopped Ceph osd.1 for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T06:07:02.098 INFO:journalctl@ceph.osd.1.smithi082.stdout:Mar 12 06:07:02 smithi082 systemd[1]: ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.1.service: Consumed 15.111s CPU time. 2024-03-12T06:07:02.114 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:07:02.114 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-12T06:07:02.114 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-12T06:07:02.114 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.2 2024-03-12T06:07:02.427 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 06:07:02 smithi088 systemd[1]: Stopping Ceph osd.2 for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:07:02.736 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 06:07:02 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2[41708]: 2024-03-12T06:07:02.421+0000 7f3f4b827700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:07:02.736 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 06:07:02 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2[41708]: 2024-03-12T06:07:02.421+0000 7f3f4b827700 -1 osd.2 100 *** Got signal Terminated *** 2024-03-12T06:07:02.736 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 06:07:02 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2[41708]: 2024-03-12T06:07:02.421+0000 7f3f4b827700 -1 osd.2 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T06:07:08.179 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 06:07:07 smithi088 podman[73944]: 2024-03-12 06:07:07.908382358 +0000 UTC m=+5.608126337 container died 87243c63e32812b9e89eb7ce3c138dd97e60ffd49e4b7def620bfade4993c71f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-12T06:07:08.486 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 06:07:08 smithi088 podman[73944]: 2024-03-12 06:07:08.255662807 +0000 UTC m=+5.955406787 container cleanup 87243c63e32812b9e89eb7ce3c138dd97e60ffd49e4b7def620bfade4993c71f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-12T06:07:08.486 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 06:07:08 smithi088 bash[73944]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2 2024-03-12T06:07:08.870 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 06:07:08 smithi088 podman[73973]: 2024-03-12 06:07:08.540239596 +0000 UTC m=+0.631331300 container remove 87243c63e32812b9e89eb7ce3c138dd97e60ffd49e4b7def620bfade4993c71f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-12T06:07:09.190 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 06:07:08 smithi088 podman[74066]: 2024-03-12 06:07:08.866215673 +0000 UTC m=+0.110959819 container create 660927a852ff0f99d0a3a081b55e175b8d22afd8a57e02a378788352118d6bba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T06:07:09.190 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 06:07:08 smithi088 podman[74066]: 2024-03-12 06:07:08.772784037 +0000 UTC m=+0.017528146 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T06:07:09.191 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 06:07:09 smithi088 podman[74066]: 2024-03-12 06:07:09.110608113 +0000 UTC m=+0.355352230 container init 660927a852ff0f99d0a3a081b55e175b8d22afd8a57e02a378788352118d6bba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T06:07:09.191 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 06:07:09 smithi088 podman[74066]: 2024-03-12 06:07:09.113631834 +0000 UTC m=+0.358375945 container start 660927a852ff0f99d0a3a081b55e175b8d22afd8a57e02a378788352118d6bba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-12T06:07:09.191 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 06:07:09 smithi088 podman[74066]: 2024-03-12 06:07:09.185352525 +0000 UTC m=+0.430096630 container attach 660927a852ff0f99d0a3a081b55e175b8d22afd8a57e02a378788352118d6bba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2-deactivate, RELEASE=squid-94a5397, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-12T06:07:09.486 INFO:journalctl@ceph.osd.2.smithi088.stdout:Mar 12 06:07:09 smithi088 podman[74066]: 2024-03-12 06:07:09.401720089 +0000 UTC m=+0.646464211 container died 660927a852ff0f99d0a3a081b55e175b8d22afd8a57e02a378788352118d6bba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-03-12T06:07:09.937 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.2.service' 2024-03-12T06:07:09.979 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:07:09.979 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-12T06:07:09.979 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-12T06:07:09.980 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.3 2024-03-12T06:07:10.305 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:10 smithi076 systemd[1]: Stopping Ceph osd.3 for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:07:10.739 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:10 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3[53101]: 2024-03-12T06:07:10.299+0000 7f39d0499700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:07:10.740 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:10 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3[53101]: 2024-03-12T06:07:10.299+0000 7f39d0499700 -1 osd.3 100 *** Got signal Terminated *** 2024-03-12T06:07:10.740 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:10 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3[53101]: 2024-03-12T06:07:10.299+0000 7f39d0499700 -1 osd.3 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T06:07:13.489 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:13 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3[53101]: 2024-03-12T06:07:13.105+0000 7f39cd28e700 -1 osd.3 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:47.519450+0000 front 2024-03-12T06:06:47.519470+0000 (oldest deadline 2024-03-12T06:07:12.219509+0000) 2024-03-12T06:07:14.489 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:14 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3[53101]: 2024-03-12T06:07:14.099+0000 7f39cd28e700 -1 osd.3 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:47.519450+0000 front 2024-03-12T06:06:47.519470+0000 (oldest deadline 2024-03-12T06:07:12.219509+0000) 2024-03-12T06:07:15.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:14 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:14.842+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:15.489 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:15 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3[53101]: 2024-03-12T06:07:15.108+0000 7f39cd28e700 -1 osd.3 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:47.519450+0000 front 2024-03-12T06:06:47.519470+0000 (oldest deadline 2024-03-12T06:07:12.219509+0000) 2024-03-12T06:07:15.931 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:15 smithi076 podman[105317]: 2024-03-12 06:07:15.645158086 +0000 UTC m=+5.468805915 container died 59da77640409590aa7f48765ed07979ceca8063e75e7af7c37f54f6947237a7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-12T06:07:15.932 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:15 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:15.619+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:15.932 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:15 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:15.629+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:16.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:15 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:15.830+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:16.236 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:15 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:15.927+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:16.239 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:16 smithi076 podman[105317]: 2024-03-12 06:07:16.017378488 +0000 UTC m=+5.841026318 container cleanup 59da77640409590aa7f48765ed07979ceca8063e75e7af7c37f54f6947237a7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T06:07:16.239 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:16 smithi076 bash[105317]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3 2024-03-12T06:07:16.630 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:16 smithi076 podman[105334]: 2024-03-12 06:07:16.288358069 +0000 UTC m=+0.641548875 container remove 59da77640409590aa7f48765ed07979ceca8063e75e7af7c37f54f6947237a7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397) 2024-03-12T06:07:16.885 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:16 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:16.647+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:16.886 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:16 smithi076 podman[105429]: 2024-03-12 06:07:16.52905104 +0000 UTC m=+0.017497375 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T06:07:16.886 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:16 smithi076 podman[105429]: 2024-03-12 06:07:16.64509531 +0000 UTC m=+0.133541642 container create 15bb4457f18ea59e2c044f4d64e0b21c381fe74de35aabafb604d210c29f20d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-12T06:07:16.886 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:16 smithi076 podman[105429]: 2024-03-12 06:07:16.881143064 +0000 UTC m=+0.369589401 container init 15bb4457f18ea59e2c044f4d64e0b21c381fe74de35aabafb604d210c29f20d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T06:07:16.889 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:16 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:16.612+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:17.175 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:16 smithi076 podman[105429]: 2024-03-12 06:07:16.884225208 +0000 UTC m=+0.372671545 container start 15bb4457f18ea59e2c044f4d64e0b21c381fe74de35aabafb604d210c29f20d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-12T06:07:17.175 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:16 smithi076 podman[105429]: 2024-03-12 06:07:16.956279034 +0000 UTC m=+0.444725378 container attach 15bb4457f18ea59e2c044f4d64e0b21c381fe74de35aabafb604d210c29f20d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-12T06:07:17.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:16 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:16.864+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:17.236 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:16 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:16.884+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:17.449 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 12 06:07:17 smithi076 podman[105429]: 2024-03-12 06:07:17.170796239 +0000 UTC m=+0.659242573 container died 15bb4457f18ea59e2c044f4d64e0b21c381fe74de35aabafb604d210c29f20d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-12T06:07:17.739 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:17 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:17.444+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:17.740 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:17 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:17.682+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:17.857 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.3.service' 2024-03-12T06:07:17.899 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:07:17.899 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-12T06:07:17.899 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-12T06:07:17.899 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.4 2024-03-12T06:07:17.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:17 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:17.850+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:17.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:17 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:17.625+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:18.196 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:17 smithi082 systemd[1]: Stopping Ceph osd.4 for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:07:18.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:17 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:17.892+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:18.696 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:18 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4[47711]: 2024-03-12T06:07:18.193+0000 7f407f4d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:07:18.696 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:18 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4[47711]: 2024-03-12T06:07:18.193+0000 7f407f4d8700 -1 osd.4 100 *** Got signal Terminated *** 2024-03-12T06:07:18.697 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:18 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4[47711]: 2024-03-12T06:07:18.193+0000 7f407f4d8700 -1 osd.4 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T06:07:18.711 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:18.476+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:18.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:18 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:18.820+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:18.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:18 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:18.634+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:18.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:18 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:18.707+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:19.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:18 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:18.896+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:19.739 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:19 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:19.479+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:19.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:19 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:19.782+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:19.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:19 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:19.638+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:20.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:19 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:19.850+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:20.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:19 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:19.746+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:20.731 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:20 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:20.435+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:20.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:20 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:20.738+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:20.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:20 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:20.594+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:20.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:20 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:20.726+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:21.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:20 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:20.880+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:21.695 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:21 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:21.403+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:21.696 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:21 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:21.403+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:21.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:21 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:21.767+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:21.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:21 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:21.767+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.466981+0000 front 2024-03-12T06:06:55.467004+0000 (oldest deadline 2024-03-12T06:07:20.767130+0000) 2024-03-12T06:07:21.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:21 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:21.594+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:21.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:21 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:21.594+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:21.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:21 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:21.691+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:22.134 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:21 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4[47711]: 2024-03-12T06:07:21.959+0000 7f40782c5700 -1 osd.4 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.541912+0000 front 2024-03-12T06:06:56.542119+0000 (oldest deadline 2024-03-12T06:07:21.242046+0000) 2024-03-12T06:07:22.135 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:21 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:21.844+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:22.446 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:22 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:22.129+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:22.739 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:22 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:22.673+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:22.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:22 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:22.443+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:22.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:22 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:22.443+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:22.987 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:22 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:22.745+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:23.063 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:22 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:22.745+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.466981+0000 front 2024-03-12T06:06:55.467004+0000 (oldest deadline 2024-03-12T06:07:20.767130+0000) 2024-03-12T06:07:23.063 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:22 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:22.612+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:23.063 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:22 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:22.612+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:23.196 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:22 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4[47711]: 2024-03-12T06:07:22.912+0000 7f40782c5700 -1 osd.4 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.541912+0000 front 2024-03-12T06:06:56.542119+0000 (oldest deadline 2024-03-12T06:07:21.242046+0000) 2024-03-12T06:07:23.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:23 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:23.108+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:23.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:22 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:22.859+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:23.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:22 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:22.859+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:23.649 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:23 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:23.393+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:23.650 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:23 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:23.393+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:23.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:23 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:23.729+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:23.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:23 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:23.729+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.466981+0000 front 2024-03-12T06:06:55.467004+0000 (oldest deadline 2024-03-12T06:07:20.767130+0000) 2024-03-12T06:07:23.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:23 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:23.655+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:23.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:23 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:23.655+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:23.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:23 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:23.645+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:23.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:23 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:23.645+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:24.032 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:23 smithi082 podman[72490]: 2024-03-12 06:07:23.738722396 +0000 UTC m=+5.657272078 container died 23b934fbe8eec4ca5062c5220de3f347744def1dc5e9910eacbb438538beffe2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T06:07:24.033 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:23 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:23.826+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:24.033 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:23 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:23.826+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:24.382 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:24 smithi082 podman[72490]: 2024-03-12 06:07:24.126634907 +0000 UTC m=+6.045184569 container cleanup 23b934fbe8eec4ca5062c5220de3f347744def1dc5e9910eacbb438538beffe2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, ceph=True) 2024-03-12T06:07:24.382 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:24 smithi082 bash[72490]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4 2024-03-12T06:07:24.382 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:24 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:24.147+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:24.697 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:24 smithi082 podman[72507]: 2024-03-12 06:07:24.377604397 +0000 UTC m=+0.636407974 container remove 23b934fbe8eec4ca5062c5220de3f347744def1dc5e9910eacbb438538beffe2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-12T06:07:24.697 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:24 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:24.439+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:24.697 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:24 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:24.439+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:24.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:24 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:24.686+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:24.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:24 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:24.686+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.466981+0000 front 2024-03-12T06:06:55.467004+0000 (oldest deadline 2024-03-12T06:07:20.767130+0000) 2024-03-12T06:07:24.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:24 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:24.678+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:24.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:24 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:24.678+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:24.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:24 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:24.691+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:24.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:24 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:24.691+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:25.035 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:24 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:24.832+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:25.036 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:24 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:24.832+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:25.036 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:24 smithi082 podman[72596]: 2024-03-12 06:07:24.634674683 +0000 UTC m=+0.017606699 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T06:07:25.036 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:24 smithi082 podman[72596]: 2024-03-12 06:07:24.745121457 +0000 UTC m=+0.128053462 container create 605854196afeae0c736fe7d272cfe3ddb42f6a8e6efbf0216eb3e81216a1ffde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T06:07:25.036 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:24 smithi082 podman[72596]: 2024-03-12 06:07:24.930737623 +0000 UTC m=+0.313669637 container init 605854196afeae0c736fe7d272cfe3ddb42f6a8e6efbf0216eb3e81216a1ffde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4-deactivate, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-12T06:07:25.036 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:24 smithi082 podman[72596]: 2024-03-12 06:07:24.933951283 +0000 UTC m=+0.316883290 container start 605854196afeae0c736fe7d272cfe3ddb42f6a8e6efbf0216eb3e81216a1ffde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-12T06:07:25.036 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:25 smithi082 podman[72596]: 2024-03-12 06:07:25.03110938 +0000 UTC m=+0.414041400 container attach 605854196afeae0c736fe7d272cfe3ddb42f6a8e6efbf0216eb3e81216a1ffde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-12T06:07:25.447 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:25 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:25.178+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:25.447 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:25 smithi082 podman[72596]: 2024-03-12 06:07:25.224856402 +0000 UTC m=+0.607788417 container died 605854196afeae0c736fe7d272cfe3ddb42f6a8e6efbf0216eb3e81216a1ffde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-12T06:07:25.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:25 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:25.111+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:25.739 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:25 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:25.480+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:25.739 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:25 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:25.480+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:25.740 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:25 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:25.730+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:25.740 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:25 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:25.730+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:25.822 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.4.service' 2024-03-12T06:07:25.846 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:25 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:25.784+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:25.847 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:25 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:25.784+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:25.847 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:25 smithi082 podman[72622]: 2024-03-12 06:07:25.792891636 +0000 UTC m=+0.562833016 container remove 605854196afeae0c736fe7d272cfe3ddb42f6a8e6efbf0216eb3e81216a1ffde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T06:07:25.847 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:25 smithi082 systemd[1]: ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-12T06:07:25.847 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:25 smithi082 systemd[1]: Stopped Ceph osd.4 for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T06:07:25.848 INFO:journalctl@ceph.osd.4.smithi082.stdout:Mar 12 06:07:25 smithi082 systemd[1]: ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.4.service: Consumed 12.619s CPU time. 2024-03-12T06:07:25.862 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:07:25.863 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-12T06:07:25.863 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-12T06:07:25.863 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.5 2024-03-12T06:07:25.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:25 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:25.678+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:25.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:25 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:25.678+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.466981+0000 front 2024-03-12T06:06:55.467004+0000 (oldest deadline 2024-03-12T06:07:20.767130+0000) 2024-03-12T06:07:25.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:25 smithi088 systemd[1]: Stopping Ceph osd.5 for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:07:25.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:25 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:25.715+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:25.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:25 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:25.715+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:26.446 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:26 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:26.156+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:26.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:26 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:26.068+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:26.486 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:26 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:26.164+0000 7fd72b7bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:07:26.486 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:26 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:26.164+0000 7fd72b7bc700 -1 osd.5 100 *** Got signal Terminated *** 2024-03-12T06:07:26.486 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:26 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:26.164+0000 7fd72b7bc700 -1 osd.5 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T06:07:26.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:26 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:26.701+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:26.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:26 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:26.701+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.466981+0000 front 2024-03-12T06:06:55.467004+0000 (oldest deadline 2024-03-12T06:07:20.767130+0000) 2024-03-12T06:07:26.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:26 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:26.760+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:26.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:26 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:26.760+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:26.989 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:26 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:26.520+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:26.989 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:26 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:26.520+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:26.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:26 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:26.689+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:26.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:26 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:26.689+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:27.155 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:26 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:26.813+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:27.155 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:26 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:26.813+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:27.446 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:27 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:27.150+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:27.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:27 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:27.050+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:27.739 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:27 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:27.482+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:27.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:27 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:27.482+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:27.740 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:27 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:27.710+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:27.740 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:27 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:27.710+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:27.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:27 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:27.726+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:27.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:27 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:27.726+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.466981+0000 front 2024-03-12T06:06:55.467004+0000 (oldest deadline 2024-03-12T06:07:20.767130+0000) 2024-03-12T06:07:27.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:27 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:27.788+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:27.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:27 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:27.788+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:28.122 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:27 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:27.810+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:28.122 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:27 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:27.810+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:28.446 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:28 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:28.117+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:28.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:28 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:28.011+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:28.989 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:28 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:28.514+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:28.990 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:28 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:28.514+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:28.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:28 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:28.759+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:28.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:28 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:28.759+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:29.159 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:28 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:28.800+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:29.159 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:28 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:28.800+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:29.160 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:28 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:28.800+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:29.236 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:28 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:28.976+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:29.237 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:28 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:28.761+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:29.237 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:28 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:28.761+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.466981+0000 front 2024-03-12T06:06:55.467004+0000 (oldest deadline 2024-03-12T06:07:20.767130+0000) 2024-03-12T06:07:29.237 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:28 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:28.761+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:04.268021+0000 front 2024-03-12T06:07:04.267978+0000 (oldest deadline 2024-03-12T06:07:28.268684+0000) 2024-03-12T06:07:29.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:28 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:28.797+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:29.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:28 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:28.797+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:29.446 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:29 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:29.155+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:29.739 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:29 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:29.477+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:29.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:29 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:29.477+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:29.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:29 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:29.720+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:29.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:29 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:29.720+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.466981+0000 front 2024-03-12T06:06:55.467004+0000 (oldest deadline 2024-03-12T06:07:20.767130+0000) 2024-03-12T06:07:29.986 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:29 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:29.720+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:04.268021+0000 front 2024-03-12T06:07:04.267978+0000 (oldest deadline 2024-03-12T06:07:28.268684+0000) 2024-03-12T06:07:29.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:29 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:29.775+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:29.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:29 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:29.775+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:30.143 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:29 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:29.780+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:30.143 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:29 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:29.780+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:30.143 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:29 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:29.780+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:30.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:29 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:29.769+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:30.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:29 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:29.769+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:30.446 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:30 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:30.138+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:30.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:29 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:29.991+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:30.739 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:30 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:30.434+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:30.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:30 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:30.434+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:30.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:30 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:30.434+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:05.459732+0000 front 2024-03-12T06:07:05.459614+0000 (oldest deadline 2024-03-12T06:07:30.159456+0000) 2024-03-12T06:07:31.137 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:30 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:30.811+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:31.137 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:30 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:30.811+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:31.137 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:30 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:30.811+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:31.236 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:30 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:30.951+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:31.236 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:30 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:30.758+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:49.566699+0000 front 2024-03-12T06:06:49.566683+0000 (oldest deadline 2024-03-12T06:07:15.466539+0000) 2024-03-12T06:07:31.237 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:30 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:30.758+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.466981+0000 front 2024-03-12T06:06:55.467004+0000 (oldest deadline 2024-03-12T06:07:20.767130+0000) 2024-03-12T06:07:31.237 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:30 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5[48682]: 2024-03-12T06:07:30.758+0000 7fd7285b1700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:04.268021+0000 front 2024-03-12T06:07:04.267978+0000 (oldest deadline 2024-03-12T06:07:28.268684+0000) 2024-03-12T06:07:31.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:30 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:30.786+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:31.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:30 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:30.786+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:31.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:30 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:30.786+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:31.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:30 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:30.816+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:31.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:30 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:30.816+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:31.446 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:31 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:31.132+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:31.739 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:31 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:31.467+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:31.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:31 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:31.467+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:31.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:31 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:31.467+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:05.459732+0000 front 2024-03-12T06:07:05.459614+0000 (oldest deadline 2024-03-12T06:07:30.159456+0000) 2024-03-12T06:07:31.887 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:31 smithi088 podman[74172]: 2024-03-12 06:07:31.599003244 +0000 UTC m=+5.548894639 container died 36471a80adc58ebcf066bd2b57aa7066ad1b641bc3fac6ad615965be77be7807 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, ceph=True, org.label-schema.build-date=20240307) 2024-03-12T06:07:31.887 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:31 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:31.746+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:31.888 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:31 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:31.746+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:31.888 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:31 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:31.746+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:32.171 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:31 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:31.852+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:32.172 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:31 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:31.852+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:32.172 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:31 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:31.852+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:32.204 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:31 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:31.987+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:32.204 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:31 smithi088 podman[74172]: 2024-03-12 06:07:31.973457373 +0000 UTC m=+5.923348767 container cleanup 36471a80adc58ebcf066bd2b57aa7066ad1b641bc3fac6ad615965be77be7807 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-12T06:07:32.204 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:31 smithi088 bash[74172]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5 2024-03-12T06:07:32.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:31 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:31.796+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:32.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:31 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:31.796+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:32.446 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:32 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:32.166+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:32.447 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:32 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:32.166+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:32.486 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:32 smithi088 podman[74189]: 2024-03-12 06:07:32.199638314 +0000 UTC m=+0.602899831 container remove 36471a80adc58ebcf066bd2b57aa7066ad1b641bc3fac6ad615965be77be7807 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-12T06:07:32.783 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:32 smithi088 podman[74275]: 2024-03-12 06:07:32.430599466 +0000 UTC m=+0.017370227 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T06:07:32.783 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:32 smithi088 podman[74275]: 2024-03-12 06:07:32.5593243 +0000 UTC m=+0.146095093 container create 4be3ebcf6c934280a6090c873396184218361037f93279f7678c72867eb9cc6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T06:07:32.783 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:32 smithi088 podman[74275]: 2024-03-12 06:07:32.778166101 +0000 UTC m=+0.364936870 container init 4be3ebcf6c934280a6090c873396184218361037f93279f7678c72867eb9cc6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-12T06:07:32.783 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:32 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:32.731+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:32.783 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:32 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:32.731+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:32.784 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:32 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:32.731+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:32.801 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:32 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:32.511+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:32.801 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:32 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:32.511+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:32.801 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:32 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:32.511+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:05.459732+0000 front 2024-03-12T06:07:05.459614+0000 (oldest deadline 2024-03-12T06:07:30.159456+0000) 2024-03-12T06:07:33.072 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:32 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:32.990+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:33.073 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:32 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:32.990+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:33.073 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:32 smithi088 podman[74275]: 2024-03-12 06:07:32.781230903 +0000 UTC m=+0.368001655 container start 4be3ebcf6c934280a6090c873396184218361037f93279f7678c72867eb9cc6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-12T06:07:33.073 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:32 smithi088 podman[74275]: 2024-03-12 06:07:32.861472763 +0000 UTC m=+0.448243532 container attach 4be3ebcf6c934280a6090c873396184218361037f93279f7678c72867eb9cc6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-12T06:07:33.127 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:32 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:32.853+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:33.127 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:32 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:32.853+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:33.127 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:32 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:32.853+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:33.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:32 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:32.796+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:33.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:32 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:32.796+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:33.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:32 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:32.796+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:33.338 INFO:journalctl@ceph.osd.5.smithi088.stdout:Mar 12 06:07:33 smithi088 podman[74275]: 2024-03-12 06:07:33.068655682 +0000 UTC m=+0.655426427 container died 4be3ebcf6c934280a6090c873396184218361037f93279f7678c72867eb9cc6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-5-deactivate, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-03-12T06:07:33.446 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:33 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:33.121+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:33.447 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:33 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:33.121+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:33.615 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.5.service' 2024-03-12T06:07:33.654 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:07:33.654 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-12T06:07:33.655 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-12T06:07:33.655 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.6 2024-03-12T06:07:33.739 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:33 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:33.468+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:33.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:33 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:33.468+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:33.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:33 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:33.468+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:05.459732+0000 front 2024-03-12T06:07:05.459614+0000 (oldest deadline 2024-03-12T06:07:30.159456+0000) 2024-03-12T06:07:33.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:33 smithi076 systemd[1]: Stopping Ceph osd.6 for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:07:33.983 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:33 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:33.727+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:33.983 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:33 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:33.727+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:33.983 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:33 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:33.727+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:34.108 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:33 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:33.827+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:34.108 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:33 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:33.827+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:34.108 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:33 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:33.827+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:34.236 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:33 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:33.977+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:34.236 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:33 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:33.977+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:34.239 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:33 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:33.975+0000 7fa731c61700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:07:34.239 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:33 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:33.975+0000 7fa731c61700 -1 osd.6 100 *** Got signal Terminated *** 2024-03-12T06:07:34.239 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:33 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:33.975+0000 7fa731c61700 -1 osd.6 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T06:07:34.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:33 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:33.816+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:34.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:33 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:33.816+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:34.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:33 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:33.816+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:34.446 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:34 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:34.102+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:34.446 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:34 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:34.102+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:34.739 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:34 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:34.435+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:34.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:34 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:34.435+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:34.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:34 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:34.435+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:05.459732+0000 front 2024-03-12T06:07:05.459614+0000 (oldest deadline 2024-03-12T06:07:30.159456+0000) 2024-03-12T06:07:35.032 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:34 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:34.776+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:35.032 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:34 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:34.776+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:35.032 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:34 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:34.776+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:35.196 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:35 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:35.069+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:35.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:35 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:35.069+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:35.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:34 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:34.876+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:35.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:34 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:34.876+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:35.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:34 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:34.876+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:35.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:34 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:34.828+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:35.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:34 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:34.828+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:35.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:34 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:34.828+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:35.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:35 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:35.027+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:35.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:35 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:35.027+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:35.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:35.388+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:35.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:35.388+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:35.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:35.388+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:05.459732+0000 front 2024-03-12T06:07:05.459614+0000 (oldest deadline 2024-03-12T06:07:30.159456+0000) 2024-03-12T06:07:36.037 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:35 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:35.741+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:36.038 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:35 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:35.741+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:36.038 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:35 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:35.741+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:36.196 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:36 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:36.076+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:36.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:36 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:36.076+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:36.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:35 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:35.913+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:36.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:35 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:35.913+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:36.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:35 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:35.913+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:36.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:35.829+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:36.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:35.829+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:36.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:35.829+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:36.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:36 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:36.032+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:36.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:36 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:36.032+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:36.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:36 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:36.376+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:36.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:36 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:36.376+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:36.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:36 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:36.376+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:05.459732+0000 front 2024-03-12T06:07:05.459614+0000 (oldest deadline 2024-03-12T06:07:30.159456+0000) 2024-03-12T06:07:36.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:36 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:36.724+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:36.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:36 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:36.724+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:36.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:36 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:36.724+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:37.196 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:37 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:37.073+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:37.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:37 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:37.073+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:37.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:37 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:37.073+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.211375+0000 front 2024-03-12T06:07:15.211502+0000 (oldest deadline 2024-03-12T06:07:36.911439+0000) 2024-03-12T06:07:37.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:36 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:36.892+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:37.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:36 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:36.892+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:37.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:36 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:36.892+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:37.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:36 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:36.892+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:37.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:36 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:36.788+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:37.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:36 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:36.788+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:37.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:36 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:36.788+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:37.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:37 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:37.053+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:37.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:37 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:37.053+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:37.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:37 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:37.053+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:37.739 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:37 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:37.419+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:37.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:37 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:37.419+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:37.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:37 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:37.419+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:05.459732+0000 front 2024-03-12T06:07:05.459614+0000 (oldest deadline 2024-03-12T06:07:30.159456+0000) 2024-03-12T06:07:38.061 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:37 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:37.763+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:38.061 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:37 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:37.763+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:38.061 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:37 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:37.763+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:38.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:37 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:37.854+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:38.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:37 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:37.854+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:38.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:37 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:37.854+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:38.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:37 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:37.854+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:38.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:38 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:38.093+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:38.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:38 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:38.093+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:38.198 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:38 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:38.093+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.211375+0000 front 2024-03-12T06:07:15.211502+0000 (oldest deadline 2024-03-12T06:07:36.911439+0000) 2024-03-12T06:07:38.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:37 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:37.780+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:38.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:37 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:37.780+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:38.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:37 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:37.780+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:38.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:38 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:38.056+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:38.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:38 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:38.056+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:38.487 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:38 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:38.056+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:38.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:38 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:38.399+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.757532+0000 front 2024-03-12T06:06:50.757497+0000 (oldest deadline 2024-03-12T06:07:16.657689+0000) 2024-03-12T06:07:38.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:38 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:38.399+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.658374+0000 front 2024-03-12T06:06:56.658416+0000 (oldest deadline 2024-03-12T06:07:21.358389+0000) 2024-03-12T06:07:38.740 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:38 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6[60104]: 2024-03-12T06:07:38.399+0000 7fa72aa4e700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:05.459732+0000 front 2024-03-12T06:07:05.459614+0000 (oldest deadline 2024-03-12T06:07:30.159456+0000) 2024-03-12T06:07:39.099 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:38 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:38.807+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:39.100 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:38 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:38.807+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:39.100 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:38 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:38.807+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:39.125 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:38 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:38.867+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:39.126 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:38 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:38.867+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:39.126 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:38 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:38.867+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:39.126 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:38 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:38.867+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:39.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:38 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:38.763+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:39.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:38 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:38.763+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:39.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:38 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:38.763+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:39.446 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:39 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:39.120+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:39.446 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:39 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:39.120+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:39.447 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:39 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:39.120+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.211375+0000 front 2024-03-12T06:07:15.211502+0000 (oldest deadline 2024-03-12T06:07:36.911439+0000) 2024-03-12T06:07:39.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:39 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:39.094+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:39.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:39 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:39.094+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:39.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:39 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:39.094+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:39.674 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:39 smithi076 podman[105547]: 2024-03-12 06:07:39.413653146 +0000 UTC m=+5.564620482 container died e4f91a3d2dca17659a586b3ffe67afce79fa31e67b2547afa949b13f8b9a50b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True) 2024-03-12T06:07:39.974 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:39 smithi076 podman[105547]: 2024-03-12 06:07:39.76024547 +0000 UTC m=+5.911212806 container cleanup e4f91a3d2dca17659a586b3ffe67afce79fa31e67b2547afa949b13f8b9a50b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-12T06:07:39.974 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:39 smithi076 bash[105547]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6 2024-03-12T06:07:39.975 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:39 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:39.730+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:39.975 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:39 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:39.730+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:39.975 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:39 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:39.730+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:40.072 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:39 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:39.759+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:40.072 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:39 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:39.759+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:40.072 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:39 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:39.759+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:40.196 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:40 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:40.094+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:40.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:40 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:40.094+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:40.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:40 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:40.094+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.211375+0000 front 2024-03-12T06:07:15.211502+0000 (oldest deadline 2024-03-12T06:07:36.911439+0000) 2024-03-12T06:07:40.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:39 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:39.855+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:40.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:39 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:39.855+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:40.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:39 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:39.855+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:40.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:39 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:39.855+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:40.241 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:39 smithi076 podman[105565]: 2024-03-12 06:07:39.969917543 +0000 UTC m=+0.552738844 container remove e4f91a3d2dca17659a586b3ffe67afce79fa31e67b2547afa949b13f8b9a50b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-12T06:07:40.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:40 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:40.066+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:40.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:40 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:40.066+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:40.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:40 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:40.066+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:40.686 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:40 smithi076 podman[105656]: 2024-03-12 06:07:40.283581985 +0000 UTC m=+0.017868903 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T06:07:40.687 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:40 smithi076 podman[105656]: 2024-03-12 06:07:40.387653866 +0000 UTC m=+0.121940783 container create 7a4f1de4890b995f61a1a1644df1479089864e92745c120c3ae03ffc9d2063a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6-deactivate, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-12T06:07:40.687 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:40 smithi076 podman[105656]: 2024-03-12 06:07:40.607501792 +0000 UTC m=+0.341788723 container init 7a4f1de4890b995f61a1a1644df1479089864e92745c120c3ae03ffc9d2063a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T06:07:40.687 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:40 smithi076 podman[105656]: 2024-03-12 06:07:40.610594519 +0000 UTC m=+0.344881442 container start 7a4f1de4890b995f61a1a1644df1479089864e92745c120c3ae03ffc9d2063a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-94a5397, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-12T06:07:40.687 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:40 smithi076 podman[105656]: 2024-03-12 06:07:40.682296543 +0000 UTC m=+0.416583478 container attach 7a4f1de4890b995f61a1a1644df1479089864e92745c120c3ae03ffc9d2063a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6-deactivate, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T06:07:40.989 INFO:journalctl@ceph.osd.6.smithi076.stdout:Mar 12 06:07:40 smithi076 podman[105656]: 2024-03-12 06:07:40.900374092 +0000 UTC m=+0.634661002 container died 7a4f1de4890b995f61a1a1644df1479089864e92745c120c3ae03ffc9d2063a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-12T06:07:40.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:40 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:40.705+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:40.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:40 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:40.705+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:40.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:40 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:40.705+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:40.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:40 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:40.705+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:41.081 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:40 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:40.776+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:41.081 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:40 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:40.776+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:41.081 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:40 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:40.776+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:41.196 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:41.057+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:41.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:41.057+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:41.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:41.057+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.211375+0000 front 2024-03-12T06:07:15.211502+0000 (oldest deadline 2024-03-12T06:07:36.911439+0000) 2024-03-12T06:07:41.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:40 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:40.880+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:41.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:40 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:40.880+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:41.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:40 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:40.880+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:41.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:40 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:40.880+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:41.486 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.6.service' 2024-03-12T06:07:41.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:41 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:41.075+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:41.487 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:41 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:41.075+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:41.487 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:41 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:41.075+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:41.528 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:07:41.528 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-12T06:07:41.529 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-12T06:07:41.529 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.7 2024-03-12T06:07:41.739 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:41 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:41.671+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:41.740 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:41 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:41.671+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:41.740 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:41 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:41.671+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:41.740 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:41 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:41.671+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:41.823 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:41 smithi082 systemd[1]: Stopping Ceph osd.7 for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:07:42.086 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:41.846+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:42.086 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:41.846+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:42.086 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:41.846+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:42.087 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:41.846+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:42.087 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:41.818+0000 7fd14e1e3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:07:42.087 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:41.818+0000 7fd14e1e3700 -1 osd.7 100 *** Got signal Terminated *** 2024-03-12T06:07:42.087 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:41.818+0000 7fd14e1e3700 -1 osd.7 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T06:07:42.120 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:41 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:41.788+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:42.120 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:41 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:41.788+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:42.120 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:41 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:41.788+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:42.446 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:42.081+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:42.447 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:42.081+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:42.447 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:42.081+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.211375+0000 front 2024-03-12T06:07:15.211502+0000 (oldest deadline 2024-03-12T06:07:36.911439+0000) 2024-03-12T06:07:42.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:42 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:42.114+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:42.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:42 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:42.114+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:42.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:42 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:42.114+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:42.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:42 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:42.709+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:42.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:42 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:42.709+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:42.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:42 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:42.709+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:42.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:42 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:42.709+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:43.108 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:42 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:42.837+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:43.109 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:42 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:42.837+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:43.109 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:42 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:42.837+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:43.196 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:43 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:43.049+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:43.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:43 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:43.049+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:43.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:43 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:43.049+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.211375+0000 front 2024-03-12T06:07:15.211502+0000 (oldest deadline 2024-03-12T06:07:36.911439+0000) 2024-03-12T06:07:43.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:42.884+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:43.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:42.884+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:43.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:42.884+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:43.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:42 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:42.884+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:43.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:43 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:43.103+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:43.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:43 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:43.103+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:43.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:43 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:43.103+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:43.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:43 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:43.726+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:43.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:43 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:43.726+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:43.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:43 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:43.726+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:43.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:43 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:43.726+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:44.144 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:43 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:43.841+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:44.144 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:43 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:43.841+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:44.144 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:43 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:43.841+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:44.196 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:44 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:44.031+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:44.196 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:44 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:44.031+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:44.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:44 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:44.031+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.211375+0000 front 2024-03-12T06:07:15.211502+0000 (oldest deadline 2024-03-12T06:07:36.911439+0000) 2024-03-12T06:07:44.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:43 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:43.869+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:44.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:43 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:43.869+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:44.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:43 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:43.869+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:44.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:43 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:43.869+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:44.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:44 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:44.138+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:44.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:44 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:44.138+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:44.487 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:44 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:44.138+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:45.156 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:44 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:44.808+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:45.156 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:44 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:44.808+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:45.156 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:44 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:44.808+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:45.196 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:44 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:44.989+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:45.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:44 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:44.989+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:45.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:44 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:44.989+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.211375+0000 front 2024-03-12T06:07:15.211502+0000 (oldest deadline 2024-03-12T06:07:36.911439+0000) 2024-03-12T06:07:45.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:44 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:44.843+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:45.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:44 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:44.843+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:45.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:44 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:44.843+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:45.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:44 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:44.843+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:45.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:44 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:44.772+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:45.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:44 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:44.772+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:45.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:44 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:44.772+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:45.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:44 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:44.772+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:45.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:45 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:45.150+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:45.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:45 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:45.150+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:45.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:45 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:45.150+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:45.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:45 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:45.150+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.636364+0000 front 2024-03-12T06:07:22.636375+0000 (oldest deadline 2024-03-12T06:07:44.936174+0000) 2024-03-12T06:07:45.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:45 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:45.733+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:45.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:45 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:45.733+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:45.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:45 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:45.733+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:45.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:45 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:45.733+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:45.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:45 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:45.733+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:46.196 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:45 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:45.987+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:56.608020+0000 front 2024-03-12T06:06:56.608041+0000 (oldest deadline 2024-03-12T06:07:21.908240+0000) 2024-03-12T06:07:46.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:45 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:45.987+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:03.609274+0000 front 2024-03-12T06:07:03.609300+0000 (oldest deadline 2024-03-12T06:07:31.209953+0000) 2024-03-12T06:07:46.197 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:45 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7[54334]: 2024-03-12T06:07:45.987+0000 7fd146fd0700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.211375+0000 front 2024-03-12T06:07:15.211502+0000 (oldest deadline 2024-03-12T06:07:36.911439+0000) 2024-03-12T06:07:46.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:45 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:45.820+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:46.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:45 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:45.820+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:46.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:45 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:45.820+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:46.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:45 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:45.820+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:46.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:45 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:45.820+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:46.203 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:45 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:45.758+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:46.204 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:45 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:45.758+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:46.204 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:45 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:45.758+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:46.204 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:45 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:45.758+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:07:46.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:46 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:46.198+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:46.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:46 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:46.198+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:46.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:46 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:46.198+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:46.487 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:46 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:46.198+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.636364+0000 front 2024-03-12T06:07:22.636375+0000 (oldest deadline 2024-03-12T06:07:44.936174+0000) 2024-03-12T06:07:47.182 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:46 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:46.756+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:47.182 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:46 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:46.756+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:47.183 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:46 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:46.756+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:47.183 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:46 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:46.756+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:07:47.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:46 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:46.808+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:47.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:46 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:46.808+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:47.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:46 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:46.808+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:47.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:46 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:46.808+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:47.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:46 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:46.808+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:47.239 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:46 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:46.759+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:47.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:46 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:46.759+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:47.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:46 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:46.759+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:47.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:46 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:46.759+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:47.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:46 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:46.759+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:47.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:47 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:47.177+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:47.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:47 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:47.177+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:47.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:47 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:47.177+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:47.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:47 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:47.177+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.636364+0000 front 2024-03-12T06:07:22.636375+0000 (oldest deadline 2024-03-12T06:07:44.936174+0000) 2024-03-12T06:07:47.631 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:47 smithi082 podman[72700]: 2024-03-12 06:07:47.337581961 +0000 UTC m=+5.618439889 container died db57a3e3b6c0173af0cfb9d221d8715f43239520db6d43c880a773b15c38cde0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-12T06:07:47.946 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:47 smithi082 podman[72700]: 2024-03-12 06:07:47.709212229 +0000 UTC m=+5.990070141 container cleanup db57a3e3b6c0173af0cfb9d221d8715f43239520db6d43c880a773b15c38cde0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-12T06:07:47.947 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:47 smithi082 bash[72700]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7 2024-03-12T06:07:47.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:47 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:47.762+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:47.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:47 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:47.762+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:47.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:47 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:47.762+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:47.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:47 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:47.762+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:47.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:47 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:47.762+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:47.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:47.721+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:47.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:47.721+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:47.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:47.721+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:47.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:47.721+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:47.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:47 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:47.721+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:48.184 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:47 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:47.790+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:48.184 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:47 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:47.790+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:48.185 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:47 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:47.790+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:48.185 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:47 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:47.790+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:07:48.307 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:47 smithi082 podman[72718]: 2024-03-12 06:07:47.961224657 +0000 UTC m=+0.621789892 container remove db57a3e3b6c0173af0cfb9d221d8715f43239520db6d43c880a773b15c38cde0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397) 2024-03-12T06:07:48.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:48 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:48.179+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:48.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:48 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:48.179+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:48.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:48 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:48.179+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:48.487 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:48 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:48.179+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.636364+0000 front 2024-03-12T06:07:22.636375+0000 (oldest deadline 2024-03-12T06:07:44.936174+0000) 2024-03-12T06:07:48.618 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:48 smithi082 podman[72812]: 2024-03-12 06:07:48.30297723 +0000 UTC m=+0.100638333 container create dc8cafdff41f59480c285d89a826dad80c487a385c6219471e6c4bf2d028e277 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7-deactivate, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T06:07:48.618 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:48 smithi082 podman[72812]: 2024-03-12 06:07:48.22003037 +0000 UTC m=+0.017691493 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T06:07:48.619 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:48 smithi082 podman[72812]: 2024-03-12 06:07:48.531105423 +0000 UTC m=+0.328766542 container init dc8cafdff41f59480c285d89a826dad80c487a385c6219471e6c4bf2d028e277 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-12T06:07:48.619 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:48 smithi082 podman[72812]: 2024-03-12 06:07:48.534837516 +0000 UTC m=+0.332498626 container start dc8cafdff41f59480c285d89a826dad80c487a385c6219471e6c4bf2d028e277 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7-deactivate, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-03-12T06:07:48.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:48 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:48.782+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:48.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:48 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:48.782+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:48.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:48 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:48.782+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:48.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:48 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:48.782+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:48.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:48 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:48.782+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:48.948 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:48 smithi082 podman[72812]: 2024-03-12 06:07:48.613477497 +0000 UTC m=+0.411138637 container attach dc8cafdff41f59480c285d89a826dad80c487a385c6219471e6c4bf2d028e277 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7-deactivate, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-12T06:07:48.948 INFO:journalctl@ceph.osd.7.smithi082.stdout:Mar 12 06:07:48 smithi082 podman[72812]: 2024-03-12 06:07:48.834412702 +0000 UTC m=+0.632073814 container died dc8cafdff41f59480c285d89a826dad80c487a385c6219471e6c4bf2d028e277 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_CLEAN=True) 2024-03-12T06:07:48.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:48 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:48.684+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:48.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:48 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:48.684+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:48.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:48 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:48.684+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:48.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:48 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:48.684+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:48.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:48 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:48.684+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:49.173 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:48 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:48.783+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:49.173 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:48 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:48.783+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:49.174 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:48 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:48.783+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:49.174 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:48 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:48.783+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:07:49.398 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.7.service' 2024-03-12T06:07:49.442 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:07:49.442 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-12T06:07:49.442 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-12T06:07:49.443 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.8 2024-03-12T06:07:49.472 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:49 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:49.167+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:49.472 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:49 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:49.167+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:49.473 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:49 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:49.167+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:49.473 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:49 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:49.167+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.636364+0000 front 2024-03-12T06:07:22.636375+0000 (oldest deadline 2024-03-12T06:07:44.936174+0000) 2024-03-12T06:07:49.736 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:49 smithi088 systemd[1]: Stopping Ceph osd.8 for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:07:49.946 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:49 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:49.811+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:49.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:49 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:49.811+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:49.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:49 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:49.811+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:49.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:49 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:49.811+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:49.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:49 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:49.811+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:49.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:49 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:49.640+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:49.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:49 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:49.640+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:49.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:49 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:49.640+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:49.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:49 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:49.640+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:49.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:49 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:49.640+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:50.157 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:49 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:49.796+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:50.157 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:49 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:49.796+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:50.157 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:49 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:49.796+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:50.157 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:49 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:49.796+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:07:50.157 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:49 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:49.732+0000 7f55298dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:07:50.157 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:49 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:49.732+0000 7f55298dc700 -1 osd.8 100 *** Got signal Terminated *** 2024-03-12T06:07:50.158 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:49 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:49.732+0000 7f55298dc700 -1 osd.8 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T06:07:50.158 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:50 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:50.151+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:50.158 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:50 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:50.151+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:50.158 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:50 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:50.151+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:50.158 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:50 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:50.151+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.636364+0000 front 2024-03-12T06:07:22.636375+0000 (oldest deadline 2024-03-12T06:07:44.936174+0000) 2024-03-12T06:07:50.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:50 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:50.780+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:50.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:50 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:50.780+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:50.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:50 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:50.780+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:50.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:50 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:50.780+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:50.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:50 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:50.780+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:50.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:50 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:50.658+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:50.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:50 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:50.658+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:50.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:50 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:50.658+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:50.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:50 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:50.658+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:50.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:50 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:50.658+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:51.156 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:50 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:50.807+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:51.156 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:50 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:50.807+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:51.156 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:50 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:50.807+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:51.156 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:50 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:50.807+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:07:51.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:51 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:51.150+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:51.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:51 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:51.150+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:51.487 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:51 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:51.150+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:51.487 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:51 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:51.150+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.636364+0000 front 2024-03-12T06:07:22.636375+0000 (oldest deadline 2024-03-12T06:07:44.936174+0000) 2024-03-12T06:07:51.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:51 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:51.640+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:51.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:51 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:51.640+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:51.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:51 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:51.640+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:51.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:51 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:51.640+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:51.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:51 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:51.640+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:52.137 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:51 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:51.852+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:52.137 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:51 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:51.852+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:52.137 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:51 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:51.852+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:52.138 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:51 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:51.852+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:07:52.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:51 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:51.740+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:52.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:51 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:51.740+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:52.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:51 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:51.740+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:52.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:51 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:51.740+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:52.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:51 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:51.740+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:52.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:52 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:52.132+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:52.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:52 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:52.132+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:52.487 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:52 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:52.132+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:52.487 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:52 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:52.132+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.636364+0000 front 2024-03-12T06:07:22.636375+0000 (oldest deadline 2024-03-12T06:07:44.936174+0000) 2024-03-12T06:07:52.891 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:52 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:52.631+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:52.891 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:52 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:52.631+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:52.891 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:52 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:52.631+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:52.891 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:52 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:52.631+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:52.891 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:52 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:52.631+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:53.174 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:52 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:52.848+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:53.175 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:52 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:52.848+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:53.175 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:52 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:52.848+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:53.175 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:52 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:52.848+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:07:53.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:52 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:52.765+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:53.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:52 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:52.765+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:53.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:52 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:52.765+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:53.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:52 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:52.765+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:53.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:52 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:52.765+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:53.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:52 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:52.765+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:07:53.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:53.168+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:53.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:53.169+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:53.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:53.169+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:53.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:53.169+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.636364+0000 front 2024-03-12T06:07:22.636375+0000 (oldest deadline 2024-03-12T06:07:44.936174+0000) 2024-03-12T06:07:53.487 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:53.169+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:30.737556+0000 front 2024-03-12T06:07:30.737623+0000 (oldest deadline 2024-03-12T06:07:52.437606+0000) 2024-03-12T06:07:53.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:53 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:53.658+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:53.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:53 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:53.658+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:53.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:53 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:53.658+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:53.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:53 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:53.658+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:53.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:53 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:53.658+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:54.152 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:53.811+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:54.152 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:53.811+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:54.152 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:53.811+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:54.152 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:53.811+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:07:54.152 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:53 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:53.811+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:07:54.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:53 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:53.795+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:54.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:53 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:53.795+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:54.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:53 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:53.795+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:54.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:53 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:53.795+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:54.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:53 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:53.795+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:54.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:53 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:53.795+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:07:54.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:54 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:54.143+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:58.732689+0000 front 2024-03-12T06:06:58.732722+0000 (oldest deadline 2024-03-12T06:07:24.632696+0000) 2024-03-12T06:07:54.486 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:54 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:54.143+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.933642+0000 front 2024-03-12T06:07:06.933715+0000 (oldest deadline 2024-03-12T06:07:32.134286+0000) 2024-03-12T06:07:54.487 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:54 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:54.143+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:12.134760+0000 front 2024-03-12T06:07:12.134955+0000 (oldest deadline 2024-03-12T06:07:36.234782+0000) 2024-03-12T06:07:54.487 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:54 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:54.143+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.636364+0000 front 2024-03-12T06:07:22.636375+0000 (oldest deadline 2024-03-12T06:07:44.936174+0000) 2024-03-12T06:07:54.487 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:54 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8[55570]: 2024-03-12T06:07:54.143+0000 7f55226c9700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:30.737556+0000 front 2024-03-12T06:07:30.737623+0000 (oldest deadline 2024-03-12T06:07:52.437606+0000) 2024-03-12T06:07:54.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:54 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:54.669+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:54.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:54 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:54.669+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:54.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:54 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:54.669+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:54.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:54 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:54.669+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:54.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:54 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:54.669+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:54.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:54 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:54.669+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.749643+0000 front 2024-03-12T06:07:29.749617+0000 (oldest deadline 2024-03-12T06:07:53.849369+0000) 2024-03-12T06:07:55.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:54 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:54.799+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:55.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:54 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:54.799+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:55.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:54 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:54.799+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:55.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:54 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:54.799+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:55.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:54 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:54.799+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:55.198 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:54 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:54.799+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:07:55.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:54 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:54.857+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:55.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:54 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:54.857+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:55.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:54 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:54.857+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:55.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:54 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:54.857+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:07:55.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:54 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:54.857+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:07:55.595 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:55 smithi088 podman[74392]: 2024-03-12 06:07:55.281847146 +0000 UTC m=+5.646034207 container died f4d6766e5324c43adabd54cc15b9a69a74ee2dfbb867e077ce2d8b8e6f7fcc4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-12T06:07:55.876 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:55 smithi088 podman[74392]: 2024-03-12 06:07:55.591364425 +0000 UTC m=+5.955551452 container cleanup f4d6766e5324c43adabd54cc15b9a69a74ee2dfbb867e077ce2d8b8e6f7fcc4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T06:07:55.877 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:55 smithi088 bash[74392]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8 2024-03-12T06:07:55.877 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:55 smithi088 podman[74409]: 2024-03-12 06:07:55.817521838 +0000 UTC m=+0.537638031 container remove f4d6766e5324c43adabd54cc15b9a69a74ee2dfbb867e077ce2d8b8e6f7fcc4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-12T06:07:55.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:55 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:55.684+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:55.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:55 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:55.684+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:55.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:55 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:55.684+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:55.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:55 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:55.684+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:55.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:55 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:55.684+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:55.991 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:55 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:55.684+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.749643+0000 front 2024-03-12T06:07:29.749617+0000 (oldest deadline 2024-03-12T06:07:53.849369+0000) 2024-03-12T06:07:56.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:55 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:55.757+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:56.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:55 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:55.757+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:56.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:55 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:55.757+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:56.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:55 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:55.757+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:56.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:55 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:55.757+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:56.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:55 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:55.757+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:07:56.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:55 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:55.871+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:56.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:55 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:55.871+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:56.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:55 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:55.871+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:56.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:55 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:55.871+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:07:56.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:55 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:55.871+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:07:56.721 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:56 smithi088 podman[74498]: 2024-03-12 06:07:56.155578665 +0000 UTC m=+0.017293280 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T06:07:56.721 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:56 smithi088 podman[74498]: 2024-03-12 06:07:56.260060034 +0000 UTC m=+0.121774682 container create 513fe48b5c1637b6d4e37202713a402dd5ea812518b0c0688c659dd31c3d3c6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T06:07:56.721 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:56 smithi088 podman[74498]: 2024-03-12 06:07:56.429554706 +0000 UTC m=+0.291269327 container init 513fe48b5c1637b6d4e37202713a402dd5ea812518b0c0688c659dd31c3d3c6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5) 2024-03-12T06:07:56.721 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:56 smithi088 podman[74498]: 2024-03-12 06:07:56.432620187 +0000 UTC m=+0.294334784 container start 513fe48b5c1637b6d4e37202713a402dd5ea812518b0c0688c659dd31c3d3c6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8-deactivate, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-12T06:07:56.721 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:56 smithi088 podman[74498]: 2024-03-12 06:07:56.487117684 +0000 UTC m=+0.348832304 container attach 513fe48b5c1637b6d4e37202713a402dd5ea812518b0c0688c659dd31c3d3c6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397) 2024-03-12T06:07:56.986 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:56 smithi088 podman[74498]: 2024-03-12 06:07:56.716342232 +0000 UTC m=+0.578056838 container died 513fe48b5c1637b6d4e37202713a402dd5ea812518b0c0688c659dd31c3d3c6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-12T06:07:56.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:56 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:56.855+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:56.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:56 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:56.855+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:56.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:56 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:56.855+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:56.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:56 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:56.855+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:07:56.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:56 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:56.855+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:07:56.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:56 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:56.707+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:56.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:56 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:56.707+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:56.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:56 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:56.707+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:56.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:56 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:56.707+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:56.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:56 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:56.707+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:56.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:56 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:56.707+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.749643+0000 front 2024-03-12T06:07:29.749617+0000 (oldest deadline 2024-03-12T06:07:53.849369+0000) 2024-03-12T06:07:57.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:56 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:56.791+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:57.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:56 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:56.791+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:57.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:56 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:56.791+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:57.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:56 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:56.791+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:57.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:56 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:56.791+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:57.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:56 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:56.791+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:07:57.258 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.8.service' 2024-03-12T06:07:57.323 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:57 smithi088 podman[74525]: 2024-03-12 06:07:57.224115697 +0000 UTC m=+0.502121753 container remove 513fe48b5c1637b6d4e37202713a402dd5ea812518b0c0688c659dd31c3d3c6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T06:07:57.324 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:57 smithi088 systemd[1]: ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.8.service: Deactivated successfully. 2024-03-12T06:07:57.324 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:57 smithi088 systemd[1]: Stopped Ceph osd.8 for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T06:07:57.324 INFO:journalctl@ceph.osd.8.smithi088.stdout:Mar 12 06:07:57 smithi088 systemd[1]: ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.8.service: Consumed 13.737s CPU time. 2024-03-12T06:07:57.339 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:07:57.339 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-12T06:07:57.340 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-12T06:07:57.340 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.9 2024-03-12T06:07:57.635 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:57 smithi076 systemd[1]: Stopping Ceph osd.9 for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:07:57.636 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:57 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:57.629+0000 7f27d0028700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:07:57.636 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:57 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:57.629+0000 7f27d0028700 -1 osd.9 100 *** Got signal Terminated *** 2024-03-12T06:07:57.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:57 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:57.831+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:57.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:57 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:57.831+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:57.986 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:57 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:57.831+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:57.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:57 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:57.831+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:07:57.987 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:57 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:57.831+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:07:57.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:57 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:57.629+0000 7f27d0028700 -1 osd.9 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T06:07:57.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:57 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:57.664+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:57.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:57 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:57.664+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:57.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:57 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:57.664+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:57.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:57 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:57.664+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:57.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:57 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:57.664+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:57.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:57 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:57.664+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.749643+0000 front 2024-03-12T06:07:29.749617+0000 (oldest deadline 2024-03-12T06:07:53.849369+0000) 2024-03-12T06:07:58.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:57 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:57.748+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:58.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:57 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:57.748+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:58.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:57 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:57.748+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:58.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:57 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:57.748+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:58.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:57 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:57.748+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:58.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:57 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:57.748+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:07:58.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:58 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:58.697+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:58.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:58 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:58.697+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:58.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:58 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:58.697+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:58.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:58 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:58.697+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:58.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:58 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:58.697+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:58.991 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:58 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:58.697+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.749643+0000 front 2024-03-12T06:07:29.749617+0000 (oldest deadline 2024-03-12T06:07:53.849369+0000) 2024-03-12T06:07:59.196 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:58 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:58.699+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:59.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:58 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:58.699+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:59.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:58 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:58.699+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:59.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:58 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:58.699+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:59.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:58 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:58.699+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:59.197 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:58 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:58.699+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:07:59.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:58 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:58.863+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:07:59.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:58 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:58.863+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:07:59.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:58 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:58.863+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:07:59.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:58 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:58.863+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:07:59.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:58 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:58.863+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:07:59.946 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:59 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:59.661+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:07:59.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:59 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:59.661+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:07:59.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:59 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:59.661+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:07:59.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:59 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:59.661+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:07:59.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:59 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:59.661+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:07:59.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:07:59 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:07:59.661+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:07:59.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:59 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:59.703+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:07:59.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:59 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:59.703+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:07:59.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:59 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:59.703+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:07:59.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:59 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:59.703+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:07:59.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:59 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:59.703+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:07:59.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:07:59 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:07:59.703+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.749643+0000 front 2024-03-12T06:07:29.749617+0000 (oldest deadline 2024-03-12T06:07:53.849369+0000) 2024-03-12T06:08:00.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:59 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:59.875+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:00.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:59 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:59.875+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:00.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:59 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:59.875+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:00.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:59 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:59.875+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:00.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:07:59 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:07:59.875+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:00.946 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:00 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:00.680+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:08:00.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:00 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:00.680+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:08:00.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:00 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:00.680+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:08:00.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:00 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:00.680+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:08:00.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:00 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:00.680+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:08:00.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:00 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:00.680+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:08:00.989 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:00 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:08:00.716+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:08:00.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:00 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:08:00.716+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:08:00.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:00 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:08:00.716+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:08:00.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:00 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:08:00.716+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:08:00.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:00 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:08:00.716+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:08:00.990 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:00 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:08:00.716+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.749643+0000 front 2024-03-12T06:07:29.749617+0000 (oldest deadline 2024-03-12T06:07:53.849369+0000) 2024-03-12T06:08:01.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:00 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:00.877+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:01.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:00 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:00.877+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:01.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:00 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:00.877+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:01.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:00 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:00.877+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:01.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:00 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:00.877+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:01.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:00 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:00.877+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:01.946 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:01 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:01.673+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:08:01.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:01 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:01.673+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:08:01.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:01 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:01.673+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:08:01.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:01 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:01.673+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:08:01.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:01 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:01.673+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:08:01.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:01 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:01.673+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:08:02.137 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:01 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:01.852+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:02.138 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:01 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:01.852+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:02.138 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:01 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:01.852+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:02.138 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:01 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:01.852+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:02.138 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:01 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:01.852+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:02.138 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:01 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:01.852+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:02.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:01 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:08:01.754+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:51.442331+0000 front 2024-03-12T06:06:51.442442+0000 (oldest deadline 2024-03-12T06:07:14.842973+0000) 2024-03-12T06:08:02.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:01 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:08:01.754+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:57.545380+0000 front 2024-03-12T06:06:57.545491+0000 (oldest deadline 2024-03-12T06:07:22.845444+0000) 2024-03-12T06:08:02.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:01 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:08:01.754+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.847027+0000 front 2024-03-12T06:07:06.847093+0000 (oldest deadline 2024-03-12T06:07:32.746910+0000) 2024-03-12T06:08:02.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:01 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:08:01.754+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:15.047676+0000 front 2024-03-12T06:07:15.047736+0000 (oldest deadline 2024-03-12T06:07:40.347798+0000) 2024-03-12T06:08:02.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:01 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:08:01.754+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:20.348461+0000 front 2024-03-12T06:07:20.348509+0000 (oldest deadline 2024-03-12T06:07:45.048421+0000) 2024-03-12T06:08:02.240 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:01 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:08:01.754+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.749643+0000 front 2024-03-12T06:07:29.749617+0000 (oldest deadline 2024-03-12T06:07:53.849369+0000) 2024-03-12T06:08:02.241 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:01 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9[66766]: 2024-03-12T06:08:01.754+0000 7f27cce1d700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.750454+0000 front 2024-03-12T06:07:36.750432+0000 (oldest deadline 2024-03-12T06:08:01.350869+0000) 2024-03-12T06:08:02.946 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:02 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:02.629+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:08:02.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:02 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:02.630+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:08:02.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:02 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:02.630+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:08:02.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:02 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:02.630+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:08:02.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:02 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:02.630+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:08:02.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:02 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:02.630+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:08:03.220 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:02 smithi076 podman[105764]: 2024-03-12 06:08:02.935518408 +0000 UTC m=+5.405484253 container died f2a23fff01175e64c8463adf23e83df067412f71b51e7b40b95dc61cb2ebf56b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T06:08:03.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:02 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:02.891+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:03.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:02 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:02.891+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:03.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:02 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:02.891+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:03.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:02 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:02.891+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:03.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:02 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:02.891+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:03.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:02 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:02.891+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:03.489 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:03 smithi076 podman[105764]: 2024-03-12 06:08:03.289439505 +0000 UTC m=+5.759405355 container cleanup f2a23fff01175e64c8463adf23e83df067412f71b51e7b40b95dc61cb2ebf56b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, ceph=True) 2024-03-12T06:08:03.490 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:03 smithi076 bash[105764]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9 2024-03-12T06:08:03.904 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:03 smithi076 podman[105781]: 2024-03-12 06:08:03.574106633 +0000 UTC m=+0.635960869 container remove f2a23fff01175e64c8463adf23e83df067412f71b51e7b40b95dc61cb2ebf56b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-12T06:08:03.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:03 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:03.594+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:08:03.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:03 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:03.594+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:08:03.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:03 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:03.594+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:08:03.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:03 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:03.594+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:08:03.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:03 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:03.594+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:08:03.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:03 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:03.594+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:08:04.215 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:03 smithi076 podman[105874]: 2024-03-12 06:08:03.899935052 +0000 UTC m=+0.106139584 container create f60e1aaae566b3f81f2e022706dd004e85e50007a9a1fcfd4cc8232cd842214b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-12T06:08:04.215 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:03 smithi076 podman[105874]: 2024-03-12 06:08:03.811210845 +0000 UTC m=+0.017415385 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T06:08:04.216 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:04 smithi076 podman[105874]: 2024-03-12 06:08:04.128148937 +0000 UTC m=+0.334353478 container init f60e1aaae566b3f81f2e022706dd004e85e50007a9a1fcfd4cc8232cd842214b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-03-12T06:08:04.216 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:04 smithi076 podman[105874]: 2024-03-12 06:08:04.131560799 +0000 UTC m=+0.337765333 container start f60e1aaae566b3f81f2e022706dd004e85e50007a9a1fcfd4cc8232cd842214b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-12T06:08:04.216 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:04 smithi076 podman[105874]: 2024-03-12 06:08:04.210979464 +0000 UTC m=+0.417184014 container attach f60e1aaae566b3f81f2e022706dd004e85e50007a9a1fcfd4cc8232cd842214b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, ceph=True) 2024-03-12T06:08:04.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:03 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:03.845+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:04.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:03 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:03.845+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:04.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:03 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:03.845+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:04.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:03 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:03.845+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:04.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:03 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:03.845+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:04.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:03 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:03.845+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:04.489 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:04 smithi076 podman[105874]: 2024-03-12 06:08:04.415362247 +0000 UTC m=+0.621566790 container died f60e1aaae566b3f81f2e022706dd004e85e50007a9a1fcfd4cc8232cd842214b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-12T06:08:04.946 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:04 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:04.566+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:08:04.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:04 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:04.566+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:08:04.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:04 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:04.566+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:08:04.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:04 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:04.566+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:08:04.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:04 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:04.566+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:08:04.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:04 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:04.566+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:08:04.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:04 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:04.566+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:38.634434+0000 front 2024-03-12T06:07:38.634520+0000 (oldest deadline 2024-03-12T06:08:04.534249+0000) 2024-03-12T06:08:05.036 INFO:journalctl@ceph.osd.9.smithi076.stdout:Mar 12 06:08:05 smithi076 podman[105899]: 2024-03-12 06:08:05.007033927 +0000 UTC m=+0.586833109 container remove f60e1aaae566b3f81f2e022706dd004e85e50007a9a1fcfd4cc8232cd842214b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-12T06:08:05.041 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.9.service' 2024-03-12T06:08:05.083 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:08:05.083 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-12T06:08:05.083 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-12T06:08:05.084 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.10 2024-03-12T06:08:05.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:04 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:04.812+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:05.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:04 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:04.812+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:05.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:04 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:04.812+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:05.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:04 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:04.812+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:05.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:04 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:04.812+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:05.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:04 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:04.812+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:05.406 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:05 smithi082 systemd[1]: Stopping Ceph osd.10 for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:08:05.407 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:05 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:05.400+0000 7f228f503700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:08:05.407 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:05 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:05.400+0000 7f228f503700 -1 osd.10 100 *** Got signal Terminated *** 2024-03-12T06:08:05.407 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:05 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:05.400+0000 7f228f503700 -1 osd.10 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T06:08:05.697 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:05 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:05.538+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:08:05.697 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:05 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:05.538+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:08:05.697 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:05 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:05.538+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:08:05.697 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:05 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:05.538+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:08:05.697 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:05 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:05.538+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:08:05.697 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:05 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:05.538+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:08:05.697 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:05 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:05.538+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:38.634434+0000 front 2024-03-12T06:07:38.634520+0000 (oldest deadline 2024-03-12T06:08:04.534249+0000) 2024-03-12T06:08:06.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:05 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:05.813+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:06.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:05 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:05.813+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:06.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:05 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:05.813+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:06.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:05 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:05.813+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:06.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:05 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:05.813+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:06.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:05 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:05.813+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:06.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:06 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:06.576+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:08:06.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:06 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:06.576+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:08:06.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:06 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:06.576+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:08:06.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:06 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:06.576+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:08:06.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:06 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:06.576+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:08:06.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:06 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:06.576+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:08:06.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:06 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:06.576+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:38.634434+0000 front 2024-03-12T06:07:38.634520+0000 (oldest deadline 2024-03-12T06:08:04.534249+0000) 2024-03-12T06:08:07.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:06 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:06.785+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:07.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:06 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:06.785+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:07.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:06 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:06.785+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:07.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:06 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:06.785+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:07.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:06 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:06.785+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:07.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:06 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:06.785+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:07.946 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:07 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:07.585+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:08:07.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:07 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:07.585+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:08:07.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:07 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:07.585+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:08:07.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:07 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:07.585+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:08:07.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:07 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:07.585+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:08:07.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:07 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:07.585+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:08:07.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:07 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:07.585+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:38.634434+0000 front 2024-03-12T06:07:38.634520+0000 (oldest deadline 2024-03-12T06:08:04.534249+0000) 2024-03-12T06:08:08.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:07 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:07.793+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:08.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:07 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:07.793+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:08.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:07 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:07.793+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:08.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:07 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:07.793+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:08.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:07 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:07.793+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:08.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:07 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:07.793+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:08.946 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:08 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:08.555+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:08:08.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:08 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:08.555+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:08:08.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:08 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:08.555+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:08:08.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:08 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:08.555+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:08:08.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:08 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:08.555+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:08:08.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:08 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:08.555+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:08:08.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:08 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:08.555+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:38.634434+0000 front 2024-03-12T06:07:38.634520+0000 (oldest deadline 2024-03-12T06:08:04.534249+0000) 2024-03-12T06:08:09.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:08 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:08.803+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:09.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:08 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:08.803+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:09.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:08 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:08.803+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:09.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:08 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:08.803+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:09.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:08 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:08.803+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:09.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:08 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:08.803+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:09.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:08 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:08.803+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-12T06:07:44.182151+0000 front 2024-03-12T06:07:44.182279+0000 (oldest deadline 2024-03-12T06:08:08.281976+0000) 2024-03-12T06:08:09.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:09 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:09.582+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.224343+0000 front 2024-03-12T06:06:50.224234+0000 (oldest deadline 2024-03-12T06:07:14.224805+0000) 2024-03-12T06:08:09.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:09 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:09.582+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:59.426335+0000 front 2024-03-12T06:06:59.426422+0000 (oldest deadline 2024-03-12T06:07:22.326503+0000) 2024-03-12T06:08:09.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:09 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:09.582+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.327955+0000 front 2024-03-12T06:07:06.327814+0000 (oldest deadline 2024-03-12T06:07:28.027858+0000) 2024-03-12T06:08:09.947 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:09 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:09.582+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6814 osd.3 since back 2024-03-12T06:07:14.429524+0000 front 2024-03-12T06:07:14.429584+0000 (oldest deadline 2024-03-12T06:07:36.030004+0000) 2024-03-12T06:08:09.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:09 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:09.582+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.431083+0000 front 2024-03-12T06:07:22.431127+0000 (oldest deadline 2024-03-12T06:07:45.331243+0000) 2024-03-12T06:08:09.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:09 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:09.582+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:28.133024+0000 front 2024-03-12T06:07:28.133154+0000 (oldest deadline 2024-03-12T06:07:52.232869+0000) 2024-03-12T06:08:09.948 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:09 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10[60993]: 2024-03-12T06:08:09.582+0000 7f22882f0700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:38.634434+0000 front 2024-03-12T06:07:38.634520+0000 (oldest deadline 2024-03-12T06:08:04.534249+0000) 2024-03-12T06:08:10.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:09 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:09.787+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:10.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:09 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:09.787+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:10.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:09 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:09.787+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:10.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:09 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:09.787+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:10.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:09 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:09.787+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:10.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:09 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:09.787+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:10.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:09 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:09.787+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-12T06:07:44.182151+0000 front 2024-03-12T06:07:44.182279+0000 (oldest deadline 2024-03-12T06:08:08.281976+0000) 2024-03-12T06:08:11.164 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:10 smithi082 podman[72935]: 2024-03-12 06:08:10.850538453 +0000 UTC m=+5.578881761 container died 6a65aec96e6ac2cf1f1bd6254a155981d0b763919e314c0e6342bf136bd7a835 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-12T06:08:11.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:10 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:10.837+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:11.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:10 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:10.837+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:11.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:10 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:10.837+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:11.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:10 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:10.837+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:11.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:10 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:10.837+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:11.238 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:10 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:10.837+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:11.238 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:10 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:10.837+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-12T06:07:44.182151+0000 front 2024-03-12T06:07:44.182279+0000 (oldest deadline 2024-03-12T06:08:08.281976+0000) 2024-03-12T06:08:11.446 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:11 smithi082 podman[72935]: 2024-03-12 06:08:11.269523236 +0000 UTC m=+5.997866524 container cleanup 6a65aec96e6ac2cf1f1bd6254a155981d0b763919e314c0e6342bf136bd7a835 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T06:08:11.447 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:11 smithi082 bash[72935]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10 2024-03-12T06:08:11.865 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:11 smithi082 podman[72952]: 2024-03-12 06:08:11.443150933 +0000 UTC m=+0.595314802 container remove 6a65aec96e6ac2cf1f1bd6254a155981d0b763919e314c0e6342bf136bd7a835 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=squid-94a5397) 2024-03-12T06:08:12.142 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:11 smithi082 podman[73043]: 2024-03-12 06:08:11.860632078 +0000 UTC m=+0.097239317 container create 0f491a84a29ad3d0a7c863518d9cdf9fd00a80fad46d056d51ac33e0875d2676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-12T06:08:12.142 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:11 smithi082 podman[73043]: 2024-03-12 06:08:11.781127865 +0000 UTC m=+0.017735120 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T06:08:12.142 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:12 smithi082 podman[73043]: 2024-03-12 06:08:12.054598369 +0000 UTC m=+0.291205609 container init 0f491a84a29ad3d0a7c863518d9cdf9fd00a80fad46d056d51ac33e0875d2676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10-deactivate, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-03-12T06:08:12.143 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:12 smithi082 podman[73043]: 2024-03-12 06:08:12.058040371 +0000 UTC m=+0.294647620 container start 0f491a84a29ad3d0a7c863518d9cdf9fd00a80fad46d056d51ac33e0875d2676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-12T06:08:12.143 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:12 smithi082 podman[73043]: 2024-03-12 06:08:12.137811791 +0000 UTC m=+0.374419063 container attach 0f491a84a29ad3d0a7c863518d9cdf9fd00a80fad46d056d51ac33e0875d2676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397) 2024-03-12T06:08:12.198 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:11 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:11.884+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:12.198 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:11 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:11.884+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:12.199 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:11 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:11.884+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:12.199 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:11 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:11.884+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:12.199 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:11 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:11.884+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:12.199 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:11 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:11.884+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:12.200 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:11 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:11.884+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-12T06:07:44.182151+0000 front 2024-03-12T06:07:44.182279+0000 (oldest deadline 2024-03-12T06:08:08.281976+0000) 2024-03-12T06:08:12.446 INFO:journalctl@ceph.osd.10.smithi082.stdout:Mar 12 06:08:12 smithi082 podman[73043]: 2024-03-12 06:08:12.34627116 +0000 UTC m=+0.582878402 container died 0f491a84a29ad3d0a7c863518d9cdf9fd00a80fad46d056d51ac33e0875d2676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T06:08:12.907 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.10.service' 2024-03-12T06:08:12.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:08:12.951 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-12T06:08:12.951 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-12T06:08:12.952 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.11 2024-03-12T06:08:13.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:12 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:12.883+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:13.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:12 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:12.883+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:13.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:12 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:12.883+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:13.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:12 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:12.883+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:13.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:12 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:12.883+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:13.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:12 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:12.883+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:13.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:12 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:12.883+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-12T06:07:44.182151+0000 front 2024-03-12T06:07:44.182279+0000 (oldest deadline 2024-03-12T06:08:08.281976+0000) 2024-03-12T06:08:13.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:12 smithi088 systemd[1]: Stopping Ceph osd.11 for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:08:13.736 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:13 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:13.233+0000 7f49874c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:08:13.736 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:13 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:13.233+0000 7f49874c2700 -1 osd.11 100 *** Got signal Terminated *** 2024-03-12T06:08:13.736 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:13 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:13.233+0000 7f49874c2700 -1 osd.11 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T06:08:14.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:13 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:13.833+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:14.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:13 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:13.833+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:14.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:13 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:13.833+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:14.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:13 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:13.833+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:14.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:13 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:13.833+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:14.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:13 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:13.833+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:14.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:13 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:13.833+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-12T06:07:44.182151+0000 front 2024-03-12T06:07:44.182279+0000 (oldest deadline 2024-03-12T06:08:08.281976+0000) 2024-03-12T06:08:15.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:14 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:14.876+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:15.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:14 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:14.876+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:15.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:14 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:14.876+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:15.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:14 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:14.876+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:15.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:14 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:14.876+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:15.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:14 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:14.876+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:15.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:14 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:14.876+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-12T06:07:44.182151+0000 front 2024-03-12T06:07:44.182279+0000 (oldest deadline 2024-03-12T06:08:08.281976+0000) 2024-03-12T06:08:16.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:15 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:15.915+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:16.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:15 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:15.915+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:16.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:15 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:15.915+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:16.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:15 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:15.915+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:16.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:15 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:15.915+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:16.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:15 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:15.915+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:16.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:15 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:15.915+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-12T06:07:44.182151+0000 front 2024-03-12T06:07:44.182279+0000 (oldest deadline 2024-03-12T06:08:08.281976+0000) 2024-03-12T06:08:17.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:16 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:16.894+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:17.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:16 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:16.894+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:17.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:16 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:16.894+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:17.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:16 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:16.894+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:17.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:16 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:16.894+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:17.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:16 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:16.894+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:17.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:16 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:16.894+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-12T06:07:44.182151+0000 front 2024-03-12T06:07:44.182279+0000 (oldest deadline 2024-03-12T06:08:08.281976+0000) 2024-03-12T06:08:18.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:17 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:17.849+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6806 osd.0 since back 2024-03-12T06:06:50.271951+0000 front 2024-03-12T06:06:50.271996+0000 (oldest deadline 2024-03-12T06:07:15.472404+0000) 2024-03-12T06:08:18.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:17 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:17.849+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6804 osd.1 since back 2024-03-12T06:06:55.473092+0000 front 2024-03-12T06:06:55.473083+0000 (oldest deadline 2024-03-12T06:07:21.372914+0000) 2024-03-12T06:08:18.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:17 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:17.849+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6804 osd.2 since back 2024-03-12T06:07:06.073846+0000 front 2024-03-12T06:07:06.073961+0000 (oldest deadline 2024-03-12T06:07:30.773951+0000) 2024-03-12T06:08:18.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:17 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:17.849+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6812 osd.4 since back 2024-03-12T06:07:22.777882+0000 front 2024-03-12T06:07:22.777788+0000 (oldest deadline 2024-03-12T06:07:45.677781+0000) 2024-03-12T06:08:18.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:17 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:17.849+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.88:6812 osd.5 since back 2024-03-12T06:07:29.178922+0000 front 2024-03-12T06:07:29.179029+0000 (oldest deadline 2024-03-12T06:07:53.279064+0000) 2024-03-12T06:08:18.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:17 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:17.849+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.76:6822 osd.6 since back 2024-03-12T06:07:36.680907+0000 front 2024-03-12T06:07:36.680864+0000 (oldest deadline 2024-03-12T06:08:00.180641+0000) 2024-03-12T06:08:18.237 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:17 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11[62656]: 2024-03-12T06:08:17.849+0000 7f49802af700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.82:6820 osd.7 since back 2024-03-12T06:07:44.182151+0000 front 2024-03-12T06:07:44.182279+0000 (oldest deadline 2024-03-12T06:08:08.281976+0000) 2024-03-12T06:08:18.981 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:18 smithi088 podman[74617]: 2024-03-12 06:08:18.706004407 +0000 UTC m=+5.563942579 container died bbe9c50879fa607af55dac65d472d862484e055fda2b69b1a6d79d6f3a653cd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-12T06:08:19.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:19 smithi088 podman[74617]: 2024-03-12 06:08:19.065773864 +0000 UTC m=+5.923712033 container cleanup bbe9c50879fa607af55dac65d472d862484e055fda2b69b1a6d79d6f3a653cd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True) 2024-03-12T06:08:19.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:19 smithi088 bash[74617]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11 2024-03-12T06:08:19.647 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:19 smithi088 podman[74637]: 2024-03-12 06:08:19.333606079 +0000 UTC m=+0.621389396 container remove bbe9c50879fa607af55dac65d472d862484e055fda2b69b1a6d79d6f3a653cd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-12T06:08:19.950 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:19 smithi088 podman[74729]: 2024-03-12 06:08:19.642707869 +0000 UTC m=+0.083267511 container create 3669af71129ad3eaa2a784d3c7345b4c414c21df723f77b1478738c478686588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-03-12T06:08:19.950 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:19 smithi088 podman[74729]: 2024-03-12 06:08:19.576941845 +0000 UTC m=+0.017501484 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-12T06:08:19.950 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:19 smithi088 podman[74729]: 2024-03-12 06:08:19.862701639 +0000 UTC m=+0.303261288 container init 3669af71129ad3eaa2a784d3c7345b4c414c21df723f77b1478738c478686588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-12T06:08:19.950 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:19 smithi088 podman[74729]: 2024-03-12 06:08:19.866050037 +0000 UTC m=+0.306609678 container start 3669af71129ad3eaa2a784d3c7345b4c414c21df723f77b1478738c478686588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-12T06:08:19.950 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:19 smithi088 podman[74729]: 2024-03-12 06:08:19.945410776 +0000 UTC m=+0.385970414 container attach 3669af71129ad3eaa2a784d3c7345b4c414c21df723f77b1478738c478686588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-12T06:08:20.236 INFO:journalctl@ceph.osd.11.smithi088.stdout:Mar 12 06:08:20 smithi088 podman[74729]: 2024-03-12 06:08:20.156645044 +0000 UTC m=+0.597204676 container died 3669af71129ad3eaa2a784d3c7345b4c414c21df723f77b1478738c478686588 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-12T06:08:20.811 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@osd.11.service' 2024-03-12T06:08:20.852 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:08:20.852 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-12T06:08:20.853 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-12T06:08:20.853 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.a 2024-03-12T06:08:21.156 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 06:08:20 smithi076 systemd[1]: Stopping Ceph mds.a for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:08:21.489 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 06:08:21 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-a[73468]: 2024-03-12T06:08:21.150+0000 7f0661d66700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:08:21.490 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 06:08:21 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-a[73468]: 2024-03-12T06:08:21.150+0000 7f0661d66700 -1 mds.a *** got signal Terminated *** 2024-03-12T06:08:25.431 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 06:08:25 smithi076 podman[105993]: 2024-03-12 06:08:25.158514347 +0000 UTC m=+4.108773305 container died c5ac474888b8c19a4992e0f80b576044dc52fee919ab59536b767e817082579f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-12T06:08:25.739 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 06:08:25 smithi076 podman[105993]: 2024-03-12 06:08:25.544575844 +0000 UTC m=+4.494834801 container cleanup c5ac474888b8c19a4992e0f80b576044dc52fee919ab59536b767e817082579f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397) 2024-03-12T06:08:25.740 INFO:journalctl@ceph.mds.a.smithi076.stdout:Mar 12 06:08:25 smithi076 bash[105993]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-a 2024-03-12T06:08:25.981 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.a.service' 2024-03-12T06:08:26.018 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:08:26.019 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-12T06:08:26.019 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-12T06:08:26.019 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.d 2024-03-12T06:08:26.343 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 06:08:26 smithi076 systemd[1]: Stopping Ceph mds.d for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:08:26.739 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 06:08:26 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-d[72497]: 2024-03-12T06:08:26.338+0000 7f38776ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:08:26.740 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 06:08:26 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-d[72497]: 2024-03-12T06:08:26.338+0000 7f38776ba700 -1 mds.d *** got signal Terminated *** 2024-03-12T06:08:30.670 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 06:08:30 smithi076 podman[106114]: 2024-03-12 06:08:30.345100029 +0000 UTC m=+4.102992904 container died 89bf230cda25afb00fa9fb84fdaf8816989db03ff3fd1f2facc0f9eda2bab8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-12T06:08:30.990 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 06:08:30 smithi076 podman[106114]: 2024-03-12 06:08:30.739214738 +0000 UTC m=+4.497107611 container cleanup 89bf230cda25afb00fa9fb84fdaf8816989db03ff3fd1f2facc0f9eda2bab8bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-d, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397) 2024-03-12T06:08:30.990 INFO:journalctl@ceph.mds.d.smithi076.stdout:Mar 12 06:08:30 smithi076 bash[106114]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-d 2024-03-12T06:08:31.088 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.d.service' 2024-03-12T06:08:31.126 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:08:31.127 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-12T06:08:31.127 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-12T06:08:31.127 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.g 2024-03-12T06:08:31.448 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 06:08:31 smithi076 systemd[1]: Stopping Ceph mds.g for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:08:31.739 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 06:08:31 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-g[73065]: 2024-03-12T06:08:31.443+0000 7f4475c70700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:08:31.740 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 06:08:31 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-g[73065]: 2024-03-12T06:08:31.443+0000 7f4475c70700 -1 mds.g *** got signal Terminated *** 2024-03-12T06:08:34.965 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 06:08:34 smithi076 podman[106247]: 2024-03-12 06:08:34.710355936 +0000 UTC m=+3.372450514 container died 0c425e9597299e33f25a0116769e92ad40aaa4306843476d4fde91da52ce9aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-g, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-03-12T06:08:35.239 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 06:08:35 smithi076 podman[106247]: 2024-03-12 06:08:35.085502233 +0000 UTC m=+3.747596809 container cleanup 0c425e9597299e33f25a0116769e92ad40aaa4306843476d4fde91da52ce9aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-g, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-12T06:08:35.240 INFO:journalctl@ceph.mds.g.smithi076.stdout:Mar 12 06:08:35 smithi076 bash[106247]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-g 2024-03-12T06:08:35.453 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.g.service' 2024-03-12T06:08:35.488 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:08:35.488 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-12T06:08:35.488 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-12T06:08:35.488 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.j 2024-03-12T06:08:35.739 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 06:08:35 smithi076 systemd[1]: Stopping Ceph mds.j for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:08:36.239 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 06:08:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-j[72109]: 2024-03-12T06:08:35.839+0000 7f5962926700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:08:36.240 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 06:08:35 smithi076 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-j[72109]: 2024-03-12T06:08:35.839+0000 7f5962926700 -1 mds.j *** got signal Terminated *** 2024-03-12T06:08:40.240 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 06:08:39 smithi076 podman[106367]: 2024-03-12 06:08:39.852187901 +0000 UTC m=+4.149236801 container died 7656e4ec3d6a93f0bcdeb8ffdf135ca49b38f791ad4f9f0cd64ef044bd23716a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-j, RELEASE=squid-94a5397, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T06:08:40.695 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 06:08:40 smithi076 podman[106367]: 2024-03-12 06:08:40.406385334 +0000 UTC m=+4.703434234 container cleanup 7656e4ec3d6a93f0bcdeb8ffdf135ca49b38f791ad4f9f0cd64ef044bd23716a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-j, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-12T06:08:40.695 INFO:journalctl@ceph.mds.j.smithi076.stdout:Mar 12 06:08:40 smithi076 bash[106367]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-j 2024-03-12T06:08:40.839 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.j.service' 2024-03-12T06:08:40.874 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:08:40.874 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-12T06:08:40.874 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-12T06:08:40.875 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.b 2024-03-12T06:08:41.197 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 06:08:40 smithi082 systemd[1]: Stopping Ceph mds.b for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:08:41.561 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 06:08:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b[66548]: 2024-03-12T06:08:41.271+0000 7f6356f88700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:08:41.562 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 06:08:41 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b[66548]: 2024-03-12T06:08:41.271+0000 7f6356f88700 -1 mds.b *** got signal Terminated *** 2024-03-12T06:08:44.446 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 06:08:44 smithi082 podman[73149]: 2024-03-12 06:08:44.028627321 +0000 UTC m=+2.961497829 container died 1336ba7664a1854fa67575a981439d68ca11ff76f8b29db9dea9c69f580a524c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-12T06:08:44.761 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 06:08:44 smithi082 podman[73149]: 2024-03-12 06:08:44.571682695 +0000 UTC m=+3.504553200 container cleanup 1336ba7664a1854fa67575a981439d68ca11ff76f8b29db9dea9c69f580a524c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-12T06:08:44.761 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 06:08:44 smithi082 bash[73149]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b 2024-03-12T06:08:44.993 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.b.service' 2024-03-12T06:08:45.018 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 06:08:44 smithi082 podman[73176]: 2024-03-12 06:08:44.755930906 +0000 UTC m=+0.723490060 container remove 1336ba7664a1854fa67575a981439d68ca11ff76f8b29db9dea9c69f580a524c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, io.buildah.version=1.33.5) 2024-03-12T06:08:45.018 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 06:08:44 smithi082 systemd[1]: ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.b.service: Deactivated successfully. 2024-03-12T06:08:45.018 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 06:08:44 smithi082 systemd[1]: Stopped Ceph mds.b for 9f336de8-e033-11ee-95c8-87774f69a715. 2024-03-12T06:08:45.018 INFO:journalctl@ceph.mds.b.smithi082.stdout:Mar 12 06:08:44 smithi082 systemd[1]: ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.b.service: Consumed 38.818s CPU time. 2024-03-12T06:08:45.031 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:08:45.031 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-12T06:08:45.031 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-12T06:08:45.032 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.e 2024-03-12T06:08:45.357 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 12 06:08:45 smithi082 systemd[1]: Stopping Ceph mds.e for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:08:45.696 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 12 06:08:45 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-e[65918]: 2024-03-12T06:08:45.349+0000 7f79f486c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:08:45.697 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 12 06:08:45 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-e[65918]: 2024-03-12T06:08:45.349+0000 7f79f486c700 -1 mds.e *** got signal Terminated *** 2024-03-12T06:08:48.329 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 12 06:08:48 smithi082 podman[73283]: 2024-03-12 06:08:48.02334793 +0000 UTC m=+2.780703614 container died 862c71a1e1df42acdf92b4e84e7e31b78abf9874843152e3597281b0aa673101 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-e, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-12T06:08:48.329 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 12 06:08:48 smithi082 podman[73283]: 2024-03-12 06:08:48.324610267 +0000 UTC m=+3.081965966 container cleanup 862c71a1e1df42acdf92b4e84e7e31b78abf9874843152e3597281b0aa673101 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-e, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T06:08:48.628 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 12 06:08:48 smithi082 bash[73283]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-e 2024-03-12T06:08:48.628 INFO:journalctl@ceph.mds.e.smithi082.stdout:Mar 12 06:08:48 smithi082 podman[73297]: 2024-03-12 06:08:48.517303706 +0000 UTC m=+0.492184034 container remove 862c71a1e1df42acdf92b4e84e7e31b78abf9874843152e3597281b0aa673101 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-e, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True) 2024-03-12T06:08:48.652 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.e.service' 2024-03-12T06:08:48.687 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:08:48.687 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-12T06:08:48.687 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-12T06:08:48.687 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.h 2024-03-12T06:08:48.946 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 06:08:48 smithi082 systemd[1]: Stopping Ceph mds.h for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:08:49.450 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 06:08:48 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-h[66229]: 2024-03-12T06:08:48.994+0000 7faf07f64700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:08:49.450 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 06:08:48 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-h[66229]: 2024-03-12T06:08:48.994+0000 7faf07f64700 -1 mds.h *** got signal Terminated *** 2024-03-12T06:08:52.196 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 06:08:52 smithi082 podman[73403]: 2024-03-12 06:08:52.027900069 +0000 UTC m=+3.128957519 container died e32cf20a5ee18216e1a0819b4910d3ea956823fe750b7e40dba074e675edeca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-h, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-12T06:08:52.583 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 06:08:52 smithi082 podman[73403]: 2024-03-12 06:08:52.369435538 +0000 UTC m=+3.470492991 container cleanup e32cf20a5ee18216e1a0819b4910d3ea956823fe750b7e40dba074e675edeca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-h, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T06:08:52.583 INFO:journalctl@ceph.mds.h.smithi082.stdout:Mar 12 06:08:52 smithi082 bash[73403]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-h 2024-03-12T06:08:52.786 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.h.service' 2024-03-12T06:08:52.821 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:08:52.821 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-12T06:08:52.822 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-12T06:08:52.822 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.k 2024-03-12T06:08:53.144 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 06:08:52 smithi082 systemd[1]: Stopping Ceph mds.k for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:08:53.446 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 06:08:53 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-k[65595]: 2024-03-12T06:08:53.139+0000 7f2ffb6b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:08:53.447 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 06:08:53 smithi082 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-k[65595]: 2024-03-12T06:08:53.139+0000 7f2ffb6b8700 -1 mds.k *** got signal Terminated *** 2024-03-12T06:08:57.352 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 06:08:57 smithi082 podman[73525]: 2024-03-12 06:08:57.086287533 +0000 UTC m=+4.050875247 container died 3120ec94420a57965e107cddfc53d37c427cc42b93c71f906cde1836570598d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-k, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-94a5397) 2024-03-12T06:08:57.697 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 06:08:57 smithi082 podman[73525]: 2024-03-12 06:08:57.439004352 +0000 UTC m=+4.403592068 container cleanup 3120ec94420a57965e107cddfc53d37c427cc42b93c71f906cde1836570598d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-12T06:08:57.697 INFO:journalctl@ceph.mds.k.smithi082.stdout:Mar 12 06:08:57 smithi082 bash[73525]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-k 2024-03-12T06:08:57.824 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.k.service' 2024-03-12T06:08:57.858 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:08:57.859 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-12T06:08:57.859 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-12T06:08:57.859 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.c 2024-03-12T06:08:58.158 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 06:08:57 smithi088 systemd[1]: Stopping Ceph mds.c for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:08:58.486 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 06:08:58 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-c[68008]: 2024-03-12T06:08:58.153+0000 7f794a8d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:08:58.486 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 06:08:58 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-c[68008]: 2024-03-12T06:08:58.153+0000 7f794a8d3700 -1 mds.c *** got signal Terminated *** 2024-03-12T06:09:00.283 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 06:09:00 smithi088 podman[74833]: 2024-03-12 06:09:00.01319108 +0000 UTC m=+1.967829725 container died 4550fbbd722ac36ec05d86750bdb3ebc621f37ed0c7d4f6ead8f492bd256f7fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397) 2024-03-12T06:09:00.634 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 06:09:00 smithi088 podman[74833]: 2024-03-12 06:09:00.370091707 +0000 UTC m=+2.324730341 container cleanup 4550fbbd722ac36ec05d86750bdb3ebc621f37ed0c7d4f6ead8f492bd256f7fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-12T06:09:00.634 INFO:journalctl@ceph.mds.c.smithi088.stdout:Mar 12 06:09:00 smithi088 bash[74833]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-c 2024-03-12T06:09:00.794 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.c.service' 2024-03-12T06:09:00.830 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:09:00.830 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-12T06:09:00.830 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-12T06:09:00.831 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.f 2024-03-12T06:09:01.236 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 06:09:00 smithi088 systemd[1]: Stopping Ceph mds.f for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:09:01.236 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 06:09:01 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-f[67702]: 2024-03-12T06:09:01.131+0000 7fb181af7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:09:01.236 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 06:09:01 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-f[67702]: 2024-03-12T06:09:01.131+0000 7fb181af7700 -1 mds.f *** got signal Terminated *** 2024-03-12T06:09:04.336 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 06:09:04 smithi088 podman[74957]: 2024-03-12 06:09:04.016796549 +0000 UTC m=+2.979332383 container died e1fdcaa76ae5353ff8f2530ad38d7e5fd3fc58704620e6cb3a8256426044414c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-f, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-12T06:09:04.336 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 06:09:04 smithi088 podman[74957]: 2024-03-12 06:09:04.331357951 +0000 UTC m=+3.293893775 container cleanup e1fdcaa76ae5353ff8f2530ad38d7e5fd3fc58704620e6cb3a8256426044414c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-f, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_BRANCH=HEAD) 2024-03-12T06:09:04.736 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 06:09:04 smithi088 bash[74957]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-f 2024-03-12T06:09:04.736 INFO:journalctl@ceph.mds.f.smithi088.stdout:Mar 12 06:09:04 smithi088 podman[74972]: 2024-03-12 06:09:04.524191608 +0000 UTC m=+0.506146217 container remove e1fdcaa76ae5353ff8f2530ad38d7e5fd3fc58704620e6cb3a8256426044414c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-f, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-12T06:09:04.756 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.f.service' 2024-03-12T06:09:04.790 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:09:04.791 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-12T06:09:04.791 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-12T06:09:04.791 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.i 2024-03-12T06:09:05.081 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 06:09:04 smithi088 systemd[1]: Stopping Ceph mds.i for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:09:05.486 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 06:09:05 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-i[68318]: 2024-03-12T06:09:05.075+0000 7ffbb805d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:09:05.486 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 06:09:05 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-i[68318]: 2024-03-12T06:09:05.075+0000 7ffbb805d700 -1 mds.i *** got signal Terminated *** 2024-03-12T06:09:08.298 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 06:09:08 smithi088 podman[75082]: 2024-03-12 06:09:08.016540213 +0000 UTC m=+3.021218795 container died 0f4ae941a28f35615eb4f9423c65b75a84d1ed8726b6302fdae18c4ef3354992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T06:09:08.623 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 06:09:08 smithi088 podman[75082]: 2024-03-12 06:09:08.367983125 +0000 UTC m=+3.372661696 container cleanup 0f4ae941a28f35615eb4f9423c65b75a84d1ed8726b6302fdae18c4ef3354992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-i, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-12T06:09:08.624 INFO:journalctl@ceph.mds.i.smithi088.stdout:Mar 12 06:09:08 smithi088 bash[75082]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-i 2024-03-12T06:09:08.753 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.i.service' 2024-03-12T06:09:08.787 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:09:08.788 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-12T06:09:08.788 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-12T06:09:08.789 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.l 2024-03-12T06:09:08.986 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 06:09:08 smithi088 systemd[1]: Stopping Ceph mds.l for 9f336de8-e033-11ee-95c8-87774f69a715... 2024-03-12T06:09:09.486 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 06:09:09 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-l[67388]: 2024-03-12T06:09:09.112+0000 7f1933920700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T06:09:09.486 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 06:09:09 smithi088 ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-l[67388]: 2024-03-12T06:09:09.112+0000 7f1933920700 -1 mds.l *** got signal Terminated *** 2024-03-12T06:09:10.736 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 06:09:10 smithi088 podman[75203]: 2024-03-12 06:09:10.455644352 +0000 UTC m=+1.463906379 container died 5caf516650fad2c35100cf448d6e82f59b26ca35738d393a6fcd1529dc22325c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-l, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-12T06:09:11.052 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 06:09:10 smithi088 podman[75203]: 2024-03-12 06:09:10.829779022 +0000 UTC m=+1.838041046 container cleanup 5caf516650fad2c35100cf448d6e82f59b26ca35738d393a6fcd1529dc22325c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-03-12T06:09:11.052 INFO:journalctl@ceph.mds.l.smithi088.stdout:Mar 12 06:09:10 smithi088 bash[75203]: ceph-9f336de8-e033-11ee-95c8-87774f69a715-mds-l 2024-03-12T06:09:11.187 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9f336de8-e033-11ee-95c8-87774f69a715@mds.l.service' 2024-03-12T06:09:11.221 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T06:09:11.221 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-12T06:09:11.221 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9f336de8-e033-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T06:09:11.411 INFO:teuthology.orchestra.run.smithi076.stdout:Deleting cluster with fsid: 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T06:09:13.799 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9f336de8-e033-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T06:09:13.991 INFO:teuthology.orchestra.run.smithi082.stdout:Deleting cluster with fsid: 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T06:09:16.518 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9f336de8-e033-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T06:09:16.707 INFO:teuthology.orchestra.run.smithi088.stdout:Deleting cluster with fsid: 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T06:09:19.117 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T06:09:19.151 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T06:09:19.185 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T06:09:19.214 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-12T06:09:19.216 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400/remote/smithi076/crash 2024-03-12T06:09:19.217 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/crash -- . 2024-03-12T06:09:19.256 INFO:teuthology.orchestra.run.smithi076.stderr:tar: /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T06:09:19.256 INFO:teuthology.orchestra.run.smithi076.stderr:tar: Error is not recoverable: exiting now 2024-03-12T06:09:19.258 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400/remote/smithi082/crash 2024-03-12T06:09:19.259 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/crash -- . 2024-03-12T06:09:19.299 INFO:teuthology.orchestra.run.smithi082.stderr:tar: /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T06:09:19.299 INFO:teuthology.orchestra.run.smithi082.stderr:tar: Error is not recoverable: exiting now 2024-03-12T06:09:19.301 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400/remote/smithi088/crash 2024-03-12T06:09:19.302 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/crash -- . 2024-03-12T06:09:19.340 INFO:teuthology.orchestra.run.smithi088.stderr:tar: /var/lib/ceph/9f336de8-e033-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T06:09:19.341 INFO:teuthology.orchestra.run.smithi088.stderr:tar: Error is not recoverable: exiting now 2024-03-12T06:09:19.342 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-12T06:09:19.342 DEBUG:teuthology.orchestra.run.smithi076:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-12T06:09:19.376 INFO:teuthology.orchestra.run.smithi076.stdout:2024-03-12T05:47:40.017758+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-12T06:09:19.377 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-12T06:09:19.377 INFO:tasks.cephadm:Compressing logs... 2024-03-12T06:09:19.377 DEBUG:teuthology.orchestra.run.smithi076:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T06:09:19.418 DEBUG:teuthology.orchestra.run.smithi082:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T06:09:19.421 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T06:09:19.444 INFO:teuthology.orchestra.run.smithi076.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T06:09:19.444 INFO:teuthology.orchestra.run.smithi076.stderr:: No such file or directory 2024-03-12T06:09:19.444 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-12T06:09:19.445 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-12T06:09:19.445 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T06:09:19.445 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.g.log 2024-03-12T06:09:19.445 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.j.log 2024-03-12T06:09:19.447 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.audit.log: /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-12T06:09:19.447 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-12T06:09:19.448 INFO:teuthology.orchestra.run.smithi088.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T06:09:19.449 INFO:teuthology.orchestra.run.smithi088.stderr:: No such file or directory 2024-03-12T06:09:19.449 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.l.log 2024-03-12T06:09:19.449 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T06:09:19.449 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.i.log 2024-03-12T06:09:19.457 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-12T06:09:19.457 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mgr.x.log: 90.1% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T06:09:19.458 INFO:teuthology.orchestra.run.smithi076.stderr: 94.6% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.g.log.gz 2024-03-12T06:09:19.458 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T06:09:19.458 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.0.log: 94.6% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.j.log.gz 2024-03-12T06:09:19.458 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.log 2024-03-12T06:09:19.458 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T06:09:19.458 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.cephadm.log: /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.log: 83.9% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T06:09:19.459 INFO:teuthology.orchestra.run.smithi076.stderr: 93.3%gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.9.log 2024-03-12T06:09:19.459 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T06:09:19.459 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-volume.log: 87.5% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T06:09:19.464 INFO:teuthology.orchestra.run.smithi082.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T06:09:19.465 INFO:teuthology.orchestra.run.smithi082.stderr:: No such file or directory 2024-03-12T06:09:19.465 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.10.log 2024-03-12T06:09:19.465 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T06:09:19.465 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.b.log 2024-03-12T06:09:19.465 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-12T06:09:19.465 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-12T06:09:19.465 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.b.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T06:09:19.465 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.k.log 2024-03-12T06:09:19.466 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.h.log 2024-03-12T06:09:19.466 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-12T06:09:19.466 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.e.log 2024-03-12T06:09:19.466 INFO:teuthology.orchestra.run.smithi082.stderr: 90.2% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T06:09:19.466 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.7.log: 94.5% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.k.log.gz 2024-03-12T06:09:19.466 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-12T06:09:19.466 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T06:09:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.11.log 2024-03-12T06:09:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mgr.z.log 2024-03-12T06:09:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.i.log: /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.8.log 2024-03-12T06:09:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mgr.z.log: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T06:09:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-12T06:09:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.f.log 2024-03-12T06:09:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T06:09:19.471 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.f.log: 90.2% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T06:09:19.471 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.log 2024-03-12T06:09:19.471 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T06:09:19.472 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-12T06:09:19.472 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.log: 94.6% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.l.log.gz 2024-03-12T06:09:19.472 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.c.log 2024-03-12T06:09:19.472 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.2.log: 87.5% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T06:09:19.472 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T06:09:19.472 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.c.log: 91.8% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mgr.z.log.gz 2024-03-12T06:09:19.473 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-12T06:09:19.477 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T06:09:19.477 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.d.log 2024-03-12T06:09:19.478 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.a.log 2024-03-12T06:09:19.478 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.d.log: 91.0% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T06:09:19.478 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.d.log.gz 2024-03-12T06:09:19.478 INFO:teuthology.orchestra.run.smithi076.stderr: 94.5% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.a.log.gz 2024-03-12T06:09:19.479 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mgr.y.log: 91.8%gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.log 2024-03-12T06:09:19.480 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T06:09:19.480 INFO:teuthology.orchestra.run.smithi082.stderr: -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-12T06:09:19.480 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T06:09:19.480 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.log: 87.5% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T06:09:19.480 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-12T06:09:19.502 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T06:09:19.665 INFO:teuthology.orchestra.run.smithi082.stderr: 94.4% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.e.log.gz 2024-03-12T06:09:19.724 INFO:teuthology.orchestra.run.smithi088.stderr: 94.8% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.c.log.gz 2024-03-12T06:09:19.775 INFO:teuthology.orchestra.run.smithi076.stderr: 89.9% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-12T06:09:19.919 INFO:teuthology.orchestra.run.smithi088.stderr: 91.9% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-12T06:09:19.934 INFO:teuthology.orchestra.run.smithi082.stderr: 92.0% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-12T06:09:20.347 INFO:teuthology.orchestra.run.smithi082.stderr: 93.5% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-12T06:09:20.364 INFO:teuthology.orchestra.run.smithi076.stderr: 90.6% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-12T06:09:20.545 INFO:teuthology.orchestra.run.smithi088.stderr: 93.9% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.8.log.gz 2024-03-12T06:09:20.600 INFO:teuthology.orchestra.run.smithi076.stderr: 93.9% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-12T06:09:20.647 INFO:teuthology.orchestra.run.smithi082.stderr: 95.3% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.h.log.gz 2024-03-12T06:09:20.685 INFO:teuthology.orchestra.run.smithi088.stderr: 94.2% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.11.log.gz 2024-03-12T06:09:20.710 INFO:teuthology.orchestra.run.smithi076.stderr: 93.9% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.9.log.gz 2024-03-12T06:09:20.853 INFO:teuthology.orchestra.run.smithi076.stderr: 94.0% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-12T06:09:20.949 INFO:teuthology.orchestra.run.smithi088.stderr: 93.7% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-12T06:09:20.982 INFO:teuthology.orchestra.run.smithi082.stderr: 94.0% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-12T06:09:20.999 INFO:teuthology.orchestra.run.smithi082.stderr: 93.7% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.10.log.gz 2024-03-12T06:09:21.034 INFO:teuthology.orchestra.run.smithi088.stderr: 93.6% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-12T06:09:21.050 INFO:teuthology.orchestra.run.smithi076.stderr: 94.1% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-12T06:09:21.052 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-12T06:09:21.052 INFO:teuthology.orchestra.run.smithi076.stderr:real 0m1.621s 2024-03-12T06:09:21.053 INFO:teuthology.orchestra.run.smithi076.stderr:user 0m6.447s 2024-03-12T06:09:21.053 INFO:teuthology.orchestra.run.smithi076.stderr:sys 0m0.279s 2024-03-12T06:09:21.224 INFO:teuthology.orchestra.run.smithi082.stderr: 94.1% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-12T06:09:22.069 INFO:teuthology.orchestra.run.smithi088.stderr: 94.6% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.f.log.gz 2024-03-12T06:09:22.693 INFO:teuthology.orchestra.run.smithi088.stderr: 94.8% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.i.log.gz 2024-03-12T06:09:22.695 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-03-12T06:09:22.695 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m3.260s 2024-03-12T06:09:22.695 INFO:teuthology.orchestra.run.smithi088.stderr:user 0m11.619s 2024-03-12T06:09:22.695 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m0.418s 2024-03-12T06:09:31.692 INFO:teuthology.orchestra.run.smithi082.stderr: 93.0% -- replaced with /var/log/ceph/9f336de8-e033-11ee-95c8-87774f69a715/ceph-mds.b.log.gz 2024-03-12T06:09:31.693 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-12T06:09:31.693 INFO:teuthology.orchestra.run.smithi082.stderr:real 0m12.258s 2024-03-12T06:09:31.694 INFO:teuthology.orchestra.run.smithi082.stderr:user 0m19.229s 2024-03-12T06:09:31.694 INFO:teuthology.orchestra.run.smithi082.stderr:sys 0m0.645s 2024-03-12T06:09:31.694 INFO:tasks.cephadm:Archiving logs... 2024-03-12T06:09:31.694 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400/remote/smithi076/log 2024-03-12T06:09:31.696 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T06:09:32.553 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400/remote/smithi082/log 2024-03-12T06:09:32.554 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T06:09:35.280 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400/remote/smithi088/log 2024-03-12T06:09:35.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T06:09:36.705 INFO:tasks.cephadm:Removing cluster... 2024-03-12T06:09:36.706 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9f336de8-e033-11ee-95c8-87774f69a715 --force 2024-03-12T06:09:36.893 INFO:teuthology.orchestra.run.smithi076.stdout:Deleting cluster with fsid: 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T06:09:37.154 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9f336de8-e033-11ee-95c8-87774f69a715 --force 2024-03-12T06:09:37.345 INFO:teuthology.orchestra.run.smithi082.stdout:Deleting cluster with fsid: 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T06:09:37.613 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9f336de8-e033-11ee-95c8-87774f69a715 --force 2024-03-12T06:09:37.808 INFO:teuthology.orchestra.run.smithi088.stdout:Deleting cluster with fsid: 9f336de8-e033-11ee-95c8-87774f69a715 2024-03-12T06:09:38.072 INFO:tasks.cephadm:Removing cephadm ... 2024-03-12T06:09:38.073 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T06:09:38.091 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T06:09:38.107 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T06:09:38.121 INFO:tasks.cephadm:Teardown complete 2024-03-12T06:09:38.121 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-12T06:09:38.132 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-12T06:09:38.133 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-12T06:09:38.135 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-12T06:09:38.149 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-12T06:09:38.267 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-12T06:09:38.277 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-12T06:09:38.277 DEBUG:teuthology.orchestra.run.smithi076:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T06:09:38.280 DEBUG:teuthology.orchestra.run.smithi082:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T06:09:38.282 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T06:09:38.294 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-03-12T06:09:38.294 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-03-12T06:09:38.296 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2024-03-12T06:09:38.299 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T06:09:38.299 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-03-12T06:09:38.299 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 +1562us[+1626us] +/- 98ms 2024-03-12T06:09:38.299 INFO:teuthology.orchestra.run.smithi082.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -1269us[-1205us] +/- 64ms 2024-03-12T06:09:38.299 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +326us[ +391us] +/- 62ms 2024-03-12T06:09:38.299 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T06:09:38.300 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T06:09:38.300 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-03-12T06:09:38.301 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +1626us[+1626us] +/- 98ms 2024-03-12T06:09:38.301 INFO:teuthology.orchestra.run.smithi076.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -1210us[-1195us] +/- 64ms 2024-03-12T06:09:38.301 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +422us[ +436us] +/- 62ms 2024-03-12T06:09:38.301 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T06:09:38.302 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T06:09:38.302 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-03-12T06:09:38.302 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +1648us[+1648us] +/- 98ms 2024-03-12T06:09:38.302 INFO:teuthology.orchestra.run.smithi088.stdout:^* hv02.front.sepia.ceph.com 3 7 377 92 -1180us[-1097us] +/- 63ms 2024-03-12T06:09:38.302 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +422us[ +422us] +/- 62ms 2024-03-12T06:09:38.303 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T06:09:38.303 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-12T06:09:38.313 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-12T06:09:38.313 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-12T06:09:38.324 DEBUG:teuthology.orchestra.run.smithi076:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T06:09:38.429 DEBUG:teuthology.orchestra.run.smithi082:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T06:09:38.501 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T06:09:38.573 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T06:09:38.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T06:09:38.605 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T06:09:38.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T06:09:38.635 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T06:09:38.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T06:09:38.665 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-12T06:09:38.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-12T06:09:38.715 INFO:teuthology.task.internal:Duration was 2140.444238 seconds 2024-03-12T06:09:38.716 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-12T06:09:38.726 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-12T06:09:38.727 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T06:09:38.729 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T06:09:38.732 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T06:09:38.771 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T06:09:38.772 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T06:09:38.774 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T06:09:39.226 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-12T06:09:39.226 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi076.front.sepia.ceph.com 2024-03-12T06:09:39.227 DEBUG:teuthology.orchestra.run.smithi076:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T06:09:39.247 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2024-03-12T06:09:39.247 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T06:09:39.289 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2024-03-12T06:09:39.290 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T06:09:39.316 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-12T06:09:39.317 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T06:09:39.319 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T06:09:39.332 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T06:09:39.385 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-12T06:09:39.386 DEBUG:teuthology.orchestra.run.smithi076:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T06:09:39.389 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T06:09:39.401 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T06:09:39.766 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-12T06:09:39.778 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-12T06:09:39.778 DEBUG:teuthology.orchestra.run.smithi076:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T06:09:39.805 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T06:09:39.832 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T06:09:39.859 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-12T06:09:39.871 DEBUG:teuthology.orchestra.run.smithi076:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T06:09:39.873 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T06:09:39.876 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T06:09:39.897 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = core 2024-03-12T06:09:39.898 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2024-03-12T06:09:39.925 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2024-03-12T06:09:39.944 DEBUG:teuthology.orchestra.run.smithi076:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T06:09:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T06:09:39.967 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T06:09:39.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T06:09:39.981 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T06:09:39.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T06:09:39.998 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-12T06:09:40.009 INFO:teuthology.task.internal:Transferring archived files... 2024-03-12T06:09:40.010 DEBUG:teuthology.misc:Transferring archived files from smithi076:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400/remote/smithi076 2024-03-12T06:09:40.010 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T06:09:40.054 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400/remote/smithi082 2024-03-12T06:09:40.054 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T06:09:40.092 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400/remote/smithi088 2024-03-12T06:09:40.092 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T06:09:40.130 INFO:teuthology.task.internal:Removing archive directory... 2024-03-12T06:09:40.131 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T06:09:40.133 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T06:09:40.136 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T06:09:40.179 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-12T06:09:40.192 INFO:teuthology.task.internal:Not uploading archives. 2024-03-12T06:09:40.192 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-12T06:09:40.203 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-12T06:09:40.204 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T06:09:40.206 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T06:09:40.208 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T06:09:40.218 INFO:teuthology.orchestra.run.smithi076.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 06:09 /home/ubuntu/cephtest 2024-03-12T06:09:40.220 INFO:teuthology.orchestra.run.smithi082.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 06:09 /home/ubuntu/cephtest 2024-03-12T06:09:40.233 INFO:teuthology.orchestra.run.smithi088.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 06:09 /home/ubuntu/cephtest 2024-03-12T06:09:40.234 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-12T06:09:40.246 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-12T06:09:40.270 INFO:teuthology.nuke:Checking targets against current locks 2024-03-12T06:09:40.294 DEBUG:teuthology.nuke:shortname: smithi076 2024-03-12T06:09:40.295 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T06:09:40.320 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 05:21:34.347497', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBqka8M8jzDWF3C2lqo7awi+U11UcjBC8Zs7qH3YtWdp+1zsFMhetUtBoJIewgFOT9yQ6NU4TTrYHmpoQYmfUMg='} 2024-03-12T06:09:40.321 DEBUG:teuthology.nuke:shortname: smithi082 2024-03-12T06:09:40.321 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T06:09:40.348 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 05:21:34.349469', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpcbTekqurm/P/LdvPVYPk/tK/Ff68jezglLEGdGJKJ+GszihZUSUNetFE2IBnaSh5KjxJMRvGf7TC69MVdLIQ='} 2024-03-12T06:09:40.349 DEBUG:teuthology.nuke:shortname: smithi088 2024-03-12T06:09:40.349 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T06:09:40.372 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591400', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 05:21:34.351240', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMT5r1Iy6rqvFFcdWQB5pSnaz720cyku351a7vLpZdiM2w8akjnZYm6UE5UWSAv9nZOOznlQsnPrCEu85hCYyhI='} 2024-03-12T06:09:40.383 INFO:teuthology.orchestra.console.smithi076:Power off 2024-03-12T06:09:40.384 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T06:09:40.408 DEBUG:teuthology.orchestra.console.smithi076:power off output: Chassis Power Control: Down/Off 2024-03-12T06:09:40.408 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T06:09:40.425 INFO:teuthology.orchestra.console.smithi082:Power off 2024-03-12T06:09:40.425 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T06:09:40.440 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-03-12T06:09:40.451 DEBUG:teuthology.orchestra.console.smithi082:power off output: Chassis Power Control: Down/Off 2024-03-12T06:09:40.452 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T06:09:40.467 INFO:teuthology.orchestra.console.smithi088:Power off 2024-03-12T06:09:40.468 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T06:09:40.477 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-03-12T06:09:40.489 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off 2024-03-12T06:09:40.490 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T06:09:40.513 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-03-12T06:09:44.441 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T06:09:44.478 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T06:09:44.514 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T06:09:44.552 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-03-12T06:09:44.589 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-03-12T06:09:44.626 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-03-12T06:09:48.554 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T06:09:48.590 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T06:09:48.626 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T06:09:48.666 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off 2024-03-12T06:09:48.702 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-03-12T06:09:48.738 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off 2024-03-12T06:09:48.768 INFO:teuthology.orchestra.console.smithi076:Power off completed 2024-03-12T06:09:48.840 INFO:teuthology.orchestra.console.smithi088:Power off completed 2024-03-12T06:09:52.706 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T06:09:52.820 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off 2024-03-12T06:09:52.920 INFO:teuthology.orchestra.console.smithi082:Power off completed 2024-03-12T06:09:53.027 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} duration: 2140.444237947464 failure_reason: 'Command failed (workunit test suites/fsx.sh) on smithi076 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6a4997be2b8bb74411268cc42753f29ae9a78ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=33bc5d8b5898496ea0bbd3125e311c19 status: fail success: false 2024-03-12T06:09:53.028 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T06:09:53.115 INFO:teuthology.run:FAIL